Momentum-Based Policy Gradient with Second-Order Information · HackerPapers discussions

Momentum-Based Policy Gradient with Second-Order Information — mentioned by namibj

Sun, 07 Apr 2024 12:24:55 GMT

I wonder how it relates/compares to https://arxiv.org/abs/2205.08253 : Title: Momentum-Based Policy Gradient with Second-Order Information Abstract: Variance-reduced gradient estimators for policy gradient methods have been one of the main focus of research in the reinforcement learning in recent years as they allow acceleration of the estimation process. We propose a variance-reduced policy-gradient method, called SHARP, which incorporates second-order information into stochastic gradient descent (SGD) using momentum with a time-varying learning rate. SHARP algorithm is parameter-free…

Paper and discussion history. A mention is not an endorsement.

Momentum-Based Policy Gradient with Second-Order Information — mentioned by namibj

Sat, 02 Dec 2023 18:12:18 GMT

Actually, selectively deleting facts like that does kinda work, essentially what you're doing is scolding-based reinforcement learning. For decent speed you'd want to probably use variance-reduction techniques like https://arxiv.org/abs/2205.08253 , though. Or a related (would be linked as cited by/cited in the linked when you look on semantic scholar) method that doesn't compute a hessian-vector product (which apparently interacts poorly with the non-linearity of the multi-layer perceptrons as provided by LLM-typical ReLU), in exchange for e.g. sizable batching (gradient accumulation!) at…

Paper and discussion history. A mention is not an endorsement.