REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization — mentioned by bradhilton

Thu, 06 Mar 2025 22:05:22 GMT

Yeah, it may help. In this paper[1], the author used a KL penalty of 0.01 for general tasks and 0.001 for mathematical. I tend to think it's probably not very important unless you're trying to optimize for human preferences. As for response length, I think the model internalizes the logic and doesn't deliberate its answers through context creation. I don't think this is necessarily good for general reasoning, but for a specific task it would cut down inference costs. Just depends on what you're optimizing for. To encourage more general reasoning, I think a broader train and validation set…

Paper and discussion history. A mention is not an endorsement.

REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization · HackerPapers discussions

REINFORCE++: Stabilizing Critic-Free Policy Optimization with Global Advantage Normalization — mentioned by bradhilton