[GH-ISSUE #236] Can RLHF even simpler to maximize the expectation of rewards? #69

New issue

Closed

opened 2026-10-26 21:12:50 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-26 21:12:50 +00:00

Owner

Originally created by @kindernerd on GitHub (Jan 7, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V3/issues/236

GRPO simplifies advantage to (r-mean)/std, i'm wondering whether RLHF can even be simpler by directly maximum the following objective:
\sum_o\pi_{\theta}(o|q)[r_o - E(r_o|q)]
which can be approximated by sampling or using the N-best Lists
\sum_{o_i\in \pi_{old}}\pi_{\theta}(o_i|q)[r_{o_i} -mean(r)]
this is similar to sequence training (MWER) in e2e asr optimization, proposed by google in this paper https://arxiv.org/abs/1712.01818

Originally created by @kindernerd on GitHub (Jan 7, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V3/issues/236 GRPO simplifies advantage to (r-mean)/std, i'm wondering whether RLHF can even be simpler by directly maximum the following objective: $\sum_o\pi_{\theta}(o|q)[r_o - E(r_o|q)]$ which can be approximated by sampling or using the N-best Lists $\sum_{o_i\in \pi_{old}}\pi_{\theta}(o_i|q)[r_{o_i} -mean(r)]$ this is similar to sequence training (MWER) in e2e asr optimization, proposed by google in this paper https://arxiv.org/abs/1712.01818

bsakaguchi

2026-10-26 21:12:50 +00:00

closed this issue
added the
stale

closed-as-stale
labels

bsakaguchi commented

2026-10-26 21:12:52 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Feb 8, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

@github-actions[bot] commented on GitHub (Feb 8, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

bsakaguchi commented

2026-10-26 21:12:53 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Feb 24, 2025):

false

@github-actions[bot] commented on GitHub (Feb 24, 2025): false