[GH-ISSUE #252] [Feature] Integrate Budget Forcing as a Test-Time Compute Control Mechanism Prior to GRPO Execution

bsakaguchi commented

2026-10-26 21:16:25 +00:00

Owner

Originally created by @novvoo on GitHub (Feb 4, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/252

To enhance the efficiency and adaptability of DeepSeek R1’s GRPO (Group Relative Policy Optimization) algorithm, we propose integrating Budget Forcing—a test-time compute control method introduced in the Simple Test-Time Scaling framework (arXiv:2501.19393v1#S3)26. This addition aims to dynamically adjust the model’s reasoning process during inference, aligning computational expenditure with predefined budgets while improving performance on resource-constrained tasks like mathematical reasoning and complex problem-solving.

Proposed Feature Details

Budget Forcing Mechanism
- Dynamic Compute Allocation:
  Budget Forcing operates by either:
  - Terminating Early: If the model exceeds a token limit (e.g., MAX_TOKENS_THINKING=32000), append an end-of-thinking token to force an answer.
  - Extending Reasoning: If the model attempts to terminate prematurely, suppress the end token and append "Wait" to encourage further exploration, effectively increasing test-time compute.
- User-Defined Budget Parameters: Allow users to specify computational constraints (e.g., token limits, FLOPs) as input, enabling fine-grained control over inference costs.
Integration with GRPO
- Pre-GRPO Application: Apply Budget Forcing during the inference phase before GRPO’s policy optimization. This ensures that GRPO’s group-based reward calculations (which rely on multiple candidate solutions) are performed under controlled computational conditions.
- Compatibility with Group Sampling: GRPO generates multiple responses per query for relative reward evaluation. Budget Forcing can be applied to each candidate solution individually, ensuring resource limits per output while maintaining group diversity.

Key Benefits

Enhanced Resource Efficiency:
- Prioritize critical reasoning steps by terminating low-value computations early, reducing wasted resources.
- Achieve up to 27% performance gains on MATH and AIME24 benchmarks when paired with GRPO, as shown in the s1-32B model.
Improved Robustness:
- Force models to "double-check" answers via extended reasoning (e.g., appending "Wait"), correcting errors in intermediate steps.
Scalability Across Environments:
- Adapt seamlessly to high- and low-resource settings, making GRPO-based models more practical for real-world deployments.

Implementation Steps

Parameterization:
- Introduce budget_mode (e.g., strict, flexible) and max_thinking_tokens as configurable hyperparameters.
Token Management:
- Modify the generation loop to track token counts and enforce termination/extension based on budget rules. Code snippets from the s1 GitHub repo demonstrate how to handle stop_token_ids and append "Wait" tokens dynamically.

GRPO Workflow Adjustment:

During GRPO’s group sampling phase, apply Budget Forcing to each candidate solution. For example:

# Pseudocode for GRPO + Budget Forcing integration  
for query in batch:  
    candidates = generate_candidates(query, num_samples=G)  
    for candidate in candidates:  
        apply_budget_forcing(candidate, max_tokens=32000)  
    rewards = evaluate_candidates(candidates)  
    update_policy_using_grpo(candidates, rewards)

Validation Metrics:
- Measure performance gains (e.g., accuracy on MATH benchmarks) and compute savings (e.g., average tokens per query) to validate efficacy.

Expected Outcomes

Performance-Resource Trade-off Optimization:
- Enable users to balance accuracy and computational cost, as demonstrated by s1-32B’s improvement from 50% → 57% on AIME24 with extended compute.
Broader Applicability:
- Extend GRPO’s utility to scenarios requiring strict resource constraints (e.g., edge devices, real-time systems).

References to Code/Data:

Budget Forcing implementation: GitHub/simplescaling/s1
GRPO algorithm details: DeepSeekMath Paper

Originally created by @novvoo on GitHub (Feb 4, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/252 To enhance the efficiency and adaptability of DeepSeek R1’s GRPO (Group Relative Policy Optimization) algorithm, we propose integrating Budget Forcing—a test-time compute control method introduced in the Simple Test-Time Scaling framework ([arXiv:2501.19393v1#S3](https://arxiv.org/html/2501.19393v1#S3))26. This addition aims to dynamically adjust the model’s reasoning process during inference, aligning computational expenditure with predefined budgets while improving performance on resource-constrained tasks like mathematical reasoning and complex problem-solving. ### **Proposed Feature Details** 1. **Budget Forcing Mechanism** - **Dynamic Compute Allocation:** Budget Forcing operates by either: - **Terminating Early:** If the model exceeds a token limit (e.g., `MAX_TOKENS_THINKING=32000`), append an end-of-thinking token to force an answer. - **Extending Reasoning:** If the model attempts to terminate prematurely, suppress the end token and append "Wait" to encourage further exploration, effectively increasing test-time compute. - **User-Defined Budget Parameters:** Allow users to specify computational constraints (e.g., token limits, FLOPs) as input, enabling fine-grained control over inference costs. 2. **Integration with GRPO** - **Pre-GRPO Application:** Apply Budget Forcing during the **inference phase** before GRPO’s policy optimization. This ensures that GRPO’s group-based reward calculations (which rely on multiple candidate solutions) are performed under controlled computational conditions. - **Compatibility with Group Sampling:** GRPO generates multiple responses per query for relative reward evaluation. Budget Forcing can be applied to each candidate solution individually, ensuring resource limits per output while maintaining group diversity. --- ### **Key Benefits** 1. **Enhanced Resource Efficiency:** - Prioritize critical reasoning steps by terminating low-value computations early, reducing wasted resources. - Achieve up to **27% performance gains** on MATH and AIME24 benchmarks when paired with GRPO, as shown in the s1-32B model. 2. **Improved Robustness:** - Force models to "double-check" answers via extended reasoning (e.g., appending "Wait"), correcting errors in intermediate steps. 3. **Scalability Across Environments:** - Adapt seamlessly to high- and low-resource settings, making GRPO-based models more practical for real-world deployments. --- ### **Implementation Steps** 1. **Parameterization:** - Introduce `budget_mode` (e.g., `strict`, `flexible`) and `max_thinking_tokens` as configurable hyperparameters. 2. **Token Management:** - Modify the generation loop to track token counts and enforce termination/extension based on budget rules. Code snippets from the [s1 GitHub repo](https://github.com/simplescaling/s1) demonstrate how to handle `stop_token_ids` and append "Wait" tokens dynamically. 3. **GRPO Workflow Adjustment:** - During GRPO’s group sampling phase, apply Budget Forcing to each candidate solution. For example: ```python # Pseudocode for GRPO + Budget Forcing integration for query in batch: candidates = generate_candidates(query, num_samples=G) for candidate in candidates: apply_budget_forcing(candidate, max_tokens=32000) rewards = evaluate_candidates(candidates) update_policy_using_grpo(candidates, rewards) ``` 4. **Validation Metrics:** - Measure performance gains (e.g., accuracy on MATH benchmarks) and compute savings (e.g., average tokens per query) to validate efficacy. --- ### **Expected Outcomes** 1. **Performance-Resource Trade-off Optimization:** - Enable users to balance accuracy and computational cost, as demonstrated by s1-32B’s improvement from 50% → 57% on AIME24 with extended compute. 2. **Broader Applicability:** - Extend GRPO’s utility to scenarios requiring strict resource constraints (e.g., edge devices, real-time systems). --- **References to Code/Data:** - Budget Forcing implementation: [GitHub/simplescaling/s1](https://github.com/simplescaling/s1) - GRPO algorithm details: [DeepSeekMath Paper](https://arxiv.org/abs/2402.03300)

bsakaguchi

2026-10-26 21:16:25 +00:00

closed this issue
added the
stale

closed-as-stale
labels

bsakaguchi commented

2026-10-26 21:16:27 +00:00

Author

Owner

@manoj633 commented on GitHub (Feb 4, 2025):

The current approach forces early termination if a threshold is exceeded. However, abrupt stopping might lead to incomplete reasoning. Introduce a confidence-based termination metric, where the model assesses its certainty before stopping (e.g., via entropy of token probabilities).

@manoj633 commented on GitHub (Feb 4, 2025): The current approach forces early termination if a threshold is exceeded. However, abrupt stopping might lead to incomplete reasoning. Introduce a confidence-based termination metric, where the model assesses its certainty before stopping (e.g., via entropy of token probabilities).

bsakaguchi commented

2026-10-26 21:16:27 +00:00

Author

Owner

@novvoo commented on GitHub (Feb 5, 2025):

The current approach forces early termination if a threshold is exceeded. However, abrupt stopping might lead to incomplete reasoning. Introduce a confidence-based termination metric, where the model assesses its certainty before stopping (e.g., via entropy of token probabilities).

Well, I agree that R1’s intelligent termination is indeed a better solution overall. In addition to that, I have two further suggestions that might help us leverage S1’s strengths while maintaining R1’s inherent flexibility:

Integrate S1's dynamic computation control:
We can adopt the method proposed by S1 to limit the computational budget for candidate answers before executing R1’s GRPO. This would ensure that the reasoning process remains optimal and within our designated resource limits.

Parameterization and adaptive adjustment:
Opening up user-defined budget parameters in R1 could provide greater flexibility. By experimenting with different strategies on benchmarks such as MATH and AIME24, we can verify the effects and maximize computational efficiency while ensuring performance improvements.

I look forward to these suggestions can further improve the intelligence level of R1.

@novvoo commented on GitHub (Feb 5, 2025): > The current approach forces early termination if a threshold is exceeded. However, abrupt stopping might lead to incomplete reasoning. Introduce a confidence-based termination metric, where the model assesses its certainty before stopping (e.g., via entropy of token probabilities). Well, I agree that R1’s intelligent termination is indeed a better solution overall. In addition to that, I have two further suggestions that might help us leverage S1’s strengths while maintaining R1’s inherent flexibility: Integrate S1's dynamic computation control: We can adopt the method proposed by S1 to limit the computational budget for candidate answers before executing R1’s GRPO. This would ensure that the reasoning process remains optimal and within our designated resource limits. Parameterization and adaptive adjustment: Opening up user-defined budget parameters in R1 could provide greater flexibility. By experimenting with different strategies on benchmarks such as MATH and AIME24, we can verify the effects and maximize computational efficiency while ensuring performance improvements. I look forward to these suggestions can further improve the intelligence level of R1.

bsakaguchi commented

2026-10-26 21:16:28 +00:00

Author

Owner

@novvoo commented on GitHub (Feb 6, 2025):

S1's force stopping is being hotly discussed now, so this isuue still has its meaning
https://news.ycombinator.com/item?id=42946854

@novvoo commented on GitHub (Feb 6, 2025): S1's force stopping is being hotly discussed now, so this isuue still has its meaning https://news.ycombinator.com/item?id=42946854

bsakaguchi commented

2026-10-26 21:16:28 +00:00

Author

Owner

@Codys12 commented on GitHub (Feb 8, 2025):

I feel as if this can be aproached during training by factoring in Sequence Length into GRPO rewards

@Codys12 commented on GitHub (Feb 8, 2025): I feel as if this can be aproached during training by factoring in Sequence Length into GRPO rewards

bsakaguchi commented

2026-10-26 21:16:28 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Apr 1, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

@github-actions[bot] commented on GitHub (Apr 1, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

bsakaguchi commented

2026-10-26 21:16:28 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (May 8, 2025):

false

@github-actions[bot] commented on GitHub (May 8, 2025): false

Rows
Columns

[GH-ISSUE #252] [Feature] Integrate Budget Forcing as a Test-Time Compute Control Mechanism Prior to GRPO Execution #184

Proposed Feature Details

Key Benefits

Implementation Steps

Expected Outcomes