[GH-ISSUE #458] How to transfer 4k context Qwen2.5-Math-7B to 128K context DeepSeek-R1-Distill-Qwen-7B #366

New issue

Closed

opened 2026-10-26 21:20:08 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-26 21:20:08 +00:00

Owner

Originally created by @yanyc428 on GitHub (Feb 19, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/458

We note that Qwen2.5-Math-7B is a model with a 4K context window, while DeepSeek-R1-Distill-Qwen-7B is a model with a 128K context window.
We want to know how the expansion from 4K to 128K was achieved. Was it by changing the max_position_embeddings from 4K to 128K?

Originally created by @yanyc428 on GitHub (Feb 19, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/458 We note that Qwen2.5-Math-7B is a model with a 4K context window, while DeepSeek-R1-Distill-Qwen-7B is a model with a 128K context window. We want to know how the expansion from 4K to 128K was achieved. Was it by changing the `max_position_embeddings` from 4K to 128K?

bsakaguchi

2026-10-26 21:20:08 +00:00

closed this issue
added the
stale

closed-as-stale
labels

bsakaguchi commented

2026-10-26 21:20:09 +00:00

Author

Owner

@Jeremmmyyyyy commented on GitHub (Feb 27, 2025):

I have the same question. I tried to change the settings in the config.json like this :

{
  "_name_or_path": "Qwen/Qwen2.5-Math-7B-Instruct",
  "architectures": [
    "Qwen2ForCausalLM"
  ],
  "attention_dropout": 0.0,
  "bos_token_id": 151643,
  "eos_token_id": 151645,
  "hidden_act": "silu",
  "hidden_size": 3584,
  "initializer_range": 0.02,
  "intermediate_size": 18944,
  "max_position_embeddings": 32768,
  "max_window_layers": 28,
  "model_type": "qwen2",
  "num_attention_heads": 28,
  "num_hidden_layers": 28,
  "num_key_value_heads": 4,
  "rms_norm_eps": 1e-06,
    "rope_scaling": {
    "type": "linear",
    "factor": 8.0
  },
  "rope_theta": 10000.0,
  "sliding_window": null,
  "tie_word_embeddings": false,
  "torch_dtype": "bfloat16",
  "transformers_version": "4.48.1",
  "use_cache": true,
  "use_sliding_window": false,
  "vocab_size": 152064
}

But this resulted in poor generations. I read that they changed the "RoPE frequency to 300k" but wasn't able to find any of these settings

@Jeremmmyyyyy commented on GitHub (Feb 27, 2025): I have the same question. I tried to change the settings in the config.json like this : ``` { "_name_or_path": "Qwen/Qwen2.5-Math-7B-Instruct", "architectures": [ "Qwen2ForCausalLM" ], "attention_dropout": 0.0, "bos_token_id": 151643, "eos_token_id": 151645, "hidden_act": "silu", "hidden_size": 3584, "initializer_range": 0.02, "intermediate_size": 18944, "max_position_embeddings": 32768, "max_window_layers": 28, "model_type": "qwen2", "num_attention_heads": 28, "num_hidden_layers": 28, "num_key_value_heads": 4, "rms_norm_eps": 1e-06, "rope_scaling": { "type": "linear", "factor": 8.0 }, "rope_theta": 10000.0, "sliding_window": null, "tie_word_embeddings": false, "torch_dtype": "bfloat16", "transformers_version": "4.48.1", "use_cache": true, "use_sliding_window": false, "vocab_size": 152064 } ``` But this resulted in poor generations. I read that they changed the "RoPE frequency to 300k" but wasn't able to find any of these settings

bsakaguchi commented

2026-10-26 21:20:09 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Apr 23, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

@github-actions[bot] commented on GitHub (Apr 23, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

bsakaguchi commented

2026-10-26 21:20:09 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (May 28, 2025):

false

@github-actions[bot] commented on GitHub (May 28, 2025): false