deepseek-ai/DeepSeek-R1
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-R1.git synced 2026-11-05 00:47:42 +00:00
Code Issues 45 Projects Packages Wiki Activity

[GH-ISSUE #684] Unrecognized keys in rope_scaling for 'rope_type'='yarn': {'attn_factor'} #558

New issue
Closed
opened 2026-10-26 21:23:08 +00:00 by bsakaguchi · 10 comments
bsakaguchi commented 2026-10-26 21:23:08 +00:00
Owner
Copy link

Originally created by @Jintao-Huang on GitHub (Jun 12, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/684

哈咯，我使用deepseek-ai/DeepSeek-R1-0528-Qwen3-8B

但是遇到了下面的警告信息：
"Unrecognized keys in rope_scaling for 'rope_type'='yarn': {'attn_factor'}"

我查看了transformers对于yarn的实现

github.com/huggingface/transformers@7c58336949/src/transformers/modeling_rope_utils.py (L256)

是否config.json中的rope_scaling参数有误，attn_factor -> attention_factor

使用transformers计算的attention_factor为：

Image

而默认值为 "attn_factor": 0.8782488562869419

这是否导致性能下降，谢谢

Originally created by @Jintao-Huang on GitHub (Jun 12, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/684 哈咯，我使用deepseek-ai/DeepSeek-R1-0528-Qwen3-8B 但是遇到了下面的警告信息： "Unrecognized keys in `rope_scaling` for 'rope_type'='yarn': {'attn_factor'}" 我查看了transformers对于yarn的实现 https://github.com/huggingface/transformers/blob/7c5833694934c289e4297f50417d422a893c9d5d/src/transformers/modeling_rope_utils.py#L256 是否config.json中的rope_scaling参数有误，attn_factor -> attention_factor 使用transformers计算的attention_factor为： <img width="169" alt="Image" src="https://github.com/user-attachments/assets/4b6931e3-929f-4450-8d07-7a72c8e7fba5" /> 而默认值为 "attn_factor": 0.8782488562869419 这是否导致性能下降，谢谢
bsakaguchi 2026-10-26 21:23:08 +00:00
bsakaguchi commented 2026-10-26 21:23:09 +00:00
Author
Owner
Copy link

@OCEANOUXIN commented on GitHub (Jun 26, 2025):

哈喽，你是这么解决这个问题的，我也遇到了这个问题

<!-- gh-comment-id:3007802865 --> @OCEANOUXIN commented on GitHub (Jun 26, 2025): 哈喽，你是这么解决这个问题的，我也遇到了这个问题
bsakaguchi commented 2026-10-26 21:23:09 +00:00
Author
Owner
Copy link

@HBin013 commented on GitHub (Jul 23, 2025):

你好，请问你有做实验测试这个设置的影响吗？

Hello! Have you ever done any experiments to verify that?

<!-- gh-comment-id:3106608226 --> @HBin013 commented on GitHub (Jul 23, 2025): 你好，请问你有做实验测试这个设置的影响吗？ Hello! Have you ever done any experiments to verify that?
bsakaguchi commented 2026-10-26 21:23:09 +00:00
Author
Owner
Copy link

@yangysc commented on GitHub (Aug 11, 2025):

Yes, I also found this issue. Hope someone can helpfully clarify this!

<!-- gh-comment-id:3173271559 --> @yangysc commented on GitHub (Aug 11, 2025): Yes, I also found this issue. Hope someone can helpfully clarify this!
bsakaguchi commented 2026-10-26 21:23:10 +00:00
Author
Owner
Copy link

@HBin013 commented on GitHub (Aug 11, 2025):

Yes, I also found this issue. Hope someone can helpfully clarify this!

I've tested different settings on AIME24. It turned out that the original config was right.

<!-- gh-comment-id:3174684761 --> @HBin013 commented on GitHub (Aug 11, 2025): > Yes, I also found this issue. Hope someone can helpfully clarify this! I've tested different settings on AIME24. It turned out that the original config was right.
bsakaguchi commented 2026-10-26 21:23:10 +00:00
Author
Owner
Copy link

@yangysc commented on GitHub (Aug 12, 2025):

Yes, I also found this issue. Hope someone can helpfully clarify this!

I've tested different settings on AIME24. It turned out that the original config was right.

Thanks. By "original config," do you mean the attention_factor in Transformers or the rope_scaling setting in config.json?

<!-- gh-comment-id:3177337810 --> @yangysc commented on GitHub (Aug 12, 2025): > > Yes, I also found this issue. Hope someone can helpfully clarify this! > > I've tested different settings on AIME24. It turned out that the original config was right. Thanks. By "original config," do you mean the attention_factor in Transformers or the rope_scaling setting in config.json?
bsakaguchi commented 2026-10-26 21:23:10 +00:00
Author
Owner
Copy link

@HBin013 commented on GitHub (Aug 12, 2025):

Yes, I also found this issue. Hope someone can helpfully clarify this!

I've tested different settings on AIME24. It turned out that the original config was right.

Thanks. By "original config," do you mean the attention_factor in Transformers or the rope_scaling setting in config.json?

Sorry that I didn't make myself clear. I mean the original settings in the official config.json.

  "rope_scaling": {
    "rope_type": "yarn",
    "factor": 4.0,
    "original_max_position_embeddings": 32768,
    "attn_factor": 0.8782488562869419
  },

With this setting I reproduced the results from the official technical report on AIME25, and a slightly lower result on AIME24:

AIME24 w/ original: 81.98 (averaged over 32 runs)
AIME25 w/ original: 74.69 (averaged over 32 runs)

Settings mentioned in this issue:

"attention_factor": 0.8782488562869419

or

"attention_factor": 1.138629436111989

caused a significant performance drop:

AIME24: 0.7354/ 0.7479 (both averaged over 16 runs)
AIME25: 0.6500/ 0.6687 (both averaged over 16 runs)

I am not familiar with rope scaling. Based on my experimental results, I suppose the original setup is correct.

Hope it helps.

<!-- gh-comment-id:3179020383 --> @HBin013 commented on GitHub (Aug 12, 2025): > > > Yes, I also found this issue. Hope someone can helpfully clarify this! > > > > > > I've tested different settings on AIME24. It turned out that the original config was right. > > Thanks. By "original config," do you mean the attention_factor in Transformers or the rope_scaling setting in config.json? Sorry that I didn't make myself clear. I mean the original settings in the official `config.json`. ```json "rope_scaling": { "rope_type": "yarn", "factor": 4.0, "original_max_position_embeddings": 32768, "attn_factor": 0.8782488562869419 }, ``` With this setting I reproduced the results from the official technical report on AIME25, and a slightly lower result on AIME24: ``` AIME24 w/ original: 81.98 (averaged over 32 runs) AIME25 w/ original: 74.69 (averaged over 32 runs) ``` Settings mentioned in this issue: ``` "attention_factor": 0.8782488562869419 ``` or ``` "attention_factor": 1.138629436111989 ``` caused a significant performance drop: ``` AIME24: 0.7354/ 0.7479 (both averaged over 16 runs) AIME25: 0.6500/ 0.6687 (both averaged over 16 runs) ``` I am not familiar with rope scaling. Based on my experimental results, I suppose the original setup is correct. Hope it helps.
bsakaguchi commented 2026-10-26 21:23:10 +00:00
Author
Owner
Copy link

@yangysc commented on GitHub (Aug 12, 2025):

Yes, I also found this issue. Hope someone can helpfully clarify this!

I've tested different settings on AIME24. It turned out that the original config was right.

Thanks. By "original config," do you mean the attention_factor in Transformers or the rope_scaling setting in config.json?

Sorry that I didn't make myself clear. I mean the original settings in the official config.json.

"rope_scaling": {
"rope_type": "yarn",
"factor": 4.0,
"original_max_position_embeddings": 32768,
"attn_factor": 0.8782488562869419
},
With this setting I reproduced the results from the official technical report on AIME25, and a slightly lower result on AIME24:

AIME24 w/ original: 81.98 (averaged over 32 runs)
AIME25 w/ original: 74.69 (averaged over 32 runs)

Settings mentioned in this issue:

"attention_factor": 0.8782488562869419

or

"attention_factor": 1.138629436111989

caused a significant performance drop:

AIME24: 0.7354/ 0.7479 (both averaged over 16 runs)
AIME25: 0.6500/ 0.6687 (both averaged over 16 runs)

I am not familiar with rope scaling. Based on my experimental results, I suppose the original setup is correct.

Hope it helps.

Thanks very much for your detailed explanation! So maybe we can safely ignore the warning Unrecognized keys in rope_scaling for 'rope_type'='yarn': {'attn_factor'}

<!-- gh-comment-id:3179213870 --> @yangysc commented on GitHub (Aug 12, 2025): > > > > Yes, I also found this issue. Hope someone can helpfully clarify this! > > > > > > > > > I've tested different settings on AIME24. It turned out that the original config was right. > > > > > > Thanks. By "original config," do you mean the attention_factor in Transformers or the rope_scaling setting in config.json? > > Sorry that I didn't make myself clear. I mean the original settings in the official `config.json`. > > "rope_scaling": { > "rope_type": "yarn", > "factor": 4.0, > "original_max_position_embeddings": 32768, > "attn_factor": 0.8782488562869419 > }, > With this setting I reproduced the results from the official technical report on AIME25, and a slightly lower result on AIME24: > > ``` > AIME24 w/ original: 81.98 (averaged over 32 runs) > AIME25 w/ original: 74.69 (averaged over 32 runs) > ``` > > Settings mentioned in this issue: > > ``` > "attention_factor": 0.8782488562869419 > ``` > > or > > ``` > "attention_factor": 1.138629436111989 > ``` > > caused a significant performance drop: > > ``` > AIME24: 0.7354/ 0.7479 (both averaged over 16 runs) > AIME25: 0.6500/ 0.6687 (both averaged over 16 runs) > ``` > > I am not familiar with rope scaling. Based on my experimental results, I suppose the original setup is correct. > > Hope it helps. Thanks very much for your detailed explanation! So maybe we can safely ignore the warning `Unrecognized keys in rope_scaling for 'rope_type'='yarn': {'attn_factor'}`
bsakaguchi commented 2026-10-26 21:23:10 +00:00
Author
Owner
Copy link

@Insanexiao commented on GitHub (Aug 15, 2025):

我也遇到这个问题了，我看了你们的讨论，这意思是不是直接忽略掉这个警告就可以了？

<!-- gh-comment-id:3192438909 --> @Insanexiao commented on GitHub (Aug 15, 2025): 我也遇到这个问题了，我看了你们的讨论，这意思是不是直接忽略掉这个警告就可以了？
bsakaguchi commented 2026-10-26 21:23:10 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Sep 16, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

<!-- gh-comment-id:3294392841 --> @github-actions[bot] commented on GitHub (Sep 16, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
bsakaguchi commented 2026-10-26 21:23:11 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Sep 30, 2025):

false

<!-- gh-comment-id:3349457992 --> @github-actions[bot] commented on GitHub (Sep 30, 2025): false
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
v1.0.0
Labels
Clear labels   
bug

   
closed-as-stale

   
closed-as-stale

   
enhancement

   
pull-request
Mirrored from GitHub Pull Request

  
stale

   
stale
No labels
bug
closed-as-stale
closed-as-stale
enhancement
pull-request
stale
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepSeek-R1#558
No description provided.