deepseek-ai/DeepSeek-R1
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-R1.git synced 2026-11-05 00:47:42 +00:00
Code Issues 45 Projects Packages Wiki Activity

[GH-ISSUE #695] 分词器model_max_length为16384，而上下文最大131072 #569

New issue
Closed
opened 2026-10-26 21:23:19 +00:00 by bsakaguchi · 3 comments
bsakaguchi commented 2026-10-26 21:23:19 +00:00
Owner
Copy link

Originally created by @AlanYue2023 on GitHub (Jun 19, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/695

使用transformer加载deepseek-r1-32b的分词器，统计一段文本分词后token数量，遇到序列超过16384的报错，我看tokenizer_config.json里model_max_length:16384，但是deepseek不是支持131k序列长度吗，config.json里有"max_position_embeddings": 131072，请问可以修改model_max_length为131072吗？会不会有什么影响？

Image

Originally created by @AlanYue2023 on GitHub (Jun 19, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/695 使用transformer加载deepseek-r1-32b的分词器，统计一段文本分词后token数量，遇到序列超过16384的报错，我看tokenizer_config.json里model_max_length:16384，但是deepseek不是支持131k序列长度吗，config.json里有"max_position_embeddings": 131072，请问可以修改model_max_length为131072吗？会不会有什么影响？ ![Image](https://github.com/user-attachments/assets/5711ab50-f60c-4cd9-9668-86cd5748b5a0)
bsakaguchi 2026-10-26 21:23:19 +00:00
bsakaguchi commented 2026-10-26 21:23:20 +00:00
Author
Owner
Copy link

@AlanYue2023 commented on GitHub (Jun 19, 2025):

Image

Image

<!-- gh-comment-id:2988028644 --> @AlanYue2023 commented on GitHub (Jun 19, 2025): ![Image](https://github.com/user-attachments/assets/07127f46-8951-4d58-a2ea-d4da4468bf13) ![Image](https://github.com/user-attachments/assets/9b028663-2629-48e9-9fd4-cf8b4abac98c)
bsakaguchi commented 2026-10-26 21:23:20 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Jul 22, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

<!-- gh-comment-id:3100022981 --> @github-actions[bot] commented on GitHub (Jul 22, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
bsakaguchi commented 2026-10-26 21:23:20 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Aug 7, 2025):

false

<!-- gh-comment-id:3161994756 --> @github-actions[bot] commented on GitHub (Aug 7, 2025): false
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
v1.0.0
Labels
Clear labels   
bug

   
closed-as-stale

   
closed-as-stale

   
enhancement

   
pull-request
Mirrored from GitHub Pull Request

  
stale

   
stale
No labels
bug
closed-as-stale
closed-as-stale
enhancement
pull-request
stale
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepSeek-R1#569
No description provided.