[GH-ISSUE #938] Different rotary embedding implementation between this repo and Huggingface #567

New issue

Closed

opened 2026-10-26 21:23:20 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-26 21:23:20 +00:00

Owner

Originally created by @wwwjn on GitHub (Jul 26, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V3/issues/938

Describe the bug
Hi team, I'm working on reproducing the great deepseek-v3 model on torchtitan . While I'm trying to run numerical verification, I noticed the rotary embedding in HF and this repo is different.

HF: https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py#L339

In HF rotary embedding implementation, they explicitly permute (interleave the odd column and even columns) the q_pe / k_pe.

This repo: https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py#L339

In DeepSeek-AI's implementation, they didn't permute the q_pe / k_pe.
In the convert.py, the script load HF weights as it's original order, and this script didn't permute the weights, to accommodate the ordering difference in apply_rotary_embedding().

And this discrepancy will result in different mathematical results after the attention module of the first dense layer.

I want to double check with the team if I missed something here. Thank you for your help in advance! cc @tianyu-l

To Reproduce
Environment: transformers ==4.54.0

I run following 2 runs with code: https://github.com/wwwjn/DeepSeek-V3 . I randomized the same inputs for both runs.

a single forward pass using HuggingFace transformers, with weights from:https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/tree/main .
python hf_implementation/hf_implementation.py --num_layers 5 > hf_outputs.txt 2>&1
Using this repo's model implementation and run a single forward pass (To simplify, I didn't use the - First, I used the convert.py to convert HF checkpoint weights.
python convert.py --hf-ckpt-path /data/users/jianiw/dsv3-weights/ --save-path /data/users/jianiw/dsv3-weights-5-layer/ --n-experts 256 --model-parallel 8
- Second, I run a single forward pass using
  torchrun --nnodes 1 --nproc-per-node 8 inference/run_single_forward.py --config inference/configs/config_671B.json > dsv3-output.txt 2>&1

Here's the detailed numerical comparison I've seen:

Expected behavior
Expected behavior: After first dense layer's attention layer, the output should be almost the same (There might be slightly difference because of fp8 vs bfloat16 precision).

Screenshots
See screenshots above

Additional context
We observed the same issue for llama3 model before - The weights for llama3 on HuggingFace is permuted compared to original Meta's weights. so we need to manually permute the weights back to accommodate the rotary embedding implementation difference. Reference: https://github.com/pytorch/torchtitan/issues/335

Originally created by @wwwjn on GitHub (Jul 26, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V3/issues/938 **Describe the bug** Hi team, I'm working on reproducing the great deepseek-v3 model on [torchtitan](https://github.com/pytorch/torchtitan/tree/refs/heads/main) . While I'm trying to run numerical verification, I noticed the rotary embedding in HF and this repo is different. HF: https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py#L339 - In HF rotary embedding implementation, they explicitly permute (interleave the odd column and even columns) the q_pe / k_pe. This repo: https://huggingface.co/deepseek-ai/DeepSeek-V3/blob/main/modeling_deepseek.py#L339 - In DeepSeek-AI's implementation, they didn't permute the q_pe / k_pe. - In the `convert.py`, the script load HF weights as it's original order, and this script didn't permute the weights, to accommodate the ordering difference in `apply_rotary_embedding()`. And this discrepancy will result in different mathematical results after the attention module of the first dense layer. I want to double check with the team if I missed something here. Thank you for your help in advance! cc @tianyu-l **To Reproduce** Environment: transformers ==4.54.0 I run following 2 runs with code: https://github.com/wwwjn/DeepSeek-V3 . I randomized the same inputs for both runs. 1. a single forward pass using HuggingFace `transformers`, with weights from:https://huggingface.co/deepseek-ai/DeepSeek-V3-0324/tree/main . ```python hf_implementation/hf_implementation.py --num_layers 5 > hf_outputs.txt 2>&1``` 2. Using this repo's model implementation and run a single forward pass (To simplify, I didn't use the - First, I used the `convert.py` to convert HF checkpoint weights. ```python convert.py --hf-ckpt-path /data/users/jianiw/dsv3-weights/ --save-path /data/users/jianiw/dsv3-weights-5-layer/ --n-experts 256 --model-parallel 8``` - Second, I run a single forward pass using ```torchrun --nnodes 1 --nproc-per-node 8 inference/run_single_forward.py --config inference/configs/config_671B.json > dsv3-output.txt 2>&1``` Here's the detailed numerical comparison I've seen: <img width="917" height="488" alt="Image" src="https://github.com/user-attachments/assets/a1b3f865-d0ec-4ed1-a5c0-f039fe6879f2" /> <img width="917" height="628" alt="Image" src="https://github.com/user-attachments/assets/73cc92a1-eca0-4cd0-8543-fb049107b66b" /> **Expected behavior** Expected behavior: After first dense layer's attention layer, the output should be almost the same (There might be slightly difference because of fp8 vs bfloat16 precision). **Screenshots** See screenshots above **Additional context** We observed the same issue for llama3 model before - The weights for llama3 on HuggingFace is permuted compared to original Meta's weights. so we need to manually permute the weights back to accommodate the rotary embedding implementation difference. Reference: https://github.com/pytorch/torchtitan/issues/335

bsakaguchi

2026-10-26 21:23:20 +00:00

closed this issue
added the
stale

closed-as-stale
labels

bsakaguchi commented

2026-10-26 21:23:21 +00:00

Author

Owner

@69gg commented on GitHub (Jul 26, 2025):

The deepseek team will not answer your question here in github. Try ask in Reddit or Stack overflow and some people will talk with you.

@69gg commented on GitHub (Jul 26, 2025): The deepseek team will not answer your question here in github. Try ask in Reddit or Stack overflow and some people will talk with you.

bsakaguchi commented

2026-10-26 21:23:21 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Aug 26, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

@github-actions[bot] commented on GitHub (Aug 26, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

bsakaguchi commented

2026-10-26 21:23:22 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Sep 10, 2025):

false

@github-actions[bot] commented on GitHub (Sep 10, 2025): false