deepseek-ai/DeepSeek-V3
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-V3.git synced 2026-11-05 00:47:48 +00:00
Code Issues 25 Projects Packages Wiki Activity

[GH-ISSUE #4] Converted bf16 Model on Hugging Face #3

New issue
Closed
opened 2026-10-26 21:10:03 +00:00 by bsakaguchi · 14 comments
bsakaguchi commented 2026-10-26 21:10:03 +00:00
Owner
Copy link

Originally created by @OpenSourceRonin on GitHub (Dec 26, 2024).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V3/issues/4

I've uploaded the converted bf16 model here for everyone to use freely: https://huggingface.co/collections/opensourcerelease/deepseek-v3-bf16-676d7fa1b3f500d39f8f559b

Originally created by @OpenSourceRonin on GitHub (Dec 26, 2024). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V3/issues/4 I've uploaded the converted bf16 model here for everyone to use freely: https://huggingface.co/collections/opensourcerelease/deepseek-v3-bf16-676d7fa1b3f500d39f8f559b
bsakaguchi 2026-10-26 21:10:03 +00:00
bsakaguchi commented 2026-10-26 21:10:06 +00:00
Author
Owner
Copy link

@samyon7 commented on GitHub (Dec 30, 2024):

Do you build this model purely using the code? Would you give me the full documentation so you can upload to huggingface with the transformers safetensor formats? I don't get anywhere how to do it. Thankyouu :)<

<!-- gh-comment-id:2565365780 --> @samyon7 commented on GitHub (Dec 30, 2024): Do you build this model purely using the code? Would you give me the full documentation so you can upload to huggingface with the transformers safetensor formats? I don't get anywhere how to do it. Thankyouu :)<
bsakaguchi commented 2026-10-26 21:10:06 +00:00
Author
Owner
Copy link

@OpenSourceRonin commented on GitHub (Dec 30, 2024):

We have uploaded converted models at here https://huggingface.co/collections/opensourcerelease/deepseek-v3-bf16-676d7fa1b3f500d39f8f559b

<!-- gh-comment-id:2565382464 --> @OpenSourceRonin commented on GitHub (Dec 30, 2024): We have uploaded converted models at here https://huggingface.co/collections/opensourcerelease/deepseek-v3-bf16-676d7fa1b3f500d39f8f559b
bsakaguchi commented 2026-10-26 21:10:07 +00:00
Author
Owner
Copy link

@samyon7 commented on GitHub (Dec 30, 2024):

I mean, how you convert from pytorch format into safetensor formats? Any documentation there?

<!-- gh-comment-id:2565417367 --> @samyon7 commented on GitHub (Dec 30, 2024): I mean, how you convert from pytorch format into safetensor formats? Any documentation there?
bsakaguchi commented 2026-10-26 21:10:07 +00:00
Author
Owner
Copy link

@OpenSourceRonin commented on GitHub (Dec 31, 2024):

I mean, how you convert from pytorch format into safetensor formats? Any documentation there?
just run this one https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/fp8_cast_bf16.py

python fp8_cast_bf16.py --input-fp8-hf-path INPUT_PATH --output-bf16-hf-path OUTPUT_PATH
<!-- gh-comment-id:2566209712 --> @OpenSourceRonin commented on GitHub (Dec 31, 2024): > I mean, how you convert from pytorch format into safetensor formats? Any documentation there? just run this one https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/fp8_cast_bf16.py ```bash python fp8_cast_bf16.py --input-fp8-hf-path INPUT_PATH --output-bf16-hf-path OUTPUT_PATH ```
bsakaguchi commented 2026-10-26 21:10:07 +00:00
Author
Owner
Copy link

@wacyfdyy commented on GitHub (Feb 6, 2025):

Hello, do I need at least the size of the bf16 model GPU memory to convert successfully? How many GPUs did you use to convert the weights of bf16,thanku

<!-- gh-comment-id:2638969829 --> @wacyfdyy commented on GitHub (Feb 6, 2025): Hello, do I need at least the size of the bf16 model GPU memory to convert successfully? How many GPUs did you use to convert the weights of bf16,thanku
bsakaguchi commented 2026-10-26 21:10:08 +00:00
Author
Owner
Copy link

@ehuaa commented on GitHub (Feb 7, 2025):

Can this conversion be operationed on A100 cards?

<!-- gh-comment-id:2642799577 --> @ehuaa commented on GitHub (Feb 7, 2025): Can this conversion be operationed on A100 cards?
bsakaguchi commented 2026-10-26 21:10:08 +00:00
Author
Owner
Copy link

@gkm0120 commented on GitHub (Feb 10, 2025):

I'm trying to convert deepseek v3 using A100

python fp8_cast_bf16.py --input-fp8-hf-path /models/DeepSeek-V3/ --output-bf16-hf-path /models/DeepSeek-V3-BF16

but got
assert builder.options.allow_fp8e4nv, "fp8e4nv data type is not supported on CUDA arch < 89"
AssertionError: fp8e4nv data type is not supported on CUDA arch < 89

<!-- gh-comment-id:2647565827 --> @gkm0120 commented on GitHub (Feb 10, 2025): I'm trying to convert deepseek v3 using A100 ``` python fp8_cast_bf16.py --input-fp8-hf-path /models/DeepSeek-V3/ --output-bf16-hf-path /models/DeepSeek-V3-BF16 ``` but got assert builder.options.allow_fp8e4nv, "fp8e4nv data type is not supported on CUDA arch < 89" AssertionError: fp8e4nv data type is not supported on CUDA arch < 89
bsakaguchi commented 2026-10-26 21:10:08 +00:00
Author
Owner
Copy link

@ehuaa commented on GitHub (Feb 10, 2025):

I'm trying to convert deepseek v3 using A100

python fp8_cast_bf16.py --input-fp8-hf-path /models/DeepSeek-V3/ --output-bf16-hf-path /models/DeepSeek-V3-BF16

but got assert builder.options.allow_fp8e4nv, "fp8e4nv data type is not supported on CUDA arch < 89" AssertionError: fp8e4nv data type is not supported on CUDA arch < 89

Is the error happens in https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/kernel.py#L78 here? maybe triton cannot support tl.load an fp8 tensor in A100 @gkm0120

<!-- gh-comment-id:2647605485 --> @ehuaa commented on GitHub (Feb 10, 2025): > I'm trying to convert deepseek v3 using A100 > > ``` > python fp8_cast_bf16.py --input-fp8-hf-path /models/DeepSeek-V3/ --output-bf16-hf-path /models/DeepSeek-V3-BF16 > ``` > > but got assert builder.options.allow_fp8e4nv, "fp8e4nv data type is not supported on CUDA arch < 89" AssertionError: fp8e4nv data type is not supported on CUDA arch < 89 Is the error happens in https://github.com/deepseek-ai/DeepSeek-V3/blob/main/inference/kernel.py#L78 here? maybe triton cannot support tl.load an fp8 tensor in A100 @gkm0120
bsakaguchi commented 2026-10-26 21:10:09 +00:00
Author
Owner
Copy link

@wptoux commented on GitHub (Feb 14, 2025):

github.com/deepseek-ai/DeepSeek-V3@2f7b80eece/inference/fp8_cast_bf16.py (L80)

I tried changing

new_state_dict[weight_name] = weight_dequant(weight, scale_inv)

to

new_state_dict[weight_name] = weight_dequant(weight.float(), scale_inv)

and then the conversion code seems to work now on A100.

<!-- gh-comment-id:2658440116 --> @wptoux commented on GitHub (Feb 14, 2025): https://github.com/deepseek-ai/DeepSeek-V3/blob/2f7b80eecebf3d1c84da5a0d465f6639ea175012/inference/fp8_cast_bf16.py#L80 I tried changing ```python new_state_dict[weight_name] = weight_dequant(weight, scale_inv) ``` to ```python new_state_dict[weight_name] = weight_dequant(weight.float(), scale_inv) ``` and then the conversion code seems to work now on A100.
bsakaguchi commented 2026-10-26 21:10:09 +00:00
Author
Owner
Copy link

@PokeLu commented on GitHub (Mar 10, 2025):

DeepSeek-V3/inference/fp8_cast_bf16.py

Line 80 in 2f7b80e

new_state_dict[weight_name] = weight_dequant(weight, scale_inv)
I tried changing

new_state_dict[weight_name] = weight_dequant(weight, scale_inv)
to

new_state_dict[weight_name] = weight_dequant(weight.float(), scale_inv)
and then the conversion code seems to work now on A100.

I tried your solution and it worked well on A800. Does this change has any influnece on the converting precison or exactly the same as the original script?

<!-- gh-comment-id:2710092934 --> @PokeLu commented on GitHub (Mar 10, 2025): > [DeepSeek-V3/inference/fp8_cast_bf16.py](https://github.com/deepseek-ai/DeepSeek-V3/blob/2f7b80eecebf3d1c84da5a0d465f6639ea175012/inference/fp8_cast_bf16.py#L80) > > Line 80 in [2f7b80e](/deepseek-ai/DeepSeek-V3/commit/2f7b80eecebf3d1c84da5a0d465f6639ea175012) > > new_state_dict[weight_name] = weight_dequant(weight, scale_inv) > I tried changing > > new_state_dict[weight_name] = weight_dequant(weight, scale_inv) > to > > new_state_dict[weight_name] = weight_dequant(weight.float(), scale_inv) > and then the conversion code seems to work now on A100. I tried your solution and it worked well on A800. Does this change has any influnece on the converting precison or exactly the same as the original script?
bsakaguchi commented 2026-10-26 21:10:10 +00:00
Author
Owner
Copy link

@wptoux commented on GitHub (Mar 14, 2025):

DeepSeek-V3/inference/fp8_cast_bf16.py
Line 80 in 2f7b80e
new_state_dict[weight_name] = weight_dequant(weight, scale_inv)
I tried changing
new_state_dict[weight_name] = weight_dequant(weight, scale_inv)
to
new_state_dict[weight_name] = weight_dequant(weight.float(), scale_inv)
and then the conversion code seems to work now on A100.

I tried your solution and it worked well on A800. Does this change has any influnece on the converting precison or exactly the same as the original script?

The purpose of the code is to convert FP8 weights to 32-bit float before computation, which should be lossless in PyTorch. After testing the converted model, I observed no noticeable precision loss.

<!-- gh-comment-id:2723686148 --> @wptoux commented on GitHub (Mar 14, 2025): > > [DeepSeek-V3/inference/fp8_cast_bf16.py](https://github.com/deepseek-ai/DeepSeek-V3/blob/2f7b80eecebf3d1c84da5a0d465f6639ea175012/inference/fp8_cast_bf16.py#L80) > > Line 80 in [2f7b80e](/deepseek-ai/DeepSeek-V3/commit/2f7b80eecebf3d1c84da5a0d465f6639ea175012) > > new_state_dict[weight_name] = weight_dequant(weight, scale_inv) > > I tried changing > > new_state_dict[weight_name] = weight_dequant(weight, scale_inv) > > to > > new_state_dict[weight_name] = weight_dequant(weight.float(), scale_inv) > > and then the conversion code seems to work now on A100. > > I tried your solution and it worked well on A800. Does this change has any influnece on the converting precison or exactly the same as the original script? The purpose of the code is to convert FP8 weights to 32-bit float before computation, which should be lossless in PyTorch. After testing the converted model, I observed no noticeable precision loss.
bsakaguchi commented 2026-10-26 21:10:10 +00:00
Author
Owner
Copy link

@Lynnzake commented on GitHub (Apr 3, 2025):

I'm trying to convert deepseek v3 using A100

python fp8_cast_bf16.py --input-fp8-hf-path /models/DeepSeek-V3/ --output-bf16-hf-path /models/DeepSeek-V3-BF16

but got assert builder.options.allow_fp8e4nv, "fp8e4nv data type is not supported on CUDA arch < 89" AssertionError: fp8e4nv data type is not supported on CUDA arch < 89

Got Similar problem when i run code on A800,

root@b9fbb2fd8c99:/Pai-Megatron-Patch-main/toolkits/model_checkpoints_convertor/deepseek# python fp8_cast_bf16.py --input-fp8-hf-path /deepseek-ckpts/DeepSeek-V3 --output-bf16-hf-path /deepseek-ckpts/DeepSeek-V3-bf16
  0%|                                                                                                                                                                               | 0/163 [00:02<?, ?it/s]
Traceback (most recent call last):
  File "/Pai-Megatron-Patch-main/toolkits/model_checkpoints_convertor/deepseek/fp8_cast_bf16.py", line 106, in <module>
    main(args.input_fp8_hf_path, args.output_bf16_hf_path)
  File "/Pai-Megatron-Patch-main/toolkits/model_checkpoints_convertor/deepseek/fp8_cast_bf16.py", line 75, in main
    new_state_dict[weight_name] = weight_dequant(weight, scale_inv)
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Pai-Megatron-Patch-main/toolkits/model_checkpoints_convertor/deepseek/fp8_cast_bf16.py", line 32, in weight_dequant
    weight_dequant_kernel[grid](x, s, y, M, N, BLOCK_SIZE=block_size)
  File "/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py", line 330, in <lambda>
    return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py", line 623, in run
    kernel = self.compile(
             ^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py", line 273, in compile
    module = src.make_ir(options, codegen_fns, module_map, context)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py", line 100, in make_ir
    return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
triton.compiler.errors.CompilationError: at 1:0:
def weight_dequant_kernel(x_ptr, s_ptr, y_ptr, M, N, BLOCK_SIZE: tl.constexpr):
^
ValueError("type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')")

but after following the solution of @wptoux , it worked.
Although problem solved, but I questioned why this could work, and why we do this kind of change?

<!-- gh-comment-id:2774718257 --> @Lynnzake commented on GitHub (Apr 3, 2025): > I'm trying to convert deepseek v3 using A100 > > ``` > python fp8_cast_bf16.py --input-fp8-hf-path /models/DeepSeek-V3/ --output-bf16-hf-path /models/DeepSeek-V3-BF16 > ``` > > but got assert builder.options.allow_fp8e4nv, "fp8e4nv data type is not supported on CUDA arch < 89" AssertionError: fp8e4nv data type is not supported on CUDA arch < 89 Got Similar problem when i run code on A800, ```bash root@b9fbb2fd8c99:/Pai-Megatron-Patch-main/toolkits/model_checkpoints_convertor/deepseek# python fp8_cast_bf16.py --input-fp8-hf-path /deepseek-ckpts/DeepSeek-V3 --output-bf16-hf-path /deepseek-ckpts/DeepSeek-V3-bf16 0%| | 0/163 [00:02<?, ?it/s] Traceback (most recent call last): File "/Pai-Megatron-Patch-main/toolkits/model_checkpoints_convertor/deepseek/fp8_cast_bf16.py", line 106, in <module> main(args.input_fp8_hf_path, args.output_bf16_hf_path) File "/Pai-Megatron-Patch-main/toolkits/model_checkpoints_convertor/deepseek/fp8_cast_bf16.py", line 75, in main new_state_dict[weight_name] = weight_dequant(weight, scale_inv) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/Pai-Megatron-Patch-main/toolkits/model_checkpoints_convertor/deepseek/fp8_cast_bf16.py", line 32, in weight_dequant weight_dequant_kernel[grid](x, s, y, M, N, BLOCK_SIZE=block_size) File "/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py", line 330, in <lambda> return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py", line 623, in run kernel = self.compile( ^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py", line 273, in compile module = src.make_ir(options, codegen_fns, module_map, context) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py", line 100, in make_ir return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ triton.compiler.errors.CompilationError: at 1:0: def weight_dequant_kernel(x_ptr, s_ptr, y_ptr, M, N, BLOCK_SIZE: tl.constexpr): ^ ValueError("type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')") ``` but after following the solution of @wptoux , it worked. Although problem solved, but I questioned why this could work, and why we do this kind of change?
bsakaguchi commented 2026-10-26 21:10:10 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (May 5, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

<!-- gh-comment-id:2849610340 --> @github-actions[bot] commented on GitHub (May 5, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
bsakaguchi commented 2026-10-26 21:10:11 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (May 19, 2025):

false

<!-- gh-comment-id:2889303541 --> @github-actions[bot] commented on GitHub (May 19, 2025): false
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
v1.0.0
Labels
Clear labels   
closed-as-stale

   
enhancement

   
question

   
spam

   
stale
No labels
closed-as-stale
enhancement
question
spam
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepSeek-V3#3
No description provided.