[GH-ISSUE #1946] [New Model]: Support Deepseek V4 Series

bsakaguchi commented

2026-10-29 14:49:55 +00:00

Owner

Originally created by @tunglinwood on GitHub (Apr 24, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1946

Reminder

I have read the above rules and searched the existing issues.

Description

https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro

Series: https://huggingface.co/collections/deepseek-ai/deepseek-v4

Pull Request

No response

Originally created by @tunglinwood on GitHub (Apr 24, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1946 ### Reminder - [x] I have read the above rules and searched the existing issues. ### Description https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro Series: https://huggingface.co/collections/deepseek-ai/deepseek-v4 ### Pull Request _No response_

bsakaguchi added the

enhancement

label

2026-10-29 14:49:55 +00:00

bsakaguchi commented

2026-10-29 14:49:55 +00:00

Author

Owner

@SchenbergZY commented on GitHub (Apr 24, 2026):

long way to go with so much techs..

@SchenbergZY commented on GitHub (Apr 24, 2026): long way to go with so much techs..

bsakaguchi commented

2026-10-29 14:49:55 +00:00

Author

Owner

@sorasoras commented on GitHub (Apr 24, 2026):

DS4-flash would be great on consumer hardware.

@sorasoras commented on GitHub (Apr 24, 2026): DS4-flash would be great on consumer hardware.

bsakaguchi commented

2026-10-29 14:49:55 +00:00

Author

Owner

@dzenc commented on GitHub (Apr 25, 2026):

Looking forward to this one (DS4-flash). Are there technical hurdles?

@dzenc commented on GitHub (Apr 25, 2026): Looking forward to this one (DS4-flash). Are there technical hurdles?

bsakaguchi commented

2026-10-29 14:49:56 +00:00

Author

Owner

@jimtjames commented on GitHub (Apr 25, 2026):

The official releases use MXFP4 for expert weights (though the base models are entirely FP8). AFAIK, FP4 formats aren't yet supported by kt-kernel, though they're on the Q2 roadmap.

@jimtjames commented on GitHub (Apr 25, 2026): The official releases use MXFP4 for expert weights (though the base models are entirely FP8). AFAIK, FP4 formats aren't yet supported by kt-kernel, though they're on the Q2 roadmap.

bsakaguchi commented

2026-10-29 14:49:56 +00:00

Author

Owner

@xxxzsgxxx commented on GitHub (Apr 30, 2026):

非常期待支持 DS-V4

@xxxzsgxxx commented on GitHub (Apr 30, 2026): 非常期待支持 DS-V4

bsakaguchi commented

2026-10-29 14:49:56 +00:00

Author

Owner

@kassieclaire commented on GitHub (Apr 30, 2026):

https://github.com/kvcache-ai/ktransformers/pull/1957 -- it seems like this would be related

@kassieclaire commented on GitHub (Apr 30, 2026): https://github.com/kvcache-ai/ktransformers/pull/1957 -- it seems like this would be related

bsakaguchi commented

2026-10-29 14:49:56 +00:00

Author

Owner

@mytbk commented on GitHub (May 6, 2026):

看最新文档 doc/en/DeepSeek-V4-Flash.md 说能支持 DS-V4-Flash 了，而且可以支持消费级显卡（现在vLLM和SGLang只能支持数据中心的Hopper和Blackwell），感觉可以试一下。

@mytbk commented on GitHub (May 6, 2026): 看最新文档 doc/en/DeepSeek-V4-Flash.md 说能支持 DS-V4-Flash 了，而且可以支持消费级显卡（现在vLLM和SGLang只能支持数据中心的Hopper和Blackwell），感觉可以试一下。

bsakaguchi commented

2026-10-29 14:49:56 +00:00

Author

Owner

@Jeffrey064 commented on GitHub (May 11, 2026):

why don't support A100 GPU ...

@Jeffrey064 commented on GitHub (May 11, 2026): why don't support A100 GPU ...

bsakaguchi commented

2026-10-29 14:49:56 +00:00

Author

Owner

@kassieclaire commented on GitHub (May 13, 2026):

why don't support A100 GPU ...

Would using a quant for GPU weights like an int8 or int4 quant resolve this issue? Maybe converting fp8 non-expert weights to bf16?

@kassieclaire commented on GitHub (May 13, 2026): > why don't support A100 GPU ... Would using a quant for GPU weights like an int8 or int4 quant resolve this issue? Maybe converting fp8 non-expert weights to bf16?

bsakaguchi commented

2026-10-29 14:49:56 +00:00

Author

Owner

@mytbk commented on GitHub (May 20, 2026):

DeepSeek V4 Flash 官方没给 chat template, 我现在试过 https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/discussions/16 提供的 chat template，在 sglang 命令行里面加 --reasoning-parser deepseek-v4，都没能把思考功能打开，调 API 都是返回 "reasoning_content": null. 不知道有没有人知道怎么打开思考功能。

@mytbk commented on GitHub (May 20, 2026): DeepSeek V4 Flash 官方没给 chat template, 我现在试过 https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash/discussions/16 提供的 chat template，在 sglang 命令行里面加 ``--reasoning-parser deepseek-v4``，都没能把思考功能打开，调 API 都是返回 ``"reasoning_content": null``. 不知道有没有人知道怎么打开思考功能。

bsakaguchi commented

2026-10-29 14:49:56 +00:00

Author

Owner

@usrlocalben commented on GitHub (May 20, 2026):

I tried this out. I managed to get some output but not without problems along the way.

The docs indicate using pip install --upgrade flashinfer-python flashinfer-cubin but today this also upgrades torch which breaks sglang. Downgrading torch back to its prior version (with --no-deps) after the flashinfer upgrade seems to resolve it.
A general problem with kt/sg docs is confusion and ambiguity wrt. CUDA 12 and CUDA 13. I couldn't get CUDA 13 to work using my build script that works for e.g. Kimi. The DS4 tutorial says "CUDA 12.8+" does that include CUDA 13? or only 12.9? It's somewhat ambiguous. Using CUDA 12 nvcc and the sg kernel wheels seem to be OK.
DS4 tool use module is not included in the kt sglang. --tool-parser deepseekv32 seems to work OK . This is not mentioned in the documentation anywhere. Similarly, one must look around to find a chat_template. The model is nearly useless without these things so it's a little puzzling why they're not at least hinted at in the tutorial/docs.
The docs indicate the use of --disable-radix-cache - without a cache this model seems nearly useless. Is that just the current state of the sg/dsv4 implementation?

All that said, it does work. Without a cache I don't know what one would use it for besides one-shot tasks like compaction, for which is is highly useful since the long context allows it to compact a 256K session from another model.

@usrlocalben commented on GitHub (May 20, 2026): I tried this out. I managed to get some output but not without problems along the way. - The docs indicate using `pip install --upgrade flashinfer-python flashinfer-cubin` but today this also upgrades _torch_ which breaks sglang. Downgrading torch back to its prior version (with --no-deps) after the flashinfer upgrade seems to resolve it. - A general problem with kt/sg docs is confusion and ambiguity wrt. CUDA 12 and CUDA 13. I couldn't get CUDA 13 to work using my build script that works for e.g. Kimi. The DS4 tutorial says "CUDA 12.8+" does that include CUDA 13? or only 12.9? It's somewhat ambiguous. Using CUDA 12 nvcc and the sg kernel wheels seem to be OK. - DS4 tool use module is not included in the kt sglang. --tool-parser deepseekv32 seems to work OK . This is not mentioned in the documentation anywhere. Similarly, one must look around to find a chat_template. The model is nearly useless without these things so it's a little puzzling why they're not at least hinted at in the tutorial/docs. - The docs indicate the use of --disable-radix-cache - without a cache this model seems nearly useless. Is that just the current state of the sg/dsv4 implementation? All that said, it does work. Without a cache I don't know what one would use it for besides one-shot tasks like compaction, for which is is highly useful since the long context allows it to compact a 256K session from another model.

bsakaguchi commented

2026-10-29 14:49:56 +00:00

Author

Owner

@mytbk commented on GitHub (May 22, 2026):

I tried this out. I managed to get some output but not without problems along the way.

* The docs indicate using `pip install --upgrade flashinfer-python flashinfer-cubin` but today this also upgrades _torch_ which breaks sglang. Downgrading torch back to its prior version (with --no-deps) after the flashinfer upgrade seems to resolve it.

* A general problem with kt/sg docs is confusion and ambiguity wrt. CUDA 12 and CUDA 13. I couldn't get CUDA 13 to work using my build script that works for e.g. Kimi. The DS4 tutorial says "CUDA 12.8+" does that include CUDA 13? or only 12.9? It's somewhat ambiguous. Using CUDA 12 nvcc and the sg kernel wheels seem to be OK.

* DS4 tool use module is not included in the kt sglang. --tool-parser deepseekv32 seems to work OK . This is not mentioned in the documentation anywhere. Similarly, one must look around to find a chat_template. The model is nearly useless without these things so it's a little puzzling why they're not at least hinted at in the tutorial/docs.

* The docs indicate the use of --disable-radix-cache - without a cache this model seems nearly useless. Is that just the current state of the sg/dsv4 implementation?

All that said, it does work. Without a cache I don't know what one would use it for besides one-shot tasks like compaction, for which is is highly useful since the long context allows it to compact a 256K session from another model.

@usrlocalben From the sm120 patch and running guide by @0xSero (https://github.com/0xSero/deepseek-v4-flash-sm120#model) it seems that we don't need a chat template for DS4, and the kvcache-ai fork of sglang already ships the encoding_dsv4.py encoder. Also, the guide mentions SGLANG_ENABLE_THINKING=1 variable, and I don't know if this enables thinking.

For the CUDA13 stuff, I'm also confused. I use the kt and kt-kernel install.sh scripts, and pip will just pull the cu12 torch and sgl-kernel wheels, so I don't think it can run without CUDA12.

My server is currently broken, so I cannot test this for now.

@mytbk commented on GitHub (May 22, 2026): > I tried this out. I managed to get some output but not without problems along the way. > > * The docs indicate using `pip install --upgrade flashinfer-python flashinfer-cubin` but today this also upgrades _torch_ which breaks sglang. Downgrading torch back to its prior version (with --no-deps) after the flashinfer upgrade seems to resolve it. > > * A general problem with kt/sg docs is confusion and ambiguity wrt. CUDA 12 and CUDA 13. I couldn't get CUDA 13 to work using my build script that works for e.g. Kimi. The DS4 tutorial says "CUDA 12.8+" does that include CUDA 13? or only 12.9? It's somewhat ambiguous. Using CUDA 12 nvcc and the sg kernel wheels seem to be OK. > > * DS4 tool use module is not included in the kt sglang. --tool-parser deepseekv32 seems to work OK . This is not mentioned in the documentation anywhere. Similarly, one must look around to find a chat_template. The model is nearly useless without these things so it's a little puzzling why they're not at least hinted at in the tutorial/docs. > > * The docs indicate the use of --disable-radix-cache - without a cache this model seems nearly useless. Is that just the current state of the sg/dsv4 implementation? > > > All that said, it does work. Without a cache I don't know what one would use it for besides one-shot tasks like compaction, for which is is highly useful since the long context allows it to compact a 256K session from another model. @usrlocalben From the sm120 patch and running guide by @0xSero (https://github.com/0xSero/deepseek-v4-flash-sm120#model) it seems that we don't need a chat template for DS4, and the kvcache-ai fork of sglang already ships the encoding_dsv4.py encoder. Also, the guide mentions ``SGLANG_ENABLE_THINKING=1`` variable, and I don't know if this enables thinking. For the CUDA13 stuff, I'm also confused. I use the kt and kt-kernel install.sh scripts, and pip will just pull the cu12 torch and sgl-kernel wheels, so I don't think it can run without CUDA12. My server is currently broken, so I cannot test this for now.

bsakaguchi commented

2026-10-29 14:49:57 +00:00

Author

Owner

@nekoteai commented on GitHub (May 29, 2026):

Is your deploy's output normal? I evaluate its output and found it's different with the demo on hf.co

@nekoteai commented on GitHub (May 29, 2026): Is your deploy's output normal? I evaluate its output and found it's different with the demo on hf.co

Rows
Columns

[GH-ISSUE #1946] [New Model]: Support Deepseek V4 Series #1208

Reminder

Description

Pull Request