[GH-ISSUE #117 ] Support for DeepseekV3 680B #65

Author

Owner

@fengyang95 commented on GitHub (Dec 26, 2024):

+1

@fengyang95 commented on GitHub (Dec 26, 2024): +1

bsakaguchi commented

Author

Owner

@Nottlespike commented on GitHub (Dec 26, 2024):

Does ktransformers depend on HF transformers support for a for a model arch? If so we are going to have to wait until DeepSeek-V3 supports HF transformers as it does not yet and I don't see a PR from the DeepSeek team yet.

@Nottlespike commented on GitHub (Dec 26, 2024): Does ktransformers depend on HF transformers support for a for a model arch? If so we are going to have to wait until DeepSeek-V3 supports HF transformers as it does not yet and I don't see a PR from the DeepSeek team yet.

bsakaguchi commented

Author

Owner

@ghost commented on GitHub (Dec 27, 2024):

The ktransformer backend is an old commit of llama.cpp iirc

Edit: 6 months old github.com/ggerganov/llama.cpp@a94e6ff877

@ghost commented on GitHub (Dec 27, 2024): The ktransformer backend is an old commit of llama.cpp iirc Edit: 6 months old https://github.com/ggerganov/llama.cpp/tree/a94e6ff8774b7c9f950d9545baf0ce35e8d1ed2f

bsakaguchi commented

Author

Owner

@Nottlespike commented on GitHub (Dec 27, 2024):

The ktransformer backend is an old commit of llama.cpp iirc

Edit: 6 months old github.com/ggerganov/llama.cpp@a94e6ff877

Is it just me or is this not promising... I mean I'm patient but this means that transformers needs to support DeepSeek-V3 then llama.cpp needs to suppport DeepSeek-V3 then ktransformers needs to support llama.cpp's DeepSeek-V3 implementation....

@Nottlespike commented on GitHub (Dec 27, 2024): > The ktransformer backend is an old commit of llama.cpp iirc > > Edit: 6 months old https://github.com/ggerganov/llama.cpp/tree/a94e6ff8774b7c9f950d9545baf0ce35e8d1ed2f Is it just me or is this not promising... I mean I'm patient but this means that transformers needs to support DeepSeek-V3 then llama.cpp needs to suppport DeepSeek-V3 then ktransformers needs to support llama.cpp's DeepSeek-V3 implementation....

bsakaguchi commented

Author

Owner

@sorasoras commented on GitHub (Dec 27, 2024):

The ktransformer backend is an old commit of llama.cpp iirc

Edit: 6 months old github.com/ggerganov/llama.cpp@a94e6ff877

Is it just me or is this not promising... I mean I'm patient but this means that transformers needs to support DeepSeek-V3 then llama.cpp needs to suppport DeepSeek-V3 then ktransformers needs to support llama.cpp's DeepSeek-V3 implementation....

I heard that it s not hard to support V3 on llama cpp due to been resemblance to v2

@sorasoras commented on GitHub (Dec 27, 2024): > > The ktransformer backend is an old commit of llama.cpp iirc > > > > Edit: 6 months old https://github.com/ggerganov/llama.cpp/tree/a94e6ff8774b7c9f950d9545baf0ce35e8d1ed2f > > Is it just me or is this not promising... I mean I'm patient but this means that transformers needs to support DeepSeek-V3 then llama.cpp needs to suppport DeepSeek-V3 then ktransformers needs to support llama.cpp's DeepSeek-V3 implementation.... I heard that it s not hard to support V3 on llama cpp due to been resemblance to v2

bsakaguchi commented

Author

Owner

@Nottlespike commented on GitHub (Dec 27, 2024):

The ktransformer backend is an old commit of llama.cpp iirc
Edit: 6 months old github.com/ggerganov/llama.cpp@a94e6ff877

Is it just me or is this not promising... I mean I'm patient but this means that transformers needs to support DeepSeek-V3 then llama.cpp needs to suppport DeepSeek-V3 then ktransformers needs to support llama.cpp's DeepSeek-V3 implementation....

I heard that it s not hard to support V3 on llama cpp due to been resemblance to v2

Sadly this is not the case in two ways. Firstly V3 from the technical report is far far more complex than V2. Second V2 never actually got a HF transformers implementation sadly only a stale draft PR.
This is an issue that shows the problem in action https://github.com/huggingface/transformers/issues/34335
This is the stale attempt at V2 integration https://github.com/huggingface/transformers/pull/31976

@Nottlespike commented on GitHub (Dec 27, 2024): > > > The ktransformer backend is an old commit of llama.cpp iirc > > > Edit: 6 months old https://github.com/ggerganov/llama.cpp/tree/a94e6ff8774b7c9f950d9545baf0ce35e8d1ed2f > > > > > > Is it just me or is this not promising... I mean I'm patient but this means that transformers needs to support DeepSeek-V3 then llama.cpp needs to suppport DeepSeek-V3 then ktransformers needs to support llama.cpp's DeepSeek-V3 implementation.... > > I heard that it s not hard to support V3 on llama cpp due to been resemblance to v2 Sadly this is not the case in two ways. Firstly V3 from the technical report is far far more complex than V2. Second V2 never actually got a HF transformers implementation sadly only a stale draft PR. This is an issue that shows the problem in action https://github.com/huggingface/transformers/issues/34335 This is the stale attempt at V2 integration https://github.com/huggingface/transformers/pull/31976

bsakaguchi commented

Author

Owner

@sorasoras commented on GitHub (Dec 27, 2024):

The ktransformer backend is an old commit of llama.cpp iirc
Edit: 6 months old github.com/ggerganov/llama.cpp@a94e6ff877

Is it just me or is this not promising... I mean I'm patient but this means that transformers needs to support DeepSeek-V3 then llama.cpp needs to suppport DeepSeek-V3 then ktransformers needs to support llama.cpp's DeepSeek-V3 implementation....

I heard that it s not hard to support V3 on llama cpp due to been resemblance to v2

Sadly this is not the case in two ways. Firstly V3 from the technical report is far far more complex than V2. Second V2 never actually got a HF transformers implementation sadly only a stale draft PR. This is an issue that shows the problem in action huggingface/transformers#34335 This is the stale attempt at V2 integration huggingface/transformers#31976

well. kt do support DS2 so it should be just a upgrade with next gen moe router

@sorasoras commented on GitHub (Dec 27, 2024): > > > > The ktransformer backend is an old commit of llama.cpp iirc > > > > Edit: 6 months old https://github.com/ggerganov/llama.cpp/tree/a94e6ff8774b7c9f950d9545baf0ce35e8d1ed2f > > > > > > > > > Is it just me or is this not promising... I mean I'm patient but this means that transformers needs to support DeepSeek-V3 then llama.cpp needs to suppport DeepSeek-V3 then ktransformers needs to support llama.cpp's DeepSeek-V3 implementation.... > > > > > > I heard that it s not hard to support V3 on llama cpp due to been resemblance to v2 > > Sadly this is not the case in two ways. Firstly V3 from the technical report is far far more complex than V2. Second V2 never actually got a HF transformers implementation sadly only a stale draft PR. This is an issue that shows the problem in action [huggingface/transformers#34335](https://github.com/huggingface/transformers/issues/34335) This is the stale attempt at V2 integration [huggingface/transformers#31976](https://github.com/huggingface/transformers/pull/31976) well. kt do support DS2 so it should be just a upgrade with next gen moe router

bsakaguchi commented

Author

Owner

@Nottlespike commented on GitHub (Dec 27, 2024):

The ktransformer backend is an old commit of llama.cpp iirc
Edit: 6 months old github.com/ggerganov/llama.cpp@a94e6ff877

Is it just me or is this not promising... I mean I'm patient but this means that transformers needs to support DeepSeek-V3 then llama.cpp needs to suppport DeepSeek-V3 then ktransformers needs to support llama.cpp's DeepSeek-V3 implementation....

I heard that it s not hard to support V3 on llama cpp due to been resemblance to v2

Sadly this is not the case in two ways. Firstly V3 from the technical report is far far more complex than V2. Second V2 never actually got a HF transformers implementation sadly only a stale draft PR. This is an issue that shows the problem in action huggingface/transformers#34335 This is the stale attempt at V2 integration huggingface/transformers#31976

well. kt do support DS2 so it should be just a upgrade with next gen moe router

What they do is use remote code. There is no native HF transformers implemenation. Feel free to find it if you think I am incorrect. If people wish to use this for enterprise deployment letting a model run what is functionally arbitrary code on your servers is 100% a no go like it is for my startup.

@Nottlespike commented on GitHub (Dec 27, 2024): > > > > > The ktransformer backend is an old commit of llama.cpp iirc > > > > > Edit: 6 months old https://github.com/ggerganov/llama.cpp/tree/a94e6ff8774b7c9f950d9545baf0ce35e8d1ed2f > > > > > > > > > > > > Is it just me or is this not promising... I mean I'm patient but this means that transformers needs to support DeepSeek-V3 then llama.cpp needs to suppport DeepSeek-V3 then ktransformers needs to support llama.cpp's DeepSeek-V3 implementation.... > > > > > > > > > I heard that it s not hard to support V3 on llama cpp due to been resemblance to v2 > > > > > > Sadly this is not the case in two ways. Firstly V3 from the technical report is far far more complex than V2. Second V2 never actually got a HF transformers implementation sadly only a stale draft PR. This is an issue that shows the problem in action [huggingface/transformers#34335](https://github.com/huggingface/transformers/issues/34335) This is the stale attempt at V2 integration [huggingface/transformers#31976](https://github.com/huggingface/transformers/pull/31976) > > well. kt do support DS2 so it should be just a upgrade with next gen moe router What they do is use remote code. There is no native HF transformers implemenation. Feel free to find it if you think I am incorrect. If people wish to use this for enterprise deployment letting a model run what is functionally arbitrary code on your servers is 100% a no go like it is for my startup.

bsakaguchi commented

Author

Owner

@mahald commented on GitHub (Dec 28, 2024):

+1

@mahald commented on GitHub (Dec 28, 2024): +1

bsakaguchi commented

Author

Owner

@lzumot commented on GitHub (Dec 29, 2024):

+1

@lzumot commented on GitHub (Dec 29, 2024): +1

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Dec 30, 2024):

Supporting this seems easy; however, it requires approximately 400g of RAM for even q4km?

@Azure-Tang commented on GitHub (Dec 30, 2024): Supporting this seems easy; however, it requires approximately 400g of RAM for even q4km?

bsakaguchi commented

Author

Owner

@Nottlespike commented on GitHub (Dec 30, 2024):

Supporting this seems easy; however, it requires approximately 400g of RAM for even q4km?

Does ktransformers not need transformers or llama.cpp support for a a model? https://github.com/huggingface/transformers/issues/35425 Back of the napkin math is more like 5XXGB of VRAM/RAM would be needed given context.

@Nottlespike commented on GitHub (Dec 30, 2024): > Supporting this seems easy; however, it requires approximately 400g of RAM for even q4km? Does ktransformers not need transformers or llama.cpp support for a a model? https://github.com/huggingface/transformers/issues/35425 Back of the napkin math is more like 5XXGB of VRAM/RAM would be needed given context.

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Dec 31, 2024):

Supporting this seems easy; however, it requires approximately 400g of RAM for even q4km?

Does ktransformers not need transformers or llama.cpp support for a a model? huggingface/transformers#35425 Back of the napkin math is more like 5XXGB of VRAM/RAM would be needed given context.

Yes, we need transformers‘ modeling.py.

@Azure-Tang commented on GitHub (Dec 31, 2024): > > Supporting this seems easy; however, it requires approximately 400g of RAM for even q4km? > > Does ktransformers not need transformers or llama.cpp support for a a model? [huggingface/transformers#35425](https://github.com/huggingface/transformers/issues/35425) Back of the napkin math is more like 5XXGB of VRAM/RAM would be needed given context. Yes, we need transformers‘ modeling.py.

bsakaguchi commented

Author

Owner

@16x3b commented on GitHub (Dec 31, 2024):

Just spitballing. Would it be possible to do some speculative decoding with a smaller model (dense or moe) and then on the larger moe just use nvme for some of the experts? Why do we need all experts loaded into ram at all times instead of selecting experts as necessary?

Again just throwing an idea out there from where my understanding is at. I'd like to understand better why this would not work.

@16x3b commented on GitHub (Dec 31, 2024): Just spitballing. Would it be possible to do some speculative decoding with a smaller model (dense or moe) and then on the larger moe just use nvme for some of the experts? Why do we need all experts loaded into ram at all times instead of selecting experts as necessary? Again just throwing an idea out there from where my understanding is at. I'd like to understand better why this would not work.

bsakaguchi commented

Author

Owner

@sorasoras commented on GitHub (Jan 1, 2025):

Just spitballing. Would it be possible to do some speculative decoding with a smaller model (dense or moe) and then on the larger moe just use nvme for some of the experts? Why do we need all experts loaded into ram at all times instead of selecting experts as necessary?

Again just throwing an idea out there from where my understanding is at. I'd like to understand better why this would not work.

why would you need smaller model speculative decoding when you can do it via MTP？

@sorasoras commented on GitHub (Jan 1, 2025): > Just spitballing. Would it be possible to do some speculative decoding with a smaller model (dense or moe) and then on the larger moe just use nvme for some of the experts? Why do we need all experts loaded into ram at all times instead of selecting experts as necessary? > > Again just throwing an idea out there from where my understanding is at. I'd like to understand better why this would not work. why would you need smaller model speculative decoding when you can do it via MTP？

bsakaguchi commented

Author

Owner

@Nottlespike commented on GitHub (Jan 2, 2025):

@Azure-Tang
We are basically done with integrating DeepSeek-V3 into llama.cpp via a very very elegant solution from @fairydreaming
https://github.com/ggerganov/llama.cpp/issues/10981#issuecomment-2568349563
Can you see if you can try what they did?

@Nottlespike commented on GitHub (Jan 2, 2025): @Azure-Tang We are basically done with integrating DeepSeek-V3 into llama.cpp via a very very elegant solution from @fairydreaming https://github.com/ggerganov/llama.cpp/issues/10981#issuecomment-2568349563 Can you see if you can try what they did?

bsakaguchi commented

Author

Owner

@Nottlespike commented on GitHub (Jan 6, 2025):

For those looking for an update on this I've to forked it and I think I know how to get this working but no ETA.

@Nottlespike commented on GitHub (Jan 6, 2025): For those looking for an update on this I've to forked it and I think I know how to get this working but no ETA.

bsakaguchi commented

Author

Owner

@ELigoP commented on GitHub (Jan 9, 2025):

Supporting this seems easy; however, it requires approximately 400g of RAM for even q4km?

I manage to fit in Q3_K_M quant within 96GB VRAM + 256GB RAM. Q4_K_M is out of my reach.

@ELigoP commented on GitHub (Jan 9, 2025): > Supporting this seems easy; however, it requires approximately 400g of RAM for even q4km? I manage to fit in Q3_K_M quant within 96GB VRAM + 256GB RAM. Q4_K_M is out of my reach.

bsakaguchi commented

Author

Owner

@hvico commented on GitHub (Jan 9, 2025):

Supporting this seems easy; however, it requires approximately 400g of RAM for even q4km?

I manage to fit in Q3_K_M quant within 96GB VRAM + 256GB RAM. Q4_K_M is out of my reach.

Wow. Great news. What would be the mininum specs to be able to run a Q4_K_M with 96 GB VRAM? 512 GB of RAM plus that VRAM would be enough? I have that much VRAM but I will upgrade my RAM for that. Thanks!

@hvico commented on GitHub (Jan 9, 2025): > > Supporting this seems easy; however, it requires approximately 400g of RAM for even q4km? > > I manage to fit in Q3_K_M quant within 96GB VRAM + 256GB RAM. Q4_K_M is out of my reach. Wow. Great news. What would be the mininum specs to be able to run a Q4_K_M with 96 GB VRAM? 512 GB of RAM plus that VRAM would be enough? I have that much VRAM but I will upgrade my RAM for that. Thanks!

bsakaguchi commented

Author

Owner

@16x3b commented on GitHub (Jan 12, 2025):

why would you need smaller model speculative decoding when you can do it via MTP？

I don't see how MTP helps me. I'm suggesting speculative decoding because we can get faster inference from a smaller model and only refer to the larger model if confidence is low. No need to call the large model if the small one has a confident answer.

@16x3b commented on GitHub (Jan 12, 2025): > why would you need smaller model speculative decoding when you can do it via MTP？ I don't see how MTP helps me. I'm suggesting speculative decoding because we can get faster inference from a smaller model and only refer to the larger model if confidence is low. No need to call the large model if the small one has a confident answer.

bsakaguchi commented

Author

Owner

@sorasoras commented on GitHub (Jan 13, 2025):

why would you need smaller model speculative decoding when you can do it via MTP？

I don't see how MTP helps me. I'm suggesting speculative decoding because we can get faster inference from a smaller model and only refer to the larger model if confidence is low. No need to call the large model if the small one has a confident answer.

MTP generate two token and you use second token as speculative decode. This is call self-speculative decode. no need for extra model

@sorasoras commented on GitHub (Jan 13, 2025): > > why would you need smaller model speculative decoding when you can do it via MTP？ > > I don't see how MTP helps me. I'm suggesting speculative decoding because we can get faster inference from a smaller model and only refer to the larger model if confidence is low. No need to call the large model if the small one has a confident answer. MTP generate two token and you use second token as speculative decode. This is call self-speculative decode. no need for extra model

bsakaguchi commented

Author

Owner

@16x3b commented on GitHub (Jan 14, 2025):

MTP generate two token and you use second token as speculative decode. This is call self-speculative decode. no need for extra model

Understood. That's excellent! Thank you for explaining that to me. I see why there's no need to implement speculative decoding with a model that already has MTP implemented.

@16x3b commented on GitHub (Jan 14, 2025): > MTP generate two token and you use second token as speculative decode. This is call self-speculative decode. no need for extra model Understood. That's excellent! Thank you for explaining that to me. I see why there's no need to implement speculative decoding with a model that already has MTP implemented.

bsakaguchi commented

Author

Owner

@sorasoras commented on GitHub (Jan 14, 2025):

MTP generate two token and you use second token as speculative decode. This is call self-speculative decode. no need for extra model

Understood. That's excellent! Thank you for explaining that to me. I see why there's no need to implement speculative decoding with a model that already has MTP implemented.

if you read the paper, it mentioned you can get 90% hit rate with MTP 2-token and SD.
I guess we are going to have 4-token MTP in the future.

@sorasoras commented on GitHub (Jan 14, 2025): > > MTP generate two token and you use second token as speculative decode. This is call self-speculative decode. no need for extra model > > Understood. That's excellent! Thank you for explaining that to me. I see why there's no need to implement speculative decoding with a model that already has MTP implemented. if you read the paper, it mentioned you can get 90% hit rate with MTP 2-token and SD. I guess we are going to have 4-token MTP in the future.

bsakaguchi commented

Author

Owner

@bitnom commented on GitHub (Jan 28, 2025):

so hot right now

@bitnom commented on GitHub (Jan 28, 2025): so hot right now

bsakaguchi commented

Author

Owner

@whisper-bye commented on GitHub (Jan 28, 2025):

Dynamic 1.58-bit
https://unsloth.ai/blog/deepseekr1-dynamic

@whisper-bye commented on GitHub (Jan 28, 2025): Dynamic 1.58-bit https://unsloth.ai/blog/deepseekr1-dynamic

bsakaguchi commented

Author

Owner

@ChandanVerma commented on GitHub (Jan 29, 2025):

Any updates by when can we see deepseek v3 or r1 in ktransformers? Really looking forward to it. It would be of great help to everyone

Thanks

@ChandanVerma commented on GitHub (Jan 29, 2025): Any updates by when can we see deepseek v3 or r1 in ktransformers? Really looking forward to it. It would be of great help to everyone Thanks

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Jan 29, 2025):

Any updates by when can we see deepseek v3 or r1 in ktransformers? Really looking forward to it. It would be of great help to everyone

Thanks

I'm actively tracking the DeepSeek V3/R1 integration. The HuggingFace Transformers team is finalizing support for DeepSeek V3 (see PR #35926), which ktransformers relies on.

I'm pausing my vacation resolving any remaining compatibility issues. Expect a quick follow-up release for ktransformers once dependencies stabilize.

I’ll keep you updated here – appreciate your patience and support! 🙌

@Azure-Tang commented on GitHub (Jan 29, 2025): > Any updates by when can we see deepseek v3 or r1 in ktransformers? Really looking forward to it. It would be of great help to everyone > > Thanks I'm actively tracking the DeepSeek V3/R1 integration. The HuggingFace Transformers team is finalizing support for DeepSeek V3 (see [PR #35926](https://github.com/huggingface/transformers/pull/35926/files)), which ktransformers relies on. I'm pausing my vacation resolving any remaining compatibility issues. Expect a quick follow-up release for ktransformers once dependencies stabilize. I’ll keep you updated here – appreciate your patience and support! 🙌

bsakaguchi commented

Author

Owner

@Nondzu commented on GitHub (Jan 29, 2025):

I can run DS V3 and R1 on my workstation 768g of RAM. llama.cpp tested and works fine with deepseek v3/r1 Q4_K_M and Q5 models.
If you need any tests or help to implementation pls ping me.

@Nondzu commented on GitHub (Jan 29, 2025): I can run DS V3 and R1 on my workstation 768g of RAM. llama.cpp tested and works fine with deepseek v3/r1 Q4_K_M and Q5 models. If you need any tests or help to implementation pls ping me.

bsakaguchi commented

Author

Owner

@ChandanVerma commented on GitHub (Jan 30, 2025):

I can run DS V3 and R1 on my workstation 768g of RAM. llama.cpp tested and works fine with deepseek v3/r1 Q4_K_M and Q5 models. If you need any tests or help to implementation pls ping me.

Is it usable? How many TPS do you get? I am planning to run on my local server with 512g RAM and 6x3090 system

@ChandanVerma commented on GitHub (Jan 30, 2025): > I can run DS V3 and R1 on my workstation 768g of RAM. llama.cpp tested and works fine with deepseek v3/r1 Q4_K_M and Q5 models. If you need any tests or help to implementation pls ping me. Is it usable? How many TPS do you get? I am planning to run on my local server with 512g RAM and 6x3090 system

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Jan 31, 2025):

I have successfully deployed DeepSeek-V3 on the KTransformers framework, but there are still some accuracy issues that I am currently investigating.

Based on preliminary tests, using the Q4KM weight file requires at least 16GB of VRAM and 480GB of RAM. On a positive note, the generation speed can reach approximately 3.8 tokens/s.”

See #122

@Azure-Tang commented on GitHub (Jan 31, 2025): I have successfully deployed DeepSeek-V3 on the KTransformers framework, but there are still some accuracy issues that I am currently investigating. Based on preliminary tests, using the Q4KM weight file requires at least 16GB of VRAM and 480GB of RAM. On a positive note, the generation speed can reach approximately 3.8 tokens/s.” See #122

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Jan 31, 2025):

I can run DS V3 and R1 on my workstation 768g of RAM. llama.cpp tested and works fine with deepseek v3/r1 Q4_K_M and Q5 models. If you need any tests or help to implementation pls ping me.

What's your generate speed using llama.cpp with v3/r1 Q4_K_M?

@Azure-Tang commented on GitHub (Jan 31, 2025): > I can run DS V3 and R1 on my workstation 768g of RAM. llama.cpp tested and works fine with deepseek v3/r1 Q4_K_M and Q5 models. If you need any tests or help to implementation pls ping me. What's your generate speed using llama.cpp with v3/r1 Q4_K_M?

bsakaguchi commented

Author

Owner

@ChandanVerma commented on GitHub (Jan 31, 2025):

I have successfully deployed DeepSeek-V3 on the KTransformers framework, but there are still some accuracy issues that I am currently investigating.

Based on preliminary tests, using the Q4KM weight file requires at least 16GB of VRAM and 480GB of RAM. On a positive note, the generation speed can reach approximately 3.8 tokens/s.”

Thats wonderful.. Really appreciate the hard work. Just a small question if i inject more layers to the gpu since i have 144 GB of VRAM, will the RAM consumption go down and will i be able to allocate more context?

@ChandanVerma commented on GitHub (Jan 31, 2025): > I have successfully deployed DeepSeek-V3 on the KTransformers framework, but there are still some accuracy issues that I am currently investigating. > > Based on preliminary tests, using the Q4KM weight file requires at least 16GB of VRAM and 480GB of RAM. On a positive note, the generation speed can reach approximately 3.8 tokens/s.” Thats wonderful.. Really appreciate the hard work. Just a small question if i inject more layers to the gpu since i have 144 GB of VRAM, will the RAM consumption go down and will i be able to allocate more context?

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Jan 31, 2025):

I have successfully deployed DeepSeek-V3 on the KTransformers framework, but there are still some accuracy issues that I am currently investigating.
Based on preliminary tests, using the Q4KM weight file requires at least 16GB of VRAM and 480GB of RAM. On a positive note, the generation speed can reach approximately 3.8 tokens/s.”

Thats wonderful.. Really appreciate the hard work. Just a small question if i inject more layers to the gpu since i have 144 GB of VRAM, will the RAM consumption go down and will i be able to allocate more context?

yes, theoretically you can use marling expert as KExperts' backend (but we haven't test this op, this may cause some bug orz, and it's loading will be veeeeeeery slow)

@Azure-Tang commented on GitHub (Jan 31, 2025): > > I have successfully deployed DeepSeek-V3 on the KTransformers framework, but there are still some accuracy issues that I am currently investigating. > > Based on preliminary tests, using the Q4KM weight file requires at least 16GB of VRAM and 480GB of RAM. On a positive note, the generation speed can reach approximately 3.8 tokens/s.” > > Thats wonderful.. Really appreciate the hard work. Just a small question if i inject more layers to the gpu since i have 144 GB of VRAM, will the RAM consumption go down and will i be able to allocate more context? yes, theoretically you can use marling expert as KExperts' backend (but we haven't test this op, this may cause some bug orz, and it's loading will be veeeeeeery slow)

bsakaguchi commented

Author

Owner

@Nondzu commented on GitHub (Jan 31, 2025):

@ChandanVerma @Azure-Tang I've ~7t/s by llama.cpp with 100% CPU offload. don't have any GPU on that machine. and yes, it's usable as well with that speed

@Nondzu commented on GitHub (Jan 31, 2025): @ChandanVerma @Azure-Tang I've ~7t/s by llama.cpp with 100% CPU offload. don't have any GPU on that machine. and yes, it's usable as well with that speed

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Jan 31, 2025):

@ChandanVerma @Azure-Tang I've ~7t/s by llama.cpp with 100% CPU offload. don't have any GPU on that machine. and yes, it's usable as well with that speed

Our framework's architectural design suggests it should have inherent performance advantages in CPU-only environments. there might be some factors I haven't accounted for yet.

@Azure-Tang commented on GitHub (Jan 31, 2025): > [@ChandanVerma](https://github.com/ChandanVerma) [@Azure-Tang](https://github.com/Azure-Tang) I've ~7t/s by llama.cpp with 100% CPU offload. don't have any GPU on that machine. and yes, it's usable as well with that speed Our framework's architectural design suggests it should have inherent performance advantages in CPU-only environments. there might be some factors I haven't accounted for yet.

bsakaguchi commented

Author

Owner

@trilog-inc commented on GitHub (Jan 31, 2025):

Watching this closely. Thanks for your hardwork @Azure-Tang

@trilog-inc commented on GitHub (Jan 31, 2025): Watching this closely. Thanks for your hardwork @Azure-Tang

bsakaguchi commented

Author

Owner

@Nondzu commented on GitHub (Jan 31, 2025):

llama.cpp benchmark deepseek-256x20B-v3-q4_k.gguf
| deepseek2 671B Q4_K - Medium | 376.65 GiB | 671.03 B | CPU | 112 | pp512 | 49.65 ± 2.25 |
| deepseek2 671B Q4_K - Medium | 376.65 GiB | 671.03 B | CPU | 112 | tg128 | 7.37 ± 0.00 |

@Nondzu commented on GitHub (Jan 31, 2025): llama.cpp benchmark deepseek-256x20B-v3-q4_k.gguf | deepseek2 671B Q4_K - Medium | 376.65 GiB | 671.03 B | CPU | 112 | pp512 | 49.65 ± 2.25 | | deepseek2 671B Q4_K - Medium | 376.65 GiB | 671.03 B | CPU | 112 | tg128 | 7.37 ± 0.00 |

bsakaguchi commented

Author

Owner

@ChandanVerma commented on GitHub (Jan 31, 2025):

llama.cpp benchmark deepseek-256x20B-v3-q4_k.gguf | deepseek2 671B Q4_K - Medium | 376.65 GiB | 671.03 B | CPU | 112 | pp512 | 49.65 ± 2.25 | | deepseek2 671B Q4_K - Medium | 376.65 GiB | 671.03 B | CPU | 112 | tg128 | 7.37 ± 0.00 |

Great....How much RAM does it consume and whats the context size you are able to achieve?

@ChandanVerma commented on GitHub (Jan 31, 2025): > llama.cpp benchmark deepseek-256x20B-v3-q4_k.gguf | deepseek2 671B Q4_K - Medium | 376.65 GiB | 671.03 B | CPU | 112 | pp512 | 49.65 ± 2.25 | | deepseek2 671B Q4_K - Medium | 376.65 GiB | 671.03 B | CPU | 112 | tg128 | 7.37 ± 0.00 | Great....How much RAM does it consume and whats the context size you are able to achieve?

bsakaguchi commented

Author

Owner

@Nondzu commented on GitHub (Jan 31, 2025):

i use R1 Q5 for last days with 32000 ctx and predict 16000, it need almost 600g of RAM

@Nondzu commented on GitHub (Jan 31, 2025): i use R1 Q5 for last days with 32000 ctx and predict 16000, it need almost 600g of RAM ![Image](https://github.com/user-attachments/assets/22179971-21eb-4036-9f13-a28ae687933c)

bsakaguchi commented

Author

Owner

@Green0-0 commented on GitHub (Jan 31, 2025):

What are the chances of getting MTP support? That would greatly increase the t/s.

@Green0-0 commented on GitHub (Jan 31, 2025): What are the chances of getting MTP support? That would greatly increase the t/s.

bsakaguchi commented

Author

Owner

@Aiden-Frost commented on GitHub (Feb 1, 2025):

Is it possible to run the DeepSeek R1 in just 4 A100 GPU (roughly 320 GB VRAM) using accelerate with offloading to disk? While this will be really really slow, but for some analysis purpose is it possible to run with this setup?

@Aiden-Frost commented on GitHub (Feb 1, 2025): Is it possible to run the DeepSeek R1 in just 4 A100 GPU (roughly 320 GB VRAM) using accelerate with offloading to disk? While this will be really really slow, but for some analysis purpose is it possible to run with this setup?

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Feb 1, 2025):

Is it possible to run the DeepSeek R1 in just 4 A100 GPU (roughly 320 GB VRAM) using accelerate with offloading to disk? While this will be really really slow, but for some analysis purpose is it possible to run with this setup?

It's possible. Loading process will be slow in this way, but the inference speed can be greatly accelerated. You can refer this:

I have successfully deployed DeepSeek-V3 on the KTransformers framework, but there are still some accuracy issues that I am currently investigating.
Based on preliminary tests, using the Q4KM weight file requires at least 16GB of VRAM and 480GB of RAM. On a positive note, the generation speed can reach approximately 3.8 tokens/s.”

Thats wonderful.. Really appreciate the hard work. Just a small question if i inject more layers to the gpu since i have 144 GB of VRAM, will the RAM consumption go down and will i be able to allocate more context?

yes, theoretically you can use marling expert as KExperts' backend (but we haven't test this op, this may cause some bug orz, and it's loading will be veeeeeeery slow)

I would check our marlin expert op once down supporting DeepSeekv3.

@Azure-Tang commented on GitHub (Feb 1, 2025): > Is it possible to run the DeepSeek R1 in just 4 A100 GPU (roughly 320 GB VRAM) using accelerate with offloading to disk? While this will be really really slow, but for some analysis purpose is it possible to run with this setup? It's possible. Loading process will be slow in this way, but the inference speed can be greatly accelerated. You can refer this: > > > I have successfully deployed DeepSeek-V3 on the KTransformers framework, but there are still some accuracy issues that I am currently investigating. > > > Based on preliminary tests, using the Q4KM weight file requires at least 16GB of VRAM and 480GB of RAM. On a positive note, the generation speed can reach approximately 3.8 tokens/s.” > > > > > > Thats wonderful.. Really appreciate the hard work. Just a small question if i inject more layers to the gpu since i have 144 GB of VRAM, will the RAM consumption go down and will i be able to allocate more context? > > yes, theoretically you can use marling expert as KExperts' backend (but we haven't test this op, this may cause some bug orz, and it's loading will be veeeeeeery slow) I would check our marlin expert op once down supporting DeepSeekv3.

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Feb 1, 2025):

I can run DS V3 and R1 on my workstation 768g of RAM. llama.cpp tested and works fine with deepseek v3/r1 Q4_K_M and Q5 models. If you need any tests or help to implementation pls ping me.

Could you please help me check the precision issue? I'm currently stuck. The problem is similar to this:

user: hi
assistant: Hello! How are youHello! How can I have a nice to meet can you today. Good evening! Hello! How are youHello! or fine thanks for your, . can i have a chatI'mm but I am good-day evening! eveniny

I think this may cased by mla/rope implements or moe gates?

@Azure-Tang commented on GitHub (Feb 1, 2025): > I can run DS V3 and R1 on my workstation 768g of RAM. llama.cpp tested and works fine with deepseek v3/r1 Q4_K_M and Q5 models. If you need any tests or help to implementation pls ping me. Could you please help me check the precision issue? I'm currently stuck. The problem is similar to this: user: hi assistant: Hello! How are youHello! How can I have a nice to meet can you today. Good evening! Hello! How are youHello! or fine thanks for your, . can i have a chatI'mm but I am good-day evening! eveniny I think this may cased by mla/rope implements or moe gates?

bsakaguchi commented

Author

Owner

@Nondzu commented on GitHub (Feb 1, 2025):

@Azure-Tang I've issue to install ktransformers without cuda on machine. is it possible to build and install with CPU only ?
atm machine doesn't have any gpu

@Nondzu commented on GitHub (Feb 1, 2025): @Azure-Tang I've issue to install ktransformers without cuda on machine. is it possible to build and install with CPU only ? atm machine doesn't have any gpu

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Feb 1, 2025):

@Azure-Tang I've issue to install ktransformers without cuda on machine. is it possible to build and install with CPU only ? atm machine doesn't have any gpu

Currently we does require GPU orz. Thanks for reaching out—I'll try to find a workaround.

@Azure-Tang commented on GitHub (Feb 1, 2025): > [@Azure-Tang](https://github.com/Azure-Tang) I've issue to install ktransformers without cuda on machine. is it possible to build and install with CPU only ? atm machine doesn't have any gpu Currently we does require GPU orz. Thanks for reaching out—I'll try to find a workaround.

bsakaguchi commented

Author

Owner

@ELigoP commented on GitHub (Feb 3, 2025):

Do we need specific weights/quants to try DeepSeek V3 support pull request or any would work? E.g. IQ3_M or Q3_K_M GGUFs?

@ELigoP commented on GitHub (Feb 3, 2025): Do we need specific weights/quants to try [DeepSeek V3 support pull request](https://github.com/kvcache-ai/ktransformers/pull/122) or any would work? E.g. [IQ3_M](https://huggingface.co/bartowski/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-IQ3_M) or [Q3_K_M](https://huggingface.co/bartowski/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-Q3_K_M) GGUFs?

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Feb 3, 2025):

Do we need specific weights/quants to try DeepSeek V3 support pull request or any would work? E.g. IQ3_M or Q3_K_M GGUFs?

Any would work；We are currently investigating precision problem; I think release version would be available in one or two days.

@Azure-Tang commented on GitHub (Feb 3, 2025): > Do we need specific weights/quants to try [DeepSeek V3 support pull request](https://github.com/kvcache-ai/ktransformers/pull/122) or any would work? E.g. [IQ3_M](https://huggingface.co/bartowski/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-IQ3_M) or [Q3_K_M](https://huggingface.co/bartowski/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-Q3_K_M) GGUFs? Any would work；We are currently investigating precision problem; I think release version would be available in one or two days.

bsakaguchi commented

Author

Owner

@huliangbing commented on GitHub (Feb 3, 2025):

期待...

@huliangbing commented on GitHub (Feb 3, 2025): 期待...

bsakaguchi commented

Author

Owner

@davidsyoung commented on GitHub (Feb 3, 2025):

Watching this very closely. I have 9x3090's along with 256GB DDR4 RAM. Is it worth testing your current PR @Azure-Tang?

@davidsyoung commented on GitHub (Feb 3, 2025): Watching this very closely. I have 9x3090's along with 256GB DDR4 RAM. Is it worth testing your current PR @Azure-Tang?

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Feb 4, 2025):

Hi everyone,

Exciting updates to share! After intensive development efforts, we are pleased to announce that ktransformers now supports Deepseek-V3 and R1 architectures. The working PR (#122 ) is currently undergoing final validation and is expected to merge into main branch within the next few days. You can build from source via the feat-DeepSeekV3 branch for early access.

Special thanks to our community for the continued support.

@Azure-Tang commented on GitHub (Feb 4, 2025): Hi everyone, Exciting updates to share! After intensive development efforts, we are pleased to announce that ktransformers now supports Deepseek-V3 and R1 architectures. The working PR (#122 ) is currently undergoing final validation and is expected to merge into main branch within the next few days. You can build from source via the feat-DeepSeekV3 branch for early access. Special thanks to our community for the continued support.

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Feb 4, 2025):

Watching this very closely. I have 9x3090's along with 256GB DDR4 RAM. Is it worth testing your current PR @Azure-Tang?

You may need bigger RAM for now, it requires 380GB RAM for my default optimization yaml…… Or you can try to modify yaml to use marlin expert to use more DRAM?

@Azure-Tang commented on GitHub (Feb 4, 2025): > Watching this very closely. I have 9x3090's along with 256GB DDR4 RAM. Is it worth testing your current PR [@Azure-Tang](https://github.com/Azure-Tang)? You may need bigger RAM for now, it requires 380GB RAM for my default optimization yaml…… Or you can try to modify yaml to use marlin expert to use more DRAM?

bsakaguchi commented

Author

Owner

@davidsyoung commented on GitHub (Feb 4, 2025):

Hi everyone,

Exciting updates to share! After intensive development efforts, we are pleased to announce that ktransformers now supports Deepseek-V3 and R1 architectures. The working PR (#122 ) is currently undergoing final validation and is expected to merge into main branch within the next few days. You can build from source via the feat-DeepSeekV3 branch for early access.

Special thanks to our community for the continued support.

Fantastic! Going to try to get this working. Could you give a config/gguf that you recommend for this?

I'm currently attempting to launch with the following docker compose but not sure if I'm missing something. Could you give an example config/launch params for a Deepseek R1? Thank you very much for the hard work!

I'm getting

2025-02-04 16:31:26,879 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py[29]: Creating SQL tables
2025-02-04 16:31:26,997 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant
A new version of the following files was downloaded from https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat:
- configuration_deepseek.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
Traceback (most recent call last):
  File "/opt/conda/bin/ktransformers", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 114, in main
    create_interface(config=cfg, default_args=cfg)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface
    GlobalInterface.interface = BackendInterface(default_args)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 47, in __init__
    optimize_and_load_gguf(self.model, optimize_rule_path, gguf_path, config)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 128, in optimize_and_load_gguf
    inject(module, optimize_config, model_config, gguf_loader)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 31, in inject
    module_cls=getattr(__import__(import_module_name, fromlist=[""]), import_class_name)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/models.py", line 22, in <module>
    from ktransformers.operators.dynamic_attention import DynamicScaledDotProductAttention
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/dynamic_attention.py", line 19, in <module>
    from ktransformers.operators.cpuinfer import CPUInfer, CPUInferKVCache
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/cpuinfer.py", line 25, in <module>
    import cpuinfer_ext
ImportError: /opt/conda/lib/python3.10/site-packages/torch/lib/../../../.././libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /opt/conda/lib/python3.10/site-packages/cpuinfer_ext.cpython-310-x86_64-linux-gnu.so)

It's possible that my docker compose didn't fully rebuild from branch, forcing rebuild now

@davidsyoung commented on GitHub (Feb 4, 2025): > Hi everyone, > > Exciting updates to share! After intensive development efforts, we are pleased to announce that ktransformers now supports Deepseek-V3 and R1 architectures. The working PR ([#122](https://github.com/kvcache-ai/ktransformers/pull/122) ) is currently undergoing final validation and is expected to merge into main branch within the next few days. You can build from source via the feat-DeepSeekV3 branch for early access. > > Special thanks to our community for the continued support. Fantastic! Going to try to get this working. Could you give a config/gguf that you recommend for this? I'm currently attempting to launch with the following docker compose but not sure if I'm missing something. Could you give an example config/launch params for a Deepseek R1? Thank you very much for the hard work! I'm getting ``` 2025-02-04 16:31:26,879 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py[29]: Creating SQL tables 2025-02-04 16:31:26,997 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant A new version of the following files was downloaded from https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat: - configuration_deepseek.py . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. Traceback (most recent call last): File "/opt/conda/bin/ktransformers", line 8, in <module> sys.exit(main()) File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 114, in main create_interface(config=cfg, default_args=cfg) File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface GlobalInterface.interface = BackendInterface(default_args) File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 47, in __init__ optimize_and_load_gguf(self.model, optimize_rule_path, gguf_path, config) File "/opt/conda/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 128, in optimize_and_load_gguf inject(module, optimize_config, model_config, gguf_loader) File "/opt/conda/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 31, in inject module_cls=getattr(__import__(import_module_name, fromlist=[""]), import_class_name) File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/models.py", line 22, in <module> from ktransformers.operators.dynamic_attention import DynamicScaledDotProductAttention File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/dynamic_attention.py", line 19, in <module> from ktransformers.operators.cpuinfer import CPUInfer, CPUInferKVCache File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/cpuinfer.py", line 25, in <module> import cpuinfer_ext ImportError: /opt/conda/lib/python3.10/site-packages/torch/lib/../../../.././libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /opt/conda/lib/python3.10/site-packages/cpuinfer_ext.cpython-310-x86_64-linux-gnu.so) ``` It's possible that my docker compose didn't fully rebuild from branch, forcing rebuild now

bsakaguchi commented

Author

Owner

@davidsyoung commented on GitHub (Feb 4, 2025):

Watching this very closely. I have 9x3090's along with 256GB DDR4 RAM. Is it worth testing your current PR @Azure-Tang?

You may need bigger RAM for now, it requires 380GB RAM for my default optimization yaml…… Or you can try to modify yaml to use marlin expert to use more DRAM?

I can most definitely work on doing this and release once I get it working!

@davidsyoung commented on GitHub (Feb 4, 2025): > > Watching this very closely. I have 9x3090's along with 256GB DDR4 RAM. Is it worth testing your current PR [@Azure-Tang](https://github.com/Azure-Tang)? > > You may need bigger RAM for now, it requires 380GB RAM for my default optimization yaml…… Or you can try to modify yaml to use marlin expert to use more DRAM? I can most definitely work on doing this and release once I get it working!

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Feb 4, 2025):

Watching this very closely. I have 9x3090's along with 256GB DDR4 RAM. Is it worth testing your current PR @Azure-Tang?

You may need bigger RAM for now, it requires 380GB RAM for my default optimization yaml…… Or you can try to modify yaml to use marlin expert to use more DRAM?

I can most definitely work on doing this and release once I get it working!

Ok，the 380GB is obtained by q4km v3 model. Just try it!

@Azure-Tang commented on GitHub (Feb 4, 2025): > > > Watching this very closely. I have 9x3090's along with 256GB DDR4 RAM. Is it worth testing your current PR [@Azure-Tang](https://github.com/Azure-Tang)? > > > > > > You may need bigger RAM for now, it requires 380GB RAM for my default optimization yaml…… Or you can try to modify yaml to use marlin expert to use more DRAM? > > I can most definitely work on doing this and release once I get it working! Ok，the 380GB is obtained by q4km v3 model. Just try it!

bsakaguchi commented

Author

Owner

@davidsyoung commented on GitHub (Feb 4, 2025):

Watching this very closely. I have 9x3090's along with 256GB DDR4 RAM. Is it worth testing your current PR @Azure-Tang?

You may need bigger RAM for now, it requires 380GB RAM for my default optimization yaml…… Or you can try to modify yaml to use marlin expert to use more DRAM?

I can most definitely work on doing this and release once I get it working!

Ok，the 380GB is obtained by q4km v3 model. Just try it!

Trying with unsloth/DeepSeek-R1-GGUF-Q4_K_M once done building. Thank you!

@davidsyoung commented on GitHub (Feb 4, 2025): > > > > Watching this very closely. I have 9x3090's along with 256GB DDR4 RAM. Is it worth testing your current PR [@Azure-Tang](https://github.com/Azure-Tang)? > > > > > > > > > You may need bigger RAM for now, it requires 380GB RAM for my default optimization yaml…… Or you can try to modify yaml to use marlin expert to use more DRAM? > > > > > > I can most definitely work on doing this and release once I get it working! > > Ok，the 380GB is obtained by q4km v3 model. Just try it! Trying with unsloth/DeepSeek-R1-GGUF-Q4_K_M once done building. Thank you!

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Feb 4, 2025):

Hi everyone,
Exciting updates to share! After intensive development efforts, we are pleased to announce that ktransformers now supports Deepseek-V3 and R1 architectures. The working PR (#122 ) is currently undergoing final validation and is expected to merge into main branch within the next few days. You can build from source via the feat-DeepSeekV3 branch for early access.
Special thanks to our community for the continued support.

Fantastic! Going to try to get this working. Could you give a config/gguf that you recommend for this?

I'm currently attempting to launch with the following docker compose but not sure if I'm missing something. Could you give an example config/launch params for a Deepseek R1? Thank you very much for the hard work!

I'm getting

2025-02-04 16:31:26,879 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py[29]: Creating SQL tables
2025-02-04 16:31:26,997 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant
A new version of the following files was downloaded from https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat:
- configuration_deepseek.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
Traceback (most recent call last):
  File "/opt/conda/bin/ktransformers", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 114, in main
    create_interface(config=cfg, default_args=cfg)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface
    GlobalInterface.interface = BackendInterface(default_args)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 47, in __init__
    optimize_and_load_gguf(self.model, optimize_rule_path, gguf_path, config)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 128, in optimize_and_load_gguf
    inject(module, optimize_config, model_config, gguf_loader)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 31, in inject
    module_cls=getattr(__import__(import_module_name, fromlist=[""]), import_class_name)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/models.py", line 22, in <module>
    from ktransformers.operators.dynamic_attention import DynamicScaledDotProductAttention
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/dynamic_attention.py", line 19, in <module>
    from ktransformers.operators.cpuinfer import CPUInfer, CPUInferKVCache
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/cpuinfer.py", line 25, in <module>
    import cpuinfer_ext
ImportError: /opt/conda/lib/python3.10/site-packages/torch/lib/../../../.././libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /opt/conda/lib/python3.10/site-packages/cpuinfer_ext.cpython-310-x86_64-linux-gnu.so)

It's possible that my docker compose didn't fully rebuild from branch, forcing rebuild now

As developer usually I run/test project from code like this:

python ktransformers/local_chat.py \
  --model_path deepseek-ai/DeepSeek-R1 \
  --gguf_path /path/to/DeepSeek-R1-q4km.gguf \
  --max_new_tokens 500

Maybe try to install from source/conda can avoid this XD?

@Azure-Tang commented on GitHub (Feb 4, 2025): > > Hi everyone, > > Exciting updates to share! After intensive development efforts, we are pleased to announce that ktransformers now supports Deepseek-V3 and R1 architectures. The working PR ([#122](https://github.com/kvcache-ai/ktransformers/pull/122) ) is currently undergoing final validation and is expected to merge into main branch within the next few days. You can build from source via the feat-DeepSeekV3 branch for early access. > > Special thanks to our community for the continued support. > > Fantastic! Going to try to get this working. Could you give a config/gguf that you recommend for this? > > I'm currently attempting to launch with the following docker compose but not sure if I'm missing something. Could you give an example config/launch params for a Deepseek R1? Thank you very much for the hard work! > > I'm getting > > ``` > 2025-02-04 16:31:26,879 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py[29]: Creating SQL tables > 2025-02-04 16:31:26,997 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant > A new version of the following files was downloaded from https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat: > - configuration_deepseek.py > . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. > Traceback (most recent call last): > File "/opt/conda/bin/ktransformers", line 8, in <module> > sys.exit(main()) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 114, in main > create_interface(config=cfg, default_args=cfg) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface > GlobalInterface.interface = BackendInterface(default_args) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 47, in __init__ > optimize_and_load_gguf(self.model, optimize_rule_path, gguf_path, config) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 128, in optimize_and_load_gguf > inject(module, optimize_config, model_config, gguf_loader) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 31, in inject > module_cls=getattr(__import__(import_module_name, fromlist=[""]), import_class_name) > File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/models.py", line 22, in <module> > from ktransformers.operators.dynamic_attention import DynamicScaledDotProductAttention > File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/dynamic_attention.py", line 19, in <module> > from ktransformers.operators.cpuinfer import CPUInfer, CPUInferKVCache > File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/cpuinfer.py", line 25, in <module> > import cpuinfer_ext > ImportError: /opt/conda/lib/python3.10/site-packages/torch/lib/../../../.././libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /opt/conda/lib/python3.10/site-packages/cpuinfer_ext.cpython-310-x86_64-linux-gnu.so) > ``` > > It's possible that my docker compose didn't fully rebuild from branch, forcing rebuild now As developer usually I run/test project from code like this: ```shell python ktransformers/local_chat.py \ --model_path deepseek-ai/DeepSeek-R1 \ --gguf_path /path/to/DeepSeek-R1-q4km.gguf \ --max_new_tokens 500 ``` Maybe try to install from source/conda can avoid this XD?

bsakaguchi commented

Author

Owner

@davidsyoung commented on GitHub (Feb 4, 2025):

Hi everyone,
Exciting updates to share! After intensive development efforts, we are pleased to announce that ktransformers now supports Deepseek-V3 and R1 architectures. The working PR (#122 ) is currently undergoing final validation and is expected to merge into main branch within the next few days. You can build from source via the feat-DeepSeekV3 branch for early access.
Special thanks to our community for the continued support.

Fantastic! Going to try to get this working. Could you give a config/gguf that you recommend for this?
I'm currently attempting to launch with the following docker compose but not sure if I'm missing something. Could you give an example config/launch params for a Deepseek R1? Thank you very much for the hard work!
I'm getting
2025-02-04 16:31:26,879 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py[29]: Creating SQL tables
2025-02-04 16:31:26,997 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant
A new version of the following files was downloaded from https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat:
- configuration_deepseek.py
. Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
Traceback (most recent call last):
  File "/opt/conda/bin/ktransformers", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 114, in main
    create_interface(config=cfg, default_args=cfg)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface
    GlobalInterface.interface = BackendInterface(default_args)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 47, in __init__
    optimize_and_load_gguf(self.model, optimize_rule_path, gguf_path, config)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 128, in optimize_and_load_gguf
    inject(module, optimize_config, model_config, gguf_loader)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 31, in inject
    module_cls=getattr(__import__(import_module_name, fromlist=[""]), import_class_name)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/models.py", line 22, in <module>
    from ktransformers.operators.dynamic_attention import DynamicScaledDotProductAttention
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/dynamic_attention.py", line 19, in <module>
    from ktransformers.operators.cpuinfer import CPUInfer, CPUInferKVCache
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/cpuinfer.py", line 25, in <module>
    import cpuinfer_ext
ImportError: /opt/conda/lib/python3.10/site-packages/torch/lib/../../../.././libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /opt/conda/lib/python3.10/site-packages/cpuinfer_ext.cpython-310-x86_64-linux-gnu.so)
It's possible that my docker compose didn't fully rebuild from branch, forcing rebuild now
As developer usually I run/test project from code like this:

python ktransformers/local_chat.py
--model_path deepseek-ai/DeepSeek-R1
--gguf_path /path/to/DeepSeek-R1-q4km.gguf
--max_new_tokens 500
Maybe try to install from source/conda can avoid this XD?

I'm hoping that the Dockerfile contained will do an okay job at this? I'm thinking... I find it hard to build from source with conda with the server I run it on!

@davidsyoung commented on GitHub (Feb 4, 2025): > > > Hi everyone, > > > Exciting updates to share! After intensive development efforts, we are pleased to announce that ktransformers now supports Deepseek-V3 and R1 architectures. The working PR ([#122](https://github.com/kvcache-ai/ktransformers/pull/122) ) is currently undergoing final validation and is expected to merge into main branch within the next few days. You can build from source via the feat-DeepSeekV3 branch for early access. > > > Special thanks to our community for the continued support. > > > > > > Fantastic! Going to try to get this working. Could you give a config/gguf that you recommend for this? > > I'm currently attempting to launch with the following docker compose but not sure if I'm missing something. Could you give an example config/launch params for a Deepseek R1? Thank you very much for the hard work! > > I'm getting > > ``` > > 2025-02-04 16:31:26,879 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py[29]: Creating SQL tables > > 2025-02-04 16:31:26,997 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant > > A new version of the following files was downloaded from https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite-Chat: > > - configuration_deepseek.py > > . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. > > Traceback (most recent call last): > > File "/opt/conda/bin/ktransformers", line 8, in <module> > > sys.exit(main()) > > File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 114, in main > > create_interface(config=cfg, default_args=cfg) > > File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface > > GlobalInterface.interface = BackendInterface(default_args) > > File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 47, in __init__ > > optimize_and_load_gguf(self.model, optimize_rule_path, gguf_path, config) > > File "/opt/conda/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 128, in optimize_and_load_gguf > > inject(module, optimize_config, model_config, gguf_loader) > > File "/opt/conda/lib/python3.10/site-packages/ktransformers/optimize/optimize.py", line 31, in inject > > module_cls=getattr(__import__(import_module_name, fromlist=[""]), import_class_name) > > File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/models.py", line 22, in <module> > > from ktransformers.operators.dynamic_attention import DynamicScaledDotProductAttention > > File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/dynamic_attention.py", line 19, in <module> > > from ktransformers.operators.cpuinfer import CPUInfer, CPUInferKVCache > > File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/cpuinfer.py", line 25, in <module> > > import cpuinfer_ext > > ImportError: /opt/conda/lib/python3.10/site-packages/torch/lib/../../../.././libstdc++.so.6: version `GLIBCXX_3.4.30' not found (required by /opt/conda/lib/python3.10/site-packages/cpuinfer_ext.cpython-310-x86_64-linux-gnu.so) > > ``` > > > > > > > > > > > > > > > > > > > > > > > > It's possible that my docker compose didn't fully rebuild from branch, forcing rebuild now > > As developer usually I run/test project from code like this: > > python ktransformers/local_chat.py \ > --model_path deepseek-ai/DeepSeek-R1 \ > --gguf_path /path/to/DeepSeek-R1-q4km.gguf \ > --max_new_tokens 500 > Maybe try to install from source/conda can avoid this XD? I'm hoping that the Dockerfile contained will do an okay job at this? I'm thinking... I find it hard to build from source with conda with the server I run it on!

bsakaguchi commented

Author

Owner

@davidsyoung commented on GitHub (Feb 4, 2025):

Okay built and ran with these commands:

      - "--port"
      - "10002"
      - "--model_path"
      - "deepseek-ai/DeepSeek-R1"
      - "--gguf_path"
      - "/models/unsloth_DeepSeek-R1-GGUF-Q4_K_M/"

Output:

ktransformers  | 2025-02-04 17:20:46,128 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py[29]: Creating SQL tables
ktransformers  | 2025-02-04 17:20:46,261 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant
ktransformers  | A new version of the following files was downloaded from https://huggingface.co/deepseek-ai/DeepSeek-R1:
ktransformers  | - configuration_deepseek.py
ktransformers  | . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision.
ktransformers  | Traceback (most recent call last):
ktransformers  |   File "/opt/conda/bin/ktransformers", line 8, in <module>
ktransformers  |     sys.exit(main())
ktransformers  |   File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 114, in main
ktransformers  |     create_interface(config=cfg, default_args=cfg)
ktransformers  |   File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface
ktransformers  |     GlobalInterface.interface = BackendInterface(default_args)
ktransformers  |   File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 33, in __init__
ktransformers  |     self.model = custom_models[config.architectures[0]](config)
ktransformers  | KeyError: 'DeepseekV3ForCausalLM'
ktransformers exited with code 1

@davidsyoung commented on GitHub (Feb 4, 2025): Okay built and ran with these commands: ``` - "--port" - "10002" - "--model_path" - "deepseek-ai/DeepSeek-R1" - "--gguf_path" - "/models/unsloth_DeepSeek-R1-GGUF-Q4_K_M/" ``` Output: ``` ktransformers | 2025-02-04 17:20:46,128 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py[29]: Creating SQL tables ktransformers | 2025-02-04 17:20:46,261 INFO /opt/conda/lib/python3.10/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant ktransformers | A new version of the following files was downloaded from https://huggingface.co/deepseek-ai/DeepSeek-R1: ktransformers | - configuration_deepseek.py ktransformers | . Make sure to double-check they do not contain any added malicious code. To avoid downloading new versions of the code file, you can pin a revision. ktransformers | Traceback (most recent call last): ktransformers | File "/opt/conda/bin/ktransformers", line 8, in <module> ktransformers | sys.exit(main()) ktransformers | File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 114, in main ktransformers | create_interface(config=cfg, default_args=cfg) ktransformers | File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface ktransformers | GlobalInterface.interface = BackendInterface(default_args) ktransformers | File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 33, in __init__ ktransformers | self.model = custom_models[config.architectures[0]](config) ktransformers | KeyError: 'DeepseekV3ForCausalLM' ktransformers exited with code 1 ```

bsakaguchi commented

Author

Owner

@davidsyoung commented on GitHub (Feb 4, 2025):

Ignore me, I believe the dockerfile isn't referencing the feature branch and building from main instead. Will re-build and try again

@davidsyoung commented on GitHub (Feb 4, 2025): Ignore me, I believe the dockerfile isn't referencing the feature branch and building from main instead. Will re-build and try again

bsakaguchi commented

Author

Owner

@ELigoP commented on GitHub (Feb 4, 2025):

Managed to run DeepSeek-V3 with

ktransformers --model_path deepseek-ai/DeepSeek-V3 --gguf_path /mnt/models/gguf/bullerwins/DeepSeek-V3-GGUF --port 10002 --cpu_infer 32

got SegFault due to OOM, after few responses, most probably.

But couldn't run R1 with IQ3_M with

ktransformers --model_path deepseek-ai/DeepSeek-V3 --gguf_path /mnt/models/gguf/bartowski/DeepSeek-R1-GGUF/DeepSeek-R1-IQ3_M --port 10002 --cpu_infer 32

probably expected.
Also needed flash-attn in requirements-local_chat.txt and no build isolation to install.

@ELigoP commented on GitHub (Feb 4, 2025): Managed to run DeepSeek-V3 with ```sh ktransformers --model_path deepseek-ai/DeepSeek-V3 --gguf_path /mnt/models/gguf/bullerwins/DeepSeek-V3-GGUF --port 10002 --cpu_infer 32 ``` - got SegFault due to OOM, after few responses, most probably. - But couldn't run R1 with IQ3_M with ```sh ktransformers --model_path deepseek-ai/DeepSeek-V3 --gguf_path /mnt/models/gguf/bartowski/DeepSeek-R1-GGUF/DeepSeek-R1-IQ3_M --port 10002 --cpu_infer 32 ``` - probably expected. - Also needed `flash-attn` in `requirements-local_chat.txt` and no build isolation to install.

bsakaguchi commented

Author

Owner

@Nondzu commented on GitHub (Feb 4, 2025):

@Azure-Tang Great job, Thanks !
Do you have any update about CPU build without CUDA?

@Nondzu commented on GitHub (Feb 4, 2025): @Azure-Tang Great job, Thanks ! Do you have any update about CPU build without CUDA?

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Feb 4, 2025):

Managed to run DeepSeek-V3 with

ktransformers --model_path deepseek-ai/DeepSeek-V3 --gguf_path /mnt/models/gguf/bullerwins/DeepSeek-V3-GGUF --port 10002 --cpu_infer 32

got SegFault due to OOM, after few responses, most probably.

But couldn't run R1 with IQ3_M with

ktransformers --model_path deepseek-ai/DeepSeek-V3 --gguf_path /mnt/models/gguf/bartowski/DeepSeek-R1-GGUF/DeepSeek-R1-IQ3_M --port 10002 --cpu_infer 32

probably expected.

Also needed flash-attn in requirements-local_chat.txt and no build isolation to install.

Sorry, IQ3_M support is still under development and marked as a todo item.

@Azure-Tang commented on GitHub (Feb 4, 2025): > Managed to run DeepSeek-V3 with > > ktransformers --model_path deepseek-ai/DeepSeek-V3 --gguf_path /mnt/models/gguf/bullerwins/DeepSeek-V3-GGUF --port 10002 --cpu_infer 32 > * got SegFault due to OOM, after few responses, most probably. > > But couldn't run R1 with IQ3_M with > > ktransformers --model_path deepseek-ai/DeepSeek-V3 --gguf_path /mnt/models/gguf/bartowski/DeepSeek-R1-GGUF/DeepSeek-R1-IQ3_M --port 10002 --cpu_infer 32 > * probably expected. > * Also needed `flash-attn` in `requirements-local_chat.txt` and no build isolation to install. Sorry, IQ3_M support is still under development and marked as a todo item.

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Feb 4, 2025):

@Azure-Tang Great job, Thanks ! Do you have any update about CPU build without CUDA?

You could try removing the "KTransformersOps" dependency, as it requires CUDA. However, if you're not using a GPU, have you considered using llamacpp directly? The main advantage of ktransformers is its ability to offload high arithmetic intensity operations to the GPU while keeping others on the CPU.

@Azure-Tang commented on GitHub (Feb 4, 2025): > [@Azure-Tang](https://github.com/Azure-Tang) Great job, Thanks ! Do you have any update about CPU build without CUDA? You could try removing the ["KTransformersOps"](https://github.com/kvcache-ai/ktransformers/blob/de7e892f72ada9dd904517d9889ff3c68603071d/setup.py#L298) dependency, as it requires CUDA. However, if you're not using a GPU, have you considered using llamacpp directly? The main advantage of ktransformers is its ability to offload high arithmetic intensity operations to the GPU while keeping others on the CPU.

bsakaguchi commented

Author

Owner

@trilog-inc commented on GitHub (Feb 4, 2025):

Great news!

Ran a simple test with

python ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/home_extend/models/bartowski_DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --max_new_tokens 5000

prompt: test

Answer:

<think>
Okay, so I got a message that just says "test". Hmm, maybe the user is checking if the system is working? I mean, sometimes people send "test" to see if there's a response, especially in new or unfamiliar platforms. But maybe they actually have a question and accidentally sent "test" first? Or perhaps they were interrupted while typing.

I need to figure out the best way to respond. If they're testing, a simple reply like "Test received! How can I help you today?" might work. That invites them to ask their real question. But maybe they just want to confirm the system is active, so acknowledging the test is enough. Alternatively, they could be new and unsure how to start, so providing some guidance or examples might be helpful.

Wait, there's also the possibility that "test" was a placeholder, and they meant to replace it with an actual query but forgot. In that case, prompting them again makes sense. I don't want to assume it's a mistake, though. Better to respond neutrally and encourage them to ask their question.

Another angle: sometimes users send "test" to check response time or the bot's language capabilities. If that's the case, a prompt reply demonstrating understanding and willingness to assist would address that test effectively.

In summary, the best approach is to acknowledge the test message and invite the user to proceed with their actual query. This covers multiple potential scenarios without making unwarranted assumptions.
</think>
Test received! How can I assist you today? Feel free to ask any questions or share what you need help with. 😊

with the following performance:
2025-02-04 14:48:08,724 INFO /mnt/home_extend/llm/ktransformers/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 1.5659206588635783, decode 2.8615367519258315. Time(s): tokenize 0.08403253555297852, prefill 2.554407835006714, decode 114.97318696975708

However, I am at the upper limit of my current setup ( 384GB DDR5, 3x3090 but only 1 being used ). top reports a 98.3% memory usage just when at the prompt. nvidia-smi settles around 12870MiB.

@trilog-inc commented on GitHub (Feb 4, 2025): Great news! Ran a simple test with python ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/home_extend/models/bartowski_DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --max_new_tokens 5000 prompt: test Answer: ``` <think> Okay, so I got a message that just says "test". Hmm, maybe the user is checking if the system is working? I mean, sometimes people send "test" to see if there's a response, especially in new or unfamiliar platforms. But maybe they actually have a question and accidentally sent "test" first? Or perhaps they were interrupted while typing. I need to figure out the best way to respond. If they're testing, a simple reply like "Test received! How can I help you today?" might work. That invites them to ask their real question. But maybe they just want to confirm the system is active, so acknowledging the test is enough. Alternatively, they could be new and unsure how to start, so providing some guidance or examples might be helpful. Wait, there's also the possibility that "test" was a placeholder, and they meant to replace it with an actual query but forgot. In that case, prompting them again makes sense. I don't want to assume it's a mistake, though. Better to respond neutrally and encourage them to ask their question. Another angle: sometimes users send "test" to check response time or the bot's language capabilities. If that's the case, a prompt reply demonstrating understanding and willingness to assist would address that test effectively. In summary, the best approach is to acknowledge the test message and invite the user to proceed with their actual query. This covers multiple potential scenarios without making unwarranted assumptions. </think> Test received! How can I assist you today? Feel free to ask any questions or share what you need help with. 😊 ``` with the following performance: 2025-02-04 14:48:08,724 INFO /mnt/home_extend/llm/ktransformers/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 1.5659206588635783, decode 2.8615367519258315. Time(s): tokenize 0.08403253555297852, prefill 2.554407835006714, decode 114.97318696975708 However, I am at the upper limit of my current setup ( 384GB DDR5, 3x3090 but only 1 being used ). top reports a 98.3% memory usage just when at the prompt. nvidia-smi settles around 12870MiB.

bsakaguchi commented

Author

Owner

@davidsyoung commented on GitHub (Feb 4, 2025):

Ok, great news - I got it running! Had to do some finicky stuff to make it work in Docker.

Next stage. Do you have any recommendations for how I would make use of the 9x3090's I have instead of putting into CPU/DRAM @Azure-Tang? As I don't have enough DRAM to run it without.

I'm going to take a look at the multi gpu config file but thought it'd be useful to get some direction here!

@davidsyoung commented on GitHub (Feb 4, 2025): Ok, great news - I got it running! Had to do some finicky stuff to make it work in Docker. Next stage. Do you have any recommendations for how I would make use of the 9x3090's I have instead of putting into CPU/DRAM @Azure-Tang? As I don't have enough DRAM to run it without. I'm going to take a look at the multi gpu config file but thought it'd be useful to get some direction here!

bsakaguchi commented

Author

Owner

@trilog-inc commented on GitHub (Feb 5, 2025):

I was getting a lot of "Segmentation fault" when using the Q4_K_M model, so i switched to the Q4_K_XS and it did not help. For example using the following prompt: "Use python and the fastapi library to write a generic openAI compatible endpoint for /v1/models" ( so i can use this library with open WebUI! ).

I thought it was an OOM error at first, but with the Q4_K_XS model, there is still 43GB available on the system.

Could there be something else causing this? What can i do to root cause this issue?

@trilog-inc commented on GitHub (Feb 5, 2025): I was getting a lot of "Segmentation fault" when using the Q4_K_M model, so i switched to the Q4_K_XS and it did not help. For example using the following prompt: "Use python and the fastapi library to write a generic openAI compatible endpoint for /v1/models" ( so i can use this library with open WebUI! ). I thought it was an OOM error at first, but with the Q4_K_XS model, there is still 43GB available on the system. Could there be something else causing this? What can i do to root cause this issue?

bsakaguchi commented

Author

Owner

@ELigoP commented on GitHub (Feb 5, 2025):

Maybe that's default context length which is too large? Try --total_context 1024 and --max_new_tokens 1024 or smth like first.

@ELigoP commented on GitHub (Feb 5, 2025): Maybe that's default context length which is too large? Try `--total_context 1024` and `--max_new_tokens 1024` or smth like first.

bsakaguchi commented

Author

Owner

@RodriMora commented on GitHub (Feb 5, 2025):

Hi!

I'm testing but get this error:

Chat: what it's 2+2?
2025-02-05 14:04:05,330 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/transformers.py[192]: get input ids of shape torch.Size([1, 11])
2025-02-05 14:04:05,330 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[121]: input_ids: torch.Size([1, 11])
2025-02-05 14:04:05,331 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[147]: cache position: 0 to 11
fish: Job 1, 'python ktransformers/local_chat…' terminated by signal SIGSEGV (Address boundary error)

Running using this:
python ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/ubuntuai/models/DeepSeek-R1-Q3_K_M --max_new_tokens 1000 --total_context 2000

I haven't used ktransformers in the past, do I have to point to the folder containing all the .gguf's shards right? not the first 00001.gguf?

System:
EPYC 7402 24c/48t
512GB Ram 3200Mhz
4x3090

I installed manually everything, and I use python venv instead of conda:

git clone https://github.com/kvcache-ai/ktransformers
cd ktransformers
python3.11 -m venv .venv
source ./.venv/bin/activate.fish #I'm using fish instead of bash
gh pr checkout 122
git submodule init
git submodule update
python -m pip install torch packaging ninja
rm -rf ktransformers/ktransformers_ext/build
rm -rf ktransformers/ktransformers_ext/cuda/build
rm -rf ktransformers/ktransformers_ext/cuda/dist
rm -rf ktransformers/ktransformers_ext/cuda/*.egg-info
python -m pip install wheels
python -m pip install -r requirements-local_chat.txt
KTRANSFORMERS_FORCE_BUILD=TRUE python -m pip install . --no-build-isolation
python -m pip install flash-attn --no-build-isolation

@RodriMora commented on GitHub (Feb 5, 2025): Hi! I'm testing but get this error: ``` Chat: what it's 2+2? 2025-02-05 14:04:05,330 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/transformers.py[192]: get input ids of shape torch.Size([1, 11]) 2025-02-05 14:04:05,330 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[121]: input_ids: torch.Size([1, 11]) 2025-02-05 14:04:05,331 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[147]: cache position: 0 to 11 fish: Job 1, 'python ktransformers/local_chat…' terminated by signal SIGSEGV (Address boundary error) ``` Running using this: `python ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/ubuntuai/models/DeepSeek-R1-Q3_K_M --max_new_tokens 1000 --total_context 2000` I haven't used ktransformers in the past, do I have to point to the folder containing all the .gguf's shards right? not the first 00001.gguf? System: EPYC 7402 24c/48t 512GB Ram 3200Mhz 4x3090 I installed manually everything, and I use python venv instead of conda: ``` git clone https://github.com/kvcache-ai/ktransformers cd ktransformers python3.11 -m venv .venv source ./.venv/bin/activate.fish #I'm using fish instead of bash gh pr checkout 122 git submodule init git submodule update python -m pip install torch packaging ninja rm -rf ktransformers/ktransformers_ext/build rm -rf ktransformers/ktransformers_ext/cuda/build rm -rf ktransformers/ktransformers_ext/cuda/dist rm -rf ktransformers/ktransformers_ext/cuda/*.egg-info python -m pip install wheels python -m pip install -r requirements-local_chat.txt KTRANSFORMERS_FORCE_BUILD=TRUE python -m pip install . --no-build-isolation python -m pip install flash-attn --no-build-isolation ```

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Feb 5, 2025):

Ok, great news - I got it running! Had to do some finicky stuff to make it work in Docker.

Next stage. Do you have any recommendations for how I would make use of the 9x3090's I have instead of putting into CPU/DRAM @Azure-Tang? As I don't have enough DRAM to run it without.

I'm going to take a look at the multi gpu config file but thought it'd be useful to get some direction here!

You could try injecting some layers' experts and using Marlin exports to replace KExpertsCPU as the backend for KTransformersExperts. A detailed tutorial can be found here. You may refer to multi-gpu yaml to learn how to replace some layers while keeping others on cpu.

That said, I haven’t tested this with the v3 model yet, so there might be some bugs. If you run into any issues, feel free to ping us!

@Azure-Tang commented on GitHub (Feb 5, 2025): > Ok, great news - I got it running! Had to do some finicky stuff to make it work in Docker. > > Next stage. Do you have any recommendations for how I would make use of the 9x3090's I have instead of putting into CPU/DRAM [@Azure-Tang](https://github.com/Azure-Tang)? As I don't have enough DRAM to run it without. > > I'm going to take a look at the multi gpu config file but thought it'd be useful to get some direction here! You could try injecting some layers' experts and using Marlin exports to replace KExpertsCPU as the backend for KTransformersExperts. A detailed tutorial can be found [here](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/injection_tutorial.md). You may refer to [multi-gpu yaml](https://github.com/kvcache-ai/ktransformers/blob/ee24a27001d8eadd798dfea5983b16a4f49fdca8/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml) to learn how to replace some layers while keeping others on cpu. That said, I haven’t tested this with the v3 model yet, so there might be some bugs. If you run into any issues, feel free to ping us!

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Feb 5, 2025):

I was getting a lot of "Segmentation fault" when using the Q4_K_M model, so i switched to the Q4_K_XS and it did not help. For example using the following prompt: "Use python and the fastapi library to write a generic openAI compatible endpoint for /v1/models" ( so i can use this library with open WebUI! ).

I thought it was an OOM error at first, but with the Q4_K_XS model, there is still 43GB available on the system.

Could there be something else causing this? What can i do to root cause this issue?

Did this works? ⬇️

Maybe that's default context length which is too large? Try --total_context 1024 and --max_new_tokens 1024 or smth like first.

@Azure-Tang commented on GitHub (Feb 5, 2025): > I was getting a lot of "Segmentation fault" when using the Q4_K_M model, so i switched to the Q4_K_XS and it did not help. For example using the following prompt: "Use python and the fastapi library to write a generic openAI compatible endpoint for /v1/models" ( so i can use this library with open WebUI! ). > > I thought it was an OOM error at first, but with the Q4_K_XS model, there is still 43GB available on the system. > > Could there be something else causing this? What can i do to root cause this issue? Did this works? ⬇️ > Maybe that's default context length which is too large? Try `--total_context 1024` and `--max_new_tokens 1024` or smth like first.

bsakaguchi commented

Author

Owner

@trilog-inc commented on GitHub (Feb 5, 2025):

I was getting a lot of "Segmentation fault" when using the Q4_K_M model, so i switched to the Q4_K_XS and it did not help. For example using the following prompt: "Use python and the fastapi library to write a generic openAI compatible endpoint for /v1/models" ( so i can use this library with open WebUI! ).
I thought it was an OOM error at first, but with the Q4_K_XS model, there is still 43GB available on the system.
Could there be something else causing this? What can i do to root cause this issue?

Did this works? ⬇️

Maybe that's default context length which is too large? Try --total_context 1024 and --max_new_tokens 1024 or smth like first.

It did not unfortunately. Here is the debug output when using --max_new_tokens 512 --total_context 512

2025-02-05 11:15:51,400 WARNING /mnt/home_extend/llm/ktransformers/ktransformers/server/backend/interfaces/transformers.py[189]: num_unequal_elements: 15
2025-02-05 11:15:51,400 DEBUG /mnt/home_extend/llm/ktransformers/ktransformers/server/backend/interfaces/transformers.py[192]: get input ids of shape torch.Size([1, 20])
2025-02-05 11:15:51,401 DEBUG /mnt/home_extend/llm/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[121]: input_ids: torch.Size([1, 20])
2025-02-05 11:15:51,401 DEBUG /mnt/home_extend/llm/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[137]: generate_ids: torch.Size([1, 517])
2025-02-05 11:15:51,401 DEBUG /mnt/home_extend/llm/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[147]: cache position: 19 to 39

With this prompt "Use python and the fastapi library to write a generic openAI compatible endpoint for /v1/models"

@trilog-inc commented on GitHub (Feb 5, 2025): > > I was getting a lot of "Segmentation fault" when using the Q4_K_M model, so i switched to the Q4_K_XS and it did not help. For example using the following prompt: "Use python and the fastapi library to write a generic openAI compatible endpoint for /v1/models" ( so i can use this library with open WebUI! ). > > I thought it was an OOM error at first, but with the Q4_K_XS model, there is still 43GB available on the system. > > Could there be something else causing this? What can i do to root cause this issue? > > Did this works? ⬇️ > > > Maybe that's default context length which is too large? Try `--total_context 1024` and `--max_new_tokens 1024` or smth like first. It did not unfortunately. Here is the debug output when using --max_new_tokens 512 --total_context 512 ``` 2025-02-05 11:15:51,400 WARNING /mnt/home_extend/llm/ktransformers/ktransformers/server/backend/interfaces/transformers.py[189]: num_unequal_elements: 15 2025-02-05 11:15:51,400 DEBUG /mnt/home_extend/llm/ktransformers/ktransformers/server/backend/interfaces/transformers.py[192]: get input ids of shape torch.Size([1, 20]) 2025-02-05 11:15:51,401 DEBUG /mnt/home_extend/llm/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[121]: input_ids: torch.Size([1, 20]) 2025-02-05 11:15:51,401 DEBUG /mnt/home_extend/llm/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[137]: generate_ids: torch.Size([1, 517]) 2025-02-05 11:15:51,401 DEBUG /mnt/home_extend/llm/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[147]: cache position: 19 to 39 ``` With this prompt "Use python and the fastapi library to write a generic openAI compatible endpoint for /v1/models"

bsakaguchi commented

Author

Owner

@ELigoP commented on GitHub (Feb 5, 2025):

Hi!

I'm testing but get this error:

Chat: what it's 2+2?
2025-02-05 14:04:05,330 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/transformers.py[192]: get input ids of shape torch.Size([1, 11])
2025-02-05 14:04:05,330 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[121]: input_ids: torch.Size([1, 11])
2025-02-05 14:04:05,331 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[147]: cache position: 0 to 11
fish: Job 1, 'python ktransformers/local_chat…' terminated by signal SIGSEGV (Address boundary error)

Running using this: python ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/ubuntuai/models/DeepSeek-R1-Q3_K_M --max_new_tokens 1000 --total_context 2000

I haven't used ktransformers in the past, do I have to point to the folder containing all the .gguf's shards right? not the first 00001.gguf?

System: EPYC 7402 24c/48t 512GB Ram 3200Mhz 4x3090

I installed manually everything, and I use python venv instead of conda:

git clone https://github.com/kvcache-ai/ktransformers
cd ktransformers
python3.11 -m venv .venv
source ./.venv/bin/activate.fish #I'm using fish instead of bash
gh pr checkout 122
git submodule init
git submodule update
python -m pip install torch packaging ninja
rm -rf ktransformers/ktransformers_ext/build
rm -rf ktransformers/ktransformers_ext/cuda/build
rm -rf ktransformers/ktransformers_ext/cuda/dist
rm -rf ktransformers/ktransformers_ext/cuda/*.egg-info
python -m pip install wheels
python -m pip install -r requirements-local_chat.txt
KTRANSFORMERS_FORCE_BUILD=TRUE python -m pip install . --no-build-isolation
python -m pip install flash-attn --no-build-isolation

Installation guide recommends conda, but I also wanted problems :) had to work it out with pyenv + venv.

First install pyenv from official guide. Then run pyenv install 3.11.6 and pyenv activate 3.11.6. Then this is my diff that I worked out:

diff --git a/install.sh b/install.sh
index ffb7aca..e43535c 100644
--- a/install.sh
+++ b/install.sh
@@ -11,5 +11,5 @@ echo "Installing python dependencies from requirements.txt"
 pip install -r requirements-local_chat.txt
 
 echo "Installing ktransformers"
-KTRANSFORMERS_FORCE_BUILD=TRUE pip install . --no-build-isolation
-echo "Installation completed successfully"
\ No newline at end of file
+KTRANSFORMERS_FORCE_BUILD=TRUE pip install . #--no-build-isolation
+echo "Installation completed successfully"
diff --git a/requirements-local_chat.txt b/requirements-local_chat.txt
index 50b1f65..3320fb0 100644
--- a/requirements-local_chat.txt
+++ b/requirements-local_chat.txt
@@ -2,6 +2,7 @@ fire
 transformers
 numpy
 torch>=2.3.0
+flash-attn
 packaging
 cpufeature
-protobuf
\ No newline at end of file
+protobuf

then run bash install.sh.

@ELigoP commented on GitHub (Feb 5, 2025): > Hi! > > I'm testing but get this error: > > ``` > Chat: what it's 2+2? > 2025-02-05 14:04:05,330 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/transformers.py[192]: get input ids of shape torch.Size([1, 11]) > 2025-02-05 14:04:05,330 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[121]: input_ids: torch.Size([1, 11]) > 2025-02-05 14:04:05,331 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[147]: cache position: 0 to 11 > fish: Job 1, 'python ktransformers/local_chat…' terminated by signal SIGSEGV (Address boundary error) > ``` > > Running using this: `python ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/ubuntuai/models/DeepSeek-R1-Q3_K_M --max_new_tokens 1000 --total_context 2000` > > I haven't used ktransformers in the past, do I have to point to the folder containing all the .gguf's shards right? not the first 00001.gguf? > > System: EPYC 7402 24c/48t 512GB Ram 3200Mhz 4x3090 > > I installed manually everything, and I use python venv instead of conda: > > ``` > git clone https://github.com/kvcache-ai/ktransformers > cd ktransformers > python3.11 -m venv .venv > source ./.venv/bin/activate.fish #I'm using fish instead of bash > gh pr checkout 122 > git submodule init > git submodule update > python -m pip install torch packaging ninja > rm -rf ktransformers/ktransformers_ext/build > rm -rf ktransformers/ktransformers_ext/cuda/build > rm -rf ktransformers/ktransformers_ext/cuda/dist > rm -rf ktransformers/ktransformers_ext/cuda/*.egg-info > python -m pip install wheels > python -m pip install -r requirements-local_chat.txt > KTRANSFORMERS_FORCE_BUILD=TRUE python -m pip install . --no-build-isolation > python -m pip install flash-attn --no-build-isolation > ``` Installation guide recommends conda, but I also wanted problems :) had to work it out with `pyenv` + `venv`. First install `pyenv` from official guide. Then run `pyenv install 3.11.6` and `pyenv activate 3.11.6`. Then this is my diff that I worked out: ```diff diff --git a/install.sh b/install.sh index ffb7aca..e43535c 100644 --- a/install.sh +++ b/install.sh @@ -11,5 +11,5 @@ echo "Installing python dependencies from requirements.txt" pip install -r requirements-local_chat.txt echo "Installing ktransformers" -KTRANSFORMERS_FORCE_BUILD=TRUE pip install . --no-build-isolation -echo "Installation completed successfully" \ No newline at end of file +KTRANSFORMERS_FORCE_BUILD=TRUE pip install . #--no-build-isolation +echo "Installation completed successfully" diff --git a/requirements-local_chat.txt b/requirements-local_chat.txt index 50b1f65..3320fb0 100644 --- a/requirements-local_chat.txt +++ b/requirements-local_chat.txt @@ -2,6 +2,7 @@ fire transformers numpy torch>=2.3.0 +flash-attn packaging cpufeature -protobuf \ No newline at end of file +protobuf ``` then run `bash install.sh`.

bsakaguchi commented

Author

Owner

@Nottlespike commented on GitHub (Feb 5, 2025):

I was getting a lot of "Segmentation fault" when using the Q4_K_M model, so i switched to the Q4_K_XS and it did not help. For example using the following prompt: "Use python and the fastapi library to write a generic openAI compatible endpoint for /v1/models" ( so i can use this library with open WebUI! ).
I thought it was an OOM error at first, but with the Q4_K_XS model, there is still 43GB available on the system.
Could there be something else causing this? What can i do to root cause this issue?

Did this works? ⬇️

Maybe that's default context length which is too large? Try --total_context 1024 and --max_new_tokens 1024 or smth like first.

I've been messing around with GDB to identify exactly where the segfault is occuring and from my testing this is my failure point with this command python ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path ./DeepSeek-R1-GGUF --total_context 8192 --max_new_tokens 8192:

2025-02-05 10:57:01,765 DEBUG /home/kearm/ktransformers/ktransformers/server/backend/interfaces/transformers.py[192]: get input ids of shape torch.Size([1, 54])

2025-02-05 10:57:01,765 DEBUG /home/kearm/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[121]: input_ids: torch.Size([1, 54])

2025-02-05 10:57:01,766 DEBUG /home/kearm/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[147]: cache position: 0 to 54

Thread 64 "python" received signal SIGSEGV, Segmentation fault.

[Switching to Thread 0x7fc98b4006c0 (LWP 834120)]

0x00007fc9a2454b70 in ?? () from /home/kearm/ktransformers/.conda/lib/python3.11/site-packages/cpuinfer_ext.cpython-311-x86_64-linux-gnu.so

When running a backtrace to get some more debugging info I am getting this output:

(gdb) bt

#0  0x00007fc9a2454b70 in ?? () from /home/kearm/ktransformers/.conda/lib/python3.11/site-packages/cpuinfer_ext.cpython-311-x86_64-linux-gnu.so

#1  0x00007fc9a24a4be6 in ?? () from /home/kearm/ktransformers/.conda/lib/python3.11/site-packages/cpuinfer_ext.cpython-311-x86_64-linux-gnu.so

#2  0x00007fc9a245224c in ?? () from /home/kearm/ktransformers/.conda/lib/python3.11/site-packages/cpuinfer_ext.cpython-311-x86_64-linux-gnu.so

#3  0x00007fc9a2448a78 in ?? () from /home/kearm/ktransformers/.conda/lib/python3.11/site-packages/cpuinfer_ext.cpython-311-x86_64-linux-gnu.so

#4  0x00007fc9a2448c1b in ?? () from /home/kearm/ktransformers/.conda/lib/python3.11/site-packages/cpuinfer_ext.cpython-311-x86_64-linux-gnu.so

#5  0x00007ffff54f1b65 in std::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:104

#6  0x00007ffff7c9ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447

#7  0x00007ffff7d29c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78

This led me to rebuild the branch from scratch but this error recurred exactly the same way. To further debug I used valgrind and this is the summary of many many issues with memory:

==873402== LEAK SUMMARY:
==873402==    definitely lost: 720 bytes in 9 blocks
==873402==    indirectly lost: 0 bytes in 0 blocks
==873402==      possibly lost: 2,522,685 bytes in 10,914 blocks
==873402==    still reachable: 41,358,203 bytes in 134,376 blocks
==873402==                       of which reachable via heuristic:
==873402==                         stdstring          : 529,187 bytes in 13,894 blocks
==873402==                         newarray           : 26 bytes in 1 blocks
==873402==         suppressed: 0 bytes in 0 blocks
==873402== Reachable blocks (those to which a pointer was found) are not shown.
==873402== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==873402== 
==873402== Use --track-origins=yes to see where uninitialised values come from
==873402== For lists of detected and suppressed errors, rerun with: -s
==873402== ERROR SUMMARY: 683546 errors from 547 contexts (suppressed: 0 from 0)

I am on a 4x3090 Ti system with a ThreadRipper Pro 5965WX, 24 cores, and 8x128GB sticks of Samsung ECC DDR4 DRAM running Ubuntu 24.04 LTS. When running this model with the latest build of llama.cpp it is working quite well.

@Nottlespike commented on GitHub (Feb 5, 2025): > > I was getting a lot of "Segmentation fault" when using the Q4_K_M model, so i switched to the Q4_K_XS and it did not help. For example using the following prompt: "Use python and the fastapi library to write a generic openAI compatible endpoint for /v1/models" ( so i can use this library with open WebUI! ). > > I thought it was an OOM error at first, but with the Q4_K_XS model, there is still 43GB available on the system. > > Could there be something else causing this? What can i do to root cause this issue? > > Did this works? ⬇️ > > > Maybe that's default context length which is too large? Try `--total_context 1024` and `--max_new_tokens 1024` or smth like first. I've been messing around with GDB to identify exactly where the segfault is occuring and from my testing this is my failure point with this command `python ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path ./DeepSeek-R1-GGUF --total_context 8192 --max_new_tokens 8192`: ``` 2025-02-05 10:57:01,765 DEBUG /home/kearm/ktransformers/ktransformers/server/backend/interfaces/transformers.py[192]: get input ids of shape torch.Size([1, 54]) 2025-02-05 10:57:01,765 DEBUG /home/kearm/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[121]: input_ids: torch.Size([1, 54]) 2025-02-05 10:57:01,766 DEBUG /home/kearm/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[147]: cache position: 0 to 54 Thread 64 "python" received signal SIGSEGV, Segmentation fault. [Switching to Thread 0x7fc98b4006c0 (LWP 834120)] 0x00007fc9a2454b70 in ?? () from /home/kearm/ktransformers/.conda/lib/python3.11/site-packages/cpuinfer_ext.cpython-311-x86_64-linux-gnu.so ``` When running a backtrace to get some more debugging info I am getting this output: ``` (gdb) bt #0 0x00007fc9a2454b70 in ?? () from /home/kearm/ktransformers/.conda/lib/python3.11/site-packages/cpuinfer_ext.cpython-311-x86_64-linux-gnu.so #1 0x00007fc9a24a4be6 in ?? () from /home/kearm/ktransformers/.conda/lib/python3.11/site-packages/cpuinfer_ext.cpython-311-x86_64-linux-gnu.so #2 0x00007fc9a245224c in ?? () from /home/kearm/ktransformers/.conda/lib/python3.11/site-packages/cpuinfer_ext.cpython-311-x86_64-linux-gnu.so #3 0x00007fc9a2448a78 in ?? () from /home/kearm/ktransformers/.conda/lib/python3.11/site-packages/cpuinfer_ext.cpython-311-x86_64-linux-gnu.so #4 0x00007fc9a2448c1b in ?? () from /home/kearm/ktransformers/.conda/lib/python3.11/site-packages/cpuinfer_ext.cpython-311-x86_64-linux-gnu.so #5 0x00007ffff54f1b65 in std::execute_native_thread_routine (__p=<optimized out>) at ../../../../../libstdc++-v3/src/c++11/thread.cc:104 #6 0x00007ffff7c9ca94 in start_thread (arg=<optimized out>) at ./nptl/pthread_create.c:447 #7 0x00007ffff7d29c3c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:78 ``` This led me to rebuild the branch from scratch but this error recurred exactly the same way. To further debug I used valgrind and this is the summary of many many issues with memory: ``` ==873402== LEAK SUMMARY: ==873402== definitely lost: 720 bytes in 9 blocks ==873402== indirectly lost: 0 bytes in 0 blocks ==873402== possibly lost: 2,522,685 bytes in 10,914 blocks ==873402== still reachable: 41,358,203 bytes in 134,376 blocks ==873402== of which reachable via heuristic: ==873402== stdstring : 529,187 bytes in 13,894 blocks ==873402== newarray : 26 bytes in 1 blocks ==873402== suppressed: 0 bytes in 0 blocks ==873402== Reachable blocks (those to which a pointer was found) are not shown. ==873402== To see them, rerun with: --leak-check=full --show-leak-kinds=all ==873402== ==873402== Use --track-origins=yes to see where uninitialised values come from ==873402== For lists of detected and suppressed errors, rerun with: -s ==873402== ERROR SUMMARY: 683546 errors from 547 contexts (suppressed: 0 from 0) ``` I am on a 4x3090 Ti system with a ThreadRipper Pro 5965WX, 24 cores, and 8x128GB sticks of Samsung ECC DDR4 DRAM running Ubuntu 24.04 LTS. When running this model with the latest build of llama.cpp it is working quite well.

bsakaguchi commented

Author

Owner

@huliangbing commented on GitHub (Feb 6, 2025):

@Azure-Tang Great work! What's the benchmark and performance of some GPU and CPU hardware？

@huliangbing commented on GitHub (Feb 6, 2025): @Azure-Tang Great work! What's the benchmark and performance of some GPU and CPU hardware？

bsakaguchi commented

Author

Owner

@ELigoP commented on GitHub (Feb 6, 2025):

I am now testing with

python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-V3 --gguf_path /mnt/models/gguf/bullerwins/DeepSeek-V3-GGUF --total_context 1024 --max_new_tokens 1024

And I always reproduce segfault on second reply.

Chat: Hi
2025-02-06 11:39:43,022 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/transformers.py[192]: get input ids of shape torch.Size([1, 4])
2025-02-06 11:39:43,022 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/ktransformers.py[121]: input_ids: torch.Size([1, 4])
2025-02-06 11:39:43,024 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/ktransformers.py[147]: cache position: 0 to 4
/home/ai/.pyenv/versions/3.11.6/lib/python3.11/contextlib.py:105: FutureWarning: `torch.backends.cuda.sdp_kernel()` is deprecated. In the future, this context manager will be removed. Please see `torch.nn.attention.sdpa_kernel()` for the new context manager, with updated signature.
  self.gen = func(*args, **kwds)
Hello! How can I assist you today? 😊
2025-02-06 11:39:54,135 INFO /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 1.049270814017157, decode 1.506704551766984. Time(s): tokenize 0.06775856018066406, prefill 3.812171220779419, decode 7.300701379776001
Chat: How long your context is? Can I pass multiline strings? E.g.
2025-02-06 11:40:14,604 WARNING /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/transformers.py[189]: num_unequal_elements: 0
2025-02-06 11:40:14,605 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/transformers.py[192]: get input ids of shape torch.Size([1, 19])
2025-02-06 11:40:14,605 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/ktransformers.py[121]: input_ids: torch.Size([1, 19])
2025-02-06 11:40:14,605 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/ktransformers.py[137]: generate_ids: torch.Size([1, 1029])
2025-02-06 11:40:14,605 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/ktransformers.py[147]: cache position: 15 to 34
Segmentation fault (core dumped)

@ELigoP commented on GitHub (Feb 6, 2025): I am now testing with ```python python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-V3 --gguf_path /mnt/models/gguf/bullerwins/DeepSeek-V3-GGUF --total_context 1024 --max_new_tokens 1024 ``` And I always reproduce segfault on second reply. ``` Chat: Hi 2025-02-06 11:39:43,022 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/transformers.py[192]: get input ids of shape torch.Size([1, 4]) 2025-02-06 11:39:43,022 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/ktransformers.py[121]: input_ids: torch.Size([1, 4]) 2025-02-06 11:39:43,024 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/ktransformers.py[147]: cache position: 0 to 4 /home/ai/.pyenv/versions/3.11.6/lib/python3.11/contextlib.py:105: FutureWarning: `torch.backends.cuda.sdp_kernel()` is deprecated. In the future, this context manager will be removed. Please see `torch.nn.attention.sdpa_kernel()` for the new context manager, with updated signature. self.gen = func(*args, **kwds) Hello! How can I assist you today? 😊 2025-02-06 11:39:54,135 INFO /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 1.049270814017157, decode 1.506704551766984. Time(s): tokenize 0.06775856018066406, prefill 3.812171220779419, decode 7.300701379776001 Chat: How long your context is? Can I pass multiline strings? E.g. 2025-02-06 11:40:14,604 WARNING /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/transformers.py[189]: num_unequal_elements: 0 2025-02-06 11:40:14,605 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/transformers.py[192]: get input ids of shape torch.Size([1, 19]) 2025-02-06 11:40:14,605 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/ktransformers.py[121]: input_ids: torch.Size([1, 19]) 2025-02-06 11:40:14,605 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/ktransformers.py[137]: generate_ids: torch.Size([1, 1029]) 2025-02-06 11:40:14,605 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/ktransformers.py[147]: cache position: 15 to 34 Segmentation fault (core dumped) ```

bsakaguchi commented

Author

Owner

@ELigoP commented on GitHub (Feb 6, 2025):

This might be interesting: I tried with context and length of 2048, and after second reply I got

...
2025-02-06 13:25:54,834 WARNING /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/transformers.py[189]: num_unequal_elements: 0
2025-02-06 13:25:54,834 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/transformers.py[192]: get input ids of shape torch.Size([1, 19])
2025-02-06 13:25:54,834 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/ktransformers.py[121]: input_ids: torch.Size([1, 19])
2025-02-06 13:25:54,835 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/ktransformers.py[137]: generate_ids: torch.Size([1, 2053])
2025-02-06 13:25:54,835 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/ktransformers.py[147]: cache position: 15 to 34

Seems to be generate_ids dimension is always slightly bigger than max context, this time it is [1, 2053] vs context length 2048, and in previous comment it was [1, 1029] vs context length 1024.

Could that be the issue?

@ELigoP commented on GitHub (Feb 6, 2025): This might be interesting: I tried with context and length of 2048, and after second reply I got ``` ... 2025-02-06 13:25:54,834 WARNING /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/transformers.py[189]: num_unequal_elements: 0 2025-02-06 13:25:54,834 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/transformers.py[192]: get input ids of shape torch.Size([1, 19]) 2025-02-06 13:25:54,834 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/ktransformers.py[121]: input_ids: torch.Size([1, 19]) 2025-02-06 13:25:54,835 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/ktransformers.py[137]: generate_ids: torch.Size([1, 2053]) 2025-02-06 13:25:54,835 DEBUG /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/ktransformers/server/backend/interfaces/ktransformers.py[147]: cache position: 15 to 34 ``` Seems to be `generate_ids` dimension is always slightly bigger than max context, this time it is `[1, 2053]` vs context length `2048`, and in previous comment it was `[1, 1029]` vs context length `1024`. Could that be the issue?

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Feb 6, 2025):

We seem have located the segment fault problem. Try latest pr?

@Azure-Tang commented on GitHub (Feb 6, 2025): We seem have located the segment fault problem. Try latest pr?

bsakaguchi commented

Author

Owner

@RodriMora commented on GitHub (Feb 6, 2025):

We seem have located the segment fault problem. Try latest pr?

That solved it for me, quick test:
Same settings as https://github.com/kvcache-ai/ktransformers/issues/117#issuecomment-2636883021

Chat: what is 2+2?
2025-02-06 15:17:30,510 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/transformers.py[192]: get input ids of shape torch.Size([1, 10])
2025-02-06 15:17:30,510 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[121]: input_ids: torch.Size([1, 10])
2025-02-06 15:17:30,511 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[147]: cache position: 0 to 10
/usr/lib/python3.11/contextlib.py:105: FutureWarning: `torch.backends.cuda.sdp_kernel()` is deprecated. In the future, this context manager will be removed. Please see `torch.nn.attention.sdpa_kernel()` for the new context manager, with updated signature.
  self.gen = func(*args, **kwds)
<think>
Okay, so the user is asking "what is 2+2?" At first glance, this seems like a super simple question. But maybe I should think it through carefully to make sure I'm not missing anything.

Alright, 2 plus 2. Let's start by recalling basic arithmetic. Addition is one of the fundamental operations. When you add two numbers, you're combining their values. So, if you have 2 apples and someone gives you 2 more apples, how many apples do you have in total?

If I count them out: 1, 2, then 3, 4. So that's 4 apples. Therefore, 2 + 2 equals 4. But wait, maybe there's a trick here. Sometimes people use bases other than base 10. For example, in base 3, the numbers would be different.

Let me check that. In base 3, the number 2 is still 2. Adding 2 + 2 in base 3 would result in 11 (since 2 + 2 is 4 in base 10, and 4 divided by 3 is 1 with a remainder of 1). So in base 3, 2 + 2 is 11. But the user didn't specify a base, so we can assume base 10.

Also, perhaps in some non-standard arithmetic systems, 2 + 2 might not equal 4. But again, unless specified, we stick to standard arithmetic.

Another angle: sometimes people ask 2 + 2 to test if it's a trick question, like involving sets or vectors. But 2 is a scalar, and adding two scalars gives another scalar. So 2 + 2 is definitely 4 in that context.

In conclusion, unless there's some unstated context, the answer is 4.
</think>

The result of 2 + 2 is **4**.

This is a straightforward arithmetic calculation in base 10. If there were any additional contexts or specific number systems involved (such as different bases or non-standard arithmetic), the answer might vary. However, under standard mathematical rules, the sum of 2 and 2 is consistently 4.
2025-02-06 15:19:26,839 INFO /home/ubuntuai/ktransformers/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 5.776492920315103, decode 3.970428054551163. Time(s): tokenize 0.06781554222106934, prefill 1.731154203414917, decode 114.59721565246582

@RodriMora commented on GitHub (Feb 6, 2025): > We seem have located the segment fault problem. Try latest pr? That solved it for me, quick test: Same settings as https://github.com/kvcache-ai/ktransformers/issues/117#issuecomment-2636883021 ``` Chat: what is 2+2? 2025-02-06 15:17:30,510 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/transformers.py[192]: get input ids of shape torch.Size([1, 10]) 2025-02-06 15:17:30,510 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[121]: input_ids: torch.Size([1, 10]) 2025-02-06 15:17:30,511 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[147]: cache position: 0 to 10 /usr/lib/python3.11/contextlib.py:105: FutureWarning: `torch.backends.cuda.sdp_kernel()` is deprecated. In the future, this context manager will be removed. Please see `torch.nn.attention.sdpa_kernel()` for the new context manager, with updated signature. self.gen = func(*args, **kwds) <think> Okay, so the user is asking "what is 2+2?" At first glance, this seems like a super simple question. But maybe I should think it through carefully to make sure I'm not missing anything. Alright, 2 plus 2. Let's start by recalling basic arithmetic. Addition is one of the fundamental operations. When you add two numbers, you're combining their values. So, if you have 2 apples and someone gives you 2 more apples, how many apples do you have in total? If I count them out: 1, 2, then 3, 4. So that's 4 apples. Therefore, 2 + 2 equals 4. But wait, maybe there's a trick here. Sometimes people use bases other than base 10. For example, in base 3, the numbers would be different. Let me check that. In base 3, the number 2 is still 2. Adding 2 + 2 in base 3 would result in 11 (since 2 + 2 is 4 in base 10, and 4 divided by 3 is 1 with a remainder of 1). So in base 3, 2 + 2 is 11. But the user didn't specify a base, so we can assume base 10. Also, perhaps in some non-standard arithmetic systems, 2 + 2 might not equal 4. But again, unless specified, we stick to standard arithmetic. Another angle: sometimes people ask 2 + 2 to test if it's a trick question, like involving sets or vectors. But 2 is a scalar, and adding two scalars gives another scalar. So 2 + 2 is definitely 4 in that context. In conclusion, unless there's some unstated context, the answer is 4. </think> The result of 2 + 2 is **4**. This is a straightforward arithmetic calculation in base 10. If there were any additional contexts or specific number systems involved (such as different bases or non-standard arithmetic), the answer might vary. However, under standard mathematical rules, the sum of 2 and 2 is consistently 4. 2025-02-06 15:19:26,839 INFO /home/ubuntuai/ktransformers/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 5.776492920315103, decode 3.970428054551163. Time(s): tokenize 0.06781554222106934, prefill 1.731154203414917, decode 114.59721565246582 ```

bsakaguchi commented

Author

Owner

@ELigoP commented on GitHub (Feb 6, 2025):

Me too. Now performance looks slow, but I will investigate that.

@ELigoP commented on GitHub (Feb 6, 2025): Me too. Now performance looks slow, but I will investigate that.

bsakaguchi commented

Author

Owner

@RodriMora commented on GitHub (Feb 6, 2025):

I'm trying to see how much vram and ram it's being allocated but not sure if the args are right.

--max_new_tokens, seems pretty self explanatory, how long the response can be, shouldn't affect how much ram/vram it takes.

But, how do I define the max context for the model to load?

Is it --total_context? or rather --length? can't tell by the docs

@RodriMora commented on GitHub (Feb 6, 2025): I'm trying to see how much vram and ram it's being allocated but not sure if the args are right. --max_new_tokens, seems pretty self explanatory, how long the response can be, shouldn't affect how much ram/vram it takes. But, how do I define the max context for the model to load? Is it --total_context? or rather --length? can't tell by the docs

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Feb 6, 2025):

Ok, it seems influenced prefill speed. We have fixed in latest commit. Try to pull again?

Me too. Now performance looks slow, but I will investigate that.

@Azure-Tang commented on GitHub (Feb 6, 2025): Ok, it seems influenced prefill speed. We have fixed in latest commit. Try to pull again? > Me too. Now performance looks slow, but I will investigate that.

bsakaguchi commented

Author

Owner

@trilog-inc commented on GitHub (Feb 6, 2025):

Can confirm the latest commit affected performance:

with prompt: test

2025-02-06 10:11:09,433 INFO /mnt/home_extend/llm/ktransformers/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 0.3275322458538577, decode 1.8468836199556402. Time(s): tokenize 0.10863876342773438, prefill 12.212538003921509, decode 71.47174763679504

using the longer prompt: Use python and the fastapi library to write a generic openAI compatible endpoint for /v1/models

Traceback (most recent call last):
  File "/mnt/home_extend/llm/ktransformers/ktransformers/local_chat.py", line 104, in <module>
    local_chat()
  File "/mnt/home_extend/llm/ktransformers/ktransformers/local_chat.py", line 96, in local_chat
    generated = asyncio.run(async_inference(messages))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/myfrienderic/miniconda3/envs/ktransformers/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/myfrienderic/miniconda3/envs/ktransformers/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/myfrienderic/miniconda3/envs/ktransformers/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/mnt/home_extend/llm/ktransformers/ktransformers/local_chat.py", line 92, in async_inference
    async for token in interface.inference(messages, "local_chat"):
  File "/mnt/home_extend/llm/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 319, in inference
    input_ids = self.format_and_tokenize_input_ids(thread_id, local_messages)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home_extend/llm/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 186, in format_and_tokenize_input_ids
    unequal_mask = torch.ne(x,y)
                   ^^^^^^^^^^^^^
RuntimeError: The size of tensor a (136) must match the size of tensor b (61) at non-singleton dimension 1

command: python ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/home_extend/models/bartowski_DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --max_new_tokens 1024 --total_context 1024 --cpu_infer 46

@trilog-inc commented on GitHub (Feb 6, 2025): Can confirm the latest commit affected performance: with prompt: test ``` 2025-02-06 10:11:09,433 INFO /mnt/home_extend/llm/ktransformers/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 0.3275322458538577, decode 1.8468836199556402. Time(s): tokenize 0.10863876342773438, prefill 12.212538003921509, decode 71.47174763679504 ``` using the longer prompt: Use python and the fastapi library to write a generic openAI compatible endpoint for /v1/models ``` Traceback (most recent call last): File "/mnt/home_extend/llm/ktransformers/ktransformers/local_chat.py", line 104, in <module> local_chat() File "/mnt/home_extend/llm/ktransformers/ktransformers/local_chat.py", line 96, in local_chat generated = asyncio.run(async_inference(messages)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/myfrienderic/miniconda3/envs/ktransformers/lib/python3.11/asyncio/runners.py", line 190, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File "/home/myfrienderic/miniconda3/envs/ktransformers/lib/python3.11/asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/myfrienderic/miniconda3/envs/ktransformers/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File "/mnt/home_extend/llm/ktransformers/ktransformers/local_chat.py", line 92, in async_inference async for token in interface.inference(messages, "local_chat"): File "/mnt/home_extend/llm/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 319, in inference input_ids = self.format_and_tokenize_input_ids(thread_id, local_messages) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/mnt/home_extend/llm/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 186, in format_and_tokenize_input_ids unequal_mask = torch.ne(x,y) ^^^^^^^^^^^^^ RuntimeError: The size of tensor a (136) must match the size of tensor b (61) at non-singleton dimension 1 ``` command: python ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/home_extend/models/bartowski_DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --max_new_tokens 1024 --total_context 1024 --cpu_infer 46

bsakaguchi commented

Author

Owner

@RodriMora commented on GitHub (Feb 6, 2025):

Can confirm the latest commit affected performance:

with prompt: test

2025-02-06 10:11:09,433 INFO /mnt/home_extend/llm/ktransformers/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 0.3275322458538577, decode 1.8468836199556402. Time(s): tokenize 0.10863876342773438, prefill 12.212538003921509, decode 71.47174763679504

using the longer prompt: Use python and the fastapi library to write a generic openAI compatible endpoint for /v1/models

Traceback (most recent call last):
  File "/mnt/home_extend/llm/ktransformers/ktransformers/local_chat.py", line 104, in <module>
    local_chat()
  File "/mnt/home_extend/llm/ktransformers/ktransformers/local_chat.py", line 96, in local_chat
    generated = asyncio.run(async_inference(messages))
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/myfrienderic/miniconda3/envs/ktransformers/lib/python3.11/asyncio/runners.py", line 190, in run
    return runner.run(main)
           ^^^^^^^^^^^^^^^^
  File "/home/myfrienderic/miniconda3/envs/ktransformers/lib/python3.11/asyncio/runners.py", line 118, in run
    return self._loop.run_until_complete(task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/myfrienderic/miniconda3/envs/ktransformers/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete
    return future.result()
           ^^^^^^^^^^^^^^^
  File "/mnt/home_extend/llm/ktransformers/ktransformers/local_chat.py", line 92, in async_inference
    async for token in interface.inference(messages, "local_chat"):
  File "/mnt/home_extend/llm/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 319, in inference
    input_ids = self.format_and_tokenize_input_ids(thread_id, local_messages)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/mnt/home_extend/llm/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 186, in format_and_tokenize_input_ids
    unequal_mask = torch.ne(x,y)
                   ^^^^^^^^^^^^^
RuntimeError: The size of tensor a (136) must match the size of tensor b (61) at non-singleton dimension 1

command: python ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/home_extend/models/bartowski_DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --max_new_tokens 1024 --total_context 1024 --cpu_infer 46

Working fine for me:

python ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/ubuntuai/models/DeepSeek-R1-Q3_K_M --max_new_tokens 1000 --total_context 2000

Chat: Use python and the fastapi library to write a generic openAI compatible endpoint for /v1/models
2025-02-06 16:21:55,390 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/transformers.py[192]: get input ids of shape torch.Size([1, 23])
2025-02-06 16:21:55,391 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[121]: input_ids: torch.Size([1, 23])
2025-02-06 16:21:55,392 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[147]: cache position: 0 to 23
/usr/lib/python3.11/contextlib.py:105: FutureWarning: `torch.backends.cuda.sdp_kernel()` is deprecated. In the future, this context manager will be removed. Please see `torch.nn.attention.sdpa_kernel()` for the new context manager, with updated signature.
  self.gen = func(*args, **kwds)
<think>
Okay, the user wants to create a FastAPI endpoint that's compatible with OpenAI's /v1/models. Let me figure out what steps are needed.

First, I know that the OpenAI API returns a list of available models. So the endpoint should respond with similar data. The response includes an object with a "data" array containing model entries, each with an "id", "object" type, and "owned_by" info.

Using FastAPI, I need to set up a GET route at /v1/models. I'll define Pydantic models to structure the response. The Model and ModelList classes will ensure the JSON response matches OpenAI's format.

The Model model should have fields: id (str), object (str, default "model"), and owned_by (str, maybe default to "openai" for compatibility). The ModelList model will have a data field which is a list of Model instances.

For the example, I'll hardcode a couple of models, like "gpt-3.5-turbo" and "gpt-4", but note that the user should replace these with actual models they support.

Finally, the FastAPI app is created, and the endpoint is added. The user needs to install FastAPI and an ASGI server like uvicorn to run it.

I should remember to include necessary imports, like FastAPI and the Pydantic BaseModel. Also, mention any dependencies the user might need, like installing fastapi with pip.

Check if there are any additional considerations, CORS perhaps, but since the user didn't specify, maybe leave it out unless necessary. Though for testing from a browser, enabling CORS might be needed, but the answer can note that if the user wants to allow requests from different domains, they should add CORS middleware.

All right, putting it all together step by step.
</think>

Here's a step-by-step guide to creating a FastAPI endpoint compatible with OpenAI's /v1/models:

1. **Install Dependencies**
bash
pip install fastapi uvicorn pydantic


2. **Create the FastAPI Application**

python
from fastapi import FastAPI
from pydantic import BaseModel
from typing import List

# Define Pydantic models for the OpenAI-compatible response
class Model(BaseModel):
    id: str
    object: str = "model"
    owned_by: str = "openai"

class ModelList(BaseModel):
    data: List[Model]

app = FastAPI()

# Add CORS middleware if needed
from fastapi.middleware.cors import CORSMiddleware
app.add_middleware(
    CORSMiddleware,
    allow_origins=["*"],
    allow_credentials=True,
    allow_methods=["*"],
    allow_headers=["*"],
)

@app.get("/v1/models", response_model=ModelList)
async def get_models():
    # Replace with your actual model list
    models = [
        Model(id="gpt-3.5-turbo", owned_by="openai"),
        Model(id="gpt-4", owned_by="openai")
    ]
    return ModelList(data=models)


3. **Run the Application**
bash
uvicorn main:app --reload


Key points:
- The endpoint `GET /v1/models` returns a list of available models in OpenAI-compatible format
- Response structure matches the OpenAI specification with `id`, `object`, and `owned_by` fields
- CORS middleware is added to handle cross-origin requests
- The example returns hardcoded models, but you should replace them with your actual model list

To test the endpoint, you can use curl:
bash
curl http://localhost:8000/v1/models

2025-02-06 16:25:17,287 INFO /home/ubuntuai/ktransformers/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 8.782558441449666, decode 3.9291855660679187. Time(s): tokenize 0.051436424255371094, prefill 2.6188268661499023, decode 199.27793860435486

@RodriMora commented on GitHub (Feb 6, 2025): > Can confirm the latest commit affected performance: > > with prompt: test > > ``` > 2025-02-06 10:11:09,433 INFO /mnt/home_extend/llm/ktransformers/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 0.3275322458538577, decode 1.8468836199556402. Time(s): tokenize 0.10863876342773438, prefill 12.212538003921509, decode 71.47174763679504 > ``` > > using the longer prompt: Use python and the fastapi library to write a generic openAI compatible endpoint for /v1/models > > ``` > Traceback (most recent call last): > File "/mnt/home_extend/llm/ktransformers/ktransformers/local_chat.py", line 104, in <module> > local_chat() > File "/mnt/home_extend/llm/ktransformers/ktransformers/local_chat.py", line 96, in local_chat > generated = asyncio.run(async_inference(messages)) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/home/myfrienderic/miniconda3/envs/ktransformers/lib/python3.11/asyncio/runners.py", line 190, in run > return runner.run(main) > ^^^^^^^^^^^^^^^^ > File "/home/myfrienderic/miniconda3/envs/ktransformers/lib/python3.11/asyncio/runners.py", line 118, in run > return self._loop.run_until_complete(task) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/home/myfrienderic/miniconda3/envs/ktransformers/lib/python3.11/asyncio/base_events.py", line 654, in run_until_complete > return future.result() > ^^^^^^^^^^^^^^^ > File "/mnt/home_extend/llm/ktransformers/ktransformers/local_chat.py", line 92, in async_inference > async for token in interface.inference(messages, "local_chat"): > File "/mnt/home_extend/llm/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 319, in inference > input_ids = self.format_and_tokenize_input_ids(thread_id, local_messages) > ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ > File "/mnt/home_extend/llm/ktransformers/ktransformers/server/backend/interfaces/transformers.py", line 186, in format_and_tokenize_input_ids > unequal_mask = torch.ne(x,y) > ^^^^^^^^^^^^^ > RuntimeError: The size of tensor a (136) must match the size of tensor b (61) at non-singleton dimension 1 > ``` > > command: python ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/home_extend/models/bartowski_DeepSeek-R1-GGUF/DeepSeek-R1-Q4_K_M --max_new_tokens 1024 --total_context 1024 --cpu_infer 46 Working fine for me: `python ktransformers/local_chat.py --model_path deepseek-ai/DeepSeek-R1 --gguf_path /home/ubuntuai/models/DeepSeek-R1-Q3_K_M --max_new_tokens 1000 --total_context 2000` ``` Chat: Use python and the fastapi library to write a generic openAI compatible endpoint for /v1/models 2025-02-06 16:21:55,390 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/transformers.py[192]: get input ids of shape torch.Size([1, 23]) 2025-02-06 16:21:55,391 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[121]: input_ids: torch.Size([1, 23]) 2025-02-06 16:21:55,392 DEBUG /home/ubuntuai/ktransformers/ktransformers/server/backend/interfaces/ktransformers.py[147]: cache position: 0 to 23 /usr/lib/python3.11/contextlib.py:105: FutureWarning: `torch.backends.cuda.sdp_kernel()` is deprecated. In the future, this context manager will be removed. Please see `torch.nn.attention.sdpa_kernel()` for the new context manager, with updated signature. self.gen = func(*args, **kwds) <think> Okay, the user wants to create a FastAPI endpoint that's compatible with OpenAI's /v1/models. Let me figure out what steps are needed. First, I know that the OpenAI API returns a list of available models. So the endpoint should respond with similar data. The response includes an object with a "data" array containing model entries, each with an "id", "object" type, and "owned_by" info. Using FastAPI, I need to set up a GET route at /v1/models. I'll define Pydantic models to structure the response. The Model and ModelList classes will ensure the JSON response matches OpenAI's format. The Model model should have fields: id (str), object (str, default "model"), and owned_by (str, maybe default to "openai" for compatibility). The ModelList model will have a data field which is a list of Model instances. For the example, I'll hardcode a couple of models, like "gpt-3.5-turbo" and "gpt-4", but note that the user should replace these with actual models they support. Finally, the FastAPI app is created, and the endpoint is added. The user needs to install FastAPI and an ASGI server like uvicorn to run it. I should remember to include necessary imports, like FastAPI and the Pydantic BaseModel. Also, mention any dependencies the user might need, like installing fastapi with pip. Check if there are any additional considerations, CORS perhaps, but since the user didn't specify, maybe leave it out unless necessary. Though for testing from a browser, enabling CORS might be needed, but the answer can note that if the user wants to allow requests from different domains, they should add CORS middleware. All right, putting it all together step by step. </think> Here's a step-by-step guide to creating a FastAPI endpoint compatible with OpenAI's /v1/models: 1. **Install Dependencies** bash pip install fastapi uvicorn pydantic 2. **Create the FastAPI Application** python from fastapi import FastAPI from pydantic import BaseModel from typing import List # Define Pydantic models for the OpenAI-compatible response class Model(BaseModel): id: str object: str = "model" owned_by: str = "openai" class ModelList(BaseModel): data: List[Model] app = FastAPI() # Add CORS middleware if needed from fastapi.middleware.cors import CORSMiddleware app.add_middleware( CORSMiddleware, allow_origins=["*"], allow_credentials=True, allow_methods=["*"], allow_headers=["*"], ) @app.get("/v1/models", response_model=ModelList) async def get_models(): # Replace with your actual model list models = [ Model(id="gpt-3.5-turbo", owned_by="openai"), Model(id="gpt-4", owned_by="openai") ] return ModelList(data=models) 3. **Run the Application** bash uvicorn main:app --reload Key points: - The endpoint `GET /v1/models` returns a list of available models in OpenAI-compatible format - Response structure matches the OpenAI specification with `id`, `object`, and `owned_by` fields - CORS middleware is added to handle cross-origin requests - The example returns hardcoded models, but you should replace them with your actual model list To test the endpoint, you can use curl: bash curl http://localhost:8000/v1/models 2025-02-06 16:25:17,287 INFO /home/ubuntuai/ktransformers/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 8.782558441449666, decode 3.9291855660679187. Time(s): tokenize 0.051436424255371094, prefill 2.6188268661499023, decode 199.27793860435486 ```

bsakaguchi commented

Author

Owner

@trilog-inc commented on GitHub (Feb 6, 2025):

You are right, my bad.

Flushed the entire conda environment and ktransformers folder, rebuilt and it works as expected 🙌

Going to play with this today, huge thanks to the team for their efforts!

@trilog-inc commented on GitHub (Feb 6, 2025): You are right, my bad. Flushed the entire conda environment and ktransformers folder, rebuilt and it works as expected 🙌 Going to play with this today, huge thanks to the team for their efforts!

bsakaguchi commented

Author

Owner

@RodriMora commented on GitHub (Feb 6, 2025):

curl 'http://0.0.0.0:10002/v1/chat/completions' \
    -X POST \
    -H "Content-Type: application/json" \
    --data-raw '{"model":"deepseek-ai/DeepSeek-V3","messages":[{"role":"system","content":"You are an AI coding assistant. You explain as minimum as possible."},{"role":"user","content":"Write numbers from 1 to 10, each on new line, no coding."}],"stream":false}'

Did you change the 0.0.0.0 for privacy? you should use the actual IP of the server, it's working for me.

curl 'http://192.168.10.2:5000/v1/chat/completions' \
    -X POST \
    -H "Content-Type: application/json" \
    --data-raw '{"model":"deepseek-ai/DeepSeek-V3","messages":[{"role":"system","content":"You are an AI coding assistant. You explain as minimum as possible."},{"role":"user","content":"Write numbers from 1 to 10, each on new line, no coding."}],"stream":false}'

{"id":"7ff95288-d334-431f-9d94-867f372bd497","object":"chat.completion.chunk","created":1738864152,"model":"not implmented","system_fingerprint":"not implmented","usage":null,"choices":[{"index":0,"message":{"content":"<think>\nOkay, so the user wants me to write numbers from 1 to 10, each on a new line, and they specified no coding. Let me make sure I understand. They mentioned \"You are an AI coding assistant\" but then said \"no coding,\" so maybe they want the output to be simple text without any code formatting like in a code block. Got it.\n\nSo, I need to list numbers 1 through 10, each on separate lines. Let me check if there's any other requirement. The user wrote \"explain as minimum as possible,\" which probably means not adding any extra explanations or comments. Just the numbers. Alright.\n\nI think that's all. I'll write each number from 1 to 10, each on a new line. Let me count to confirm: 10 numbers, each separated. Yes, that should do it.\n</think>\n\n1 \n2 \n3 \n4 \n5 \n6 \n7 \n8 \n9 \n10","role":"assistant","name":null},"logprobs":null,"finish_reason":null}]}

@RodriMora commented on GitHub (Feb 6, 2025): > ``` > curl 'http://0.0.0.0:10002/v1/chat/completions' \ > -X POST \ > -H "Content-Type: application/json" \ > --data-raw '{"model":"deepseek-ai/DeepSeek-V3","messages":[{"role":"system","content":"You are an AI coding assistant. You explain as minimum as possible."},{"role":"user","content":"Write numbers from 1 to 10, each on new line, no coding."}],"stream":false}' > ``` Did you change the 0.0.0.0 for privacy? you should use the actual IP of the server, it's working for me. ``` curl 'http://192.168.10.2:5000/v1/chat/completions' \ -X POST \ -H "Content-Type: application/json" \ --data-raw '{"model":"deepseek-ai/DeepSeek-V3","messages":[{"role":"system","content":"You are an AI coding assistant. You explain as minimum as possible."},{"role":"user","content":"Write numbers from 1 to 10, each on new line, no coding."}],"stream":false}' ``` `{"id":"7ff95288-d334-431f-9d94-867f372bd497","object":"chat.completion.chunk","created":1738864152,"model":"not implmented","system_fingerprint":"not implmented","usage":null,"choices":[{"index":0,"message":{"content":"<think>\nOkay, so the user wants me to write numbers from 1 to 10, each on a new line, and they specified no coding. Let me make sure I understand. They mentioned \"You are an AI coding assistant\" but then said \"no coding,\" so maybe they want the output to be simple text without any code formatting like in a code block. Got it.\n\nSo, I need to list numbers 1 through 10, each on separate lines. Let me check if there's any other requirement. The user wrote \"explain as minimum as possible,\" which probably means not adding any extra explanations or comments. Just the numbers. Alright.\n\nI think that's all. I'll write each number from 1 to 10, each on a new line. Let me count to confirm: 10 numbers, each separated. Yes, that should do it.\n</think>\n\n1 \n2 \n3 \n4 \n5 \n6 \n7 \n8 \n9 \n10","role":"assistant","name":null},"logprobs":null,"finish_reason":null}]}`

bsakaguchi commented

Author

Owner

@ELigoP commented on GitHub (Feb 6, 2025):

curl 'http://0.0.0.0:10002/v1/chat/completions' \
    -X POST \
    -H "Content-Type: application/json" \
    --data-raw '{"model":"deepseek-ai/DeepSeek-V3","messages":[{"role":"system","content":"You are an AI coding assistant. You explain as minimum as possible."},{"role":"user","content":"Write numbers from 1 to 10, each on new line, no coding."}],"stream":false}'
Did you change the 0.0.0.0 for privacy? you should use the actual IP of the server, it's working for me.
curl 'http://192.168.10.2:5000/v1/chat/completions' \
    -X POST \
    -H "Content-Type: application/json" \
    --data-raw '{"model":"deepseek-ai/DeepSeek-V3","messages":[{"role":"system","content":"You are an AI coding assistant. You explain as minimum as possible."},{"role":"user","content":"Write numbers from 1 to 10, each on new line, no coding."}],"stream":false}'
{"id":"7ff95288-d334-431f-9d94-867f372bd497","object":"chat.completion.chunk","created":1738864152,"model":"not implmented","system_fingerprint":"not implmented","usage":null,"choices":[{"index":0,"message":{"content":"<think>\nOkay, so the user wants me to write numbers from 1 to 10, each on a new line, and they specified no coding. Let me make sure I understand. They mentioned \"You are an AI coding assistant\" but then said \"no coding,\" so maybe they want the output to be simple text without any code formatting like in a code block. Got it.\n\nSo, I need to list numbers 1 through 10, each on separate lines. Let me check if there's any other requirement. The user wrote \"explain as minimum as possible,\" which probably means not adding any extra explanations or comments. Just the numbers. Alright.\n\nI think that's all. I'll write each number from 1 to 10, each on a new line. Let me count to confirm: 10 numbers, each separated. Yes, that should do it.\n</think>\n\n1 \n2 \n3 \n4 \n5 \n6 \n7 \n8 \n9 \n10","role":"assistant","name":null},"logprobs":null,"finish_reason":null}]}

Thanks! My bad, I didn't pull latest change. Now server responds.
Still performance is not as expected (comparable and even worse than llama.cpp, which as I understand offloads only first layers to GPU):

2025-02-06 20:45:21,438 INFO /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/venv/lib/python3.11/site-packages/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 1.9965464671950348, decode 1.9084481659736685. Time(s): tokenize 0.04080605506896973, prefill 13.523351669311523, decode 259.89702463150024

llama.cpp gets 3.5 tps prompt processing (I guess this is prefill) and 2.5 tps token generation (which I guess is decode).

I will play with settings more.

@RodriMora what is your system specs, by the way? Mine is 4xRTX 3090 (only first two cards used with my command) and Threadripper 3970X with 256GB DDR4 RAM (4 ram channels, RAM read speed about 80GB/s).

@ELigoP commented on GitHub (Feb 6, 2025): > > ``` > > curl 'http://0.0.0.0:10002/v1/chat/completions' \ > > -X POST \ > > -H "Content-Type: application/json" \ > > --data-raw '{"model":"deepseek-ai/DeepSeek-V3","messages":[{"role":"system","content":"You are an AI coding assistant. You explain as minimum as possible."},{"role":"user","content":"Write numbers from 1 to 10, each on new line, no coding."}],"stream":false}' > > ``` > > Did you change the 0.0.0.0 for privacy? you should use the actual IP of the server, it's working for me. > > ``` > curl 'http://192.168.10.2:5000/v1/chat/completions' \ > -X POST \ > -H "Content-Type: application/json" \ > --data-raw '{"model":"deepseek-ai/DeepSeek-V3","messages":[{"role":"system","content":"You are an AI coding assistant. You explain as minimum as possible."},{"role":"user","content":"Write numbers from 1 to 10, each on new line, no coding."}],"stream":false}' > ``` > > `{"id":"7ff95288-d334-431f-9d94-867f372bd497","object":"chat.completion.chunk","created":1738864152,"model":"not implmented","system_fingerprint":"not implmented","usage":null,"choices":[{"index":0,"message":{"content":"<think>\nOkay, so the user wants me to write numbers from 1 to 10, each on a new line, and they specified no coding. Let me make sure I understand. They mentioned \"You are an AI coding assistant\" but then said \"no coding,\" so maybe they want the output to be simple text without any code formatting like in a code block. Got it.\n\nSo, I need to list numbers 1 through 10, each on separate lines. Let me check if there's any other requirement. The user wrote \"explain as minimum as possible,\" which probably means not adding any extra explanations or comments. Just the numbers. Alright.\n\nI think that's all. I'll write each number from 1 to 10, each on a new line. Let me count to confirm: 10 numbers, each separated. Yes, that should do it.\n</think>\n\n1 \n2 \n3 \n4 \n5 \n6 \n7 \n8 \n9 \n10","role":"assistant","name":null},"logprobs":null,"finish_reason":null}]}` Thanks! My bad, I didn't pull latest change. Now server responds. Still performance is not as expected (comparable and even worse than llama.cpp, which as I understand offloads only first layers to GPU): ``` 2025-02-06 20:45:21,438 INFO /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/venv/lib/python3.11/site-packages/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 1.9965464671950348, decode 1.9084481659736685. Time(s): tokenize 0.04080605506896973, prefill 13.523351669311523, decode 259.89702463150024 ``` llama.cpp gets 3.5 tps prompt processing (I guess this is prefill) and 2.5 tps token generation (which I guess is decode). I will play with settings more. @RodriMora what is your system specs, by the way? Mine is 4xRTX 3090 (only first two cards used with my command) and Threadripper 3970X with 256GB DDR4 RAM (4 ram channels, RAM read speed about 80GB/s).

bsakaguchi commented

Author

Owner

@RodriMora commented on GitHub (Feb 6, 2025):

@RodriMora what is your system specs, by the way? Mine is 4xRTX 3090 (only first two cards used with my command) and Threadripper 3970X with 256GB DDR4 RAM (4 ram channels, RAM read speed about 80GB/s).

Similar to yours but on Epyc cpu:

EPYC 7402 24c/48t
512GB RAM 3200MHz
4x3090s

I'm currently doing some performance tests, comparing it to llama.cpp at different context lenghts. I've never used ktransformers so I dont know how to manage how many GPU's to use? It seems to use 2/4 at the moment.

@RodriMora commented on GitHub (Feb 6, 2025): > [@RodriMora](https://github.com/RodriMora) what is your system specs, by the way? Mine is 4xRTX 3090 (only first two cards used with my command) and Threadripper 3970X with 256GB DDR4 RAM (4 ram channels, RAM read speed about 80GB/s). Similar to yours but on Epyc cpu: EPYC 7402 24c/48t 512GB RAM 3200MHz 4x3090s I'm currently doing some performance tests, comparing it to llama.cpp at different context lenghts. I've never used ktransformers so I dont know how to manage how many GPU's to use? It seems to use 2/4 at the moment.

bsakaguchi commented

Author

Owner

@Nottlespike commented on GitHub (Feb 6, 2025):

@RodriMora what is your system specs, by the way? Mine is 4xRTX 3090 (only first two cards used with my command) and Threadripper 3970X with 256GB DDR4 RAM (4 ram channels, RAM read speed about 80GB/s).

Similar to yours but on Epyc cpu:

EPYC 7402 24c/48t 512GB RAM 3200MHz 4x3090s

I'm currently doing some performance tests, comparing it to llama.cpp at different context lenghts. I've never used ktransformers so I dont know how to manage how many GPU's to use? It seems to use 2/4 at the moment.

The relative path for the multi GPU .yaml is at ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml.

With the tutorial on injection being here: https://github.com/kvcache-ai/ktransformers/blob/feat-DeepSeekV3/doc/en/injection_tutorial.md

I've been stymied by the regex for offloading to more than 2 GPU's with specifically this piece being particularly confusing (0|[1-9]|[12][0-9]) as I assume this is for the fact that the first 3 layers of the model are dense but given the pipe I'm not sure how ktransformers handles the "or" regex here and still loads all 61 layers.

Either way it looks like the segfault issue is fixed so I'm rebuilding then testing.

@Nottlespike commented on GitHub (Feb 6, 2025): > > [@RodriMora](https://github.com/RodriMora) what is your system specs, by the way? Mine is 4xRTX 3090 (only first two cards used with my command) and Threadripper 3970X with 256GB DDR4 RAM (4 ram channels, RAM read speed about 80GB/s). > > Similar to yours but on Epyc cpu: > > EPYC 7402 24c/48t 512GB RAM 3200MHz 4x3090s > > I'm currently doing some performance tests, comparing it to llama.cpp at different context lenghts. I've never used ktransformers so I dont know how to manage how many GPU's to use? It seems to use 2/4 at the moment. The relative path for the multi GPU .yaml is at `ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml`. With the tutorial on injection being here: https://github.com/kvcache-ai/ktransformers/blob/feat-DeepSeekV3/doc/en/injection_tutorial.md I've been stymied by the regex for offloading to more than 2 GPU's with specifically this piece being particularly confusing `(0|[1-9]|[12][0-9])` as I assume this is for the fact that the first 3 layers of the model are dense but given the pipe I'm not sure how ktransformers handles the "or" regex here and still loads all 61 layers. Either way it looks like the segfault issue is fixed so I'm rebuilding then testing.

bsakaguchi commented

Author

Owner

@Nottlespike commented on GitHub (Feb 6, 2025):

@RodriMora @ELigoP

So it seems NCCL speeds matter a LOT for TPS. I'm getting WAY faster speeds using 2 GPU's with NVLink than vanilla llama.cpp with the same 4x3090/3090 Ti setup with 14 layers offloaded. For me cuda:0 and cuda:2 are NVLinked together so when I replaced all cuda:1 instances with cuda:2 in ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml I saw a massive speedup of ~10tps which is 2x vanilla llama.cpp with --max_new_tokens 8192 --total_context 8192 and same context on vanilla llama.cpp.

Now to write the 4 GPU config! Any assitance with adapting the ktransformers/optimize/optimize_rules/DeepSeek-V2-Chat-multi-gpu-4.yaml to DeepSeek-V3/R1 would be appreciated @Azure-Tang! I've taken a VERY short look at the paper and it seems that the main difference here is that there is 1 dense layer in V2 vs the 3 in V3 and 60 layers in V2 vs 61 layers in V3 yet my regex for layer assignment seems to break despite my best efforts and reading the injection tutorial multiple times.

@Nottlespike commented on GitHub (Feb 6, 2025): @RodriMora @ELigoP So it seems NCCL speeds matter a LOT for TPS. I'm getting WAY faster speeds using 2 GPU's with NVLink than vanilla llama.cpp with the same 4x3090/3090 Ti setup with 14 layers offloaded. For me `cuda:0` and `cuda:2` are NVLinked together so when I replaced all `cuda:1` instances with `cuda:2` in `ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml` I saw a massive speedup of ~10tps which is 2x vanilla llama.cpp with `--max_new_tokens 8192 --total_context 8192` and same context on vanilla llama.cpp. Now to write the 4 GPU config! Any assitance with adapting the `ktransformers/optimize/optimize_rules/DeepSeek-V2-Chat-multi-gpu-4.yaml` to DeepSeek-V3/R1 would be appreciated @Azure-Tang! I've taken a VERY short look at the paper and it seems that the main difference here is that there is 1 dense layer in V2 vs the 3 in V3 and 60 layers in V2 vs 61 layers in V3 yet my regex for layer assignment seems to break despite my best efforts and reading the injection tutorial multiple times.

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Feb 7, 2025):

curl 'http://0.0.0.0:10002/v1/chat/completions' \
    -X POST \
    -H "Content-Type: application/json" \
    --data-raw '{"model":"deepseek-ai/DeepSeek-V3","messages":[{"role":"system","content":"You are an AI coding assistant. You explain as minimum as possible."},{"role":"user","content":"Write numbers from 1 to 10, each on new line, no coding."}],"stream":false}'
Did you change the 0.0.0.0 for privacy? you should use the actual IP of the server, it's working for me.
curl 'http://192.168.10.2:5000/v1/chat/completions' \
    -X POST \
    -H "Content-Type: application/json" \
    --data-raw '{"model":"deepseek-ai/DeepSeek-V3","messages":[{"role":"system","content":"You are an AI coding assistant. You explain as minimum as possible."},{"role":"user","content":"Write numbers from 1 to 10, each on new line, no coding."}],"stream":false}'
{"id":"7ff95288-d334-431f-9d94-867f372bd497","object":"chat.completion.chunk","created":1738864152,"model":"not implmented","system_fingerprint":"not implmented","usage":null,"choices":[{"index":0,"message":{"content":"<think>\nOkay, so the user wants me to write numbers from 1 to 10, each on a new line, and they specified no coding. Let me make sure I understand. They mentioned \"You are an AI coding assistant\" but then said \"no coding,\" so maybe they want the output to be simple text without any code formatting like in a code block. Got it.\n\nSo, I need to list numbers 1 through 10, each on separate lines. Let me check if there's any other requirement. The user wrote \"explain as minimum as possible,\" which probably means not adding any extra explanations or comments. Just the numbers. Alright.\n\nI think that's all. I'll write each number from 1 to 10, each on a new line. Let me count to confirm: 10 numbers, each separated. Yes, that should do it.\n</think>\n\n1 \n2 \n3 \n4 \n5 \n6 \n7 \n8 \n9 \n10","role":"assistant","name":null},"logprobs":null,"finish_reason":null}]}
Thanks! My bad, I didn't pull latest change. Now server responds. Still performance is not as expected (comparable and even worse than llama.cpp, which as I understand offloads only first layers to GPU):
2025-02-06 20:45:21,438 INFO /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/venv/lib/python3.11/site-packages/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 1.9965464671950348, decode 1.9084481659736685. Time(s): tokenize 0.04080605506896973, prefill 13.523351669311523, decode 259.89702463150024
llama.cpp gets 3.5 tps prompt processing (I guess this is prefill) and 2.5 tps token generation (which I guess is decode).

I will play with settings more.

@RodriMora what is your system specs, by the way? Mine is 4xRTX 3090 (only first two cards used with my command) and Threadripper 3970X with 256GB DDR4 RAM (4 ram channels, RAM read speed about 80GB/s).

Try to adjust cpu_infer according to your cpu core~

@Azure-Tang commented on GitHub (Feb 7, 2025): > > > ``` > > > curl 'http://0.0.0.0:10002/v1/chat/completions' \ > > > -X POST \ > > > -H "Content-Type: application/json" \ > > > --data-raw '{"model":"deepseek-ai/DeepSeek-V3","messages":[{"role":"system","content":"You are an AI coding assistant. You explain as minimum as possible."},{"role":"user","content":"Write numbers from 1 to 10, each on new line, no coding."}],"stream":false}' > > > ``` > > > > > > Did you change the 0.0.0.0 for privacy? you should use the actual IP of the server, it's working for me. > > ``` > > curl 'http://192.168.10.2:5000/v1/chat/completions' \ > > -X POST \ > > -H "Content-Type: application/json" \ > > --data-raw '{"model":"deepseek-ai/DeepSeek-V3","messages":[{"role":"system","content":"You are an AI coding assistant. You explain as minimum as possible."},{"role":"user","content":"Write numbers from 1 to 10, each on new line, no coding."}],"stream":false}' > > ``` > > > > > > > > > > > > > > > > > > > > > > > > `{"id":"7ff95288-d334-431f-9d94-867f372bd497","object":"chat.completion.chunk","created":1738864152,"model":"not implmented","system_fingerprint":"not implmented","usage":null,"choices":[{"index":0,"message":{"content":"<think>\nOkay, so the user wants me to write numbers from 1 to 10, each on a new line, and they specified no coding. Let me make sure I understand. They mentioned \"You are an AI coding assistant\" but then said \"no coding,\" so maybe they want the output to be simple text without any code formatting like in a code block. Got it.\n\nSo, I need to list numbers 1 through 10, each on separate lines. Let me check if there's any other requirement. The user wrote \"explain as minimum as possible,\" which probably means not adding any extra explanations or comments. Just the numbers. Alright.\n\nI think that's all. I'll write each number from 1 to 10, each on a new line. Let me count to confirm: 10 numbers, each separated. Yes, that should do it.\n</think>\n\n1 \n2 \n3 \n4 \n5 \n6 \n7 \n8 \n9 \n10","role":"assistant","name":null},"logprobs":null,"finish_reason":null}]}` > > Thanks! My bad, I didn't pull latest change. Now server responds. Still performance is not as expected (comparable and even worse than llama.cpp, which as I understand offloads only first layers to GPU): > > ``` > 2025-02-06 20:45:21,438 INFO /home/ai/3rdparty/KTransformers-feat-DeepSeekV3/venv/lib/python3.11/site-packages/ktransformers/server/backend/base.py[64]: Performance(T/s): prefill 1.9965464671950348, decode 1.9084481659736685. Time(s): tokenize 0.04080605506896973, prefill 13.523351669311523, decode 259.89702463150024 > ``` > > llama.cpp gets 3.5 tps prompt processing (I guess this is prefill) and 2.5 tps token generation (which I guess is decode). > > I will play with settings more. > > [@RodriMora](https://github.com/RodriMora) what is your system specs, by the way? Mine is 4xRTX 3090 (only first two cards used with my command) and Threadripper 3970X with 256GB DDR4 RAM (4 ram channels, RAM read speed about 80GB/s). Try to adjust `cpu_infer` according to your cpu core~

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Feb 7, 2025):

I have successfully deployed DeepSeek-V3 on the KTransformers framework, but there are still some accuracy issues that I am currently investigating.
Based on preliminary tests, using the Q4KM weight file requires at least 16GB of VRAM and 480GB of RAM. On a positive note, the generation speed can reach approximately 3.8 tokens/s.”

Thats wonderful.. Really appreciate the hard work. Just a small question if i inject more layers to the gpu since i have 144 GB of VRAM, will the RAM consumption go down and will i be able to allocate more context?

Is it possible to run the DeepSeek R1 in just 4 A100 GPU (roughly 320 GB VRAM) using accelerate with offloading to disk? While this will be really really slow, but for some analysis purpose is it possible to run with this setup?

It's possible. Loading process will be slow in this way, but the inference speed can be greatly accelerated. You can refer this:

I have successfully deployed DeepSeek-V3 on the KTransformers framework, but there are still some accuracy issues that I am currently investigating.
Based on preliminary tests, using the Q4KM weight file requires at least 16GB of VRAM and 480GB of RAM. On a positive note, the generation speed can reach approximately 3.8 tokens/s.”

Thats wonderful.. Really appreciate the hard work. Just a small question if i inject more layers to the gpu since i have 144 GB of VRAM, will the RAM consumption go down and will i be able to allocate more context?

yes, theoretically you can use marling expert as KExperts' backend (but we haven't test this op, this may cause some bug orz, and it's loading will be veeeeeeery slow)

I would check our marlin expert op once down supporting DeepSeekv3.

Hi,
I’ve updated the Marlin backend and added a small example YAML in ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-marlin.yaml.
To fully utilize your VRAM, try injecting more KExpertsMarlin as the KTransformersExperts backend.

Please note:
• If you use KExpertsMarlin, you need to set --use_cuda_graph False. (This will be added to the documentation later.)
• Loading speed will be significantly slower as you add more KExpertsMarlin.

@Azure-Tang commented on GitHub (Feb 7, 2025): > > I have successfully deployed DeepSeek-V3 on the KTransformers framework, but there are still some accuracy issues that I am currently investigating. > > Based on preliminary tests, using the Q4KM weight file requires at least 16GB of VRAM and 480GB of RAM. On a positive note, the generation speed can reach approximately 3.8 tokens/s.” > > Thats wonderful.. Really appreciate the hard work. Just a small question if i inject more layers to the gpu since i have 144 GB of VRAM, will the RAM consumption go down and will i be able to allocate more context? > > Is it possible to run the DeepSeek R1 in just 4 A100 GPU (roughly 320 GB VRAM) using accelerate with offloading to disk? While this will be really really slow, but for some analysis purpose is it possible to run with this setup? > > It's possible. Loading process will be slow in this way, but the inference speed can be greatly accelerated. You can refer this: > > > > > I have successfully deployed DeepSeek-V3 on the KTransformers framework, but there are still some accuracy issues that I am currently investigating. > > > > Based on preliminary tests, using the Q4KM weight file requires at least 16GB of VRAM and 480GB of RAM. On a positive note, the generation speed can reach approximately 3.8 tokens/s.” > > > > > > > > > Thats wonderful.. Really appreciate the hard work. Just a small question if i inject more layers to the gpu since i have 144 GB of VRAM, will the RAM consumption go down and will i be able to allocate more context? > > > > > > yes, theoretically you can use marling expert as KExperts' backend (but we haven't test this op, this may cause some bug orz, and it's loading will be veeeeeeery slow) > > I would check our marlin expert op once down supporting DeepSeekv3. Hi, I’ve updated the Marlin backend and added a small example YAML in ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-marlin.yaml. To fully utilize your VRAM, try injecting more KExpertsMarlin as the KTransformersExperts backend. Please note: • If you use KExpertsMarlin, you need to set --use_cuda_graph False. (This will be added to the documentation later.) • Loading speed will be significantly slower as you add more KExpertsMarlin.

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Feb 7, 2025):

@RodriMora @ELigoP

So it seems NCCL speeds matter a LOT for TPS. I'm getting WAY faster speeds using 2 GPU's with NVLink than vanilla llama.cpp with the same 4x3090/3090 Ti setup with 14 layers offloaded. For me cuda:0 and cuda:2 are NVLinked together so when I replaced all cuda:1 instances with cuda:2 in ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml I saw a massive speedup of ~10tps which is 2x vanilla llama.cpp with --max_new_tokens 8192 --total_context 8192 and same context on vanilla llama.cpp.

Now to write the 4 GPU config! Any assitance with adapting the ktransformers/optimize/optimize_rules/DeepSeek-V2-Chat-multi-gpu-4.yaml to DeepSeek-V3/R1 would be appreciated @Azure-Tang! I've taken a VERY short look at the paper and it seems that the main difference here is that there is 1 dense layer in V2 vs the 3 in V3 and 60 layers in V2 vs 61 layers in V3 yet my regex for layer assignment seems to break despite my best efforts and reading the injection tutorial multiple times.

You can copy the v2 4-GPU YAML and make slight modifications. The changes needed can be identified by comparing the differences between DeepSeek-V2-Chat-multi-gpu and DeepSeek-V3-Chat-multi-gpu.
I’m sure this will be a piece of cake for you! >.-

@Azure-Tang commented on GitHub (Feb 7, 2025): > [@RodriMora](https://github.com/RodriMora) [@ELigoP](https://github.com/ELigoP) > > So it seems NCCL speeds matter a LOT for TPS. I'm getting WAY faster speeds using 2 GPU's with NVLink than vanilla llama.cpp with the same 4x3090/3090 Ti setup with 14 layers offloaded. For me `cuda:0` and `cuda:2` are NVLinked together so when I replaced all `cuda:1` instances with `cuda:2` in `ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml` I saw a massive speedup of ~10tps which is 2x vanilla llama.cpp with `--max_new_tokens 8192 --total_context 8192` and same context on vanilla llama.cpp. > > Now to write the 4 GPU config! Any assitance with adapting the `ktransformers/optimize/optimize_rules/DeepSeek-V2-Chat-multi-gpu-4.yaml` to DeepSeek-V3/R1 would be appreciated [@Azure-Tang](https://github.com/Azure-Tang)! I've taken a VERY short look at the paper and it seems that the main difference here is that there is 1 dense layer in V2 vs the 3 in V3 and 60 layers in V2 vs 61 layers in V3 yet my regex for layer assignment seems to break despite my best efforts and reading the injection tutorial multiple times. You can copy the v2 4-GPU YAML and make slight modifications. The changes needed can be identified by comparing the differences between DeepSeek-V2-Chat-multi-gpu and DeepSeek-V3-Chat-multi-gpu. I’m sure this will be a piece of cake for you! >.-

bsakaguchi commented

Author

Owner

@Nottlespike commented on GitHub (Feb 7, 2025):

@RodriMora @ELigoP
So it seems NCCL speeds matter a LOT for TPS. I'm getting WAY faster speeds using 2 GPU's with NVLink than vanilla llama.cpp with the same 4x3090/3090 Ti setup with 14 layers offloaded. For me cuda:0 and cuda:2 are NVLinked together so when I replaced all cuda:1 instances with cuda:2 in ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml I saw a massive speedup of ~10tps which is 2x vanilla llama.cpp with --max_new_tokens 8192 --total_context 8192 and same context on vanilla llama.cpp.
Now to write the 4 GPU config! Any assitance with adapting the ktransformers/optimize/optimize_rules/DeepSeek-V2-Chat-multi-gpu-4.yaml to DeepSeek-V3/R1 would be appreciated @Azure-Tang! I've taken a VERY short look at the paper and it seems that the main difference here is that there is 1 dense layer in V2 vs the 3 in V3 and 60 layers in V2 vs 61 layers in V3 yet my regex for layer assignment seems to break despite my best efforts and reading the injection tutorial multiple times.

You can copy the v2 4-GPU YAML and make slight modifications. The changes needed can be identified by comparing the differences between DeepSeek-V2-Chat-multi-gpu and DeepSeek-V3-Chat-multi-gpu. I’m sure this will be a piece of cake for you! >.-

哎呀

@Nottlespike commented on GitHub (Feb 7, 2025): > > [@RodriMora](https://github.com/RodriMora) [@ELigoP](https://github.com/ELigoP) > > So it seems NCCL speeds matter a LOT for TPS. I'm getting WAY faster speeds using 2 GPU's with NVLink than vanilla llama.cpp with the same 4x3090/3090 Ti setup with 14 layers offloaded. For me `cuda:0` and `cuda:2` are NVLinked together so when I replaced all `cuda:1` instances with `cuda:2` in `ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu.yaml` I saw a massive speedup of ~10tps which is 2x vanilla llama.cpp with `--max_new_tokens 8192 --total_context 8192` and same context on vanilla llama.cpp. > > Now to write the 4 GPU config! Any assitance with adapting the `ktransformers/optimize/optimize_rules/DeepSeek-V2-Chat-multi-gpu-4.yaml` to DeepSeek-V3/R1 would be appreciated [@Azure-Tang](https://github.com/Azure-Tang)! I've taken a VERY short look at the paper and it seems that the main difference here is that there is 1 dense layer in V2 vs the 3 in V3 and 60 layers in V2 vs 61 layers in V3 yet my regex for layer assignment seems to break despite my best efforts and reading the injection tutorial multiple times. > > You can copy the v2 4-GPU YAML and make slight modifications. The changes needed can be identified by comparing the differences between DeepSeek-V2-Chat-multi-gpu and DeepSeek-V3-Chat-multi-gpu. I’m sure this will be a piece of cake for you! >.- 哎呀

bsakaguchi commented

Author

Owner

@huliangbing commented on GitHub (Feb 7, 2025):

Waiting for the results of 4 GPUs ....

@huliangbing commented on GitHub (Feb 7, 2025): Waiting for the results of 4 GPUs ....

bsakaguchi commented

Author

Owner

@RodriMora commented on GitHub (Feb 7, 2025):

I did some test with my system:

Command to test:
ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/llms/models/DeepSeek-R1-UD-Q2_K_XL --total_context 1024 --max_new_tokens 512 --port 5000 --host 0.0.0.0 --cpu_infer 24 --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-marlin.yaml

	Default	--cpu_infer 22	--cpu_infer 24	--cpu_infer 46	--cpu_infer 48	--cpu_infer 24 --v3-2Gpu.yaml	--cpu_infer 24 --v3-4Gpu.yaml
Prefill	12.31613439	23.87405887	25.11264391	24.01843436	23.99479158	25.12282812	25.07483114
Decode	4.384906435	5.116141499	5.160205418	5.066774707	3.65675763	5.134406789	5.15631064
Time prefill	247.3178599	127.5861812	121.2934811	126.819257	126.9442158	121.2443116	121.4763913
Time decode	116.536124	99.87995839	99.02706552	66.3143754	139.7412822	99.52464247	99.10186481
Tokens prefill	3046	3046	3046	3046	3046	3046	3046
Tokens decode	511	511	511	336	511	511	511

For the 2 gpu optimize rules a used this (I would say same results as without any optimization rules, it seems to use 2 gpus anyways by default but not fully, like less than 50% each card):
https://github.com/kvcache-ai/ktransformers/blob/feat-DeepSeekV3/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-marlin.yaml

For the 4 gpu optimize rules I made this, also uses low vram only 4-6GB each card:
https://github.com/RodriMora/ktransformers/blob/main/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-marlin-4.yaml

Note: --use_cuda_graph False doesn't seem to be enabled as an arg while loading the openai api.

Edit: Comparasion vs llama.cpp:
Same prompt and tokens. llama.cpp with -ngl 20 with the gpus at 23/24Gb VRAM full. 40% increase with ktransformers

prompt eval time =   78999.88 ms /  3046 tokens (   25.94 ms per token,    38.56 tokens per second)
       eval time =  161916.39 ms /   512 tokens (  316.24 ms per token,     3.16 tokens per second)
      total time =  240916.27 ms /  3558 tokens

@RodriMora commented on GitHub (Feb 7, 2025): I did some test with my system: Command to test: `ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/llms/models/DeepSeek-R1-UD-Q2_K_XL --total_context 1024 --max_new_tokens 512 --port 5000 --host 0.0.0.0 --cpu_infer 24 --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-marlin.yaml` <google-sheets-html-origin> | Default | --cpu_infer 22 | --cpu_infer 24 | --cpu_infer 46 | --cpu_infer 48 | --cpu_infer 24 --v3-2Gpu.yaml | --cpu_infer 24 --v3-4Gpu.yaml -- | -- | -- | -- | -- | -- | -- | -- Prefill | 12.31613439 | 23.87405887 | 25.11264391 | 24.01843436 | 23.99479158 | 25.12282812 | 25.07483114 Decode | 4.384906435 | 5.116141499 | 5.160205418 | 5.066774707 | 3.65675763 | 5.134406789 | 5.15631064 Time prefill | 247.3178599 | 127.5861812 | 121.2934811 | 126.819257 | 126.9442158 | 121.2443116 | 121.4763913 Time decode | 116.536124 | 99.87995839 | 99.02706552 | 66.3143754 | 139.7412822 | 99.52464247 | 99.10186481 Tokens prefill | 3046 | 3046 | 3046 | 3046 | 3046 | 3046 | 3046 Tokens decode | 511 | 511 | 511 | 336 | 511 | 511 | 511 For the 2 gpu optimize rules a used this (I would say same results as without any optimization rules, it seems to use 2 gpus anyways by default but not fully, like less than 50% each card): https://github.com/kvcache-ai/ktransformers/blob/feat-DeepSeekV3/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-marlin.yaml For the 4 gpu optimize rules I made this, also uses low vram only 4-6GB each card: https://github.com/RodriMora/ktransformers/blob/main/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-marlin-4.yaml Note: --use_cuda_graph False doesn't seem to be enabled as an arg while loading the openai api. Edit: Comparasion vs llama.cpp: Same prompt and tokens. llama.cpp with -ngl 20 with the gpus at 23/24Gb VRAM full. 40% increase with ktransformers ``` prompt eval time = 78999.88 ms / 3046 tokens ( 25.94 ms per token, 38.56 tokens per second) eval time = 161916.39 ms / 512 tokens ( 316.24 ms per token, 3.16 tokens per second) total time = 240916.27 ms / 3558 tokens ```

bsakaguchi commented

Author

Owner

@KMSorSMS commented on GitHub (Feb 8, 2025):

I did some test with my system:

Command to test: ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/llms/models/DeepSeek-R1-UD-Q2_K_XL --total_context 1024 --max_new_tokens 512 --port 5000 --host 0.0.0.0 --cpu_infer 24 --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-marlin.yaml

Default --cpu_infer 22 --cpu_infer 24 --cpu_infer 46 --cpu_infer 48 --cpu_infer 24 --v3-2Gpu.yaml --cpu_infer 24 --v3-4Gpu.yaml
Prefill 12.31613439 23.87405887 25.11264391 24.01843436 23.99479158 25.12282812 25.07483114
Decode 4.384906435 5.116141499 5.160205418 5.066774707 3.65675763 5.134406789 5.15631064
Time prefill 247.3178599 127.5861812 121.2934811 126.819257 126.9442158 121.2443116 121.4763913
Time decode 116.536124 99.87995839 99.02706552 66.3143754 139.7412822 99.52464247 99.10186481
Tokens prefill 3046 3046 3046 3046 3046 3046 3046
Tokens decode 511 511 511 336 511 511 511
For the 2 gpu optimize rules a used this (I would say same results as without any optimization rules, it seems to use 2 gpus anyways by default but not fully, like less than 50% each card): https://github.com/kvcache-ai/ktransformers/blob/feat-DeepSeekV3/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-marlin.yaml

For the 4 gpu optimize rules I made this, also uses low vram only 4-6GB each card: https://github.com/RodriMora/ktransformers/blob/main/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-marlin-4.yaml

Note: --use_cuda_graph False doesn't seem to be enabled as an arg while loading the openai api.

Edit: Comparasion vs llama.cpp: Same prompt and tokens. llama.cpp with -ngl 20 with the gpus at 23/24Gb VRAM full. 40% increase with ktransformers
prompt eval time =   78999.88 ms /  3046 tokens (   25.94 ms per token,    38.56 tokens per second)
       eval time =  161916.39 ms /   512 tokens (  316.24 ms per token,     3.16 tokens per second)
      total time =  240916.27 ms /  3558 tokens

This testing result is consistent with our test which will come soon. Our performance video shows faster prefilling decoding compared with llama.cpp. Seems your CPU has about 24 cores on a single Numa Node and in multi GPU context also slows some performance ( As our multi GPU uses pipeline(pp), not tensor Tensor Parallel, So putting in a single GPU is best). You can try to modify .yaml of deepseekv3 to inject more marlin experts to fully utilize GPU and use command numactl -N 1 -m 1 to bind the infer on the same Numa node. And if you want to see more, you can set the --cache_lens , command arg, to 1536. But it will constrain your input and output lens, so be cautious. And our detailed test will come with a stable new release. Coming soon~ 😃

@KMSorSMS commented on GitHub (Feb 8, 2025): > I did some test with my system: > > Command to test: `ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/llms/models/DeepSeek-R1-UD-Q2_K_XL --total_context 1024 --max_new_tokens 512 --port 5000 --host 0.0.0.0 --cpu_infer 24 --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-marlin.yaml` > > Default --cpu_infer 22 --cpu_infer 24 --cpu_infer 46 --cpu_infer 48 --cpu_infer 24 --v3-2Gpu.yaml --cpu_infer 24 --v3-4Gpu.yaml > Prefill 12.31613439 23.87405887 25.11264391 24.01843436 23.99479158 25.12282812 25.07483114 > Decode 4.384906435 5.116141499 5.160205418 5.066774707 3.65675763 5.134406789 5.15631064 > Time prefill 247.3178599 127.5861812 121.2934811 126.819257 126.9442158 121.2443116 121.4763913 > Time decode 116.536124 99.87995839 99.02706552 66.3143754 139.7412822 99.52464247 99.10186481 > Tokens prefill 3046 3046 3046 3046 3046 3046 3046 > Tokens decode 511 511 511 336 511 511 511 > For the 2 gpu optimize rules a used this (I would say same results as without any optimization rules, it seems to use 2 gpus anyways by default but not fully, like less than 50% each card): https://github.com/kvcache-ai/ktransformers/blob/feat-DeepSeekV3/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-marlin.yaml > > For the 4 gpu optimize rules I made this, also uses low vram only 4-6GB each card: https://github.com/RodriMora/ktransformers/blob/main/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-marlin-4.yaml > > Note: --use_cuda_graph False doesn't seem to be enabled as an arg while loading the openai api. > > Edit: Comparasion vs llama.cpp: Same prompt and tokens. llama.cpp with -ngl 20 with the gpus at 23/24Gb VRAM full. 40% increase with ktransformers > > ``` > prompt eval time = 78999.88 ms / 3046 tokens ( 25.94 ms per token, 38.56 tokens per second) > eval time = 161916.39 ms / 512 tokens ( 316.24 ms per token, 3.16 tokens per second) > total time = 240916.27 ms / 3558 tokens > ``` This testing result is consistent with our test which will come soon. Our performance video shows faster prefilling decoding compared with llama.cpp. Seems your CPU has about 24 cores on a single Numa Node and in multi GPU context also slows some performance ( As our multi GPU uses pipeline(pp), not tensor Tensor Parallel, So putting in a single GPU is best). You can try to modify .yaml of deepseekv3 to inject more marlin experts to fully utilize GPU and use command `numactl -N 1 -m 1` to bind the infer on the same Numa node. And if you want to see more, you can set the --cache_lens <lens>, command arg, to 1536. But it will constrain your input and output lens, so be cautious. And our detailed test will come with a stable new release. Coming soon~ 😃

bsakaguchi commented

Author

Owner

@Azure-Tang commented on GitHub (Feb 10, 2025):

Hi! We’ve merged the PR supporting Deepseek-V3/R1 into the main branch, and we’ve also introduced additional optimizations for even better performance. For more details, please check our README and tutorial. Thank you for your support—if you find this helpful, we’d really appreciate any recommendations you share with your friends or community!

@Azure-Tang commented on GitHub (Feb 10, 2025): Hi! We’ve merged the PR supporting Deepseek-V3/R1 into the main branch, and we’ve also introduced additional optimizations for even better performance. For more details, please check our README and tutorial. Thank you for your support—if you find this helpful, we’d really appreciate any recommendations you share with your friends or community!

bsakaguchi commented

Author

Owner

@davidsyoung commented on GitHub (Feb 10, 2025):

Thank you for all of your work on this! If you could create some theoretical optimised YAML versions for more GPU's I could test it here on my side. I have 10x3090's, soon to be 12 with a 7713 EPYC 256gb 3200mhz.

@davidsyoung commented on GitHub (Feb 10, 2025): Thank you for all of your work on this! If you could create some theoretical optimised YAML versions for more GPU's I could test it here on my side. I have 10x3090's, soon to be 12 with a 7713 EPYC 256gb 3200mhz.

bsakaguchi commented

Author

Owner

@RodriMora commented on GitHub (Feb 10, 2025):

What does the "--cache_lens 1536" do? I got from 5.2t/s decode to 9t/s, using the current main branch compiled (so I think 0.2, not 0.3). That's an insane increase of speed. Is there any compromise?

For the rest, using either:
DeepSeek-V3-Chat.yaml
DeepSeek-V3-Chat-multi-gpu.yaml
DeepSeek-V3-Chat-multi-gpu-marlin.yaml

yields the same results. I think using the single gpu DeepSeek-V3-Chat.yaml for longer context give OOM errors, the rest works fine but with similar performance. 24-25 prefill and 5 decode.

I'll try 0.3 next

@RodriMora commented on GitHub (Feb 10, 2025): What does the "--cache_lens 1536" do? I got from 5.2t/s decode to 9t/s, using the current main branch compiled (so I think 0.2, not 0.3). That's an insane increase of speed. Is there any compromise? For the rest, using either: DeepSeek-V3-Chat.yaml DeepSeek-V3-Chat-multi-gpu.yaml DeepSeek-V3-Chat-multi-gpu-marlin.yaml yields the same results. I think using the single gpu DeepSeek-V3-Chat.yaml for longer context give OOM errors, the rest works fine but with similar performance. 24-25 prefill and 5 decode. I'll try 0.3 next

bsakaguchi commented

Author

Owner

@RodriMora commented on GitHub (Feb 10, 2025):

I installed 0.3.

Few notes, in my Ubuntu 22.04 installation was needed to add the:
sudo add-apt-repository ppa:ubuntu-toolchain-r/test
sudo apt-get update
sudo apt-get install --only-upgrade libstdc++6

Othewise it gives this error:
ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version GLIBCXX_3.4.32' not found (required by /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/KTransformersOps.cpython-311-x86_64-linux-gnu.so)`

But I guess my CPU is not compatible? EPYC 7402

(.venv) ubuntuai@ubuntuai ~/ktransformers (main) [0|SIGILL]> ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/llms/models/DeepSeek-R1-UD-Q2_K_XL/ --max_new_tokens 512 --length 512 --total_context 512 --port 5000 --host 0.0.0.0 --cpu_infer 24
2025-02-10 11:14:11,145 INFO /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/ktransformers/server/main.py[29]: Creating SQL tables
2025-02-10 11:14:11,147 INFO /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant
__AVX512F__
fish: Job 1, 'ktransformers --model_path deep…' terminated by signal SIGILL (Illegal instruction)
(.venv) ubuntuai@ubuntuai ~/ktransformers (main) [0|SIGILL]>

@RodriMora commented on GitHub (Feb 10, 2025): I installed 0.3. Few notes, in my Ubuntu 22.04 installation was needed to add the: sudo add-apt-repository ppa:ubuntu-toolchain-r/test sudo apt-get update sudo apt-get install --only-upgrade libstdc++6 Othewise it gives this error: `ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/KTransformersOps.cpython-311-x86_64-linux-gnu.so)` But I guess my CPU is not compatible? EPYC 7402 ``` (.venv) ubuntuai@ubuntuai ~/ktransformers (main) [0|SIGILL]> ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/llms/models/DeepSeek-R1-UD-Q2_K_XL/ --max_new_tokens 512 --length 512 --total_context 512 --port 5000 --host 0.0.0.0 --cpu_infer 24 2025-02-10 11:14:11,145 INFO /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/ktransformers/server/main.py[29]: Creating SQL tables 2025-02-10 11:14:11,147 INFO /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant __AVX512F__ fish: Job 1, 'ktransformers --model_path deep…' terminated by signal SIGILL (Illegal instruction) (.venv) ubuntuai@ubuntuai ~/ktransformers (main) [0|SIGILL]> ```

bsakaguchi commented

Author

Owner

@KMSorSMS commented on GitHub (Feb 10, 2025):

What does the "--cache_lens 1536" do? I got from 5.2t/s decode to 9t/s, using the current main branch compiled (so I think 0.2, not 0.3). That's an insane increase of speed. Is there any compromise?

For the rest, using either: DeepSeek-V3-Chat.yaml DeepSeek-V3-Chat-multi-gpu.yaml DeepSeek-V3-Chat-multi-gpu-marlin.yaml

yields the same results. I think using the single gpu DeepSeek-V3-Chat.yaml for longer context give OOM errors, the rest works fine but with similar performance. 24-25 prefill and 5 decode.

I'll try 0.3 next

TL;DR

The --cache_lens has no use.
It's because --cache_lens used to be an arg which indicates a static area to compute attention even if there is no such a huge prompt to calculate. So in the older version, we set it smaller to our actual prompt will make it faster (as the default is too large). Now the new version we set this to the prompt length and your max_nex_token size. (But it can't support history context now).

Some further intro:
Actually, If you use the current main branch( which is our release V0.2), we get rid of the arg cache-lens in local_chat. (But it remains in server backend, so we just keep this in the tutorial command ). The more insane increase can come from our support for dual-socket support ( make sure you first set env var USE_NUMA export USE_NUMA=1, see the link in detail). Tips: you may need to use apt to install the libnuma. If you don't use this, you can also check our best test on a single socket. link.

@KMSorSMS commented on GitHub (Feb 10, 2025): > What does the "--cache_lens 1536" do? I got from 5.2t/s decode to 9t/s, using the current main branch compiled (so I think 0.2, not 0.3). That's an insane increase of speed. Is there any compromise? > > For the rest, using either: DeepSeek-V3-Chat.yaml DeepSeek-V3-Chat-multi-gpu.yaml DeepSeek-V3-Chat-multi-gpu-marlin.yaml > > yields the same results. I think using the single gpu DeepSeek-V3-Chat.yaml for longer context give OOM errors, the rest works fine but with similar performance. 24-25 prefill and 5 decode. > > I'll try 0.3 next TL;DR The --cache_lens has no use. It's because --cache_lens used to be an arg which indicates a static area to compute attention even if there is no such a huge prompt to calculate. So in the older version, we set it smaller to our actual prompt will make it faster (as the default is too large). Now the new version we set this to the prompt length and your max_nex_token size. (But it can't support history context now). ----------------------------------- Some further intro: Actually, If you use the current main branch( which is our release V0.2), we get rid of the arg cache-lens in local_chat. (But it remains in server backend, so we just keep this in the [tutorial command](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md#single-socket-version-32-cores) ). The more insane increase can come from our support for dual-socket support ( make sure you first set env var USE_NUMA `export USE_NUMA=1`, see the [link](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md#dual-socket-version-64-cores) in detail). Tips: you may need to use apt to install the libnuma. If you don't use this, you can also check our best test on a single socket. [link](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md#single-socket-version-32-cores).

bsakaguchi commented

Author

Owner

@chenht2022 commented on GitHub (Feb 10, 2025):

I installed 0.3.

Few notes, in my Ubuntu 22.04 installation was needed to add the: sudo add-apt-repository ppa:ubuntu-toolchain-r/test sudo apt-get update sudo apt-get install --only-upgrade libstdc++6

Othewise it gives this error: ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version GLIBCXX_3.4.32' not found (required by /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/KTransformersOps.cpython-311-x86_64-linux-gnu.so)`

But I guess my CPU is not compatible? EPYC 7402
(.venv) ubuntuai@ubuntuai ~/ktransformers (main) [0|SIGILL]> ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/llms/models/DeepSeek-R1-UD-Q2_K_XL/ --max_new_tokens 512 --length 512 --total_context 512 --port 5000 --host 0.0.0.0 --cpu_infer 24
2025-02-10 11:14:11,145 INFO /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/ktransformers/server/main.py[29]: Creating SQL tables
2025-02-10 11:14:11,147 INFO /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant
__AVX512F__
fish: Job 1, 'ktransformers --model_path deep…' terminated by signal SIGILL (Illegal instruction)
(.venv) ubuntuai@ubuntuai ~/ktransformers (main) [0|SIGILL]>

AMD CPUs like EPYC does not support AMX instruction, our V0.3 currently only supports Intel Xeon CPUs.

@chenht2022 commented on GitHub (Feb 10, 2025): > I installed 0.3. > > Few notes, in my Ubuntu 22.04 installation was needed to add the: sudo add-apt-repository ppa:ubuntu-toolchain-r/test sudo apt-get update sudo apt-get install --only-upgrade libstdc++6 > > Othewise it gives this error: `ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/KTransformersOps.cpython-311-x86_64-linux-gnu.so)` > > But I guess my CPU is not compatible? EPYC 7402 > > ``` > (.venv) ubuntuai@ubuntuai ~/ktransformers (main) [0|SIGILL]> ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/llms/models/DeepSeek-R1-UD-Q2_K_XL/ --max_new_tokens 512 --length 512 --total_context 512 --port 5000 --host 0.0.0.0 --cpu_infer 24 > 2025-02-10 11:14:11,145 INFO /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/ktransformers/server/main.py[29]: Creating SQL tables > 2025-02-10 11:14:11,147 INFO /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant > __AVX512F__ > fish: Job 1, 'ktransformers --model_path deep…' terminated by signal SIGILL (Illegal instruction) > (.venv) ubuntuai@ubuntuai ~/ktransformers (main) [0|SIGILL]> > ``` AMD CPUs like EPYC does not support AMX instruction, our V0.3 currently only supports Intel Xeon CPUs.

bsakaguchi commented

Author

Owner

@RodriMora commented on GitHub (Feb 10, 2025):

What does the "--cache_lens 1536" do? I got from 5.2t/s decode to 9t/s, using the current main branch compiled (so I think 0.2, not 0.3). That's an insane increase of speed. Is there any compromise?
For the rest, using either: DeepSeek-V3-Chat.yaml DeepSeek-V3-Chat-multi-gpu.yaml DeepSeek-V3-Chat-multi-gpu-marlin.yaml
yields the same results. I think using the single gpu DeepSeek-V3-Chat.yaml for longer context give OOM errors, the rest works fine but with similar performance. 24-25 prefill and 5 decode.
I'll try 0.3 next

TL;DR

The --cache_lens has no use. It's because --cache_lens used to be an arg which indicates a static area to compute attention even if there is no such a huge prompt to calculate. So in the older version, we set it smaller to our actual prompt will make it faster (as the default is too large). Now the new version we set this to the prompt length and your max_nex_token size. (But it can't support history context now).

Some further intro: Actually, If you use the current main branch( which is our release V0.2), we get rid of the arg cache-lens in local_chat. (But it remains in server backend, so we just keep this in the tutorial command ). The more insane increase can come from our support for dual-socket support ( make sure you first set env var USE_NUMA export USE_NUMA=1, see the link in detail). Tips: you may need to use apt to install the libnuma. If you don't use this, you can also check our best test on a single socket. link.

numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
node 0 size: 515634 MB
node 0 free: 14832 MB
node distances:
node   0
  0:  10

My server only has 1 socket and 1 numa, so it doesn't get any benefit from numactl -N 1 -m 1 (or numactl -N 0 -m 0 in my case).
The --cache_lens 1536 though gives a 80% increase in speed

@RodriMora commented on GitHub (Feb 10, 2025): > > What does the "--cache_lens 1536" do? I got from 5.2t/s decode to 9t/s, using the current main branch compiled (so I think 0.2, not 0.3). That's an insane increase of speed. Is there any compromise? > > For the rest, using either: DeepSeek-V3-Chat.yaml DeepSeek-V3-Chat-multi-gpu.yaml DeepSeek-V3-Chat-multi-gpu-marlin.yaml > > yields the same results. I think using the single gpu DeepSeek-V3-Chat.yaml for longer context give OOM errors, the rest works fine but with similar performance. 24-25 prefill and 5 decode. > > I'll try 0.3 next > > TL;DR > > The --cache_lens has no use. It's because --cache_lens used to be an arg which indicates a static area to compute attention even if there is no such a huge prompt to calculate. So in the older version, we set it smaller to our actual prompt will make it faster (as the default is too large). Now the new version we set this to the prompt length and your max_nex_token size. (But it can't support history context now). > > Some further intro: Actually, If you use the current main branch( which is our release V0.2), we get rid of the arg cache-lens in local_chat. (But it remains in server backend, so we just keep this in the [tutorial command](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md#single-socket-version-32-cores) ). The more insane increase can come from our support for dual-socket support ( make sure you first set env var USE_NUMA `export USE_NUMA=1`, see the [link](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md#dual-socket-version-64-cores) in detail). Tips: you may need to use apt to install the libnuma. If you don't use this, you can also check our best test on a single socket. [link](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md#single-socket-version-32-cores). ``` numactl --hardware available: 1 nodes (0) node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 node 0 size: 515634 MB node 0 free: 14832 MB node distances: node 0 0: 10 ``` My server only has 1 socket and 1 numa, so it doesn't get any benefit from numactl -N 1 -m 1 (or numactl -N 0 -m 0 in my case). The --cache_lens 1536 though gives a 80% increase in speed

bsakaguchi commented

Author

Owner

@KMSorSMS commented on GitHub (Feb 10, 2025):

What does the "--cache_lens 1536" do? I got from 5.2t/s decode to 9t/s, using the current main branch compiled (so I think 0.2, not 0.3). That's an insane increase of speed. Is there any compromise?
For the rest, using either: DeepSeek-V3-Chat.yaml DeepSeek-V3-Chat-multi-gpu.yaml DeepSeek-V3-Chat-multi-gpu-marlin.yaml
yields the same results. I think using the single gpu DeepSeek-V3-Chat.yaml for longer context give OOM errors, the rest works fine but with similar performance. 24-25 prefill and 5 decode.
I'll try 0.3 next

TL;DR
The --cache_lens has no use. It's because --cache_lens used to be an arg which indicates a static area to compute attention even if there is no such a huge prompt to calculate. So in the older version, we set it smaller to our actual prompt will make it faster (as the default is too large). Now the new version we set this to the prompt length and your max_nex_token size. (But it can't support history context now).
Some further intro: Actually, If you use the current main branch( which is our release V0.2), we get rid of the arg cache-lens in local_chat. (But it remains in server backend, so we just keep this in the tutorial command ). The more insane increase can come from our support for dual-socket support ( make sure you first set env var USE_NUMA export USE_NUMA=1, see the link in detail). Tips: you may need to use apt to install the libnuma. If you don't use this, you can also check our best test on a single socket. link.
numactl --hardware
available: 1 nodes (0)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47
node 0 size: 515634 MB
node 0 free: 14832 MB
node distances:
node   0
  0:  10
My server only has 1 socket and 1 numa, so it doesn't get any benefit from numactl -N 1 -m 1 (or numactl -N 0 -m 0 in my case). The --cache_lens 1536 though gives a 80% increase in speed

Have you test again without the --cache_lens args? As it has no use now in local_chat run.

@KMSorSMS commented on GitHub (Feb 10, 2025): > > > What does the "--cache_lens 1536" do? I got from 5.2t/s decode to 9t/s, using the current main branch compiled (so I think 0.2, not 0.3). That's an insane increase of speed. Is there any compromise? > > > For the rest, using either: DeepSeek-V3-Chat.yaml DeepSeek-V3-Chat-multi-gpu.yaml DeepSeek-V3-Chat-multi-gpu-marlin.yaml > > > yields the same results. I think using the single gpu DeepSeek-V3-Chat.yaml for longer context give OOM errors, the rest works fine but with similar performance. 24-25 prefill and 5 decode. > > > I'll try 0.3 next > > > > > > TL;DR > > The --cache_lens has no use. It's because --cache_lens used to be an arg which indicates a static area to compute attention even if there is no such a huge prompt to calculate. So in the older version, we set it smaller to our actual prompt will make it faster (as the default is too large). Now the new version we set this to the prompt length and your max_nex_token size. (But it can't support history context now). > > Some further intro: Actually, If you use the current main branch( which is our release V0.2), we get rid of the arg cache-lens in local_chat. (But it remains in server backend, so we just keep this in the [tutorial command](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md#single-socket-version-32-cores) ). The more insane increase can come from our support for dual-socket support ( make sure you first set env var USE_NUMA `export USE_NUMA=1`, see the [link](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md#dual-socket-version-64-cores) in detail). Tips: you may need to use apt to install the libnuma. If you don't use this, you can also check our best test on a single socket. [link](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md#single-socket-version-32-cores). > > ``` > numactl --hardware > available: 1 nodes (0) > node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 > node 0 size: 515634 MB > node 0 free: 14832 MB > node distances: > node 0 > 0: 10 > ``` > > My server only has 1 socket and 1 numa, so it doesn't get any benefit from numactl -N 1 -m 1 (or numactl -N 0 -m 0 in my case). The --cache_lens 1536 though gives a 80% increase in speed Have you test again without the --cache_lens args? As it has no use now in local_chat run.

bsakaguchi commented

Author

Owner

@RodriMora commented on GitHub (Feb 10, 2025):

Have you test again without the --cache_lens args? As it has no use now in local_chat run.

Yes. Got the same performance on local_chat. The difference is in the openai api server.

With --cache_lens
prompt eval count: 16 token(s)
prompt eval duration: 1.1299359798431396s
prompt eval rate: 14.160094275625383 tokens/s
eval count: 1000 token(s)
eval duration: 98.3808274269104s
eval rate: 10.164582125952592 tokens/s

Without: --cache_lens
prompt eval count: 16 token(s)
prompt eval duration: 1.133697271347046s
prompt eval rate: 14.11311503024876 tokens/s
eval count: 1000 token(s)
eval duration: 98.45745468139648s
eval rate: 10.156671257000815 tokens/s

@RodriMora commented on GitHub (Feb 10, 2025): > Have you test again without the --cache_lens args? As it has no use now in local_chat run. Yes. Got the same performance on local_chat. The difference is in the openai api server. With --cache_lens prompt eval count: 16 token(s) prompt eval duration: 1.1299359798431396s prompt eval rate: 14.160094275625383 tokens/s eval count: 1000 token(s) eval duration: 98.3808274269104s eval rate: 10.164582125952592 tokens/s Without: --cache_lens prompt eval count: 16 token(s) prompt eval duration: 1.133697271347046s prompt eval rate: 14.11311503024876 tokens/s eval count: 1000 token(s) eval duration: 98.45745468139648s eval rate: 10.156671257000815 tokens/s

bsakaguchi commented

Author

Owner

@KMSorSMS commented on GitHub (Feb 10, 2025):

Have you test again without the --cache_lens args? As it has no use now in local_chat run.

Yes. Got the same performance on local_chat. The difference is in the openai api server.

With --cache_lens prompt eval count: 16 token(s) prompt eval duration: 1.1299359798431396s prompt eval rate: 14.160094275625383 tokens/s eval count: 1000 token(s) eval duration: 98.3808274269104s eval rate: 10.164582125952592 tokens/s

Without: --cache_lens prompt eval count: 16 token(s) prompt eval duration: 1.133697271347046s prompt eval rate: 14.11311503024876 tokens/s eval count: 1000 token(s) eval duration: 98.45745468139648s eval rate: 10.156671257000815 tokens/s

Good! This is exactly what we have test and post on the Report(our result is 8.73 tokens/s in decoding as our single socket only get 32 cores). The cache_lens args have no use in the current version of our local_chat.py.

@KMSorSMS commented on GitHub (Feb 10, 2025): > > Have you test again without the --cache_lens args? As it has no use now in local_chat run. > > Yes. Got the same performance on local_chat. The difference is in the openai api server. > > With --cache_lens prompt eval count: 16 token(s) prompt eval duration: 1.1299359798431396s prompt eval rate: 14.160094275625383 tokens/s eval count: 1000 token(s) eval duration: 98.3808274269104s eval rate: 10.164582125952592 tokens/s > > Without: --cache_lens prompt eval count: 16 token(s) prompt eval duration: 1.133697271347046s prompt eval rate: 14.11311503024876 tokens/s eval count: 1000 token(s) eval duration: 98.45745468139648s eval rate: 10.156671257000815 tokens/s Good! This is exactly what we have test and post on the [Report](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/DeepseekR1_V3_tutorial.md#benchmark-results)(our result is 8.73 tokens/s in decoding as our single socket only get 32 cores). The cache_lens args have no use in the current version of our local_chat.py.

bsakaguchi commented

Author

Owner

@ArYuZzz commented on GitHub (Feb 11, 2025):

I installed 0.3.
Few notes, in my Ubuntu 22.04 installation was needed to add the: sudo add-apt-repository ppa:ubuntu-toolchain-r/test sudo apt-get update sudo apt-get install --only-upgrade libstdc++6
Othewise it gives this error: ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version GLIBCXX_3.4.32' not found (required by /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/KTransformersOps.cpython-311-x86_64-linux-gnu.so)`
But I guess my CPU is not compatible? EPYC 7402
(.venv) ubuntuai@ubuntuai ~/ktransformers (main) [0|SIGILL]> ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/llms/models/DeepSeek-R1-UD-Q2_K_XL/ --max_new_tokens 512 --length 512 --total_context 512 --port 5000 --host 0.0.0.0 --cpu_infer 24
2025-02-10 11:14:11,145 INFO /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/ktransformers/server/main.py[29]: Creating SQL tables
2025-02-10 11:14:11,147 INFO /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant
__AVX512F__
fish: Job 1, 'ktransformers --model_path deep…' terminated by signal SIGILL (Illegal instruction)
(.venv) ubuntuai@ubuntuai ~/ktransformers (main) [0|SIGILL]>
AMD CPUs like EPYC does not support AMX instruction, our V0.3 currently only supports Intel Xeon CPUs.

Does v0.3 supports earlier Xeon CPUs like gen3 or gen2 which only support AVX-512?
And if not, does v0.2 support AVX-512 and can also accelerate the inference of deepseek?

@ArYuZzz commented on GitHub (Feb 11, 2025): > > I installed 0.3. > > Few notes, in my Ubuntu 22.04 installation was needed to add the: sudo add-apt-repository ppa:ubuntu-toolchain-r/test sudo apt-get update sudo apt-get install --only-upgrade libstdc++6 > > Othewise it gives this error: `ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/KTransformersOps.cpython-311-x86_64-linux-gnu.so)` > > But I guess my CPU is not compatible? EPYC 7402 > > ``` > > (.venv) ubuntuai@ubuntuai ~/ktransformers (main) [0|SIGILL]> ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/llms/models/DeepSeek-R1-UD-Q2_K_XL/ --max_new_tokens 512 --length 512 --total_context 512 --port 5000 --host 0.0.0.0 --cpu_infer 24 > > 2025-02-10 11:14:11,145 INFO /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/ktransformers/server/main.py[29]: Creating SQL tables > > 2025-02-10 11:14:11,147 INFO /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant > > __AVX512F__ > > fish: Job 1, 'ktransformers --model_path deep…' terminated by signal SIGILL (Illegal instruction) > > (.venv) ubuntuai@ubuntuai ~/ktransformers (main) [0|SIGILL]> > > ``` > > AMD CPUs like EPYC does not support AMX instruction, our V0.3 currently only supports Intel Xeon CPUs. Does v0.3 supports earlier Xeon CPUs like gen3 or gen2 which only support AVX-512? And if not, does v0.2 support AVX-512 and can also accelerate the inference of deepseek?

bsakaguchi commented

Author

Owner

@KMSorSMS commented on GitHub (Feb 11, 2025):

I installed 0.3.
Few notes, in my Ubuntu 22.04 installation was needed to add the: sudo add-apt-repository ppa:ubuntu-toolchain-r/test sudo apt-get update sudo apt-get install --only-upgrade libstdc++6
Othewise it gives this error: ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version GLIBCXX_3.4.32' not found (required by /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/KTransformersOps.cpython-311-x86_64-linux-gnu.so)`
But I guess my CPU is not compatible? EPYC 7402
(.venv) ubuntuai@ubuntuai ~/ktransformers (main) [0|SIGILL]> ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/llms/models/DeepSeek-R1-UD-Q2_K_XL/ --max_new_tokens 512 --length 512 --total_context 512 --port 5000 --host 0.0.0.0 --cpu_infer 24
2025-02-10 11:14:11,145 INFO /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/ktransformers/server/main.py[29]: Creating SQL tables
2025-02-10 11:14:11,147 INFO /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant
__AVX512F__
fish: Job 1, 'ktransformers --model_path deep…' terminated by signal SIGILL (Illegal instruction)
(.venv) ubuntuai@ubuntuai ~/ktransformers (main) [0|SIGILL]>
AMD CPUs like EPYC does not support AMX instruction, our V0.3 currently only supports Intel Xeon CPUs.
Does v0.3 supports earlier Xeon CPUs like gen3 or gen2 which only support AVX-512? And if not, does v0.2 support AVX-512 and can also accelerate the inference of deepseek?

The preview version of V0.3 doesn't support earlier Xeon CPU. But V0.2 can support as it doesn't contain AMX instruction acceleration implementation. So you can try V0.2 to accelerate the inference of deepseek

@KMSorSMS commented on GitHub (Feb 11, 2025): > > > I installed 0.3. > > > Few notes, in my Ubuntu 22.04 installation was needed to add the: sudo add-apt-repository ppa:ubuntu-toolchain-r/test sudo apt-get update sudo apt-get install --only-upgrade libstdc++6 > > > Othewise it gives this error: `ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/KTransformersOps.cpython-311-x86_64-linux-gnu.so)` > > > But I guess my CPU is not compatible? EPYC 7402 > > > ``` > > > (.venv) ubuntuai@ubuntuai ~/ktransformers (main) [0|SIGILL]> ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/llms/models/DeepSeek-R1-UD-Q2_K_XL/ --max_new_tokens 512 --length 512 --total_context 512 --port 5000 --host 0.0.0.0 --cpu_infer 24 > > > 2025-02-10 11:14:11,145 INFO /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/ktransformers/server/main.py[29]: Creating SQL tables > > > 2025-02-10 11:14:11,147 INFO /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant > > > __AVX512F__ > > > fish: Job 1, 'ktransformers --model_path deep…' terminated by signal SIGILL (Illegal instruction) > > > (.venv) ubuntuai@ubuntuai ~/ktransformers (main) [0|SIGILL]> > > > ``` > > > > > > AMD CPUs like EPYC does not support AMX instruction, our V0.3 currently only supports Intel Xeon CPUs. > > Does v0.3 supports earlier Xeon CPUs like gen3 or gen2 which only support AVX-512? And if not, does v0.2 support AVX-512 and can also accelerate the inference of deepseek? The preview version of V0.3 doesn't support earlier Xeon CPU. But V0.2 can support as it doesn't contain AMX instruction acceleration implementation. So you can try V0.2 to accelerate the inference of deepseek

bsakaguchi commented

Author

Owner

@chenht2022 commented on GitHub (Feb 11, 2025):

AMX instructions are not supported on Xeon gen2 and gen3 , and you can also check the supported instruction sets through command lscpu. If AMX instruction are supported, there will be flags such as amx_bf16, amx_tile, amx_int8
V0.2 will automatically use the AVX512 instruction set on hardware that supports AVX512

@chenht2022 commented on GitHub (Feb 11, 2025): AMX instructions are not supported on Xeon gen2 and gen3 , and you can also check the supported instruction sets through command `lscpu`. If AMX instruction are supported, there will be flags such as amx_bf16, amx_tile, amx_int8 V0.2 will automatically use the AVX512 instruction set on hardware that supports AVX512

bsakaguchi commented

https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-BF16

Author

Owner

@WuNein commented on GitHub (Feb 11, 2025):

must I download the BF16 version gguf to use V0.3?

It is giant!!

@WuNein commented on GitHub (Feb 11, 2025): must I download the BF16 version gguf to use V0.3? https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-BF16 It is giant!!

bsakaguchi commented

Author

Owner

@chenht2022 commented on GitHub (Feb 11, 2025):

must I download the BF16 version gguf to use V0.3?

https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-BF16

It is giant!!

Yes, for V0.3 preview version. We will consider optimizing it in the official release version, maybe support online dequantization.

@chenht2022 commented on GitHub (Feb 11, 2025): > must I download the BF16 version gguf to use V0.3? > > https://huggingface.co/unsloth/DeepSeek-R1-GGUF/tree/main/DeepSeek-R1-BF16 > > It is giant!! Yes, for V0.3 preview version. We will consider optimizing it in the official release version, maybe support online dequantization.

bsakaguchi commented

Author

Owner

@ArYuZzz commented on GitHub (Feb 11, 2025):

I installed 0.3.
Few notes, in my Ubuntu 22.04 installation was needed to add the: sudo add-apt-repository ppa:ubuntu-toolchain-r/test sudo apt-get update sudo apt-get install --only-upgrade libstdc++6
Othewise it gives this error: ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version GLIBCXX_3.4.32' not found (required by /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/KTransformersOps.cpython-311-x86_64-linux-gnu.so)`
But I guess my CPU is not compatible? EPYC 7402
(.venv) ubuntuai@ubuntuai ~/ktransformers (main) [0|SIGILL]> ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/llms/models/DeepSeek-R1-UD-Q2_K_XL/ --max_new_tokens 512 --length 512 --total_context 512 --port 5000 --host 0.0.0.0 --cpu_infer 24
2025-02-10 11:14:11,145 INFO /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/ktransformers/server/main.py[29]: Creating SQL tables
2025-02-10 11:14:11,147 INFO /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant
__AVX512F__
fish: Job 1, 'ktransformers --model_path deep…' terminated by signal SIGILL (Illegal instruction)
(.venv) ubuntuai@ubuntuai ~/ktransformers (main) [0|SIGILL]>
AMD CPUs like EPYC does not support AMX instruction, our V0.3 currently only supports Intel Xeon CPUs.
Does v0.3 supports earlier Xeon CPUs like gen3 or gen2 which only support AVX-512? And if not, does v0.2 support AVX-512 and can also accelerate the inference of deepseek?
The preview version of V0.3 doesn't support earlier Xeon CPU. But V0.2 can support as it doesn't contain AMX instruction acceleration implementation. So you can try V0.2 to accelerate the inference of deepseek

Thanks for reply and contribution. Will the upcoming v0.3 supports AVX512?

@ArYuZzz commented on GitHub (Feb 11, 2025): > > > > I installed 0.3. > > > > Few notes, in my Ubuntu 22.04 installation was needed to add the: sudo add-apt-repository ppa:ubuntu-toolchain-r/test sudo apt-get update sudo apt-get install --only-upgrade libstdc++6 > > > > Othewise it gives this error: `ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/KTransformersOps.cpython-311-x86_64-linux-gnu.so)` > > > > But I guess my CPU is not compatible? EPYC 7402 > > > > ``` > > > > (.venv) ubuntuai@ubuntuai ~/ktransformers (main) [0|SIGILL]> ktransformers --model_path deepseek-ai/DeepSeek-R1 --gguf_path /mnt/llms/models/DeepSeek-R1-UD-Q2_K_XL/ --max_new_tokens 512 --length 512 --total_context 512 --port 5000 --host 0.0.0.0 --cpu_infer 24 > > > > 2025-02-10 11:14:11,145 INFO /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/ktransformers/server/main.py[29]: Creating SQL tables > > > > 2025-02-10 11:14:11,147 INFO /home/ubuntuai/ktransformers/.venv/lib/python3.11/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant > > > > __AVX512F__ > > > > fish: Job 1, 'ktransformers --model_path deep…' terminated by signal SIGILL (Illegal instruction) > > > > (.venv) ubuntuai@ubuntuai ~/ktransformers (main) [0|SIGILL]> > > > > ``` > > > > > > > > > AMD CPUs like EPYC does not support AMX instruction, our V0.3 currently only supports Intel Xeon CPUs. > > > > > > Does v0.3 supports earlier Xeon CPUs like gen3 or gen2 which only support AVX-512? And if not, does v0.2 support AVX-512 and can also accelerate the inference of deepseek? > > The preview version of V0.3 doesn't support earlier Xeon CPU. But V0.2 can support as it doesn't contain AMX instruction acceleration implementation. So you can try V0.2 to accelerate the inference of deepseek Thanks for reply and contribution. Will the upcoming v0.3 supports AVX512?

bsakaguchi commented