mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #79] Is deepseek-ai/DeepSeek-V2.5 supported? #39
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#39
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @AshD on GitHub (Sep 7, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/79
I tried running bartowski/DeepSeek-V2.5-GGUF on a Linux box with 512GB RAM/192GB VRAM.
python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-V2.5 --gguf_path ~/ai/llms/DeepSeek-V2.5-Q8_0-00001-of-00007.gguf
File "/home/ash/ai/ktransformers/ktransformers/operators/base_operator.py", line 60, in load
utils.load_weights(child, self.gguf_loader, self.key+".")
File "/home/ash/ai/ktransformers/ktransformers/util/utils.py", line 83, in load_weights
load_weights(child, gguf_loader, prefix+name+".")
File "/home/ash/ai/ktransformers/ktransformers/util/utils.py", line 81, in load_weights
load_cur_state_dict(module, gguf_loader, prefix)
File "/home/ash/ai/ktransformers/ktransformers/util/utils.py", line 76, in load_cur_state_dict
raise Exception(f"can't find {translated_key} in GGUF file!")
Exception: can't find token_embd.weight in GGUF file!
@Azure-Tang commented on GitHub (Sep 8, 2024):
You should input dir rather than a gguf file to
--gguf_path. And make sure your gguf files end with.gguf~@AshD commented on GitHub (Sep 8, 2024):
Thanks @Azure-Tang It works now with
ktransformers --model_path deepseek-ai/DeepSeek-V2-Chat --gguf_path ~/ai/llms/DeepSeek-V2.5 --port 8080 --cpu_infer 64
Couple of issues
I tried the streaming OpenAI API and it did not work but Non-Streaming one worked fine. I assume streaming API is not supported.
How do I get the multi-GPU to work? I made a copy of ktransformers/optimize/optimize_rules/DeepSeek-V2-Chat-multi-gpu-4.yaml and called it DeepSeek-V2.5-multi-gpu-4.yaml
But it loads the model only on 1 of my 4 GPUs (RTX 6000 Ada)
Thanks,
Ash
@Azure-Tang commented on GitHub (Sep 10, 2024):
--optimize_config_pathin server. The detailed yaml writing tutorial here@sammcj commented on GitHub (Sep 10, 2024):
I've been trying to get DeepSeek V2.5 working but have hit 'NotImplementedError: ggml_type 21 not implemented' errors:
Not sure if it's a lack of DeepSeek V2 support, or perhaps my config?
*edit: nope, not my config I tried with the default as well
@sammcj commented on GitHub (Sep 10, 2024):
Oh, I just spotted:
That will be why.
@UnicornChan commented on GitHub (Sep 10, 2024):
Could you please show the code you use with stream? This is the code I use for testing, and the version of OpenAI client is 1.44.1. @AshD
@AshD commented on GitHub (Sep 10, 2024):
Thanks @Azure-Tang It's working now with --optimize_config_path
BTW, the home page, it the model arguments section calls this parameter --optimize_rule_path
@AshD commented on GitHub (Sep 10, 2024):
I tested this with our Fusion Quill client which uses a custom C# implementation. Will test some other client.
@AshD commented on GitHub (Sep 10, 2024):
Thanks @UnicornChan
There was an issue where our streaming client was expecting "data: " and ktransformers was sending "data:" without the space. I handled that condition, so I am closing this issue. Thanks for all your help.