mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #101] Long prompt with DeepSeek crashing with tensor size mismatch #53
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#53
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @bitbottrap on GitHub (Oct 14, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/101
Using: KTransformers REST API
Model: DeepSeek Coder V2 236B Q8
I changed the following settings in args.py:
max_new_tokens to 16384
max_response_tokens to 16384
cache_q4 to False
I was attempting to increase the size of the response from the REST API for coding purposes. I changed the cache quantization while I was at it too. Don't think it makes much of a performance difference for me and I have system RAM available.
(I have not tracked down or understand the difference, or need, for both max_new_tokens versus max_response_tokens? From their descriptions it sounds like they do the same thing.)
What I did:
KTransformers successfully completed a significant prompt (the first prompt provided to the server) and produced a lengthy and complete response. When I attempted a follow up (the second prompt provided to the server) prompt I get the following messages in my logs and KTransformers server is unresponsive:
@bitbottrap commented on GitHub (Oct 14, 2024):
I've reverted the changes one by one and the crash happens every time. It's being fed with a long prompt but ... I don't think [?] that should matter. Looks like a b00g to me. Works on other engines.
@qiyuxinlin commented on GitHub (Oct 15, 2024):
It seems that it is caused by the inconsistent length of the attention_mask passed in and the input token. Can you share the code for starting the program? Let me repeat the problem you encountered. After using the REST API to call the DeepSeek model, the program will report an error in the second round of dialogue?
@bitbottrap commented on GitHub (Oct 15, 2024):
The first time it crashed it was on the second prompt. The first prompt was simple and had a relatively short response. The second prompt was long and it crashed immediately when the request was made. No work was done.
I have found that the long prompt will crash KTransformers even if it is the first prompt after starting KTransformers. I am using the latest Continue.dev VSCode extension.
I have made no code changes and followed the build instructions precisely. The command line I'm starting KTransformers with is:
ktransformers.env/bin/ktransformers --port 8090 --cpu_infer 60 --model_path deepseek-ai/DeepSeek-Coder-V2-Instruct --gguf_path /mnt/models/deepseek-coder-v2:236b-instruct-q8_0
This is an 8-bit quantized gguf model that was pulled from the ollama library.
I will try to recreate the prompt again. It appears I accidentally deleted it when I periodically clean up my prompts.
@bitbottrap commented on GitHub (Oct 24, 2024):
I had been using the quantized model available from ollama. I have now downloaded the DeepSeek model and quantized it myself to q8_0 using llama.cpp code current as of today. I have not seen this error (yet) but have opened another issue with a different error. I think there is probably some low hanging fruit with debugging increased output token counts.
@qiyuxinlin commented on GitHub (Oct 28, 2024):
We have located the problem. We did not set the size of the kv cache to an interface. The default configuration is 4096. Conversations exceeding the length of 4096 will report an error. We are integrating the configuration files and will modify this problem in a later version. If you want to solve the problems you encounter in the current version, you can modify
cache_lensin/site-packages/ktransformers/ktransformers/server/backend/args.py@arthurv commented on GitHub (Oct 28, 2024):
I've managed to change the kv cache to 8192, however changing it to 16384 results in CUDA out of memory errors - it tried to allocate on GPU 0 while there was free space on the other GPUs. Unifying the configuration would help as it's a bit confusing whether changing the GPU split option or editing the YAML files would solve the issue.
@bitbottrap commented on GitHub (Oct 28, 2024):
I've also managed to change the cache_lens, max_new_tokens, and max_response_tokens to 8192 and achieved long[er] output without error. (This might be a better discussion for the other bug though?) However, as arthurv reports above my attempt at 16384 yields a CUDA out of memory error. I had to resort to using the multiple-GPU yaml optimization file (DeepSeek-V2-Chat-multi-gpu.yaml) for the 8192 configuration to complete. I have two 24GB GPUs and the memory usage is interesting.
At one of the stages prior to text generation memory usage spikes:
| 0 N/A N/A 24567 C ...sv-ai/ktransformers.env/bin/python3 19140MiB |
| 1 N/A N/A 24567 C ...sv-ai/ktransformers.env/bin/python3 20530MiB |
And when text is being generated memory utilization drops:
| 0 N/A N/A 24567 C ...sv-ai/ktransformers.env/bin/python3 5232MiB |
| 1 N/A N/A 24567 C ...sv-ai/ktransformers.env/bin/python3 8518MiB |
When these settings are changed there's definitely a significant temporary increase in memory requirements. If that could be reduced or eliminated we'd probably be good to go. Though there does appear to be a significant performance impact for the 8192/multi-GPU configuration. (around 50% slower)
@qiyuxinlin commented on GitHub (Oct 30, 2024):
We used DeepSeek-V2-Chat-multi-gpu.yaml to test the 16k situation in local_chat and server, and the phenomenon you mentioned did not occur. Moreover, our program applies for VRAM when starting, and the GPU memory usage should not fluctuate significantly. Can you provide the yaml file and run command you used?
@arthurv commented on GitHub (Oct 30, 2024):
Not OP but here's my experience:
ktransformers commit id
a6a1cc054fChanged: ktransformers/server/backend/args.py
-> changed max_new_tokens to 16384
-> changed cache_lens to 16384
Starting command:
ktransformers --model_path deepseek-ai/DeepSeek-V2.5 --gguf_path DeepSeek-V2.5-Q6 --port 10002 --optimize_config_path ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V2-Chat-multi-gpu-4.yamlI start chatting through the API - I enter a prompt with 39 tokens, it generates 804 tokens as a reply. I enter 804 more tokens as input, and it crashes with torch.outOfMemoryError: CUDA out of memory
Nvidia-smi gives this:
I have 4 GPUs, and you can see that it's mostly using only GPU 0. It crashed trying to allocate 6.23 GB on GPU 0 when there was only 5GB available. Is there a way to redistribute the memory use better?
@arthurv commented on GitHub (Oct 30, 2024):
Adding to the info above - I reverted max_new_tokens and cache_lens to 8192, and launched ktransformers again with the same command.
Submitted an 804 token prompt and it was OK. Memory usage was:
@bitbottrap commented on GitHub (Oct 30, 2024):
The multi-GPU memory usage I displayed was from the following multi-GPU configuration command line:
ktransformers --port 8090 --cpu_infer 60 --model_path deepseek-ai/DeepSeek-Coder-V2-Instruct --gguf_path /mnt/models/DeepSeek-Coder-V2-Instruct --optimize_config_path /home/user/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V2-Chat-multi-gpu.yamlSingle GPU configured to generate 8192 tokens is not possible.
I went through my logs and believe this is from a single GPU configuration with DeepSeek v2 modified for increased output. What other details would you like me to provide?