mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #729] [BUG]Ktransformer Docker镜像0.2.1/0.2.2部署Deepseek-r1671B Q4_K_M，OpenAI SDK访问出现 shape mismatch #532
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#532
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @XMoyas on GitHub (Feb 28, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/729
Describe the bug
Ubuntu22.04 Docker0.2.1/0.2.2部署Deepseek-r1671B Q4_K_M，两个镜像部署的模型localchat均能正常对话，在切换到Ktransformer server版本时，服务能够正常起来，但是使用OpenAI SDK访问时两个镜像均出现类似报错：RuntimeError: shape mismatch: value tensor of shape [5, 1, 1, 512] cannot be broadcast to indexing result of shape [1, 1, 512]
运行的命令分别如下：
服务版：ktransformers --gguf_path /root/models/DeepSeek-R1-GGUF/Deepseek-R1-Q4_K_M --model_path DeepSeek-V3 --model_name deepseek-r1-671b --optimize_config_path /workspace/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml --max_new_tokens 4096 --cpu_infer 62 --port 10002 --batch_size 5 --use_cuda_graph --cache_lens 8092 --force_think
本地聊天：python -m ktransformers.local_chat --gguf_path /root/models/DeepSeek-R1-GGUF/Deepseek-R1-Q4_K_M --model_path DeepSeek-V3 --optimize_rule_path /workspace/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml --max_new_tokens 4096 --cpu_infer 62 --port 10002 --force_think --use_cuda_graph
OpenAI SDK访问信息：
from openai import OpenAI
API_KEY='EMPTY'
BASE_URL='http://宿主机的IP地址:10002/v1'
MODEL_NAME='deepseek-r1-671b'
client = OpenAI(
api_key=API_KEY,
base_url=BASE_URL
)
response = client.chat.completions.create(
model=MODEL_NAME,
messages=[{"role": "user", "content": 'Hi, 初次见面做个自我介绍吧'}],
temperature=0.1,
max_tokens=1000
)
print(response)
报错信息如下：
2025-02-28 02:44:20,756 DEBUG /opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/transformers.py[197]: get input ids of shape torch.Size([1, 12])
2025-02-28 02:44:20,758 DEBUG /opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[132]: input_ids: torch.Size([1, 15])
2025-02-28 02:44:20,759 DEBUG /opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[158]: same prefix len: 14
2025-02-28 02:44:20,761 DEBUG /opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[167]: input_ids: torch.Size([1, 1])
2025-02-28 02:44:20,762 DEBUG /opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[169]: generate_ids: torch.Size([5, 14])
2025-02-28 02:44:20,762 DEBUG /opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[180]: cache position: 14 to 15
好的INFO: 172.17.0.3:53440 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/opt/conda/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in call
return await self.app(scope, receive, send)
File "/opt/conda/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/opt/conda/lib/python3.10/site-packages/starlette/applications.py", line 123, in call
await self.middleware_stack(scope, receive, send)
File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/errors.py", line 186, in call
raise exc
File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/errors.py", line 164, in call
await self.app(scope, receive, _send)
File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/cors.py", line 85, in call
await self.app(scope, receive, send)
File "/opt/conda/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 65, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/opt/conda/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/opt/conda/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 754, in call
await self.middleware_stack(scope, receive, send)
File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 774, in app
await route.handle(scope, receive, send)
File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 295, in handle
await self.app(scope, receive, send)
File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 77, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/opt/conda/lib/python3.10/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
raise exc
File "/opt/conda/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
await app(scope, receive, sender)
File "/opt/conda/lib/python3.10/site-packages/starlette/routing.py", line 74, in app
response = await f(request)
File "/opt/conda/lib/python3.10/site-packages/fastapi/routing.py", line 297, in app
raw_response = await run_endpoint_function(
File "/opt/conda/lib/python3.10/site-packages/fastapi/routing.py", line 210, in run_endpoint_function
return await dependant.call(**values)
File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 38, in chat_completion
async for token in interface.inference(input_message,id):
File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 212, in inference
async for v in super().inference(local_messages, thread_id):
File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 397, in inference
for t in self.generate():
File "/opt/conda/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 35, in generator_context
response = gen.send(None)
File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 341, in generate
next_token = self.decode_one_tokens()
File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 113, in decode_one_tokens
logits = self.model(
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1688, in forward
outputs = self.model(
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/models.py", line 722, in forward
layer_outputs = decoder_layer(
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1205, in forward
hidden_states, self_attn_weights, present_key_value = self.self_attn(
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1532, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1541, in _call_impl
return forward_call(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/attention.py", line 598, in forward
return self.forward_linux_triton(
File "/opt/conda/lib/python3.10/site-packages/ktransformers/operators/attention.py", line 234, in forward_linux_triton
compressed_kv_with_k_pe, page_table = past_key_value.update(compressed_kv, k_pe, self.layer_idx, cache_kwargs)
File "/opt/conda/lib/python3.10/site-packages/ktransformers/models/custom_cache.py", line 141, in update
k_out[page_idx, page_offset, :, :self.kv_lora_rank] = key_states
RuntimeError: shape mismatch: value tensor of shape [5, 1, 1, 512] cannot be broadcast to indexing result of shape [1, 1, 512]
Environments(please complete the following information):
ktransformers 0.2.1.post1+cu121torch23fancy
torch 2.3.1
torchaudio 2.3.1
torchelastic 0.2.2
torchvision 0.18.1
flash-attn 2.6.3
flashinfer-python 0.2.2
@4thGardenOfQMH commented on GitHub (Feb 28, 2025):
我也遇到了类似的问题，我启动服务的命令是 python3 ktransformers/server/main.py --gguf_path*******
@verigle commented on GitHub (Mar 7, 2025):
拉的哪个镜像呢？
目前是不是还没有官方镜像
@magic-hya commented on GitHub (Mar 10, 2025):
遇到同样的问题，拉的approachingai/ktransformers:v0.2.3-NATIVE这个镜像
@XMoyas commented on GitHub (Mar 10, 2025):
@4thGardenOfQMH @verigle @magic-hya 我是因为server启动时，参数设置batch size为5（2也不行）导致的，换成1就可以正常调用，不是很清楚原因。
ktransformers --gguf_path /root/models/DeepSeek-R1-GGUF/Deepseek-R1-Q4_K_M --model_path DeepSeek-V3 --model_name deepseek-r1-671b --optimize_config_path /workspace/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml --max_new_tokens 4096 --cpu_infer 62 --port 10002 **--batch_size 5** --use_cuda_graph --cache_lens 8092 --force_think@magic-hya commented on GitHub (Mar 11, 2025):
感谢答复，确实改成1就好了。我用这个镜像跑的很慢，不知道是不是r1的原因，还是镜像的原因