mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #483] Enable optimize_rule of multi-gpu on A100 leads to RuntimeError #340
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#340
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @ArYuZzz on GitHub (Feb 19, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/483
Precondition
Device: Dual A100(PCIe) + Xeon 3
Engine Version:
ktransformers-0.2.1.post1+cu124torch24avx512-cp311-cp311-linux_x86_6409f5c5eEnvironment: docker
ubuntu 22.04 with privileged. hostubuntu 24.04.Optimize Rule:
ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-multi-gpu-marlin.yaml0564ac8Command:
Result
The output log of ktransformers:
The operator injecting and weight loading seems successful.
However, the log shows that 30~60 layers weight still be loaded into
cuda:0VRAM Usage:
Meanwhile, the Memory-usage of
cuda:1does increases.Error Thrown:
While input prompt, the following
RuntimeErrorwill be thrown,@ArYuZzz commented on GitHub (Feb 19, 2025):
@Azure-Tang Please check the issue.
@Jotakak-yu commented on GitHub (Feb 22, 2025):
Same issue. Any updates?
@proxy2008 commented on GitHub (Feb 24, 2025):
me 2
@tempgidam commented on GitHub (Mar 2, 2025):
I have the same problem on 2x A6000 and Ubuntu 22.04. I tried both ktransformers v0.2.1 and v0.2.2.
@ruralharry commented on GitHub (Jun 2, 2025):
INFO: 192.168.71.50:57796 - "OPTIONS /v1/chat/completions HTTP/1.1" 200 OK
INFO: 192.168.71.50:57796 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2025-06-02 14:14:02,267 DEBUG /home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/transformers.py[209]: get input ids of shape torch.Size([1, 5])
2025-06-02 14:14:02,267 DEBUG /home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[144]: input_ids: torch.Size([1, 5])
2025-06-02 14:14:02,267 DEBUG /home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[169]: same prefix len: 4
2025-06-02 14:14:02,269 DEBUG /home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[178]: input_ids: torch.Size([1, 1])
2025-06-02 14:14:02,269 DEBUG /home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[179]: generate_ids: torch.Size([1, 4])
2025-06-02 14:14:02,269 DEBUG /home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[194]: cache position: 4 to 5
ERROR: Exception in ASGI application
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
| yield
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 263, in call
| async with anyio.create_task_group() as task_group:
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in aexit
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
| return await self.app(scope, receive, send)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in call
| await super().call(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/applications.py", line 112, in call
| await self.middleware_stack(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in call
| raise exc
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in call
| await self.app(scope, receive, _send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/cors.py", line 93, in call
| await self.simple_response(scope, receive, send, request_headers=headers)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/cors.py", line 144, in simple_response
| await self.app(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in call
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 714, in call
| await self.middleware_stack(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 734, in app
| await route.handle(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
| await self.app(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 74, in app
| await response(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 262, in call
| with collapse_excgroups():
| ^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/contextlib.py", line 158, in exit
| self.gen.throw(value)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
| raise exc
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 266, in wrap
| await func()
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 246, in stream_response
| async for chunk in self.body_iterator:
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
| async for event in async_events:
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
| async for event in async_events:
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
| async for event in async_events:
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
| async for event in async_events:
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 266, in inner
| async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens):
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 241, in inference
| async for v in super().inference(local_messages, thread_id, temperature, top_p, max_tokens, max_completion_tokens):
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 466, in inference
| for t in self.prefill(input_ids, self.check_is_new(thread_id), temperature, top_p, max_tokens, max_completion_tokens):
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
| response = gen.send(None)
| ^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 224, in prefill
| logits = chunk_prefill(input_ids[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end])
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 207, in chunk_prefill
| logits = self.model(
| ^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1690, in forward
| outputs = self.model(
| ^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/operators/models.py", line 751, in forward
| layer_outputs = decoder_layer(
| ^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1204, in forward
| hidden_states = self.input_layernorm(hidden_states)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 110, in forward
| return self.weight * hidden_states.to(input_dtype)
| ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:2 and cuda:0!
+------------------------------------
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/applications.py", line 112, in call
await self.middleware_stack(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in call
raise exc
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in call
await self.app(scope, receive, _send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/cors.py", line 93, in call
await self.simple_response(scope, receive, send, request_headers=headers)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/cors.py", line 144, in simple_response
await self.app(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 714, in call
await self.middleware_stack(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 734, in app
await route.handle(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 74, in app
await response(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 262, in call
with collapse_excgroups():
^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/contextlib.py", line 158, in exit
self.gen.throw(value)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 266, in wrap
await func()
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 246, in stream_response
async for chunk in self.body_iterator:
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
async for event in async_events:
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
async for event in async_events:
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
async for event in async_events:
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
async for event in async_events:
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 266, in inner
async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens):
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 241, in inference
async for v in super().inference(local_messages, thread_id, temperature, top_p, max_tokens, max_completion_tokens):
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 466, in inference
for t in self.prefill(input_ids, self.check_is_new(thread_id), temperature, top_p, max_tokens, max_completion_tokens):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 224, in prefill
logits = chunk_prefill(input_ids[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 207, in chunk_prefill
logits = self.model(
^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1690, in forward
outputs = self.model(
^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/operators/models.py", line 751, in forward
layer_outputs = decoder_layer(
^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1204, in forward
hidden_states = self.input_layernorm(hidden_states)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 110, in forward
return self.weight * hidden_states.to(input_dtype)
~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:2 and cuda:0!
INFO: 192.168.71.50:57797 - "OPTIONS /v1/chat/completions HTTP/1.1" 200 OK
INFO: 192.168.71.50:57797 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2025-06-02 14:14:02,792 DEBUG /home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/transformers.py[209]: get input ids of shape torch.Size([1, 5])
2025-06-02 14:14:02,792 DEBUG /home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[144]: input_ids: torch.Size([1, 5])
2025-06-02 14:14:02,793 DEBUG /home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[169]: same prefix len: 4
2025-06-02 14:14:02,794 DEBUG /home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[178]: input_ids: torch.Size([1, 1])
2025-06-02 14:14:02,794 DEBUG /home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[179]: generate_ids: torch.Size([1, 4])
2025-06-02 14:14:02,794 DEBUG /home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[194]: cache position: 4 to 5
ERROR: Exception in ASGI application
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
| yield
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 263, in call
| async with anyio.create_task_group() as task_group:
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in aexit
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
| return await self.app(scope, receive, send)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in call
| await super().call(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/applications.py", line 112, in call
| await self.middleware_stack(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in call
| raise exc
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in call
| await self.app(scope, receive, _send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/cors.py", line 93, in call
| await self.simple_response(scope, receive, send, request_headers=headers)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/cors.py", line 144, in simple_response
| await self.app(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in call
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 714, in call
| await self.middleware_stack(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 734, in app
| await route.handle(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
| await self.app(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 74, in app
| await response(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 262, in call
| with collapse_excgroups():
| ^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/contextlib.py", line 158, in exit
| self.gen.throw(value)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
| raise exc
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 266, in wrap
| await func()
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 246, in stream_response
| async for chunk in self.body_iterator:
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
| async for event in async_events:
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
| async for event in async_events:
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
| async for event in async_events:
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
| async for event in async_events:
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 266, in inner
| async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens):
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 241, in inference
| async for v in super().inference(local_messages, thread_id, temperature, top_p, max_tokens, max_completion_tokens):
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 466, in inference
| for t in self.prefill(input_ids, self.check_is_new(thread_id), temperature, top_p, max_tokens, max_completion_tokens):
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
| response = gen.send(None)
| ^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 224, in prefill
| logits = chunk_prefill(input_ids[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end])
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 207, in chunk_prefill
| logits = self.model(
| ^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1690, in forward
| outputs = self.model(
| ^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/operators/models.py", line 751, in forward
| layer_outputs = decoder_layer(
| ^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1204, in forward
| hidden_states = self.input_layernorm(hidden_states)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 110, in forward
| return self.weight * hidden_states.to(input_dtype)
| ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:2 and cuda:0!
+------------------------------------
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/applications.py", line 112, in call
await self.middleware_stack(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in call
raise exc
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in call
await self.app(scope, receive, _send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/cors.py", line 93, in call
await self.simple_response(scope, receive, send, request_headers=headers)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/cors.py", line 144, in simple_response
await self.app(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 714, in call
await self.middleware_stack(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 734, in app
await route.handle(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 74, in app
await response(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 262, in call
with collapse_excgroups():
^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/contextlib.py", line 158, in exit
self.gen.throw(value)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 266, in wrap
await func()
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 246, in stream_response
async for chunk in self.body_iterator:
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
async for event in async_events:
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
async for event in async_events:
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
async for event in async_events:
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
async for event in async_events:
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 266, in inner
async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens):
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 241, in inference
async for v in super().inference(local_messages, thread_id, temperature, top_p, max_tokens, max_completion_tokens):
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 466, in inference
for t in self.prefill(input_ids, self.check_is_new(thread_id), temperature, top_p, max_tokens, max_completion_tokens):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 224, in prefill
logits = chunk_prefill(input_ids[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 207, in chunk_prefill
logits = self.model(
^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1690, in forward
outputs = self.model(
^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/operators/models.py", line 751, in forward
layer_outputs = decoder_layer(
^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1204, in forward
hidden_states = self.input_layernorm(hidden_states)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 110, in forward
return self.weight * hidden_states.to(input_dtype)
~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:2 and cuda:0!
INFO: 192.168.71.50:57805 - "OPTIONS /v1/chat/completions HTTP/1.1" 200 OK
INFO: 192.168.71.50:57805 - "POST /v1/chat/completions HTTP/1.1" 200 OK
2025-06-02 14:14:03,839 DEBUG /home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/transformers.py[209]: get input ids of shape torch.Size([1, 5])
2025-06-02 14:14:03,839 DEBUG /home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[144]: input_ids: torch.Size([1, 5])
2025-06-02 14:14:03,840 DEBUG /home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[169]: same prefix len: 4
2025-06-02 14:14:03,841 DEBUG /home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[178]: input_ids: torch.Size([1, 1])
2025-06-02 14:14:03,841 DEBUG /home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[179]: generate_ids: torch.Size([1, 4])
2025-06-02 14:14:03,841 DEBUG /home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py[194]: cache position: 4 to 5
ERROR: Exception in ASGI application
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_utils.py", line 76, in collapse_excgroups
| yield
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 263, in call
| async with anyio.create_task_group() as task_group:
| ^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/anyio/_backends/_asyncio.py", line 772, in aexit
| raise BaseExceptionGroup(
| ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
+-+---------------- 1 ----------------
| Traceback (most recent call last):
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
| result = await app( # type: ignore[func-returns-value]
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
| return await self.app(scope, receive, send)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in call
| await super().call(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/applications.py", line 112, in call
| await self.middleware_stack(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in call
| raise exc
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in call
| await self.app(scope, receive, _send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/cors.py", line 93, in call
| await self.simple_response(scope, receive, send, request_headers=headers)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/cors.py", line 144, in simple_response
| await self.app(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in call
| await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 714, in call
| await self.middleware_stack(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 734, in app
| await route.handle(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
| await self.app(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
| await wrap_app_handling_exceptions(app, request)(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
| raise exc
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
| await app(scope, receive, sender)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 74, in app
| await response(scope, receive, send)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 262, in call
| with collapse_excgroups():
| ^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/contextlib.py", line 158, in exit
| self.gen.throw(value)
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
| raise exc
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 266, in wrap
| await func()
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 246, in stream_response
| async for chunk in self.body_iterator:
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
| async for event in async_events:
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
| async for event in async_events:
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
| async for event in async_events:
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
| async for event in async_events:
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 266, in inner
| async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens):
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 241, in inference
| async for v in super().inference(local_messages, thread_id, temperature, top_p, max_tokens, max_completion_tokens):
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 466, in inference
| for t in self.prefill(input_ids, self.check_is_new(thread_id), temperature, top_p, max_tokens, max_completion_tokens):
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
| response = gen.send(None)
| ^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 224, in prefill
| logits = chunk_prefill(input_ids[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end])
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 207, in chunk_prefill
| logits = self.model(
| ^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1690, in forward
| outputs = self.model(
| ^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/operators/models.py", line 751, in forward
| layer_outputs = decoder_layer(
| ^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1204, in forward
| hidden_states = self.input_layernorm(hidden_states)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
| return self._call_impl(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
| return forward_call(*args, **kwargs)
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 110, in forward
| return self.weight * hidden_states.to(input_dtype)
| ~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:2 and cuda:0!
+------------------------------------
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi
result = await app( # type: ignore[func-returns-value]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
return await self.app(scope, receive, send)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/applications.py", line 112, in call
await self.middleware_stack(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/errors.py", line 187, in call
raise exc
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/errors.py", line 165, in call
await self.app(scope, receive, _send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/cors.py", line 93, in call
await self.simple_response(scope, receive, send, request_headers=headers)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/cors.py", line 144, in simple_response
await self.app(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/middleware/exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 714, in call
await self.middleware_stack(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 734, in app
await route.handle(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/routing.py", line 74, in app
await response(scope, receive, send)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 262, in call
with collapse_excgroups():
^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/contextlib.py", line 158, in exit
self.gen.throw(value)
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/_utils.py", line 82, in collapse_excgroups
raise exc
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 266, in wrap
await func()
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/starlette/responses.py", line 246, in stream_response
async for chunk in self.body_iterator:
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
async for event in async_events:
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
async for event in async_events:
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
async for event in async_events:
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk
async for event in async_events:
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 266, in inner
async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens):
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 241, in inference
async for v in super().inference(local_messages, thread_id, temperature, top_p, max_tokens, max_completion_tokens):
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 466, in inference
for t in self.prefill(input_ids, self.check_is_new(thread_id), temperature, top_p, max_tokens, max_completion_tokens):
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 224, in prefill
logits = chunk_prefill(input_ids[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 207, in chunk_prefill
logits = self.model(
^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1690, in forward
outputs = self.model(
^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/operators/models.py", line 751, in forward
layer_outputs = decoder_layer(
^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1204, in forward
hidden_states = self.input_layernorm(hidden_states)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/ruralharry/miniconda3/envs/kt/lib/python3.12/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 110, in forward
return self.weight * hidden_states.to(input_dtype)
~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:2 and cuda:0!
我也遇到了这个问题。
环境信息
代码分支：[
github.com/kvcache-ai/ktransformers@7071970339]，20250601最新运行硬件：AMD EPYC 9755 *2 ，1TB内存，5080 *4
运行系统：Ubuntu 22.04 conda python3.12 cmake3.31.6
其他信息：
📌 NVIDIA CUDA 版本:
Cuda compilation tools, release 12.8, V12.8.93
📌 NVIDIA 驱动版本:
/usr/bin/nvidia-smi
| NVIDIA-SMI 570.133.07 Driver Version: 570.133.07 CUDA Version: 12.8 |
📌 PyTorch 版本:
2.7.0+cu128
📌 NumPy 版本:
2.1.2
📌 cpufeature 版本:
0.2.1
📌 flash-attn 版本:
2.7.4.post1
（自己编译）
📌 triton 版本:
Version: 3.3.0+gitcda42295
（自己编译）
我有4个GPU，在配置文件DeepSeek-V3-Chat-multi-gpu.yaml 中指定的 cuda:2和cuda:3
运行的是Deepseek-r1-0528-q8
@ruralharry commented on GitHub (Jun 2, 2025):
@Atream 您好，还有人在关注这个问题吗？v0.2.1 就存在的问题，现在v0.3.1 了，还存在呢？