[GH-ISSUE #45] Cannot run DeepSeek V2 Chat in server mode on 2 GPUs #24

New issue

Closed

opened 2026-10-29 14:32:44 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 14:32:44 +00:00

Owner

Originally created by @ELigoP on GitHub (Aug 20, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/45

I have 2x RTX 3090. I've built ktransformers from source (needed to install some missing packages, but it worked in the end).

When I run local_chat with

python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-V2-Chat-0628 --gguf_path ~/.cache/lm-studio/models/bartowski/DeepSeek-V2-Chat-0628-GGUF --optimize_rule_path ./build/lib.linux-x86_64-cpython-312/ktransformers/optimize/optimize_rules/DeepSeek-V2-Chat-multi-gpu.yaml

in-console chat works, I get 9 tps, more than llama.cpp out-of-the-box 7.5 tps.

I didn't find a way to pass multi-gpu optimization rule to ktransformers server, as --optimize_config_path OPTIMIZE_CONFIG_PATH option seems to be not about that. So I just replaced default file with multi-gpu version. The I run

ktransformers --model_path deepseek-ai/DeepSeek-V2-Chat-0628 --gguf_path ~/.cache/lm-studio/models/bartowski/DeepSeek-V2-Chat-0628-GGUF --port 1234

I get this error (in the end it says Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!).

Should I also change ktransformers/configs/config.yaml ? How? Is the problem that only cuda:0 is mentioned in config? But right when the server starts it seems to read optimize rule right - it split 0-29, 30-59 layers bettween GPUs:

...
loading blk.29.attn_norm.weight to cuda:0
loading blk.29.ffn_norm.weight to cuda:0
loading blk.30.attn_q_a.weight to cuda:1
loading blk.30.attn_q_a_norm.weight to cuda:1
...

The error:

INFO:     192.168.27.159:46148 - "OPTIONS /v1/chat/completions HTTP/1.1" 200 OK
2024-08-20 19:03:00,415 WARNING /home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[161]: change system to user
2024-08-20 19:03:00,415 WARNING /home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[167]: merge two adjacent user messages
2024-08-20 19:03:00,461 DEBUG /home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[185]: get input ids of shape torch.Size([1, 18])
2024-08-20 19:03:00,461 DEBUG /home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[240]: input_ids: torch.Size([1, 18])
2024-08-20 19:03:00,462 DEBUG /home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[262]: cache position: 0 to 18
INFO:     192.168.27.159:46148 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__
    raise exc
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__
    await self.app(scope, receive, _send)
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/cors.py", line 93, in __call__
    await self.simple_response(scope, receive, send, request_headers=headers)
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/cors.py", line 148, in simple_response
    await self.app(scope, receive, send)
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 776, in app
    await route.handle(scope, receive, send)
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle
    await self.app(scope, receive, send)
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 77, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app
    raise exc
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    await app(scope, receive, sender)
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 72, in app
    response = await func(request)
               ^^^^^^^^^^^^^^^^^^^
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 32, in chat_completion
    async for token in interface.inference(input_message,id):
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 323, in inference
    for t in self.prefill(input_ids,self.check_is_new(thread_id)):
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
    response = gen.send(None)
               ^^^^^^^^^^^^^^
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 272, in prefill
    logits = self.model(
             ^^^^^^^^^^^
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1731, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/models.py", line 613, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1238, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
                                                          ^^^^^^^^^^^^^^^
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 163, in forward
    return self.forward_chunck(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 92, in forward_chunck
    k_pe, compressed_kv = past_key_value.update(k_pe, compressed_kv, self.layer_idx, cache_kwargs)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/custom_cache.py", line 105, in update
    k_out[:, :, cache_position] = key_states
    ~~~~~^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!

Originally created by @ELigoP on GitHub (Aug 20, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/45 I have 2x RTX 3090. I've built `ktransformers` from source (needed to install some missing packages, but it worked in the end). When I run `local_chat` with ``` python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-V2-Chat-0628 --gguf_path ~/.cache/lm-studio/models/bartowski/DeepSeek-V2-Chat-0628-GGUF --optimize_rule_path ./build/lib.linux-x86_64-cpython-312/ktransformers/optimize/optimize_rules/DeepSeek-V2-Chat-multi-gpu.yaml ``` in-console chat works, I get 9 tps, more than llama.cpp out-of-the-box 7.5 tps. I didn't find a way to pass multi-gpu optimization rule to `ktransformers` server, as `--optimize_config_path OPTIMIZE_CONFIG_PATH` option seems to be not about that. So I just replaced default file with `multi-gpu` version. The I run ``` ktransformers --model_path deepseek-ai/DeepSeek-V2-Chat-0628 --gguf_path ~/.cache/lm-studio/models/bartowski/DeepSeek-V2-Chat-0628-GGUF --port 1234 ``` I get this error (in the end it says `Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1!`). **Should I also change `ktransformers/configs/config.yaml` ?** How? Is the problem that only `cuda:0` is mentioned in config? But right when the server starts it seems to read optimize rule right - it split 0-29, 30-59 layers bettween GPUs: ``` ... loading blk.29.attn_norm.weight to cuda:0 loading blk.29.ffn_norm.weight to cuda:0 loading blk.30.attn_q_a.weight to cuda:1 loading blk.30.attn_q_a_norm.weight to cuda:1 ... ``` The error: ``` INFO: 192.168.27.159:46148 - "OPTIONS /v1/chat/completions HTTP/1.1" 200 OK 2024-08-20 19:03:00,415 WARNING /home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[161]: change system to user 2024-08-20 19:03:00,415 WARNING /home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[167]: merge two adjacent user messages 2024-08-20 19:03:00,461 DEBUG /home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[185]: get input ids of shape torch.Size([1, 18]) 2024-08-20 19:03:00,461 DEBUG /home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[240]: input_ids: torch.Size([1, 18]) 2024-08-20 19:03:00,462 DEBUG /home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[262]: cache position: 0 to 18 INFO: 192.168.27.159:46148 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 70, in __call__ return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/applications.py", line 123, in __call__ await self.middleware_stack(scope, receive, send) File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 186, in __call__ raise exc File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 164, in __call__ await self.app(scope, receive, _send) File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/cors.py", line 93, in __call__ await self.simple_response(scope, receive, send, request_headers=headers) File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/cors.py", line 148, in simple_response await self.app(scope, receive, send) File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 65, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 756, in __call__ await self.middleware_stack(scope, receive, send) File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 776, in app await route.handle(scope, receive, send) File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 297, in handle await self.app(scope, receive, send) File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 77, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 64, in wrapped_app raise exc File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app await app(scope, receive, sender) File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 72, in app response = await func(request) ^^^^^^^^^^^^^^^^^^^ File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/routing.py", line 278, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/routing.py", line 191, in run_endpoint_function return await dependant.call(**values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 32, in chat_completion async for token in interface.inference(input_message,id): File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 323, in inference for t in self.prefill(input_ids,self.check_is_new(thread_id)): File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context response = gen.send(None) ^^^^^^^^^^^^^^ File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 272, in prefill logits = self.model( ^^^^^^^^^^^ File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1731, in forward outputs = self.model( ^^^^^^^^^^^ File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/models.py", line 613, in forward layer_outputs = decoder_layer( ^^^^^^^^^^^^^^ File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1238, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 163, in forward return self.forward_chunck( ^^^^^^^^^^^^^^^^^^^^ File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 92, in forward_chunck k_pe, compressed_kv = past_key_value.update(k_pe, compressed_kv, self.layer_idx, cache_kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/ai/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/custom_cache.py", line 105, in update k_out[:, :, cache_position] = key_states ~~~~~^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cuda:1! ```

bsakaguchi closed this issue

2026-10-29 14:32:45 +00:00

bsakaguchi commented

2026-10-29 14:32:45 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Aug 21, 2024):

Sorry for the inconvenience orz.

You can appoint optimise yaml rule by --optimize_config_path now. I have fixed by #48 .
Expected all tensors to be on the same device
Ok this is a bug that server creating static cache on one device. Also fixed by #48 .

@Azure-Tang commented on GitHub (Aug 21, 2024): Sorry for the inconvenience orz. 1. You can appoint optimise yaml rule by `--optimize_config_path` now. I have fixed by #48 . 2. `Expected all tensors to be on the same device` Ok this is a bug that server creating static cache on one device. Also fixed by #48 .

bsakaguchi referenced this issue

2026-10-29 14:42:22 +00:00

[GH-ISSUE #903] Rocm run deepseek r1 671b q4 failed #646

bsakaguchi referenced this issue

2026-10-29 14:45:29 +00:00

[GH-ISSUE #1269] [Question] benchmark with evalscope failed #859

bsakaguchi referenced this issue

2026-10-29 14:45:38 +00:00

[GH-ISSUE #1286] [Bug] inference_context.v_cache[0][i] IndexError: list index out of range #871

bsakaguchi referenced this issue

2026-10-29 14:45:39 +00:00

[GH-ISSUE #1288] [Bug] #876