kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #96] Error Compile with TORCH_USE_CUDA_DSA to enable device-side assertions. #52

New issue
Open
opened 2026-10-29 14:33:05 +00:00 by bsakaguchi · 12 comments
bsakaguchi commented 2026-10-29 14:33:05 +00:00
Owner
Copy link

Originally created by @drrros on GitHub (Oct 2, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/96

I'm trying to run a DeepSeek-V2.5 model.
Command used: python -m ktransformers.local_chat --model_path ./DeepSeek-V2.5/ --gguf_path ../ 

Chat: hi
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/drros/deepseek2.5/ktransformers/ktransformers/local_chat.py", line 159, in <module>
    fire.Fire(local_chat)
  File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drros/deepseek2.5/ktransformers/ktransformers/local_chat.py", line 153, in local_chat
    generated = prefill_and_generate(
                ^^^^^^^^^^^^^^^^^^^^^
  File "/home/drros/deepseek2.5/ktransformers/ktransformers/util/utils.py", line 150, in prefill_and_generate
    logits = model(
             ^^^^^^
  File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drros/deepseek2.5/ktransformers/ktransformers/models/modeling_deepseek.py", line 1731, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drros/deepseek2.5/ktransformers/ktransformers/operators/models.py", line 719, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drros/deepseek2.5/ktransformers/ktransformers/models/modeling_deepseek.py", line 1238, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
                                                          ^^^^^^^^^^^^^^^
  File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drros/deepseek2.5/ktransformers/ktransformers/operators/attention.py", line 170, in forward
    return self.forward_chunck(
           ^^^^^^^^^^^^^^^^^^^^
  File "/home/drros/deepseek2.5/ktransformers/ktransformers/operators/attention.py", line 71, in forward_chunck
    q = self.q_b_proj(self.q_a_layernorm(self.q_a_proj(hidden_states)))
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/drros/deepseek2.5/ktransformers/ktransformers/models/modeling_deepseek.py", line 113, in forward
    hidden_states = hidden_states.to(torch.float32)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: invalid device function
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

I've tried with both ktransformers.local_chat and web interface mode, with and without --optimize_config_path option. Model loads but when first iteraction occurs it fails with this traceback. Other backends (koboldcpp and llama.cpp) runs fine.
Server specs: 200Gb ram, dual P40.

Originally created by @drrros on GitHub (Oct 2, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/96 I'm trying to run a DeepSeek-V2.5 model. Command used: ```python -m ktransformers.local_chat --model_path ./DeepSeek-V2.5/ --gguf_path ../ ``` ``` Chat: hi Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/home/drros/deepseek2.5/ktransformers/ktransformers/local_chat.py", line 159, in <module> fire.Fire(local_chat) File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/drros/deepseek2.5/ktransformers/ktransformers/local_chat.py", line 153, in local_chat generated = prefill_and_generate( ^^^^^^^^^^^^^^^^^^^^^ File "/home/drros/deepseek2.5/ktransformers/ktransformers/util/utils.py", line 150, in prefill_and_generate logits = model( ^^^^^^ File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/drros/deepseek2.5/ktransformers/ktransformers/models/modeling_deepseek.py", line 1731, in forward outputs = self.model( ^^^^^^^^^^^ File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/drros/deepseek2.5/ktransformers/ktransformers/operators/models.py", line 719, in forward layer_outputs = decoder_layer( ^^^^^^^^^^^^^^ File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/drros/deepseek2.5/ktransformers/ktransformers/models/modeling_deepseek.py", line 1238, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/drros/deepseek2.5/ktransformers/ktransformers/operators/attention.py", line 170, in forward return self.forward_chunck( ^^^^^^^^^^^^^^^^^^^^ File "/home/drros/deepseek2.5/ktransformers/ktransformers/operators/attention.py", line 71, in forward_chunck q = self.q_b_proj(self.q_a_layernorm(self.q_a_proj(hidden_states))) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/drros/deepseek2.5/ktransformers/ktransformers/models/modeling_deepseek.py", line 113, in forward hidden_states = hidden_states.to(torch.float32) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: CUDA error: invalid device function CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. ``` I've tried with both ktransformers.local_chat and web interface mode, with and without --optimize_config_path option. Model loads but when first iteraction occurs it fails with this traceback. Other backends (koboldcpp and llama.cpp) runs fine. Server specs: 200Gb ram, dual P40.
bsakaguchi commented 2026-10-29 14:33:06 +00:00
Author
Owner
Copy link

@drrros commented on GitHub (Oct 2, 2024):

This is full logs in API mode:

COMPLETION INPUT:----
<|begin_of_text|><|start_header_id|>system<|end_header_id|>

You are a world-class AI system, capable of complex reasoning and reflection. Reason through the query inside <thinking> tags, and then provide your final response inside <output> tags. If you detect that you made a mistake in your reasoning at any point, correct yourself inside <reflection> tags.<|eot_id|><|start_header_id|>user<|end_header_id|>

{user_prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

{assistant_response}<|eot_id|><|start_header_id|>user<|end_header_id|>

{next_user_prompt}<|eot_id|>
***
DrRos: text
DrRos: test
Assistant:
----
INFO:     192.168.0.77:58360 - "POST /v1/completions HTTP/1.1" 200 OK
2024-10-02 19:29:03,525 DEBUG /home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[240]: input_ids: torch.Size([1, 202])
2024-10-02 19:29:03,529 DEBUG /home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[262]: cache position: 0 to 202
ERROR:    Exception in ASGI application
Traceback (most recent call last):
  File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 257, in __call__
    await wrap(partial(self.listen_for_disconnect, receive))
  File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 253, in wrap
    await func()
  File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 230, in listen_for_disconnect
    message = await receive()
              ^^^^^^^^^^^^^^^
  File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 534, in receive
    await self.message_event.wait()
  File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/asyncio/locks.py", line 213, in wait
    await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f0f919ca950

During handling of the above exception, another exception occurred:

  + Exception Group Traceback (most recent call last):
  |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi
  |     result = await app(  # type: ignore[func-returns-value]
  |              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__
  |     return await self.app(scope, receive, send)
  |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__
  |     await super().__call__(scope, receive, send)
  |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__
  |     raise exc
  |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__
  |     await self.app(scope, receive, _send)
  |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__
  |     await self.app(scope, receive, send)
  |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__
  |     await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
  |     raise exc
  |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__
  |     await self.middleware_stack(scope, receive, send)
  |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 735, in app
  |     await route.handle(scope, receive, send)
  |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
  |     await self.app(scope, receive, send)
  |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
  |     await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app
  |     raise exc
  |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app
  |     await app(scope, receive, sender)
  |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 74, in app
  |     await response(scope, receive, send)
  |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 250, in __call__
  |     async with anyio.create_task_group() as task_group:
  |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 736, in __aexit__
  |     raise BaseExceptionGroup(
  | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
  +-+---------------- 1 ----------------
    | Traceback (most recent call last):
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 253, in wrap
    |     await func()
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 242, in stream_response
    |     async for chunk in self.body_iterator:
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link
    |     async for event in async_events:
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply
    |     async for event in async_events:
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done
    |     async for event in async_events:
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/api/openai/legacy/completions.py", line 23, in inner
    |     async for token in interface.inference(create.prompt,id):
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 323, in inference
    |     for t in self.prefill(input_ids,self.check_is_new(thread_id)):
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
    |     response = gen.send(None)
    |                ^^^^^^^^^^^^^^
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 272, in prefill
    |     logits = self.model(
    |              ^^^^^^^^^^^
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    |     return self._call_impl(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    |     return forward_call(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1731, in forward
    |     outputs = self.model(
    |               ^^^^^^^^^^^
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    |     return self._call_impl(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    |     return forward_call(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/models.py", line 719, in forward
    |     layer_outputs = decoder_layer(
    |                     ^^^^^^^^^^^^^^
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    |     return self._call_impl(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    |     return forward_call(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1238, in forward
    |     hidden_states, self_attn_weights, present_key_value = self.self_attn(
    |                                                           ^^^^^^^^^^^^^^^
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    |     return self._call_impl(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    |     return forward_call(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 170, in forward
    |     return self.forward_chunck(
    |            ^^^^^^^^^^^^^^^^^^^^
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 71, in forward_chunck
    |     q = self.q_b_proj(self.q_a_layernorm(self.q_a_proj(hidden_states)))
    |                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
    |     return self._call_impl(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
    |     return forward_call(*args, **kwargs)
    |            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 113, in forward
    |     hidden_states = hidden_states.to(torch.float32)
    |                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    | RuntimeError: CUDA error: invalid device function
    | CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
    | For debugging consider passing CUDA_LAUNCH_BLOCKING=1
    | Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
    | 
    +------------------------------------

versions:

(base) drros@tesla-ubuntu:~$ nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2023 NVIDIA Corporation
Built on Tue_Aug_15_22:02:13_PDT_2023
Cuda compilation tools, release 12.2, V12.2.140
Build cuda_12.2.r12.2/compiler.33191640_0
(base) drros@tesla-ubuntu:~$ uname -a
Linux tesla-ubuntu 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
<!-- gh-comment-id:2389106400 --> @drrros commented on GitHub (Oct 2, 2024): This is full logs in API mode: ``` COMPLETION INPUT:---- <|begin_of_text|><|start_header_id|>system<|end_header_id|> You are a world-class AI system, capable of complex reasoning and reflection. Reason through the query inside <thinking> tags, and then provide your final response inside <output> tags. If you detect that you made a mistake in your reasoning at any point, correct yourself inside <reflection> tags.<|eot_id|><|start_header_id|>user<|end_header_id|> {user_prompt}<|eot_id|><|start_header_id|>assistant<|end_header_id|> {assistant_response}<|eot_id|><|start_header_id|>user<|end_header_id|> {next_user_prompt}<|eot_id|> *** DrRos: text DrRos: test Assistant: ---- INFO: 192.168.0.77:58360 - "POST /v1/completions HTTP/1.1" 200 OK 2024-10-02 19:29:03,525 DEBUG /home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[240]: input_ids: torch.Size([1, 202]) 2024-10-02 19:29:03,529 DEBUG /home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py[262]: cache position: 0 to 202 ERROR: Exception in ASGI application Traceback (most recent call last): File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 257, in __call__ await wrap(partial(self.listen_for_disconnect, receive)) File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 253, in wrap await func() File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 230, in listen_for_disconnect message = await receive() ^^^^^^^^^^^^^^^ File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 534, in receive await self.message_event.wait() File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/asyncio/locks.py", line 213, in wait await fut asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f0f919ca950 During handling of the above exception, another exception occurred: + Exception Group Traceback (most recent call last): | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 406, in run_asgi | result = await app( # type: ignore[func-returns-value] | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__ | return await self.app(scope, receive, send) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__ | await super().__call__(scope, receive, send) | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/applications.py", line 113, in __call__ | await self.middleware_stack(scope, receive, send) | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__ | raise exc | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__ | await self.app(scope, receive, _send) | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__ | await self.app(scope, receive, send) | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ | await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app | raise exc | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app | await app(scope, receive, sender) | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 715, in __call__ | await self.middleware_stack(scope, receive, send) | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 735, in app | await route.handle(scope, receive, send) | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle | await self.app(scope, receive, send) | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 76, in app | await wrap_app_handling_exceptions(app, request)(scope, receive, send) | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 62, in wrapped_app | raise exc | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/_exception_handler.py", line 51, in wrapped_app | await app(scope, receive, sender) | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/routing.py", line 74, in app | await response(scope, receive, send) | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 250, in __call__ | async with anyio.create_task_group() as task_group: | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/anyio/_backends/_asyncio.py", line 736, in __aexit__ | raise BaseExceptionGroup( | ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception) +-+---------------- 1 ---------------- | Traceback (most recent call last): | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 253, in wrap | await func() | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/starlette/responses.py", line 242, in stream_response | async for chunk in self.body_iterator: | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link | async for event in async_events: | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply | async for event in async_events: | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done | async for event in async_events: | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/api/openai/legacy/completions.py", line 23, in inner | async for token in interface.inference(create.prompt,id): | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 323, in inference | for t in self.prefill(input_ids,self.check_is_new(thread_id)): | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context | response = gen.send(None) | ^^^^^^^^^^^^^^ | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 272, in prefill | logits = self.model( | ^^^^^^^^^^^ | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl | return self._call_impl(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl | return forward_call(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1731, in forward | outputs = self.model( | ^^^^^^^^^^^ | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl | return self._call_impl(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl | return forward_call(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/models.py", line 719, in forward | layer_outputs = decoder_layer( | ^^^^^^^^^^^^^^ | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl | return self._call_impl(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl | return forward_call(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1238, in forward | hidden_states, self_attn_weights, present_key_value = self.self_attn( | ^^^^^^^^^^^^^^^ | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl | return self._call_impl(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl | return forward_call(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 170, in forward | return self.forward_chunck( | ^^^^^^^^^^^^^^^^^^^^ | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/operators/attention.py", line 71, in forward_chunck | q = self.q_b_proj(self.q_a_layernorm(self.q_a_proj(hidden_states))) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl | return self._call_impl(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl | return forward_call(*args, **kwargs) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | File "/home/drros/miniconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 113, in forward | hidden_states = hidden_states.to(torch.float32) | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ | RuntimeError: CUDA error: invalid device function | CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. | For debugging consider passing CUDA_LAUNCH_BLOCKING=1 | Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. | +------------------------------------ ``` versions: ``` (base) drros@tesla-ubuntu:~$ nvcc --version nvcc: NVIDIA (R) Cuda compiler driver Copyright (c) 2005-2023 NVIDIA Corporation Built on Tue_Aug_15_22:02:13_PDT_2023 Cuda compilation tools, release 12.2, V12.2.140 Build cuda_12.2.r12.2/compiler.33191640_0 (base) drros@tesla-ubuntu:~$ uname -a Linux tesla-ubuntu 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux ```
bsakaguchi commented 2026-10-29 14:33:06 +00:00
Author
Owner
Copy link

@Chain-Mao commented on GitHub (Oct 9, 2024):

I have the same problem, did you have solved it?

<!-- gh-comment-id:2401805287 --> @Chain-Mao commented on GitHub (Oct 9, 2024): I have the same problem, did you have solved it?
bsakaguchi commented 2026-10-29 14:33:06 +00:00
Author
Owner
Copy link

@drrros commented on GitHub (Oct 9, 2024):

I have the same problem, did you have solved it?

No. Tried different versions of torch, but it did not solve the issue. What GPUs are you using? Mine are P40's.

<!-- gh-comment-id:2402289352 --> @drrros commented on GitHub (Oct 9, 2024): > I have the same problem, did you have solved it? No. Tried different versions of torch, but it did not solve the issue. What GPUs are you using? Mine are P40's.
bsakaguchi commented 2026-10-29 14:33:06 +00:00
Author
Owner
Copy link

@Chain-Mao commented on GitHub (Oct 10, 2024):

I have the same problem, did you have solved it?

No. Tried different versions of torch, but it did not solve the issue. What GPUs are you using? Mine are P40's.

My gpu is V100, maybe I find the reason, the V100 device use vlota architecture，this problem lead to I can't use flash-atten normally，so most of this project is incompatible.

<!-- gh-comment-id:2403816458 --> @Chain-Mao commented on GitHub (Oct 10, 2024): > > I have the same problem, did you have solved it? > > No. Tried different versions of torch, but it did not solve the issue. What GPUs are you using? Mine are P40's. My gpu is V100, maybe I find the reason, the V100 device use vlota architecture，this problem lead to I can't use flash-atten normally，so most of this project is incompatible.
bsakaguchi commented 2026-10-29 14:33:06 +00:00
Author
Owner
Copy link

@UnicornChan commented on GitHub (Oct 25, 2024):

I have the same problem, did you have solved it?

No. Tried different versions of torch, but it did not solve the issue. What GPUs are you using? Mine are P40's.

Sorry, we use the Marlin operator to calculate the layer on the GPU. It requires Compute Capability 8.0 or above to run. However, the Compute Capability of P40 is 6.1, so it cannot run. Maybe you can run it by removing the Marlin operator. @Azure-Tang Maybe you can show how to remove Marlin?

<!-- gh-comment-id:2437142279 --> @UnicornChan commented on GitHub (Oct 25, 2024): > > I have the same problem, did you have solved it? > > No. Tried different versions of torch, but it did not solve the issue. What GPUs are you using? Mine are P40's. Sorry, we use the Marlin operator to calculate the layer on the GPU. It requires Compute Capability 8.0 or above to run. However, the Compute Capability of P40 is 6.1, so it cannot run. Maybe you can run it by removing the Marlin operator. @Azure-Tang Maybe you can show how to remove Marlin?
bsakaguchi commented 2026-10-29 14:33:07 +00:00
Author
Owner
Copy link

@buliaoyin commented on GitHub (Feb 11, 2025):

Same issue with 2080Ti, any solution to support GPU lower than 8.0?

<!-- gh-comment-id:2651289470 --> @buliaoyin commented on GitHub (Feb 11, 2025): Same issue with 2080Ti, any solution to support GPU lower than 8.0?
bsakaguchi commented 2026-10-29 14:33:07 +00:00
Author
Owner
Copy link

@MrYang-Jia commented on GitHub (Feb 15, 2025):

Oh my, me too.
Same issue with 2080Ti,this Capability is 7.5

<!-- gh-comment-id:2660636258 --> @MrYang-Jia commented on GitHub (Feb 15, 2025): Oh my, me too. Same issue with 2080Ti,this Capability is 7.5
bsakaguchi commented 2026-10-29 14:33:07 +00:00
Author
Owner
Copy link

@newstargo commented on GitHub (Feb 16, 2025):

+1

<!-- gh-comment-id:2661680836 --> @newstargo commented on GitHub (Feb 16, 2025): +1
bsakaguchi commented 2026-10-29 14:33:07 +00:00
Author
Owner
Copy link

@Azure-Tang commented on GitHub (Feb 17, 2025):

For those who don't have ampere gpu, plz refer to #150

<!-- gh-comment-id:2662362745 --> @Azure-Tang commented on GitHub (Feb 17, 2025): For those who don't have ampere gpu, plz refer to #150
bsakaguchi commented 2026-10-29 14:33:08 +00:00
Author
Owner
Copy link

@gobirdiego commented on GitHub (Feb 22, 2025):

我也有同样的问题，解决方案是，flash-attn 有两个版本。在确认cu、cp、torch匹配的同时，文件名还有个true和false的不同。选对就通过了

<!-- gh-comment-id:2676185439 --> @gobirdiego commented on GitHub (Feb 22, 2025): 我也有同样的问题，解决方案是，flash-attn 有两个版本。在确认cu、cp、torch匹配的同时，文件名还有个true和false的不同。选对就通过了
bsakaguchi commented 2026-10-29 14:33:08 +00:00
Author
Owner
Copy link

@CloudWise-Lukemiao commented on GitHub (Feb 28, 2025):

我也有同样的问题，解决方案是，flash-attn 有两个版本。在确认cu、cp、torch匹配的同时，文件名还有个true和false的不同。选对就通过了

该怎么修改呢或者选择呢？

<!-- gh-comment-id:2689647155 --> @CloudWise-Lukemiao commented on GitHub (Feb 28, 2025): > 我也有同样的问题，解决方案是，flash-attn 有两个版本。在确认cu、cp、torch匹配的同时，文件名还有个true和false的不同。选对就通过了 该怎么修改呢或者选择呢？
bsakaguchi commented 2026-10-29 14:33:08 +00:00
Author
Owner
Copy link

@gtzhao34 commented on GitHub (Mar 9, 2025):

For those who don't have ampere gpu, plz refer to #150

How come I use A800 and still have similar problems. I tried deepseek-v2-lite-chat. I can fix this by replacing all the "KLinearMarlin" by "KLinearTorch", but I think A800 is supposed to be able to support marlin kernel.

Traceback (most recent call last):
  File "/home/s2024244004/zgt/ktransformers/ktransformers/local_chat.py", line 186, in <module>
    fire.Fire(local_chat)
  File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/s2024244004/zgt/ktransformers/ktransformers/local_chat.py", line 179, in local_chat
    generated = prefill_and_generate(
                ^^^^^^^^^^^^^^^^^^^^^
  File "/home/s2024244004/zgt/ktransformers/ktransformers/util/utils.py", line 208, in prefill_and_generate
    logits = chunk_prefill(inputs[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end], past_key_values)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/s2024244004/zgt/ktransformers/ktransformers/util/utils.py", line 165, in chunk_prefill
    logits = model(
             ^^^^^^
  File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/s2024244004/zgt/ktransformers/ktransformers/models/modeling_deepseek.py", line 1731, in forward
    outputs = self.model(
              ^^^^^^^^^^^
  File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/s2024244004/zgt/ktransformers/ktransformers/operators/models.py", line 727, in forward
    layer_outputs = decoder_layer(
                    ^^^^^^^^^^^^^^
  File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/s2024244004/zgt/ktransformers/ktransformers/models/modeling_deepseek.py", line 1238, in forward
    hidden_states, self_attn_weights, present_key_value = self.self_attn(
                                                          ^^^^^^^^^^^^^^^
  File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/s2024244004/zgt/ktransformers/ktransformers/operators/attention.py", line 617, in forward
    return self.forward_linux_triton(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/s2024244004/zgt/ktransformers/ktransformers/operators/attention.py", line 211, in forward_linux_triton
    compressed_kv = self.kv_a_proj_with_mqa(hidden_states)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl
    return forward_call(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/s2024244004/zgt/ktransformers/ktransformers/operators/linear.py", line 507, in forward
    y = self.generate_linear.forward(x)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/s2024244004/zgt/ktransformers/ktransformers/operators/linear.py", line 327, in forward
    padding_input[:,:self.orin_in_features] = x
    ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: invalid device function
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
<!-- gh-comment-id:2708854558 --> @gtzhao34 commented on GitHub (Mar 9, 2025): > For those who don't have ampere gpu, plz refer to [#150](https://github.com/kvcache-ai/ktransformers/issues/150) How come I use A800 and still have similar problems. I tried deepseek-v2-lite-chat. I can fix this by replacing all the "KLinearMarlin" by "KLinearTorch", but I think A800 is supposed to be able to support marlin kernel. ``` Traceback (most recent call last): File "/home/s2024244004/zgt/ktransformers/ktransformers/local_chat.py", line 186, in <module> fire.Fire(local_chat) File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/s2024244004/zgt/ktransformers/ktransformers/local_chat.py", line 179, in local_chat generated = prefill_and_generate( ^^^^^^^^^^^^^^^^^^^^^ File "/home/s2024244004/zgt/ktransformers/ktransformers/util/utils.py", line 208, in prefill_and_generate logits = chunk_prefill(inputs[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end], past_key_values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/s2024244004/zgt/ktransformers/ktransformers/util/utils.py", line 165, in chunk_prefill logits = model( ^^^^^^ File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/s2024244004/zgt/ktransformers/ktransformers/models/modeling_deepseek.py", line 1731, in forward outputs = self.model( ^^^^^^^^^^^ File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/s2024244004/zgt/ktransformers/ktransformers/operators/models.py", line 727, in forward layer_outputs = decoder_layer( ^^^^^^^^^^^^^^ File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/s2024244004/zgt/ktransformers/ktransformers/models/modeling_deepseek.py", line 1238, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/s2024244004/zgt/ktransformers/ktransformers/operators/attention.py", line 617, in forward return self.forward_linux_triton( ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/s2024244004/zgt/ktransformers/ktransformers/operators/attention.py", line 211, in forward_linux_triton compressed_kv = self.kv_a_proj_with_mqa(hidden_states) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1736, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/s2024244004/miniconda3/envs/ktransformer/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1747, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/s2024244004/zgt/ktransformers/ktransformers/operators/linear.py", line 507, in forward y = self.generate_linear.forward(x) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/s2024244004/zgt/ktransformers/ktransformers/operators/linear.py", line 327, in forward padding_input[:,:self.orin_in_features] = x ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^ RuntimeError: CUDA error: invalid device function CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. ```
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#52
No description provided.