kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #1389] [Bug] DeepSeek-V3-0324 671b:Q4_K_M one prompt crash / insanity #932

New issue
Open
opened 2026-10-29 14:46:19 +00:00 by bsakaguchi · 8 comments
bsakaguchi commented 2026-10-29 14:46:19 +00:00
Owner
Copy link

Originally created by @createthis on GitHub (Jun 13, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1389

Originally assigned to: @qiyuxinlin on GitHub.

Checklist

  • 1. I have searched related issues but cannot get the expected help.
  • 2. The bug has not been fixed in the latest version.
  • 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
  • 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
  • 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

I have a one shot prompt that "crashes" ktransformers when running DeepSeek V3 0324 Q4_K_M. Sometimes I get a backtrace, but sometimes it just outputs gibberish afterward. Here's the backtrace when I get it:

INFO:     192.168.0.201:65208 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error
ERROR:    Exception in ASGI application 
Traceback (most recent call last):
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403,
 in run_asgi
    result = await app(  # type: ignore[func-returns-value]
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60,
 in __call__
    return await self.app(scope, receive, send)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call
__
    await super().__call__(scope, receive, send)
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/applications.py", line 112, in __cal
l__
    await self.middleware_stack(scope, receive, send)
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in 
__call__
    raise exc
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in 
__call__
    await self.app(scope, receive, _send)
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __c
all__
    await self.app(scope, receive, send)
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, 
in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in 
wrapped_app
    raise exc
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in 
wrapped_app
    await app(scope, receive, sender)   
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/routing.py", line 734, in app
    await route.handle(scope, receive, send)
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in 
wrapped_app
    raise exc
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in 
wrapped_app
    await app(scope, receive, sender)   
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 436, in chat_completion
    async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens):
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 241, in inference
    async for v in super().inference(local_messages, thread_id, temperature, top_p, max_tokens, max_completion_tokens):
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 474, in inference
    for t, finish_reason in self.generate():
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
    response = gen.send(None)
               ^^^^^^^^^^^^^^
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 411, in generate
    next_token = self.decode_one_tokens()
                 ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 127, in decode_one_tokens
    return self.logits_to_token(logits)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 299, in logits_to_token
    last = last.item()
           ^^^^^^^^^^^
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1

Reproduction

I recorded this session using mitmdump. You can read all about it and download a curl script to replay the prompt here: https://github.com/createthis/ktransformers_8dc1ab9_bug_deepseek_v3_0324

I mention a specific commit of ktransformers in the README, but I just reproduced it using HEAD 64ec0ec too.

Environment

dual EPYC 9355
768gb RAM
96gb blackwell 6000 pro
CUDA Version: 12.8
Driver Version: 570.124.06
OS: ubuntu 24.04.02 LTS

Originally created by @createthis on GitHub (Jun 13, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1389 Originally assigned to: @qiyuxinlin on GitHub. ### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [x] 2. The bug has not been fixed in the latest version. - [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug I have a one shot prompt that "crashes" ktransformers when running DeepSeek V3 0324 Q4_K_M. Sometimes I get a backtrace, but sometimes it just outputs gibberish afterward. Here's the backtrace when I get it: ``` INFO: 192.168.0.201:65208 - "POST /v1/chat/completions HTTP/1.1" 500 Internal Server Error ERROR: Exception in ASGI application Traceback (most recent call last): File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__ return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call __ await super().__call__(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/applications.py", line 112, in __cal l__ await self.middleware_stack(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__ raise exc File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__ await self.app(scope, receive, _send) File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __c all__ await self.app(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__ await self.middleware_stack(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/routing.py", line 734, in app await route.handle(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle await self.app(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/routing.py", line 76, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/starlette/routing.py", line 73, in app response = await f(request) ^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/fastapi/routing.py", line 301, in app raw_response = await run_endpoint_function( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/fastapi/routing.py", line 212, in run_endpoint_function return await dependant.call(**values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 436, in chat_completion async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens): File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 241, in inference async for v in super().inference(local_messages, thread_id, temperature, top_p, max_tokens, max_completion_tokens): File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 474, in inference for t, finish_reason in self.generate(): File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context response = gen.send(None) ^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 411, in generate next_token = self.decode_one_tokens() ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 127, in decode_one_tokens return self.logits_to_token(logits) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_head_cu128/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 299, in logits_to_token last = last.item() ^^^^^^^^^^^ RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 ``` ### Reproduction I recorded this session using `mitmdump`. You can read all about it and download a curl script to replay the prompt here: https://github.com/createthis/ktransformers_8dc1ab9_bug_deepseek_v3_0324 I mention a specific commit of ktransformers in the README, but I just reproduced it using HEAD `64ec0ec` too. ### Environment dual EPYC 9355 768gb RAM 96gb blackwell 6000 pro CUDA Version: 12.8 Driver Version: 570.124.06 OS: ubuntu 24.04.02 LTS
bsakaguchi commented 2026-10-29 14:46:20 +00:00
Author
Owner
Copy link

@createthis commented on GitHub (Jun 13, 2025):

@qiuqiu10 ^

<!-- gh-comment-id:2971540357 --> @createthis commented on GitHub (Jun 13, 2025): @qiuqiu10 ^
bsakaguchi commented 2026-10-29 14:46:20 +00:00
Author
Owner
Copy link

@createthis commented on GitHub (Jun 13, 2025):

Similar to https://github.com/kvcache-ai/ktransformers/issues/1375

<!-- gh-comment-id:2971540950 --> @createthis commented on GitHub (Jun 13, 2025): Similar to https://github.com/kvcache-ai/ktransformers/issues/1375
bsakaguchi commented 2026-10-29 14:46:20 +00:00
Author
Owner
Copy link

@qiyuxinlin commented on GitHub (Jun 16, 2025):

Hello @createthis, the ktransformers backend is no longer being maintained. We will update the balance_serve prefix-cache version within the next two weeks.

<!-- gh-comment-id:2976089588 --> @qiyuxinlin commented on GitHub (Jun 16, 2025): Hello @createthis, the ktransformers backend is no longer being maintained. We will update the balance_serve prefix-cache version within the next two weeks.
bsakaguchi commented 2026-10-29 14:46:20 +00:00
Author
Owner
Copy link

@createthis commented on GitHub (Jun 16, 2025):

@qiyuxinlin ok. I tried balance_serve too and it crashes after some time as well. I've been using llama.cpp for the past few days now that I have more VRAM available. It's a little slower, but it doesn't crash, so I'll probably be using that for the foreseeable future unless ktransformers fixes the stability issues.

<!-- gh-comment-id:2976678932 --> @createthis commented on GitHub (Jun 16, 2025): @qiyuxinlin ok. I tried balance_serve too and it crashes after some time as well. I've been using llama.cpp for the past few days now that I have more VRAM available. It's a little slower, but it doesn't crash, so I'll probably be using that for the foreseeable future unless ktransformers fixes the stability issues.
bsakaguchi commented 2026-10-29 14:46:21 +00:00
Author
Owner
Copy link

@qiyuxinlin commented on GitHub (Jun 18, 2025):

@qiyuxinlin ok. I tried balance_serve too and it crashes after some time as well. I've been using llama.cpp for the past few days now that I have more VRAM available. It's a little slower, but it doesn't crash, so I'll probably be using that for the foreseeable future unless ktransformers fixes the stability issues.

Stability is indeed the most important thing. Could you describe how the balance serve crash manifests?

<!-- gh-comment-id:2982981625 --> @qiyuxinlin commented on GitHub (Jun 18, 2025): > [@qiyuxinlin](https://github.com/qiyuxinlin) ok. I tried balance_serve too and it crashes after some time as well. I've been using llama.cpp for the past few days now that I have more VRAM available. It's a little slower, but it doesn't crash, so I'll probably be using that for the foreseeable future unless ktransformers fixes the stability issues. Stability is indeed the most important thing. Could you describe how the balance serve crash manifests?
bsakaguchi commented 2026-10-29 14:46:21 +00:00
Author
Owner
Copy link

@createthis commented on GitHub (Jun 18, 2025):

Stability is indeed the most important thing. Could you describe how the balance serve crash manifests?

Exactly the same way the ktransformer backend crash manifests. It runs for a while, then it either goes insane or backtraces. I suspect if you fix the ktransformers backend, the balance_serve backend will also be fixed.

<!-- gh-comment-id:2984189964 --> @createthis commented on GitHub (Jun 18, 2025): > Stability is indeed the most important thing. Could you describe how the balance serve crash manifests? Exactly the same way the ktransformer backend crash manifests. It runs for a while, then it either goes insane or backtraces. I suspect if you fix the ktransformers backend, the balance_serve backend will also be fixed.
bsakaguchi commented 2026-10-29 14:46:22 +00:00
Author
Owner
Copy link

@createthis commented on GitHub (Jun 22, 2025):

Here's a video showing this crash: https://youtu.be/vfi9LRJxgHs?si=JoGtEpu4MoxebmAT&t=1314

<!-- gh-comment-id:2993852299 --> @createthis commented on GitHub (Jun 22, 2025): Here's a video showing this crash: https://youtu.be/vfi9LRJxgHs?si=JoGtEpu4MoxebmAT&t=1314
bsakaguchi commented 2026-10-29 14:46:22 +00:00
Author
Owner
Copy link

@qiyuxinlin commented on GitHub (Jun 22, 2025):

thanks

<!-- gh-comment-id:2993911141 --> @qiyuxinlin commented on GitHub (Jun 22, 2025): thanks
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#932
No description provided.