kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #242] chat Error #146

New issue
Open
opened 2026-10-29 14:34:58 +00:00 by bsakaguchi · 11 comments
bsakaguchi commented 2026-10-29 14:34:58 +00:00
Owner
Copy link

Originally created by @Zha-Miku on GitHub (Feb 14, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/242

detail

nvidia-smi 


+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.86.15              Driver Version: 570.86.15      CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA L40                     On  |   00000000:15:00.0 Off |                    0 |
| N/A   29C    P0             86W /  300W |    6551MiB /  46068MiB |      9%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA L40                     On  |   00000000:18:00.0 Off |                    0 |
| N/A   23C    P8             24W /  300W |       4MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA L40                     On  |   00000000:19:00.0 Off |                    0 |
| N/A   22C    P8             36W /  300W |       4MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA L40                     On  |   00000000:1C:00.0 Off |                    0 |
| N/A   24C    P8             24W /  300W |       4MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   4  NVIDIA L40                     On  |   00000000:1D:00.0 Off |                    0 |
| N/A   27C    P0             70W /  300W |       4MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   5  NVIDIA L40                     On  |   00000000:96:00.0 Off |                    0 |
| N/A   25C    P0             71W /  300W |       4MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   6  NVIDIA L40                     On  |   00000000:99:00.0 Off |                    0 |
| N/A   25C    P0             69W /  300W |       4MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   7  NVIDIA L40                     On  |   00000000:9A:00.0 Off |                    0 |
| N/A   25C    P0             70W /  300W |       4MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   8  NVIDIA L40                     On  |   00000000:9D:00.0 Off |                    0 |
| N/A   24C    P0             46W /  300W |       4MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   9  NVIDIA L40                     On  |   00000000:9E:00.0 Off |                    0 |
| N/A   24C    P0             46W /  300W |       4MiB /  46068MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A         2973245      C   python                                 6542MiB |
+-----------------------------------------------------------------------------------------+

ram 


top - 09:04:10 up 1 day, 18:58,  9 users,  load average: 5.13, 3.94, 3.03
Tasks: 1474 total,   5 running, 1469 sleeping,   0 stopped,   0 zombie
%Cpu(s):  3.4 us,  1.4 sy,  0.0 ni, 95.1 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st
MiB Mem : 515593.5 total, 107267.3 free,  19094.7 used, 389231.5 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used. 493030.5 avail Mem

cpu

Intel(R) Xeon(R) Platinum 8338C CPU @ 2.60GHz

Error

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 179, in <module>
    fire.Fire(local_chat)
  File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 173, in local_chat
    generated = prefill_and_generate(
                ^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/util/utils.py", line 196, in prefill_and_generate
    tokens.append(int(next_token))
                  ^^^^^^^^^^^^^^^
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
Originally created by @Zha-Miku on GitHub (Feb 14, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/242 # detail <details><summary>nvidia-smi</summary> <p> ```sh +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 570.86.15 Driver Version: 570.86.15 CUDA Version: 12.8 | |-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA L40 On | 00000000:15:00.0 Off | 0 | | N/A 29C P0 86W / 300W | 6551MiB / 46068MiB | 9% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 1 NVIDIA L40 On | 00000000:18:00.0 Off | 0 | | N/A 23C P8 24W / 300W | 4MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 2 NVIDIA L40 On | 00000000:19:00.0 Off | 0 | | N/A 22C P8 36W / 300W | 4MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 3 NVIDIA L40 On | 00000000:1C:00.0 Off | 0 | | N/A 24C P8 24W / 300W | 4MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 4 NVIDIA L40 On | 00000000:1D:00.0 Off | 0 | | N/A 27C P0 70W / 300W | 4MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 5 NVIDIA L40 On | 00000000:96:00.0 Off | 0 | | N/A 25C P0 71W / 300W | 4MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 6 NVIDIA L40 On | 00000000:99:00.0 Off | 0 | | N/A 25C P0 69W / 300W | 4MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 7 NVIDIA L40 On | 00000000:9A:00.0 Off | 0 | | N/A 25C P0 70W / 300W | 4MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 8 NVIDIA L40 On | 00000000:9D:00.0 Off | 0 | | N/A 24C P0 46W / 300W | 4MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ | 9 NVIDIA L40 On | 00000000:9E:00.0 Off | 0 | | N/A 24C P0 46W / 300W | 4MiB / 46068MiB | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 2973245 C python 6542MiB | +-----------------------------------------------------------------------------------------+ ``` </p> </details> <details><summary>ram</summary> <p> ```sh top - 09:04:10 up 1 day, 18:58, 9 users, load average: 5.13, 3.94, 3.03 Tasks: 1474 total, 5 running, 1469 sleeping, 0 stopped, 0 zombie %Cpu(s): 3.4 us, 1.4 sy, 0.0 ni, 95.1 id, 0.0 wa, 0.0 hi, 0.1 si, 0.0 st MiB Mem : 515593.5 total, 107267.3 free, 19094.7 used, 389231.5 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used. 493030.5 avail Mem ``` </p> </details> ## cpu Intel(R) Xeon(R) Platinum 8338C CPU @ 2.60GHz # Error ```sh Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 179, in <module> fire.Fire(local_chat) File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 135, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 468, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 684, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/local_chat.py", line 173, in local_chat generated = prefill_and_generate( ^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/anaconda3/envs/ktransformers/lib/python3.11/site-packages/ktransformers/util/utils.py", line 196, in prefill_and_generate tokens.append(int(next_token)) ^^^^^^^^^^^^^^^ RuntimeError: CUDA error: device-side assert triggered CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. ```
bsakaguchi commented 2026-10-29 14:34:58 +00:00
Author
Owner
Copy link

@Zha-Miku commented on GitHub (Feb 14, 2025):

new error

Chat: hi
Okay, the /pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion `probability tensor contains either `inf`, `nan` or element < 0` failed.
<!-- gh-comment-id:2658056486 --> @Zha-Miku commented on GitHub (Feb 14, 2025): # new error ```sh Chat: hi Okay, the /pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion `probability tensor contains either `inf`, `nan` or element < 0` failed. ```
bsakaguchi commented 2026-10-29 14:34:58 +00:00
Author
Owner
Copy link

@xialang2012 commented on GitHub (Feb 16, 2025):

解决了吗

<!-- gh-comment-id:2661441300 --> @xialang2012 commented on GitHub (Feb 16, 2025): 解决了吗
bsakaguchi commented 2026-10-29 14:34:59 +00:00
Author
Owner
Copy link

@newstargo commented on GitHub (Feb 16, 2025):

+1

<!-- gh-comment-id:2661678015 --> @newstargo commented on GitHub (Feb 16, 2025): +1
bsakaguchi commented 2026-10-29 14:34:59 +00:00
Author
Owner
Copy link

@wangfan008 commented on GitHub (Feb 18, 2025):

+1

<!-- gh-comment-id:2665858464 --> @wangfan008 commented on GitHub (Feb 18, 2025): +1
bsakaguchi commented 2026-10-29 14:34:59 +00:00
Author
Owner
Copy link

@chuangzhidan commented on GitHub (Feb 19, 2025):

+1

<!-- gh-comment-id:2669203508 --> @chuangzhidan commented on GitHub (Feb 19, 2025): +1
bsakaguchi commented 2026-10-29 14:34:59 +00:00
Author
Owner
Copy link

@konghr commented on GitHub (Feb 20, 2025):

+1

<!-- gh-comment-id:2671441602 --> @konghr commented on GitHub (Feb 20, 2025): +1
bsakaguchi commented 2026-10-29 14:34:59 +00:00
Author
Owner
Copy link

@CmyLjj commented on GitHub (Feb 21, 2025):

how to deal with ?

<!-- gh-comment-id:2673453421 --> @CmyLjj commented on GitHub (Feb 21, 2025): how to deal with ?
bsakaguchi commented 2026-10-29 14:34:59 +00:00
Author
Owner
Copy link

@xieqihui commented on GitHub (Mar 5, 2025):

+1

<!-- gh-comment-id:2700498129 --> @xieqihui commented on GitHub (Mar 5, 2025): +1
bsakaguchi commented 2026-10-29 14:34:59 +00:00
Author
Owner
Copy link

@xxchen86 commented on GitHub (Mar 10, 2025):

+1

<!-- gh-comment-id:2709577204 --> @xxchen86 commented on GitHub (Mar 10, 2025): +1
bsakaguchi commented 2026-10-29 14:34:59 +00:00
Author
Owner
Copy link

@Print1n commented on GitHub (Mar 10, 2025):

+1

<!-- gh-comment-id:2712043625 --> @Print1n commented on GitHub (Mar 10, 2025): +1
bsakaguchi commented 2026-10-29 14:34:59 +00:00
Author
Owner
Copy link

@tunantu commented on GitHub (Mar 14, 2025):

+1

<!-- gh-comment-id:2723757597 --> @tunantu commented on GitHub (Mar 14, 2025): +1
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#146
No description provided.