kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #2009] Crash when enable MTP on KT 0.6.2 DeepSeekV4Flash #1235

New issue
Open
opened 2026-10-29 14:50:23 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-29 14:50:23 +00:00
Owner
Copy link

Originally created by @mrgaolei on GitHub (May 16, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2009

Reminder

  • I have read the above rules and searched the existing issues.

System Info

When I add MTP to startup script:

  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --speculative-moe-runner-backend auto \

It crash, but without that, everything OK.

This is full script:

export TORCH_CUDA_ARCH_LIST="12.0+PTX"
export SGLANG_DSV4_MODE=2604
export SGLANG_DSV4_2604_SUBMODE=2604B

numactl --interleave=all python -m sglang.launch_server \
  --served-model-name DeepSeek-V4-Flash \
  --host 0.0.0.0 --port 30000 \
  --model /mnt/nvme0/ai-models/DeepSeek-V4-Flash/ \
  --kt-weight-path /mnt/nvme0/ai-models/DeepSeek-V4-Flash \
  --kt-method MXFP4 \
  --kt-num-gpu-experts 10 \
  --kt-cpuinfer 35 \
  --kt-threadpool-count 2 \
  --kt-gpu-prefill-token-threshold 4096 \
  --kt-enable-dynamic-expert-update \
  --tensor-parallel-size 1 \
  --context-length 32768 \
  --attention-backend flashinfer \
  --mem-fraction-static 0.85 \
  --chunked-prefill-size 2048 \
  --max-prefill-tokens 2048 \
  --max-running-requests 2 \
  --watchdog-timeout 1200 \
  --disable-shared-experts-fusion \
  --trust-remote-code \
  --cuda-graph-bs 1 \
  --cuda-graph-max-bs 1 \
  --disable-radix-cache \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --speculative-moe-runner-backend auto \
  --skip-server-warmup

Reproduction

[2026-05-16 14:59:21] INFO:     Uvicorn running on http://0.0.0.0:30000 (Press CTRL+C to quit)                                                                    /home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/fastapi/routing.py:120: FastAPIDeprecationWarning: ORJSONResponse is deprecated, FastAPI now serializes data directly to JSON bytes via Pydantic when a return type or response model is set, which is faster and doesn't need a custom response class. Read more in the FastAPI docs: https://fastapi.tiangolo.com/advanced/custom-response/#orjson-or-response-model and https://fastapi.tiangolo.com/tutorial/response-model/     response = await f(request)                                                                                                                                     [2026-05-16 14:59:49] INFO:     172.17.0.2:44190 - "GET /v1/models HTTP/1.1" 200 OK                                                                               [2026-05-16 15:00:01] INFO:     172.17.0.2:44256 - "POST /v1/chat/completions HTTP/1.1" 200 OK                                                                    Fatal Python error: Segmentation fault                                                                                                                                                                                                                                                                                              Thread 0x0000701261fff6c0 (most recent call first):                                                                                                                 File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/utils/watchdog.py", line 145 in _watchdog_once                                File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/utils/watchdog.py", line 125 in _watchdog_thread
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 1012 in run
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x0000701dfbfff6c0 (most recent call first):
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/torch/_inductor/compile_worker/subproc_pool.py", line 73 in _recv_msg
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/torch/_inductor/compile_worker/subproc_pool.py", line 228 in _read_thread
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 1012 in run
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x0000701df7ffe6c0 (most recent call first):
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 359 in wait
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 655 in wait
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/tqdm/_monitor.py", line 60 in run
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x0000701e3bfff6c0 (most recent call first):
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 359 in wait
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 655 in wait
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/tqdm/_monitor.py", line 60 in run
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 1075 in _bootstrap_inner
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 1032 in _bootstrap

Thread 0x0000704ef8876740 (most recent call first):
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/torch/cuda/graphs.py", line 141 in replay
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 1074 in replay
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 2541 in _forward_raw
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 2456 in forward
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/managers/tp_worker.py", line 456 in forward_batch_generation
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/speculative/eagle_worker_v2.py", line 757 in verify
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/speculative/eagle_worker_v2.py", line 697 in forward_batch_generation
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 2405 in run_batch
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 1176 in event_loop_overlap
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120 in decorate_context
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 3245 in run_scheduler_process
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/multiprocessing/process.py", line 108 in run
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/multiprocessing/process.py", line 314 in _bootstrap
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/multiprocessing/spawn.py", line 135 in _main
  File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/multiprocessing/spawn.py", line 122 in spawn_main
  File "<string>", line 1 in <module>

Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, pybase64._pybase64, charset_normalizer.md, charset_normalizer.cd, requests.packages.
charset_normalizer.md, requests.packages.chardet.md, requests.packages.charset_normalizer.cd, requests.packages.chardet.cd, psutil._psutil_linux, multidict._multi
dict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenl
ist, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg,
torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, zmq.backend.cython._zmq, PIL._imaging, sentencepiece._sentencepiece, yaml._yaml, regex._regex
, markupsafe._speedups, cuda_utils, PIL._imagingft, av._core, av.logging, av.bytesource, av.buffer, av.audio.format, av.error, av.dictionary, av.container.pyio, a
v.option, av.descriptor, av.format, av.utils, av.stream, av.container.streams, av.sidedata.encparams, av.sidedata.motionvectors, av.sidedata.sidedata, av.opaque,
av.packet, av.container.input, av.container.output, av.container.core, av.codec.context, av.video.format, av.video.reformatter, av.plane, av.video.plane, av.video
.frame, av.video.stream, av.codec.hwaccel, av.codec.codec, av.frame, av.audio.layout, av.audio.plane, av.audio.frame, av.audio.stream, av.filter.link, av.filter.c
ontext, av.filter.graph, av.filter.filter, av.filter.loudnorm, av.audio.resampler, av.audio.codeccontext, av.audio.fifo, av.bitstream, av.video.codeccontext, nump
y.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._generator, numpy.random._mt19937, numpy.random._p
hilox, numpy.random._sfc64, numpy.random.mtrand, _cffi_backend, _cyutility, scipy._cyutility, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack,
 scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._batched_linalg, scipy.linalg._decomp_lu_cython, sci
py.linalg._matfuncs_schur_sqrtm, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse._s
parsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpacklib, scipy.sparse.linalg._pr
opack, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy
.optimize._slsqplib, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx
, scipy.special._ellip_harm_2, scipy.special._special_ufuncs, scipy.special._gufuncs, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.li
nalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial.
_hausdorff, scipy.spatial._distance_wrap, scipy.spatial.transform._rotation_cy, scipy.spatial.transform._rigid_transform_cy, scipy.optimize._direct, setproctitle.
_setproctitle, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, tvm_ffi.core, msgspec._core, grpc._cython.cygrpc, google._upb._mess
age, cuda.bindings._bindings.cynvrtc, cuda.bindings.cynvrtc, cuda.bindings.nvrtc, cuda.bindings._bindings.cyruntime_ptds, cuda.bindings._bindings.cyruntime, cuda.
bindings.cyruntime, cuda.bindings.runtime, cuda.tile._cext, _xxsubinterpreters, tilelang_cython_wrapper, __triton_launcher, uvloop.loop (total: 163)
!!!!!!! Segfault encountered !!!!!!!
  File "./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c", line 0, in 0x0000704ef864532f
  File "../../../../../libstdc++-v3/src/c++11/thread.cc", line 104, in execute_native_thread_routine
  File "./nptl/pthread_create.c", line 447, in start_thread
  File "../sysdeps/unix/sysv/linux/x86_64/clone3.S", line 78, in clone3
  File "<unknown>", line 0, in 0xffffffffffffffff

Others

No response

Originally created by @mrgaolei on GitHub (May 16, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2009 ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info When I add MTP to startup script: ``` --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-moe-runner-backend auto \ ``` It crash, but without that, everything OK. This is full script: ``` export TORCH_CUDA_ARCH_LIST="12.0+PTX" export SGLANG_DSV4_MODE=2604 export SGLANG_DSV4_2604_SUBMODE=2604B numactl --interleave=all python -m sglang.launch_server \ --served-model-name DeepSeek-V4-Flash \ --host 0.0.0.0 --port 30000 \ --model /mnt/nvme0/ai-models/DeepSeek-V4-Flash/ \ --kt-weight-path /mnt/nvme0/ai-models/DeepSeek-V4-Flash \ --kt-method MXFP4 \ --kt-num-gpu-experts 10 \ --kt-cpuinfer 35 \ --kt-threadpool-count 2 \ --kt-gpu-prefill-token-threshold 4096 \ --kt-enable-dynamic-expert-update \ --tensor-parallel-size 1 \ --context-length 32768 \ --attention-backend flashinfer \ --mem-fraction-static 0.85 \ --chunked-prefill-size 2048 \ --max-prefill-tokens 2048 \ --max-running-requests 2 \ --watchdog-timeout 1200 \ --disable-shared-experts-fusion \ --trust-remote-code \ --cuda-graph-bs 1 \ --cuda-graph-max-bs 1 \ --disable-radix-cache \ --speculative-algorithm EAGLE \ --speculative-num-steps 3 \ --speculative-eagle-topk 1 \ --speculative-num-draft-tokens 4 \ --speculative-moe-runner-backend auto \ --skip-server-warmup ``` ### Reproduction ```bash [2026-05-16 14:59:21] INFO: Uvicorn running on http://0.0.0.0:30000 (Press CTRL+C to quit) /home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/fastapi/routing.py:120: FastAPIDeprecationWarning: ORJSONResponse is deprecated, FastAPI now serializes data directly to JSON bytes via Pydantic when a return type or response model is set, which is faster and doesn't need a custom response class. Read more in the FastAPI docs: https://fastapi.tiangolo.com/advanced/custom-response/#orjson-or-response-model and https://fastapi.tiangolo.com/tutorial/response-model/ response = await f(request) [2026-05-16 14:59:49] INFO: 172.17.0.2:44190 - "GET /v1/models HTTP/1.1" 200 OK [2026-05-16 15:00:01] INFO: 172.17.0.2:44256 - "POST /v1/chat/completions HTTP/1.1" 200 OK Fatal Python error: Segmentation fault Thread 0x0000701261fff6c0 (most recent call first): File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/utils/watchdog.py", line 145 in _watchdog_once File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/utils/watchdog.py", line 125 in _watchdog_thread File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 1012 in run File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 1075 in _bootstrap_inner File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 1032 in _bootstrap Thread 0x0000701dfbfff6c0 (most recent call first): File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/torch/_inductor/compile_worker/subproc_pool.py", line 73 in _recv_msg File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/torch/_inductor/compile_worker/subproc_pool.py", line 228 in _read_thread File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 1012 in run File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 1075 in _bootstrap_inner File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 1032 in _bootstrap Thread 0x0000701df7ffe6c0 (most recent call first): File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 359 in wait File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 655 in wait File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/tqdm/_monitor.py", line 60 in run File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 1075 in _bootstrap_inner File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 1032 in _bootstrap Thread 0x0000701e3bfff6c0 (most recent call first): File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 359 in wait File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 655 in wait File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/tqdm/_monitor.py", line 60 in run File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 1075 in _bootstrap_inner File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/threading.py", line 1032 in _bootstrap Thread 0x0000704ef8876740 (most recent call first): File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/torch/cuda/graphs.py", line 141 in replay File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/model_executor/cuda_graph_runner.py", line 1074 in replay File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 2541 in _forward_raw File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/model_executor/model_runner.py", line 2456 in forward File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/managers/tp_worker.py", line 456 in forward_batch_generation File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/speculative/eagle_worker_v2.py", line 757 in verify File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/speculative/eagle_worker_v2.py", line 697 in forward_batch_generation File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 2405 in run_batch File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 1176 in event_loop_overlap File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 120 in decorate_context File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/site-packages/sglang/srt/managers/scheduler.py", line 3245 in run_scheduler_process File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/multiprocessing/process.py", line 108 in run File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/multiprocessing/process.py", line 314 in _bootstrap File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/multiprocessing/spawn.py", line 135 in _main File "/home/aigao/miniconda3/envs/kt-kernel/lib/python3.12/multiprocessing/spawn.py", line 122 in spawn_main File "<string>", line 1 in <module> Extension modules: numpy._core._multiarray_umath, numpy.linalg._umath_linalg, pybase64._pybase64, charset_normalizer.md, charset_normalizer.cd, requests.packages. charset_normalizer.md, requests.packages.chardet.md, requests.packages.charset_normalizer.cd, requests.packages.chardet.cd, psutil._psutil_linux, multidict._multi dict, yarl._quoting_c, propcache._helpers_c, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket.mask, aiohttp._websocket.reader_c, frozenlist._frozenl ist, torch._C, torch._C._dynamo.autograd_compiler, torch._C._dynamo.eval_frame, torch._C._dynamo.guards, torch._C._dynamo.utils, torch._C._fft, torch._C._linalg, torch._C._nested, torch._C._nn, torch._C._sparse, torch._C._special, zmq.backend.cython._zmq, PIL._imaging, sentencepiece._sentencepiece, yaml._yaml, regex._regex , markupsafe._speedups, cuda_utils, PIL._imagingft, av._core, av.logging, av.bytesource, av.buffer, av.audio.format, av.error, av.dictionary, av.container.pyio, a v.option, av.descriptor, av.format, av.utils, av.stream, av.container.streams, av.sidedata.encparams, av.sidedata.motionvectors, av.sidedata.sidedata, av.opaque, av.packet, av.container.input, av.container.output, av.container.core, av.codec.context, av.video.format, av.video.reformatter, av.plane, av.video.plane, av.video .frame, av.video.stream, av.codec.hwaccel, av.codec.codec, av.frame, av.audio.layout, av.audio.plane, av.audio.frame, av.audio.stream, av.filter.link, av.filter.c ontext, av.filter.graph, av.filter.filter, av.filter.loudnorm, av.audio.resampler, av.audio.codeccontext, av.audio.fifo, av.bitstream, av.video.codeccontext, nump y.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._pcg64, numpy.random._generator, numpy.random._mt19937, numpy.random._p hilox, numpy.random._sfc64, numpy.random.mtrand, _cffi_backend, _cyutility, scipy._cyutility, scipy._lib._ccallback_c, scipy.linalg._fblas, scipy.linalg._flapack, scipy.linalg.cython_lapack, scipy.linalg._cythonized_array_utils, scipy.linalg._solve_toeplitz, scipy.linalg._batched_linalg, scipy.linalg._decomp_lu_cython, sci py.linalg._matfuncs_schur_sqrtm, scipy.linalg._matfuncs_expm, scipy.linalg._linalg_pythran, scipy.linalg.cython_blas, scipy.linalg._decomp_update, scipy.sparse._s parsetools, _csparsetools, scipy.sparse._csparsetools, scipy.sparse.linalg._dsolve._superlu, scipy.sparse.linalg._eigen.arpack._arpacklib, scipy.sparse.linalg._pr opack, scipy.optimize._group_columns, scipy._lib.messagestream, scipy.optimize._trlib._trlib, scipy.optimize._lbfgsb, _moduleTNC, scipy.optimize._moduleTNC, scipy .optimize._slsqplib, scipy.optimize._minpack, scipy.optimize._lsq.givens_elimination, scipy.optimize._zeros, scipy._lib._uarray._uarray, scipy.special._ufuncs_cxx , scipy.special._ellip_harm_2, scipy.special._special_ufuncs, scipy.special._gufuncs, scipy.special._ufuncs, scipy.special._specfun, scipy.special._comb, scipy.li nalg._decomp_interpolative, scipy.optimize._bglu_dense, scipy.optimize._lsap, scipy.spatial._ckdtree, scipy.spatial._qhull, scipy.spatial._voronoi, scipy.spatial. _hausdorff, scipy.spatial._distance_wrap, scipy.spatial.transform._rotation_cy, scipy.spatial.transform._rigid_transform_cy, scipy.optimize._direct, setproctitle. _setproctitle, cuda.bindings._bindings.cydriver, cuda.bindings.cydriver, cuda.bindings.driver, tvm_ffi.core, msgspec._core, grpc._cython.cygrpc, google._upb._mess age, cuda.bindings._bindings.cynvrtc, cuda.bindings.cynvrtc, cuda.bindings.nvrtc, cuda.bindings._bindings.cyruntime_ptds, cuda.bindings._bindings.cyruntime, cuda. bindings.cyruntime, cuda.bindings.runtime, cuda.tile._cext, _xxsubinterpreters, tilelang_cython_wrapper, __triton_launcher, uvloop.loop (total: 163) !!!!!!! Segfault encountered !!!!!!! File "./signal/../sysdeps/unix/sysv/linux/x86_64/libc_sigaction.c", line 0, in 0x0000704ef864532f File "../../../../../libstdc++-v3/src/c++11/thread.cc", line 104, in execute_native_thread_routine File "./nptl/pthread_create.c", line 447, in start_thread File "../sysdeps/unix/sysv/linux/x86_64/clone3.S", line 78, in clone3 File "<unknown>", line 0, in 0xffffffffffffffff ``` ### Others _No response_
bsakaguchi commented 2026-10-29 14:50:23 +00:00
Author
Owner
Copy link

@jimtjames commented on GitHub (May 16, 2026):

What hardware are you running this on? With an RTX 5090 I'm not able to enable both MTP and that many GPU experts with that much mem fraction (there isn't enough VRAM for this). However, I just see a standard CUDA OOM rather than something like this.

In a previous issue I saw you mentioned using an RTX 4090 48GB. If that's the same situation here, you don't want to use export TORCH_CUDA_ARCH_LIST="12.0+PTX", as SM120 is Consumer Blackwell. You probably want to set:

export TORCH_CUDA_ARCH_LIST="8.9+PTX"
<!-- gh-comment-id:4468188954 --> @jimtjames commented on GitHub (May 16, 2026): What hardware are you running this on? With an RTX 5090 I'm not able to enable both MTP and that many GPU experts with that much mem fraction (there isn't enough VRAM for this). However, I just see a standard CUDA OOM rather than something like this. In a previous issue I saw you mentioned using an RTX 4090 48GB. If that's the same situation here, you don't want to use `export TORCH_CUDA_ARCH_LIST="12.0+PTX"`, as SM120 is Consumer Blackwell. You probably want to set: ``` export TORCH_CUDA_ARCH_LIST="8.9+PTX" ```
bsakaguchi commented 2026-10-29 14:50:24 +00:00
Author
Owner
Copy link

@mrgaolei commented on GitHub (May 18, 2026):

What hardware are you running this on? With an RTX 5090 I'm not able to enable both MTP and that many GPU experts with that much mem fraction (there isn't enough VRAM for this). However, I just see a standard CUDA OOM rather than something like this.

In a previous issue I saw you mentioned using an RTX 4090 48GB. If that's the same situation here, you don't want to use export TORCH_CUDA_ARCH_LIST="12.0+PTX", as SM120 is Consumer Blackwell. You probably want to set:

export TORCH_CUDA_ARCH_LIST="8.9+PTX"

Yes, I'm using 4090 48G, and also I changed export TORCH_CUDA_ARCH_LIST="8.9+PTX"
With same exception.

<!-- gh-comment-id:4474540961 --> @mrgaolei commented on GitHub (May 18, 2026): > What hardware are you running this on? With an RTX 5090 I'm not able to enable both MTP and that many GPU experts with that much mem fraction (there isn't enough VRAM for this). However, I just see a standard CUDA OOM rather than something like this. > > In a previous issue I saw you mentioned using an RTX 4090 48GB. If that's the same situation here, you don't want to use `export TORCH_CUDA_ARCH_LIST="12.0+PTX"`, as SM120 is Consumer Blackwell. You probably want to set: > > ``` > export TORCH_CUDA_ARCH_LIST="8.9+PTX" > ``` Yes, I'm using 4090 48G, and also I changed `export TORCH_CUDA_ARCH_LIST="8.9+PTX"` With same exception.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#1235
No description provided.