kvcache-ai/ktransformers
1
0
Fork 0
mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00
Code Issues 469 Projects Packages Wiki Activity

[GH-ISSUE #1705] W4A16 GPU model doesn't work [Qwen3-30B-A3B] #1088

New issue
Open
opened 2026-10-29 14:48:22 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-29 14:48:22 +00:00
Owner
Copy link

Originally created by @xin3he on GitHub (Dec 11, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1705

Originally assigned to: @SkqLiao, @ovowei on GitHub.

Reminder

  • I have read the above rules and searched the existing issues.

System Info

Docker: nvcr.io/nvidia/pytorch:25.08-py3
kt-kernel 0.1.0
sglang 0.5.5.post3 /home/xinhe/sglang/python
sgl-kernel 0.3.18.post1

Reproduction

BF16 GPU model works well, GPU and CPU models are generated as below and loaded with SGLang.

# CPU model
python scripts/convert_cpu_weights.py \
  --input-path /home/xinhe/Qwen3-30B-A3B \
  --input-type bf16 \
  --output /home/xinhe/Qwen3-30B-A3B-CPU-INT4-KT \
  --quant-method int4

# GPU model
python scripts/convert_gpu_weights.py \
  --model_id /home/xinhe/Qwen3-30B-A3B \
  --output_dir /home/xinhe/Qwen3-30B-A3B-GPU-INT4-KT \
  --quant_method RTN \
  --quant_type W4A16

# Load with SGLang
python -m sglang.launch_server \
  --model /home/xinhe/Qwen3-30B-A3B-GPU-INT4-KT \
  --port 8802 \
  --kt-method AMXINT4 \
  --kt-weight-path /home/xinhe/Qwen3-30B-A3B-CPU-INT4-KT \
  --kt-cpuinfer 64 \
  --kt-threadpool-count 2 \
  --kt-num-gpu-experts 32 \
  --kt-max-deferred-experts-per-token 2 

...
TP MOE layer 45, pool: 0x4cb39520, expert num: 128, num_experts_per_tok: 8                                                       
Creating AMX_MOE_TP 0 at numa 0                                                                                                  
Creating AMX_MOE_TP 1 at numa 1                                                                                                  
TP Load from loader                                                                                                              
TP MOE layer 46, pool: 0x4cb39520, expert num: 128, num_experts_per_tok: 8                                                       
Creating AMX_MOE_TP 0 at numa 0                                                                                                  
Creating AMX_MOE_TP 1 at numa 1                                                                                                  
TP Load from loader                                                                                                              
TP MOE layer 47, pool: 0x4cb39520, expert num: 128, num_experts_per_tok: 8                                                       
Creating AMX_MOE_TP 0 at numa 0                                                                                                  
Creating AMX_MOE_TP 1 at numa 1                                                                                                  
TP Load from loader                                                                                            06:33:00 [90/1947]
[2025-12-11 06:32:58] Load weight end. type=Qwen3MoeForCausalLM, dtype=torch.bfloat16, avail mem=168.33 GB, mem usage=9.29 GB.   
[2025-12-11 06:32:58] Using KV cache dtype: torch.bfloat16                                                                       
[2025-12-11 06:32:58] KV Cache is allocated. #tokens: 1560985, K size: 71.46 GB, V size: 71.46 GB                                
[2025-12-11 06:32:58] Memory pool end. avail mem=24.64 GB                                                                        
[2025-12-11 06:32:59] Capture cuda graph begin. This can take up to several minutes. avail mem=24.08 GB                          
[2025-12-11 06:32:59] Capture cuda graph bs [1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136,
 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432
, 448, 464, 480, 496, 512]                                                                                                       
Capturing batches (bs=512 avail_mem=23.69 GB):   0%|                                                      | 0/52 [00:00<?, ?it/s]
[2025-12-11 06:33:00] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /home/xinhe/sgl
ang/python/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_5_1/E=32,N=4096,device_name=NVIDIA_B200.json, you can create t
hem with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton                                      
Capturing batches (bs=512 avail_mem=23.69 GB):   0%|                                                      | 0/52 [00:01<?, ?it/s]
[2025-12-11 06:33:00] Scheduler hit an exception: Traceback (most recent call last):                                             
  File "/home/xinhe/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 344, in __init__                         
    self.capture()                                                                                                               
  File "/home/xinhe/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 502, in capture                          
    _capture_one_stream()                                                                                                        
  File "/home/xinhe/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 486, in _capture_one_stream              
    ) = self.capture_one_batch_size(bs, forward, stream_idx)                                                                     
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                     
  File "/home/xinhe/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 693, in capture_one_batch_size           
    run_once()                                                                                                                   
  File "/home/xinhe/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 680, in run_once                         
    logits_output_or_pp_proxy_tensors = forward(                                                                                 
                                        ^^^^^^^^                                                                                 
  File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context                       
    return func(*args, **kwargs)                                                                                                 
           ^^^^^^^^^^^^^^^^^^^^^                                                                                                 
  File "/home/xinhe/sglang/python/sglang/srt/models/qwen3_moe.py", line 755, in forward                                          
    hidden_states = self.model(                                                                                                  
                    ^^^^^^^^^^^                                                                                                  
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl                    
    return self._call_impl(*args, **kwargs)                                                                                      
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                      
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl                            
    return forward_call(*args, **kwargs)                                                                                         
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                         
  File "/home/xinhe/sglang/python/sglang/srt/models/qwen2_moe.py", line 608, in forward                                          
    hidden_states, residual = layer(                                                                                             
                              ^^^^^^                                                                                             
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl                    
    return self._call_impl(*args, **kwargs)                                                                                      
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl          06:33:00 [45/1947]
    return forward_call(*args, **kwargs)                                                                                         
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                         
  File "/home/xinhe/sglang/python/sglang/srt/models/qwen3_moe.py", line 628, in forward                                          
    hidden_states = self.mlp(                                                                                                    
                    ^^^^^^^^^                                                                                                    
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl                    
    return self._call_impl(*args, **kwargs)                                                                                      
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                      
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl                            
    return forward_call(*args, **kwargs)                                                                                         
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                         
  File "/home/xinhe/sglang/python/sglang/srt/models/qwen3_moe.py", line 151, in forward                                          
    return self.forward_normal(                                                                                                  
           ^^^^^^^^^^^^^^^^^^^^                                                                                                  
  File "/home/xinhe/sglang/python/sglang/srt/models/qwen3_moe.py", line 176, in forward_normal                                   
    final_hidden_states = self.experts(hidden_states, topk_output)                                                               
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                               
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl                    
    return self._call_impl(*args, **kwargs)                                                                                      
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                      
  File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl                            
    return forward_call(*args, **kwargs)                                                                                         
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                                                         
  File "/home/xinhe/sglang/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 894, in forward                         
    combine_input = self.run_moe_core(                                                                                           
                    ^^^^^^^^^^^^^^^^^^                                                                                           
  File "/home/xinhe/sglang/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 915, in run_moe_core                    
    return self.quant_method.apply(                                                                                              
           ^^^^^^^^^^^^^^^^^^^^^^^^                                                                                              
  File "/home/xinhe/sglang/python/sglang/srt/layers/moe/kt_ep_wrapper.py", line 365, in apply                                    
    gpu_combine_input = self.gpu_method.apply(layer, masked_dispatch_output)                                                     
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^                                                     
  File "/home/xinhe/sglang/python/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors_moe.py", line 637, in appl
y                                                                                                                                
    output = fused_marlin_moe(                                                                                                   
             ^^^^^^^^^^^^^^^^^                                                                                                   
  File "/home/xinhe/sglang/python/sglang/srt/layers/moe/fused_moe_triton/fused_marlin_moe.py", line 213, in fused_marlin_moe     
    torch.sum(intermediate_cache3.view(*intermediate_cache3.shape), dim=1, out=output)                                           
torch.AcceleratorError: CUDA error: an illegal memory access was encountered                                                     
Search for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more informat
ion.                                                                                                                             
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.          
For debugging consider passing CUDA_LAUNCH_BLOCKING=1                                                                            
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. 

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/xinhe/sglang/python/sglang/srt/managers/scheduler.py", line 2628, in run_scheduler_process
    scheduler = Scheduler(
                ^^^^^^^^^^
  File "/home/xinhe/sglang/python/sglang/srt/managers/scheduler.py", line 314, in __init__
    self.tp_worker = TpModelWorker(
                     ^^^^^^^^^^^^^^
  File "/home/xinhe/sglang/python/sglang/srt/managers/tp_worker.py", line 245, in __init__
    self._model_runner = ModelRunner(
                         ^^^^^^^^^^^^
  File "/home/xinhe/sglang/python/sglang/srt/model_executor/model_runner.py", line 341, in __init__
    self.initialize(min_per_gpu_memory)
  File "/home/xinhe/sglang/python/sglang/srt/model_executor/model_runner.py", line 531, in initialize
    self.init_device_graphs()
  File "/home/xinhe/sglang/python/sglang/srt/model_executor/model_runner.py", line 2165, in init_device_graphs
    self.graph_runner = graph_runners[self.device](self)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/xinhe/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 346, in __init__
    raise Exception(
Exception: Capture cuda graph failed: CUDA error: an illegal memory access was encountered
Search for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information.
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.

Possible solutions:
1. set --mem-fraction-static to a smaller value (e.g., 0.8 or 0.7)
2. set --cuda-graph-max-bs to a smaller value (e.g., 16)
3. disable torch compile by not using --enable-torch-compile
4. disable CUDA graph by --disable-cuda-graph. (Not recommended. Huge performance loss)
Open an issue on GitHub https://github.com/sgl-project/sglang/issues/new/choose 


[2025-12-11 06:33:00] Received sigquit from a child process. It usually means the child failed.
Killed

Others

No response

Originally created by @xin3he on GitHub (Dec 11, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1705 Originally assigned to: @SkqLiao, @ovowei on GitHub. ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info Docker: nvcr.io/nvidia/pytorch:25.08-py3 kt-kernel 0.1.0 sglang 0.5.5.post3 /home/xinhe/sglang/python sgl-kernel 0.3.18.post1 ### Reproduction BF16 GPU model works well, GPU and CPU models are generated as below and loaded with SGLang. ```bash # CPU model python scripts/convert_cpu_weights.py \ --input-path /home/xinhe/Qwen3-30B-A3B \ --input-type bf16 \ --output /home/xinhe/Qwen3-30B-A3B-CPU-INT4-KT \ --quant-method int4 # GPU model python scripts/convert_gpu_weights.py \ --model_id /home/xinhe/Qwen3-30B-A3B \ --output_dir /home/xinhe/Qwen3-30B-A3B-GPU-INT4-KT \ --quant_method RTN \ --quant_type W4A16 # Load with SGLang python -m sglang.launch_server \ --model /home/xinhe/Qwen3-30B-A3B-GPU-INT4-KT \ --port 8802 \ --kt-method AMXINT4 \ --kt-weight-path /home/xinhe/Qwen3-30B-A3B-CPU-INT4-KT \ --kt-cpuinfer 64 \ --kt-threadpool-count 2 \ --kt-num-gpu-experts 32 \ --kt-max-deferred-experts-per-token 2 ``` ```log ... TP MOE layer 45, pool: 0x4cb39520, expert num: 128, num_experts_per_tok: 8 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 46, pool: 0x4cb39520, expert num: 128, num_experts_per_tok: 8 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader TP MOE layer 47, pool: 0x4cb39520, expert num: 128, num_experts_per_tok: 8 Creating AMX_MOE_TP 0 at numa 0 Creating AMX_MOE_TP 1 at numa 1 TP Load from loader 06:33:00 [90/1947] [2025-12-11 06:32:58] Load weight end. type=Qwen3MoeForCausalLM, dtype=torch.bfloat16, avail mem=168.33 GB, mem usage=9.29 GB. [2025-12-11 06:32:58] Using KV cache dtype: torch.bfloat16 [2025-12-11 06:32:58] KV Cache is allocated. #tokens: 1560985, K size: 71.46 GB, V size: 71.46 GB [2025-12-11 06:32:58] Memory pool end. avail mem=24.64 GB [2025-12-11 06:32:59] Capture cuda graph begin. This can take up to several minutes. avail mem=24.08 GB [2025-12-11 06:32:59] Capture cuda graph bs [1, 2, 4, 8, 12, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432 , 448, 464, 480, 496, 512] Capturing batches (bs=512 avail_mem=23.69 GB): 0%| | 0/52 [00:00<?, ?it/s] [2025-12-11 06:33:00] Using default MoE kernel config. Performance might be sub-optimal! Config file not found at /home/xinhe/sgl ang/python/sglang/srt/layers/moe/fused_moe_triton/configs/triton_3_5_1/E=32,N=4096,device_name=NVIDIA_B200.json, you can create t hem with https://github.com/sgl-project/sglang/tree/main/benchmark/kernels/fused_moe_triton Capturing batches (bs=512 avail_mem=23.69 GB): 0%| | 0/52 [00:01<?, ?it/s] [2025-12-11 06:33:00] Scheduler hit an exception: Traceback (most recent call last): File "/home/xinhe/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 344, in __init__ self.capture() File "/home/xinhe/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 502, in capture _capture_one_stream() File "/home/xinhe/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 486, in _capture_one_stream ) = self.capture_one_batch_size(bs, forward, stream_idx) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/xinhe/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 693, in capture_one_batch_size run_once() File "/home/xinhe/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 680, in run_once logits_output_or_pp_proxy_tensors = forward( ^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/utils/_contextlib.py", line 120, in decorate_context return func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/xinhe/sglang/python/sglang/srt/models/qwen3_moe.py", line 755, in forward hidden_states = self.model( ^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/xinhe/sglang/python/sglang/srt/models/qwen2_moe.py", line 608, in forward hidden_states, residual = layer( ^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl 06:33:00 [45/1947] return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/xinhe/sglang/python/sglang/srt/models/qwen3_moe.py", line 628, in forward hidden_states = self.mlp( ^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/xinhe/sglang/python/sglang/srt/models/qwen3_moe.py", line 151, in forward return self.forward_normal( ^^^^^^^^^^^^^^^^^^^^ File "/home/xinhe/sglang/python/sglang/srt/models/qwen3_moe.py", line 176, in forward_normal final_hidden_states = self.experts(hidden_states, topk_output) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.12/dist-packages/torch/nn/modules/module.py", line 1786, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/xinhe/sglang/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 894, in forward combine_input = self.run_moe_core( ^^^^^^^^^^^^^^^^^^ File "/home/xinhe/sglang/python/sglang/srt/layers/moe/fused_moe_triton/layer.py", line 915, in run_moe_core return self.quant_method.apply( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/xinhe/sglang/python/sglang/srt/layers/moe/kt_ep_wrapper.py", line 365, in apply gpu_combine_input = self.gpu_method.apply(layer, masked_dispatch_output) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/xinhe/sglang/python/sglang/srt/layers/quantization/compressed_tensors/compressed_tensors_moe.py", line 637, in appl y output = fused_marlin_moe( ^^^^^^^^^^^^^^^^^ File "/home/xinhe/sglang/python/sglang/srt/layers/moe/fused_moe_triton/fused_marlin_moe.py", line 213, in fused_marlin_moe torch.sum(intermediate_cache3.view(*intermediate_cache3.shape), dim=1, out=output) torch.AcceleratorError: CUDA error: an illegal memory access was encountered Search for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more informat ion. CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/xinhe/sglang/python/sglang/srt/managers/scheduler.py", line 2628, in run_scheduler_process scheduler = Scheduler( ^^^^^^^^^^ File "/home/xinhe/sglang/python/sglang/srt/managers/scheduler.py", line 314, in __init__ self.tp_worker = TpModelWorker( ^^^^^^^^^^^^^^ File "/home/xinhe/sglang/python/sglang/srt/managers/tp_worker.py", line 245, in __init__ self._model_runner = ModelRunner( ^^^^^^^^^^^^ File "/home/xinhe/sglang/python/sglang/srt/model_executor/model_runner.py", line 341, in __init__ self.initialize(min_per_gpu_memory) File "/home/xinhe/sglang/python/sglang/srt/model_executor/model_runner.py", line 531, in initialize self.init_device_graphs() File "/home/xinhe/sglang/python/sglang/srt/model_executor/model_runner.py", line 2165, in init_device_graphs self.graph_runner = graph_runners[self.device](self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/xinhe/sglang/python/sglang/srt/model_executor/cuda_graph_runner.py", line 346, in __init__ raise Exception( Exception: Capture cuda graph failed: CUDA error: an illegal memory access was encountered Search for `cudaErrorIllegalAddress' in https://docs.nvidia.com/cuda/cuda-runtime-api/group__CUDART__TYPES.html for more information. CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect. For debugging consider passing CUDA_LAUNCH_BLOCKING=1 Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions. Possible solutions: 1. set --mem-fraction-static to a smaller value (e.g., 0.8 or 0.7) 2. set --cuda-graph-max-bs to a smaller value (e.g., 16) 3. disable torch compile by not using --enable-torch-compile 4. disable CUDA graph by --disable-cuda-graph. (Not recommended. Huge performance loss) Open an issue on GitHub https://github.com/sgl-project/sglang/issues/new/choose [2025-12-11 06:33:00] Received sigquit from a child process. It usually means the child failed. Killed ``` ### Others _No response_
bsakaguchi commented 2026-10-29 14:48:22 +00:00
Author
Owner
Copy link

@xin3he commented on GitHub (Dec 11, 2025):

@SkqLiao This is related with AutoRound Enabling. I intend to load AutoRound W4A16 model as GPU models.

However, I got failure with W4A16 GPU model following the readme in kt-kernel/scripts, haven't used AutoRound format.

<!-- gh-comment-id:3640481261 --> @xin3he commented on GitHub (Dec 11, 2025): @SkqLiao This is related with AutoRound Enabling. I intend to load AutoRound W4A16 model as GPU models. However, I got failure with W4A16 GPU model following the readme in [kt-kernel/scripts](https://github.com/kvcache-ai/ktransformers/tree/main/kt-kernel/scripts), haven't used AutoRound format.
bsakaguchi commented 2026-10-29 14:48:23 +00:00
Author
Owner
Copy link

@xin3he commented on GitHub (Dec 22, 2025):

@SkqLiao, @ovowei Do you have any suggestions to make it work properly?

<!-- gh-comment-id:3680495969 --> @xin3he commented on GitHub (Dec 22, 2025): @SkqLiao, @ovowei Do you have any suggestions to make it work properly?
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
auto/sync-sglang
gh-pages
docs/kt-ft-figure2-single-column-20261010
perf/bf16-grad-norm-20261010
docs/kt-ft-paper-20261008
release/20260911-carrier-repair
release/20260911-ci-python
release/20260911-pypi
ci/four-main-pypi-release
ci/model-e2e-acceptance
release/0.7.0.post3-frozen-pypi
fix/glm53-index-share-sync
release/0.7.0.post2-frozen-pypi
hotfix/0.7.0.post2-pypi-runtime
fix/avx2-wheel-feature-isolation
fix/release-self-hosted-runner
release/automate-kt-sft-pypi
support-glm5-next-djw
docs/deepseek-v4-flash-docker
ovowei-patch-3
support-glm52-djw
doc-reorg-entrypoints-support-matrix
bump-sglang-pr31
docs-v061-refresh
fp4-moe-amx
flatten-ktransformers-shim
archive-kt-sft
kt-kernel-torch-range-v061
sft-whl
sft
docs/gosim-2026-and-roadmap-link
ovowei-patch-2
ovowei-patch-1
revert-1918-fix/rocm-cpuinfer-stream-bridge
chore/kt-layerwise-prefill-main
chore/kt-layerwise-prefill-label
sft-rel
support-glm51-djw
release/v0.5.3
chore/kt-run-numa-nodes-unified
mtp
ksft-sglang
avx2
fix/sglang-kt-detection
layerwise-prefill-opt
feat/mistral-loader-support
fix-docker-build
minimax-m2.5
qwen3.5
add-maintainers
oql-doc
cli-dev-ljq
ljq-cli-dev-new
JimmyPeilinLi-patch-2
kimi_k2.5
oql-dev
experts_sched
JimmyPeilinLi-patch-1
ErvinXie-patch-1
ksft-llama
glm47_fp8
debug_llama
refactor_sft
ljq-cli-dev
refactor_sft_int4_debug
lut-nvfp4
fix-import
fix-deploy-djw
cpu-fix
ljq-dev-1
route-moe
sosp25-ae
v0.7.2
paper-ktft-v1.1
paper-ktft-v1
v0.7.1
v0.7.0.post4
v0.7.0.post1
v0.7.0
v0.6.4
v0.6.3.post1
v0.6.3
v0.6.2.post3
v0.6.2.post2
v0.6.2.post1
v0.6.2
v0.6.1
v0.6.1.post1
v0.5.3
v0.5.2.post1
v0.5.1
v0.5.0.post1
v0.5.0
v0.4.4
v0.4.3
v0.4.2
v0.4.1
v0.3.2
v0.3.1
v0.3
v0.2.4post1
v0.2.4
v0.2.3post2
v0.2.3post1
v0.2.3
v0.2.2
v0.2.2rc2
v0.2.2rc1
v0.2.1.post1
v0.2.1
v0.2.0
v0.1.4
v0.1.3
v0.1.2
v0.1.1
v0.1.0
Labels
Clear labels   
bug

   
duplicate

   
enhancement

   
good first issue

   
help wanted

   
pull-request
Mirrored from GitHub Pull Request

  
stale
No labels
bug
duplicate
enhancement
good first issue
help wanted
pull-request
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/ktransformers#1088
No description provided.