Three fixes for Kimi K2.5 RAWINT4 failing to start with CUDA graph:
1. fused_marlin_moe.py: Fix IndentationError from bad merge conflict
resolution — imports were left outside the `if _is_cuda:` block.
2. fused_marlin_moe.py: Add early return for E=0/M=0. When
kt-num-gpu-experts=0, GPU expert weights are empty tensors (E=0).
The marlin MoE kernel crashes on these empty inputs. Return zeros
so KT CPU experts can contribute the full result.
3. deepseek_v2.py: Skip dual-stream path for KT wrapper. The
forward_normal_dual_stream uses alt_stream for shared expert
parallelism, which conflicts with KT wrapper internal _cpu_stream
during CUDA graph capture.
Fixes#1866
- kt_ep_wrapper.py: normalize list-form moe_layer_freq to int
MiMo-V2-Flash uses per-layer mask [0,1,1,...] instead of int freq
- mimo_v2_flash.py: use getattr for pad_token_id (not in MiMo config)
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
_load_deepseek_v32_model() rewrites architectures from
DeepseekV32ForCausalLM to DeepseekV3ForCausalLM for transformers
compatibility, but the MLA detection list did not include
DeepseekV3ForCausalLM, causing use_mla_backend=False and
MHATokenToKVPool to be created instead of NSATokenToKVPool/MLATokenToKVPool.
DeepseekV32ForCausalLM was missing from the model_arch guard in
_handle_model_specific_adjustments(), so is_deepseek_nsa() was never
reached for V3.2 models. This caused the NSA attention backend to not
be auto-selected, leading to q_rope TypeError with flashinfer or
incorrect behavior with other backends.
Upstream bug introduced in sgl-project/sglang#13687 (commit 618ca2380)
which refactored the flat is_deepseek_nsa() check into a nested block
under model_arch guard but only listed DeepseekV3ForCausalLM.
During CUDA graph capture (regular or PCG), torch.cuda.synchronize()
and CPU-GPU expert coordination are not allowed. Detect capture mode
via is_in_piecewise_cuda_graph() and torch.cuda.is_current_stream_capturing(),
and delegate directly to the GPU method in those cases.
This enables running Qwen3.5 with --attention-backend triton without
--disable-cuda-graph, improving decode from ~11 tok/s to ~65 tok/s.
