mirror of
https://github.com/kvcache-ai/sglang.git
synced 2026-07-12 18:36:58 +00:00
6ac4f82e8f
* fix(kt-ep): match cpu_buf dtype to kt-kernel's bf16 scale write for MXFP4 kt-kernel's write_weights_to_buffer (operators/amx/fp4-moe.hpp) writes gate/up scales as bf16 via fast_fp32_to_bf16, but mxfp4_deepseek allocates w13/w2_weight_scale_inv as fp32. The 2x element-size mismatch caused kt-kernel to fill only the first half of cpu_buf in fp32-element terms; after Phase 3 .to(float8_e8m0fnu) the second half (= up_proj rows) became 2^-127, zeroing dequantized up_proj weights for all experts loaded via the kt double-buffered pipeline. Single-chunk GPU prefill on V4-Flash MXFP4 produced mode-collapsed garbage as a result. Allocate the cpu_buf with bf16 dtype for these two scale tensors so kt-kernel's write fills it exactly; gpu_t[e].copy_(cpu_buf[slot]) then performs the bf16->fp32 dtype cast automatically. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * feat(v4-2604b): apply SwiGLU clamp on triton-kernels GPU MoE path The trtllm and deep_gemm paths both apply a 2604B-specific asymmetric gate/up clamp (gate.clamp(max=limit); up.clamp(-limit, limit)) on the gemm1 output before silu_and_mul. The triton-kernels path (default GPU MoE on every capability outside _TRTLLM_FP4_CAPS, including SM_120 RTX 5090) was constructing a bare matmul_ogs → silu_and_mul → matmul_ogs sequence with no clamp, leaving routed-expert outputs numerically inconsistent with the trtllm reference on long-prompt / large-activation tokens. Threads moe_runner_config.swiglu_limit through DeepSeekMxfp4MoEMethod.apply to apply_v4_triton_kernels_moe; semantics match moe_runner/deep_gemm.py:_apply_swiglu_limit verbatim. No-op when submode != 2604B (swiglu_limit is None). Origin: sglang 本身. * feat(v4-2604b): pass swiglu_limit through KTEPWrapper to kt-kernel The kt CPU expert path was applying plain silu(g)*u with no clamp, diverging from the trtllm `gemm1_clamp_limit` and deep_gemm `_apply_swiglu_limit` references on long-prompt / large-activation tokens. Companion changes in kt-kernel (`feat/v4-2604b-swiglu-clamp:d10bd3d`) plumb a `swiglu_limit` field through `MOEConfig` into the AMX `act_fn`; this commit passes the value through the kt-sglang bridge. The KTMoEWrapper is constructed in `create_weights`, before `create_moe_runner` delivers the full `MoeRunnerConfig`, but the value is fully determined by SGLANG_DSV4_2604_SUBMODE which is fixed at process start, so we read the env directly here. Mirrors the `assert swiglu_limit == 10` in moe_runner/deep_gemm.py and the `torch.full(..., swiglu_limit, ...)` constructor in mxfp4_deepseek.py:177-186. Origin: kt-sglang 耦合. * fix(scheduler): correct inverted chunked_req check that hangs hybrid SWA chunked prefill In _get_new_batch_prefill_raw the inline comment explicitly says "Ignore the check if self.chunked_req is not None" but the code below used `is not None`, which is the opposite. With --disable-radix-cache + hybrid SWA + multi-chunk prompt, the chunked_req keeps holding its req_pool slot across chunks (ChunkCache.cache_unfinished_req does not release it), and ReqToTokenPool initialises free_slots = list(range(1, size)) wasting index 0, so once chunked_req takes the only available slot the check fires forever and the scheduler returns None on every iteration -> silent hang (chunk1 prefill completes, chunk2 never starts; TP CPU 60-145% busy spin; client request never returns). The sister check at line 2065 (`and self.chunked_req is None: return None`) is correctly inverted; this brings line 2082 in line with the comment and with that sister check. Repro (DeepSeek V4 Flash, hybrid SWA, page_size=256): --disable-radix-cache --chunked-prefill-size 2048 \ --tensor-parallel-size 4 --max-running-requests 2 + a prompt > 2048 tokens (forces multi-chunk) Before: chunk1 prefill runs, then silent hang or false-positive "token_to_kv_pool_allocator memory leak detected" SIGQUIT (the hybrid leak check is also too strict; addressed in a follow-up commit). After: 5001-token English prompt -> 3 chunks, HTTP 200 in 26.4s; 6695-token Chinese prompt -> 4 chunks, HTTP 200 in 52.2s. Origin: sglang itself (not kt-sglang coupling). Reproduces on pip- installed upstream sglang as well as on the kt third_party submodule. * fix(scheduler): skip self_check_during_idle when in-flight work still holds KV slots Defensive guard for the same bug class as the previous commit. When the scheduler enters the idle branch with chunked_req != None or a non-empty running_batch / waiting_queue, the in-flight KV slots are not yet freed nor cached. _check_hybrid_memory then reports them as leaked because its formula `full_num_used != 0` does not subtract protected_size / in-flight usage the way _check_radix_cache_memory does. The result was a SIGQUIT-on-false-positive: 4 TP ranks raise simultaneously and the server dies mid-request. The other branches of self_check_during_idle (DisaggregationMode.PREFILL and .DECODE) already early-return on similar in-flight conditions; this patch adds the equivalent guard for DisaggregationMode.NULL which had no such check. The same pattern is used at scheduler.py line 1372 and process_input_requests around line 1370. This guard is no longer load-bearing once the scheduler.py 2082 fix is in (chunked prefill advances every iter, the scheduler never reaches batch=None mid-request), but is kept as defence-in-depth against any future path that produces a double-None batch frame. Origin: sglang itself. --------- Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>