[GH-ISSUE #2184] cache_salt/extra_key are computed & carried but never delivered to Req — radix-cache tenant isolation is a no-op (regression vs upstream sglang) #1299

New issue

Open

opened 2026-10-29 14:50:59 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 14:50:59 +00:00

Owner

Originally created by @wildoranges on GitHub (Sep 2, 2026).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2184

Reminder

I have read the above rules and searched the existing issues.

System Info

ktransformers : 0.7.0.post1 (main @ 948129159aa750075a0264aca16dba061cc6e6ae, 2026-08-28)
sglang        : sglang-kt (kvcache-ai/sglang @ 3bb39fca90ea8cd5d643866fdb6c8371ffc162de, "fix(glm5-next): accept MTP index sharing metadata")
                pip install -e python/ (from fork source), transformers-kt 5.6.0.post3
model         : Qwen/Qwen2.5-0.5B-Instruct (HF)
platform      : Ubuntu 22.04, kernel 6.8.0-136-generic
python        : 3.12.11 (conda env "sglang-kt"); torch 2.9.1+cu128; flashinfer 0.6.3
cpu           : 2 × AMD EPYC 7763 64-Core Processor (256 threads)
gpu           : NVIDIA GeForce RTX 3090 24GB ×8, driver 595.80 (CUDA 13.2), GPU 0 used
server        : python -m sglang.launch_server --model-path /home/cjb/kt-models/qwen25-05b \
                --host 0.0.0.0 --port 31000 --mem-fraction-static 0.6 --disable-cuda-graph \
                --enable-metrics --log-level info        (radix cache enabled by default, page_size=1, lru eviction)

Reproduction

Request sequence (OpenAI /v1/completions, same prompt, 5 semantically distinct cache_salt/extra_key contexts):

T1: {"prompt": P, "max_tokens":1, "temperature":0, "cache_salt":"tenant-alpha"}
T2: {"prompt": P, "max_tokens":1, "temperature":0, "cache_salt":"tenant-beta"}
T3: {"prompt": P, "max_tokens":1, "temperature":0, "extra_key":"tenant-gamma"}
T4: {"prompt": P, "max_tokens":1, "temperature":0, "cache_salt":"a", "extra_key":"bc"}
T5: {"prompt": P, "max_tokens":1, "temperature":0, "cache_salt":"ab", "extra_key":"c"}

(P is a fixed ~450-token text; the full JSON of P and all 5 requests is in the attached scripts.)

Expected: different (cache_salt, extra_key) contexts → distinct radix namespaces → no cross-hits (T2–T5 should recompute prefill after T1's cold start).

Actual (scheduler INFO log, per-request #cached-token):

T1  Prefill batch, #new-token: 457, #cached-token: 2    (cold start)
T2  Prefill batch, #new-token: 1,   #cached-token: 458  (full hit on T1's KV!)
T3  Prefill batch, #new-token: 1,   #cached-token: 458  (hit)
T4  Prefill batch, #new-token: 1,   #cached-token: 458  (hit)
T5  Prefill batch, #new-token: 1,   #cached-token: 458  (hit)

I.e. all 5 semantically unrelated contexts share a single KV-cache namespace — the cache_salt/extra_key isolation contract is a complete no-op at runtime.

Code-level break (fork @ 3bb39fca) — the feature is computed, carried layer by layer, then dropped when Req is constructed:

serving_base.py:146-157        _compute_extra_key() merges cache_salt+extra_key (called by serving_chat:318 / serving_completions:119 / serving_responses:283)
io_struct.py:208 / :724         GenerateReqInput.extra_key → TokenizedGenerateReqInput.extra_key
tokenizer_manager.py:964       extra_key=obj.extra_key
managers/scheduler.py:1504-1536  handle_generate_request constructs Req(...) WITHOUT extra_key=recv_req.extra_key (no read of recv_req.extra_key anywhere in the repo)
schedule_policy.py:194-236 / schedule_batch.py:938 / mem_cache/radix_cache.py   consumers intact, but always receive None

Upstream comparison (regression evidence): sgl-project/sglang main managers/scheduler.py:2549 passes extra_key=recv_req.extra_key at the same construction site (also :2863; :2464-2467 additionally uses extra_key for the elastic-EP namespace). In the fork, the serving layer, io_struct, the Req.__init__ parameter, and the radix-cache namespace implementation (radix_cache.py:357-364 docstring: "Isolate KV cache lines for different LoRA / adapter IDs … cache version, or retrieval augmentation context") are all retained — only the scheduler hand-off is missing.

Others

Impact scope (no over-claiming):

For plain-text models identical tokens ⇒ identical KV, so cross-context hits do not change outputs. Actual harm: the documented isolation contract is void (deployments that rely on salt semantics — custom logit processors, retrieval-augmented contexts, per-tenant state — get silently wrong outputs), plus a cross-tenant prefix-cache hit side channel (timing/metadata leakage).
LoRA isolation remains effective (lora_id is passed at scheduler.py:1513 and folded into the namespace by Req.__init__) — statically inferred, not GPU-tested this time.

Suggested fix:

Main fix (1 line, restores upstream behavior): add extra_key=recv_req.extra_key to the Req(...) call in scheduler.handle_generate_request (fix_extra_key_delivery.patch attached; py_compile verified; TokenizedGenerateReqInput.extra_key field already exists, io_struct:724).
Must be done together: the separator-less concatenation in serving_base.py:146-157 ("".join([cache_salt, extra_key])) and schedule_batch.py:632-638 ((extra_key or "") + lora_id) permits semantic collisions — (salt="a", ek="bc") ≡ (salt="ab", ek="c") ≡ (ek=<lora_id>) — which become exploitable the moment fix #1 lands (reproduced against the fork's real RadixCache in verify_cache_identity.py: CROSS-CONTEXT HIT). Recommend delimiter/hash encoding (length-prefixed or sha256).

Please confirm: is this a regression or intentionally disabled? (Upstream passes the same field at the same site; the fork has no visible git history marking a deliberate removal, so we cannot rule intent out ourselves.)

Originally created by @wildoranges on GitHub (Sep 2, 2026). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/2184 ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info ```text ktransformers : 0.7.0.post1 (main @ 948129159aa750075a0264aca16dba061cc6e6ae, 2026-08-28) sglang : sglang-kt (kvcache-ai/sglang @ 3bb39fca90ea8cd5d643866fdb6c8371ffc162de, "fix(glm5-next): accept MTP index sharing metadata") pip install -e python/ (from fork source), transformers-kt 5.6.0.post3 model : Qwen/Qwen2.5-0.5B-Instruct (HF) platform : Ubuntu 22.04, kernel 6.8.0-136-generic python : 3.12.11 (conda env "sglang-kt"); torch 2.9.1+cu128; flashinfer 0.6.3 cpu : 2 × AMD EPYC 7763 64-Core Processor (256 threads) gpu : NVIDIA GeForce RTX 3090 24GB ×8, driver 595.80 (CUDA 13.2), GPU 0 used server : python -m sglang.launch_server --model-path /home/cjb/kt-models/qwen25-05b \ --host 0.0.0.0 --port 31000 --mem-fraction-static 0.6 --disable-cuda-graph \ --enable-metrics --log-level info (radix cache enabled by default, page_size=1, lru eviction) ``` ### Reproduction Request sequence (OpenAI `/v1/completions`, same prompt, **5 semantically distinct `cache_salt`/`extra_key` contexts**): ```text T1: {"prompt": P, "max_tokens":1, "temperature":0, "cache_salt":"tenant-alpha"} T2: {"prompt": P, "max_tokens":1, "temperature":0, "cache_salt":"tenant-beta"} T3: {"prompt": P, "max_tokens":1, "temperature":0, "extra_key":"tenant-gamma"} T4: {"prompt": P, "max_tokens":1, "temperature":0, "cache_salt":"a", "extra_key":"bc"} T5: {"prompt": P, "max_tokens":1, "temperature":0, "cache_salt":"ab", "extra_key":"c"} ``` (P is a fixed ~450-token text; the full JSON of P and all 5 requests is in the attached scripts.) Expected: different `(cache_salt, extra_key)` contexts → distinct radix namespaces → no cross-hits (T2–T5 should recompute prefill after T1's cold start). Actual (scheduler INFO log, per-request `#cached-token`): ```text T1 Prefill batch, #new-token: 457, #cached-token: 2 (cold start) T2 Prefill batch, #new-token: 1, #cached-token: 458 (full hit on T1's KV!) T3 Prefill batch, #new-token: 1, #cached-token: 458 (hit) T4 Prefill batch, #new-token: 1, #cached-token: 458 (hit) T5 Prefill batch, #new-token: 1, #cached-token: 458 (hit) ``` I.e. **all 5 semantically unrelated contexts share a single KV-cache namespace** — the `cache_salt`/`extra_key` isolation contract is a complete no-op at runtime. Code-level break (fork @ 3bb39fca) — the feature is computed, carried layer by layer, then **dropped when `Req` is constructed**: ```text serving_base.py:146-157 _compute_extra_key() merges cache_salt+extra_key (called by serving_chat:318 / serving_completions:119 / serving_responses:283) io_struct.py:208 / :724 GenerateReqInput.extra_key → TokenizedGenerateReqInput.extra_key tokenizer_manager.py:964 extra_key=obj.extra_key managers/scheduler.py:1504-1536 handle_generate_request constructs Req(...) WITHOUT extra_key=recv_req.extra_key (no read of recv_req.extra_key anywhere in the repo) schedule_policy.py:194-236 / schedule_batch.py:938 / mem_cache/radix_cache.py consumers intact, but always receive None ``` Upstream comparison (regression evidence): sgl-project/sglang main `managers/scheduler.py:2549` passes `extra_key=recv_req.extra_key` at **the same construction site** (also :2863; :2464-2467 additionally uses extra_key for the elastic-EP namespace). In the fork, the serving layer, io_struct, the `Req.__init__` parameter, and the radix-cache namespace implementation (radix_cache.py:357-364 docstring: "Isolate KV cache lines for different LoRA / adapter IDs … cache version, or retrieval augmentation context") are all retained — only the scheduler hand-off is missing. ### Others **Impact scope** (no over-claiming): - For plain-text models identical tokens ⇒ identical KV, so cross-context hits do not change outputs. Actual harm: the documented isolation contract is void (deployments that rely on salt semantics — custom logit processors, retrieval-augmented contexts, per-tenant state — get silently wrong outputs), plus a cross-tenant prefix-cache hit side channel (timing/metadata leakage). - LoRA isolation remains effective (lora_id is passed at scheduler.py:1513 and folded into the namespace by `Req.__init__`) — statically inferred, not GPU-tested this time. **Suggested fix**: 1. Main fix (1 line, restores upstream behavior): add `extra_key=recv_req.extra_key` to the `Req(...)` call in `scheduler.handle_generate_request` (`fix_extra_key_delivery.patch` attached; py_compile verified; `TokenizedGenerateReqInput.extra_key` field already exists, io_struct:724). 2. **Must be done together**: the separator-less concatenation in serving_base.py:146-157 (`"".join([cache_salt, extra_key])`) and schedule_batch.py:632-638 (`(extra_key or "") + lora_id`) permits semantic collisions — `(salt="a", ek="bc") ≡ (salt="ab", ek="c") ≡ (ek=<lora_id>)` — which become exploitable the moment fix #1 lands (reproduced against the fork's real RadixCache in `verify_cache_identity.py`: CROSS-CONTEXT HIT). Recommend delimiter/hash encoding (length-prefixed or sha256). **Please confirm**: is this a regression or intentionally disabled? (Upstream passes the same field at the same site; the fork has no visible git history marking a deliberate removal, so we cannot rule intent out ourselves.)

bsakaguchi commented

2026-10-29 14:51:00 +00:00

Author

Owner

@UniversePeak commented on GitHub (Sep 2, 2026):

I traced the current vendored SGLang path and found a small regression harness that can make this contract durable: reuse the existing radix-cache isolation test for same-key hits and cross-tenant misses, then add a focused request-construction check that requires handle_generate_request to pass recv_req.extra_key into Req. The latter currently fails on main, which confirms the delivery gap rather than assuming the radix cache is at fault. Because SGLang is pinned as a submodule here, would you prefer this regression to land in the vendored SGLang test suite first, or should KTransformers carry a parent-level contract test alongside the coordinated submodule fix?

@UniversePeak commented on GitHub (Sep 2, 2026): I traced the current vendored SGLang path and found a small regression harness that can make this contract durable: reuse the existing radix-cache isolation test for same-key hits and cross-tenant misses, then add a focused request-construction check that requires `handle_generate_request` to pass `recv_req.extra_key` into `Req`. The latter currently fails on `main`, which confirms the delivery gap rather than assuming the radix cache is at fault. Because SGLang is pinned as a submodule here, would you prefer this regression to land in the vendored SGLang test suite first, or should KTransformers carry a parent-level contract test alongside the coordinated submodule fix?

bsakaguchi commented

2026-10-29 14:51:00 +00:00

Author

Owner

@wildoranges commented on GitHub (Sep 2, 2026):

@UniversePeak Thanks for tracing this. My preference is to land the focused regression in the vendored SGLang test suite alongside https://github.com/kvcache-ai/sglang/pull/89, since the delivery gap is in the SGLang submodule.

The PR already includes the RadixCache isolation and scheduler/session propagation tests. The manual collision-test baseline issue was fixed in github.com/kvcache-ai/sglang@f29d63fd39.

A parent-level KTransformers contract test can be added later if an integration-only check is still needed after the submodule is updated.

@wildoranges commented on GitHub (Sep 2, 2026): @UniversePeak Thanks for tracing this. My preference is to land the focused regression in the vendored SGLang test suite alongside https://github.com/kvcache-ai/sglang/pull/89, since the delivery gap is in the SGLang submodule. The PR already includes the RadixCache isolation and scheduler/session propagation tests. The manual collision-test baseline issue was fixed in https://github.com/kvcache-ai/sglang/commit/f29d63fd39d9caee538f6c7ccdadca7b769701fd. A parent-level KTransformers contract test can be added later if an integration-only check is still needed after the submodule is updated.