kvcache-ai/sglang

Fork 0

mirror of https://github.com/kvcache-ai/sglang.git synced 2026-11-05 01:07:39 +00:00

[PR #89] fix: restore radix cache namespace isolation #89

New issue

Open

opened 2026-10-29 21:07:06 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:07:06 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/sglang/pull/89
Author: @wildoranges
Created: 9/2/2026
Status: 🔄 Open

Base: main ← Head: kt-main-latest

📝 Commits (2)

fa13d5e fix: restore radix cache namespace isolation
f29d63f test: reset cache between identity collision cases

📊 Changes

8 files changed (+281 additions, -18 deletions)

View changed files

➕ python/sglang/srt/cache_identity.py (+33 -0)
📝 python/sglang/srt/disaggregation/encode_receiver.py (+1 -0)
📝 python/sglang/srt/entrypoints/openai/serving_base.py (+7 -11)
📝 python/sglang/srt/managers/schedule_batch.py (+5 -7)
📝 python/sglang/srt/managers/scheduler.py (+1 -0)
📝 python/sglang/srt/managers/session_controller.py (+1 -0)
📝 test/manual/openai_server/features/test_cache_report.py (+76 -0)
➕ test/unit/test_extra_key_cache_isolation.py (+157 -0)

📄 Description

Motivation

Fixes https://github.com/kvcache-ai/ktransformers/issues/2184.

The OpenAI serving layer computes a per-request cache namespace from
cache_salt and extra_key, and the request structs carry that value to the
scheduler. However, the normal scheduler path did not pass the value into
Req, so caller-provided cache namespaces were dropped before radix-cache
lookup. Session requests and the asynchronous multimodal receiver had the same
problem when constructing a new Req.

As a result, affected requests with different cache_salt or extra_key
values could reuse each other's KV cache entries. A pre-patch reproduction with
Qwen2.5-0.5B on an 8x RTX 3090 server showed the same full cache hit
(#cached-token: 458) for five semantically distinct cache contexts, while the
cold request prefetched 457 new tokens. The reproduction log is attached to
the linked issue.

Simply propagating the existing value would also expose two ambiguous string
concatenations: (cache_salt, extra_key) in the OpenAI layer and
(extra_key, lora_id) in Req. Different component tuples could therefore
produce the same radix-cache key.

Modifications

Add a canonical cache-identity encoder with an explicit version, domain,
component names, and null positions.
Encode OpenAI (cache_salt, extra_key) identities without delimiter or
field-position collisions.
Propagate extra_key through normal scheduler requests, session requests,
and asynchronous multimodal receiver requests.
Encode the final (extra_key, lora_id) identity in Req, preserving LoRA
isolation without ambiguous concatenation.
Add unit coverage for canonical encoding, previously colliding identities,
the OpenAIServingBase -> Req -> RadixCache path, and scheduler/session
propagation.
Add manual server regressions for field-position collisions, concatenation
collisions, and native session extra_key isolation.

The encoded namespace is internal and opaque; no request or response schema is
changed. Requests that do not provide a cache namespace or LoRA adapter retain
None as their radix-cache extra_key.

Accuracy Tests

This change does not modify model kernels, forward computation, sampling, or
generated-token selection. It only controls which radix-cache namespace is
eligible for prefix reuse, so model accuracy evaluation is not applicable.

python -m pytest -q test/unit/test_extra_key_cache_isolation.py: 4 passed.
python -m pytest -q test/registered/radix_cache/test_radix_cache_unit.py::TestRadixCache::test_extra_key_isolation: 1 passed.
Server E2E regressions were added to
test/manual/openai_server/features/test_cache_report.py, but were not run
locally because this workspace does not have the GPU/model-server
environment required by that test module.

Benchmarking and Profiling

No inference benchmark was run. The change adds deterministic JSON encoding on
the CPU request-control path when a cache namespace or LoRA adapter is present;
it does not alter GPU inference kernels. No material throughput impact is
expected, but this has not been measured.

Checklist

Format your code according to the Format code with pre-commit. Black, isort, Ruff (F401,F821), py_compile, and git diff --check passed for the changed files.
Add unit tests according to the Run and add unit tests.
Update documentation according to Write documentations. No documentation update is required because this restores the documented cache-isolation behavior without changing the API.
Provide accuracy and speed benchmark results according to Test the accuracy and Benchmark the speed. Accuracy testing is not applicable, and a performance benchmark was not run; see the sections above.
Follow the SGLang code style guidance.

Review Process

Ping Merge Oncalls to start the PR flow. See the PR Merge Process.
Get approvals from CODEOWNERS and other reviewers.
Trigger CI tests with comments or contact authorized users to do so.
- /tag-run-ci-label, /rerun-failed-ci, /tag-and-rerun-ci
After green CI and required approvals, ask Merge Oncalls to merge.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/sglang/pull/89 **Author:** [@wildoranges](https://github.com/wildoranges) **Created:** 9/2/2026 **Status:** 🔄 Open **Base:** `main` ← **Head:** `kt-main-latest` --- ### 📝 Commits (2) - [`fa13d5e`](https://github.com/kvcache-ai/sglang/commit/fa13d5e73402286b582472eba18c299be941bec7) fix: restore radix cache namespace isolation - [`f29d63f`](https://github.com/kvcache-ai/sglang/commit/f29d63fd39d9caee538f6c7ccdadca7b769701fd) test: reset cache between identity collision cases ### 📊 Changes **8 files changed** (+281 additions, -18 deletions) <details> <summary>View changed files</summary> ➕ `python/sglang/srt/cache_identity.py` (+33 -0) 📝 `python/sglang/srt/disaggregation/encode_receiver.py` (+1 -0) 📝 `python/sglang/srt/entrypoints/openai/serving_base.py` (+7 -11) 📝 `python/sglang/srt/managers/schedule_batch.py` (+5 -7) 📝 `python/sglang/srt/managers/scheduler.py` (+1 -0) 📝 `python/sglang/srt/managers/session_controller.py` (+1 -0) 📝 `test/manual/openai_server/features/test_cache_report.py` (+76 -0) ➕ `test/unit/test_extra_key_cache_isolation.py` (+157 -0) </details> ### 📄 Description ## Motivation Fixes https://github.com/kvcache-ai/ktransformers/issues/2184. The OpenAI serving layer computes a per-request cache namespace from `cache_salt` and `extra_key`, and the request structs carry that value to the scheduler. However, the normal scheduler path did not pass the value into `Req`, so caller-provided cache namespaces were dropped before radix-cache lookup. Session requests and the asynchronous multimodal receiver had the same problem when constructing a new `Req`. As a result, affected requests with different `cache_salt` or `extra_key` values could reuse each other's KV cache entries. A pre-patch reproduction with Qwen2.5-0.5B on an 8x RTX 3090 server showed the same full cache hit (`#cached-token: 458`) for five semantically distinct cache contexts, while the cold request prefetched 457 new tokens. The reproduction log is attached to the linked issue. Simply propagating the existing value would also expose two ambiguous string concatenations: `(cache_salt, extra_key)` in the OpenAI layer and `(extra_key, lora_id)` in `Req`. Different component tuples could therefore produce the same radix-cache key. ## Modifications - Add a canonical cache-identity encoder with an explicit version, domain, component names, and null positions. - Encode OpenAI `(cache_salt, extra_key)` identities without delimiter or field-position collisions. - Propagate `extra_key` through normal scheduler requests, session requests, and asynchronous multimodal receiver requests. - Encode the final `(extra_key, lora_id)` identity in `Req`, preserving LoRA isolation without ambiguous concatenation. - Add unit coverage for canonical encoding, previously colliding identities, the `OpenAIServingBase -> Req -> RadixCache` path, and scheduler/session propagation. - Add manual server regressions for field-position collisions, concatenation collisions, and native session `extra_key` isolation. The encoded namespace is internal and opaque; no request or response schema is changed. Requests that do not provide a cache namespace or LoRA adapter retain `None` as their radix-cache `extra_key`. ## Accuracy Tests This change does not modify model kernels, forward computation, sampling, or generated-token selection. It only controls which radix-cache namespace is eligible for prefix reuse, so model accuracy evaluation is not applicable. - `python -m pytest -q test/unit/test_extra_key_cache_isolation.py`: 4 passed. - `python -m pytest -q test/registered/radix_cache/test_radix_cache_unit.py::TestRadixCache::test_extra_key_isolation`: 1 passed. - Server E2E regressions were added to `test/manual/openai_server/features/test_cache_report.py`, but were not run locally because this workspace does not have the GPU/model-server environment required by that test module. ## Benchmarking and Profiling No inference benchmark was run. The change adds deterministic JSON encoding on the CPU request-control path when a cache namespace or LoRA adapter is present; it does not alter GPU inference kernels. No material throughput impact is expected, but this has not been measured. ## Checklist - [x] Format your code according to the [Format code with pre-commit](https://docs.sglang.io/developer_guide/contribution_guide.html#format-code-with-pre-commit). Black, isort, Ruff (`F401,F821`), `py_compile`, and `git diff --check` passed for the changed files. - [x] Add unit tests according to the [Run and add unit tests](https://docs.sglang.io/developer_guide/contribution_guide.html#run-and-add-unit-tests). - [x] Update documentation according to [Write documentations](https://docs.sglang.io/developer_guide/contribution_guide.html#write-documentations). No documentation update is required because this restores the documented cache-isolation behavior without changing the API. - [ ] Provide accuracy and speed benchmark results according to [Test the accuracy](https://docs.sglang.io/developer_guide/contribution_guide.html#test-the-accuracy) and [Benchmark the speed](https://docs.sglang.io/developer_guide/contribution_guide.html#benchmark-the-speed). Accuracy testing is not applicable, and a performance benchmark was not run; see the sections above. - [x] Follow the SGLang code style [guidance](https://docs.sglang.io/developer_guide/contribution_guide.html#code-style-guidance). ## Review Process 1. Ping Merge Oncalls to start the PR flow. See the [PR Merge Process](https://github.com/sgl-project/sglang/blob/main/.github/MAINTAINER.md#pull-request-merge-process). 2. Get approvals from [CODEOWNERS](https://github.com/sgl-project/sglang/blob/main/.github/CODEOWNERS) and other reviewers. 3. Trigger CI tests with [comments](https://docs.sglang.io/developer_guide/contribution_guide.html#how-to-trigger-ci-tests) or contact authorized users to do so. - `/tag-run-ci-label`, `/rerun-failed-ci`, `/tag-and-rerun-ci` 4. After green CI and required approvals, ask Merge Oncalls to merge. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi added the

pull-request

label

2026-10-29 21:07:06 +00:00

No labels

pull-request

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/sglang#89

No description provided.

Rows
Columns