[GH-ISSUE #2189] [Call For Contributions] Mooncake Conductor Roadmap

bsakaguchi commented

2026-10-29 21:17:03 +00:00

Owner

Originally created by @yejj710 on GitHub (May 22, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2189

Originally assigned to: @Asher-XunZhang, @yejj710 on GitHub.

Background

The Mooncake Conductor Indexer is a service for tracking and reporting token cache-hit statistics across cache tiers and model instances. It exposes APIs that allow clients to query hit statistics by token IDs or chunked token hashes, helping inference systems make better cache-aware scheduling decisions.

The current Conductor implementation has not yet been merged into the main branch. Before merging it, we should clarify the roadmap and improve the overall extensibility of the project. The goal is to evolve the existing HTTP RESTful service into a reusable, multi-language Conductor control plane: the core implementation remains in Go, the interface layer supports Go, Rust, and Python SDKs, and the existing HTTP service remains available as a compatibility layer.

Phase 0: Baseline and Compatibility

smoke tests: Add end-to-end test cases that validate the basic Conductor service flow. @guozhihao-224
Add performance benchmarks: Provide scripts to measure key Conductor service metrics, such as QPS, latency, and memory usage. @guozhihao-224

Phase 1: Go Core Library Refactor

Extract an importable Go library: Move the non-startup business logic from main.go and the core logic from kvevent.EventManager into reusable packages.
Restructure Go packages: Move the binary entry point to cmd/mooncake_conductor and place the core functionality under pkg/conductor.
Define the public Go API: Expose stable methods such as Start, Stop, Register, Unregister, and Query.
Separate interface adapters from core logic: Decouple HTTP handlers, ZMQ subscriptions, and configuration loading from the core service, so every entry point calls the same service layer.

Phase 2: Interface Layer Decoupling

Introduce a gRPC/Proto contract: Add strongly typed proto definitions that standardize the register, unregister, and query operations.
Generate multi-language clients: Provide gRPC-based SDKs for Go, Rust, and Python. Rust and Python clients should call Conductor through RPC instead of depending on Go internal packages.

Phase 3: Correctness and RAS

Route all operations through the core service interface: Ensure register, unregister, and query operations all use the shared service interface to reduce duplicated REST-layer business logic.
Expand concurrency-safety coverage: Test subscription lifecycle management, tenant instance maps, prefix cache table updates, and concurrent queries.
Improve high availability: Make Conductor service data highly available and avoid service disruption caused by single points of failure.

Phase 4: Integrate More Frameworks

Integrate vLLM Ascend: Support ascend-store-connector backends such as Mooncake and Yuanrong. @yangsonglin13
Integrate SGLang HiCache: Support SGLang HiCache as a KV event publisher.
Integrate Mooncake Store: Support a KV event publisher in Mooncake Store. @Asher-XunZhang

Originally created by @yejj710 on GitHub (May 22, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2189 Originally assigned to: @Asher-XunZhang, @yejj710 on GitHub. ## Background The Mooncake Conductor Indexer is a service for tracking and reporting token cache-hit statistics across cache tiers and model instances. It exposes APIs that allow clients to query hit statistics by token IDs or chunked token hashes, helping inference systems make better cache-aware scheduling decisions. The current Conductor implementation has not yet been merged into the `main` branch. Before merging it, we should clarify the roadmap and improve the overall extensibility of the project. The goal is to evolve the existing HTTP RESTful service into a reusable, multi-language Conductor control plane: the core implementation remains in Go, the interface layer supports Go, Rust, and Python SDKs, and the existing HTTP service remains available as a compatibility layer. ## Phase 0: Baseline and Compatibility - [ ] **smoke tests**: Add end-to-end test cases that validate the basic Conductor service flow. @guozhihao-224 - [ ] **Add performance benchmarks**: Provide scripts to measure key Conductor service metrics, such as QPS, latency, and memory usage. @guozhihao-224 ## Phase 1: Go Core Library Refactor - [ ] **Extract an importable Go library**: Move the non-startup business logic from `main.go` and the core logic from `kvevent.EventManager` into reusable packages. - [ ] **Restructure Go packages**: Move the binary entry point to `cmd/mooncake_conductor` and place the core functionality under `pkg/conductor`. - [ ] **Define the public Go API**: Expose stable methods such as `Start`, `Stop`, `Register`, `Unregister`, and `Query`. - [ ] **Separate interface adapters from core logic**: Decouple HTTP handlers, ZMQ subscriptions, and configuration loading from the core service, so every entry point calls the same service layer. ## Phase 2: Interface Layer Decoupling - [ ] **Introduce a gRPC/Proto contract**: Add strongly typed proto definitions that standardize the register, unregister, and query operations. - [ ] **Generate multi-language clients**: Provide gRPC-based SDKs for Go, Rust, and Python. Rust and Python clients should call Conductor through RPC instead of depending on Go internal packages. ## Phase 3: Correctness and RAS - [ ] **Route all operations through the core service interface**: Ensure register, unregister, and query operations all use the shared service interface to reduce duplicated REST-layer business logic. - [ ] **Expand concurrency-safety coverage**: Test subscription lifecycle management, tenant instance maps, prefix cache table updates, and concurrent queries. - [ ] **Improve high availability**: Make Conductor service data highly available and avoid service disruption caused by single points of failure. ## Phase 4: Integrate More Frameworks - [ ] **Integrate vLLM Ascend**: Support ascend-store-connector backends such as Mooncake and Yuanrong. @yangsonglin13 - [ ] **Integrate SGLang HiCache**: Support SGLang HiCache as a KV event publisher. - [ ] **Integrate Mooncake Store**: Support a KV event publisher in Mooncake Store. @Asher-XunZhang

bsakaguchi closed this issue

2026-10-29 21:17:03 +00:00

bsakaguchi commented

2026-10-29 21:17:04 +00:00

Author

Owner

@Asher-XunZhang commented on GitHub (May 22, 2026):

the background：https://github.com/kvcache-ai/Mooncake/issues/977 , https://github.com/kvcache-ai/Mooncake/issues/1408
btw, the current progress is presented on the dev/kv-indexer branch. For more detailed discussions, please join the conversation in this post.

@Asher-XunZhang commented on GitHub (May 22, 2026): the background：https://github.com/kvcache-ai/Mooncake/issues/977 , https://github.com/kvcache-ai/Mooncake/issues/1408 btw, the current progress is presented on the [`dev/kv-indexer`](https://github.com/kvcache-ai/Mooncake/tree/dev/kv-indexer) branch. For more detailed discussions, please join the conversation in [this post](https://github.com/kvcache-ai/Mooncake/discussions/1967).

bsakaguchi commented

2026-10-29 21:17:04 +00:00

Author

Owner

@guozhihao-224 commented on GitHub (May 27, 2026):

Hi @yejj710 @Asher-XunZhang, thanks for putting together this roadmap.

I'd like to contribute. After reading the design docs in docs/source/design/conductor/, browsing the code on dev/kv-indexer, and running go test ./... locally, I'd like to start with two small contributions toward Phase 0:

Pre-PR: fix stale tests in `zmq_client_test.go`

go test ./... currently fails to build the zmq package on dev/kv-indexer. Two distinct kinds of drift:

MockEventHandler.HandleEvent signature is (event KVEvent) but the current EventHandler interface in zmq/types.go requires (event KVEvent, dpRank int64).
ZMQClientConfig field names in the test (ServiceIP, Port, RouterPort) no longer exist on the struct (current fields: CachePoolKey, Endpoint, ReplayEndpoint, ModelName, PollTimeout, ReplayTimeout, ReconnectDelay).

Result: FAIL conductor/zmq [build failed], while conductor/common passes and kvevent / prefixindex have no tests.

I'd like to land a small fix-only PR (~30-50 lines, single file) that updates the test file to compile against the current API and turns go test ./... green. This unblocks the rest of Phase 0.

PR-1: Phase 0-A smoke tests

Once the test file compiles, I'd like to reuse its existing MockPublisher to add end-to-end smoke tests for EventManager:

mooncake-conductor/conductor-ctrl/kvevent/event_manager_test.go
3 happy-path cases:
1. register → publish BlockStored → query returns expected hits
2. register → BlockStored → BlockRemoved → query reflects eviction
3. unregister → subsequent events not consumed
Edge cases (concurrent registration, malformed payloads, missing tenant) in a follow-up

A few questions before I start

Is dev/kv-indexer the right base branch, or is there a more recent integration branch I should target?
Any preference on smoke test placement — per-package _test.go next to the code, or a top-level tests/ directory?
Should CI wiring (a workflow that runs go test ./mooncake-conductor/...) be part of PR-1, or split out?
For the Pre-PR fix, should I keep the existing test cases as-is (just updating the mock to compile), or is it OK to also delete obviously-stale assertions while I'm there?

Happy to adjust scope/order based on your preference. Thanks!

@guozhihao-224 commented on GitHub (May 27, 2026): Hi @yejj710 @Asher-XunZhang, thanks for putting together this roadmap. I'd like to contribute. After reading the design docs in `docs/source/design/conductor/`, browsing the code on `dev/kv-indexer`, and running `go test ./...` locally, I'd like to start with two small contributions toward Phase 0: ### Pre-PR: fix stale tests in `zmq_client_test.go` `go test ./...` currently fails to build the `zmq` package on `dev/kv-indexer`. Two distinct kinds of drift: 1. **`MockEventHandler.HandleEvent` signature** is `(event KVEvent)` but the current `EventHandler` interface in `zmq/types.go` requires `(event KVEvent, dpRank int64)`. 2. **`ZMQClientConfig` field names** in the test (`ServiceIP`, `Port`, `RouterPort`) no longer exist on the struct (current fields: `CachePoolKey`, `Endpoint`, `ReplayEndpoint`, `ModelName`, `PollTimeout`, `ReplayTimeout`, `ReconnectDelay`). Result: `FAIL conductor/zmq [build failed]`, while `conductor/common` passes and `kvevent` / `prefixindex` have no tests. I'd like to land a small fix-only PR (~30-50 lines, single file) that updates the test file to compile against the current API and turns `go test ./...` green. This unblocks the rest of Phase 0. ### PR-1: Phase 0-A smoke tests Once the test file compiles, I'd like to reuse its existing `MockPublisher` to add end-to-end smoke tests for `EventManager`: - `mooncake-conductor/conductor-ctrl/kvevent/event_manager_test.go` - 3 happy-path cases: 1. `register` → publish `BlockStored` → `query` returns expected hits 2. `register` → `BlockStored` → `BlockRemoved` → `query` reflects eviction 3. `unregister` → subsequent events not consumed - Edge cases (concurrent registration, malformed payloads, missing tenant) in a follow-up ### A few questions before I start 1. Is `dev/kv-indexer` the right base branch, or is there a more recent integration branch I should target? 2. Any preference on smoke test placement — per-package `_test.go` next to the code, or a top-level `tests/` directory? 3. Should CI wiring (a workflow that runs `go test ./mooncake-conductor/...`) be part of PR-1, or split out? 4. For the Pre-PR fix, should I keep the existing test cases as-is (just updating the mock to compile), or is it OK to also delete obviously-stale assertions while I'm there? Happy to adjust scope/order based on your preference. Thanks!

bsakaguchi commented

2026-10-29 21:17:04 +00:00

Author

Owner

@yejj710 commented on GitHub (May 27, 2026):

A few questions before I start

Is dev/kv-indexer the right base branch, or is there a more recent integration branch I should target?

Any preference on smoke test placement — per-package _test.go next to the code, or a top-level tests/ directory?

Should CI wiring (a workflow that runs go test ./mooncake-conductor/...) be part of PR-1, or split out?

For the Pre-PR fix, should I keep the existing test cases as-is (just updating the mock to compile), or is it OK to also delete obviously-stale assertions while I'm there?

Happy to adjust scope/order based on your preference. Thanks!

@guozhihao-224 Thanks for your Contributions!
q1: Yes, that’s right. I’ve made some additional code changes on my personal branch (indexer-part), and I’ll sync them to dev/kv-indexer later.
q2: Per-package _test.go files are more suitable for unit tests. My personal preference is to create a separate top-level tests/ directory for end-to-end smoke tests and test scripts, which makes CI integration and local testing easier.
q3: Up to your preference, but the current branch is not integrated with the CI system yet. We can do this after it is merged into the main branch.
q4: The old test cases should already be outdated, so feel free to modify them based on your understanding.

@yejj710 commented on GitHub (May 27, 2026): > ### A few questions before I start > 1. Is `dev/kv-indexer` the right base branch, or is there a more recent integration branch I should target? > 2. Any preference on smoke test placement — per-package `_test.go` next to the code, or a top-level `tests/` directory? > 3. Should CI wiring (a workflow that runs `go test ./mooncake-conductor/...`) be part of PR-1, or split out? > 4. For the Pre-PR fix, should I keep the existing test cases as-is (just updating the mock to compile), or is it OK to also delete obviously-stale assertions while I'm there? > > Happy to adjust scope/order based on your preference. Thanks! @guozhihao-224 Thanks for your Contributions! q1: Yes, that’s right. I’ve made some additional code changes on my personal branch (`indexer-part`), and I’ll sync them to `dev/kv-indexer` later. q2: Per-package `_test.go` files are more suitable for unit tests. My personal preference is to create a separate top-level `tests/` directory for end-to-end smoke tests and test scripts, which makes CI integration and local testing easier. q3: Up to your preference, but the current branch is not integrated with the CI system yet. We can do this after it is merged into the `main` branch. q4: The old test cases should already be outdated, so feel free to modify them based on your understanding.

bsakaguchi commented

2026-10-29 21:17:04 +00:00

Author

Owner

@zhoujing101 commented on GitHub (Jun 24, 2026):

I have added the function of querying information about registered instances, so that upper-layer services can obtain the missing instance information and register the instances again.

https://github.com/kvcache-ai/Mooncake/pull/2595

@zhoujing101 commented on GitHub (Jun 24, 2026): I have added the function of querying information about registered instances, so that upper-layer services can obtain the missing instance information and register the instances again. https://github.com/kvcache-ai/Mooncake/pull/2595

bsakaguchi commented

2026-10-29 21:17:05 +00:00

Author

Owner

@PatStiles commented on GitHub (Jul 21, 2026):

@Asher-XunZhang @yejj710 Is this roadmap still in line with the work merged in https://github.com/kvcache-ai/Mooncake/issues/2189? I was interested in possibly contributing to via some of the additional tasks in the later phases once they were unblocked.

@PatStiles commented on GitHub (Jul 21, 2026): @Asher-XunZhang @yejj710 Is this roadmap still in line with the work merged in https://github.com/kvcache-ai/Mooncake/issues/2189? I was interested in possibly contributing to via some of the additional tasks in the later phases once they were unblocked.

bsakaguchi commented

2026-10-29 21:17:05 +00:00

Author

Owner

@yejj710 commented on GitHub (Jul 21, 2026):

Is this roadmap still in line with the work merged in #2189?

yes, Chase-Rong is working on some refactoring tasks, and you guys can have some discussions about it.~ @Chase-Rong @PatStiles

@yejj710 commented on GitHub (Jul 21, 2026): > Is this roadmap still in line with the work merged in [#2189](https://github.com/kvcache-ai/Mooncake/issues/2189)? yes, Chase-Rong is working on some refactoring tasks, and you guys can have some discussions about it.~ @Chase-Rong @PatStiles

bsakaguchi commented

2026-10-29 21:17:05 +00:00

Author

Owner

@Chase-Rong commented on GitHub (Jul 22, 2026):

Is this roadmap still in line with the work merged in #2189?

yes, Chase-Rong is working on some refactoring tasks, and you guys can have some discussions about it.~ @Chase-Rong @PatStiles

Yes, in response to community feedback, we are refactoring Conductor. We are currently conducting validation with customers, focusing primarily on tasks in Phases 0 and 1. We also welcome future contributions.

@Chase-Rong commented on GitHub (Jul 22, 2026): > > Is this roadmap still in line with the work merged in [#2189](https://github.com/kvcache-ai/Mooncake/issues/2189)? > > yes, Chase-Rong is working on some refactoring tasks, and you guys can have some discussions about it.~ [@Chase-Rong](https://github.com/Chase-Rong) [@PatStiles](https://github.com/PatStiles) Yes, in response to community feedback, we are refactoring Conductor. We are currently conducting validation with customers, focusing primarily on tasks in Phases 0 and 1. We also welcome future contributions.

bsakaguchi commented

2026-10-29 21:17:05 +00:00

Author

Owner

@0z5a commented on GitHub (Sep 6, 2026):

Hi @Chase-Rong @yejj710 — could I help with a bounded Phase 3 subscription-lifecycle/concurrent-query regression slice against the current Conductor upstreaming series?

I checked main cb9b380b9112606f5826f76f8471a6b653ec6e62 and #3879 at 25fda5c91214628be6b56f88daaf414fd8ad2a78. Main currently carries the C++ skeleton/common layer, while #3879 adds prefix types/hash strategies; its described subscriber and KV event manager/API slices are still downstream. I would not revive the older Go implementation, duplicate your hash/index work, or take over @guozhihao-224's Phase 0 smoke/benchmark work.

Once the actual shared lifecycle API is available, I would like to cover a small agreed set of register/unregister/query interleavings, unsubscribe with an already-in-flight event, and bounded shutdown with pending work. Expected behavior should follow the real API/ownership contract — I am not proposing new generation/tenant fields, RPCs, or HA semantics just to make tests possible.

Is there a current integration branch and a useful unclaimed subset to align with, or should this wait for the subscriber/event-manager slice to land? No lifecycle implementation or passing lifecycle tests are claimed here, and no author branch or upstream code has been modified. This audit and coordination note are AI-assisted; any future patch would need the repository's required human review.

@0z5a commented on GitHub (Sep 6, 2026): Hi @Chase-Rong @yejj710 — could I help with a bounded Phase 3 subscription-lifecycle/concurrent-query regression slice against the current Conductor upstreaming series? I checked main `cb9b380b9112606f5826f76f8471a6b653ec6e62` and #3879 at `25fda5c91214628be6b56f88daaf414fd8ad2a78`. Main currently carries the C++ skeleton/common layer, while #3879 adds prefix types/hash strategies; its described subscriber and KV event manager/API slices are still downstream. I would not revive the older Go implementation, duplicate your hash/index work, or take over @guozhihao-224's Phase 0 smoke/benchmark work. Once the actual shared lifecycle API is available, I would like to cover a small agreed set of register/unregister/query interleavings, unsubscribe with an already-in-flight event, and bounded shutdown with pending work. Expected behavior should follow the real API/ownership contract — I am not proposing new generation/tenant fields, RPCs, or HA semantics just to make tests possible. Is there a current integration branch and a useful unclaimed subset to align with, or should this wait for the subscriber/event-manager slice to land? No lifecycle implementation or passing lifecycle tests are claimed here, and no author branch or upstream code has been modified. This audit and coordination note are AI-assisted; any future patch would need the repository's required human review.

bsakaguchi commented

2026-10-29 21:17:05 +00:00

Author

Owner

@Chase-Rong commented on GitHub (Sep 7, 2026):

Hi @Chase-Rong @yejj710 — could I help with a bounded Phase 3 subscription-lifecycle/concurrent-query regression slice against the current Conductor upstreaming series?

I checked main cb9b380b9112606f5826f76f8471a6b653ec6e62 and #3879 at 25fda5c91214628be6b56f88daaf414fd8ad2a78. Main currently carries the C++ skeleton/common layer, while #3879 adds prefix types/hash strategies; its described subscriber and KV event manager/API slices are still downstream. I would not revive the older Go implementation, duplicate your hash/index work, or take over @guozhihao-224's Phase 0 smoke/benchmark work.

Once the actual shared lifecycle API is available, I would like to cover a small agreed set of register/unregister/query interleavings, unsubscribe with an already-in-flight event, and bounded shutdown with pending work. Expected behavior should follow the real API/ownership contract — I am not proposing new generation/tenant fields, RPCs, or HA semantics just to make tests possible.

Is there a current integration branch and a useful unclaimed subset to align with, or should this wait for the subscriber/event-manager slice to land? No lifecycle implementation or passing lifecycle tests are claimed here, and no author branch or upstream code has been modified. This audit and coordination note are AI-assisted; any future patch would need the repository's required human review.

After discussions with the community maintainer, the Conductor is being merged in stages, including patches for the Mooncake Store. A total of six pull requests are expected to be merged. A new roadmap will be released soon. The Conductor does indeed require some development work to improve high availability, but this will require all pull requests to be merged into the main branch. This will take approximately two weeks.

@Chase-Rong commented on GitHub (Sep 7, 2026): > Hi [@Chase-Rong](https://github.com/Chase-Rong) [@yejj710](https://github.com/yejj710) — could I help with a bounded Phase 3 subscription-lifecycle/concurrent-query regression slice against the current Conductor upstreaming series? > > I checked main `cb9b380b9112606f5826f76f8471a6b653ec6e62` and [#3879](https://github.com/kvcache-ai/Mooncake/pull/3879) at `25fda5c91214628be6b56f88daaf414fd8ad2a78`. Main currently carries the C++ skeleton/common layer, while [#3879](https://github.com/kvcache-ai/Mooncake/pull/3879) adds prefix types/hash strategies; its described subscriber and KV event manager/API slices are still downstream. I would not revive the older Go implementation, duplicate your hash/index work, or take over [@guozhihao-224](https://github.com/guozhihao-224)'s Phase 0 smoke/benchmark work. > > Once the actual shared lifecycle API is available, I would like to cover a small agreed set of register/unregister/query interleavings, unsubscribe with an already-in-flight event, and bounded shutdown with pending work. Expected behavior should follow the real API/ownership contract — I am not proposing new generation/tenant fields, RPCs, or HA semantics just to make tests possible. > > Is there a current integration branch and a useful unclaimed subset to align with, or should this wait for the subscriber/event-manager slice to land? No lifecycle implementation or passing lifecycle tests are claimed here, and no author branch or upstream code has been modified. This audit and coordination note are AI-assisted; any future patch would need the repository's required human review. After discussions with the community maintainer, the Conductor is being merged in stages, including patches for the Mooncake Store. A total of six pull requests are expected to be merged. A new roadmap will be released soon. The Conductor does indeed require some development work to improve high availability, but this will require all pull requests to be merged into the main branch. This will take approximately two weeks.

bsakaguchi referenced this issue

2026-10-29 21:29:30 +00:00

[PR #609] [CLOSED] [TransferEngine] Tcp Transport supporting vram data transfer (#602) #1685

Rows
Columns

[GH-ISSUE #2189] [Call For Contributions] Mooncake Conductor Roadmap #602

Background

Phase 0: Baseline and Compatibility

Phase 1: Go Core Library Refactor

Phase 2: Interface Layer Decoupling

Phase 3: Correctness and RAS

Phase 4: Integrate More Frameworks

Pre-PR: fix stale tests in zmq_client_test.go

PR-1: Phase 0-A smoke tests

A few questions before I start

A few questions before I start

Pre-PR: fix stale tests in `zmq_client_test.go`