bsakaguchi commented

Owner

Originally created by @alogfans on GitHub (Nov 14, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1058

Mooncake Transfer Engine NEXT (TENT) Roadmap

This document outlines the detailed roadmap for TENT, following the release of its technical preview. We categorize the roadmap into two phases:

Phase 1: Make it production-ready
Phase 2: Advanced feature development

The roadmap is informed by the TE+ system architecture described in the FAISys 25 talk "TENT: A Declarative Slice Spraying Engine for Performant and Resilient Data Movement in Disaggregated LLM Serving."
TENT aims to implement declarative orchestration, dynamic path scheduling, and resilience in a modular and community-driven way.

✅ Phase 1 – Production Readiness (Before 26Q1)

Transport Backend Support

NVIDIA (GDR, GDS, NVLink, MNNVL)
AMD/RoCM GPU (via ROCm APIs, port from mainstream version)
Huawei Ascend NPU (via UB/HIXL, completed)
- Direct file I/O
Moorethread NPUs (requires vendor interface, port from mainstream version)
Multi-rail TCP (with parallel stream slicing and connection pooling)
More NPUs and SmartNICs (as thin transport plugins)
Backport to the mainstream
- NVLink (w/o fabric)

Configuration Handling

Load configuration from file
Support override via environment variables or runtime parameters
Hot-reload or staged application of config changes

Logging

Adjustable log levels (INFO/DEBUG) to suppress noise in prod
Write logs to rotating file instead of stdout
Non-blocking async file writing (community friendly)

Python Integration

Compatible fallback via MC_USE_TEV1=1
Clean high-level declarative Python API (transfer(src, dst) style)
Runtime integration with Mooncake Store (replace build-time coupling)
Integrate with Mooncake Store at build type

Packaging & Usability

Provide Python wheel with compiled transport backends
Enable runtime dynamic loading without manual config
Extract TENT into a standalone service or library
CLI tool for isolated transfers & diagnostics (community friendly)
Docs & guide

🚀 Phase 2 – Advanced Capabilities (26Q3/Q4)

Transfer Intent API: Expose deadline_ns, policy_name, intent_type, priority, and transport_hint through stable C and Python APIs. Standardize intent types such as foreground_get, background_prefetch, migration, checkpoint, weight_loading, and staging_internal, and integrate them with Mooncake Store, SGLang, vLLM, and HIXL.
Diagnostics and Benchmarking: Provide a tent toolset covering topology inspection, configuration validation, peer serving, link selection, connectivity testing, and multi-node benchmarking. Generate both readable reports and sanitized JSON diagnostic bundles.
Dynamic Path Selection and Execution Plans: Migrate mature Classic TE transports into TENT and describe each backend through a unified capability model. Generate explainable Direct, Staged, Fallback, or Degraded plans from intent, topology, policy, health, and segment metadata. Use plugins to support heterogeneous accelerators without adding hardware-specific logic to the scheduler.
SLO-Aware Scheduling: Build telemetry-driven admission control using rail bandwidth, QP pressure, queue depth, and receiver credits. Support deadline-aware dispatch, aging, reserved capacity, and early degradation when an SLO cannot be met.
QoS-Aware RDMA: Connect policy and priority decisions to route generation, sub-batching, QP pools, and hardware TC/SL. Isolate foreground, prefetch, migration, and checkpoint traffic through stable scheduling and rate-control policies.
Multi-Rail Scheduling: Improve telemetry-driven slice placement to achieve more stable and globally efficient rail utilization.
Tracing and Observability: Trace the complete causal chain from intent and execution plan to stage, slice, rail, QP, transport completion, and final request state. Feed runtime metrics back into path cost, rail weighting, bad-link exclusion, and fallback decisions.
Production Validation: Test API compatibility, plan correctness, heterogeneous paths, fallback and degradation, deadline behavior, QoS isolation, multi-rail balance, fault recovery, performance regression, and long-running stability.

💡 Community-Contributable Areas

We welcome contributions in the following areas:

New transport backends (e.g. AMD, TCP, CXL)
Packaging (wheel, Docker, CI)
Logging and CLI tools
Benchmark suite and cluster test harness
PyTorch/TensorFlow plugin wrappers
Documentation and deployment examples

Please check our issues for “good first issues” and open RFCs.

🛣️ Vision

TENT will become the unified slice-spraying data movement substrate for heterogeneous inference workloads across diverse accelerator vendors, fabrics, and deployment models. As we align with the TE+ paper's declarative scheduling, fault-tolerant runtime, and plugin-friendly architecture, TENT aims to serve as the reliable transfer layer for LLM systems at scale.

Originally created by @alogfans on GitHub (Nov 14, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/1058 # Mooncake Transfer Engine NEXT (TENT) Roadmap This document outlines the detailed roadmap for TENT, following the release of its technical preview. We categorize the roadmap into two phases: - **Phase 1**: Make it production-ready - **Phase 2**: Advanced feature development > The roadmap is informed by the TE+ system architecture described in the FAISys 25 talk _"TENT: A Declarative Slice Spraying Engine for Performant and Resilient Data Movement in Disaggregated LLM Serving."_ > TENT aims to implement declarative orchestration, dynamic path scheduling, and resilience in a modular and community-driven way. --- ## ✅ Phase 1 – Production Readiness (Before 26Q1) ### Transport Backend Support - [x] NVIDIA (GDR, GDS, NVLink, MNNVL) - [ ] AMD/RoCM GPU (via ROCm APIs, port from mainstream version) - [x] Huawei Ascend NPU (via UB/HIXL, completed) - [ ] Direct file I/O - [ ] Moorethread NPUs (requires vendor interface, port from mainstream version) - [ ] Multi-rail TCP (with parallel stream slicing and connection pooling) - [ ] More NPUs and SmartNICs (as thin transport plugins) - [ ] Backport to the mainstream - [ ] NVLink (w/o fabric) ### Configuration Handling - [x] Load configuration from file - [ ] Support override via environment variables or runtime parameters - [ ] Hot-reload or staged application of config changes ### Logging - [ ] Adjustable log levels (INFO/DEBUG) to suppress noise in prod - [ ] Write logs to rotating file instead of stdout - [ ] Non-blocking async file writing (community friendly) ### Python Integration - [X] Compatible fallback via `MC_USE_TEV1=1` - [ ] Clean high-level declarative Python API (`transfer(src, dst)` style) - [ ] Runtime integration with Mooncake Store (replace build-time coupling) - [X] Integrate with Mooncake Store at build type ### Packaging & Usability - [ ] Provide Python wheel with compiled transport backends - [x] Enable runtime dynamic loading without manual config - [x] Extract TENT into a standalone service or library - [ ] CLI tool for isolated transfers & diagnostics (community friendly) - [ ] Docs & guide --- ## 🚀 Phase 2 – Advanced Capabilities (26Q3/Q4) - [ ] **Transfer Intent API:** Expose `deadline_ns`, `policy_name`, `intent_type`, `priority`, and `transport_hint` through stable C and Python APIs. Standardize intent types such as `foreground_get`, `background_prefetch`, `migration`, `checkpoint`, `weight_loading`, and `staging_internal`, and integrate them with Mooncake Store, SGLang, vLLM, and HIXL. - [ ] **Diagnostics and Benchmarking:** Provide a `tent` toolset covering topology inspection, configuration validation, peer serving, link selection, connectivity testing, and multi-node benchmarking. Generate both readable reports and sanitized JSON diagnostic bundles. - [ ] **Dynamic Path Selection and Execution Plans:** Migrate mature Classic TE transports into TENT and describe each backend through a unified capability model. Generate explainable Direct, Staged, Fallback, or Degraded plans from intent, topology, policy, health, and segment metadata. Use plugins to support heterogeneous accelerators without adding hardware-specific logic to the scheduler. - [ ] **SLO-Aware Scheduling:** Build telemetry-driven admission control using rail bandwidth, QP pressure, queue depth, and receiver credits. Support deadline-aware dispatch, aging, reserved capacity, and early degradation when an SLO cannot be met. - [ ] **QoS-Aware RDMA:** Connect policy and priority decisions to route generation, sub-batching, QP pools, and hardware TC/SL. Isolate foreground, prefetch, migration, and checkpoint traffic through stable scheduling and rate-control policies. - [ ] **Multi-Rail Scheduling:** Improve telemetry-driven slice placement to achieve more stable and globally efficient rail utilization. - [ ] **Tracing and Observability:** Trace the complete causal chain from intent and execution plan to stage, slice, rail, QP, transport completion, and final request state. Feed runtime metrics back into path cost, rail weighting, bad-link exclusion, and fallback decisions. - [ ] **Production Validation:** Test API compatibility, plan correctness, heterogeneous paths, fallback and degradation, deadline behavior, QoS isolation, multi-rail balance, fault recovery, performance regression, and long-running stability. --- ## 💡 Community-Contributable Areas We welcome contributions in the following areas: - New transport backends (e.g. AMD, TCP, CXL) - Packaging (wheel, Docker, CI) - Logging and CLI tools - Benchmark suite and cluster test harness - PyTorch/TensorFlow plugin wrappers - Documentation and deployment examples Please check [our issues](https://github.com/kvcache-ai/Mooncake/issues) for “good first issues” and open RFCs. --- ## 🛣️ Vision TENT will become the unified slice-spraying data movement substrate for heterogeneous inference workloads across diverse accelerator vendors, fabrics, and deployment models. As we align with the TE+ paper's declarative scheduling, fault-tolerant runtime, and plugin-friendly architecture, TENT aims to serve as the reliable transfer layer for LLM systems at scale.

bsakaguchi added the

Roadmap

label

2026-10-29 21:13:10 +00:00

bsakaguchi commented

2026-10-29 21:13:11 +00:00

Author

Owner

@ykwd commented on GitHub (Nov 14, 2025):

Great! I’m excited to see it move forward. Do we have an estimated timeline for merging it into the main branch?

@ykwd commented on GitHub (Nov 14, 2025): Great! I’m excited to see it move forward. Do we have an estimated timeline for merging it into the main branch?

bsakaguchi commented

2026-10-29 21:13:11 +00:00

Author

Owner

@alogfans commented on GitHub (Nov 14, 2025):

We expect to make it fully ready at 26Q1, and both version may coexist in about 3-6 months.

@alogfans commented on GitHub (Nov 14, 2025): We expect to make it fully ready at 26Q1, and both version may coexist in about 3-6 months.

bsakaguchi commented

2026-10-29 21:13:11 +00:00

Author

Owner

@nickyc975 commented on GitHub (Nov 14, 2025):

Hi, glad to see the roadmap! Just a few questions:

TransferEngine v1 is going to completely replace current transfer engine, both as a standalone component and in Mooncake Store, am I right?
I see components related to file based segments, but they seems not completed. What's the plan for file segments?

@nickyc975 commented on GitHub (Nov 14, 2025): Hi, glad to see the roadmap! Just a few questions: 1. TransferEngine v1 is going to completely replace current transfer engine, both as a standalone component and in Mooncake Store, am I right? 2. I see components related to file based segments, but they seems not completed. What's the plan for file segments?

bsakaguchi commented

2026-10-29 21:13:12 +00:00

Author

Owner

@alogfans commented on GitHub (Nov 14, 2025):

@nickyc975

TransferEngine v1 is going to completely replace current transfer engine, both as a standalone component and in Mooncake Store, am I right? A: Yes, maybe a standalone version can be released earlier.
I see components related to file based segments, but they seems not completed. What's the plan for file segments? A: In TENT we have finshed the file based segment, both GDS and iouring. It is good to backport it to the current version, or add more transports for other kind of NPUs.

@alogfans commented on GitHub (Nov 14, 2025): @nickyc975 1. TransferEngine v1 is going to completely replace current transfer engine, both as a standalone component and in Mooncake Store, am I right? A: Yes, maybe a standalone version can be released earlier. 2. I see components related to file based segments, but they seems not completed. What's the plan for file segments? A: In TENT we have finshed the file based segment, both GDS and iouring. It is good to backport it to the current version, or add more transports for other kind of NPUs.

bsakaguchi commented

2026-10-29 21:13:12 +00:00

Author

Owner

@nickyc975 commented on GitHub (Nov 14, 2025):

In TENT we have finshed the file based segment, both GDS and iouring. It is good to backport it to the current version, or add more transports for other kind of NPUs.

From what I see, the file segment support in TENT does not include exporting local files or accessing remote files. Is this not needed, or left to be implemented in the future? If it's the latter, I can try to port the NVMeoF Transport to TENT.

@nickyc975 commented on GitHub (Nov 14, 2025): > In TENT we have finshed the file based segment, both GDS and iouring. It is good to backport it to the current version, or add more transports for other kind of NPUs. From what I see, the file segment support in TENT does not include exporting local files or accessing remote files. Is this not needed, or left to be implemented in the future? If it's the latter, I can try to port the [NVMeoF Transport](https://github.com/kvcache-ai/Mooncake/pull/793) to TENT.

bsakaguchi commented

2026-10-29 21:13:12 +00:00

Author

Owner

@staryxchen commented on GitHub (Nov 17, 2025):

Hi @alogfans
Can we support multiple transport within a single transfer engine? This would enable us to select the optimal transport for different type transfer task. For example, if a transfer task want to move the data from HOST A to HOST B, we can use RDMA; if a transfer task just move the data from GPU A to GPU B, we can use Nvlink.

@staryxchen commented on GitHub (Nov 17, 2025): Hi @alogfans Can we support multiple transport within a single transfer engine? This would enable us to select the optimal transport for different type transfer task. For example, if a transfer task want to move the data from HOST A to HOST B, we can use RDMA; if a transfer task just move the data from GPU A to GPU B, we can use Nvlink.

bsakaguchi commented

2026-10-29 21:13:12 +00:00

Author

Owner

@alogfans commented on GitHub (Nov 18, 2025):

In TENT we have finshed the file based segment, both GDS and iouring. It is good to backport it to the current version, or add more transports for other kind of NPUs.

From what I see, the file segment support in TENT does not include exporting local files or accessing remote files. Is this not needed, or left to be implemented in the future? If it's the latter, I can try to port the NVMeoF Transport to TENT.

In my own design, we prefer users to initialize NVMe-of initiator by themselves, and TENT only access the mapped file. However if it is possible, we can port NVMe-of support in TENT.

@alogfans commented on GitHub (Nov 18, 2025): > > In TENT we have finshed the file based segment, both GDS and iouring. It is good to backport it to the current version, or add more transports for other kind of NPUs. > > From what I see, the file segment support in TENT does not include exporting local files or accessing remote files. Is this not needed, or left to be implemented in the future? If it's the latter, I can try to port the [NVMeoF Transport](https://github.com/kvcache-ai/Mooncake/pull/793) to TENT. In my own design, we prefer users to initialize NVMe-of initiator by themselves, and TENT only access the mapped file. However if it is possible, we can port NVMe-of support in TENT.

bsakaguchi commented

2026-10-29 21:13:12 +00:00

Author

Owner

@alogfans commented on GitHub (Nov 18, 2025):

Can we support multiple transport within a single transfer engine? This would enable us to select the optimal transport for different type transfer task. For example, if a transfer task want to move the data from HOST A to HOST B, we can use RDMA; if a transfer task just move the data from GPU A to GPU B, we can use Nvlink.

Currently it is implemented, you can see transfer_engine_impl.cpp. You can enhance the selection algorithm (e.g., make it plugable)

@alogfans commented on GitHub (Nov 18, 2025): > Can we support multiple transport within a single transfer engine? This would enable us to select the optimal transport for different type transfer task. For example, if a transfer task want to move the data from HOST A to HOST B, we can use RDMA; if a transfer task just move the data from GPU A to GPU B, we can use Nvlink. Currently it is implemented, you can see transfer_engine_impl.cpp. You can enhance the selection algorithm (e.g., make it plugable)

bsakaguchi commented

2026-10-29 21:13:13 +00:00

Author

Owner

@thqq479 commented on GitHub (Nov 25, 2025):

In this version, the H100 machine does not have RDMA. Is it feasible to use NVLINK for transmission between nodes within a single machine? @alogfans

@thqq479 commented on GitHub (Nov 25, 2025): In this version, the H100 machine does not have RDMA. Is it feasible to use NVLINK for transmission between nodes within a single machine? @alogfans

bsakaguchi commented

2026-10-29 21:13:13 +00:00

Author

Owner

@stmatengss commented on GitHub (Nov 28, 2025):

In this version, the H100 machine does not have RDMA. Is it feasible to use NVLINK for transmission between nodes within a single machine? @alogfans

Coming soon....

@stmatengss commented on GitHub (Nov 28, 2025): > In this version, the H100 machine does not have RDMA. Is it feasible to use NVLINK for transmission between nodes within a single machine? [@alogfans](https://github.com/alogfans) Coming soon....

bsakaguchi commented

2026-10-29 21:13:13 +00:00

Author

Owner

@alogfans commented on GitHub (Dec 2, 2025):

In this version, the H100 machine does not have RDMA. Is it feasible to use NVLINK for transmission between nodes within a single machine? @alogfans

If all processes are in the same machine, the current version is OK.

@alogfans commented on GitHub (Dec 2, 2025): > In this version, the H100 machine does not have RDMA. Is it feasible to use NVLINK for transmission between nodes within a single machine? [@alogfans](https://github.com/alogfans) If all processes are in the same machine, the current version is OK.

bsakaguchi commented

2026-10-29 21:13:13 +00:00

Author

Owner

@thqq479 commented on GitHub (Dec 16, 2025):

In this version, the H100 machine does not have RDMA. Is it feasible to use NVLINK for transmission between nodes within a single machine? @alogfans

If all processes are in the same machine, the current version is OK.

https://github.com/kvcache-ai/Mooncake/issues/481 I get this OOM error when testing on a single H100 machine @alogfans

@thqq479 commented on GitHub (Dec 16, 2025): > > In this version, the H100 machine does not have RDMA. Is it feasible to use NVLINK for transmission between nodes within a single machine? [@alogfans](https://github.com/alogfans) > > If all processes are in the same machine, the current version is OK. https://github.com/kvcache-ai/Mooncake/issues/481 I get this OOM error when testing on a single H100 machine @alogfans

bsakaguchi commented

2026-10-29 21:13:13 +00:00

Author

Owner

@QingGeng36 commented on GitHub (Mar 23, 2026):

@alogfans Hi, I've read the paper and think the design is excellent. I'd like to contribute to the TENT project, specifically focusing on implementing the heterogeneous functionality.

@QingGeng36 commented on GitHub (Mar 23, 2026): @alogfans Hi, I've read the paper and think the design is excellent. I'd like to contribute to the TENT project, specifically focusing on implementing the heterogeneous functionality.

bsakaguchi commented

2026-10-29 21:13:13 +00:00

Author

Owner

@zchuango commented on GitHub (Mar 30, 2026):

Hi @alogfans I am currently working on optimizing large package transfers in KVC. The flexibility and scalability of the current TE architecture have attracted me. I am currently trying to integrate UB/URMA memory semantic programming into TE. if the community has any plans to integrate udmk/urma via UB protocol? I would be very happy to contribute this feature to the community and look forward to feedback and discussions from the experts.

@zchuango commented on GitHub (Mar 30, 2026): Hi @alogfans I am currently working on optimizing large package transfers in KVC. The flexibility and scalability of the current TE architecture have attracted me. I am currently trying to integrate [UB](https://www.unifiedbus.com/en)/[URMA](https://atomgit.com/openeuler/umdk) memory semantic programming into TE. if the community has any plans to integrate udmk/urma via UB protocol? I would be very happy to contribute this feature to the community and look forward to feedback and discussions from the experts.

bsakaguchi commented

2026-10-29 21:13:14 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jun 29, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jun 29, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:13:14 +00:00

Author

Owner

@catyans commented on GitHub (Jul 11, 2026):

Linking a related RFC for the TENT QoS/backpressure track: #2856 proposes a TENT QoS Contract schema and effective policy model. It is intended as the control-plane contract before implementing stronger QoS scheduling, priority authorization, inflight policing, bandwidth caps, and receiver-credit allocation. The first milestone is intentionally limited to schema parsing, validation, effective-policy resolution, and explain/debug output, without changing default runtime behavior. This aligns with the roadmap items around QoS-based scheduling, sender/receiver-aware scheduling, and backpressure when hitting resource limits.

@catyans commented on GitHub (Jul 11, 2026): Linking a related RFC for the TENT QoS/backpressure track: #2856 proposes a TENT QoS Contract schema and effective policy model. It is intended as the control-plane contract before implementing stronger QoS scheduling, priority authorization, inflight policing, bandwidth caps, and receiver-credit allocation. The first milestone is intentionally limited to schema parsing, validation, effective-policy resolution, and explain/debug output, without changing default runtime behavior. This aligns with the roadmap items around QoS-based scheduling, sender/receiver-aware scheduling, and backpressure when hitting resource limits.

bsakaguchi commented

2026-10-29 21:13:14 +00:00

Author

Owner

@alogfans commented on GitHub (Jul 21, 2026):

Note that Q3/Q4 planning has been updated.

@alogfans commented on GitHub (Jul 21, 2026): Note that Q3/Q4 planning has been updated.

bsakaguchi referenced this issue

2026-10-29 21:28:00 +00:00

[PR #350] [MERGED] Fix news render in README #1501

Rows
Columns

[GH-ISSUE #1058] [RoadMap] Mooncake Transfer Engine NEXT #350

Mooncake Transfer Engine NEXT (TENT) Roadmap

✅ Phase 1 – Production Readiness (Before 26Q1)

Transport Backend Support

Configuration Handling

Logging

Python Integration

Packaging & Usability

🚀 Phase 2 – Advanced Capabilities (26Q3/Q4)

💡 Community-Contributable Areas

🛣️ Vision