[GH-ISSUE #162] [WIP] Refactoring Mooncake Transfer Engine #78

New issue

Closed

opened 2026-10-29 21:08:48 +00:00 by bsakaguchi · 12 comments

bsakaguchi commented 2026-10-29 21:08:48 +00:00

Owner

Copy link

Originally created by @alogfans on GitHub (Mar 23, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/162

This issue is for discussing how to refactor the design of Mooncake Transfer Engine.

We acknowledge @doujiang24 @yuan-luo @alogfans (may be incomplete) contributing PRs #161 #134 #147 to enhance Mooncake Transfer Engine. However these patches have major code modification. It make the code review and merge harder. Also, the interface also changed significantly, and backward compatibility may be affected.

So, before merging to the main stream, may be we can implement Transfer Engine V2 (we also want a short abbr. similar to nixl) seperately. When the second version is ready, we can replace it with the first one.

In this stage, we expect to do the following:

• Use the Status-based error reporting mechanism, instead of error number. @yuan-luo
• Update the metadata format, so a buffer can be registered by multiple transports. e.g. a memory region can use shared memory and rdma transports. @alogfans
• Add shared memory transport and fix the problem of nvmeof transport. (environment needed)
• When transferring between local processes, use shared memory if possible.
• Try to eliminate the use of unsafe http-based handshake.

What about everyone's opinions and suggestions?

Originally created by @alogfans on GitHub (Mar 23, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/162 This issue is for discussing how to refactor the design of Mooncake Transfer Engine. We acknowledge @doujiang24 @yuan-luo @alogfans (may be incomplete) contributing PRs #161 #134 #147 to enhance Mooncake Transfer Engine. However these patches have major code modification. It make the code review and merge harder. Also, the interface also changed significantly, and backward compatibility may be affected. So, before merging to the main stream, may be we can implement `Transfer Engine V2` (we also want a short abbr. similar to `nixl`) seperately. When the second version is ready, we can replace it with the first one. In this stage, we expect to do the following: - [ ] Use the `Status`-based error reporting mechanism, instead of error number. @yuan-luo - [ ] Update the metadata format, so a buffer can be registered by multiple transports. e.g. a memory region can use shared memory and rdma transports. @alogfans - [ ] Add shared memory transport and fix the problem of nvmeof transport. (environment needed) - [ ] When transferring between local processes, use shared memory if possible. - [ ] Try to eliminate the use of unsafe http-based handshake. What about everyone's opinions and suggestions?

bsakaguchi 2026-10-29 21:08:48 +00:00

• closed this issue
• added the
  stale
  auto-closed
  labels

bsakaguchi commented 2026-10-29 21:08:49 +00:00

Author

Owner

Copy link

@yuan-luo commented on GitHub (Mar 24, 2025):

@alogfans Sure. I'm fine with the plan. I also saw TransferEngine is on the sglang's roadmap. https://github.com/sgl-project/sglang/issues/4655
Could you please shed some lights on more details?

<!-- gh-comment-id:2747810476 --> @yuan-luo commented on GitHub (Mar 24, 2025): @alogfans Sure. I'm fine with the plan. I also saw TransferEngine is on the sglang's roadmap. https://github.com/sgl-project/sglang/issues/4655 Could you please shed some lights on more details?

bsakaguchi commented 2026-10-29 21:08:49 +00:00

Author

Owner

Copy link

@alogfans commented on GitHub (Mar 25, 2025):

@yuan-luo You can change the mooncake-ngxl directory to add Status-based APIs, keeping mooncake-transfer-engine unchanged.

<!-- gh-comment-id:2751465574 --> @alogfans commented on GitHub (Mar 25, 2025): @yuan-luo You can change the `mooncake-ngxl` directory to add Status-based APIs, keeping `mooncake-transfer-engine` unchanged.

bsakaguchi commented 2026-10-29 21:08:50 +00:00

Author

Owner

Copy link

@yuan-luo commented on GitHub (Mar 26, 2025):

@yuan-luo You can change the mooncake-ngxl directory to add Status-based APIs, keeping mooncake-transfer-engine unchanged.

@alogfans Sure. Btw, do you mean fork alogfans:transfer-engine-v2 and work on the branch temporarily? Otherwise we'd better wait for PR#170 being merged into main branch.

<!-- gh-comment-id:2753185171 --> @yuan-luo commented on GitHub (Mar 26, 2025): > [@yuan-luo](https://github.com/yuan-luo) You can change the `mooncake-ngxl` directory to add Status-based APIs, keeping `mooncake-transfer-engine` unchanged. @alogfans Sure. Btw, do you mean fork alogfans:transfer-engine-v2 and work on the branch temporarily? Otherwise we'd better wait for [PR#170](https://github.com/kvcache-ai/Mooncake/pull/170) being merged into main branch.

bsakaguchi commented 2026-10-29 21:08:50 +00:00

Author

Owner

Copy link

@alogfans commented on GitHub (Apr 1, 2025):

@yuan-luo

I recently reviewed the code in PR #170 and found that the new implementation may be buggy, so I believe that the progressive update is more suitable.

Goal:

• Keep backward compatibility in C/C++ APIs (transfer_engine_c.h) and metadata format
• Add support of more transports, so transfer engine can be used in both mem-to-mem and mem-to-file in any environment. The chain structure can be like the following:
  
• Fix problems of out-of-date metadata caching, handshake and transfer failures

Stage 1 Plan:

• Extract the SegmentDesc classes to a seperate file, with MemorySegmentDesc and FileSegmentDesc types; making tcp and rdma transports share the same MemorySegmentDesc
• Implement shared memory/cxl transport
• Check and fix problems in gds (originally nvme-of) transport and classic file transport
• Let the multitransport automatically select proper transport according to segment desc and HW setup
• TBD: Use some lightweight TCP-based library to perform tcp transport better
• TBD: Refactor socket-based handshake, using coro_rpc, or ...?

(You can add more in the list)

<!-- gh-comment-id:2768308097 --> @alogfans commented on GitHub (Apr 1, 2025): @yuan-luo I recently reviewed the code in PR #170 and found that the new implementation may be buggy, so I believe that the progressive update is more suitable. **Goal:** - Keep backward compatibility in C/C++ APIs (transfer_engine_c.h) and metadata format - Add support of more transports, so transfer engine can be used in both mem-to-mem and mem-to-file in any environment. The chain structure can be like the following:  - Fix problems of out-of-date metadata caching, handshake and transfer failures **Stage 1 Plan:** - [ ] Extract the SegmentDesc classes to a seperate file, with MemorySegmentDesc and FileSegmentDesc types; making tcp and rdma transports share the same MemorySegmentDesc - [ ] Implement shared memory/cxl transport - [ ] Check and fix problems in gds (originally nvme-of) transport and classic file transport - [ ] Let the multitransport automatically select proper transport according to segment desc and HW setup - [ ] TBD: Use some lightweight TCP-based library to perform tcp transport better - [ ] TBD: Refactor socket-based handshake, using coro_rpc, or ...? (You can add more in the list)

bsakaguchi commented 2026-10-29 21:08:50 +00:00

Author

Owner

Copy link

@xiaguan commented on GitHub (Apr 1, 2025):

From the perspective of the mooncake store, I think there are a few issues that need to be considered:

• Is it possible to provide a synchronous blocking interface to avoid idle CPU cycles caused by busy-waiting in the store?

• We'd like to improve observability so that when performance or logic issues arise, we can identify the specific module responsible more efficiently.

<!-- gh-comment-id:2768581880 --> @xiaguan commented on GitHub (Apr 1, 2025): From the perspective of the mooncake store, I think there are a few issues that need to be considered: * Is it possible to provide a synchronous blocking interface to avoid idle CPU cycles caused by busy-waiting in the store? * We'd like to improve observability so that when performance or logic issues arise, we can identify the specific module responsible more efficiently.

bsakaguchi commented 2026-10-29 21:08:50 +00:00

Author

Owner

Copy link

@yuan-luo commented on GitHub (Apr 1, 2025):

@yuan-luo
I recently reviewed the code in PR https://github.com/kvcache-ai/Mooncake/pull/170 and found that the new implementation may be buggy, so I believe that the progressive update is more suitable.

(You can add more in the list)

@alogfans I have some comments:

1. It separates the memory and file segment desc. I think the Segment Type to classify the Transport is more like "target storage's segment type", nevertheless the source is always D/VRAM.
2. How to classify the local tiering type? For example, the memory is to be auto-tiered into local SSD. There might be 2 segment description, one for the memory type, another for the file type, but there seems to be no specific Transport for it, or it doesn't need at all.

<!-- gh-comment-id:2769688972 --> @yuan-luo commented on GitHub (Apr 1, 2025): > [@yuan-luo](https://github.com/yuan-luo) > I recently reviewed the code in PR https://github.com/kvcache-ai/Mooncake/pull/170 and found that the new implementation may be buggy, so I believe that the progressive update is more suitable. > > (You can add more in the list) @alogfans I have some comments: 1. It separates the memory and file segment desc. I think the Segment Type to classify the Transport is more like "target storage's segment type", nevertheless the source is always D/VRAM. 2. How to classify the local tiering type? For example, the memory is to be auto-tiered into local SSD. There might be 2 segment description, one for the memory type, another for the file type, but there seems to be no specific Transport for it, or it doesn't need at all.

bsakaguchi commented 2026-10-29 21:08:50 +00:00

Author

Owner

Copy link

@yuan-luo commented on GitHub (Apr 1, 2025):

From the perspective of the mooncake store, I think there are a few issues that need to be considered:

• Is it possible to provide a synchronous blocking interface to avoid idle CPU cycles caused by busy-waiting in the store?
• We'd like to improve observability so that when performance or logic issues arise, we can identify the specific module responsible more efficiently.

@xiaguan If I understand correctly, you might mean "provide an asynchronous non-blocking interface" here. It is a good idea to provide such interfaces, while the problem is the KVCache transfer needs to be layer-wised, it must make sure each layer's cache being really transferred completely to start off the next layer's forward. So IMHO, the current public interface design in Mooncake-Store Client is reasonable.
While I agree with you that inside the Client::TransferData, we can improve to have a latch collecting the result of the async batch transfer, which means to refactor the read/write op inside TE into async op, or alternatively, introduce co-routine to do read/write in a sync manner.

<!-- gh-comment-id:2769777087 --> @yuan-luo commented on GitHub (Apr 1, 2025): > From the perspective of the mooncake store, I think there are a few issues that need to be considered: > > * Is it possible to provide a synchronous blocking interface to avoid idle CPU cycles caused by busy-waiting in the store? > * We'd like to improve observability so that when performance or logic issues arise, we can identify the specific module responsible more efficiently. @xiaguan If I understand correctly, you might mean "provide an asynchronous non-blocking interface" here. It is a good idea to provide such interfaces, while the problem is the KVCache transfer needs to be layer-wised, it must make sure each layer's cache being really transferred completely to start off the next layer's forward. So IMHO, the current public interface design in Mooncake-Store Client is reasonable. While I agree with you that inside the Client::TransferData, we can improve to have a latch collecting the result of the async batch transfer, which means to refactor the read/write op inside TE into async op, or alternatively, introduce co-routine to do read/write in a sync manner.

bsakaguchi commented 2026-10-29 21:08:51 +00:00

Author

Owner

Copy link

@xiaguan commented on GitHub (Apr 2, 2025):

With the addition of the coro_rpc library, we've also included async_simple, which offers a hybrid stackless/stacked coroutine async solution. We might be able to leverage this for some improvements.

<!-- gh-comment-id:2771123604 --> @xiaguan commented on GitHub (Apr 2, 2025): With the addition of the `coro_rpc` library, we've also included `async_simple`, which offers a hybrid stackless/stacked coroutine async solution. We might be able to leverage this for some improvements.

bsakaguchi commented 2026-10-29 21:08:51 +00:00

Author

Owner

Copy link

@yuan-luo commented on GitHub (Apr 2, 2025):

In addition, submitTransfer() itself is already non-blocking, which puts the request into an inflight queue. It requires the application to poll the result with getTransferStatus() in a separate loop. So in data path, the current design is fine. While coro_rpc is quite useful in Client-MasterService's meta-data path.

<!-- gh-comment-id:2771201034 --> @yuan-luo commented on GitHub (Apr 2, 2025): In addition, submitTransfer() itself is already non-blocking, which puts the request into an inflight queue. It requires the application to poll the result with getTransferStatus() in a separate loop. So in data path, the current design is fine. While coro_rpc is quite useful in Client-MasterService's meta-data path.

bsakaguchi commented 2026-10-29 21:08:51 +00:00

Author

Owner

Copy link

@xiaguan commented on GitHub (Apr 9, 2025):

I hope that TransferEngine can expose some interfaces for optimization switches, as certain optimizations might be beneficial in Scenario A but counterproductive in Scenario B. Providing these controls would allow callers some room for fine-tuning.

<!-- gh-comment-id:2788149129 --> @xiaguan commented on GitHub (Apr 9, 2025): I hope that TransferEngine can expose some interfaces for optimization switches, as certain optimizations might be beneficial in Scenario A but counterproductive in Scenario B. Providing these controls would allow callers some room for fine-tuning.

bsakaguchi commented 2026-10-29 21:08:51 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jul 21, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:5030322590 --> @github-actions[bot] commented on GitHub (Jul 21, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented 2026-10-29 21:08:51 +00:00

Author

Owner

Copy link

@github-actions[bot] commented on GitHub (Jul 28, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

<!-- gh-comment-id:5100155392 --> @github-actions[bot] commented on GitHub (Jul 28, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

te-coro

staryxchen-tent-capability-research

feature/musa-ep-jit

zxpdemonio/fix-tent-bootstrap-admission

add-store-rs-tone-tests

scatter-rdma-completion-pr2

P2P-Mooncake-Store

release/v0.3.14

feat/tent-rdma-adaptive-adapter

fix/ep-local-counter-gpu-scope

codex/kvcs-oplog-eviction

give-me-a-star

cruz/mooncake-kvcs-lowlevel-pr1-20261008

ylt-handshake-limits

ci/rocm-e2e-tent

sima/docs-pypi-downloads

fix-ci/hicache-mooncake-metadata-loopback

sima/cuda-driver-loading

sima/ci-performance-gate

fix/tent-rdma-unknown-topology-rail-pairing

ci/tent-e2e-dual-job

mc-patch-1

codex/p2p-merge-prerequisites

sima/store-hca-numa-placement

codex/p2p-merge-cli

codex/p2p-merge-ipv6

copilot/fix-ctests-job-failure

czh/pg-device-collectives-pr6

dependabot/cargo/mooncake-store-rs/pyo3-0.29.0

czh/pg-device-collectives-pr5

codex/segment-pool-master-cutover

sima/nightly-ci-recovery

ci/tent-params

stmatengss-patch-1

dependabot/cargo/mooncake-store-rs/rand-0.8.8

dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1

dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16

codex/migrate-store-rest-service

fix/tone-wheel-arch-selection

czh/pg-device-collectives-pr4

czh/pg-device-collectives-pr3

ketong/fix_moe_tc

copilot/fix-test-wheel-ubuntu-job

fix/ci-wheel-torch-pg-pin

codex/migrate-structured-object-storage

codex/migrate-allocator-python

codex/migrate-vllm-python

codex/migrate-shared-segment

ci/fix_sgl_eval_install

codex/segment-pool-nof-placement-cleanup

feat/pip-native-release

copilot/bugfix-ha-standby-promotion

copilot/mooncake-store-transport-settings

copilot/fix-tensor-support-issues

copilot/optimize-rl-direction

copilot/mooncake-storeagentic-workflow

copilot/fix-client-reconnect-timeout

copilot/mooncake-error-debugging

copilot/mooncake-ipv6-support

feat/nccl-host-python-api

dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1

copilot/migrate-and-move-operations

copilot/migrate-and-move-operations-again

refactor/centralize-env-vars

idhanani/mooncake-retain-groups

copilot/codeownerreviewer

copilot/give-feedback-on-issue-2123

copilot/create-issue-rfc

copilot/break-cuda-monopoly-logic

copilot/compare-cudamemcpy-cu12-cu13

copilot/report-ci-error-issue

copilot/research-commit-history-analysis

copilot/explain-mooncake-store-service

copilot/analyze-project-functionality

copilot/sub-pr-1680

copilot/review-pr-1680

copilot/sub-pr-1597

copilot/fix-github-actions-job-failure

copilot/move-test-sglang-integration-job

v0.3.14

v0.3.14-pre-4

v0.3.14-pre-3

v0.3.14-pre-2

v0.3.14-pre

v0.3.14-rc1

v0.3.13.post1

v0.3.13-rc2

v0.3.13-rc1

v0.3.13

v0.3.12.post1

v0.3.12

v0.3.11-rc1

v0.3.11.post1

v0.3.11

v0.3.10.post2

v0.3.10.post1

v0.3.10

v0.3.9

v0.3.8.post1

v0.3.8

v0.3.7.post2

v0.3.7.post1

v0.3.7

v0.3.6.post1

v0.3.6

v0.3.5

v0.3.4.post2

v0.3.4.post1

v0.3.4

v0.3.3.post2

v0.3.3.post1

v0.3.3

v0.3.2.post1

v0.3.2

v0.3.1

v0.3.0

v0.3.0-beta

v0.3.0-alpha

v0.2.0

Labels

Clear labels   

ascend

  

auto-closed

  

bug

  

bug

  

CI/Build

  

Common

  

documentation

  

enhancement

  

feature request

  

good first issue

  

help wanted

  

high priority

  

MetaX

  

nightly-failure

  

pull-request

Mirrored from GitHub Pull Request

  

PyTorch Backend

  

question

  

RFC

  

Roadmap

  

stale

  

Store

  

Tests

  

Transfer Engine

No labels

ascend

auto-closed

bug

bug

CI/Build

Common

documentation

enhancement

feature request

good first issue

help wanted

high priority

MetaX

nightly-failure

pull-request

PyTorch Backend

question

RFC

Roadmap

stale

Store

Tests

Transfer Engine

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/Mooncake#78

No description provided.