[GH-ISSUE #553] Can mooncake-store use both RDMA and TCP transports concurrently, and how to configure fallback? #187

New issue

Closed

opened 2026-10-29 21:10:39 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 21:10:39 +00:00

Owner

Originally created by @wwq2333 on GitHub (Jun 25, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/553

Hi Mooncake team,
We have a deployment scenario in which several clients run on hosts equipped with RDMA, while others can only use TCP.
Ideally we would like mooncake-store to:

Prefer RDMA when both peers support it, for best performance;
Gracefully fall back to TCP whenever one side (or the network path) does not support RDMA.

To test this, we built Mooncake tags: v0.3.4 and started two clients:

Client-A registers its segment using protocol rdma (segment size = 1 GB)

store.setup(
        "${clienta_ip}",
        "${etcd_ip}:2379",
        1073741824,
        1073741824,
        "rdma",
        "mlx5_0",
        "${master_ip}:50051",
    )

Client-B registers its segment using protocol tcp (segment size = 0 B)

store.setup(
        "${clientb_ip}",
        "${etcd_ip}:2379",
        0,
        1073741824,
        "tcp",
        "",
        "${master_ip}:50051",
    )

When Client-B performs a simple put, the request fails with the following logs:

E0625 07:33:59.890619  7082 transfer_task.cpp:380] Failed to submit all transfers, error code is NotSupportedTransport
E0625 07:33:59.890625  7082 client.cpp:670] Failed to submit transfer operation
E0625 07:33:59.890764  7082 store_py.cpp:371] Put operation failed with error: TRANSFER_FAIL

Questions

Does mooncake-store currently support installing both rdma and tcp transports at the same time?
If yes, what is the correct way to set up the metadata/topology so that
- RDMA-capable clients talk to each other via RDMA, and
- any client can still communicate over TCP when RDMA is unavailable?

Originally created by @wwq2333 on GitHub (Jun 25, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/553 Hi Mooncake team, We have a deployment scenario in which several clients run on hosts equipped with RDMA, while others can only use TCP. Ideally we would like mooncake-store to: 1. Prefer RDMA when both peers support it, for best performance; 2. Gracefully fall back to TCP whenever one side (or the network path) does not support RDMA. To test this, we built Mooncake tags: v0.3.4 and started two clients: - Client-A registers its segment using protocol rdma (segment size = 1 GB) ``` store.setup( "${clienta_ip}", "${etcd_ip}:2379", 1073741824, 1073741824, "rdma", "mlx5_0", "${master_ip}:50051", ) ``` - Client-B registers its segment using protocol tcp (segment size = 0 B) ``` store.setup( "${clientb_ip}", "${etcd_ip}:2379", 0, 1073741824, "tcp", "", "${master_ip}:50051", ) ``` When Client-B performs a simple put, the request fails with the following logs: ``` E0625 07:33:59.890619 7082 transfer_task.cpp:380] Failed to submit all transfers, error code is NotSupportedTransport E0625 07:33:59.890625 7082 client.cpp:670] Failed to submit transfer operation E0625 07:33:59.890764 7082 store_py.cpp:371] Put operation failed with error: TRANSFER_FAIL ``` **Questions** 1. Does mooncake-store currently support installing both rdma and tcp transports at the same time? 2. If yes, what is the correct way to set up the metadata/topology so that - RDMA-capable clients talk to each other via RDMA, and - any client can still communicate over TCP when RDMA is unavailable?

bsakaguchi

2026-10-29 21:10:39 +00:00

closed this issue
added the
stale

auto-closed
labels

bsakaguchi commented

2026-10-29 21:10:40 +00:00

Author

Owner

@alogfans commented on GitHub (Jun 26, 2025):

Currently Mooncake doesn't support this network architecture. We recommend you use RDMA network only for better performance.

@alogfans commented on GitHub (Jun 26, 2025): Currently Mooncake doesn't support this network architecture. We recommend you use RDMA network only for better performance.

bsakaguchi commented

2026-10-29 21:10:40 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 3, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

@github-actions[bot] commented on GitHub (Jul 3, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

bsakaguchi commented

2026-10-29 21:10:40 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Jul 10, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

@github-actions[bot] commented on GitHub (Jul 10, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.