[GH-ISSUE #4934] [Bug]: TENT hp_tcp is ~40% slower than the classic (non-TENT) TCP transport with the same number of connections #1254

New issue

Closed

opened 2026-10-29 21:24:27 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:24:27 +00:00

Owner

Originally created by @leoleoasd on GitHub (Oct 26, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4934

Originally assigned to: @alogfans on GitHub.

Bug Report

Describe your performance question

On the same two hosts, with the same number of TCP connections and the same workload, TENT's hp_tcp transport reaches only about 60% of the throughput of the classic TCP transport (MC_USE_TENT unset). Its throughput also varies a lot from run to run with an identical configuration.

Environment

Mooncake upstream main d53ddb9f, built with -DUSE_TENT=ON, host (DRAM) buffers.
Two VMs (AMD Genoa, 128 vCPU) on a VXLAN pod network, MTU 1450. iperf3 reaches ~21.7 GB/s with 32 streams and ~1.2 GB/s with one stream.

hp_tcp: MC_USE_TENT=1, MC_TENT_CONF set to:

{"transports": {"tcp": {"enable": false}, "rdma": {"enable": false},
                "hp_tcp": {"enable": true, "worker_count": 16, "connections_per_peer": 16}}}

Classic tcp: MC_FORCE_TCP=1, MC_NUM_TCP_IO_THREADS=16, MC_TCP_LANES_PER_PEER=16.

Workload

One sender process writes 32 GB into one receiver process as 16 MB WRITEs. It keeps a fixed number of writes in flight and submits a new one each time one completes:

pending = []
while todo or pending:
    while todo and len(pending) < INFLIGHT:
        local, remote, n = todo.pop()
        pending.append((te.transfer_submit_write(session, local, remote, n), n))
    pending = [(b, n) for b, n in pending if te.transfer_check_status(b) == 0]
    time.sleep(200e-6)

Results (GB/s, one process pair, 16 connections, 16 MB writes)

In flight	classic tcp	hp_tcp
16	13.7	7.9–9.2
64	11.7–13.5 (5 runs)	7.3–8.2 (5 runs); other sessions ranged 5.6–8.9, occasionally 13.9–15

With two process pairs, classic tcp reaches 19.8 GB/s and hp_tcp 10.9 GB/s. IPerf3 shows about 21.7GB/s ceiling.

Pushing a PR soon

Before submitting...

Ensure you searched for relevant issues and read the [documentation]

Originally created by @leoleoasd on GitHub (Oct 26, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4934 Originally assigned to: @alogfans on GitHub. ### Bug Report **Describe your performance question** On the same two hosts, with the same number of TCP connections and the same workload, TENT's `hp_tcp` transport reaches only about 60% of the throughput of the classic TCP transport (`MC_USE_TENT` unset). Its throughput also varies a lot from run to run with an identical configuration. **Environment** - Mooncake upstream main d53ddb9f, built with `-DUSE_TENT=ON`, host (DRAM) buffers. - Two VMs (AMD Genoa, 128 vCPU) on a VXLAN pod network, MTU 1450. iperf3 reaches ~21.7 GB/s with 32 streams and ~1.2 GB/s with one stream. - **hp_tcp:** `MC_USE_TENT=1`, `MC_TENT_CONF` set to: ```json {"transports": {"tcp": {"enable": false}, "rdma": {"enable": false}, "hp_tcp": {"enable": true, "worker_count": 16, "connections_per_peer": 16}}} ``` - **Classic tcp:** `MC_FORCE_TCP=1`, `MC_NUM_TCP_IO_THREADS=16`, `MC_TCP_LANES_PER_PEER=16`. **Workload** One sender process writes 32 GB into one receiver process as 16 MB WRITEs. It keeps a fixed number of writes in flight and submits a new one each time one completes: ```python pending = [] while todo or pending: while todo and len(pending) < INFLIGHT: local, remote, n = todo.pop() pending.append((te.transfer_submit_write(session, local, remote, n), n)) pending = [(b, n) for b, n in pending if te.transfer_check_status(b) == 0] time.sleep(200e-6) ``` **Results** (GB/s, one process pair, 16 connections, 16 MB writes) | In flight | classic tcp | hp_tcp | |---|---|---| | 16 | 13.7 | 7.9–9.2 | | 64 | 11.7–13.5 (5 runs) | 7.3–8.2 (5 runs); other sessions ranged 5.6–8.9, occasionally 13.9–15 | With two process pairs, classic tcp reaches 19.8 GB/s and hp_tcp 10.9 GB/s. IPerf3 shows about 21.7GB/s ceiling. Pushing a PR soon ### Before submitting... - [ ] Ensure you searched for relevant issues and read the [documentation]

bsakaguchi

2026-10-29 21:24:27 +00:00

closed this issue
added the
bug
label

bsakaguchi commented

2026-10-29 21:24:28 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 26, 2026):

Thanks for opening this issue, @leoleoasd!

Field	Value
Issue	#4934
GitHub user ID	`37735580`
Reporter	@leoleoasd

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 26, 2026): Thanks for opening this issue, @leoleoasd! | Field | Value | |-------|-------| | **Issue** | #4934 | | **GitHub user ID** | `37735580` | | **Reporter** | @leoleoasd | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.