[GH-ISSUE #4981] [Bug][PG]: multi-node `init_process_group(backend="mooncake"/"mooncake-cpu")` without `set_host_ip()` publishes 127.0.0.1; rank 0 then hangs forever #1262

New issue

Open

opened 2026-10-29 21:24:35 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:24:35 +00:00

Owner

Originally created by @0z5a on GitHub (Oct 27, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4981

Describe the bug

MooncakePGContext::host_ip defaults to "127.0.0.1" (mooncake-pg/include/mooncake_communicator.h) and is passed unchanged to engine->init(P2PHANDSHAKE, host_ip) (mooncake_communicator.cpp). In P2P-handshake mode the Transfer Engine publishes host_ip:<rpc_port> as its segment name, and the coordinator listens on the same address. Unless the caller invokes mooncake.pg.set_host_ip(<routable IP>) (or set_transfer_engine) before init_process_group, a multi-node job advertises loopback addresses to remote ranks.

Observed on 2 nodes × 1 GPU (aarch64, InfiniBand), mooncake-pg/benchmark/pgbench.py --collective all_reduce --backend mooncake-cpu --device cpu, torchrun-style env (RANK, WORLD_SIZE=2, MASTER_ADDR=<node0 IP>):

rank 1 fails after 30 s: mooncakePgCommCreate failed: operation timed out: timed out waiting for agent registration ... [initialize @ mooncake_communicator.cpp:722]
rank 0 logs Transfer Engine RPC using P2P handshake, listening on 127.0.0.1:<port> and never returns or errors; it had to be killed after 120 s.
The same command with mooncake.pg.set_host_ip(<node IP>) before init_process_group passes (21 sizes × 200 correctness checks, 0 wrong).

Expected behavior

Either derive a routable address by default (as the non-P2P Transfer Engine path does from the hostname, or from the interface that reaches MASTER_ADDR), or fail fast on rank 0 too, with a message that points at set_host_ip. The PG quick start (docs/source/api-reference/python/ep-backend.md) does not mention that set_host_ip is required for multi-node runs.

pgbench.py (and pg_test_utils.py / ep_test_utils.py) map rank → cuda:<rank> or require torch.cuda.device_count() >= world_size, so --device cuda refuses to run on multi-node setups with one GPU per node ("Requested more ranks than available CUDA devices"). Using LOCAL_RANK would fix it.

AI assistance disclosure

Found by AI-assisted code reading (Claude) and confirmed on the hardware above.

Originally created by @0z5a on GitHub (Oct 27, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4981 ### Describe the bug `MooncakePGContext::host_ip` defaults to `"127.0.0.1"` (`mooncake-pg/include/mooncake_communicator.h`) and is passed unchanged to `engine->init(P2PHANDSHAKE, host_ip)` (`mooncake_communicator.cpp`). In P2P-handshake mode the Transfer Engine publishes `host_ip:<rpc_port>` as its segment name, and the coordinator listens on the same address. Unless the caller invokes `mooncake.pg.set_host_ip(<routable IP>)` (or `set_transfer_engine`) before `init_process_group`, a multi-node job advertises loopback addresses to remote ranks. Observed on 2 nodes × 1 GPU (aarch64, InfiniBand), `mooncake-pg/benchmark/pgbench.py --collective all_reduce --backend mooncake-cpu --device cpu`, torchrun-style env (`RANK`, `WORLD_SIZE=2`, `MASTER_ADDR=<node0 IP>`): - rank 1 fails after 30 s: `mooncakePgCommCreate failed: operation timed out: timed out waiting for agent registration ... [initialize @ mooncake_communicator.cpp:722]` - rank 0 logs `Transfer Engine RPC using P2P handshake, listening on 127.0.0.1:<port>` and never returns or errors; it had to be killed after 120 s. - The same command with `mooncake.pg.set_host_ip(<node IP>)` before `init_process_group` passes (21 sizes × 200 correctness checks, 0 wrong). ### Expected behavior Either derive a routable address by default (as the non-P2P Transfer Engine path does from the hostname, or from the interface that reaches `MASTER_ADDR`), or fail fast on rank 0 too, with a message that points at `set_host_ip`. The PG quick start (`docs/source/api-reference/python/ep-backend.md`) does not mention that `set_host_ip` is required for multi-node runs. ### Related harness issue `pgbench.py` (and `pg_test_utils.py` / `ep_test_utils.py`) map rank → `cuda:<rank>` or require `torch.cuda.device_count() >= world_size`, so `--device cuda` refuses to run on multi-node setups with one GPU per node ("Requested more ranks than available CUDA devices"). Using `LOCAL_RANK` would fix it. ### AI assistance disclosure Found by AI-assisted code reading (Claude) and confirmed on the hardware above.

bsakaguchi commented

2026-10-29 21:24:36 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 27, 2026):

Thanks for opening this issue, @0z5a!

Field	Value
Issue	#4981
GitHub user ID	`192209249`
Reporter	@0z5a

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 27, 2026): Thanks for opening this issue, @0z5a! | Field | Value | |-------|-------| | **Issue** | #4981 | | **GitHub user ID** | `192209249` | | **Reporter** | @0z5a | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:55 +00:00

[PR #1262] [MERGED] [Doc] add docs of Mooncake EPD integration with SGLang #2115