[GH-ISSUE #4823] [Docs]: MC_GID_INDEX default is documented as 3, but the code default is automatic selection (-1) #1226

New issue

Closed

opened 2026-10-29 21:24:10 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-29 21:24:10 +00:00

Owner

Originally created by @sivabreddy on GitHub (Oct 21, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4823

Originally assigned to: @alogfans on GitHub.

docs/source/design/transfer-engine/index.md (line 473 on main at 7540f58) says:

MC_GID_INDEX The GID index used per device instance, default value 3 (or the maximum value supported by the platform)

The code default is automatic selection: int gid_index = -1; // -1 for auto-selection in mooncake-transfer-engine/include/config.h, and config.cpp reads MC_GID_INDEX, then NCCL_IB_GID_INDEX when MC_GID_INDEX is not set. The automatic selection in rdma_context.cpp ranks the GIDs and prefers one that has a network device. docs/source/zh_archive/run-examples.md also says the default is 3.

Why it matters: on Kubernetes nodes where several pods share one RDMA device, the GID table holds entries for the interfaces of all pods on the node, so a fixed index (3, as the docs suggest) can pick another pod's GID. In our store-only test (4 clients, 2 per node, 16 keys of 4 MB each), MC_GID_INDEX=3 made 16 of 16 puts fail on every client, and automatic selection made 64 of 64 keys readable. We mentioned this in a comment on #4625; this issue tracks the docs change on its own.

Suggested text:

MC_GID_INDEX (or NCCL_IB_GID_INDEX when MC_GID_INDEX is not set): GID index per device. Default: automatic selection, which prefers a GID that has a network device. Set a fixed index only when the automatic selection picks a wrong GID; with several containers on one RDMA device, the GID table also holds entries for the other containers.

We can send this as a docs PR if you prefer.

Originally created by @sivabreddy on GitHub (Oct 21, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4823 Originally assigned to: @alogfans on GitHub. `docs/source/design/transfer-engine/index.md` (line 473 on `main` at 7540f58) says: > `MC_GID_INDEX` The GID index used per device instance, default value 3 (or the maximum value supported by the platform) The code default is automatic selection: `int gid_index = -1; // -1 for auto-selection` in `mooncake-transfer-engine/include/config.h`, and `config.cpp` reads `MC_GID_INDEX`, then `NCCL_IB_GID_INDEX` when `MC_GID_INDEX` is not set. The automatic selection in `rdma_context.cpp` ranks the GIDs and prefers one that has a network device. `docs/source/zh_archive/run-examples.md` also says the default is 3. Why it matters: on Kubernetes nodes where several pods share one RDMA device, the GID table holds entries for the interfaces of all pods on the node, so a fixed index (3, as the docs suggest) can pick another pod's GID. In our store-only test (4 clients, 2 per node, 16 keys of 4 MB each), `MC_GID_INDEX=3` made 16 of 16 puts fail on every client, and automatic selection made 64 of 64 keys readable. We mentioned this in a comment on #4625; this issue tracks the docs change on its own. Suggested text: > `MC_GID_INDEX` (or `NCCL_IB_GID_INDEX` when `MC_GID_INDEX` is not set): GID index per device. Default: automatic selection, which prefers a GID that has a network device. Set a fixed index only when the automatic selection picks a wrong GID; with several containers on one RDMA device, the GID table also holds entries for the other containers. We can send this as a docs PR if you prefer.

bsakaguchi closed this issue

2026-10-29 21:24:10 +00:00

bsakaguchi commented

2026-10-29 21:24:11 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 21, 2026):

Thanks for opening this issue, @sivabreddy!

Field	Value
Issue	#4823
GitHub user ID	`113816799`
Reporter	@sivabreddy

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 21, 2026): Thanks for opening this issue, @sivabreddy! | Field | Value | |-------|-------| | **Issue** | #4823 | | **GitHub user ID** | `113816799` | | **Reporter** | @sivabreddy | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi referenced this issue

2026-10-29 21:32:41 +00:00

[PR #1226] [MERGED] Add Local Cache Mechanism for Mooncake Store Client #2088