kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #5077] [Bug]: A3 Host registration breaks HBM reads with empty IPC key; buffered fallback rejected by HIXL CS #1279

New issue

Open

opened 2026-10-29 21:24:45 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:24:45 +00:00

Owner

Originally created by @lang-free on GitHub (Oct 29, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/5077

We reproduced a failure on Ascend A3 where adding a small Host allocation to a TransferEngine makes an otherwise successful HBM-to-HBM read fail with an empty IPC key. No model or router is needed to reproduce it.

This is related to #4916's buffered/CS backend selection question, but the failure stage differs: our small Host registration succeeds, then connection setup for a subsequent HBM read fails. We are not attributing this to the large-MR registration failure in that report.

Environment

Atlas 800I A3; the standalone comparison uses two processes on NPU 0 and 1 of one machine.
Installed Mooncake distribution version: 0.3.13.
CANN toolkit directory: cann-9.2.0; installed HIXL and Hcomm version.info files report 9.2.1, build timestamp 20261021_221618020.
HDK/driver: 25.5.2.
Local derivative image, not an untouched upstream image. The serving stack has Python integration changes, but the standalone test calls Mooncake directly.
Transport: ascend, P2PHANDSHAKE, HCCS path. No RoCE override in this comparison.
Probe environment: ASCEND_ENABLE_USE_FABRIC_MEM=0, ASCEND_GLOBAL_RESOURCE_CONFIG={"comm_resource_config.qos":1}, PYTORCH_NPU_ALLOC_CONF=expandable_segments:True; ASCEND_BUFFER_POOL unset.
Container uses host networking, host IPC, privileged mode, and the host Ascend driver mount.

The Fabric setting above describes the actual probe environment. We are not assuming every Store code path interprets a present variable with value 0 as disabled.

Reproduction sequence

Use separate spawned processes and keep all allocations alive until the transfer finishes:

Each process selects its NPU and allocates a 2 MiB torch.uint8 device tensor. Fill the source with 17 and the destination with 0, then synchronize.
Each creates a TransferEngine, calls initialize(host_ip, "P2PHANDSHAKE", "ascend", ""), and registers its device tensor with register_memory(ptr, size). Check each return value.
For the failing case only, the source also allocates 2 MiB Host RAM and registers it on the same engine, before any peer connection. We tested both posix_memalign and aclrtMallocHost; both fail at the later transfer step.
Pass the source engine's get_rpc_port() and device tensor pointer to the destination through a multiprocessing pipe.
The destination calls:

ret = engine.batch_transfer_sync_read(
    f"{host_ip}:{source_rpc_port}",
    [destination_device_ptr],
    [source_device_ptr],
    [2 * 1024 * 1024],
)

The requested transfer is HBM-to-HBM in both cases. The three argument lists each contain one item; no Host address is in that read request.

Results

Source registration before first connection	Read result
HBM only	`ret=0`, destination bytes verified as 17
HBM + 2 MiB `posix_memalign` Host allocation, same engine	`ret=-1`, empty IPC key / 107000
HBM + 2 MiB `aclrtMallocHost` allocation, same engine	Same failure
Register Host RAM, then unregister it before connecting	`ret=0`, bytes correct
Register Host RAM on a second engine; read HBM through the first	`ret=0`, bytes correct
Read Host RAM through that second engine	Empty IPC key / 107000
Attach a real Store to the shared engine, 64 MiB global segment + 1 MiB local buffer	HBM read fails with the same error

Store setup with only the 1 MiB local buffer, or only the 64 MiB global segment, also reproduces the failure. An empty transfer-list control returns success without moving data or producing the IPC error.

Sanitized error chain:

HIXL AutoConnect
  -> HcommChannelCreate
  -> HccsRegedMemMgr::MemoryOpenRemoteIpc
  -> RemoteIpcRmaBufferImpl::Open
  -> aclrtIpcMemImportByKey failed. return[107000], name[]

In the original DeepSeek PD deployment, the same failure occurred with nonempty read descriptors (op_descs size:50), after both workers had registered HBM and Host regions. It also appeared on a Store write (op_descs size:12). The standalone comparison removes the model, scheduler and router from the reproduction.

Buffered path and backend selection

Our older deployment used:

export ASCEND_BUFFER_POOL=4:8
export ASCEND_ENABLE_USE_FABRIC_MEM=0

Its logs showed Ignore register host mem ... when buffer pool is enabled, and PD transfers completed. This is historical evidence, not a controlled version-only comparison.

On the newer stack, keeping 4:8 fails during initialization:

Invalid option fields, OPTION_BUFFER_POOL for hixl engine only supports 0:0

The Mooncake executor source we inspected derives client_server_mode from capability detection and automatically supplies adxl.LocalCommRes={"version":"1.3"} when no explicit value was provided. That branch does not check whether a nonzero buffer pool was requested. This appears relevant to why merely restoring the previous buffer-pool setting fails; we have not validated a fix.

Possible lower-level error-handling defect

The public Hcomm IPC code has a separate field propagation gap:

LocalIpcRmaBufferImpl::Init() generates an IPC key for Device memory, while its Host branch maps the local address without generating that key.
RemoteIpcRmaBuffer::Deserialize() reads the memory type into the outer object but does not propagate it to the inner implementation.
RemoteIpcRmaBufferImpl::Open() checks the inner memory type for HOST. That field stays at its constructor default TYPE_NUM, so the Host rejection can be bypassed and an empty IPC key reaches the import call.

We checked the corresponding constructor, deserialize and open instructions in the deployed libhccl_plf.so, not just public master. The same field layout and missing propagation were present. Fixing this check alone would give an earlier rejection; it would not provide Host-over-HCCS support.

Source references: local registration, outer deserialization, inner open.

Questions

What is the supported A3 configuration for a Mooncake Host RAM pool over HCCS when Fabric memory is unavailable? Can the newer stack still select the buffered CommEngine path?
Should Mooncake's automatic CS selection honor a nonzero ASCEND_BUFFER_POOL, or reject the combination before initializing HIXL with conflicting options?
Should unrelated Host registrations be imported when connecting for an HBM-only read, and is there an existing fix for the Hcomm memory-type propagation gap?

We have not tested a patched backend selector or claimed a working fix. Separating engines protects the HBM-only path in the comparison, but does not make the Host pool readable over the failing path.

Originally created by @lang-free on GitHub (Oct 29, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/5077 We reproduced a failure on Ascend A3 where adding a small Host allocation to a TransferEngine makes an otherwise successful HBM-to-HBM read fail with an empty IPC key. No model or router is needed to reproduce it. This is related to #4916's buffered/CS backend selection question, but the failure stage differs: our small Host registration succeeds, then connection setup for a subsequent HBM read fails. We are not attributing this to the large-MR registration failure in that report. ## Environment - Atlas 800I A3; the standalone comparison uses two processes on NPU 0 and 1 of one machine. - Installed Mooncake distribution version: 0.3.13. - CANN toolkit directory: `cann-9.2.0`; installed HIXL and Hcomm `version.info` files report **9.2.1**, build timestamp `20261021_221618020`. - HDK/driver: 25.5.2. - Local derivative image, not an untouched upstream image. The serving stack has Python integration changes, but the standalone test calls Mooncake directly. - Transport: `ascend`, `P2PHANDSHAKE`, HCCS path. No RoCE override in this comparison. - Probe environment: `ASCEND_ENABLE_USE_FABRIC_MEM=0`, `ASCEND_GLOBAL_RESOURCE_CONFIG={"comm_resource_config.qos":1}`, `PYTORCH_NPU_ALLOC_CONF=expandable_segments:True`; `ASCEND_BUFFER_POOL` unset. - Container uses host networking, host IPC, privileged mode, and the host Ascend driver mount. The Fabric setting above describes the actual probe environment. We are not assuming every Store code path interprets a present variable with value `0` as disabled. ## Reproduction sequence Use separate spawned processes and keep all allocations alive until the transfer finishes: 1. Each process selects its NPU and allocates a 2 MiB `torch.uint8` device tensor. Fill the source with 17 and the destination with 0, then synchronize. 2. Each creates a `TransferEngine`, calls `initialize(host_ip, "P2PHANDSHAKE", "ascend", "")`, and registers its device tensor with `register_memory(ptr, size)`. Check each return value. 3. For the failing case only, the source also allocates 2 MiB Host RAM and registers it on **the same engine**, before any peer connection. We tested both `posix_memalign` and `aclrtMallocHost`; both fail at the later transfer step. 4. Pass the source engine's `get_rpc_port()` and device tensor pointer to the destination through a multiprocessing pipe. 5. The destination calls: ```python ret = engine.batch_transfer_sync_read( f"{host_ip}:{source_rpc_port}", [destination_device_ptr], [source_device_ptr], [2 * 1024 * 1024], ) ``` The requested transfer is HBM-to-HBM in both cases. The three argument lists each contain one item; no Host address is in that read request. ## Results | Source registration before first connection | Read result | |---|---| | HBM only | `ret=0`, destination bytes verified as 17 | | HBM + 2 MiB `posix_memalign` Host allocation, same engine | `ret=-1`, empty IPC key / 107000 | | HBM + 2 MiB `aclrtMallocHost` allocation, same engine | Same failure | | Register Host RAM, then unregister it before connecting | `ret=0`, bytes correct | | Register Host RAM on a second engine; read HBM through the first | `ret=0`, bytes correct | | Read Host RAM through that second engine | Empty IPC key / 107000 | | Attach a real Store to the shared engine, 64 MiB global segment + 1 MiB local buffer | HBM read fails with the same error | Store setup with only the 1 MiB local buffer, or only the 64 MiB global segment, also reproduces the failure. An empty transfer-list control returns success without moving data or producing the IPC error. Sanitized error chain: ```text HIXL AutoConnect -> HcommChannelCreate -> HccsRegedMemMgr::MemoryOpenRemoteIpc -> RemoteIpcRmaBufferImpl::Open -> aclrtIpcMemImportByKey failed. return[107000], name[] ``` In the original DeepSeek PD deployment, the same failure occurred with nonempty read descriptors (`op_descs size:50`), after both workers had registered HBM and Host regions. It also appeared on a Store write (`op_descs size:12`). The standalone comparison removes the model, scheduler and router from the reproduction. ## Buffered path and backend selection Our older deployment used: ```bash export ASCEND_BUFFER_POOL=4:8 export ASCEND_ENABLE_USE_FABRIC_MEM=0 ``` Its logs showed `Ignore register host mem ... when buffer pool is enabled`, and PD transfers completed. This is historical evidence, not a controlled version-only comparison. On the newer stack, keeping `4:8` fails during initialization: ```text Invalid option fields, OPTION_BUFFER_POOL for hixl engine only supports 0:0 ``` The Mooncake executor source we inspected derives `client_server_mode` from capability detection and automatically supplies `adxl.LocalCommRes={"version":"1.3"}` when no explicit value was provided. That branch does not check whether a nonzero buffer pool was requested. This appears relevant to why merely restoring the previous buffer-pool setting fails; we have not validated a fix. ## Possible lower-level error-handling defect The public Hcomm IPC code has a separate field propagation gap: - `LocalIpcRmaBufferImpl::Init()` generates an IPC key for Device memory, while its Host branch maps the local address without generating that key. - `RemoteIpcRmaBuffer::Deserialize()` reads the memory type into the outer object but does not propagate it to the inner implementation. - `RemoteIpcRmaBufferImpl::Open()` checks the inner memory type for HOST. That field stays at its constructor default `TYPE_NUM`, so the Host rejection can be bypassed and an empty IPC key reaches the import call. We checked the corresponding constructor, deserialize and open instructions in the deployed `libhccl_plf.so`, not just public master. The same field layout and missing propagation were present. Fixing this check alone would give an earlier rejection; it would not provide Host-over-HCCS support. Source references: [local registration](https://github.com/hicann/hcomm/blob/master/src/legacy/ascend910/platform/resource/rma_buffer/local_ipc_rma_buffer_impl.cc), [outer deserialization](https://github.com/hicann/hcomm/blob/master/src/legacy/ascend910/platform/resource/rma_buffer/remote_ipc_rma_buffer.cc), [inner open](https://github.com/hicann/hcomm/blob/master/src/legacy/ascend910/platform/resource/rma_buffer/remote_ipc_rma_buffer_impl.cc). ## Questions 1. What is the supported A3 configuration for a Mooncake Host RAM pool over HCCS when Fabric memory is unavailable? Can the newer stack still select the buffered CommEngine path? 2. Should Mooncake's automatic CS selection honor a nonzero `ASCEND_BUFFER_POOL`, or reject the combination before initializing HIXL with conflicting options? 3. Should unrelated Host registrations be imported when connecting for an HBM-only read, and is there an existing fix for the Hcomm memory-type propagation gap? We have not tested a patched backend selector or claimed a working fix. Separating engines protects the HBM-only path in the comparison, but does not make the Host pool readable over the failing path.

bsakaguchi commented

2026-10-29 21:24:46 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Oct 29, 2026):

Thanks for opening this issue, @lang-free!

Field	Value
Issue	#5077
GitHub user ID	`143081744`
Reporter	@lang-free

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Oct 29, 2026): Thanks for opening this issue, @lang-free! | Field | Value | |-------|-------| | **Issue** | #5077 | | **GitHub user ID** | `143081744` | | **Reporter** | @lang-free | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:24:47 +00:00

Author

Owner

@lang-free commented on GitHub (Oct 29, 2026):

Adding an A2 deployment comparison, checked on 2026-10-29.

We inspected four running A2 nodes (8 × Ascend 910B4-1 per node), serving GLM-5.3-Flash with two prefill nodes and two decode nodes. All four model containers report mooncake-transfer-engine==0.3.13; the installed package metadata does not record a source commit.

The serving-process environment is consistent across the four nodes:

ASCEND_BUFFER_POOL=0:0
HCCL_INTRA_ROCE_ENABLE=1
ASCEND_GLOBAL_RESOURCE_CONFIG=<unset>
ASCEND_LOCAL_COMM_RES=<unset>
ASCEND_USE_ASYNC_TRANSFER=<unset>

The Store configuration uses protocol=ascend, metadata_server=P2PHANDSHAKE, global_segment_size=72GB, and SSD offload. The serving integration combines MooncakeConnectorV2 with AscendStoreConnector.

Earlier serving probes on this deployment completed successfully. A correlated P→D request logged four successful KV read operations taking 5.98, 4.49, 3.51 and 3.47 ms. This deployment is not relying on the historical 4:8 buffer-pool workaround.

This is configuration and serving evidence, not a reproduction of the standalone HBM-plus-Host test above. We have not isolated Host-pool reads/writes in this check or verified the selected transport for every registration. HCCL_INTRA_ROCE_ENABLE=1 alone does not establish that all paths avoid HCCS/IPC. The same package version also does not establish identical binaries or HIXL/Hcomm versions.

@lang-free commented on GitHub (Oct 29, 2026): Adding an A2 deployment comparison, checked on 2026-10-29. We inspected four running A2 nodes (8 × Ascend 910B4-1 per node), serving GLM-5.3-Flash with two prefill nodes and two decode nodes. All four model containers report `mooncake-transfer-engine==0.3.13`; the installed package metadata does not record a source commit. The serving-process environment is consistent across the four nodes: ```text ASCEND_BUFFER_POOL=0:0 HCCL_INTRA_ROCE_ENABLE=1 ASCEND_GLOBAL_RESOURCE_CONFIG=<unset> ASCEND_LOCAL_COMM_RES=<unset> ASCEND_USE_ASYNC_TRANSFER=<unset> ``` The Store configuration uses `protocol=ascend`, `metadata_server=P2PHANDSHAKE`, `global_segment_size=72GB`, and SSD offload. The serving integration combines `MooncakeConnectorV2` with `AscendStoreConnector`. Earlier serving probes on this deployment completed successfully. A correlated P→D request logged four successful KV read operations taking 5.98, 4.49, 3.51 and 3.47 ms. This deployment is not relying on the historical `4:8` buffer-pool workaround. This is configuration and serving evidence, not a reproduction of the standalone HBM-plus-Host test above. We have not isolated Host-pool reads/writes in this check or verified the selected transport for every registration. `HCCL_INTRA_ROCE_ENABLE=1` alone does not establish that all paths avoid HCCS/IPC. The same package version also does not establish identical binaries or HIXL/Hcomm versions.

bsakaguchi referenced this issue

2026-10-29 21:33:00 +00:00

[PR #1279] [MERGED] [Store]: Update metadata callback sync strategy for tiered backend #2129