[GH-ISSUE #4046] [Bug]: Ascend transfer engine fails to register host memory (wildcard probe hard-error + misaligned malloc fallback) #1028

New issue

Closed

opened 2026-10-29 21:22:08 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 21:22:08 +00:00

Owner

Originally created by @gygdh-001 on GitHub (Sep 11, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4046

Environment

Hardware: Ascend 910B (NPU)
Mooncake: kvcache-ai/Mooncake main (mooncake-transfer-engine-npu wheel / source build)
Protocol: ascend / rdma / tcp (host-memory buffers)

Description

Two defects prevent host (CPU) memory from being registered/transferred on the Ascend
transfer engine:

1. Wildcard location probe failure is a hard error instead of falling back to host memory

AscendDirectTransport::ResolveAscendMemType() in
mooncake-transfer-engine/src/transport/ascend_transport/ascend_direct_transport/ascend_direct_transport.cpp
uses CHECK_ACL(aclrtPointerGetAttributes(addr, &attributes)) for the wildcard
location "*". When the buffer is host memory that is not managed by ACL (hugepage,
shm, or plain malloc), aclrtPointerGetAttributes fails, so ResolveAscendMemType
returns an error and registration fails (in some code paths this surfaces as
location:* is not supported).

CUDA already falls back to host memory when the wildcard probe cannot identify the
address; Ascend should behave the same way. The same pattern was previously applied to
the heterogeneous RDMA transport in PR #1657.

2. Default allocator is not 64KB-aligned, breaking Ascend SVM

initMemoryAllocator() in
mooncake-integration/transfer_engine/transfer_engine_py.cpp falls back to plain
malloc for protocols such as rdma/tcp/ascend. Ascend SVM
(_devmm_mem_remote_map) requires a 64KB page-aligned src_va (page_size == 65536);
an unaligned malloc address makes aclrtHostRegister fail with EINVAL
("Src_va is zero or not page alignment"), cascading into HCCL HcclMemRegRoce /
GE channel errors (e.g. 503900).

Steps to reproduce

Build the Ascend wheel / enable the Ascend transport.
Mount a CPU hugepage/shm buffer (or call allocate_managed_buffer on a host buffer
with protocol tcp/ascend) so that a host buffer is registered with the wildcard
location "*".
Observe the wildcard probe failure (aclrtPointerGetAttributes error / location:* is not supported), and, on the cross-card SVM path, the aclrtHostRegister EINVAL
alignment failure.

Expected behavior

The Ascend wildcard location should fall back to host memory when the probe cannot
identify the address (matching CUDA and the heterogeneous-RDMA transport).
The default allocator should return 64KB-aligned memory so Ascend SVM can map it.

Proposed fix

See the linked PR. Two surgical changes:

ResolveAscendMemType(): treat aclrtPointerGetAttributes failure as host memory
instead of a hard error.
initMemoryAllocator(): use posix_memalign(64 * 1024, ...) instead of malloc.

Originally created by @gygdh-001 on GitHub (Sep 11, 2026). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/4046 ### Environment - Hardware: Ascend 910B (NPU) - Mooncake: `kvcache-ai/Mooncake` `main` (`mooncake-transfer-engine-npu` wheel / source build) - Protocol: `ascend` / `rdma` / `tcp` (host-memory buffers) ### Description Two defects prevent host (CPU) memory from being registered/transferred on the Ascend transfer engine: **1. Wildcard location probe failure is a hard error instead of falling back to host memory** `AscendDirectTransport::ResolveAscendMemType()` in `mooncake-transfer-engine/src/transport/ascend_transport/ascend_direct_transport/ascend_direct_transport.cpp` uses `CHECK_ACL(aclrtPointerGetAttributes(addr, &attributes))` for the wildcard location `"*"`. When the buffer is host memory that is not managed by ACL (hugepage, shm, or plain `malloc`), `aclrtPointerGetAttributes` fails, so `ResolveAscendMemType` returns an error and registration fails (in some code paths this surfaces as `location:* is not supported`). CUDA already falls back to host memory when the wildcard probe cannot identify the address; Ascend should behave the same way. The same pattern was previously applied to the heterogeneous RDMA transport in PR #1657. **2. Default allocator is not 64KB-aligned, breaking Ascend SVM** `initMemoryAllocator()` in `mooncake-integration/transfer_engine/transfer_engine_py.cpp` falls back to plain `malloc` for protocols such as `rdma`/`tcp`/`ascend`. Ascend SVM (`_devmm_mem_remote_map`) requires a 64KB page-aligned `src_va` (`page_size == 65536`); an unaligned `malloc` address makes `aclrtHostRegister` fail with EINVAL ("Src_va is zero or not page alignment"), cascading into `HCCL HcclMemRegRoce` / `GE channel` errors (e.g. 503900). ### Steps to reproduce 1. Build the Ascend wheel / enable the Ascend transport. 2. Mount a CPU hugepage/shm buffer (or call `allocate_managed_buffer` on a host buffer with protocol `tcp`/`ascend`) so that a host buffer is registered with the wildcard location `"*"`. 3. Observe the wildcard probe failure (`aclrtPointerGetAttributes` error / `location:* is not supported`), and, on the cross-card SVM path, the `aclrtHostRegister` EINVAL alignment failure. ### Expected behavior - The Ascend wildcard location should fall back to host memory when the probe cannot identify the address (matching CUDA and the heterogeneous-RDMA transport). - The default allocator should return 64KB-aligned memory so Ascend SVM can map it. ### Proposed fix See the linked PR. Two surgical changes: 1. `ResolveAscendMemType()`: treat `aclrtPointerGetAttributes` failure as host memory instead of a hard error. 2. `initMemoryAllocator()`: use `posix_memalign(64 * 1024, ...)` instead of `malloc`.

bsakaguchi closed this issue

2026-10-29 21:22:08 +00:00

bsakaguchi commented

2026-10-29 21:22:09 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Sep 11, 2026):

Thanks for opening this issue, @gygdh-001!

Field	Value
Issue	#4046
GitHub user ID	`43057576`
Reporter	@gygdh-001

A maintainer will triage this when possible. To help us respond faster, please include:

Mooncake version or commit SHA
Environment (OS, CUDA/driver, RDMA stack if relevant)
Steps to reproduce and expected vs. actual behavior

Useful links: Documentation · Contributing guide

This message was posted automatically by the issue bot.

@github-actions[bot] commented on GitHub (Sep 11, 2026): Thanks for opening this issue, @gygdh-001! | Field | Value | |-------|-------| | **Issue** | #4046 | | **GitHub user ID** | `43057576` | | **Reporter** | @gygdh-001 | A maintainer will triage this when possible. To help us respond faster, please include: - Mooncake version or commit SHA - Environment (OS, CUDA/driver, RDMA stack if relevant) - Steps to reproduce and expected vs. actual behavior Useful links: [Documentation](https://kvcache-ai.github.io/Mooncake/) · [Contributing guide](https://github.com/kvcache-ai/Mooncake/blob/main/CONTRIBUTING.md) > This message was posted automatically by the issue bot.

bsakaguchi commented

2026-10-29 21:22:09 +00:00

Author

Owner

@gygdh-001 commented on GitHub (Sep 11, 2026):

Superseded - split into #4054 (wildcard probe) and #4055 (64KB alignment).

@gygdh-001 commented on GitHub (Sep 11, 2026): Superseded - split into #4054 (wildcard probe) and #4055 (64KB alignment).

bsakaguchi referenced this issue

2026-10-29 21:31:45 +00:00

[PR #1028] [MERGED] [store] Add disk eviction feature #1965