kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[PR #502] [MERGED] [TransferEngine] Enable Huawei Ascend Transport for TransferEngine #1610

New issue

Closed

opened 2026-10-29 21:28:53 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 21:28:53 +00:00

Owner

📋 Pull Request Information

Original PR: https://github.com/kvcache-ai/Mooncake/pull/502
Author: @AscendTransport
Created: 6/16/2025
Status: ✅ Merged
Merged: 7/1/2025
Merged by: @ShangmingCai

Base: main ← Head: main

📝 Commits (4)

b2879cb Ascend Transport
50b467e Merge branch 'main' into main
0eaa333 fix ci
930ae39 reviewed changes

📊 Changes

27 files changed (+3557 additions, -0 deletions)

View changed files

➕ doc/en/ascend_transport.md (+145 -0)
➕ doc/zh/ascend_transport.md (+94 -0)
📝 mooncake-common/common.cmake (+17 -0)
📝 mooncake-integration/transfer_engine/transfer_engine_py.cpp (+2 -0)
📝 mooncake-transfer-engine/CMakeLists.txt (+22 -0)
📝 mooncake-transfer-engine/example/CMakeLists.txt (+13 -0)
➕ mooncake-transfer-engine/example/transfer_engine_ascend_one_sided.cpp (+421 -0)
➕ mooncake-transfer-engine/example/transfer_engine_ascend_perf.cpp (+354 -0)
📝 mooncake-transfer-engine/include/common.h (+40 -0)
📝 mooncake-transfer-engine/include/transfer_metadata.h (+14 -0)
➕ mooncake-transfer-engine/include/transport/ascend_transport/hccl_transport/hccl_transport.h (+102 -0)
➕ mooncake-transfer-engine/include/transport/ascend_transport/hccl_transport/hccl_transport_mem_c.h (+118 -0)
📝 mooncake-transfer-engine/include/transport/transport.h (+3 -0)
📝 mooncake-transfer-engine/src/CMakeLists.txt (+9 -0)
📝 mooncake-transfer-engine/src/multi_transport.cpp (+8 -0)
📝 mooncake-transfer-engine/src/transfer_engine.cpp (+16 -0)
📝 mooncake-transfer-engine/src/transfer_metadata.cpp (+67 -0)
📝 mooncake-transfer-engine/src/transport/CMakeLists.txt (+5 -0)
➕ mooncake-transfer-engine/src/transport/ascend_transport/CMakeLists.txt (+21 -0)
➕ mooncake-transfer-engine/src/transport/ascend_transport/hccl_transport/ascend_transport_c/CMakeLists.txt (+39 -0)

...and 7 more files

📄 Description

Ascend Transport

The source code path for Ascend Transport is Mooncake/mooncake-transfer-engine/src/transport/ascend_transport, which also includes automated build scripts and the README file.

Overview

Ascend Transport is a high-performance zero-copy NPU data transfer library with one-sided semantics, directly compatible with Mooncake Transfer Engine. To compile and use the Ascend Transport library, please set the USE_ASCEND flag to "ON" in the mooncake-common/common.cmake file.

Ascend Transport supports inter-NPU data transfer using one-sided semantics (currently supports Device to Device; other modes are under development). Users only need to specify the node and memory information at both ends through the Mooncake Transfer Engine interface to achieve high-performance point-to-point data transfer. Ascend Transport abstracts away internal complexities and automatically handles operations such as establishing connections, registering and exchanging memory, and checking transfer status.

New Dependencies

In addition to the dependencies already required by Mooncake, Ascend Transport needs some HCCL-related dependencies:

MPI

yum install -y mpich mpich-devel

One-Step Build Script

Ascend Transport provides a one-step build script located at Mooncake/mooncake-transfer-engine/src/transport/ascend_transport/scripts/build_all_with_dependencies.sh. Copy this script to the desired installation directory and run it. You can also pass an installation path as an argument; if not provided, it defaults to the current directory:

./build_all_with_dependencies.sh /path/to/install_directory

This script also supports environments where users cannot perform git clone directly. Users can place the source code for dependencies and Mooncake in the target directory, and the script will handle the compilation accordingly.

One-Step Installation Script (Without Building Mooncake)

To avoid potential conflicts when running other processes during Mooncake compilation, Ascend Transport offers a solution that separates the build and runtime environments.

After completing the Mooncake build via build_all_with_dependencies.sh, you can run setup_basic_dependencies.sh to install only the required dependencies. Place the generated Mooncake .whl package and libascend_transport_mem.so into the installation directory.

Copy the script to the installation directory and run:

./setup_basic_dependencies.sh /path/to/install_directory

Before use, ensure that libascend_transport_mem.so has been copied to /usr/local/Ascend/ascend-toolkit/latest/python/site-packages, then execute:

export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:$LD_LIBRARY_PATH

Build Instructions

Once all dependencies are installed successfully, you can proceed with building Mooncake normally. If errors occur, try setting the following environment variable:

export CPLUS_INCLUDE_PATH=$(echo $CPLUS_INCLUDE_PATH | tr ':' '\n' | grep -v "/usr/local/Ascend" | paste -sd: -)

Endpoint Management

Each Huawei NPU card has a dedicated parameter-plane NIC and should be managed by a single TransferEngine instance responsible for all its data transfers.

Ranktable Management

Ascend Transport does not rely on global Ranktable information. It only needs to obtain the local Ranktable information of the current NPU card. During the initialization of Ascend Transport, it will automatically parse the /etc/hccn.conf file to acquire this information.

Initialization

When using Ascend Transport, the TransferEngine must still call the init function after construction:

TransferEngine();

int TransferEngine::init(const std::string &metadata_conn_string,
                         const std::string &local_server_name,
                         const std::string &ip_or_host_name,
                         uint64_t rpc_port)

The only difference is that the local_server_name parameter must now include the physical NPU card ID. The format changes from ip:port to ip:port:npu_x, e.g., "0.0.0.0:12345:npu_2".

Note

: This extension of the local_server_name is used internally by Ascend Transport without modifying Mooncake's external API. The segment_desc_name in metadata remains in the original format (ip:port). Therefore, each NPU card must use a unique port that is not occupied.

Metadata Service

Ascend Transport is compatible with all metadata services currently supported by Mooncake, including etcd, redis, http, and p2phandshake. Upon initialization, Ascend Transport registers key NPU card information such as device_id, device_ip, rank_id, and server_ip.

Data Transfer

Ascend Transport supports write/read semantics and automatically determines whether cross-HCCS communication is needed, selecting either HCCS or ROCE as the underlying transport protocol. Users can use the standard Mooncake getTransferStatus API to monitor the progress of each transfer request.

Fault Handling

Building upon HCCL’s built-in fault handling mechanisms, Ascend Transport implements comprehensive error recovery strategies across multiple stages, including initialization, connection setup, and data transfer. It incorporates retry logic and returns precise error codes based on HCCL collective communication standards when retries fail. For detailed logs, refer to /root/Ascend/log/plog.

Test Cases

Ascend Transport provides two test files:

Multi-scenario test: mooncake-transfer-engine/example/transfer_engine_ascend_one_sided.cpp
Performance test: mooncake-transfer-engine/example/transfer_engine_ascend_perf.cpp

You can configure various scenarios (e.g., 1-to-1, 1-to-2, 2-to-1) and performance tests by passing valid parameters to these programs.

Example Commands for Scenario Testing

Start Initiator Node:

./transfer_engine_ascend_one_sided --metadata_server=P2PHANDSHAKE --local_server_name=10.0.0.0:12345 --protocol=hccl --operation=write --segment_id=10.0.0.0:12346 --device_id=0 --mode=initiator --block_size=8388608

Start Target Node:

./transfer_engine_ascend_one_sided --metadata_server=P2PHANDSHAKE --local_server_name=10.0.0.0:12346 --protocol=hccl --operation=write --device_id=1 --mode=target --block_size=8388608

Example Commands for Performance Testing

Start Initiator Node:

./transfer_engine_ascend_perf --metadata_server=P2PHANDSHAKE --local_server_name=10.0.0.0:12345 --protocol=hccl --operation=write --segment_id=10.0.0.0:12346 --device_id=0 --mode=initiator --block_size=8388608

Start Target Node:

./transfer_engine_ascend_perf --metadata_server=P2PHANDSHAKE --local_server_name=10.0.0.0:12346 --protocol=hccl --operation=write --device_id=1 --mode=target

Print Description

If you need to obtain information about whether each transport request is cross-hccs and its corresponding execution time, you can enable the related logs by setting the environment variable. Use the following command to turn on the logging:

export ASCEND_TRANSPORT_PRINT=1

Notes

ascned_transport will establish a TCP connection on the host side.This connection uses port (10000 + deviceId). Please avoid using this port for other services to prevent conflicts.

ascned_transport has an automatic reconnection mechanism in place that triggers after a transfer is completed, in case the remote end goes offline and restarts. There is no need to manually restart the local service.

Note If the target end goes offline and restarts, the initiating end will attempt to re-establish the connection when it sends the next request. The target must complete its restart and become ready within 5 seconds after the initiator sends the request. If the target does not become available within this window, the connection will fail and an error will be returned.

_{🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.}

## 📋 Pull Request Information **Original PR:** https://github.com/kvcache-ai/Mooncake/pull/502 **Author:** [@AscendTransport](https://github.com/AscendTransport) **Created:** 6/16/2025 **Status:** ✅ Merged **Merged:** 7/1/2025 **Merged by:** [@ShangmingCai](https://github.com/ShangmingCai) **Base:** `main` ← **Head:** `main` --- ### 📝 Commits (4) - [`b2879cb`](https://github.com/kvcache-ai/Mooncake/commit/b2879cb410d28a88a91840f3c486f290c9de2a6b) Ascend Transport - [`50b467e`](https://github.com/kvcache-ai/Mooncake/commit/50b467e148b584e2aaea21e8daa5b5d73c2e6140) Merge branch 'main' into main - [`0eaa333`](https://github.com/kvcache-ai/Mooncake/commit/0eaa333ac78cae076a58ede2ecbeb166ead71600) fix ci - [`930ae39`](https://github.com/kvcache-ai/Mooncake/commit/930ae395c612c6a56a9f263139deeba5290a3c60) reviewed changes ### 📊 Changes **27 files changed** (+3557 additions, -0 deletions) <details> <summary>View changed files</summary> ➕ `doc/en/ascend_transport.md` (+145 -0) ➕ `doc/zh/ascend_transport.md` (+94 -0) 📝 `mooncake-common/common.cmake` (+17 -0) 📝 `mooncake-integration/transfer_engine/transfer_engine_py.cpp` (+2 -0) 📝 `mooncake-transfer-engine/CMakeLists.txt` (+22 -0) 📝 `mooncake-transfer-engine/example/CMakeLists.txt` (+13 -0) ➕ `mooncake-transfer-engine/example/transfer_engine_ascend_one_sided.cpp` (+421 -0) ➕ `mooncake-transfer-engine/example/transfer_engine_ascend_perf.cpp` (+354 -0) 📝 `mooncake-transfer-engine/include/common.h` (+40 -0) 📝 `mooncake-transfer-engine/include/transfer_metadata.h` (+14 -0) ➕ `mooncake-transfer-engine/include/transport/ascend_transport/hccl_transport/hccl_transport.h` (+102 -0) ➕ `mooncake-transfer-engine/include/transport/ascend_transport/hccl_transport/hccl_transport_mem_c.h` (+118 -0) 📝 `mooncake-transfer-engine/include/transport/transport.h` (+3 -0) 📝 `mooncake-transfer-engine/src/CMakeLists.txt` (+9 -0) 📝 `mooncake-transfer-engine/src/multi_transport.cpp` (+8 -0) 📝 `mooncake-transfer-engine/src/transfer_engine.cpp` (+16 -0) 📝 `mooncake-transfer-engine/src/transfer_metadata.cpp` (+67 -0) 📝 `mooncake-transfer-engine/src/transport/CMakeLists.txt` (+5 -0) ➕ `mooncake-transfer-engine/src/transport/ascend_transport/CMakeLists.txt` (+21 -0) ➕ `mooncake-transfer-engine/src/transport/ascend_transport/hccl_transport/ascend_transport_c/CMakeLists.txt` (+39 -0) _...and 7 more files_ </details> ### 📄 Description # Ascend Transport The source code path for Ascend Transport is `Mooncake/mooncake-transfer-engine/src/transport/ascend_transport`, which also includes automated build scripts and the README file. ## Overview Ascend Transport is a high-performance zero-copy NPU data transfer library with one-sided semantics, directly compatible with Mooncake Transfer Engine. To compile and use the Ascend Transport library, please set the `USE_ASCEND` flag to `"ON"` in the `mooncake-common/common.cmake` file. Ascend Transport supports inter-NPU data transfer using one-sided semantics (currently supports Device to Device; other modes are under development). Users only need to specify the node and memory information at both ends through the Mooncake Transfer Engine interface to achieve high-performance point-to-point data transfer. Ascend Transport abstracts away internal complexities and automatically handles operations such as establishing connections, registering and exchanging memory, and checking transfer status. ### New Dependencies In addition to the dependencies already required by Mooncake, Ascend Transport needs some HCCL-related dependencies: **MPI** ```bash yum install -y mpich mpich-devel ``` ### One-Step Build Script Ascend Transport provides a one-step build script located at `Mooncake/mooncake-transfer-engine/src/transport/ascend_transport/scripts/build_all_with_dependencies.sh`. Copy this script to the desired installation directory and run it. You can also pass an installation path as an argument; if not provided, it defaults to the current directory: ```bash ./build_all_with_dependencies.sh /path/to/install_directory ``` This script also supports environments where users cannot perform `git clone` directly. Users can place the source code for dependencies and Mooncake in the target directory, and the script will handle the compilation accordingly. ### One-Step Installation Script (Without Building Mooncake) To avoid potential conflicts when running other processes during Mooncake compilation, Ascend Transport offers a solution that separates the build and runtime environments. After completing the Mooncake build via build_all_with_dependencies.sh, you can run setup_basic_dependencies.sh to install only the required dependencies. Place the generated Mooncake .whl package and libascend_transport_mem.so into the installation directory. Copy the script to the installation directory and run: ```bash ./setup_basic_dependencies.sh /path/to/install_directory ``` Before use, ensure that `libascend_transport_mem.so` has been copied to `/usr/local/Ascend/ascend-toolkit/latest/python/site-packages`, then execute: ```bash export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:$LD_LIBRARY_PATH ``` ### Build Instructions Once all dependencies are installed successfully, you can proceed with building Mooncake normally. If errors occur, try setting the following environment variable: ```bash export CPLUS_INCLUDE_PATH=$(echo $CPLUS_INCLUDE_PATH | tr ':' '\n' | grep -v "/usr/local/Ascend" | paste -sd: -) ``` ### Endpoint Management Each Huawei NPU card has a dedicated parameter-plane NIC and should be managed by a single `TransferEngine` instance responsible for all its data transfers. ### Ranktable Management Ascend Transport does not rely on global Ranktable information. It only needs to obtain the local Ranktable information of the current NPU card. During the initialization of Ascend Transport, it will automatically parse the /etc/hccn.conf file to acquire this information. ### Initialization When using Ascend Transport, the `TransferEngine` must still call the `init` function after construction: ```cpp TransferEngine(); int TransferEngine::init(const std::string &metadata_conn_string, const std::string &local_server_name, const std::string &ip_or_host_name, uint64_t rpc_port) ``` The only difference is that the `local_server_name` parameter must now include the physical NPU card ID. The format changes from `ip:port` to `ip:port:npu_x`, e.g., `"0.0.0.0:12345:npu_2"`. > **Note**: This extension of the `local_server_name` is used internally by Ascend Transport without modifying Mooncake's external API. The `segment_desc_name` in metadata remains in the original format (`ip:port`). Therefore, each NPU card must use a unique port that is not occupied. ### Metadata Service Ascend Transport is compatible with all metadata services currently supported by Mooncake, including `etcd`, `redis`, `http`, and `p2phandshake`. Upon initialization, Ascend Transport registers key NPU card information such as `device_id`, `device_ip`, `rank_id`, and `server_ip`. ### Data Transfer Ascend Transport supports write/read semantics and automatically determines whether cross-HCCS communication is needed, selecting either HCCS or ROCE as the underlying transport protocol. Users can use the standard Mooncake `getTransferStatus` API to monitor the progress of each transfer request. ### Fault Handling Building upon HCCL’s built-in fault handling mechanisms, Ascend Transport implements comprehensive error recovery strategies across multiple stages, including initialization, connection setup, and data transfer. It incorporates retry logic and returns precise error codes based on HCCL collective communication standards when retries fail. For detailed logs, refer to `/root/Ascend/log/plog`. ### Test Cases Ascend Transport provides two test files: - Multi-scenario test: `mooncake-transfer-engine/example/transfer_engine_ascend_one_sided.cpp` - Performance test: `mooncake-transfer-engine/example/transfer_engine_ascend_perf.cpp` You can configure various scenarios (e.g., 1-to-1, 1-to-2, 2-to-1) and performance tests by passing valid parameters to these programs. #### Example Commands for Scenario Testing **Start Initiator Node:** ```bash ./transfer_engine_ascend_one_sided --metadata_server=P2PHANDSHAKE --local_server_name=10.0.0.0:12345 --protocol=hccl --operation=write --segment_id=10.0.0.0:12346 --device_id=0 --mode=initiator --block_size=8388608 ``` **Start Target Node:** ```bash ./transfer_engine_ascend_one_sided --metadata_server=P2PHANDSHAKE --local_server_name=10.0.0.0:12346 --protocol=hccl --operation=write --device_id=1 --mode=target --block_size=8388608 ``` #### Example Commands for Performance Testing **Start Initiator Node:** ```bash ./transfer_engine_ascend_perf --metadata_server=P2PHANDSHAKE --local_server_name=10.0.0.0:12345 --protocol=hccl --operation=write --segment_id=10.0.0.0:12346 --device_id=0 --mode=initiator --block_size=8388608 ``` **Start Target Node:** ```bash ./transfer_engine_ascend_perf --metadata_server=P2PHANDSHAKE --local_server_name=10.0.0.0:12346 --protocol=hccl --operation=write --device_id=1 --mode=target ``` ### Print Description If you need to obtain information about whether each transport request is cross-hccs and its corresponding execution time, you can enable the related logs by setting the environment variable. Use the following command to turn on the logging: ```bash export ASCEND_TRANSPORT_PRINT=1 ``` ### Notes ascned_transport will establish a TCP connection on the host side.This connection uses port (10000 + deviceId). Please avoid using this port for other services to prevent conflicts. ascned_transport has an automatic reconnection mechanism in place that triggers after a transfer is completed, in case the remote end goes offline and restarts. There is no need to manually restart the local service. Note If the target end goes offline and restarts, the initiating end will attempt to re-establish the connection when it sends the next request. The target must complete its restart and become ready within 5 seconds after the initiator sends the request. If the target does not become available within this window, the connection will fail and an error will be returned. --- <sub>🔄 This issue represents a GitHub Pull Request. It cannot be merged through Gitea due to API limitations.</sub>

bsakaguchi

2026-10-29 21:28:53 +00:00