kvcache-ai/Mooncake
1
0
Fork 0
mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00
Code Issues 368 Projects Packages Wiki Activity

[GH-ISSUE #648] [Bug]: Mooncake store Put operation error at ascend #208

New issue
Closed
opened 2026-10-29 21:10:55 +00:00 by bsakaguchi · 9 comments
bsakaguchi commented 2026-10-29 21:10:55 +00:00
Owner
Copy link

Originally created by @elfisworking on GitHub (Jul 21, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/648

Bug Report

when i run following code at ascend using this pr https://github.com/kvcache-ai/Mooncake/pull/502, everything is ok

from mooncake.store import MooncakeDistributedStore
import asyncio
import torch
import numpy
import pickle

class MooncakestoreConnector:
    def __init__(
        self,
    ):

        self.store = MooncakeDistributedStore()


        self.store.setup(
            "localhost",
            "http://127.0.0.1:8080/metadata",
            3355443200,
            1073741824,
            "tcp",
            "",
            "localhost:50051",
        )
        self.store.init_all("tcp", "", 16777216)
        
    def put_and_get(self):
        tensor = torch.ones([2,3])
        numpy_array = tensor.numpy()
        # 将 NumPy 数组转换为二进制数据 (字节)
        binary_data = numpy_array.tobytes()
        self.store.put(key_str, binary_data)
        value = self.store.get(key_str)
        print("get value is ", value)



if __name__ == "__main__":
    con = MooncakestoreConnector()
    con.put_and_get()

But after i update code https://github.com/kvcache-ai/Mooncake/pull/619, the error is

I20250721 16:36:40.154799 79567 transfer_engine.cpp:397] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable)
I20250721 16:36:40.155064 79567 client.cpp:43] client_id=9605841925196329328-2047564984061973410
I20250721 16:36:40.156528 79567 client.cpp:236] Storage root directory is not set. persisting data is disabled.
I20250721 16:36:40.156563 79567 transfer_engine.cpp:44] Transfer Engine starting. Server: localhost:12727, Metadata: http://127.0.0.1:8080/metadata, ip_or_host_name: localhost, rpc_port: 12727
I20250721 16:36:40.156576 79567 transfer_engine.cpp:58] Transfer Engine parseHostNameWithPortAscend. server_name: localhost port: 12727 devicePhyId: -1
I20250721 16:36:40.161757 79567 transfer_metadata_plugin.cpp:1053] Found active interface eno0 with IP 172.19.0.5
I20250721 16:36:40.161782 79567 transfer_metadata_plugin.cpp:1053] Found active interface bond2 with IP 100.97.160.9
I20250721 16:36:40.161793 79567 transfer_metadata_plugin.cpp:1042] Skipping interface docker0 (not UP or not RUNNING)
I20250721 16:36:40.161820 79567 transfer_engine.cpp:117] Transfer Engine RPC using new RPC mapping, listening on 172.19.0.5:15257
I20250721 16:36:40.163394 79567 hccl_transport.cpp:304] HcclTransport: begin to install transport, local devicePhyId: -1, local_server_name: localhost:12727:npu_-1
E20250721 16:36:40.171002 79567 hccl_transport.cpp:238] HcclTransport: aclrtGetDevice failed, ret: 107002
E20250721 16:36:40.171025 79567 hccl_transport.cpp:309] HcclTransport: rankInfoParse failed, ret: 107002
I20250721 16:36:40.171037 79567 client.cpp:192] transport_type=tcp
I20250721 16:36:40.171722 79567 tcp_transport.cpp:250] TcpTransport: listen on port 16645
E20250721 16:36:40.176746 79567 store_py.cpp:198] Client is already initialized
E20250721 16:36:40.185065 79567 transfer_metadata_plugin.cpp:233] Unexpected code in http response, GET http://127.0.0.1:8080/metadata?key=mooncake%2Fram%2Flocalhost%3A12727 response code: 404 response body: metadata not found
W20250721 16:36:40.185087 79567 transfer_metadata.cpp:417] Failed to retrieve segment descriptor, name localhost:12727
E20250721 16:36:40.185096 79567 transfer_task.cpp:469] Failed to open segment localhost:12727
E20250721 16:36:40.185103 79567 client.cpp:1097] Failed to submit transfer operation
E20250721 16:36:40.185184 79567 store_py.cpp:254] Put operation failed with error: TRANSFER_FAIL
get value is  b''

meet a metadata 404 error :

 Unexpected code in http response, GET http://127.0.0.1:8080/metadata?key=mooncake%2Fram%2Flocalhost%3A12727 response code: 404 response body: metadata not found

Metadata server log is

2025-07-21 16:36:09,856 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:09 +0800] "PUT /metadata?key=mooncake%2Frpc_meta%2Flocalhost%3A12440 HTTP/1.1" 200 176 "-" "-"
2025-07-21 16:36:09,865 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:09 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A12440%3Anpu_-1 HTTP/1.1" 200 176 "-" "-"
2025-07-21 16:36:09,868 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:09 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A12440%3Anpu_-1 HTTP/1.1" 200 176 "-" "-"
2025-07-21 16:36:09,869 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:09 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A12440%3Anpu_-1 HTTP/1.1" 200 176 "-" "-"
2025-07-21 16:36:10,047 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:10 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A12440%3Anpu_-1 HTTP/1.1" 200 176 "-" "-"
2025-07-21 16:36:10,048 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:10 +0800] "DELETE /metadata?key=mooncake%2Frpc_meta%2Flocalhost%3A12440 HTTP/1.1" 200 176 "-" "-"
2025-07-21 16:36:10,049 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:10 +0800] "DELETE /metadata?key=mooncake%2Fram%2Flocalhost%3A12440%3Anpu_-1 HTTP/1.1" 200 176 "-" "-"
2025-07-21 16:36:40,163 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:40 +0800] "PUT /metadata?key=mooncake%2Frpc_meta%2Flocalhost%3A12727 HTTP/1.1" 200 176 "-" "-"
2025-07-21 16:36:40,171 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:40 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A12727%3Anpu_-1 HTTP/1.1" 200 176 "-" "-"
2025-07-21 16:36:40,176 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:40 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A12727%3Anpu_-1 HTTP/1.1" 200 176 "-" "-"
2025-07-21 16:36:40,176 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:40 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A12727%3Anpu_-1 HTTP/1.1" 200 176 "-" "-"
2025-07-21 16:36:40,185 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:40 +0800] "GET /metadata?key=mooncake%2Fram%2Flocalhost%3A12727 HTTP/1.1" 404 185 "-" "-"
2025-07-21 16:36:40,321 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:40 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A12727%3Anpu_-1 HTTP/1.1" 200 176 "-" "-"
2025-07-21 16:36:40,322 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:40 +0800] "DELETE /metadata?key=mooncake%2Frpc_meta%2Flocalhost%3A12727 HTTP/1.1" 200 176 "-" "-"
2025-07-21 16:36:40,322 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:40 +0800] "DELETE /metadata?key=mooncake%2Fram%2Flocalhost%3A12727%3Anpu_-1 HTTP/1.1" 200 176 "-" "-"
2025-07-21 16:44:49,521 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:44:49 +0800] "PUT /metadata?key=mooncake%2Frpc_meta%2Flocalhost%3A14128 HTTP/1.1" 200 176 "-" "-"
2025-07-21 16:44:49,529 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:44:49 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A14128%3Anpu_-1 HTTP/1.1" 200 176 "-" "-"
2025-07-21 16:44:49,534 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:44:49 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A14128%3Anpu_-1 HTTP/1.1" 200 176 "-" "-"
2025-07-21 16:44:49,535 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:44:49 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A14128%3Anpu_-1 HTTP/1.1" 200 176 "-" "-"
2025-07-21 16:44:49,543 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:44:49 +0800] "GET /metadata?key=mooncake%2Fram%2Flocalhost%3A14128 HTTP/1.1" 404 185 "-" "-"
2025-07-21 16:44:49,718 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:44:49 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A14128%3Anpu_-1 HTTP/1.1" 200 176 "-" "-"
2025-07-21 16:44:49,719 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:44:49 +0800] "DELETE /metadata?key=mooncake%2Frpc_meta%2Flocalhost%3A14128 HTTP/1.1" 200 176 "-" "-"
2025-07-21 16:44:49,720 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:44:49 +0800] "DELETE /metadata?key=mooncake%2Fram%2Flocalhost%3A14128%3Anpu_-1 HTTP/1.1" 200 176 "-" "-"

error log is

2025-07-21 16:36:40,185 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:40 +0800] "GET /metadata?key=mooncake%2Fram%2Flocalhost%3A12727 HTTP/1.1" 404 185 "-" "-"

Before submitting...

  • Ensure you searched for relevant issues and read the [documentation]
Originally created by @elfisworking on GitHub (Jul 21, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/648 ### Bug Report when i run following code at ascend using this pr https://github.com/kvcache-ai/Mooncake/pull/502, everything is ok ``` from mooncake.store import MooncakeDistributedStore import asyncio import torch import numpy import pickle class MooncakestoreConnector: def __init__( self, ): self.store = MooncakeDistributedStore() self.store.setup( "localhost", "http://127.0.0.1:8080/metadata", 3355443200, 1073741824, "tcp", "", "localhost:50051", ) self.store.init_all("tcp", "", 16777216) def put_and_get(self): tensor = torch.ones([2,3]) numpy_array = tensor.numpy() # 将 NumPy 数组转换为二进制数据 (字节) binary_data = numpy_array.tobytes() self.store.put(key_str, binary_data) value = self.store.get(key_str) print("get value is ", value) if __name__ == "__main__": con = MooncakestoreConnector() con.put_and_get() ``` But after i update code https://github.com/kvcache-ai/Mooncake/pull/619, the error is ``` I20250721 16:36:40.154799 79567 transfer_engine.cpp:397] Metrics reporting is disabled (set MC_TE_METRIC=1 to enable) I20250721 16:36:40.155064 79567 client.cpp:43] client_id=9605841925196329328-2047564984061973410 I20250721 16:36:40.156528 79567 client.cpp:236] Storage root directory is not set. persisting data is disabled. I20250721 16:36:40.156563 79567 transfer_engine.cpp:44] Transfer Engine starting. Server: localhost:12727, Metadata: http://127.0.0.1:8080/metadata, ip_or_host_name: localhost, rpc_port: 12727 I20250721 16:36:40.156576 79567 transfer_engine.cpp:58] Transfer Engine parseHostNameWithPortAscend. server_name: localhost port: 12727 devicePhyId: -1 I20250721 16:36:40.161757 79567 transfer_metadata_plugin.cpp:1053] Found active interface eno0 with IP 172.19.0.5 I20250721 16:36:40.161782 79567 transfer_metadata_plugin.cpp:1053] Found active interface bond2 with IP 100.97.160.9 I20250721 16:36:40.161793 79567 transfer_metadata_plugin.cpp:1042] Skipping interface docker0 (not UP or not RUNNING) I20250721 16:36:40.161820 79567 transfer_engine.cpp:117] Transfer Engine RPC using new RPC mapping, listening on 172.19.0.5:15257 I20250721 16:36:40.163394 79567 hccl_transport.cpp:304] HcclTransport: begin to install transport, local devicePhyId: -1, local_server_name: localhost:12727:npu_-1 E20250721 16:36:40.171002 79567 hccl_transport.cpp:238] HcclTransport: aclrtGetDevice failed, ret: 107002 E20250721 16:36:40.171025 79567 hccl_transport.cpp:309] HcclTransport: rankInfoParse failed, ret: 107002 I20250721 16:36:40.171037 79567 client.cpp:192] transport_type=tcp I20250721 16:36:40.171722 79567 tcp_transport.cpp:250] TcpTransport: listen on port 16645 E20250721 16:36:40.176746 79567 store_py.cpp:198] Client is already initialized E20250721 16:36:40.185065 79567 transfer_metadata_plugin.cpp:233] Unexpected code in http response, GET http://127.0.0.1:8080/metadata?key=mooncake%2Fram%2Flocalhost%3A12727 response code: 404 response body: metadata not found W20250721 16:36:40.185087 79567 transfer_metadata.cpp:417] Failed to retrieve segment descriptor, name localhost:12727 E20250721 16:36:40.185096 79567 transfer_task.cpp:469] Failed to open segment localhost:12727 E20250721 16:36:40.185103 79567 client.cpp:1097] Failed to submit transfer operation E20250721 16:36:40.185184 79567 store_py.cpp:254] Put operation failed with error: TRANSFER_FAIL get value is b'' ``` meet a metadata 404 error : ``` Unexpected code in http response, GET http://127.0.0.1:8080/metadata?key=mooncake%2Fram%2Flocalhost%3A12727 response code: 404 response body: metadata not found ``` Metadata server log is ``` 2025-07-21 16:36:09,856 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:09 +0800] "PUT /metadata?key=mooncake%2Frpc_meta%2Flocalhost%3A12440 HTTP/1.1" 200 176 "-" "-" 2025-07-21 16:36:09,865 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:09 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A12440%3Anpu_-1 HTTP/1.1" 200 176 "-" "-" 2025-07-21 16:36:09,868 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:09 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A12440%3Anpu_-1 HTTP/1.1" 200 176 "-" "-" 2025-07-21 16:36:09,869 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:09 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A12440%3Anpu_-1 HTTP/1.1" 200 176 "-" "-" 2025-07-21 16:36:10,047 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:10 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A12440%3Anpu_-1 HTTP/1.1" 200 176 "-" "-" 2025-07-21 16:36:10,048 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:10 +0800] "DELETE /metadata?key=mooncake%2Frpc_meta%2Flocalhost%3A12440 HTTP/1.1" 200 176 "-" "-" 2025-07-21 16:36:10,049 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:10 +0800] "DELETE /metadata?key=mooncake%2Fram%2Flocalhost%3A12440%3Anpu_-1 HTTP/1.1" 200 176 "-" "-" 2025-07-21 16:36:40,163 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:40 +0800] "PUT /metadata?key=mooncake%2Frpc_meta%2Flocalhost%3A12727 HTTP/1.1" 200 176 "-" "-" 2025-07-21 16:36:40,171 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:40 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A12727%3Anpu_-1 HTTP/1.1" 200 176 "-" "-" 2025-07-21 16:36:40,176 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:40 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A12727%3Anpu_-1 HTTP/1.1" 200 176 "-" "-" 2025-07-21 16:36:40,176 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:40 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A12727%3Anpu_-1 HTTP/1.1" 200 176 "-" "-" 2025-07-21 16:36:40,185 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:40 +0800] "GET /metadata?key=mooncake%2Fram%2Flocalhost%3A12727 HTTP/1.1" 404 185 "-" "-" 2025-07-21 16:36:40,321 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:40 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A12727%3Anpu_-1 HTTP/1.1" 200 176 "-" "-" 2025-07-21 16:36:40,322 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:40 +0800] "DELETE /metadata?key=mooncake%2Frpc_meta%2Flocalhost%3A12727 HTTP/1.1" 200 176 "-" "-" 2025-07-21 16:36:40,322 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:40 +0800] "DELETE /metadata?key=mooncake%2Fram%2Flocalhost%3A12727%3Anpu_-1 HTTP/1.1" 200 176 "-" "-" 2025-07-21 16:44:49,521 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:44:49 +0800] "PUT /metadata?key=mooncake%2Frpc_meta%2Flocalhost%3A14128 HTTP/1.1" 200 176 "-" "-" 2025-07-21 16:44:49,529 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:44:49 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A14128%3Anpu_-1 HTTP/1.1" 200 176 "-" "-" 2025-07-21 16:44:49,534 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:44:49 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A14128%3Anpu_-1 HTTP/1.1" 200 176 "-" "-" 2025-07-21 16:44:49,535 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:44:49 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A14128%3Anpu_-1 HTTP/1.1" 200 176 "-" "-" 2025-07-21 16:44:49,543 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:44:49 +0800] "GET /metadata?key=mooncake%2Fram%2Flocalhost%3A14128 HTTP/1.1" 404 185 "-" "-" 2025-07-21 16:44:49,718 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:44:49 +0800] "PUT /metadata?key=mooncake%2Fram%2Flocalhost%3A14128%3Anpu_-1 HTTP/1.1" 200 176 "-" "-" 2025-07-21 16:44:49,719 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:44:49 +0800] "DELETE /metadata?key=mooncake%2Frpc_meta%2Flocalhost%3A14128 HTTP/1.1" 200 176 "-" "-" 2025-07-21 16:44:49,720 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:44:49 +0800] "DELETE /metadata?key=mooncake%2Fram%2Flocalhost%3A14128%3Anpu_-1 HTTP/1.1" 200 176 "-" "-" ``` error log is ``` 2025-07-21 16:36:40,185 - aiohttp.access - INFO - 127.0.0.1 [21/Jul/2025:16:36:40 +0800] "GET /metadata?key=mooncake%2Fram%2Flocalhost%3A12727 HTTP/1.1" 404 185 "-" "-" ``` ### Before submitting... - [x] Ensure you searched for relevant issues and read the [documentation]
bsakaguchi 2026-10-29 21:10:55 +00:00
bsakaguchi commented 2026-10-29 21:10:56 +00:00
Author
Owner
Copy link

@elfisworking commented on GitHub (Jul 21, 2025):

@AscendTransport can you look at this problem??

<!-- gh-comment-id:3095764612 --> @elfisworking commented on GitHub (Jul 21, 2025): @AscendTransport can you look at this problem??
bsakaguchi commented 2026-10-29 21:10:56 +00:00
Author
Owner
Copy link

@AscendTransport commented on GitHub (Jul 21, 2025):

@AscendTransport can you look at this problem??

I see that you are currently using TCP instead of Ascend. Please set USE_ASCEND to OFF on line 61 of the file mooncake-common/common.cmake to disable ASCEND.

<!-- gh-comment-id:3095948505 --> @AscendTransport commented on GitHub (Jul 21, 2025): > [@AscendTransport](https://github.com/AscendTransport) can you look at this problem?? I see that you are currently using TCP instead of Ascend. Please set USE_ASCEND to OFF on line 61 of the file mooncake-common/common.cmake to disable ASCEND.
bsakaguchi commented 2026-10-29 21:10:56 +00:00
Author
Owner
Copy link

@elfisworking commented on GitHub (Jul 21, 2025):

@AscendTransport can you look at this problem??

I see that you are currently using TCP instead of Ascend. Please set USE_ASCEND to OFF on line 61 of the file mooncake-common/common.cmake to disable ASCEND.

But when https://github.com/kvcache-ai/Mooncake/pull/502, tcp is ok where i set USE_ASCEND to ON.

<!-- gh-comment-id:3096262589 --> @elfisworking commented on GitHub (Jul 21, 2025): > > [@AscendTransport](https://github.com/AscendTransport) can you look at this problem?? > > I see that you are currently using TCP instead of Ascend. Please set USE_ASCEND to OFF on line 61 of the file mooncake-common/common.cmake to disable ASCEND. But when https://github.com/kvcache-ai/Mooncake/pull/502, tcp is ok where i set USE_ASCEND to ON.
bsakaguchi commented 2026-10-29 21:10:56 +00:00
Author
Owner
Copy link

@elfisworking commented on GitHub (Jul 21, 2025):

Additionally, I need USE_ASCEND to be set to ON, as it will later be necessary to transfer from the NPU to the mooncake store. Using TCP is just for testing purposes. @AscendTransport

<!-- gh-comment-id:3096273212 --> @elfisworking commented on GitHub (Jul 21, 2025): Additionally, I need USE_ASCEND to be set to ON, as it will later be necessary to transfer from the NPU to the mooncake store. Using TCP is just for testing purposes. @AscendTransport
bsakaguchi commented 2026-10-29 21:10:56 +00:00
Author
Owner
Copy link

@AscendTransport commented on GitHub (Jul 21, 2025):

If you wish to use ASCEND, the current requirement for the format of “localhost” is ip:port:npu_x (where x represents the physical card number of the NPU). The fact that #502 does not report an error is confusing us, as it is possible that we are not encountering an error at #502.

Additionally, I need USE_ASCEND to be set to ON, as it will later be necessary to transfer from the NPU to the mooncake store. Using TCP is just for testing purposes. @AscendTransport

<!-- gh-comment-id:3096524763 --> @AscendTransport commented on GitHub (Jul 21, 2025): If you wish to use ASCEND, the current requirement for the format of “localhost” is ip:port:npu_x (where x represents the physical card number of the NPU). The fact that #502 does not report an error is confusing us, as it is possible that we are not encountering an error at #502. > Additionally, I need USE_ASCEND to be set to ON, as it will later be necessary to transfer from the NPU to the mooncake store. Using TCP is just for testing purposes. [@AscendTransport](https://github.com/AscendTransport)
bsakaguchi commented 2026-10-29 21:10:57 +00:00
Author
Owner
Copy link

@elfisworking commented on GitHub (Jul 22, 2025):

If you wish to use ASCEND, the current requirement for the format of “localhost” is ip:port:npu_x (where x represents the physical card number of the NPU). The fact that #502 does not report an error is confusing us, as it is possible that we are not encountering an error at #502.

Additionally, I need USE_ASCEND to be set to ON, as it will later be necessary to transfer from the NPU to the mooncake store. Using TCP is just for testing purposes. @AscendTransport

We tested the mooncake transfer when USE_ASCEND is set to ON, and it is indeed working. So I suspect that the current implementation does not adapt well to the mooncake store. I once tried to write it this way, and it directly caused a core dump:

self.store.setup(
"localhost",
"http://127.0.0.1:8080/metadata",
3355443200,
1073741824,
"tcp",
"",
"localhost:50051:npu_0",
)
<!-- gh-comment-id:3100507679 --> @elfisworking commented on GitHub (Jul 22, 2025): > If you wish to use ASCEND, the current requirement for the format of “localhost” is ip:port:npu_x (where x represents the physical card number of the NPU). The fact that [#502](https://github.com/kvcache-ai/Mooncake/pull/502) does not report an error is confusing us, as it is possible that we are not encountering an error at [#502](https://github.com/kvcache-ai/Mooncake/pull/502). > > > Additionally, I need USE_ASCEND to be set to ON, as it will later be necessary to transfer from the NPU to the mooncake store. Using TCP is just for testing purposes. [@AscendTransport](https://github.com/AscendTransport) We tested the mooncake transfer when USE_ASCEND is set to ON, and it is indeed working. So I suspect that the current implementation does not adapt well to the mooncake store. I once tried to write it this way, and it directly caused a core dump: ``` self.store.setup( "localhost", "http://127.0.0.1:8080/metadata", 3355443200, 1073741824, "tcp", "", "localhost:50051:npu_0", ) ```
bsakaguchi commented 2026-10-29 21:10:57 +00:00
Author
Owner
Copy link

@AscendTransport commented on GitHub (Jul 22, 2025):

If you wish to use ASCEND, the current requirement for the format of “localhost” is ip:port:npu_x (where x represents the physical card number of the NPU). The fact that #502 does not report an error is confusing us, as it is possible that we are not encountering an error at #502.

Additionally, I need USE_ASCEND to be set to ON, as it will later be necessary to transfer from the NPU to the mooncake store. Using TCP is just for testing purposes. @AscendTransport

We tested the mooncake transfer when USE_ASCEND is set to ON, and it is indeed working. So I suspect that the current implementation does not adapt well to the mooncake store. I once tried to write it this way, and it directly caused a core dump:

self.store.setup(
"localhost",
"http://127.0.0.1:8080/metadata",
3355443200,
1073741824,
"tcp",
"",
"localhost:50051:npu_0",
)

If you intend to use the Ascend transport together with the Mooncake store, please note that the current release does not yet include the necessary adapter. We have already completed an internal implementation, but it is still undergoing rigorous self-tests before we upstream it to the community.Support for pooling will be rolled out in next month’s update.

<!-- gh-comment-id:3100685082 --> @AscendTransport commented on GitHub (Jul 22, 2025): > > If you wish to use ASCEND, the current requirement for the format of “localhost” is ip:port:npu_x (where x represents the physical card number of the NPU). The fact that [#502](https://github.com/kvcache-ai/Mooncake/pull/502) does not report an error is confusing us, as it is possible that we are not encountering an error at [#502](https://github.com/kvcache-ai/Mooncake/pull/502). > > > Additionally, I need USE_ASCEND to be set to ON, as it will later be necessary to transfer from the NPU to the mooncake store. Using TCP is just for testing purposes. [@AscendTransport](https://github.com/AscendTransport) > > We tested the mooncake transfer when USE_ASCEND is set to ON, and it is indeed working. So I suspect that the current implementation does not adapt well to the mooncake store. I once tried to write it this way, and it directly caused a core dump: > > ``` > self.store.setup( > "localhost", > "http://127.0.0.1:8080/metadata", > 3355443200, > 1073741824, > "tcp", > "", > "localhost:50051:npu_0", > ) > ``` If you intend to use the Ascend transport together with the Mooncake store, please note that the current release does not yet include the necessary adapter. We have already completed an internal implementation, but it is still undergoing rigorous self-tests before we upstream it to the community.Support for pooling will be rolled out in next month’s update.
bsakaguchi commented 2026-10-29 21:10:57 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Jun 30, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:4839876108 --> @github-actions[bot] commented on GitHub (Jun 30, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.
bsakaguchi commented 2026-10-29 21:10:57 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Jul 7, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

<!-- gh-comment-id:4900234948 --> @github-actions[bot] commented on GitHub (Jul 7, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
te-coro
staryxchen-tent-capability-research
feature/musa-ep-jit
zxpdemonio/fix-tent-bootstrap-admission
add-store-rs-tone-tests
scatter-rdma-completion-pr2
P2P-Mooncake-Store
release/v0.3.14
feat/tent-rdma-adaptive-adapter
fix/ep-local-counter-gpu-scope
codex/kvcs-oplog-eviction
give-me-a-star
cruz/mooncake-kvcs-lowlevel-pr1-20261008
ylt-handshake-limits
ci/rocm-e2e-tent
sima/docs-pypi-downloads
fix-ci/hicache-mooncake-metadata-loopback
sima/cuda-driver-loading
sima/ci-performance-gate
fix/tent-rdma-unknown-topology-rail-pairing
ci/tent-e2e-dual-job
mc-patch-1
codex/p2p-merge-prerequisites
sima/store-hca-numa-placement
codex/p2p-merge-cli
codex/p2p-merge-ipv6
copilot/fix-ctests-job-failure
czh/pg-device-collectives-pr6
dependabot/cargo/mooncake-store-rs/pyo3-0.29.0
czh/pg-device-collectives-pr5
codex/segment-pool-master-cutover
sima/nightly-ci-recovery
ci/tent-params
stmatengss-patch-1
dependabot/cargo/mooncake-store-rs/rand-0.8.8
dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1
dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16
codex/migrate-store-rest-service
fix/tone-wheel-arch-selection
czh/pg-device-collectives-pr4
czh/pg-device-collectives-pr3
ketong/fix_moe_tc
copilot/fix-test-wheel-ubuntu-job
fix/ci-wheel-torch-pg-pin
codex/migrate-structured-object-storage
codex/migrate-allocator-python
codex/migrate-vllm-python
codex/migrate-shared-segment
ci/fix_sgl_eval_install
codex/segment-pool-nof-placement-cleanup
feat/pip-native-release
copilot/bugfix-ha-standby-promotion
copilot/mooncake-store-transport-settings
copilot/fix-tensor-support-issues
copilot/optimize-rl-direction
copilot/mooncake-storeagentic-workflow
copilot/fix-client-reconnect-timeout
copilot/mooncake-error-debugging
copilot/mooncake-ipv6-support
feat/nccl-host-python-api
dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1
copilot/migrate-and-move-operations
copilot/migrate-and-move-operations-again
refactor/centralize-env-vars
idhanani/mooncake-retain-groups
copilot/codeownerreviewer
copilot/give-feedback-on-issue-2123
copilot/create-issue-rfc
copilot/break-cuda-monopoly-logic
copilot/compare-cudamemcpy-cu12-cu13
copilot/report-ci-error-issue
copilot/research-commit-history-analysis
copilot/explain-mooncake-store-service
copilot/analyze-project-functionality
copilot/sub-pr-1680
copilot/review-pr-1680
copilot/sub-pr-1597
copilot/fix-github-actions-job-failure
copilot/move-test-sglang-integration-job
v0.3.14
v0.3.14-pre-4
v0.3.14-pre-3
v0.3.14-pre-2
v0.3.14-pre
v0.3.14-rc1
v0.3.13.post1
v0.3.13-rc2
v0.3.13-rc1
v0.3.13
v0.3.12.post1
v0.3.12
v0.3.11-rc1
v0.3.11.post1
v0.3.11
v0.3.10.post2
v0.3.10.post1
v0.3.10
v0.3.9
v0.3.8.post1
v0.3.8
v0.3.7.post2
v0.3.7.post1
v0.3.7
v0.3.6.post1
v0.3.6
v0.3.5
v0.3.4.post2
v0.3.4.post1
v0.3.4
v0.3.3.post2
v0.3.3.post1
v0.3.3
v0.3.2.post1
v0.3.2
v0.3.1
v0.3.0
v0.3.0-beta
v0.3.0-alpha
v0.2.0
Labels
Clear labels   
ascend

   
auto-closed

   
bug

   
bug

   
CI/Build

   
Common

   
documentation

   
enhancement

   
feature request

   
good first issue

   
help wanted

   
high priority

   
MetaX

   
nightly-failure

   
pull-request
Mirrored from GitHub Pull Request

  
PyTorch Backend

   
question

   
RFC

   
Roadmap

   
stale

   
Store

   
Tests

   
Transfer Engine
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#208
No description provided.