kvcache-ai/Mooncake

Fork 0

mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00

[GH-ISSUE #154] mooncake-integration #73

New issue

Closed

opened 2026-10-29 21:08:46 +00:00 by bsakaguchi · 4 comments

bsakaguchi commented

2026-10-29 21:08:46 +00:00

Owner

Originally created by @dongliangwu on GitHub (Mar 20, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/154

i want to test mooncake integration with vllm.
i use this container to build from source code ------ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/pytorch-ubuntu:2.3.0-cuda12.1.1-ubuntu22.04.
so i got mooncake_vllm_adaptor.cpthon-310-x86_64-linux-gnu.so successfully and i move it to my python system path.but when i run vllm, i got this error:
ImportError: /usr/lib/python3/dist-packages/mooncake_vllm_adaptor.cpthon-310-x86_64-linux-gnu.so: undefined symbol: _ZNK6google8protobuf7Message25InitializationErrorStringEv
then,i modify the CMakeLists for mooncake_vllm_adaptor and add protobuf (i do not build grpc from source code ,the container i used has a v1.30 grpc)into it:

file(GLOB SOURCES "*.cpp")
set(PYTHON_EXECUTABLE "python3")
execute_process(
COMMAND ${PYTHON_EXECUTABLE} -c "import sys; print(sys.path[-1])"
OUTPUT_VARIABLE PYTHON_SYS_PATH
)
string(STRIP ${PYTHON_SYS_PATH} PYTHON_SYS_PATH)

include_directories("../mooncake-store/include")
include_directories("../mooncake-store/include/cachelib_memory_allocator")

include_directories("../mooncake-store/include/cachelib_memory_allocator/include")
include_directories("../mooncake-store/include/cachelib_memory_allocator/fake_include")

set(PYBIND11_FINDPYTHON ON)
find_package(pybind11 CONFIG REQUIRED)
pybind11_add_module(mooncake_vllm_adaptor ${SOURCES} ${CACHE_ALLOCATOR_SOURCES}
vllm/vllm_adaptor.cpp
vllm/distributed_object_store.cpp
)

find_package(Protobuf REQUIRED)
include_directories(${Protobuf_INCLUDE_DIRS})

target_link_libraries(mooncake_vllm_adaptor PUBLIC
transfer_engine
glog
gflags
cache_allocator
cachelib_memory_allocator

protobuf::libprotobuf

)
message("${PYTHON_SYS_PATH}")
install(TARGETS mooncake_vllm_adaptor DESTINATION ${PYTHON_SYS_PATH}/)

and when i run vllm this time，i got another error:
undefined symbol: _ZN4grpc24g_core_codegen_interfaceE

Originally created by @dongliangwu on GitHub (Mar 20, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/154 i want to test mooncake integration with vllm. i use this container to build from source code ------ac2-registry.cn-hangzhou.cr.aliyuncs.com/ac2/pytorch-ubuntu:2.3.0-cuda12.1.1-ubuntu22.04. so i got mooncake_vllm_adaptor.cpthon-310-x86_64-linux-gnu.so successfully and i move it to my python system path.but when i run vllm, i got this error: ImportError: /usr/lib/python3/dist-packages/mooncake_vllm_adaptor.cpthon-310-x86_64-linux-gnu.so: undefined symbol: _ZNK6google8protobuf7Message25InitializationErrorStringEv then,i modify the CMakeLists for mooncake_vllm_adaptor and add protobuf (i do not build grpc from source code ,the container i used has a v1.30 grpc)into it: file(GLOB SOURCES "*.cpp") set(PYTHON_EXECUTABLE "python3") execute_process( COMMAND ${PYTHON_EXECUTABLE} -c "import sys; print(sys.path[-1])" OUTPUT_VARIABLE PYTHON_SYS_PATH ) string(STRIP ${PYTHON_SYS_PATH} PYTHON_SYS_PATH) include_directories("../mooncake-store/include") include_directories("../mooncake-store/include/cachelib_memory_allocator") include_directories("../mooncake-store/include/cachelib_memory_allocator/include") include_directories("../mooncake-store/include/cachelib_memory_allocator/fake_include") set(PYBIND11_FINDPYTHON ON) find_package(pybind11 CONFIG REQUIRED) pybind11_add_module(mooncake_vllm_adaptor ${SOURCES} ${CACHE_ALLOCATOR_SOURCES} vllm/vllm_adaptor.cpp vllm/distributed_object_store.cpp ) find_package(Protobuf REQUIRED) include_directories(${Protobuf_INCLUDE_DIRS}) target_link_libraries(mooncake_vllm_adaptor PUBLIC transfer_engine glog gflags cache_allocator cachelib_memory_allocator protobuf::libprotobuf ) message("${PYTHON_SYS_PATH}") install(TARGETS mooncake_vllm_adaptor DESTINATION ${PYTHON_SYS_PATH}/) and when i run vllm this time，i got another error: undefined symbol: _ZN4grpc24g_core_codegen_interfaceE

bsakaguchi closed this issue

2026-10-29 21:08:46 +00:00

bsakaguchi commented

2026-10-29 21:08:46 +00:00

Author

Owner

@dongliangwu commented on GitHub (Mar 20, 2025):

rebuild mooncake using a old version(before this pr https://github.com/kvcache-ai/Mooncake/pull/126), i walkaround this problem. but when i try to start prfill or decode node,i got this error: torch.distributed.DistNetworkError:The client socket has timed out after 60s while trying to connect to (xxxx, 51000).
i start prefill/decode node this way():

1. Start the http server

2. Run on the prefilling side (producer role)

VLLM_HOST_IP="192.168.0.137" VLLM_PORT="51000" MASTER_ADDR="192.168.0.137" MASTER_PORT="54324" MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_DISTRIBUTED_KV_ROLE=producer VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8100 --max-model-len 10000 --gpu-memory-utilization 0.95

3. Run on the decoding side (consumer role)

VLLM_HOST_IP="192.168.0.137" VLLM_PORT="51000" MASTER_ADDR="192.168.0.137" MASTER_PORT="54324" MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_DISTRIBUTED_KV_ROLE=consumer VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8200 --max-model-len 10000 --gpu-memory-utilization 0.95

@dongliangwu commented on GitHub (Mar 20, 2025): rebuild mooncake using a old version(before this pr https://github.com/kvcache-ai/Mooncake/pull/126), i walkaround this problem. but when i try to start prfill or decode node,i got this error: torch.distributed.DistNetworkError:The client socket has timed out after 60s while trying to connect to (xxxx, 51000). i start prefill/decode node this way(): # 1. Start the http server # 2. Run on the prefilling side (producer role) VLLM_HOST_IP="192.168.0.137" VLLM_PORT="51000" MASTER_ADDR="192.168.0.137" MASTER_PORT="54324" MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_DISTRIBUTED_KV_ROLE=producer VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8100 --max-model-len 10000 --gpu-memory-utilization 0.95 # 3. Run on the decoding side (consumer role) VLLM_HOST_IP="192.168.0.137" VLLM_PORT="51000" MASTER_ADDR="192.168.0.137" MASTER_PORT="54324" MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_DISTRIBUTED_KV_ROLE=consumer VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8200 --max-model-len 10000 --gpu-memory-utilization 0.95

bsakaguchi commented

2026-10-29 21:08:47 +00:00

Author

Owner

@stmatengss commented on GitHub (Mar 20, 2025):

Thank you for the feedback! We haven't updated the Dockerfile recently. Could you check the connectivity between the prefilling node and the decoding node?

@stmatengss commented on GitHub (Mar 20, 2025): Thank you for the feedback! We haven't updated the Dockerfile recently. Could you check the connectivity between the prefilling node and the decoding node?

bsakaguchi commented

2026-10-29 21:08:47 +00:00

Author

Owner

@dongliangwu commented on GitHub (Mar 20, 2025):

Thank you for the feedback! We haven't updated the Dockerfile recently. Could you check the connectivity between the prefilling node and the decoding node?

the network is ok and i found it stuck in here:

and i found the debug log says "backend==nccl" ,is it ok?
thx!

@dongliangwu commented on GitHub (Mar 20, 2025): > Thank you for the feedback! We haven't updated the Dockerfile recently. Could you check the connectivity between the prefilling node and the decoding node? the network is ok and i found it stuck in here: ![Image](https://github.com/user-attachments/assets/b654adff-357e-4975-a5ad-839026f5f57b) and i found the debug log says "backend==nccl" ,is it ok? thx!

bsakaguchi commented

2026-10-29 21:08:47 +00:00

Author

Owner

@ShangmingCai commented on GitHub (Mar 21, 2025):

You are using v0.1, which is a deprecated version. Please check https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/vllm-integration-v1.md

@ShangmingCai commented on GitHub (Mar 21, 2025): You are using v0.1, which is a deprecated version. Please check https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/vllm-integration-v1.md

bsakaguchi referenced this issue

2026-10-29 21:26:16 +00:00

[PR #73] [MERGED] [TransferEngine] change: auto discover topology & install transport. #1310

bsakaguchi referenced this issue

2026-10-29 21:26:19 +00:00

[PR #87] [MERGED] [TransferEngine] fix: add auto_discover in transfer_engine_c #1318