[GH-ISSUE #180 ] Connection refused error when using mooncake store as kvcache store for XpYd disaggregation #87

Author

Owner

@ShangmingCai commented on GitHub (Mar 28, 2025):

@alogfans Can you check this? I think it might be related to the etcd server or master server setup.

@ShangmingCai commented on GitHub (Mar 28, 2025): @alogfans Can you check this? I think it might be related to the etcd server or master server setup.

bsakaguchi commented

Author

Owner

@xiaguan commented on GitHub (Mar 28, 2025):

The etcd and master services seem to be running fine.

The issue appears to be a failure to establish a TCP connection to the target host.

@xiaguan commented on GitHub (Mar 28, 2025): The etcd and master services seem to be running fine. The issue appears to be a failure to establish a TCP connection to the target host.

bsakaguchi commented

Author

Owner

@ShangmingCai commented on GitHub (Mar 28, 2025):

@Second222None, can you provide the full log to show us whether the mooncake store connection is initialized successfully when initializing vLLM? Also, providing the master server log will help us to identify the root cause.

@ShangmingCai commented on GitHub (Mar 28, 2025): @Second222None, can you provide the full log to show us whether the mooncake store connection is initialized successfully when initializing vLLM? Also, providing the master server log will help us to identify the root cause.

bsakaguchi commented

decode1-202503302222.log
etcd-202503311320.log
mooncake_master-202503311320.log
prefill-202503302222.log
proxy-202503311325.log

Author

Owner

@Second222None commented on GitHub (Mar 31, 2025):

Sorry for late reply! The error occured in 1P2D scenario under LLMPerf test. For simplicity, we setup 1P1D( PD instances each run in a docker and mooncake store runs on the same host.) and collect the full log. Please find the attached full log.

We see another error(object_already_exist) and the performance number(TTFT) is not within our expectation. Any suggestion is welcome!

TTFT performance number(ms):

num-concurrent-requests	Baseline	1P1D
1	55	442.2
30	141.5	6114.8
60	211.5	12671.7

@Second222None commented on GitHub (Mar 31, 2025): Sorry for late reply! The error occured in 1P2D scenario under LLMPerf test. For simplicity, we setup 1P1D( PD instances each run in a docker and mooncake store runs on the same host.) and collect the full log. Please find the attached full log. [decode1-202503302222.log](https://github.com/user-attachments/files/19530897/decode1-202503302222.log) [etcd-202503311320.log](https://github.com/user-attachments/files/19530895/etcd-202503311320.log) [mooncake_master-202503311320.log](https://github.com/user-attachments/files/19530896/mooncake_master-202503311320.log) [prefill-202503302222.log](https://github.com/user-attachments/files/19530900/prefill-202503302222.log) [proxy-202503311325.log](https://github.com/user-attachments/files/19530899/proxy-202503311325.log) We see another error(object_already_exist) and the performance number(TTFT) is not within our expectation. Any suggestion is welcome! TTFT performance number(ms): |num-concurrent-requests | Baseline | 1P1D | |--|--|--| 1 | 55 | 442.2 | 30| 141.5 | 6114.8 | 60| 211.5 | 12671.7 |

bsakaguchi commented

Author

Owner

@ShangmingCai commented on GitHub (Mar 31, 2025):

@Second222None Thx for the logs.

I see some lines in the logs of decode node:
INFO 03-30 23:16:17 [metrics.py:481] Avg prompt throughput: 778.8 tokens/s, Avg generation throughput: 29.0 tokens/s, Running: 5 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.4%, CPU KV cache usage: 0.0%.

It means that the decode instances failed to receive the kvcache, so they are running the prefill step by themselves, so the Avg prompt throughput is not zero.

If it runs successfully, you should be able to see this debug info:
DEBUG 03-31 15:49:48 [mooncake_store_connector.py:202] [rank0]: Successfully received all KVs and hidden states, skip model forwarding. (VllmWorkerProcess pid=1586499) DEBUG 03-31 15:49:48 [mooncake_store_connector.py:202] [rank1]: Successfully received all KVs and hidden states, skip model forwarding.

Also, the TTFT is supposed to be larger than the vanilla vllm without kvcache transfer before we impl layer-wise async transfer. If you want to reduce TTFT, you should increase the prefill node, instead of the decode node. Increase the number of decode instances will show some performance improvement on TPOT and ITL.

@ShangmingCai commented on GitHub (Mar 31, 2025): @Second222None Thx for the logs. I see some lines in the logs of decode node: `INFO 03-30 23:16:17 [metrics.py:481] Avg prompt throughput: 778.8 tokens/s, Avg generation throughput: 29.0 tokens/s, Running: 5 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 1.4%, CPU KV cache usage: 0.0%.` It means that the decode instances failed to receive the kvcache, so they are running the prefill step by themselves, so the Avg prompt throughput is not zero. If it runs successfully, you should be able to see this debug info: ` DEBUG 03-31 15:49:48 [mooncake_store_connector.py:202] [rank0]: Successfully received all KVs and hidden states, skip model forwarding. (VllmWorkerProcess pid=1586499) DEBUG 03-31 15:49:48 [mooncake_store_connector.py:202] [rank1]: Successfully received all KVs and hidden states, skip model forwarding. ` Also, the TTFT is supposed to be larger than the vanilla vllm without kvcache transfer before we impl layer-wise async transfer. If you want to reduce TTFT, you should increase the prefill node, instead of the decode node. Increase the number of decode instances will show some performance improvement on TPOT and ITL.

bsakaguchi commented

Author

Owner

@ShangmingCai commented on GitHub (Mar 31, 2025):

@xiaguan Any idea why all error code info is object_already_exists?

@ShangmingCai commented on GitHub (Mar 31, 2025): @xiaguan Any idea why all error code info is `object_already_exists`?

bsakaguchi commented

Author

Owner

@xiaguan commented on GitHub (Mar 31, 2025):

It simply illustrates that some objects have been inserted multiple times, without compromising the program's correctness. Going forward, it might be more appropriate to classify this log as a debug message, so it doesn't appear by default.

@xiaguan commented on GitHub (Mar 31, 2025): It simply illustrates that some objects have been inserted multiple times, without compromising the program's correctness. Going forward, it might be more appropriate to classify this log as a debug message, so it doesn't appear by default.

bsakaguchi commented

Author

Owner

@zhenwei-intel commented on GitHub (Mar 31, 2025):

Hi @ShangmingCai, @xiaguan
I also encountered the same problem. Can you help me explain object_not_found and object_already_exists? And what their appearance represents?

@zhenwei-intel commented on GitHub (Mar 31, 2025): Hi @ShangmingCai, @xiaguan I also encountered the same problem. Can you help me explain `object_not_found` and `object_already_exists`? And what their appearance represents?

bsakaguchi commented

Author

Owner

@ShangmingCai commented on GitHub (Mar 31, 2025):

@Second222None BTW, can you provide the full command to start all vllm instances? Also, did you install the latest MooncakeStore from the main branch source code, or are you using the release version?

@ShangmingCai commented on GitHub (Mar 31, 2025): @Second222None BTW, can you provide the full command to start all vllm instances? Also, did you install the latest MooncakeStore from the main branch source code, or are you using the release version?

bsakaguchi commented

Author

Owner

@xiaguan commented on GitHub (Mar 31, 2025):

object_not_found indicates that the object has never been put, meaning it wasn't found—this is consistent with the semantics of other databases.

object_already_exists signifies to put the same object multiple times. Strictly speaking, this is not an error; We'll classify this log as a debug message later.

@xiaguan commented on GitHub (Mar 31, 2025): `object_not_found` indicates that the object has never been put, meaning it wasn't found—this is consistent with the semantics of other databases. `object_already_exists` signifies to put the same object multiple times. Strictly speaking, this is not an error; We'll classify this log as a debug message later.

bsakaguchi commented

Author

Owner

@ShangmingCai commented on GitHub (Mar 31, 2025):

Hi @ShangmingCai, @xiaguan I also encountered the same problem. Can you help me explain object_not_found and object_already_exists? And what their appearance represents?

Do you use the same prompt to invoke requests multiple times?

@xiaguan Maybe there might exist some problems when multiple requests want to put the kvcache with the same key since they have the same prompt.

@ShangmingCai commented on GitHub (Mar 31, 2025): > Hi [@ShangmingCai](https://github.com/ShangmingCai), [@xiaguan](https://github.com/xiaguan) I also encountered the same problem. Can you help me explain `object_not_found` and `object_already_exists`? And what their appearance represents? Do you use the same prompt to invoke requests multiple times? @xiaguan Maybe there might exist some problems when multiple requests want to put the kvcache with the same key since they have the same prompt.

bsakaguchi commented

Author

Owner

@ShangmingCai commented on GitHub (Mar 31, 2025):

@Second222None @zhenwei-intel Also, tcp is supposed to be slow. I think rdma is required if we want to minimize the communication overhead's impact on TTFT.

@ShangmingCai commented on GitHub (Mar 31, 2025): @Second222None @zhenwei-intel Also, tcp is supposed to be slow. I think rdma is required if we want to minimize the communication overhead's impact on TTFT.

bsakaguchi commented

Author

Owner

@zhenwei-intel commented on GitHub (Mar 31, 2025):

This error occurred this morning, but now I can't reproduce it: TcpTransport: ASIO exception: connect: Connection refused

I launched a 1P1D setup on 2x V100 GPUs with the following commands:

+ model=/workspace/lzw/Meta-Llama-3.1-8B-Instruct/
+ etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://localhost:2379

+ mooncake_master --port 50001

+ CUDA_VISIBLE_DEVICES=0
+ python3 -m vllm.entrypoints.openai.api_server --model /workspace/lzw/Meta-Llama-3.1-8B-Instruct/ --port 8100 --max-model-len 10000 --dtype half --gpu-memory
-utilization 0.9 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2,"kv_buffer_size":5e9}

+ CUDA_VISIBLE_DEVICES=1
+ python3 -m vllm.entrypoints.openai.api_server --model /workspace/lzw/Meta-Llama-3.1-8B-Instruct/ --port 8200 --max-model-len 10000 --dtype half --gpu-memory
-utilization 0.9 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2,"kv_buffer_size":5e9}

+ python disagg_proxy_mooncake.py --model /workspace/lzw/Meta-Llama-3.1-8B-Instruct/ --prefill localhost:8100 --decode localhost:8200 --port 8000

benchmark scripts

python ../benchmark_serving.py --backend vllm --model /workspace/lzw/Meta-Llama-3.1-8B-Instruct/ --dataset-name random --random-input-len 256 --random-output-len 256 --num-prompts 128 --port 8000 --host localhost --request-rate 4

mooncake json

{
    "local_hostname": "localhost",
    "metadata_server": "etcd://localhost:2379",
    "protocol": "tcp",
    "device_name": "",
    "master_server_address": "localhost:50001"
}

Current issues:

1. In mooncake_store_connector.py, after self.kv_store.put(store_kvcache_key, kvcache_to_sent), I printed the key names and found duplicates:
1. The consumer logs do not show received all KVs

@zhenwei-intel commented on GitHub (Mar 31, 2025): This error occurred this morning, but now I can't reproduce it: `TcpTransport: ASIO exception: connect: Connection refused` I launched a 1P1D setup on 2x V100 GPUs with the following commands: ``` + model=/workspace/lzw/Meta-Llama-3.1-8B-Instruct/ + etcd --listen-client-urls http://0.0.0.0:2379 --advertise-client-urls http://localhost:2379 + mooncake_master --port 50001 + CUDA_VISIBLE_DEVICES=0 + python3 -m vllm.entrypoints.openai.api_server --model /workspace/lzw/Meta-Llama-3.1-8B-Instruct/ --port 8100 --max-model-len 10000 --dtype half --gpu-memory -utilization 0.9 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2,"kv_buffer_size":5e9} + CUDA_VISIBLE_DEVICES=1 + python3 -m vllm.entrypoints.openai.api_server --model /workspace/lzw/Meta-Llama-3.1-8B-Instruct/ --port 8200 --max-model-len 10000 --dtype half --gpu-memory -utilization 0.9 --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2,"kv_buffer_size":5e9} + python disagg_proxy_mooncake.py --model /workspace/lzw/Meta-Llama-3.1-8B-Instruct/ --prefill localhost:8100 --decode localhost:8200 --port 8000 ``` benchmark scripts ``` python ../benchmark_serving.py --backend vllm --model /workspace/lzw/Meta-Llama-3.1-8B-Instruct/ --dataset-name random --random-input-len 256 --random-output-len 256 --num-prompts 128 --port 8000 --host localhost --request-rate 4 ``` mooncake json ``` { "local_hostname": "localhost", "metadata_server": "etcd://localhost:2379", "protocol": "tcp", "device_name": "", "master_server_address": "localhost:50001" } ``` Current issues: - 1. In `mooncake_store_connector.py`, after `self.kv_store.put(store_kvcache_key, kvcache_to_sent)`, I printed the key names and found duplicates: ![Image](https://github.com/user-attachments/assets/0b3fba1c-b2bc-4a93-a480-e7a27910e860) - 2. The consumer logs do not show `received all KVs`

bsakaguchi commented

Author

Owner

@ShangmingCai commented on GitHub (Mar 31, 2025):

@zhenwei-intel
I think it could be related to concurrent duplicate requests with the same prompt. The hash is computed according to the token_ids, so when one instance hasn't finished putting a kvcache, another instance want to put a kvcache with the same key will report an error, right? @xiaguan

BTW, you don't need to setup kv_rank, kv_parallel_size, and kv_buffer_size for MooncakeStoreConnector.

@ShangmingCai commented on GitHub (Mar 31, 2025): @zhenwei-intel I think it could be related to concurrent duplicate requests with the same prompt. The hash is computed according to the token_ids, so when one instance hasn't finished putting a kvcache, another instance want to put a kvcache with the same key will report an error, right? @xiaguan BTW, you don't need to setup `kv_rank`, `kv_parallel_size`, and `kv_buffer_size` for `MooncakeStoreConnector`.

bsakaguchi commented

Author

Owner

@xiaguan commented on GitHub (Mar 31, 2025):

@zhenwei-intel I think it could be related to concurrent duplicate requests with the same prompt. The hash is computed according to the token_ids, so when one instance hasn't finished putting a kvcache, another instance want to put a kvcache with the same key will report an error, right?

Yes. More precisely, each object can only be put once; subsequent puts will be skipped (since by design, if the key is the same, the value will be the same), and anobject_already_exist message will be printed.

@xiaguan commented on GitHub (Mar 31, 2025): > [@zhenwei-intel](https://github.com/zhenwei-intel) I think it could be related to concurrent duplicate requests with the same prompt. The hash is computed according to the token_ids, so when one instance hasn't finished putting a kvcache, another instance want to put a kvcache with the same key will report an error, right? Yes. More precisely, each object can only be put once; subsequent puts will be skipped (since by design, if the key is the same, the value will be the same), and an`object_already_exist` message will be printed.

bsakaguchi commented

Author

Owner

@Second222None commented on GitHub (Mar 31, 2025):

@zhenwei-intel I think it could be related to concurrent duplicate requests with the same prompt. The hash is computed according to the token_ids, so when one instance hasn't finished putting a kvcache, another instance want to put a kvcache with the same key will report an error, right? 我认为这可能与具有相同提示的并发重复请求有关。哈希值是根据 token_ids 计算的，所以当一个实例还没放完 kvcache 时，另一个实例想要放同一个 key 的 kvcache 就会报错，对吧？@xiaguan

Yes. More precisely, each object can only be put once; subsequent puts will be skipped (since by design, if the key is the same, the value will be the same), and anobject_already_exist message will be printed.

According to the Mooncake Store doc, there is a TTL feature of get(). If put() skip the duplicate item, for same prompt, MooncakeStore doesn't ensure decoder can get the same kvcache multiple times. Am I right? How to turn off TTL feature.

@Second222None commented on GitHub (Mar 31, 2025): > > [@zhenwei-intel](https://github.com/zhenwei-intel) I think it could be related to concurrent duplicate requests with the same prompt. The hash is computed according to the token_ids, so when one instance hasn't finished putting a kvcache, another instance want to put a kvcache with the same key will report an error, right? 我认为这可能与具有相同提示的并发重复请求有关。哈希值是根据 token_ids 计算的，所以当一个实例还没放完 kvcache 时，另一个实例想要放同一个 key 的 kvcache 就会报错，对吧？[@xiaguan](https://github.com/xiaguan) > > Yes. More precisely, each object can only be put once; subsequent puts will be skipped (since by design, if the key is the same, the value will be the same), and an`object_already_exist` message will be printed. According to the Mooncake Store doc, there is a TTL feature of get(). If put() skip the duplicate item, for same prompt, MooncakeStore doesn't ensure decoder can get the same kvcache multiple times. Am I right? How to turn off TTL feature.

bsakaguchi commented

Author

Owner

@xiaguan commented on GitHub (Mar 31, 2025):

According to the Mooncake Store doc, there is a TTL feature of get().

Yes, the master service has a TTL feature (disabled by default on the main branch). It deletes the object 1 second after receiving a GET request to free up space. This is mainly to address allocation failure issues and is not closely related to this issue. Apologies for the confusion caused — the documentation will be updated accordingly.

Trun off it

mooncake_master --enable_gc=false

If put() skip the duplicate item, for same prompt, MooncakeStore doesn't ensure decoder can get the same kvcache multiple times. Am I right? How to turn off TTL feature.

And yes, decode will only get the first successfully put value.

@xiaguan commented on GitHub (Mar 31, 2025): > According to the Mooncake Store doc, there is a TTL feature of get(). Yes, the master service has a TTL feature (disabled by default on the main branch). It deletes the object 1 second after receiving a GET request to free up space. This is mainly to address allocation failure issues and is not closely related to this issue. Apologies for the confusion caused — the documentation will be updated accordingly. Trun off it ``` mooncake_master --enable_gc=false ``` > If put() skip the duplicate item, for same prompt, MooncakeStore doesn't ensure decoder can get the same kvcache multiple times. Am I right? How to turn off TTL feature. And yes, decode will only get the first successfully put value.

bsakaguchi commented

Author

Owner

@Second222None commented on GitHub (Mar 31, 2025):

According to the Mooncake Store doc, there is a TTL feature of get().

Yes, the master service has a TTL feature (disabled by default on the main branch). It deletes the object 1 second after receiving a GET request to free up space. This is mainly to address allocation failure issues and is not closely related to this issue. Apologies for the confusion caused — the documentation will be updated accordingly.

Trun off it
mooncake_master --enable_gc=false
If put() skip the duplicate item, for same prompt, MooncakeStore doesn't ensure decoder can get the same kvcache multiple times. Am I right? How to turn off TTL feature.

And yes, decode will only get the first successfully put value.

Another question. Does main branch has an eviction mechanism to delete the stale item?

@Second222None commented on GitHub (Mar 31, 2025): > > According to the Mooncake Store doc, there is a TTL feature of get(). > > Yes, the master service has a TTL feature (disabled by default on the main branch). It deletes the object 1 second after receiving a GET request to free up space. This is mainly to address allocation failure issues and is not closely related to this issue. Apologies for the confusion caused — the documentation will be updated accordingly. > > Trun off it > > ``` > mooncake_master --enable_gc=false > ``` > > > If put() skip the duplicate item, for same prompt, MooncakeStore doesn't ensure decoder can get the same kvcache multiple times. Am I right? How to turn off TTL feature. > > And yes, decode will only get the first successfully put value. Another question. Does main branch has an eviction mechanism to delete the stale item?

bsakaguchi commented

Author

Owner

@xiaguan commented on GitHub (Mar 31, 2025):

Another question. Does main branch has an eviction mechanism to delete the stale item?

No. Except for the previously mentioned GC.

@xiaguan commented on GitHub (Mar 31, 2025): > Another question. Does main branch has an eviction mechanism to delete the stale item? No. Except for the previously mentioned GC.

bsakaguchi commented

Author

Owner

@ShangmingCai commented on GitHub (Mar 31, 2025):

@Second222None LRU-based GC will be supported in the future once we align with the v1 design.

@ShangmingCai commented on GitHub (Mar 31, 2025): @Second222None LRU-based GC will be supported in the future once we align with the v1 design.

bsakaguchi commented

Author

Owner

@Huixxi commented on GitHub (Apr 1, 2025):

$ mooncake_master --port 50001
bash: mooncake_master: command not found
I met this problem after install mooncake follow this doc: https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/build.md

@Huixxi commented on GitHub (Apr 1, 2025): $ mooncake_master --port 50001 bash: mooncake_master: command not found I met this problem after install mooncake follow this doc: https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/build.md

bsakaguchi commented

Author

Owner

@ShangmingCai commented on GitHub (Apr 1, 2025):

$ mooncake_master --port 50001 bash: mooncake_master: command not found I met this problem after install mooncake follow this doc: https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/build.md

@Huixxi Can you try pulling the latest updates on the main branch? We are refactoring the compiling and building part now, so it is possible that your clone is in a stale state.

@ShangmingCai commented on GitHub (Apr 1, 2025): > $ mooncake_master --port 50001 bash: mooncake_master: command not found I met this problem after install mooncake follow this doc: https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/build.md @Huixxi Can you try pulling the latest updates on the main branch? We are refactoring the compiling and building part now, so it is possible that your clone is in a stale state.

bsakaguchi commented

Author

Owner

@Huixxi commented on GitHub (Apr 1, 2025):

$ mooncake_master --port 50001 bash: mooncake_master: command not found I met this problem after install mooncake follow this doc: https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/build.md

@Huixxi Can you try pulling the latest updates on the main branch? We are refactoring the compiling and building part now, so it is possible that your clone is in a stale state.

Ok, thanks, but I eventually find it here Mooncake/build/mooncake-store/src/ 😅

@Huixxi commented on GitHub (Apr 1, 2025): > > $ mooncake_master --port 50001 bash: mooncake_master: command not found I met this problem after install mooncake follow this doc: https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/build.md > > [@Huixxi](https://github.com/Huixxi) Can you try pulling the latest updates on the main branch? We are refactoring the compiling and building part now, so it is possible that your clone is in a stale state. Ok, thanks, but I eventually find it here Mooncake/build/mooncake-store/src/ 😅

bsakaguchi commented

Author

Owner

@Huixxi commented on GitHub (Apr 2, 2025):

$ mooncake_master --port 50001 bash: mooncake_master: command not found I met this problem after install mooncake follow this doc: https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/build.md

@Huixxi Can you try pulling the latest updates on the main branch? We are refactoring the compiling and building part now, so it is possible that your clone is in a stale state.

Could you please help with this issue ? https://github.com/kvcache-ai/Mooncake/issues/201

@Huixxi commented on GitHub (Apr 2, 2025): > > $ mooncake_master --port 50001 bash: mooncake_master: command not found I met this problem after install mooncake follow this doc: https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/build.md > > [@Huixxi](https://github.com/Huixxi) Can you try pulling the latest updates on the main branch? We are refactoring the compiling and building part now, so it is possible that your clone is in a stale state. Could you please help with this issue ? https://github.com/kvcache-ai/Mooncake/issues/201

bsakaguchi commented

Author

Owner

@Zhou-sx commented on GitHub (Apr 16, 2025):

I encountered the same problem, too. @zhenwei-intel @Second222None Have you solved this problem?

@Zhou-sx commented on GitHub (Apr 16, 2025): I encountered the same problem, too. @zhenwei-intel @Second222None Have you solved this problem?

bsakaguchi commented

Author

Owner

@ShangmingCai commented on GitHub (Apr 16, 2025):

I encountered the same problem, too.

@Zhou-sx Can you check whether your mooncake_master has port conflict? lsof -i:50051, you should not open multiple mooncake_master , or set up to a port that has been used.

@ShangmingCai commented on GitHub (Apr 16, 2025): > I encountered the same problem, too. @Zhou-sx Can you check whether your mooncake_master has port conflict?` lsof -i:50051`, you should not open multiple mooncake_master , or set up to a port that has been used.

bsakaguchi commented

Author

Owner

@Zhou-sx commented on GitHub (Apr 16, 2025):

lsof -i:50051

I used 50001. If i used conflict port, I will see infomation like these: Port 50001 is already in use. Please specify a different port using --port flag,but I am not.
And the problem appeared after running some steps. I wonder if there are some other problems.

@Zhou-sx commented on GitHub (Apr 16, 2025): > lsof -i:50051 I used 50001. If i used conflict port, I will see infomation like these: Port 50001 is already in use. Please specify a different port using --port flag,but I am not. And the problem appeared after running some steps. I wonder if there are some other problems.

bsakaguchi commented

github.com/kvcache-ai/Mooncake@142e04e526/mooncake-integration/store/store_py.cpp (L101)

Author

Owner

@xiaguan commented on GitHub (Apr 16, 2025):

If you are using the Python binding, mooncake store will randomly select a port for registration—please ensure that ports within this range are available. Alternatively, you can specify it directly by appending it to localhost, as in localhost:12345.

It would be even more helpful if you could provide detailed information about the error.

@xiaguan commented on GitHub (Apr 16, 2025): https://github.com/kvcache-ai/Mooncake/blob/142e04e5264b891a8f384a600f075962f8db132e/mooncake-integration/store/store_py.cpp#L101 If you are using the Python binding, mooncake store will randomly select a port for registration—please ensure that ports within this range are available. Alternatively, you can specify it directly by appending it to localhost, as in `localhost:12345`. It would be even more helpful if you could provide detailed information about the error.

bsakaguchi commented

Author

Owner

@Huixxi commented on GitHub (Apr 16, 2025):

lsof -i:50051

I used 50001. If i used conflict port, I will see infomation like these: Port 50001 is already in use. Please specify a different port using --port flag,but I am not. And the problem appeared after running some steps. I wonder if there are some other problems.

emmm, FYI. I changed this port from 50001 to 5001

@Huixxi commented on GitHub (Apr 16, 2025): > > lsof -i:50051 > > I used 50001. If i used conflict port, I will see infomation like these: Port 50001 is already in use. Please specify a different port using --port flag,but I am not. And the problem appeared after running some steps. I wonder if there are some other problems. emmm, FYI. I changed this port from 50001 to 5001

bsakaguchi commented