mirror of
https://github.com/kvcache-ai/Mooncake.git
synced 2026-11-05 01:17:55 +00:00
[GH-ISSUE #201] client has been closed, please re-connect but response success??? #90
Labels
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/Mooncake#90
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @Huixxi on GitHub (Apr 2, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/201
I followed https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/vllm-integration-v1.md with 1P1D, the response seems right but there are a lot error log. Could you please help me find out what happened?
Prefill Node0( mooncake_master and etcd server on this same node ):
Decode Node0:
Client Side( and the Proxy below are on the same node with decode):
Proxy:
@ShangmingCai commented on GitHub (Apr 2, 2025):
@Huixxi Can you provide the log of mooncake_master?
@xiaguan Please check whether there is something wrong with the coro rpc.
@ShangmingCai commented on GitHub (Apr 2, 2025):
@Huixxi Response succeeds because the deocde node re-does the prefill on its side when the kvcache transfer fails. This is what
Failed to receive all KVs and hidden states, redo model forwardingmeans.@Huixxi commented on GitHub (Apr 2, 2025):
That is all, and there is no more new logs after my requested.
@ShangmingCai commented on GitHub (Apr 2, 2025):
Can I see the content of your mooncake.json file to check the config？
@Huixxi commented on GitHub (Apr 2, 2025):
@xiaguan commented on GitHub (Apr 2, 2025):
From the logs, the master seems to be running normally. The problem is probably on the client side. If the network is stable, it could be an issue with the
coro_rpc's client. I'll try to reproduce it later.@Huixxi commented on GitHub (Apr 2, 2025):
FYI. There is no logs print on mooncake_master side.
@Huixxi commented on GitHub (Apr 2, 2025):
Is it right?
@ShangmingCai commented on GitHub (Apr 2, 2025):
If the mooncake_master's log has no updates, it means your decode node is not able to connect to the mooncake master and the preifll node.
@xiaguan commented on GitHub (Apr 2, 2025):
Yes, by default, it only outputs error logs. If no logs are printed, it means there's likely nothing wrong with the master. If you wish to view more detailed logs, you can enable them by adding --v=1.
@ShangmingCai commented on GitHub (Apr 2, 2025):
Can you try LAN ip?
@Huixxi commented on GitHub (Apr 2, 2025):
still no logs info each time a request.
@Huixxi commented on GitHub (Apr 2, 2025):
Emm, two node can ping through each other. I think the ip connection is not the problem.
@xiaguan commented on GitHub (Apr 2, 2025):
This behavior is expected, as the issue lies on the client side—no requests have been sent. If you're confident that there are no communication issues between the client and the master machines (including port configurations, especially when using cloud servers), then it's more likely an internal logic error. Then I will identify the root of the issue as soon as possible.
@Huixxi commented on GitHub (Apr 2, 2025):
Thanks a lot!! And maybe I should also check the if the ports is ok.
@Huixxi commented on GitHub (Apr 2, 2025):
BTW. I met this crush below several times after I killed and restarted the prefill server when I sent a request with curl to the proxy. I don't know why. And! after I restart all those elements(etcd, prefill, decode, proxy, mooncake_master) it recoverd and the connection issue above only occurred on the decode side(the error log only appeared on decode side) and mooncake_master also print logs.
Prefill Logs: without error message
Decode Logs: with error message
mooncate_master Logs:
Cursh Logs:
@Huixxi commented on GitHub (Apr 2, 2025):
Does mooncake pd support large language model that need
--tp 8to deploy on a node with 8 GPU? Here is my prefill and decode server start command.Prefill Server:
Decode Server:
@ShangmingCai commented on GitHub (Apr 2, 2025):
decode node port should be 8200
@Huixxi commented on GitHub (Apr 2, 2025):
Ok I'll try it now, but the ports need to be different even if on two different nodes?
@ShangmingCai commented on GitHub (Apr 2, 2025):
In that case, 8100 should be fine. But I recommend you use local IP. Also you should try running the p and d on the same node first to ensure your problem is not related to the public ip address.
@Huixxi commented on GitHub (Apr 3, 2025):
Now, there seems no more problems and the connection seems also correct. But from the prefill and decode's log, it seems on kvcache reuse? Is 0.0% normal?
The request prompt is random, and max output len limited in 80, --max-model-len 1000.
Prefill Logs:
Decode Logs:
Mooncake master:
@Huixxi commented on GitHub (Apr 15, 2025):
The port conflict may cause this issue even on separate nodes.
@Zhou-sx commented on GitHub (Apr 16, 2025):
Hi~ Can you explain why it worked?