mirror of
https://github.com/kvcache-ai/Mooncake.git
synced 2026-11-05 01:17:55 +00:00
[GH-ISSUE #95] Decoding error with TCP transport #52
Labels
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/Mooncake#52
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @ToBeResumed on GitHub (Feb 5, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/95
运行vllm_integration进行测试时，无法通过 Initial test，造成Initial test run failed。Decoding侧报错如下所示：
ERROR 02-05 16:48:45 client.py:250] RuntimeError('Engine loop has died')
ERROR 02-05 16:48:45 client.py:250] Traceback (most recent call last):
ERROR 02-05 16:48:45 client.py:250] File "/path/vllm/vllm/engine/multiprocessing/client.py", line 150, in run_heartbeat_loop
ERROR 02-05 16:48:45 client.py:250] await self._check_success(
ERROR 02-05 16:48:45 client.py:250] File "/path/vllm/vllm/engine/multiprocessing/client.py", line 314, in _check_success
ERROR 02-05 16:48:45 client.py:250] raise response
ERROR 02-05 16:48:45 client.py:250] RuntimeError: Engine loop has died
DEBUG 02-05 16:48:46 client.py:170] Waiting for output from MQLLMEngine.
ERROR: Exception in ASGI application
Traceback (most recent call last):
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/starlette/responses.py", line 259, in call
await wrap(partial(self.listen_for_disconnect, receive))
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/starlette/responses.py", line 255, in wrap
await func()
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/starlette/responses.py", line 232, in listen_for_disconnect
message = await receive()
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 563, in receive
await self.message_event.wait()
File "/miniconda3/envs/mooncake/lib/python3.10/asyncio/locks.py", line 214, in wait
await fut
asyncio.exceptions.CancelledError: Cancelled by cancel scope 7f6525099240
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/uvicorn/protocols/http/httptools_impl.py", line 409, in run_asgi
result = await app( # type: ignore[func-returns-value]
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in call
return await self.app(scope, receive, send)
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/fastapi/applications.py", line 1054, in call
await super().call(scope, receive, send)
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/starlette/applications.py", line 113, in call
await self.middleware_stack(scope, receive, send)
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/starlette/middleware/errors.py", line 187, in call
raise exc
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/starlette/middleware/errors.py", line 165, in call
await self.app(scope, receive, _send)
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/starlette/middleware/cors.py", line 85, in call
await self.app(scope, receive, send)
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/starlette/middleware/exceptions.py", line 62, in call
await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/starlette/routing.py", line 715, in call
await self.middleware_stack(scope, receive, send)
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/starlette/routing.py", line 735, in app
await route.handle(scope, receive, send)
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/starlette/routing.py", line 288, in handle
await self.app(scope, receive, send)
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/starlette/routing.py", line 76, in app
await wrap_app_handling_exceptions(app, request)(scope, receive, send)
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app
raise exc
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app
await app(scope, receive, sender)
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/starlette/routing.py", line 74, in app
await response(scope, receive, send)
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/starlette/responses.py", line 252, in call
async with anyio.create_task_group() as task_group:
File "/miniconda3/envs/mooncake/lib/python3.10/site-packages/anyio/_backends/_asyncio.py", line 767, in aexit
raise BaseExceptionGroup(
exceptiongroup.ExceptionGroup: unhandled errors in a TaskGroup (1 sub-exception)
请问这该如何排查
@ToBeResumed commented on GitHub (Feb 5, 2025):
PD节点和请求节点均在同一个服务器上，均在本地运行。设备为V100，模型为Meta-Llama-3-8B-Instruct，PD节点在不同卡上加载模型，显存未超，端口设置同项目文档
@ShangmingCai commented on GitHub (Feb 6, 2025):
@ToBeResumed 您截取的日志没有出错的关键信息。
这行表明engine已经die了，请给出P节点与D节点上有关键信息的日志或者完整错误日志，以辅助定位是kv send不成功还是其它问题。
@ToBeResumed commented on GitHub (Feb 7, 2025):
我解决了这个问题，用 #8 的方法调整了端口号。
感谢您的回复！