kvcache-ai/Mooncake
1
0
Fork 0
mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00
Code Issues 368 Projects Packages Wiki Activity

[GH-ISSUE #292] 1p1d, tcp, assert start_pos == num_prefill_tokens AssertionError() Please Help Me ! #115

New issue
Closed
opened 2026-10-29 21:09:37 +00:00 by bsakaguchi · 9 comments
bsakaguchi commented 2026-10-29 21:09:37 +00:00
Owner
Copy link

Originally created by @coding-rw on GitHub (Apr 23, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/292

1 prefill node(4L20), 1 decoding node(4L20), qwen2.5-32B-instruct

Bug on Decode Node

INFO 04-23 16:34:22 [metrics.py:489] Avg prompt throughput: 41.4 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%.
INFO 04-23 16:34:22 [metrics.py:505] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO:     172.30.11.146:54938 - "POST /v1/completions HTTP/1.1" 200 OK
INFO 04-23 16:34:32 [metrics.py:489] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.9 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO 04-23 16:34:32 [metrics.py:505] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO:     172.30.11.146:47808 - "POST /v1/completions HTTP/1.1" 200 OK
INFO 04-23 16:34:37 [metrics.py:489] Avg prompt throughput: 78.5 tokens/s, Avg generation throughput: 32.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%.
INFO 04-23 16:34:37 [metrics.py:505] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00%
INFO:     172.30.11.146:38062 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     172.30.11.146:38070 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     172.30.11.146:38084 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     172.30.11.146:38086 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     172.30.11.146:38098 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     172.30.11.146:38112 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     172.30.11.146:38138 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     172.30.11.146:38092 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     172.30.11.146:38128 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     172.30.11.146:38144 - "POST /v1/completions HTTP/1.1" 200 OK
WARNING 04-23 16:34:41 [mooncake_store_connector.py:137] You should set --enable_chunked_prefill=False and --max_num_batched_tokens should be equal to max_seq_len_to_capture
(VllmWorkerProcess pid=455906) WARNING 04-23 16:34:41 [mooncake_store_connector.py:137] You should set --enable_chunked_prefill=False and --max_num_batched_tokens should be equal to max_seq_len_to_capture
(VllmWorkerProcess pid=455907) WARNING 04-23 16:34:41 [mooncake_store_connector.py:137] You should set --enable_chunked_prefill=False and --max_num_batched_tokens should be equal to max_seq_len_to_capture
(VllmWorkerProcess pid=455908) WARNING 04-23 16:34:41 [mooncake_store_connector.py:137] You should set --enable_chunked_prefill=False and --max_num_batched_tokens should be equal to max_seq_len_to_capture
ERROR 04-23 16:34:41 [engine.py:160] AssertionError()
ERROR 04-23 16:34:41 [engine.py:160] Traceback (most recent call last):
ERROR 04-23 16:34:41 [engine.py:160]   File "/root/moonvllm/vllm/vllm/engine/multiprocessing/engine.py", line 158, in start
ERROR 04-23 16:34:41 [engine.py:160]     self.run_engine_loop()
ERROR 04-23 16:34:41 [engine.py:160]   File "/root/moonvllm/vllm/vllm/engine/multiprocessing/engine.py", line 221, in run_engine_loop
ERROR 04-23 16:34:41 [engine.py:160]     request_outputs = self.engine_step()
ERROR 04-23 16:34:41 [engine.py:160]                       ^^^^^^^^^^^^^^^^^^
ERROR 04-23 16:34:41 [engine.py:160]   File "/root/moonvllm/vllm/vllm/engine/multiprocessing/engine.py", line 247, in engine_step
ERROR 04-23 16:34:41 [engine.py:160]     raise e
ERROR 04-23 16:34:41 [engine.py:160]   File "/root/moonvllm/vllm/vllm/engine/multiprocessing/engine.py", line 230, in engine_step
ERROR 04-23 16:34:41 [engine.py:160]     return self.engine.step()
ERROR 04-23 16:34:41 [engine.py:160]            ^^^^^^^^^^^^^^^^^^
ERROR 04-23 16:34:41 [engine.py:160]   File "/root/moonvllm/vllm/vllm/engine/llm_engine.py", line 1435, in step
ERROR 04-23 16:34:41 [engine.py:160]     outputs = self.model_executor.execute_model(
ERROR 04-23 16:34:41 [engine.py:160]               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-23 16:34:41 [engine.py:160]   File "/root/moonvllm/vllm/vllm/executor/executor_base.py", line 299, in execute_model
ERROR 04-23 16:34:41 [engine.py:160]     driver_outputs = self._driver_execute_model(execute_model_req)
ERROR 04-23 16:34:41 [engine.py:160]                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-23 16:34:41 [engine.py:160]   File "/root/moonvllm/vllm/vllm/executor/mp_distributed_executor.py", line 144, in _driver_execute_model
ERROR 04-23 16:34:41 [engine.py:160]     return self.driver_worker.execute_model(execute_model_req)
ERROR 04-23 16:34:41 [engine.py:160]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-23 16:34:41 [engine.py:160]   File "/root/moonvllm/vllm/vllm/worker/worker_base.py", line 420, in execute_model
ERROR 04-23 16:34:41 [engine.py:160]     output = self.model_runner.execute_model(
ERROR 04-23 16:34:41 [engine.py:160]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-23 16:34:41 [engine.py:160]   File "/root/miniconda3/envs/moonvllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 04-23 16:34:41 [engine.py:160]     return func(*args, **kwargs)
ERROR 04-23 16:34:41 [engine.py:160]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 04-23 16:34:41 [engine.py:160]   File "/root/moonvllm/vllm/vllm/worker/model_runner.py", line 1753, in execute_model
ERROR 04-23 16:34:41 [engine.py:160]     get_kv_transfer_group().recv_kv_caches_and_hidden_states(
ERROR 04-23 16:34:41 [engine.py:160]   File "/root/moonvllm/vllm/vllm/distributed/kv_transfer/kv_connector/mooncake_store_connector.py", line 141, in recv_kv_caches_and_hidden_states
ERROR 04-23 16:34:41 [engine.py:160]     assert start_pos == num_prefill_tokens
ERROR 04-23 16:34:41 [engine.py:160]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 04-23 16:34:41 [engine.py:160] AssertionError
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] Exception in worker VllmWorkerProcess while processing method start_worker_execution_loop.
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] Traceback (most recent call last):
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/moonvllm/vllm/vllm/executor/multiproc_worker_utils.py", line 232, in _run_worker_process
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     output = run_method(worker, method, args, kwargs)
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] Exception in worker VllmWorkerProcess while processing method start_worker_execution_loop.
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/moonvllm/vllm/vllm/utils.py", line 2428, in run_method
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] Traceback (most recent call last):
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     return func(*args, **kwargs)
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/moonvllm/vllm/vllm/executor/multiproc_worker_utils.py", line 232, in _run_worker_process
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/moonvllm/vllm/vllm/worker/worker_base.py", line 91, in start_worker_execution_loop
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     output = run_method(worker, method, args, kwargs)
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     output = self.execute_model(execute_model_req=None)
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/moonvllm/vllm/vllm/utils.py", line 2428, in run_method
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/moonvllm/vllm/vllm/worker/worker_base.py", line 420, in execute_model
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     return func(*args, **kwargs)
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     output = self.model_runner.execute_model(
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/moonvllm/vllm/vllm/worker/worker_base.py", line 91, in start_worker_execution_loop
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/miniconda3/envs/moonvllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     output = self.execute_model(execute_model_req=None)
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     return func(*args, **kwargs)
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/moonvllm/vllm/vllm/worker/worker_base.py", line 420, in execute_model
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/moonvllm/vllm/vllm/worker/model_runner.py", line 1753, in execute_model
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     output = self.model_runner.execute_model(
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     get_kv_transfer_group().recv_kv_caches_and_hidden_states(
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/moonvllm/vllm/vllm/distributed/kv_transfer/kv_connector/mooncake_store_connector.py", line 141, in recv_kv_caches_and_hidden_states
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/miniconda3/envs/moonvllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     assert start_pos == num_prefill_tokens
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     return func(*args, **kwargs)
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] AssertionError
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/moonvllm/vllm/vllm/worker/model_runner.py", line 1753, in execute_model
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     get_kv_transfer_group().recv_kv_caches_and_hidden_states(
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/moonvllm/vllm/vllm/distributed/kv_transfer/kv_connector/mooncake_store_connector.py", line 141, in recv_kv_caches_and_hidden_states
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     assert start_pos == num_prefill_tokens
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] AssertionError
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] Exception in worker VllmWorkerProcess while processing method start_worker_execution_loop.
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] Traceback (most recent call last):
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/moonvllm/vllm/vllm/executor/multiproc_worker_utils.py", line 232, in _run_worker_process
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     output = run_method(worker, method, args, kwargs)
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/moonvllm/vllm/vllm/utils.py", line 2428, in run_method
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     return func(*args, **kwargs)
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/moonvllm/vllm/vllm/worker/worker_base.py", line 91, in start_worker_execution_loop
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     output = self.execute_model(execute_model_req=None)
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/moonvllm/vllm/vllm/worker/worker_base.py", line 420, in execute_model
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     output = self.model_runner.execute_model(
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/miniconda3/envs/moonvllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     return func(*args, **kwargs)
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]            ^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/moonvllm/vllm/vllm/worker/model_runner.py", line 1753, in execute_model
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     get_kv_transfer_group().recv_kv_caches_and_hidden_states(
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]   File "/root/moonvllm/vllm/vllm/distributed/kv_transfer/kv_connector/mooncake_store_connector.py", line 141, in recv_kv_caches_and_hidden_states
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]     assert start_pos == num_prefill_tokens
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] AssertionError
I0423 16:34:41.286177 455907 store_py.cpp:65] Received signal 15, cleaning up resources
I0423 16:34:41.286192 455907 store_py.cpp:58] Cleaning up DistributedObjectStore instance
I0423 16:34:41.286182 455906 store_py.cpp:65] Received signal 15, cleaning up resources
I0423 16:34:41.286183 455908 store_py.cpp:65] Received signal 15, cleaning up resources
I0423 16:34:41.286198 455906 store_py.cpp:58] Cleaning up DistributedObjectStore instance
I0423 16:34:41.286206 455908 store_py.cpp:58] Cleaning up DistributedObjectStore instance
I0423 16:34:41.289728 455906 allocator.cpp:160] simple_allocator_destroyed status=success
I0423 16:34:41.290158 455907 allocator.cpp:160] simple_allocator_destroyed status=success
I0423 16:34:41.290949 455908 allocator.cpp:160] simple_allocator_destroyed status=success
INFO:     172.30.11.146:38150 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error
INFO:     Shutting down
ERROR 04-23 16:34:42 [multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 455908 died, exit code: -15
INFO 04-23 16:34:42 [multiproc_worker_utils.py:124] Killing local vLLM worker processes
INFO:     Waiting for application shutdown.
INFO:     Application shutdown complete.
INFO:     Finished server process [455593]
/root/miniconda3/envs/moonvllm/lib/python3.12/multiprocessing/resource_tracker.py:255: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '
/root/miniconda3/envs/moonvllm/lib/python3.12/multiprocessing/resource_tracker.py:255: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

On Prefill Node

W0423 16:34:48.518262 217300 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3)
E0423 16:34:48.528317 217300 client.cpp:347] Transfer failed for task0
W0423 16:34:48.528321 217300 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3)
E0423 16:34:48.538375 217300 client.cpp:363] transfer_incomplete max_attempts=3
E0423 16:34:48.565101 217200 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused
E0423 16:34:48.565136 217200 client.cpp:347] Transfer failed for task0
W0423 16:34:48.565140 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3)
E0423 16:34:48.575198 217200 client.cpp:347] Transfer failed for task0
W0423 16:34:48.575201 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3)
E0423 16:34:48.585256 217200 client.cpp:347] Transfer failed for task0
W0423 16:34:48.585261 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3)
E0423 16:34:48.595315 217200 client.cpp:363] transfer_incomplete max_attempts=3
E0423 16:34:48.651890 217200 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused
E0423 16:34:48.651927 217200 client.cpp:347] Transfer failed for task0
W0423 16:34:48.651930 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3)
E0423 16:34:48.655128 217300 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused
E0423 16:34:48.655194 217300 client.cpp:347] Transfer failed for task0
W0423 16:34:48.655198 217300 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3)
E0423 16:34:48.661988 217200 client.cpp:347] Transfer failed for task0
W0423 16:34:48.661993 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3)
E0423 16:34:48.665254 217300 client.cpp:347] Transfer failed for task0
W0423 16:34:48.665261 217300 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3)
E0423 16:34:48.665962 217301 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused
E0423 16:34:48.666036 217301 client.cpp:347] Transfer failed for task0
W0423 16:34:48.666039 217301 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3)
E0423 16:34:48.672046 217200 client.cpp:347] Transfer failed for task0
W0423 16:34:48.672055 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3)
E0423 16:34:48.675320 217300 client.cpp:347] Transfer failed for task0
W0423 16:34:48.675333 217300 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3)
E0423 16:34:48.676095 217301 client.cpp:347] Transfer failed for task0
W0423 16:34:48.676100 217301 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3)
E0423 16:34:48.682111 217200 client.cpp:363] transfer_incomplete max_attempts=3
E0423 16:34:48.685393 217300 client.cpp:363] transfer_incomplete max_attempts=3
E0423 16:34:48.686153 217301 client.cpp:347] Transfer failed for task0
W0423 16:34:48.686158 217301 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3)
E0423 16:34:48.686291 217200 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused
E0423 16:34:48.686309 217200 client.cpp:347] Transfer failed for task0
W0423 16:34:48.686312 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3)
E0423 16:34:48.696215 217301 client.cpp:363] transfer_incomplete max_attempts=3
E0423 16:34:48.696372 217200 client.cpp:347] Transfer failed for task0
W0423 16:34:48.696377 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3)
E0423 16:34:48.703482 217301 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused
E0423 16:34:48.703500 217301 client.cpp:347] Transfer failed for task0
W0423 16:34:48.703503 217301 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3)
E0423 16:34:48.706430 217200 client.cpp:347] Transfer failed for task0
W0423 16:34:48.706434 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3)
E0423 16:34:48.713559 217301 client.cpp:347] Transfer failed for task0
W0423 16:34:48.713562 217301 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3)
E0423 16:34:48.716490 217200 client.cpp:363] transfer_incomplete max_attempts=3
E0423 16:34:48.723619 217301 client.cpp:347] Transfer failed for task0
W0423 16:34:48.723623 217301 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3)
INFO:     127.0.0.1:50274 - "POST /v1/completions HTTP/1.1" 200 OK
E0423 16:34:48.733676 217301 client.cpp:363] transfer_incomplete max_attempts=3
E0423 16:34:48.744788 217300 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused
E0423 16:34:48.744814 217300 client.cpp:347] Transfer failed for task1
W0423 16:34:48.744818 217300 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3)
E0423 16:34:48.754878 217300 client.cpp:347] Transfer failed for task1
W0423 16:34:48.754887 217300 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3)
E0423 16:34:48.764941 217300 client.cpp:347] Transfer failed for task1
W0423 16:34:48.764945 217300 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3)
E0423 16:34:48.774998 217300 client.cpp:363] transfer_incomplete max_attempts=3
E0423 16:34:48.911303 217200 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused
E0423 16:34:48.911484 217200 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused
E0423 16:34:48.911489 217200 client.cpp:347] Transfer failed for task0
E0423 16:34:48.911491 217200 client.cpp:347] Transfer failed for task1
W0423 16:34:48.911494 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3)
E0423 16:34:48.921550 217200 client.cpp:347] Transfer failed for task0
E0423 16:34:48.921553 217200 client.cpp:347] Transfer failed for task1
W0423 16:34:48.921555 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3)
E0423 16:34:48.929325 217299 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused
E0423 16:34:48.929347 217299 client.cpp:347] Transfer failed for task1
W0423 16:34:48.929348 217299 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3)
E0423 16:34:48.930759 217301 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused
E0423 16:34:48.930778 217301 client.cpp:347] Transfer failed for task1
W0423 16:34:48.930781 217301 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3)
E0423 16:34:48.931612 217200 client.cpp:347] Transfer failed for task0
E0423 16:34:48.931615 217200 client.cpp:347] Transfer failed for task1
W0423 16:34:48.931617 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3)
E0423 16:34:48.939406 217299 client.cpp:347] Transfer failed for task1
W0423 16:34:48.939412 217299 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3)
E0423 16:34:48.940837 217301 client.cpp:347] Transfer failed for task1
W0423 16:34:48.940841 217301 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3)
E0423 16:34:48.941674 217200 client.cpp:363] transfer_incomplete max_attempts=3
E0423 16:34:48.949468 217299 client.cpp:347] Transfer failed for task1
W0423 16:34:48.949472 217299 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3)
E0423 16:34:48.950897 217301 client.cpp:347] Transfer failed for task1
W0423 16:34:48.950901 217301 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3)
E0423 16:34:48.959529 217299 client.cpp:363] transfer_incomplete max_attempts=3
E0423 16:34:48.960958 217301 client.cpp:363] transfer_incomplete max_attempts=3
INFO:     127.0.0.1:50282 - "POST /v1/completions HTTP/1.1" 200 OK
E0423 16:34:48.964480 217299 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused
E0423 16:34:48.964494 217299 client.cpp:347] Transfer failed for task0
W0423 16:34:48.964497 217299 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3)
INFO:     127.0.0.1:50310 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:50300 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:50326 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:50312 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:50292 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:50336 - "POST /v1/completions HTTP/1.1" 200 OK
INFO:     127.0.0.1:50342 - "POST /v1/completions HTTP/1.1" 200 OK
E0423 16:34:48.974553 217299 client.cpp:347] Transfer failed for task0
W0423 16:34:48.974556 217299 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3)
INFO:     127.0.0.1:50334 - "POST /v1/completions HTTP/1.1" 200 OK
E0423 16:34:48.984611 217299 client.cpp:347] Transfer failed for task0
W0423 16:34:48.984614 217299 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3)
E0423 16:34:48.994669 217299 client.cpp:363] transfer_incomplete max_attempts=3
^CI0423 16:34:49.154258 217299 store_py.cpp:65] Received signal 2, cleaning up resources
I0423 16:34:49.154256 217301 store_py.cpp:65] Received signal 2, cleaning up resources
I0423 16:34:49.154258 217200 store_py.cpp:65] Received signal 2, cleaning up resources
I0423 16:34:49.154279 217301 store_py.cpp:58] Cleaning up DistributedObjectStore instance
I0423 16:34:49.154258 217300 store_py.cpp:65] Received signal 2, cleaning up resources
I0423 16:34:49.154273 217299 store_py.cpp:58] Cleaning up DistributedObjectStore instance
I0423 16:34:49.154284 217200 store_py.cpp:58] Cleaning up DistributedObjectStore instance
I0423 16:34:49.154292 217300 store_py.cpp:58] Cleaning up DistributedObjectStore instance
I0423 16:34:49.157610 217299 allocator.cpp:160] simple_allocator_destroyed status=success
I0423 16:34:49.158033 217200 allocator.cpp:160] simple_allocator_destroyed status=success
I0423 16:34:49.158039 217300 allocator.cpp:160] simple_allocator_destroyed status=success
I0423 16:34:49.159569 217301 allocator.cpp:160] simple_allocator_destroyed status=success
INFO 04-23 16:34:49 [launcher.py:79] Shutting down FastAPI HTTP server.

test shell

python benchmarks/benchmark_serving.py \
--backend vllm --model /root/32b/ --dataset-name random --random-input-len 256 \
--random-output-len 256 --num-prompts 1280 --port 8000 --host localhost --max-concurrency 10 --random-prefix-len 128

#prefill node
mooncake.json

{
    "local_hostname": "172.30.11.146",
    "metadata_server": "etcd://172.30.11.146:2379",
    "protocol": "tcp",
    "device_name": "",
    "global_segment_size": 42949672960,
    "master_server_address": "172.30.11.146:9992"
}

CUDA_VISIBLE_DEVICES=0,1,2,3 MOONCAKE_CONFIG_PATH=../mooncake.json VLLM_USE_V1=0 \ 
python3 -m  vllm.entrypoints.openai.api_server --model /root/32b/ \
--port 8100 --max-model-len 8192 --max-num-batched-tokens 8192 --max-seq-len-to-capture 8192 \
 -tp 4  --gpu-memory-utilization 0.8  --enable-prefix-caching --disable-log-requests --no-enable-chunked-prefill \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'

#decode node
mooncake.json

{
    "local_hostname": "172.30.11.145",
    "metadata_server": "etcd://172.30.11.146:2379",
    "protocol": "tcp",
    "device_name": "",
    "global_segment_size": 42949672960,
    "master_server_address": "172.30.11.146:9992"
}

CUDA_VISIBLE_DEVICES=0,1,2,3 MOONCAKE_CONFIG_PATH=../mooncake.json VLLM_USE_V1=0 \ 
python3 -m  vllm.entrypoints.openai.api_server --model /root/32b/ \
--port 8200 --max-model-len 8192 --max-num-batched-tokens 8192 --max-seq-len-to-capture 8192 \
 -tp 4  --gpu-memory-utilization 0.8  --enable-prefix-caching --disable-log-requests --no-enable-chunked-prefill \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}'
Originally created by @coding-rw on GitHub (Apr 23, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/292 1 prefill node(4*L20), 1 decoding node(4*L20), qwen2.5-32B-instruct # Bug on Decode Node ```shell INFO 04-23 16:34:22 [metrics.py:489] Avg prompt throughput: 41.4 tokens/s, Avg generation throughput: 0.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.1%, CPU KV cache usage: 0.0%. INFO 04-23 16:34:22 [metrics.py:505] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00% INFO: 172.30.11.146:54938 - "POST /v1/completions HTTP/1.1" 200 OK INFO 04-23 16:34:32 [metrics.py:489] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.9 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%. INFO 04-23 16:34:32 [metrics.py:505] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00% INFO: 172.30.11.146:47808 - "POST /v1/completions HTTP/1.1" 200 OK INFO 04-23 16:34:37 [metrics.py:489] Avg prompt throughput: 78.5 tokens/s, Avg generation throughput: 32.1 tokens/s, Running: 1 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.2%, CPU KV cache usage: 0.0%. INFO 04-23 16:34:37 [metrics.py:505] Prefix cache hit rate: GPU: 0.00%, CPU: 0.00% INFO: 172.30.11.146:38062 - "POST /v1/completions HTTP/1.1" 200 OK INFO: 172.30.11.146:38070 - "POST /v1/completions HTTP/1.1" 200 OK INFO: 172.30.11.146:38084 - "POST /v1/completions HTTP/1.1" 200 OK INFO: 172.30.11.146:38086 - "POST /v1/completions HTTP/1.1" 200 OK INFO: 172.30.11.146:38098 - "POST /v1/completions HTTP/1.1" 200 OK INFO: 172.30.11.146:38112 - "POST /v1/completions HTTP/1.1" 200 OK INFO: 172.30.11.146:38138 - "POST /v1/completions HTTP/1.1" 200 OK INFO: 172.30.11.146:38092 - "POST /v1/completions HTTP/1.1" 200 OK INFO: 172.30.11.146:38128 - "POST /v1/completions HTTP/1.1" 200 OK INFO: 172.30.11.146:38144 - "POST /v1/completions HTTP/1.1" 200 OK WARNING 04-23 16:34:41 [mooncake_store_connector.py:137] You should set --enable_chunked_prefill=False and --max_num_batched_tokens should be equal to max_seq_len_to_capture (VllmWorkerProcess pid=455906) WARNING 04-23 16:34:41 [mooncake_store_connector.py:137] You should set --enable_chunked_prefill=False and --max_num_batched_tokens should be equal to max_seq_len_to_capture (VllmWorkerProcess pid=455907) WARNING 04-23 16:34:41 [mooncake_store_connector.py:137] You should set --enable_chunked_prefill=False and --max_num_batched_tokens should be equal to max_seq_len_to_capture (VllmWorkerProcess pid=455908) WARNING 04-23 16:34:41 [mooncake_store_connector.py:137] You should set --enable_chunked_prefill=False and --max_num_batched_tokens should be equal to max_seq_len_to_capture ERROR 04-23 16:34:41 [engine.py:160] AssertionError() ERROR 04-23 16:34:41 [engine.py:160] Traceback (most recent call last): ERROR 04-23 16:34:41 [engine.py:160] File "/root/moonvllm/vllm/vllm/engine/multiprocessing/engine.py", line 158, in start ERROR 04-23 16:34:41 [engine.py:160] self.run_engine_loop() ERROR 04-23 16:34:41 [engine.py:160] File "/root/moonvllm/vllm/vllm/engine/multiprocessing/engine.py", line 221, in run_engine_loop ERROR 04-23 16:34:41 [engine.py:160] request_outputs = self.engine_step() ERROR 04-23 16:34:41 [engine.py:160] ^^^^^^^^^^^^^^^^^^ ERROR 04-23 16:34:41 [engine.py:160] File "/root/moonvllm/vllm/vllm/engine/multiprocessing/engine.py", line 247, in engine_step ERROR 04-23 16:34:41 [engine.py:160] raise e ERROR 04-23 16:34:41 [engine.py:160] File "/root/moonvllm/vllm/vllm/engine/multiprocessing/engine.py", line 230, in engine_step ERROR 04-23 16:34:41 [engine.py:160] return self.engine.step() ERROR 04-23 16:34:41 [engine.py:160] ^^^^^^^^^^^^^^^^^^ ERROR 04-23 16:34:41 [engine.py:160] File "/root/moonvllm/vllm/vllm/engine/llm_engine.py", line 1435, in step ERROR 04-23 16:34:41 [engine.py:160] outputs = self.model_executor.execute_model( ERROR 04-23 16:34:41 [engine.py:160] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 04-23 16:34:41 [engine.py:160] File "/root/moonvllm/vllm/vllm/executor/executor_base.py", line 299, in execute_model ERROR 04-23 16:34:41 [engine.py:160] driver_outputs = self._driver_execute_model(execute_model_req) ERROR 04-23 16:34:41 [engine.py:160] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 04-23 16:34:41 [engine.py:160] File "/root/moonvllm/vllm/vllm/executor/mp_distributed_executor.py", line 144, in _driver_execute_model ERROR 04-23 16:34:41 [engine.py:160] return self.driver_worker.execute_model(execute_model_req) ERROR 04-23 16:34:41 [engine.py:160] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 04-23 16:34:41 [engine.py:160] File "/root/moonvllm/vllm/vllm/worker/worker_base.py", line 420, in execute_model ERROR 04-23 16:34:41 [engine.py:160] output = self.model_runner.execute_model( ERROR 04-23 16:34:41 [engine.py:160] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 04-23 16:34:41 [engine.py:160] File "/root/miniconda3/envs/moonvllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context ERROR 04-23 16:34:41 [engine.py:160] return func(*args, **kwargs) ERROR 04-23 16:34:41 [engine.py:160] ^^^^^^^^^^^^^^^^^^^^^ ERROR 04-23 16:34:41 [engine.py:160] File "/root/moonvllm/vllm/vllm/worker/model_runner.py", line 1753, in execute_model ERROR 04-23 16:34:41 [engine.py:160] get_kv_transfer_group().recv_kv_caches_and_hidden_states( ERROR 04-23 16:34:41 [engine.py:160] File "/root/moonvllm/vllm/vllm/distributed/kv_transfer/kv_connector/mooncake_store_connector.py", line 141, in recv_kv_caches_and_hidden_states ERROR 04-23 16:34:41 [engine.py:160] assert start_pos == num_prefill_tokens ERROR 04-23 16:34:41 [engine.py:160] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 04-23 16:34:41 [engine.py:160] AssertionError (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] Exception in worker VllmWorkerProcess while processing method start_worker_execution_loop. (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] Traceback (most recent call last): (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/moonvllm/vllm/vllm/executor/multiproc_worker_utils.py", line 232, in _run_worker_process (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] output = run_method(worker, method, args, kwargs) (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] Exception in worker VllmWorkerProcess while processing method start_worker_execution_loop. (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/moonvllm/vllm/vllm/utils.py", line 2428, in run_method (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] Traceback (most recent call last): (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] return func(*args, **kwargs) (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/moonvllm/vllm/vllm/executor/multiproc_worker_utils.py", line 232, in _run_worker_process (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/moonvllm/vllm/vllm/worker/worker_base.py", line 91, in start_worker_execution_loop (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] output = run_method(worker, method, args, kwargs) (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] output = self.execute_model(execute_model_req=None) (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/moonvllm/vllm/vllm/utils.py", line 2428, in run_method (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/moonvllm/vllm/vllm/worker/worker_base.py", line 420, in execute_model (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] return func(*args, **kwargs) (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] output = self.model_runner.execute_model( (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/moonvllm/vllm/vllm/worker/worker_base.py", line 91, in start_worker_execution_loop (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/miniconda3/envs/moonvllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] output = self.execute_model(execute_model_req=None) (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] return func(*args, **kwargs) (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/moonvllm/vllm/vllm/worker/worker_base.py", line 420, in execute_model (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/moonvllm/vllm/vllm/worker/model_runner.py", line 1753, in execute_model (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] output = self.model_runner.execute_model( (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] get_kv_transfer_group().recv_kv_caches_and_hidden_states( (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/moonvllm/vllm/vllm/distributed/kv_transfer/kv_connector/mooncake_store_connector.py", line 141, in recv_kv_caches_and_hidden_states (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/miniconda3/envs/moonvllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] assert start_pos == num_prefill_tokens (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] return func(*args, **kwargs) (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=455907) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] AssertionError (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/moonvllm/vllm/vllm/worker/model_runner.py", line 1753, in execute_model (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] get_kv_transfer_group().recv_kv_caches_and_hidden_states( (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/moonvllm/vllm/vllm/distributed/kv_transfer/kv_connector/mooncake_store_connector.py", line 141, in recv_kv_caches_and_hidden_states (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] assert start_pos == num_prefill_tokens (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=455906) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] AssertionError (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] Exception in worker VllmWorkerProcess while processing method start_worker_execution_loop. (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] Traceback (most recent call last): (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/moonvllm/vllm/vllm/executor/multiproc_worker_utils.py", line 232, in _run_worker_process (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] output = run_method(worker, method, args, kwargs) (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/moonvllm/vllm/vllm/utils.py", line 2428, in run_method (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] return func(*args, **kwargs) (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/moonvllm/vllm/vllm/worker/worker_base.py", line 91, in start_worker_execution_loop (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] output = self.execute_model(execute_model_req=None) (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/moonvllm/vllm/vllm/worker/worker_base.py", line 420, in execute_model (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] output = self.model_runner.execute_model( (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/miniconda3/envs/moonvllm/lib/python3.12/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] return func(*args, **kwargs) (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/moonvllm/vllm/vllm/worker/model_runner.py", line 1753, in execute_model (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] get_kv_transfer_group().recv_kv_caches_and_hidden_states( (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] File "/root/moonvllm/vllm/vllm/distributed/kv_transfer/kv_connector/mooncake_store_connector.py", line 141, in recv_kv_caches_and_hidden_states (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] assert start_pos == num_prefill_tokens (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ (VllmWorkerProcess pid=455908) ERROR 04-23 16:34:41 [multiproc_worker_utils.py:238] AssertionError I0423 16:34:41.286177 455907 store_py.cpp:65] Received signal 15, cleaning up resources I0423 16:34:41.286192 455907 store_py.cpp:58] Cleaning up DistributedObjectStore instance I0423 16:34:41.286182 455906 store_py.cpp:65] Received signal 15, cleaning up resources I0423 16:34:41.286183 455908 store_py.cpp:65] Received signal 15, cleaning up resources I0423 16:34:41.286198 455906 store_py.cpp:58] Cleaning up DistributedObjectStore instance I0423 16:34:41.286206 455908 store_py.cpp:58] Cleaning up DistributedObjectStore instance I0423 16:34:41.289728 455906 allocator.cpp:160] simple_allocator_destroyed status=success I0423 16:34:41.290158 455907 allocator.cpp:160] simple_allocator_destroyed status=success I0423 16:34:41.290949 455908 allocator.cpp:160] simple_allocator_destroyed status=success INFO: 172.30.11.146:38150 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error INFO: Shutting down ERROR 04-23 16:34:42 [multiproc_worker_utils.py:120] Worker VllmWorkerProcess pid 455908 died, exit code: -15 INFO 04-23 16:34:42 [multiproc_worker_utils.py:124] Killing local vLLM worker processes INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [455593] /root/miniconda3/envs/moonvllm/lib/python3.12/multiprocessing/resource_tracker.py:255: UserWarning: resource_tracker: There appear to be 3 leaked semaphore objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' /root/miniconda3/envs/moonvllm/lib/python3.12/multiprocessing/resource_tracker.py:255: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory objects to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' ``` # On Prefill Node ```shell W0423 16:34:48.518262 217300 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3) E0423 16:34:48.528317 217300 client.cpp:347] Transfer failed for task0 W0423 16:34:48.528321 217300 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3) E0423 16:34:48.538375 217300 client.cpp:363] transfer_incomplete max_attempts=3 E0423 16:34:48.565101 217200 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused E0423 16:34:48.565136 217200 client.cpp:347] Transfer failed for task0 W0423 16:34:48.565140 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3) E0423 16:34:48.575198 217200 client.cpp:347] Transfer failed for task0 W0423 16:34:48.575201 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3) E0423 16:34:48.585256 217200 client.cpp:347] Transfer failed for task0 W0423 16:34:48.585261 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3) E0423 16:34:48.595315 217200 client.cpp:363] transfer_incomplete max_attempts=3 E0423 16:34:48.651890 217200 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused E0423 16:34:48.651927 217200 client.cpp:347] Transfer failed for task0 W0423 16:34:48.651930 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3) E0423 16:34:48.655128 217300 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused E0423 16:34:48.655194 217300 client.cpp:347] Transfer failed for task0 W0423 16:34:48.655198 217300 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3) E0423 16:34:48.661988 217200 client.cpp:347] Transfer failed for task0 W0423 16:34:48.661993 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3) E0423 16:34:48.665254 217300 client.cpp:347] Transfer failed for task0 W0423 16:34:48.665261 217300 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3) E0423 16:34:48.665962 217301 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused E0423 16:34:48.666036 217301 client.cpp:347] Transfer failed for task0 W0423 16:34:48.666039 217301 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3) E0423 16:34:48.672046 217200 client.cpp:347] Transfer failed for task0 W0423 16:34:48.672055 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3) E0423 16:34:48.675320 217300 client.cpp:347] Transfer failed for task0 W0423 16:34:48.675333 217300 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3) E0423 16:34:48.676095 217301 client.cpp:347] Transfer failed for task0 W0423 16:34:48.676100 217301 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3) E0423 16:34:48.682111 217200 client.cpp:363] transfer_incomplete max_attempts=3 E0423 16:34:48.685393 217300 client.cpp:363] transfer_incomplete max_attempts=3 E0423 16:34:48.686153 217301 client.cpp:347] Transfer failed for task0 W0423 16:34:48.686158 217301 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3) E0423 16:34:48.686291 217200 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused E0423 16:34:48.686309 217200 client.cpp:347] Transfer failed for task0 W0423 16:34:48.686312 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3) E0423 16:34:48.696215 217301 client.cpp:363] transfer_incomplete max_attempts=3 E0423 16:34:48.696372 217200 client.cpp:347] Transfer failed for task0 W0423 16:34:48.696377 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3) E0423 16:34:48.703482 217301 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused E0423 16:34:48.703500 217301 client.cpp:347] Transfer failed for task0 W0423 16:34:48.703503 217301 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3) E0423 16:34:48.706430 217200 client.cpp:347] Transfer failed for task0 W0423 16:34:48.706434 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3) E0423 16:34:48.713559 217301 client.cpp:347] Transfer failed for task0 W0423 16:34:48.713562 217301 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3) E0423 16:34:48.716490 217200 client.cpp:363] transfer_incomplete max_attempts=3 E0423 16:34:48.723619 217301 client.cpp:347] Transfer failed for task0 W0423 16:34:48.723623 217301 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3) INFO: 127.0.0.1:50274 - "POST /v1/completions HTTP/1.1" 200 OK E0423 16:34:48.733676 217301 client.cpp:363] transfer_incomplete max_attempts=3 E0423 16:34:48.744788 217300 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused E0423 16:34:48.744814 217300 client.cpp:347] Transfer failed for task1 W0423 16:34:48.744818 217300 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3) E0423 16:34:48.754878 217300 client.cpp:347] Transfer failed for task1 W0423 16:34:48.754887 217300 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3) E0423 16:34:48.764941 217300 client.cpp:347] Transfer failed for task1 W0423 16:34:48.764945 217300 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3) E0423 16:34:48.774998 217300 client.cpp:363] transfer_incomplete max_attempts=3 E0423 16:34:48.911303 217200 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused E0423 16:34:48.911484 217200 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused E0423 16:34:48.911489 217200 client.cpp:347] Transfer failed for task0 E0423 16:34:48.911491 217200 client.cpp:347] Transfer failed for task1 W0423 16:34:48.911494 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3) E0423 16:34:48.921550 217200 client.cpp:347] Transfer failed for task0 E0423 16:34:48.921553 217200 client.cpp:347] Transfer failed for task1 W0423 16:34:48.921555 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3) E0423 16:34:48.929325 217299 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused E0423 16:34:48.929347 217299 client.cpp:347] Transfer failed for task1 W0423 16:34:48.929348 217299 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3) E0423 16:34:48.930759 217301 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused E0423 16:34:48.930778 217301 client.cpp:347] Transfer failed for task1 W0423 16:34:48.930781 217301 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3) E0423 16:34:48.931612 217200 client.cpp:347] Transfer failed for task0 E0423 16:34:48.931615 217200 client.cpp:347] Transfer failed for task1 W0423 16:34:48.931617 217200 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3) E0423 16:34:48.939406 217299 client.cpp:347] Transfer failed for task1 W0423 16:34:48.939412 217299 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3) E0423 16:34:48.940837 217301 client.cpp:347] Transfer failed for task1 W0423 16:34:48.940841 217301 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3) E0423 16:34:48.941674 217200 client.cpp:363] transfer_incomplete max_attempts=3 E0423 16:34:48.949468 217299 client.cpp:347] Transfer failed for task1 W0423 16:34:48.949472 217299 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3) E0423 16:34:48.950897 217301 client.cpp:347] Transfer failed for task1 W0423 16:34:48.950901 217301 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3) E0423 16:34:48.959529 217299 client.cpp:363] transfer_incomplete max_attempts=3 E0423 16:34:48.960958 217301 client.cpp:363] transfer_incomplete max_attempts=3 INFO: 127.0.0.1:50282 - "POST /v1/completions HTTP/1.1" 200 OK E0423 16:34:48.964480 217299 tcp_transport.cpp:393] TcpTransport: ASIO exception: connect: Connection refused E0423 16:34:48.964494 217299 client.cpp:347] Transfer failed for task0 W0423 16:34:48.964497 217299 client.cpp:353] Transfer incomplete, retrying... (attempt 1/3) INFO: 127.0.0.1:50310 - "POST /v1/completions HTTP/1.1" 200 OK INFO: 127.0.0.1:50300 - "POST /v1/completions HTTP/1.1" 200 OK INFO: 127.0.0.1:50326 - "POST /v1/completions HTTP/1.1" 200 OK INFO: 127.0.0.1:50312 - "POST /v1/completions HTTP/1.1" 200 OK INFO: 127.0.0.1:50292 - "POST /v1/completions HTTP/1.1" 200 OK INFO: 127.0.0.1:50336 - "POST /v1/completions HTTP/1.1" 200 OK INFO: 127.0.0.1:50342 - "POST /v1/completions HTTP/1.1" 200 OK E0423 16:34:48.974553 217299 client.cpp:347] Transfer failed for task0 W0423 16:34:48.974556 217299 client.cpp:353] Transfer incomplete, retrying... (attempt 2/3) INFO: 127.0.0.1:50334 - "POST /v1/completions HTTP/1.1" 200 OK E0423 16:34:48.984611 217299 client.cpp:347] Transfer failed for task0 W0423 16:34:48.984614 217299 client.cpp:353] Transfer incomplete, retrying... (attempt 3/3) E0423 16:34:48.994669 217299 client.cpp:363] transfer_incomplete max_attempts=3 ^CI0423 16:34:49.154258 217299 store_py.cpp:65] Received signal 2, cleaning up resources I0423 16:34:49.154256 217301 store_py.cpp:65] Received signal 2, cleaning up resources I0423 16:34:49.154258 217200 store_py.cpp:65] Received signal 2, cleaning up resources I0423 16:34:49.154279 217301 store_py.cpp:58] Cleaning up DistributedObjectStore instance I0423 16:34:49.154258 217300 store_py.cpp:65] Received signal 2, cleaning up resources I0423 16:34:49.154273 217299 store_py.cpp:58] Cleaning up DistributedObjectStore instance I0423 16:34:49.154284 217200 store_py.cpp:58] Cleaning up DistributedObjectStore instance I0423 16:34:49.154292 217300 store_py.cpp:58] Cleaning up DistributedObjectStore instance I0423 16:34:49.157610 217299 allocator.cpp:160] simple_allocator_destroyed status=success I0423 16:34:49.158033 217200 allocator.cpp:160] simple_allocator_destroyed status=success I0423 16:34:49.158039 217300 allocator.cpp:160] simple_allocator_destroyed status=success I0423 16:34:49.159569 217301 allocator.cpp:160] simple_allocator_destroyed status=success INFO 04-23 16:34:49 [launcher.py:79] Shutting down FastAPI HTTP server. ``` # test shell ```shell python benchmarks/benchmark_serving.py \ --backend vllm --model /root/32b/ --dataset-name random --random-input-len 256 \ --random-output-len 256 --num-prompts 1280 --port 8000 --host localhost --max-concurrency 10 --random-prefix-len 128 ``` #prefill node mooncake.json ```shell { "local_hostname": "172.30.11.146", "metadata_server": "etcd://172.30.11.146:2379", "protocol": "tcp", "device_name": "", "global_segment_size": 42949672960, "master_server_address": "172.30.11.146:9992" } ``` ``` shell CUDA_VISIBLE_DEVICES=0,1,2,3 MOONCAKE_CONFIG_PATH=../mooncake.json VLLM_USE_V1=0 \ python3 -m vllm.entrypoints.openai.api_server --model /root/32b/ \ --port 8100 --max-model-len 8192 --max-num-batched-tokens 8192 --max-seq-len-to-capture 8192 \ -tp 4 --gpu-memory-utilization 0.8 --enable-prefix-caching --disable-log-requests --no-enable-chunked-prefill \ --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}' ``` #decode node mooncake.json ```shell { "local_hostname": "172.30.11.145", "metadata_server": "etcd://172.30.11.146:2379", "protocol": "tcp", "device_name": "", "global_segment_size": 42949672960, "master_server_address": "172.30.11.146:9992" } ``` ``` shell CUDA_VISIBLE_DEVICES=0,1,2,3 MOONCAKE_CONFIG_PATH=../mooncake.json VLLM_USE_V1=0 \ python3 -m vllm.entrypoints.openai.api_server --model /root/32b/ \ --port 8200 --max-model-len 8192 --max-num-batched-tokens 8192 --max-seq-len-to-capture 8192 \ -tp 4 --gpu-memory-utilization 0.8 --enable-prefix-caching --disable-log-requests --no-enable-chunked-prefill \ --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer"}' ```
bsakaguchi 2026-10-29 21:09:37 +00:00
bsakaguchi commented 2026-10-29 21:09:38 +00:00
Author
Owner
Copy link

@ShangmingCai commented on GitHub (Apr 23, 2025):

Just follow the guide from the log:

You should set --enable_chunked_prefill=False and --max_num_batched_tokens should be equal to max_seq_len_to_capture

Chunked prefill is not fully compatible with disaggregated serving yet.

<!-- gh-comment-id:2824104709 --> @ShangmingCai commented on GitHub (Apr 23, 2025): Just follow the guide from the log: ``` You should set --enable_chunked_prefill=False and --max_num_batched_tokens should be equal to max_seq_len_to_capture ``` Chunked prefill is not fully compatible with disaggregated serving yet.
bsakaguchi commented 2026-10-29 21:09:38 +00:00
Author
Owner
Copy link

@coding-rw commented on GitHub (Apr 23, 2025):

Just follow the guide from the log:

You should set --enable_chunked_prefill=False and --max_num_batched_tokens should be equal to max_seq_len_to_capture

Chunked prefill is not fully compatible with disaggregated serving yet.

I have used --no-enable-chunked-prefill --max-num-batched-tokens 8192 --max-seq-len-to-capture 8192, is this different from what you mentioned? --enable_chunked_prefill=False is not a legal parameter.

CUDA_VISIBLE_DEVICES=0,1,2,3 MOONCAKE_CONFIG_PATH=../mooncake.json VLLM_USE_V1=0 \ 
python3 -m  vllm.entrypoints.openai.api_server --model /root/32b/ \
--port 8100 --max-model-len 8192 --max-num-batched-tokens 8192 --max-seq-len-to-capture 8192 \
 -tp 4  --gpu-memory-utilization 0.8  --enable-prefix-caching --disable-log-requests --no-enable-chunked-prefill \
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'
<!-- gh-comment-id:2824167696 --> @coding-rw commented on GitHub (Apr 23, 2025): > Just follow the guide from the log: > > ``` > You should set --enable_chunked_prefill=False and --max_num_batched_tokens should be equal to max_seq_len_to_capture > ``` > > Chunked prefill is not fully compatible with disaggregated serving yet. I have used --no-enable-chunked-prefill --max-num-batched-tokens 8192 --max-seq-len-to-capture 8192, is this different from what you mentioned? --enable_chunked_prefill=False is not a legal parameter. ```shell CUDA_VISIBLE_DEVICES=0,1,2,3 MOONCAKE_CONFIG_PATH=../mooncake.json VLLM_USE_V1=0 \ python3 -m vllm.entrypoints.openai.api_server --model /root/32b/ \ --port 8100 --max-model-len 8192 --max-num-batched-tokens 8192 --max-seq-len-to-capture 8192 \ -tp 4 --gpu-memory-utilization 0.8 --enable-prefix-caching --disable-log-requests --no-enable-chunked-prefill \ --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}' ```
bsakaguchi commented 2026-10-29 21:09:38 +00:00
Author
Owner
Copy link

@ShangmingCai commented on GitHub (Apr 23, 2025):

@coding-rw Try --enable-chunked-prefill=False first. If it doesn't work, then just remove --enable-chunked-prefill.

<!-- gh-comment-id:2824260781 --> @ShangmingCai commented on GitHub (Apr 23, 2025): @coding-rw Try `--enable-chunked-prefill=False` first. If it doesn't work, then just remove `--enable-chunked-prefill`.
bsakaguchi commented 2026-10-29 21:09:38 +00:00
Author
Owner
Copy link

@coding-rw commented on GitHub (Apr 24, 2025):

@coding-rw Try --enable-chunked-prefill=False first. If it doesn't work, then just remove --enable-chunked-prefill. @ShangmingCai

When I use

CUDA_VISIBLE_DEVICES=0,1,2,3 MOONCAKE_CONFIG_PATH=../mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server --model /root/32b/ --port 8100 --enable-chunked-prefill=False --max-model-len 8192 -tp 4 --gpu-memory-utilization 0.8  --disable-log-requests --enable-prefix-caching --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}'

I got

api_server.py: error: argument --enable-chunked-prefill/--no-enable-chunked-prefill: ignored explicit argument 'False'

The reason is the latest version of vllm (https://docs.vllm.ai/en/latest/serving/engine_args.html), which uses --enable-chunked-prefill/--no-enable-chunked-prefill instead of --enable-chunked-prefill=False.

So, I reverted to using vllm v0.8.4, which supports --enable-chunked-prefill=False, but the same error still occurred: assert start_pos == num_prefill_tokens.

After testing, I found that this issue is caused by --enable-prefix-caching. When not using --enable-prefix-caching, the error assert start_pos == num_prefill_tokens does not occur. I haven't analyzed the root cause yet. Has anyone encountered a similar issue? Any ideas or solutions?

<!-- gh-comment-id:2826450063 --> @coding-rw commented on GitHub (Apr 24, 2025): > [@coding-rw](https://github.com/coding-rw) Try `--enable-chunked-prefill=False` first. If it doesn't work, then just remove `--enable-chunked-prefill`. @ShangmingCai When I use ```shell CUDA_VISIBLE_DEVICES=0,1,2,3 MOONCAKE_CONFIG_PATH=../mooncake.json VLLM_USE_V1=0 python3 -m vllm.entrypoints.openai.api_server --model /root/32b/ --port 8100 --enable-chunked-prefill=False --max-model-len 8192 -tp 4 --gpu-memory-utilization 0.8 --disable-log-requests --enable-prefix-caching --kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_producer"}' ``` I got ```shell api_server.py: error: argument --enable-chunked-prefill/--no-enable-chunked-prefill: ignored explicit argument 'False' ``` The reason is the latest version of vllm (https://docs.vllm.ai/en/latest/serving/engine_args.html), which uses `--enable-chunked-prefill/--no-enable-chunked-prefill` instead of `--enable-chunked-prefill=False`. So, I reverted to using vllm v0.8.4, which supports `--enable-chunked-prefill=False`, but the same error still occurred: `assert start_pos == num_prefill_tokens`. **After testing, I found that this issue is caused by `--enable-prefix-caching`. When not using `--enable-prefix-caching`, the error `assert start_pos == num_prefill_tokens` does not occur. I haven't analyzed the root cause yet. Has anyone encountered a similar issue? Any ideas or solutions?**
bsakaguchi commented 2026-10-29 21:09:38 +00:00
Author
Owner
Copy link

@ShangmingCai commented on GitHub (Apr 24, 2025):

@coding-rw Prefix caching includes immutable blocks and mutable blocks, maybe there are some potential bugs in this fix: https://github.com/vllm-project/vllm/pull/13987, you can reach out to the author of this PR.

<!-- gh-comment-id:2826644416 --> @ShangmingCai commented on GitHub (Apr 24, 2025): @coding-rw Prefix caching includes immutable blocks and mutable blocks, maybe there are some potential bugs in this fix: https://github.com/vllm-project/vllm/pull/13987, you can reach out to the author of this PR.
bsakaguchi commented 2026-10-29 21:09:39 +00:00
Author
Owner
Copy link

@coding-rw commented on GitHub (Apr 24, 2025):

@coding-rw Prefix caching includes immutable blocks and mutable blocks, maybe there are some potential bugs in this fix: vllm-project/vllm#13987, you can reach out to the author of this PR.

Thanks a lot!

<!-- gh-comment-id:2826658622 --> @coding-rw commented on GitHub (Apr 24, 2025): > [@coding-rw](https://github.com/coding-rw) Prefix caching includes immutable blocks and mutable blocks, maybe there are some potential bugs in this fix: [vllm-project/vllm#13987](https://github.com/vllm-project/vllm/pull/13987), you can reach out to the author of this PR. Thanks a lot!
bsakaguchi commented 2026-10-29 21:09:39 +00:00
Author
Owner
Copy link

@izhuhaoran commented on GitHub (Sep 3, 2025):

@coding-rw I have the same bug about this assert, do you have any fix about this? or just no-enable-prefix-caching?

<!-- gh-comment-id:3249049730 --> @izhuhaoran commented on GitHub (Sep 3, 2025): @coding-rw I have the same bug about this assert, do you have any fix about this? or just no-enable-prefix-caching?
bsakaguchi commented 2026-10-29 21:09:39 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Jul 14, 2026):

This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.

<!-- gh-comment-id:4965315523 --> @github-actions[bot] commented on GitHub (Jul 14, 2026): This issue has had no activity for 90 days and will be closed in 7 days if there is no further activity. Please comment or react if it should stay open.
bsakaguchi commented 2026-10-29 21:09:39 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Jul 21, 2026):

Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.

<!-- gh-comment-id:5030321670 --> @github-actions[bot] commented on GitHub (Jul 21, 2026): Closing due to 3 months of inactivity. If this is still relevant, please comment and we can reopen.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
te-coro
staryxchen-tent-capability-research
feature/musa-ep-jit
zxpdemonio/fix-tent-bootstrap-admission
add-store-rs-tone-tests
scatter-rdma-completion-pr2
P2P-Mooncake-Store
release/v0.3.14
feat/tent-rdma-adaptive-adapter
fix/ep-local-counter-gpu-scope
codex/kvcs-oplog-eviction
give-me-a-star
cruz/mooncake-kvcs-lowlevel-pr1-20261008
ylt-handshake-limits
ci/rocm-e2e-tent
sima/docs-pypi-downloads
fix-ci/hicache-mooncake-metadata-loopback
sima/cuda-driver-loading
sima/ci-performance-gate
fix/tent-rdma-unknown-topology-rail-pairing
ci/tent-e2e-dual-job
mc-patch-1
codex/p2p-merge-prerequisites
sima/store-hca-numa-placement
codex/p2p-merge-cli
codex/p2p-merge-ipv6
copilot/fix-ctests-job-failure
czh/pg-device-collectives-pr6
dependabot/cargo/mooncake-store-rs/pyo3-0.29.0
czh/pg-device-collectives-pr5
codex/segment-pool-master-cutover
sima/nightly-ci-recovery
ci/tent-params
stmatengss-patch-1
dependabot/cargo/mooncake-store-rs/rand-0.8.8
dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1
dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16
codex/migrate-store-rest-service
fix/tone-wheel-arch-selection
czh/pg-device-collectives-pr4
czh/pg-device-collectives-pr3
ketong/fix_moe_tc
copilot/fix-test-wheel-ubuntu-job
fix/ci-wheel-torch-pg-pin
codex/migrate-structured-object-storage
codex/migrate-allocator-python
codex/migrate-vllm-python
codex/migrate-shared-segment
ci/fix_sgl_eval_install
codex/segment-pool-nof-placement-cleanup
feat/pip-native-release
copilot/bugfix-ha-standby-promotion
copilot/mooncake-store-transport-settings
copilot/fix-tensor-support-issues
copilot/optimize-rl-direction
copilot/mooncake-storeagentic-workflow
copilot/fix-client-reconnect-timeout
copilot/mooncake-error-debugging
copilot/mooncake-ipv6-support
feat/nccl-host-python-api
dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1
copilot/migrate-and-move-operations
copilot/migrate-and-move-operations-again
refactor/centralize-env-vars
idhanani/mooncake-retain-groups
copilot/codeownerreviewer
copilot/give-feedback-on-issue-2123
copilot/create-issue-rfc
copilot/break-cuda-monopoly-logic
copilot/compare-cudamemcpy-cu12-cu13
copilot/report-ci-error-issue
copilot/research-commit-history-analysis
copilot/explain-mooncake-store-service
copilot/analyze-project-functionality
copilot/sub-pr-1680
copilot/review-pr-1680
copilot/sub-pr-1597
copilot/fix-github-actions-job-failure
copilot/move-test-sglang-integration-job
v0.3.14
v0.3.14-pre-4
v0.3.14-pre-3
v0.3.14-pre-2
v0.3.14-pre
v0.3.14-rc1
v0.3.13.post1
v0.3.13-rc2
v0.3.13-rc1
v0.3.13
v0.3.12.post1
v0.3.12
v0.3.11-rc1
v0.3.11.post1
v0.3.11
v0.3.10.post2
v0.3.10.post1
v0.3.10
v0.3.9
v0.3.8.post1
v0.3.8
v0.3.7.post2
v0.3.7.post1
v0.3.7
v0.3.6.post1
v0.3.6
v0.3.5
v0.3.4.post2
v0.3.4.post1
v0.3.4
v0.3.3.post2
v0.3.3.post1
v0.3.3
v0.3.2.post1
v0.3.2
v0.3.1
v0.3.0
v0.3.0-beta
v0.3.0-alpha
v0.2.0
Labels
Clear labels   
ascend

   
auto-closed

   
bug

   
bug

   
CI/Build

   
Common

   
documentation

   
enhancement

   
feature request

   
good first issue

   
help wanted

   
high priority

   
MetaX

   
nightly-failure

   
pull-request
Mirrored from GitHub Pull Request

  
PyTorch Backend

   
question

   
RFC

   
Roadmap

   
stale

   
Store

   
Tests

   
Transfer Engine
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#115
No description provided.