kvcache-ai/Mooncake
1
0
Fork 0
mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00
Code Issues 368 Projects Packages Wiki Activity

[GH-ISSUE #23] Error in Prefill: ValueError('Error in model execution: not enough values to unpack (expected 4, got 2)') #17

New issue
Closed
opened 2026-10-29 21:07:55 +00:00 by bsakaguchi · 7 comments
bsakaguchi commented 2026-10-29 21:07:55 +00:00
Owner
Copy link

Originally created by @LiuMicheal on GitHub (Dec 8, 2024).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/23

Hi, I have correctly installed Transfer Engine and Mooncake-vllm v0.2-Nightly. After I pre-started etcd (in prefill-vllm node), proxy_server.py (in prefill-vllm node), prefill-vllm, and decode-vllm, when I accessed using:

curl -s http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{
"model": "Qwen2___5-7B-Instruct-GPTQ-Int4",
"prompt": "San Francisco is a",
"max_tokens": 1000
}'

in prefill-vllm node. Prefill-vllm got the following error:

INFO: Started server process [148560]
INFO: Waiting for application startup.
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:8100 (Press CTRL+C to quit)
INFO: 127.0.0.1:36212 - "POST /v1/completions HTTP/1.1" 404 Not Found
DEBUG 12-08 22:00:58 client.py:187] Waiting for output from MQLLMEngine.
DEBUG 12-08 22:00:58 client.py:187] Waiting for output from MQLLMEngine.
DEBUG 12-08 22:00:58 client.py:187] Waiting for output from MQLLMEngine.
DEBUG 12-08 22:00:58 client.py:187] Waiting for output from MQLLMEngine.
DEBUG 12-08 22:00:58 client.py:187] Waiting for output from MQLLMEngine.
DEBUG 12-08 22:00:59 client.py:166] Heartbeat successful.
DEBUG 12-08 22:00:59 metrics.py:460] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
DEBUG 12-08 22:00:59 engine.py:190] Waiting for new requests in engine loop.
DEBUG 12-08 22:01:08 client.py:187] Waiting for output from MQLLMEngine.
DEBUG 12-08 22:01:08 client.py:187] Waiting for output from MQLLMEngine.
DEBUG 12-08 22:01:08 client.py:187] Waiting for output from MQLLMEngine.
DEBUG 12-08 22:01:08 client.py:187] Waiting for output from MQLLMEngine.
DEBUG 12-08 22:01:08 client.py:187] Waiting for output from MQLLMEngine.
DEBUG 12-08 22:01:09 client.py:166] Heartbeat successful.
DEBUG 12-08 22:01:09 client.py:166] Heartbeat successful.
DEBUG 12-08 22:01:09 metrics.py:460] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
DEBUG 12-08 22:01:09 engine.py:190] Waiting for new requests in engine loop.
INFO 12-08 22:01:13 logger.py:37] Received request cmpl-63eee0f8cfd54b5cab7b34584b1a5ec2-0: prompt: 'San Francisco is a', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: [23729, 12879, 374, 264], lora_request: None, prompt_adapter_request: None.
INFO 12-08 22:01:13 engine.py:267] Added request cmpl-63eee0f8cfd54b5cab7b34584b1a5ec2-0.
INFO 12-08 22:01:13 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20241208-220113.pkl...
INFO 12-08 22:01:13 model_runner_base.py:149] Completed writing input of failed execution to /tmp/err_execute_model_input_20241208-220113.pkl.
ERROR 12-08 22:01:13 engine.py:135] ValueError('Error in model execution (input dumped to /tmp/err_execute_model_input_20241208-220113.pkl): not enough values to unpack (expected 4, got 2)')
ERROR 12-08 22:01:13 engine.py:135] Traceback (most recent call last):
ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/worker/model_runner_base.py", line 116, in _wrapper
ERROR 12-08 22:01:13 engine.py:135] return func(*args, **kwargs)
ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/worker/model_runner.py", line 1696, in execute_model
ERROR 12-08 22:01:13 engine.py:135] get_kv_transfer_group().send_kv_caches_and_hidden_states(
ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/distributed/kv_transfer/kv_transfer_agent.py", line 60, in send_kv_caches_and_hidden_states
ERROR 12-08 22:01:13 engine.py:135] self.connector.send_kv_caches_and_hidden_states(
ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/distributed/kv_transfer/kv_connector/mooncake_connector.py", line 129, in send_kv_caches_and_hidden_states
ERROR 12-08 22:01:13 engine.py:135] _, _, num_heads, head_size = kv_cache[0].shape
ERROR 12-08 22:01:13 engine.py:135] ValueError: not enough values to unpack (expected 4, got 2)
ERROR 12-08 22:01:13 engine.py:135]
ERROR 12-08 22:01:13 engine.py:135] The above exception was the direct cause of the following exception:
ERROR 12-08 22:01:13 engine.py:135]
ERROR 12-08 22:01:13 engine.py:135] Traceback (most recent call last):
ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/engine/multiprocessing/engine.py", line 133, in start
ERROR 12-08 22:01:13 engine.py:135] self.run_engine_loop()
ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/engine/multiprocessing/engine.py", line 196, in run_engine_loop
ERROR 12-08 22:01:13 engine.py:135] request_outputs = self.engine_step()
ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/engine/multiprocessing/engine.py", line 214, in engine_step
ERROR 12-08 22:01:13 engine.py:135] raise e
ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/engine/multiprocessing/engine.py", line 205, in engine_step
ERROR 12-08 22:01:13 engine.py:135] return self.engine.step()
ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/engine/llm_engine.py", line 1448, in step
ERROR 12-08 22:01:13 engine.py:135] outputs = self.model_executor.execute_model(
ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/executor/gpu_executor.py", line 88, in execute_model
ERROR 12-08 22:01:13 engine.py:135] output = self.driver_worker.execute_model(execute_model_req)
ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/worker/worker_base.py", line 343, in execute_model
ERROR 12-08 22:01:13 engine.py:135] output = self.model_runner.execute_model(
ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/anaconda3/envs/vllm-test/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
ERROR 12-08 22:01:13 engine.py:135] return func(*args, **kwargs)
ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/worker/model_runner_base.py", line 152, in _wrapper
ERROR 12-08 22:01:13 engine.py:135] raise type(err)(
ERROR 12-08 22:01:13 engine.py:135] ValueError: Error in model execution (input dumped to /tmp/err_execute_model_input_20241208-220113.pkl): not enough values to unpack (expected 4, got 2)
DEBUG 12-08 22:01:13 engine.py:139] MQLLMEngine is shut down.
DEBUG 12-08 22:01:13 client.py:187] Waiting for output from MQLLMEngine.
CRITICAL 12-08 22:01:13 launcher.py:99] MQLLMEngine is already dead, terminating server process
INFO: 127.0.0.1:36768 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error
INFO: Shutting down
INFO: Waiting for application shutdown.
INFO: Application shutdown complete.
INFO: Finished server process [148560]
DEBUG 12-08 22:01:16 client.py:253] Shutting down MQLLMEngineClient output handler.
DEBUG 12-08 22:01:16 client.py:253] Shutting down MQLLMEngineClient output handler.
DEBUG 12-08 22:01:16 client.py:253] Shutting down MQLLMEngineClient output handler.
DEBUG 12-08 22:01:16 client.py:169] Shutting down MQLLMEngineClient check health loop.
DEBUG 12-08 22:01:16 client.py:253] Shutting down MQLLMEngineClient output handler.
(Then prefill-vllm terminated and returned to the shell. Meanwhile, decode-vllm can still run normally. However, it cannot be terminated by Ctrl+C.)

I launch them using these commands:
prefill-vllm:
VLLM_LOGGING_LEVEL=DEBUG CUDA_VISIBLE_DEVICES=2 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server --model /public/home/liumx/model_cache/hub/Qwen/Qwen2___5-7B-Instruct-GPTQ-Int4 --served-model-name Qwen2___5-7B-Instruct-GPTQ-Int4 --port 8100 --max-model-len 10000 --gpu-memory-utilization 0.95 --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2,"kv_buffer_size":5e9, "kv_ip": "10.10.10.12", "kv_port": 51000 }' --dtype=half

mooncake.json of prefill-vllm:
{
"prefill_url": "10.10.10.12:13003",
"decode_url": "10.10.10.13:13003",
"metadata_server": "10.10.10.12:2379",
"protocol": "rdma",
"device_name": "mlx5_0"
}

decode-vllm:
cd /home/liumx/Mooncake-vllm/vllm
VLLM_LOGGING_LEVEL=DEBUG CUDA_VISIBLE_DEVICES=2 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server --model /public/home/liumx/model_cache/hub/Qwen/Qwen2___5-7B-Instruct-GPTQ-Int4 --served-model-name Qwen2___5-7B-Instruct-GPTQ-Int4 --port 8200 --max-model-len 10000 --gpu-memory-utilization 0.95 --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2,"kv_buffer_size":5e9, "kv_ip": "10.10.10.12", "kv_port": 51000}' --dtype=half

mooncake.json of decode-vllm:
{
"prefill_url": "10.10.10.12:13003",
"decode_url": "10.10.10.13:13003",
"metadata_server": "10.10.10.12:2379",
"protocol": "rdma",
"device_name": "mlx5_0"
}

My environment is V100-32G, connected via 100Gbps IB and GDR. I have tested the correctness of GDR through Transfer Engine.

Thank you for your help!

Originally created by @LiuMicheal on GitHub (Dec 8, 2024). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/23 Hi, I have correctly installed Transfer Engine and Mooncake-vllm v0.2-Nightly. After I pre-started etcd (in prefill-vllm node), proxy_server.py (in prefill-vllm node), prefill-vllm, and decode-vllm, when I accessed using: curl -s http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{ "model": "Qwen2___5-7B-Instruct-GPTQ-Int4", "prompt": "San Francisco is a", "max_tokens": 1000 }' in prefill-vllm node. Prefill-vllm got the following error: INFO: Started server process [148560] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8100 (Press CTRL+C to quit) INFO: 127.0.0.1:36212 - "POST /v1/completions HTTP/1.1" 404 Not Found DEBUG 12-08 22:00:58 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:00:58 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:00:58 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:00:58 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:00:58 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:00:59 client.py:166] Heartbeat successful. DEBUG 12-08 22:00:59 metrics.py:460] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%. DEBUG 12-08 22:00:59 engine.py:190] Waiting for new requests in engine loop. DEBUG 12-08 22:01:08 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:01:08 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:01:08 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:01:08 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:01:08 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:01:09 client.py:166] Heartbeat successful. DEBUG 12-08 22:01:09 client.py:166] Heartbeat successful. DEBUG 12-08 22:01:09 metrics.py:460] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%. DEBUG 12-08 22:01:09 engine.py:190] Waiting for new requests in engine loop. INFO 12-08 22:01:13 logger.py:37] Received request cmpl-63eee0f8cfd54b5cab7b34584b1a5ec2-0: prompt: 'San Francisco is a', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: [23729, 12879, 374, 264], lora_request: None, prompt_adapter_request: None. INFO 12-08 22:01:13 engine.py:267] Added request cmpl-63eee0f8cfd54b5cab7b34584b1a5ec2-0. **INFO 12-08 22:01:13 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20241208-220113.pkl... INFO 12-08 22:01:13 model_runner_base.py:149] Completed writing input of failed execution to /tmp/err_execute_model_input_20241208-220113.pkl. ERROR 12-08 22:01:13 engine.py:135] ValueError('Error in model execution (input dumped to /tmp/err_execute_model_input_20241208-220113.pkl): not enough values to unpack (expected 4, got 2)')** ERROR 12-08 22:01:13 engine.py:135] Traceback (most recent call last): ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/worker/model_runner_base.py", line 116, in _wrapper ERROR 12-08 22:01:13 engine.py:135] return func(*args, **kwargs) ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/worker/model_runner.py", line 1696, in execute_model ERROR 12-08 22:01:13 engine.py:135] get_kv_transfer_group().send_kv_caches_and_hidden_states( ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/distributed/kv_transfer/kv_transfer_agent.py", line 60, in send_kv_caches_and_hidden_states ERROR 12-08 22:01:13 engine.py:135] self.connector.send_kv_caches_and_hidden_states( ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/distributed/kv_transfer/kv_connector/mooncake_connector.py", line 129, in send_kv_caches_and_hidden_states ERROR 12-08 22:01:13 engine.py:135] _, _, num_heads, head_size = kv_cache[0].shape **ERROR 12-08 22:01:13 engine.py:135] ValueError: not enough values to unpack (expected 4, got 2)** ERROR 12-08 22:01:13 engine.py:135] ERROR 12-08 22:01:13 engine.py:135] The above exception was the direct cause of the following exception: ERROR 12-08 22:01:13 engine.py:135] ERROR 12-08 22:01:13 engine.py:135] Traceback (most recent call last): ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/engine/multiprocessing/engine.py", line 133, in start ERROR 12-08 22:01:13 engine.py:135] self.run_engine_loop() ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/engine/multiprocessing/engine.py", line 196, in run_engine_loop ERROR 12-08 22:01:13 engine.py:135] request_outputs = self.engine_step() ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/engine/multiprocessing/engine.py", line 214, in engine_step ERROR 12-08 22:01:13 engine.py:135] raise e ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/engine/multiprocessing/engine.py", line 205, in engine_step ERROR 12-08 22:01:13 engine.py:135] return self.engine.step() ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/engine/llm_engine.py", line 1448, in step ERROR 12-08 22:01:13 engine.py:135] outputs = self.model_executor.execute_model( ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/executor/gpu_executor.py", line 88, in execute_model ERROR 12-08 22:01:13 engine.py:135] output = self.driver_worker.execute_model(execute_model_req) ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/worker/worker_base.py", line 343, in execute_model ERROR 12-08 22:01:13 engine.py:135] output = self.model_runner.execute_model( ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/anaconda3/envs/vllm-test/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context ERROR 12-08 22:01:13 engine.py:135] return func(*args, **kwargs) ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/worker/model_runner_base.py", line 152, in _wrapper ERROR 12-08 22:01:13 engine.py:135] raise type(err)( **ERROR 12-08 22:01:13 engine.py:135] ValueError: Error in model execution (input dumped to /tmp/err_execute_model_input_20241208-220113.pkl): not enough values to unpack (expected 4, got 2)** DEBUG 12-08 22:01:13 engine.py:139] MQLLMEngine is shut down. DEBUG 12-08 22:01:13 client.py:187] Waiting for output from MQLLMEngine. CRITICAL 12-08 22:01:13 launcher.py:99] MQLLMEngine is already dead, terminating server process INFO: 127.0.0.1:36768 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [148560] DEBUG 12-08 22:01:16 client.py:253] Shutting down MQLLMEngineClient output handler. DEBUG 12-08 22:01:16 client.py:253] Shutting down MQLLMEngineClient output handler. DEBUG 12-08 22:01:16 client.py:253] Shutting down MQLLMEngineClient output handler. DEBUG 12-08 22:01:16 client.py:169] Shutting down MQLLMEngineClient check health loop. DEBUG 12-08 22:01:16 client.py:253] Shutting down MQLLMEngineClient output handler. (Then prefill-vllm terminated and returned to the shell. Meanwhile, decode-vllm can still run normally. However, it cannot be terminated by Ctrl+C.) I launch them using these commands: prefill-vllm: VLLM_LOGGING_LEVEL=DEBUG CUDA_VISIBLE_DEVICES=2 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server --model /public/home/liumx/model_cache/hub/Qwen/Qwen2___5-7B-Instruct-GPTQ-Int4 --served-model-name Qwen2___5-7B-Instruct-GPTQ-Int4 --port 8100 --max-model-len 10000 --gpu-memory-utilization 0.95 --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2,"kv_buffer_size":5e9, "kv_ip": "10.10.10.12", "kv_port": 51000 }' --dtype=half mooncake.json of prefill-vllm: { "prefill_url": "10.10.10.12:13003", "decode_url": "10.10.10.13:13003", "metadata_server": "10.10.10.12:2379", "protocol": "rdma", "device_name": "mlx5_0" } decode-vllm: cd /home/liumx/Mooncake-vllm/vllm VLLM_LOGGING_LEVEL=DEBUG CUDA_VISIBLE_DEVICES=2 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server --model /public/home/liumx/model_cache/hub/Qwen/Qwen2___5-7B-Instruct-GPTQ-Int4 --served-model-name Qwen2___5-7B-Instruct-GPTQ-Int4 --port 8200 --max-model-len 10000 --gpu-memory-utilization 0.95 --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2,"kv_buffer_size":5e9, "kv_ip": "10.10.10.12", "kv_port": 51000}' --dtype=half mooncake.json of decode-vllm: { "prefill_url": "10.10.10.12:13003", "decode_url": "10.10.10.13:13003", "metadata_server": "10.10.10.12:2379", "protocol": "rdma", "device_name": "mlx5_0" } My environment is V100-32G, connected via 100Gbps IB and GDR. I have tested the correctness of GDR through Transfer Engine. Thank you for your help!
bsakaguchi commented 2026-10-29 21:07:56 +00:00
Author
Owner
Copy link

@ShangmingCai commented on GitHub (Dec 9, 2024):

Hi, I have correctly installed Transfer Engine and Mooncake-vllm v0.2-Nightly. After I pre-started etcd (in prefill-vllm node), proxy_server.py (in prefill-vllm node), prefill-vllm, and decode-vllm, when I accessed using:

curl -s http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{ "model": "Qwen2___5-7B-Instruct-GPTQ-Int4", "prompt": "San Francisco is a", "max_tokens": 1000 }'

in prefill-vllm node. Prefill-vllm got the following error:

INFO: Started server process [148560] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8100 (Press CTRL+C to quit) INFO: 127.0.0.1:36212 - "POST /v1/completions HTTP/1.1" 404 Not Found DEBUG 12-08 22:00:58 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:00:58 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:00:58 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:00:58 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:00:58 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:00:59 client.py:166] Heartbeat successful. DEBUG 12-08 22:00:59 metrics.py:460] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%. DEBUG 12-08 22:00:59 engine.py:190] Waiting for new requests in engine loop. DEBUG 12-08 22:01:08 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:01:08 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:01:08 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:01:08 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:01:08 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:01:09 client.py:166] Heartbeat successful. DEBUG 12-08 22:01:09 client.py:166] Heartbeat successful. DEBUG 12-08 22:01:09 metrics.py:460] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%. DEBUG 12-08 22:01:09 engine.py:190] Waiting for new requests in engine loop. INFO 12-08 22:01:13 logger.py:37] Received request cmpl-63eee0f8cfd54b5cab7b34584b1a5ec2-0: prompt: 'San Francisco is a', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: [23729, 12879, 374, 264], lora_request: None, prompt_adapter_request: None. INFO 12-08 22:01:13 engine.py:267] Added request cmpl-63eee0f8cfd54b5cab7b34584b1a5ec2-0. INFO 12-08 22:01:13 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20241208-220113.pkl... INFO 12-08 22:01:13 model_runner_base.py:149] Completed writing input of failed execution to /tmp/err_execute_model_input_20241208-220113.pkl. ERROR 12-08 22:01:13 engine.py:135] ValueError('Error in model execution (input dumped to /tmp/err_execute_model_input_20241208-220113.pkl): not enough values to unpack (expected 4, got 2)') ERROR 12-08 22:01:13 engine.py:135] Traceback (most recent call last): ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/worker/model_runner_base.py", line 116, in _wrapper ERROR 12-08 22:01:13 engine.py:135] return func(*args, **kwargs) ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/worker/model_runner.py", line 1696, in execute_model ERROR 12-08 22:01:13 engine.py:135] get_kv_transfer_group().send_kv_caches_and_hidden_states( ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/distributed/kv_transfer/kv_transfer_agent.py", line 60, in send_kv_caches_and_hidden_states ERROR 12-08 22:01:13 engine.py:135] self.connector.send_kv_caches_and_hidden_states( ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/distributed/kv_transfer/kv_connector/mooncake_connector.py", line 129, in send_kv_caches_and_hidden_states ERROR 12-08 22:01:13 engine.py:135] _, _, num_heads, head_size = kv_cache[0].shape ERROR 12-08 22:01:13 engine.py:135] ValueError: not enough values to unpack (expected 4, got 2) ERROR 12-08 22:01:13 engine.py:135] ERROR 12-08 22:01:13 engine.py:135] The above exception was the direct cause of the following exception: ERROR 12-08 22:01:13 engine.py:135] ERROR 12-08 22:01:13 engine.py:135] Traceback (most recent call last): ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/engine/multiprocessing/engine.py", line 133, in start ERROR 12-08 22:01:13 engine.py:135] self.run_engine_loop() ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/engine/multiprocessing/engine.py", line 196, in run_engine_loop ERROR 12-08 22:01:13 engine.py:135] request_outputs = self.engine_step() ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/engine/multiprocessing/engine.py", line 214, in engine_step ERROR 12-08 22:01:13 engine.py:135] raise e ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/engine/multiprocessing/engine.py", line 205, in engine_step ERROR 12-08 22:01:13 engine.py:135] return self.engine.step() ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/engine/llm_engine.py", line 1448, in step ERROR 12-08 22:01:13 engine.py:135] outputs = self.model_executor.execute_model( ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/executor/gpu_executor.py", line 88, in execute_model ERROR 12-08 22:01:13 engine.py:135] output = self.driver_worker.execute_model(execute_model_req) ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/worker/worker_base.py", line 343, in execute_model ERROR 12-08 22:01:13 engine.py:135] output = self.model_runner.execute_model( ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/anaconda3/envs/vllm-test/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context ERROR 12-08 22:01:13 engine.py:135] return func(*args, **kwargs) ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/worker/model_runner_base.py", line 152, in _wrapper ERROR 12-08 22:01:13 engine.py:135] raise type(err)( ERROR 12-08 22:01:13 engine.py:135] ValueError: Error in model execution (input dumped to /tmp/err_execute_model_input_20241208-220113.pkl): not enough values to unpack (expected 4, got 2) DEBUG 12-08 22:01:13 engine.py:139] MQLLMEngine is shut down. DEBUG 12-08 22:01:13 client.py:187] Waiting for output from MQLLMEngine. CRITICAL 12-08 22:01:13 launcher.py:99] MQLLMEngine is already dead, terminating server process INFO: 127.0.0.1:36768 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [148560] DEBUG 12-08 22:01:16 client.py:253] Shutting down MQLLMEngineClient output handler. DEBUG 12-08 22:01:16 client.py:253] Shutting down MQLLMEngineClient output handler. DEBUG 12-08 22:01:16 client.py:253] Shutting down MQLLMEngineClient output handler. DEBUG 12-08 22:01:16 client.py:169] Shutting down MQLLMEngineClient check health loop. DEBUG 12-08 22:01:16 client.py:253] Shutting down MQLLMEngineClient output handler. (Then prefill-vllm terminated and returned to the shell. Meanwhile, decode-vllm can still run normally. However, it cannot be terminated by Ctrl+C.)

I launch them using these commands: prefill-vllm: VLLM_LOGGING_LEVEL=DEBUG CUDA_VISIBLE_DEVICES=2 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server --model /public/home/liumx/model_cache/hub/Qwen/Qwen2___5-7B-Instruct-GPTQ-Int4 --served-model-name Qwen2___5-7B-Instruct-GPTQ-Int4 --port 8100 --max-model-len 10000 --gpu-memory-utilization 0.95 --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2,"kv_buffer_size":5e9, "kv_ip": "10.10.10.12", "kv_port": 51000 }' --dtype=half

mooncake.json of prefill-vllm: { "prefill_url": "10.10.10.12:13003", "decode_url": "10.10.10.13:13003", "metadata_server": "10.10.10.12:2379", "protocol": "rdma", "device_name": "mlx5_0" }

decode-vllm: cd /home/liumx/Mooncake-vllm/vllm VLLM_LOGGING_LEVEL=DEBUG CUDA_VISIBLE_DEVICES=2 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server --model /public/home/liumx/model_cache/hub/Qwen/Qwen2___5-7B-Instruct-GPTQ-Int4 --served-model-name Qwen2___5-7B-Instruct-GPTQ-Int4 --port 8200 --max-model-len 10000 --gpu-memory-utilization 0.95 --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2,"kv_buffer_size":5e9, "kv_ip": "10.10.10.12", "kv_port": 51000}' --dtype=half

mooncake.json of decode-vllm: { "prefill_url": "10.10.10.12:13003", "decode_url": "10.10.10.13:13003", "metadata_server": "10.10.10.12:2379", "protocol": "rdma", "device_name": "mlx5_0" }

My environment is V100-32G, connected via 100Gbps IB and GDR. I have tested the correctness of GDR through Transfer Engine.

Thank you for your help!

Hello, this is a known bug for PR 10502. To be consistent with the upstream PR, it is not fixed in this version but will be fixed in another PR. If you are in a hurry, I can instruct you how to perform a quick fix by modifying the code later today.

<!-- gh-comment-id:2526853094 --> @ShangmingCai commented on GitHub (Dec 9, 2024): > Hi, I have correctly installed Transfer Engine and Mooncake-vllm v0.2-Nightly. After I pre-started etcd (in prefill-vllm node), proxy_server.py (in prefill-vllm node), prefill-vllm, and decode-vllm, when I accessed using: > > curl -s http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{ "model": "Qwen2___5-7B-Instruct-GPTQ-Int4", "prompt": "San Francisco is a", "max_tokens": 1000 }' > > in prefill-vllm node. Prefill-vllm got the following error: > > INFO: Started server process [148560] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8100 (Press CTRL+C to quit) INFO: 127.0.0.1:36212 - "POST /v1/completions HTTP/1.1" 404 Not Found DEBUG 12-08 22:00:58 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:00:58 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:00:58 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:00:58 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:00:58 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:00:59 client.py:166] Heartbeat successful. DEBUG 12-08 22:00:59 metrics.py:460] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%. DEBUG 12-08 22:00:59 engine.py:190] Waiting for new requests in engine loop. DEBUG 12-08 22:01:08 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:01:08 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:01:08 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:01:08 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:01:08 client.py:187] Waiting for output from MQLLMEngine. DEBUG 12-08 22:01:09 client.py:166] Heartbeat successful. DEBUG 12-08 22:01:09 client.py:166] Heartbeat successful. DEBUG 12-08 22:01:09 metrics.py:460] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%. DEBUG 12-08 22:01:09 engine.py:190] Waiting for new requests in engine loop. INFO 12-08 22:01:13 logger.py:37] Received request cmpl-63eee0f8cfd54b5cab7b34584b1a5ec2-0: prompt: 'San Francisco is a', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: [23729, 12879, 374, 264], lora_request: None, prompt_adapter_request: None. INFO 12-08 22:01:13 engine.py:267] Added request cmpl-63eee0f8cfd54b5cab7b34584b1a5ec2-0. **INFO 12-08 22:01:13 model_runner_base.py:120] Writing input of failed execution to /tmp/err_execute_model_input_20241208-220113.pkl... INFO 12-08 22:01:13 model_runner_base.py:149] Completed writing input of failed execution to /tmp/err_execute_model_input_20241208-220113.pkl. ERROR 12-08 22:01:13 engine.py:135] ValueError('Error in model execution (input dumped to /tmp/err_execute_model_input_20241208-220113.pkl): not enough values to unpack (expected 4, got 2)')** ERROR 12-08 22:01:13 engine.py:135] Traceback (most recent call last): ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/worker/model_runner_base.py", line 116, in _wrapper ERROR 12-08 22:01:13 engine.py:135] return func(*args, **kwargs) ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/worker/model_runner.py", line 1696, in execute_model ERROR 12-08 22:01:13 engine.py:135] get_kv_transfer_group().send_kv_caches_and_hidden_states( ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/distributed/kv_transfer/kv_transfer_agent.py", line 60, in send_kv_caches_and_hidden_states ERROR 12-08 22:01:13 engine.py:135] self.connector.send_kv_caches_and_hidden_states( ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/distributed/kv_transfer/kv_connector/mooncake_connector.py", line 129, in send_kv_caches_and_hidden_states ERROR 12-08 22:01:13 engine.py:135] _, _, num_heads, head_size = kv_cache[0].shape **ERROR 12-08 22:01:13 engine.py:135] ValueError: not enough values to unpack (expected 4, got 2)** ERROR 12-08 22:01:13 engine.py:135] ERROR 12-08 22:01:13 engine.py:135] The above exception was the direct cause of the following exception: ERROR 12-08 22:01:13 engine.py:135] ERROR 12-08 22:01:13 engine.py:135] Traceback (most recent call last): ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/engine/multiprocessing/engine.py", line 133, in start ERROR 12-08 22:01:13 engine.py:135] self.run_engine_loop() ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/engine/multiprocessing/engine.py", line 196, in run_engine_loop ERROR 12-08 22:01:13 engine.py:135] request_outputs = self.engine_step() ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/engine/multiprocessing/engine.py", line 214, in engine_step ERROR 12-08 22:01:13 engine.py:135] raise e ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/engine/multiprocessing/engine.py", line 205, in engine_step ERROR 12-08 22:01:13 engine.py:135] return self.engine.step() ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/engine/llm_engine.py", line 1448, in step ERROR 12-08 22:01:13 engine.py:135] outputs = self.model_executor.execute_model( ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/executor/gpu_executor.py", line 88, in execute_model ERROR 12-08 22:01:13 engine.py:135] output = self.driver_worker.execute_model(execute_model_req) ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/worker/worker_base.py", line 343, in execute_model ERROR 12-08 22:01:13 engine.py:135] output = self.model_runner.execute_model( ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/anaconda3/envs/vllm-test/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context ERROR 12-08 22:01:13 engine.py:135] return func(*args, **kwargs) ERROR 12-08 22:01:13 engine.py:135] File "/home/liumx/Mooncake-vllm/vllm/vllm/worker/model_runner_base.py", line 152, in _wrapper ERROR 12-08 22:01:13 engine.py:135] raise type(err)( **ERROR 12-08 22:01:13 engine.py:135] ValueError: Error in model execution (input dumped to /tmp/err_execute_model_input_20241208-220113.pkl): not enough values to unpack (expected 4, got 2)** DEBUG 12-08 22:01:13 engine.py:139] MQLLMEngine is shut down. DEBUG 12-08 22:01:13 client.py:187] Waiting for output from MQLLMEngine. CRITICAL 12-08 22:01:13 launcher.py:99] MQLLMEngine is already dead, terminating server process INFO: 127.0.0.1:36768 - "POST /v1/completions HTTP/1.1" 500 Internal Server Error INFO: Shutting down INFO: Waiting for application shutdown. INFO: Application shutdown complete. INFO: Finished server process [148560] DEBUG 12-08 22:01:16 client.py:253] Shutting down MQLLMEngineClient output handler. DEBUG 12-08 22:01:16 client.py:253] Shutting down MQLLMEngineClient output handler. DEBUG 12-08 22:01:16 client.py:253] Shutting down MQLLMEngineClient output handler. DEBUG 12-08 22:01:16 client.py:169] Shutting down MQLLMEngineClient check health loop. DEBUG 12-08 22:01:16 client.py:253] Shutting down MQLLMEngineClient output handler. (Then prefill-vllm terminated and returned to the shell. Meanwhile, decode-vllm can still run normally. However, it cannot be terminated by Ctrl+C.) > > I launch them using these commands: prefill-vllm: VLLM_LOGGING_LEVEL=DEBUG CUDA_VISIBLE_DEVICES=2 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server --model /public/home/liumx/model_cache/hub/Qwen/Qwen2___5-7B-Instruct-GPTQ-Int4 --served-model-name Qwen2___5-7B-Instruct-GPTQ-Int4 --port 8100 --max-model-len 10000 --gpu-memory-utilization 0.95 --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2,"kv_buffer_size":5e9, "kv_ip": "10.10.10.12", "kv_port": 51000 }' --dtype=half > > mooncake.json of prefill-vllm: { "prefill_url": "10.10.10.12:13003", "decode_url": "10.10.10.13:13003", "metadata_server": "10.10.10.12:2379", "protocol": "rdma", "device_name": "mlx5_0" } > > decode-vllm: cd /home/liumx/Mooncake-vllm/vllm VLLM_LOGGING_LEVEL=DEBUG CUDA_VISIBLE_DEVICES=2 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server --model /public/home/liumx/model_cache/hub/Qwen/Qwen2___5-7B-Instruct-GPTQ-Int4 --served-model-name Qwen2___5-7B-Instruct-GPTQ-Int4 --port 8200 --max-model-len 10000 --gpu-memory-utilization 0.95 --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2,"kv_buffer_size":5e9, "kv_ip": "10.10.10.12", "kv_port": 51000}' --dtype=half > > mooncake.json of decode-vllm: { "prefill_url": "10.10.10.12:13003", "decode_url": "10.10.10.13:13003", "metadata_server": "10.10.10.12:2379", "protocol": "rdma", "device_name": "mlx5_0" } > > My environment is V100-32G, connected via 100Gbps IB and GDR. I have tested the correctness of GDR through Transfer Engine. > > Thank you for your help! Hello, this is a known bug for [PR 10502](https://github.com/vllm-project/vllm/pull/10502). To be consistent with the upstream PR, it is not fixed in this version but will be fixed in another PR. If you are in a hurry, I can instruct you how to perform a quick fix by modifying the code later today.
bsakaguchi commented 2026-10-29 21:07:56 +00:00
Author
Owner
Copy link

@LiuMicheal commented on GitHub (Dec 9, 2024):

Thank you very much for your help, please tell me a quick fix about modifying the code. @ShangmingCai

<!-- gh-comment-id:2527022049 --> @LiuMicheal commented on GitHub (Dec 9, 2024): Thank you very much for your help, please tell me a quick fix about modifying the code. @ShangmingCai
bsakaguchi commented 2026-10-29 21:07:56 +00:00
Author
Owner
Copy link

@ShangmingCai commented on GitHub (Dec 9, 2024):

Thank you very much for your help, please tell me a quick fix about modifying the code. @ShangmingCai

Try this branch "upstream-for-Volta/Turing", or you can modify the code by yourself according to the diff: github.com/kvcache-ai/vllm@b614cf1338

<!-- gh-comment-id:2527070883 --> @ShangmingCai commented on GitHub (Dec 9, 2024): > Thank you very much for your help, please tell me a quick fix about modifying the code. @ShangmingCai Try this branch "upstream-for-Volta/Turing", or you can modify the code by yourself according to the diff: https://github.com/kvcache-ai/vllm/commit/b614cf1338f5dd810386f6c1bc185bec65081950
bsakaguchi commented 2026-10-29 21:07:56 +00:00
Author
Owner
Copy link

@ShangmingCai commented on GitHub (Dec 9, 2024):

@LiuMicheal Let me know if this works for you. I have personally verified that this fix works on the Llama 3.2 and Qwen 2.5 series, but it is not clear whether it will work on all models.

<!-- gh-comment-id:2527215105 --> @ShangmingCai commented on GitHub (Dec 9, 2024): @LiuMicheal Let me know if this works for you. I have personally verified that this fix works on the Llama 3.2 and Qwen 2.5 series, but it is not clear whether it will work on all models.
bsakaguchi commented 2026-10-29 21:07:56 +00:00
Author
Owner
Copy link

@LiuMicheal commented on GitHub (Dec 9, 2024):

Yes, I am extremely thankful. @ShangmingCai I modified the lines of code you mentioned and solved this problem. modify the code by yourself according to the diff

image

I have not tried switching to branch "upstream-for-Volta/Turing" yet.
BTW, I feel a little confused about the reasoning for the "San Francisco is a" prompt. Is this normal (the upper right corner of the image)?

<!-- gh-comment-id:2527450473 --> @LiuMicheal commented on GitHub (Dec 9, 2024): Yes, I am extremely thankful. @ShangmingCai I modified the lines of code you mentioned and solved this problem. [modify the code by yourself according to the diff](https://github.com/kvcache-ai/vllm/commit/b614cf1338f5dd810386f6c1bc185bec65081950) ![image](https://github.com/user-attachments/assets/4f742b73-33c0-4c90-9858-3f56d41f036f) I have not tried switching to branch "upstream-for-Volta/Turing" yet. BTW, I feel a little confused about the reasoning for the "San Francisco is a" prompt. Is this normal (the upper right corner of the image)?
bsakaguchi commented 2026-10-29 21:07:56 +00:00
Author
Owner
Copy link

@ShangmingCai commented on GitHub (Dec 9, 2024):

BTW, I feel a little confused about the reasoning for the "San Francisco is a" prompt. Is this normal (the upper right corner of the image)?

@LiuMicheal I think it is normal because it is a /completions API, not a /chat/completions API. And the first output token is " city" which seems reasonable to me.

You can try this prompt to form a chat format prompt:

curl -s http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4",
  "prompt": "system: you are a helpful assistant.\n user: 你是？\nassistant:",
  "temperature":0.7,
  "top_p":0.8,
  "max_tokens":100
}'

Or you can change the code of the proxy_server.py to use a /chat/completions API.

<!-- gh-comment-id:2527542775 --> @ShangmingCai commented on GitHub (Dec 9, 2024): > BTW, I feel a little confused about the reasoning for the "San Francisco is a" prompt. Is this normal (the upper right corner of the image)? @LiuMicheal I think it is normal because it is a `/completions` API, not a `/chat/completions` API. And the first output token is " city" which seems reasonable to me. You can try this prompt to form a chat format prompt: ```bash curl -s http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{ "model": "Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4", "prompt": "system: you are a helpful assistant.\n user: 你是？\nassistant:", "temperature":0.7, "top_p":0.8, "max_tokens":100 }' ``` Or you can change the code of the `proxy_server.py` to use a `/chat/completions` API.
bsakaguchi commented 2026-10-29 21:07:57 +00:00
Author
Owner
Copy link

@LiuMicheal commented on GitHub (Dec 10, 2024):

curl -s http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{
  "model": "Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4",
  "prompt": "system: you are a helpful assistant.\n user: 你是？\nassistant:",
  "temperature":0.7,
  "top_p":0.8,
  "max_tokens":100
}'

Yes, after using this method, the correctness of the reasoning has improved a lot.

image

Thank you very much for solving all my problems. I think this issue can be closed now.

<!-- gh-comment-id:2529899303 --> @LiuMicheal commented on GitHub (Dec 10, 2024): > ```shell > curl -s http://localhost:8000/v1/completions -H "Content-Type: application/json" -d '{ > "model": "Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4", > "prompt": "system: you are a helpful assistant.\n user: 你是？\nassistant:", > "temperature":0.7, > "top_p":0.8, > "max_tokens":100 > }' > ``` Yes, after using this method, the correctness of the reasoning has improved a lot. ![image](https://github.com/user-attachments/assets/abcef480-9905-4bb2-af72-3e3a9333e68f) Thank you very much for solving all my problems. I think this issue can be closed now.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
te-coro
staryxchen-tent-capability-research
feature/musa-ep-jit
zxpdemonio/fix-tent-bootstrap-admission
add-store-rs-tone-tests
scatter-rdma-completion-pr2
P2P-Mooncake-Store
release/v0.3.14
feat/tent-rdma-adaptive-adapter
fix/ep-local-counter-gpu-scope
codex/kvcs-oplog-eviction
give-me-a-star
cruz/mooncake-kvcs-lowlevel-pr1-20261008
ylt-handshake-limits
ci/rocm-e2e-tent
sima/docs-pypi-downloads
fix-ci/hicache-mooncake-metadata-loopback
sima/cuda-driver-loading
sima/ci-performance-gate
fix/tent-rdma-unknown-topology-rail-pairing
ci/tent-e2e-dual-job
mc-patch-1
codex/p2p-merge-prerequisites
sima/store-hca-numa-placement
codex/p2p-merge-cli
codex/p2p-merge-ipv6
copilot/fix-ctests-job-failure
czh/pg-device-collectives-pr6
dependabot/cargo/mooncake-store-rs/pyo3-0.29.0
czh/pg-device-collectives-pr5
codex/segment-pool-master-cutover
sima/nightly-ci-recovery
ci/tent-params
stmatengss-patch-1
dependabot/cargo/mooncake-store-rs/rand-0.8.8
dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1
dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16
codex/migrate-store-rest-service
fix/tone-wheel-arch-selection
czh/pg-device-collectives-pr4
czh/pg-device-collectives-pr3
ketong/fix_moe_tc
copilot/fix-test-wheel-ubuntu-job
fix/ci-wheel-torch-pg-pin
codex/migrate-structured-object-storage
codex/migrate-allocator-python
codex/migrate-vllm-python
codex/migrate-shared-segment
ci/fix_sgl_eval_install
codex/segment-pool-nof-placement-cleanup
feat/pip-native-release
copilot/bugfix-ha-standby-promotion
copilot/mooncake-store-transport-settings
copilot/fix-tensor-support-issues
copilot/optimize-rl-direction
copilot/mooncake-storeagentic-workflow
copilot/fix-client-reconnect-timeout
copilot/mooncake-error-debugging
copilot/mooncake-ipv6-support
feat/nccl-host-python-api
dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1
copilot/migrate-and-move-operations
copilot/migrate-and-move-operations-again
refactor/centralize-env-vars
idhanani/mooncake-retain-groups
copilot/codeownerreviewer
copilot/give-feedback-on-issue-2123
copilot/create-issue-rfc
copilot/break-cuda-monopoly-logic
copilot/compare-cudamemcpy-cu12-cu13
copilot/report-ci-error-issue
copilot/research-commit-history-analysis
copilot/explain-mooncake-store-service
copilot/analyze-project-functionality
copilot/sub-pr-1680
copilot/review-pr-1680
copilot/sub-pr-1597
copilot/fix-github-actions-job-failure
copilot/move-test-sglang-integration-job
v0.3.14
v0.3.14-pre-4
v0.3.14-pre-3
v0.3.14-pre-2
v0.3.14-pre
v0.3.14-rc1
v0.3.13.post1
v0.3.13-rc2
v0.3.13-rc1
v0.3.13
v0.3.12.post1
v0.3.12
v0.3.11-rc1
v0.3.11.post1
v0.3.11
v0.3.10.post2
v0.3.10.post1
v0.3.10
v0.3.9
v0.3.8.post1
v0.3.8
v0.3.7.post2
v0.3.7.post1
v0.3.7
v0.3.6.post1
v0.3.6
v0.3.5
v0.3.4.post2
v0.3.4.post1
v0.3.4
v0.3.3.post2
v0.3.3.post1
v0.3.3
v0.3.2.post1
v0.3.2
v0.3.1
v0.3.0
v0.3.0-beta
v0.3.0-alpha
v0.2.0
Labels
Clear labels   
ascend

   
auto-closed

   
bug

   
bug

   
CI/Build

   
Common

   
documentation

   
enhancement

   
feature request

   
good first issue

   
help wanted

   
high priority

   
MetaX

   
nightly-failure

   
pull-request
Mirrored from GitHub Pull Request

  
PyTorch Backend

   
question

   
RFC

   
Roadmap

   
stale

   
Store

   
Tests

   
Transfer Engine
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#17
No description provided.