kvcache-ai/Mooncake
1
0
Fork 0
mirror of https://github.com/kvcache-ai/Mooncake.git synced 2026-11-05 01:17:55 +00:00
Code Issues 368 Projects Packages Wiki Activity

[GH-ISSUE #66] How to deploy 1 prefill instance with 3 decode instances? #39

New issue
Closed
opened 2026-10-29 21:08:16 +00:00 by bsakaguchi · 3 comments
bsakaguchi commented 2026-10-29 21:08:16 +00:00
Owner
Copy link

Originally created by @gujingit on GitHub (Jan 7, 2025).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/66

The demo shows deploy prefill and decode in 1:1. How to deploy prefill and decode instances in N:M?

Originally created by @gujingit on GitHub (Jan 7, 2025). Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/66 The demo shows deploy prefill and decode in 1:1. How to deploy prefill and decode instances in N:M?
bsakaguchi commented 2026-10-29 21:08:17 +00:00
Author
Owner
Copy link

@alogfans commented on GitHub (Jan 7, 2025):

The upstream vLLM repo only supports 1:1 currently. We are developing Mooncake Store and new intergration component to support N:M, which are estimated to be released in 25Q1.

<!-- gh-comment-id:2574271073 --> @alogfans commented on GitHub (Jan 7, 2025): The upstream vLLM repo only supports 1:1 currently. We are developing Mooncake Store and new intergration component to support N:M, which are estimated to be released in 25Q1.
bsakaguchi commented 2026-10-29 21:08:17 +00:00
Author
Owner
Copy link

@gujingit commented on GitHub (Jan 7, 2025):

I deploy prefill and decode in 1:1 on the same node following the doc vLLM Disaggregated Prefill/Decode Demo
.

Send a request but no output.

image

Prefill output

INFO 01-07 10:56:48 api_server.py:764] vLLM API server version 0.6.6.post2.dev89+g996357e4
INFO 01-07 10:56:48 api_server.py:765] args: Namespace(host=None, port=8100, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=256, guided_decoding_backend='xgrammar', logits_processor_pattern=None, distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.4, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=KVTransferConfig(kv_connector='MooncakeConnector', kv_buffer_device='cuda', kv_buffer_size=2000000000.0, kv_role='kv_producer', kv_rank=0, kv_parallel_size=2, kv_ip='127.0.0.1', kv_port=14579), worker_cls='auto', generation_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False)
INFO 01-07 10:56:48 __init__.py:179] Automatically detected platform cuda.
INFO 01-07 10:56:48 api_server.py:194] Started engine process with PID 21137
Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4
2025-01-07 10:56:50,256 - modelscope - INFO - Target directory already exists, skipping creation.
Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4
2025-01-07 10:56:51,128 - modelscope - INFO - Target directory already exists, skipping creation.
INFO 01-07 10:56:52 __init__.py:179] Automatically detected platform cuda.
Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4
2025-01-07 10:56:53,574 - modelscope - INFO - Target directory already exists, skipping creation.
Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4
2025-01-07 10:56:54,418 - modelscope - INFO - Target directory already exists, skipping creation.
INFO 01-07 10:57:02 config.py:517] This model supports multiple tasks: {'reward', 'score', 'embed', 'generate', 'classify'}. Defaulting to 'generate'.
INFO 01-07 10:57:03 gptq_marlin.py:109] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4
INFO 01-07 10:57:03 config.py:517] This model supports multiple tasks: {'generate', 'embed', 'classify', 'reward', 'score'}. Defaulting to 'generate'.
2025-01-07 10:57:04,070 - modelscope - INFO - Target directory already exists, skipping creation.
INFO 01-07 10:57:04 gptq_marlin.py:109] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
INFO 01-07 10:57:04 llm_engine.py:234] Initializing an LLM engine (v0.6.6.post2.dev89+g996357e4) with config: model='Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4', speculative_config=None, tokenizer='Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=256, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=gptq_marlin, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"candidate_compile_sizes":[],"compile_sizes":[],"capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True,
Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4
2025-01-07 10:57:05,648 - modelscope - INFO - Target directory already exists, skipping creation.
INFO 01-07 10:57:06 selector.py:120] Using Flash Attention backend.
INFO 01-07 10:57:06 simple_connector.py:58] Initializing MooncakeConfig under kv_transfer_config kv_connector='MooncakeConnector' kv_buffer_device='cuda' kv_buffer_size=2000000000.0 kv_role='kv_producer' kv_rank=0 kv_parallel_size=2 kv_ip='127.0.0.1' kv_port=14579
INFO 01-07 10:57:06 mooncake_pipe.py:227] Selecting device: cuda
INFO 01-07 10:57:06 mooncake_pipe.py:69] Mooncake Configuration loaded successfully.
INFO 01-07 10:57:06 model_runner.py:1094] Starting to load model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4...
INFO 01-07 10:57:06 gptq_marlin.py:200] Using MarlinLinearKernel for GPTQMarlinLinearMethod
Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4
2025-01-07 10:57:07,805 - modelscope - INFO - Target directory already exists, skipping creation.
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:00<00:00,  4.80it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00,  2.35it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00,  2.54it/s]

INFO 01-07 10:57:08 model_runner.py:1099] Loading model weights took 5.1810 GB
INFO 01-07 10:57:09 worker.py:241] Memory profiling takes 0.99 seconds
INFO 01-07 10:57:09 worker.py:241] the current vLLM instance can use total_gpu_memory (22.18GiB) x gpu_memory_utilization (0.40) = 8.87GiB
INFO 01-07 10:57:09 worker.py:241] model weights take 5.18GiB; non_torch_memory takes 0.26GiB; PyTorch activation peak memory takes 1.40GiB; the rest of the memory reserved for KV Cache is 2.03GiB.
INFO 01-07 10:57:09 gpu_executor.py:76] # GPU blocks: 2371, # CPU blocks: 4681
INFO 01-07 10:57:09 gpu_executor.py:80] Maximum concurrency for 256 tokens per request: 148.19x
INFO 01-07 10:57:12 model_runner.py:1416] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
Capturing CUDA graph shapes: 100%|██████████████| 35/35 [00:13<00:00,  2.58it/s]
INFO 01-07 10:57:25 model_runner.py:1541] Graph capturing finished in 14 secs, took 0.26 GiB
INFO 01-07 10:57:25 llm_engine.py:431] init engine (profile, create kv cache, warmup model) took 16.90 seconds
INFO 01-07 10:57:25 api_server.py:689] Using supplied chat template:
INFO 01-07 10:57:25 api_server.py:689] None
INFO 01-07 10:57:25 launcher.py:19] Available routes are:
INFO 01-07 10:57:25 launcher.py:27] Route: /openapi.json, Methods: GET, HEAD
INFO 01-07 10:57:25 launcher.py:27] Route: /docs, Methods: GET, HEAD
INFO 01-07 10:57:25 launcher.py:27] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 01-07 10:57:25 launcher.py:27] Route: /redoc, Methods: GET, HEAD
INFO 01-07 10:57:25 launcher.py:27] Route: /health, Methods: GET
INFO 01-07 10:57:25 launcher.py:27] Route: /ping, Methods: GET, POST
INFO 01-07 10:57:25 launcher.py:27] Route: /tokenize, Methods: POST
INFO 01-07 10:57:25 launcher.py:27] Route: /detokenize, Methods: POST
INFO 01-07 10:57:25 launcher.py:27] Route: /v1/models, Methods: GET
INFO 01-07 10:57:25 launcher.py:27] Route: /version, Methods: GET
INFO 01-07 10:57:25 launcher.py:27] Route: /v1/chat/completions, Methods: POST
INFO 01-07 10:57:25 launcher.py:27] Route: /v1/completions, Methods: POST
INFO 01-07 10:57:25 launcher.py:27] Route: /v1/embeddings, Methods: POST
INFO 01-07 10:57:25 launcher.py:27] Route: /pooling, Methods: POST
INFO 01-07 10:57:25 launcher.py:27] Route: /score, Methods: POST
INFO 01-07 10:57:25 launcher.py:27] Route: /v1/score, Methods: POST
INFO 01-07 10:57:25 launcher.py:27] Route: /invocations, Methods: POST
INFO:     Started server process [21000]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8100 (Press CTRL+C to quit)


INFO 01-07 11:01:43 logger.py:37] Received request cmpl-4b7898f5fdc54f9787ef117ac8576f29-0: prompt: 'San Francisco is a', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: [23729, 12879, 374, 264], lora_request: None, prompt_adapter_request: None.
INFO 01-07 11:01:43 engine.py:267] Added request cmpl-4b7898f5fdc54f9787ef117ac8576f29-0.
INFO 01-07 11:01:43 metrics.py:467] Avg prompt throughput: 0.7 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.
INFO:     127.0.0.1:33264 - "POST /v1/completions HTTP/1.1" 200 OK




INFO 01-07 11:01:53 metrics.py:467] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%.

Decode Output

INFO 01-07 10:56:50 api_server.py:764] vLLM API server version 0.6.6.post2.dev89+g996357e4
INFO 01-07 10:56:50 api_server.py:765] args: Namespace(host=None, port=8200, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=256, guided_decoding_backend='xgrammar', logits_processor_pattern=None, distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.4, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=KVTransferConfig(kv_connector='MooncakeConnector', kv_buffer_device='cuda', kv_buffer_size=2000000000.0, kv_role='kv_consumer', kv_rank=1, kv_parallel_size=2, kv_ip='127.0.0.1', kv_port=14579), worker_cls='auto', generation_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False)
INFO 01-07 10:56:50 __init__.py:179] Automatically detected platform cuda.
INFO 01-07 10:56:50 api_server.py:194] Started engine process with PID 21208
Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4
2025-01-07 10:56:51,757 - modelscope - INFO - Target directory already exists, skipping creation.
Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4
2025-01-07 10:56:52,636 - modelscope - INFO - Target directory already exists, skipping creation.
INFO 01-07 10:56:53 __init__.py:179] Automatically detected platform cuda.
Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4
2025-01-07 10:56:55,193 - modelscope - INFO - Target directory already exists, skipping creation.
Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4
2025-01-07 10:56:56,066 - modelscope - INFO - Target directory already exists, skipping creation.
INFO 01-07 10:57:02 config.py:517] This model supports multiple tasks: {'reward', 'score', 'classify', 'generate', 'embed'}. Defaulting to 'generate'.
INFO 01-07 10:57:03 gptq_marlin.py:109] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4
2025-01-07 10:57:04,017 - modelscope - INFO - Target directory already exists, skipping creation.
INFO 01-07 10:57:07 config.py:517] This model supports multiple tasks: {'score', 'classify', 'embed', 'generate', 'reward'}. Defaulting to 'generate'.
INFO 01-07 10:57:08 gptq_marlin.py:109] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel.
INFO 01-07 10:57:08 llm_engine.py:234] Initializing an LLM engine (v0.6.6.post2.dev89+g996357e4) with config: model='Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4', speculative_config=None, tokenizer='Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=256, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=gptq_marlin, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"candidate_compile_sizes":[],"compile_sizes":[],"capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True,
Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4
2025-01-07 10:57:09,290 - modelscope - INFO - Target directory already exists, skipping creation.
INFO 01-07 10:57:10 selector.py:120] Using Flash Attention backend.
INFO 01-07 10:57:10 simple_connector.py:58] Initializing MooncakeConfig under kv_transfer_config kv_connector='MooncakeConnector' kv_buffer_device='cuda' kv_buffer_size=2000000000.0 kv_role='kv_consumer' kv_rank=1 kv_parallel_size=2 kv_ip='127.0.0.1' kv_port=14579
INFO 01-07 10:57:10 mooncake_pipe.py:227] Selecting device: cuda
INFO 01-07 10:57:10 mooncake_pipe.py:69] Mooncake Configuration loaded successfully.
INFO 01-07 10:57:10 model_runner.py:1094] Starting to load model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4...
INFO 01-07 10:57:10 gptq_marlin.py:200] Using MarlinLinearKernel for GPTQMarlinLinearMethod
Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4
2025-01-07 10:57:11,514 - modelscope - INFO - Target directory already exists, skipping creation.
Loading safetensors checkpoint shards:   0% Completed | 0/2 [00:00<?, ?it/s]
Loading safetensors checkpoint shards:  50% Completed | 1/2 [00:00<00:00,  4.56it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00,  2.24it/s]
Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00,  2.42it/s]

INFO 01-07 10:57:12 model_runner.py:1099] Loading model weights took 5.1810 GB
INFO 01-07 10:57:13 worker.py:241] Memory profiling takes 0.97 seconds
INFO 01-07 10:57:13 worker.py:241] the current vLLM instance can use total_gpu_memory (22.18GiB) x gpu_memory_utilization (0.40) = 8.87GiB
INFO 01-07 10:57:13 worker.py:241] model weights take 5.18GiB; non_torch_memory takes 0.26GiB; PyTorch activation peak memory takes 1.40GiB; the rest of the memory reserved for KV Cache is 2.03GiB.
INFO 01-07 10:57:13 gpu_executor.py:76] # GPU blocks: 2371, # CPU blocks: 4681
INFO 01-07 10:57:13 gpu_executor.py:80] Maximum concurrency for 256 tokens per request: 148.19x
INFO 01-07 10:57:15 model_runner.py:1416] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage.
Capturing CUDA graph shapes: 100%|██████████████| 35/35 [00:13<00:00,  2.62it/s]
INFO 01-07 10:57:29 model_runner.py:1541] Graph capturing finished in 13 secs, took 0.26 GiB
INFO 01-07 10:57:29 llm_engine.py:431] init engine (profile, create kv cache, warmup model) took 16.68 seconds
INFO 01-07 10:57:29 api_server.py:689] Using supplied chat template:
INFO 01-07 10:57:29 api_server.py:689] None
INFO 01-07 10:57:29 launcher.py:19] Available routes are:
INFO 01-07 10:57:29 launcher.py:27] Route: /openapi.json, Methods: GET, HEAD
INFO 01-07 10:57:29 launcher.py:27] Route: /docs, Methods: GET, HEAD
INFO 01-07 10:57:29 launcher.py:27] Route: /docs/oauth2-redirect, Methods: GET, HEAD
INFO 01-07 10:57:29 launcher.py:27] Route: /redoc, Methods: GET, HEAD
INFO 01-07 10:57:29 launcher.py:27] Route: /health, Methods: GET
INFO 01-07 10:57:29 launcher.py:27] Route: /ping, Methods: GET, POST
INFO 01-07 10:57:29 launcher.py:27] Route: /tokenize, Methods: POST
INFO 01-07 10:57:29 launcher.py:27] Route: /detokenize, Methods: POST
INFO 01-07 10:57:29 launcher.py:27] Route: /v1/models, Methods: GET
INFO 01-07 10:57:29 launcher.py:27] Route: /version, Methods: GET
INFO 01-07 10:57:29 launcher.py:27] Route: /v1/chat/completions, Methods: POST
INFO 01-07 10:57:29 launcher.py:27] Route: /v1/completions, Methods: POST
INFO 01-07 10:57:29 launcher.py:27] Route: /v1/embeddings, Methods: POST
INFO 01-07 10:57:29 launcher.py:27] Route: /pooling, Methods: POST
INFO 01-07 10:57:29 launcher.py:27] Route: /score, Methods: POST
INFO 01-07 10:57:29 launcher.py:27] Route: /v1/score, Methods: POST
INFO 01-07 10:57:29 launcher.py:27] Route: /invocations, Methods: POST
INFO:     Started server process [21066]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8200 (Press CTRL+C to quit)

Mooncake.json

{
  "prefill_url": "127.0.0.1:13003",
  "decode_url": "127.0.0.1:13103",
  "metadata_server": "127.0.0.1:2379",
  "metadata_backend": "etcd",
  "protocol": "tcp",
  "device_name": ""
}
<!-- gh-comment-id:2574301120 --> @gujingit commented on GitHub (Jan 7, 2025): I deploy prefill and decode in 1:1 on the same node following the doc [vLLM Disaggregated Prefill/Decode Demo ](https://github.com/kvcache-ai/Mooncake/blob/main/doc/en/vllm-integration-v0.2.md). Send a request but no output. ![image](https://github.com/user-attachments/assets/d5d36b55-a0ed-4205-9fa4-577acc99c8d7) Prefill output ```CUDA_VISIBLE_DEVICES=0 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8100 --max-model-len 256 --gpu-memory-utilization 0.4 --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_producer","kv_rank":0,"kv_parallel_size":2,"kv_buffer_size":2e9}' INFO 01-07 10:56:48 api_server.py:764] vLLM API server version 0.6.6.post2.dev89+g996357e4 INFO 01-07 10:56:48 api_server.py:765] args: Namespace(host=None, port=8100, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=256, guided_decoding_backend='xgrammar', logits_processor_pattern=None, distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.4, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=KVTransferConfig(kv_connector='MooncakeConnector', kv_buffer_device='cuda', kv_buffer_size=2000000000.0, kv_role='kv_producer', kv_rank=0, kv_parallel_size=2, kv_ip='127.0.0.1', kv_port=14579), worker_cls='auto', generation_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False) INFO 01-07 10:56:48 __init__.py:179] Automatically detected platform cuda. INFO 01-07 10:56:48 api_server.py:194] Started engine process with PID 21137 Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 2025-01-07 10:56:50,256 - modelscope - INFO - Target directory already exists, skipping creation. Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 2025-01-07 10:56:51,128 - modelscope - INFO - Target directory already exists, skipping creation. INFO 01-07 10:56:52 __init__.py:179] Automatically detected platform cuda. Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 2025-01-07 10:56:53,574 - modelscope - INFO - Target directory already exists, skipping creation. Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 2025-01-07 10:56:54,418 - modelscope - INFO - Target directory already exists, skipping creation. INFO 01-07 10:57:02 config.py:517] This model supports multiple tasks: {'reward', 'score', 'embed', 'generate', 'classify'}. Defaulting to 'generate'. INFO 01-07 10:57:03 gptq_marlin.py:109] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel. Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 INFO 01-07 10:57:03 config.py:517] This model supports multiple tasks: {'generate', 'embed', 'classify', 'reward', 'score'}. Defaulting to 'generate'. 2025-01-07 10:57:04,070 - modelscope - INFO - Target directory already exists, skipping creation. INFO 01-07 10:57:04 gptq_marlin.py:109] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel. INFO 01-07 10:57:04 llm_engine.py:234] Initializing an LLM engine (v0.6.6.post2.dev89+g996357e4) with config: model='Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4', speculative_config=None, tokenizer='Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=256, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=gptq_marlin, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"candidate_compile_sizes":[],"compile_sizes":[],"capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True, Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 2025-01-07 10:57:05,648 - modelscope - INFO - Target directory already exists, skipping creation. INFO 01-07 10:57:06 selector.py:120] Using Flash Attention backend. INFO 01-07 10:57:06 simple_connector.py:58] Initializing MooncakeConfig under kv_transfer_config kv_connector='MooncakeConnector' kv_buffer_device='cuda' kv_buffer_size=2000000000.0 kv_role='kv_producer' kv_rank=0 kv_parallel_size=2 kv_ip='127.0.0.1' kv_port=14579 INFO 01-07 10:57:06 mooncake_pipe.py:227] Selecting device: cuda INFO 01-07 10:57:06 mooncake_pipe.py:69] Mooncake Configuration loaded successfully. INFO 01-07 10:57:06 model_runner.py:1094] Starting to load model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4... INFO 01-07 10:57:06 gptq_marlin.py:200] Using MarlinLinearKernel for GPTQMarlinLinearMethod Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 2025-01-07 10:57:07,805 - modelscope - INFO - Target directory already exists, skipping creation. Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:00<00:00, 4.80it/s] Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.35it/s] Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.54it/s] INFO 01-07 10:57:08 model_runner.py:1099] Loading model weights took 5.1810 GB INFO 01-07 10:57:09 worker.py:241] Memory profiling takes 0.99 seconds INFO 01-07 10:57:09 worker.py:241] the current vLLM instance can use total_gpu_memory (22.18GiB) x gpu_memory_utilization (0.40) = 8.87GiB INFO 01-07 10:57:09 worker.py:241] model weights take 5.18GiB; non_torch_memory takes 0.26GiB; PyTorch activation peak memory takes 1.40GiB; the rest of the memory reserved for KV Cache is 2.03GiB. INFO 01-07 10:57:09 gpu_executor.py:76] # GPU blocks: 2371, # CPU blocks: 4681 INFO 01-07 10:57:09 gpu_executor.py:80] Maximum concurrency for 256 tokens per request: 148.19x INFO 01-07 10:57:12 model_runner.py:1416] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage. Capturing CUDA graph shapes: 100%|██████████████| 35/35 [00:13<00:00, 2.58it/s] INFO 01-07 10:57:25 model_runner.py:1541] Graph capturing finished in 14 secs, took 0.26 GiB INFO 01-07 10:57:25 llm_engine.py:431] init engine (profile, create kv cache, warmup model) took 16.90 seconds INFO 01-07 10:57:25 api_server.py:689] Using supplied chat template: INFO 01-07 10:57:25 api_server.py:689] None INFO 01-07 10:57:25 launcher.py:19] Available routes are: INFO 01-07 10:57:25 launcher.py:27] Route: /openapi.json, Methods: GET, HEAD INFO 01-07 10:57:25 launcher.py:27] Route: /docs, Methods: GET, HEAD INFO 01-07 10:57:25 launcher.py:27] Route: /docs/oauth2-redirect, Methods: GET, HEAD INFO 01-07 10:57:25 launcher.py:27] Route: /redoc, Methods: GET, HEAD INFO 01-07 10:57:25 launcher.py:27] Route: /health, Methods: GET INFO 01-07 10:57:25 launcher.py:27] Route: /ping, Methods: GET, POST INFO 01-07 10:57:25 launcher.py:27] Route: /tokenize, Methods: POST INFO 01-07 10:57:25 launcher.py:27] Route: /detokenize, Methods: POST INFO 01-07 10:57:25 launcher.py:27] Route: /v1/models, Methods: GET INFO 01-07 10:57:25 launcher.py:27] Route: /version, Methods: GET INFO 01-07 10:57:25 launcher.py:27] Route: /v1/chat/completions, Methods: POST INFO 01-07 10:57:25 launcher.py:27] Route: /v1/completions, Methods: POST INFO 01-07 10:57:25 launcher.py:27] Route: /v1/embeddings, Methods: POST INFO 01-07 10:57:25 launcher.py:27] Route: /pooling, Methods: POST INFO 01-07 10:57:25 launcher.py:27] Route: /score, Methods: POST INFO 01-07 10:57:25 launcher.py:27] Route: /v1/score, Methods: POST INFO 01-07 10:57:25 launcher.py:27] Route: /invocations, Methods: POST INFO: Started server process [21000] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8100 (Press CTRL+C to quit) INFO 01-07 11:01:43 logger.py:37] Received request cmpl-4b7898f5fdc54f9787ef117ac8576f29-0: prompt: 'San Francisco is a', params: SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=1.0, top_k=-1, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], include_stop_str_in_output=False, ignore_eos=False, max_tokens=1, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, truncate_prompt_tokens=None, guided_decoding=None), prompt_token_ids: [23729, 12879, 374, 264], lora_request: None, prompt_adapter_request: None. INFO 01-07 11:01:43 engine.py:267] Added request cmpl-4b7898f5fdc54f9787ef117ac8576f29-0. INFO 01-07 11:01:43 metrics.py:467] Avg prompt throughput: 0.7 tokens/s, Avg generation throughput: 0.2 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%. INFO: 127.0.0.1:33264 - "POST /v1/completions HTTP/1.1" 200 OK INFO 01-07 11:01:53 metrics.py:467] Avg prompt throughput: 0.0 tokens/s, Avg generation throughput: 0.0 tokens/s, Running: 0 reqs, Swapped: 0 reqs, Pending: 0 reqs, GPU KV cache usage: 0.0%, CPU KV cache usage: 0.0%. ``` Decode Output ```CUDA_VISIBLE_DEVICES=1 MOONCAKE_CONFIG_PATH=./mooncake.json VLLM_USE_MODELSCOPE=True python3 -m vllm.entrypoints.openai.api_server --model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 --port 8200 --max-model-len 256 --gpu-memory-utilization 0.4 --kv-transfer-config '{"kv_connector":"MooncakeConnector","kv_role":"kv_consumer","kv_rank":1,"kv_parallel_size":2,"kv_buffer_size":2e9}' INFO 01-07 10:56:50 api_server.py:764] vLLM API server version 0.6.6.post2.dev89+g996357e4 INFO 01-07 10:56:50 api_server.py:765] args: Namespace(host=None, port=8200, uvicorn_log_level='info', allow_credentials=False, allowed_origins=['*'], allowed_methods=['*'], allowed_headers=['*'], api_key=None, lora_modules=None, prompt_adapters=None, chat_template=None, chat_template_content_format='auto', response_role='assistant', ssl_keyfile=None, ssl_certfile=None, ssl_ca_certs=None, ssl_cert_reqs=0, root_path=None, middleware=[], return_tokens_as_token_ids=False, disable_frontend_multiprocessing=False, enable_request_id_headers=False, enable_auto_tool_choice=False, tool_call_parser=None, tool_parser_plugin='', model='Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4', task='auto', tokenizer=None, skip_tokenizer_init=False, revision=None, code_revision=None, tokenizer_revision=None, tokenizer_mode='auto', trust_remote_code=False, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='auto', kv_cache_dtype='auto', quantization_param_path=None, max_model_len=256, guided_decoding_backend='xgrammar', logits_processor_pattern=None, distributed_executor_backend=None, worker_use_ray=False, pipeline_parallel_size=1, tensor_parallel_size=1, max_parallel_loading_workers=None, ray_workers_use_nsight=False, block_size=None, enable_prefix_caching=None, disable_sliding_window=False, use_v2_block_manager=True, num_lookahead_slots=0, seed=0, swap_space=4, cpu_offload_gb=0, gpu_memory_utilization=0.4, num_gpu_blocks_override=None, max_num_batched_tokens=None, max_num_seqs=None, max_logprobs=20, disable_log_stats=False, quantization=None, rope_scaling=None, rope_theta=None, hf_overrides=None, enforce_eager=False, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, lora_extra_vocab_size=256, lora_dtype='auto', long_lora_scaling_factors=None, max_cpu_loras=None, fully_sharded_loras=False, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, scheduler_delay_factor=0.0, enable_chunked_prefill=None, speculative_model=None, speculative_model_quantization=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_draft_tensor_parallel_size=None, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, disable_logprobs_during_spec_decoding=None, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, served_model_name=None, qlora_adapter_name_or_path=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, kv_transfer_config=KVTransferConfig(kv_connector='MooncakeConnector', kv_buffer_device='cuda', kv_buffer_size=2000000000.0, kv_role='kv_consumer', kv_rank=1, kv_parallel_size=2, kv_ip='127.0.0.1', kv_port=14579), worker_cls='auto', generation_config=None, disable_log_requests=False, max_log_len=None, disable_fastapi_docs=False, enable_prompt_tokens_details=False) INFO 01-07 10:56:50 __init__.py:179] Automatically detected platform cuda. INFO 01-07 10:56:50 api_server.py:194] Started engine process with PID 21208 Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 2025-01-07 10:56:51,757 - modelscope - INFO - Target directory already exists, skipping creation. Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 2025-01-07 10:56:52,636 - modelscope - INFO - Target directory already exists, skipping creation. INFO 01-07 10:56:53 __init__.py:179] Automatically detected platform cuda. Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 2025-01-07 10:56:55,193 - modelscope - INFO - Target directory already exists, skipping creation. Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 2025-01-07 10:56:56,066 - modelscope - INFO - Target directory already exists, skipping creation. INFO 01-07 10:57:02 config.py:517] This model supports multiple tasks: {'reward', 'score', 'classify', 'generate', 'embed'}. Defaulting to 'generate'. INFO 01-07 10:57:03 gptq_marlin.py:109] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel. Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 2025-01-07 10:57:04,017 - modelscope - INFO - Target directory already exists, skipping creation. INFO 01-07 10:57:07 config.py:517] This model supports multiple tasks: {'score', 'classify', 'embed', 'generate', 'reward'}. Defaulting to 'generate'. INFO 01-07 10:57:08 gptq_marlin.py:109] The model is convertible to gptq_marlin during runtime. Using gptq_marlin kernel. INFO 01-07 10:57:08 llm_engine.py:234] Initializing an LLM engine (v0.6.6.post2.dev89+g996357e4) with config: model='Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4', speculative_config=None, tokenizer='Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, override_neuron_config=None, tokenizer_revision=None, trust_remote_code=False, dtype=torch.float16, max_seq_len=256, download_dir=None, load_format=auto, tensor_parallel_size=1, pipeline_parallel_size=1, disable_custom_all_reduce=False, quantization=gptq_marlin, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='xgrammar'), observability_config=ObservabilityConfig(otlp_traces_endpoint=None, collect_model_forward_time=False, collect_model_execute_time=False), seed=0, served_model_name=Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4, num_scheduler_steps=1, multi_step_stream_outputs=True, enable_prefix_caching=False, chunked_prefill_enabled=False, use_async_output_proc=True, disable_mm_preprocessor_cache=False, mm_processor_kwargs=None, pooler_config=None, compilation_config={"splitting_ops":["vllm.unified_attention","vllm.unified_attention_with_output"],"candidate_compile_sizes":[],"compile_sizes":[],"capture_sizes":[256,248,240,232,224,216,208,200,192,184,176,168,160,152,144,136,128,120,112,104,96,88,80,72,64,56,48,40,32,24,16,8,4,2,1],"max_capture_size":256}, use_cached_outputs=True, Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 2025-01-07 10:57:09,290 - modelscope - INFO - Target directory already exists, skipping creation. INFO 01-07 10:57:10 selector.py:120] Using Flash Attention backend. INFO 01-07 10:57:10 simple_connector.py:58] Initializing MooncakeConfig under kv_transfer_config kv_connector='MooncakeConnector' kv_buffer_device='cuda' kv_buffer_size=2000000000.0 kv_role='kv_consumer' kv_rank=1 kv_parallel_size=2 kv_ip='127.0.0.1' kv_port=14579 INFO 01-07 10:57:10 mooncake_pipe.py:227] Selecting device: cuda INFO 01-07 10:57:10 mooncake_pipe.py:69] Mooncake Configuration loaded successfully. INFO 01-07 10:57:10 model_runner.py:1094] Starting to load model Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4... INFO 01-07 10:57:10 gptq_marlin.py:200] Using MarlinLinearKernel for GPTQMarlinLinearMethod Downloading Model to directory: /root/.cache/modelscope/hub/Qwen/Qwen2.5-7B-Instruct-GPTQ-Int4 2025-01-07 10:57:11,514 - modelscope - INFO - Target directory already exists, skipping creation. Loading safetensors checkpoint shards: 0% Completed | 0/2 [00:00<?, ?it/s] Loading safetensors checkpoint shards: 50% Completed | 1/2 [00:00<00:00, 4.56it/s] Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.24it/s] Loading safetensors checkpoint shards: 100% Completed | 2/2 [00:00<00:00, 2.42it/s] INFO 01-07 10:57:12 model_runner.py:1099] Loading model weights took 5.1810 GB INFO 01-07 10:57:13 worker.py:241] Memory profiling takes 0.97 seconds INFO 01-07 10:57:13 worker.py:241] the current vLLM instance can use total_gpu_memory (22.18GiB) x gpu_memory_utilization (0.40) = 8.87GiB INFO 01-07 10:57:13 worker.py:241] model weights take 5.18GiB; non_torch_memory takes 0.26GiB; PyTorch activation peak memory takes 1.40GiB; the rest of the memory reserved for KV Cache is 2.03GiB. INFO 01-07 10:57:13 gpu_executor.py:76] # GPU blocks: 2371, # CPU blocks: 4681 INFO 01-07 10:57:13 gpu_executor.py:80] Maximum concurrency for 256 tokens per request: 148.19x INFO 01-07 10:57:15 model_runner.py:1416] Capturing cudagraphs for decoding. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. If out-of-memory error occurs during cudagraph capture, consider decreasing `gpu_memory_utilization` or switching to eager mode. You can also reduce the `max_num_seqs` as needed to decrease memory usage. Capturing CUDA graph shapes: 100%|██████████████| 35/35 [00:13<00:00, 2.62it/s] INFO 01-07 10:57:29 model_runner.py:1541] Graph capturing finished in 13 secs, took 0.26 GiB INFO 01-07 10:57:29 llm_engine.py:431] init engine (profile, create kv cache, warmup model) took 16.68 seconds INFO 01-07 10:57:29 api_server.py:689] Using supplied chat template: INFO 01-07 10:57:29 api_server.py:689] None INFO 01-07 10:57:29 launcher.py:19] Available routes are: INFO 01-07 10:57:29 launcher.py:27] Route: /openapi.json, Methods: GET, HEAD INFO 01-07 10:57:29 launcher.py:27] Route: /docs, Methods: GET, HEAD INFO 01-07 10:57:29 launcher.py:27] Route: /docs/oauth2-redirect, Methods: GET, HEAD INFO 01-07 10:57:29 launcher.py:27] Route: /redoc, Methods: GET, HEAD INFO 01-07 10:57:29 launcher.py:27] Route: /health, Methods: GET INFO 01-07 10:57:29 launcher.py:27] Route: /ping, Methods: GET, POST INFO 01-07 10:57:29 launcher.py:27] Route: /tokenize, Methods: POST INFO 01-07 10:57:29 launcher.py:27] Route: /detokenize, Methods: POST INFO 01-07 10:57:29 launcher.py:27] Route: /v1/models, Methods: GET INFO 01-07 10:57:29 launcher.py:27] Route: /version, Methods: GET INFO 01-07 10:57:29 launcher.py:27] Route: /v1/chat/completions, Methods: POST INFO 01-07 10:57:29 launcher.py:27] Route: /v1/completions, Methods: POST INFO 01-07 10:57:29 launcher.py:27] Route: /v1/embeddings, Methods: POST INFO 01-07 10:57:29 launcher.py:27] Route: /pooling, Methods: POST INFO 01-07 10:57:29 launcher.py:27] Route: /score, Methods: POST INFO 01-07 10:57:29 launcher.py:27] Route: /v1/score, Methods: POST INFO 01-07 10:57:29 launcher.py:27] Route: /invocations, Methods: POST INFO: Started server process [21066] INFO: Waiting for application startup. INFO: Application startup complete. INFO: Uvicorn running on http://0.0.0.0:8200 (Press CTRL+C to quit) ``` Mooncake.json ``` { "prefill_url": "127.0.0.1:13003", "decode_url": "127.0.0.1:13103", "metadata_server": "127.0.0.1:2379", "metadata_backend": "etcd", "protocol": "tcp", "device_name": "" } ```
bsakaguchi commented 2026-10-29 21:08:17 +00:00
Author
Owner
Copy link

@gujingit commented on GitHub (Jan 7, 2025):

fixed

<!-- gh-comment-id:2574306709 --> @gujingit commented on GitHub (Jan 7, 2025): fixed
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
te-coro
staryxchen-tent-capability-research
feature/musa-ep-jit
zxpdemonio/fix-tent-bootstrap-admission
add-store-rs-tone-tests
scatter-rdma-completion-pr2
P2P-Mooncake-Store
release/v0.3.14
feat/tent-rdma-adaptive-adapter
fix/ep-local-counter-gpu-scope
codex/kvcs-oplog-eviction
give-me-a-star
cruz/mooncake-kvcs-lowlevel-pr1-20261008
ylt-handshake-limits
ci/rocm-e2e-tent
sima/docs-pypi-downloads
fix-ci/hicache-mooncake-metadata-loopback
sima/cuda-driver-loading
sima/ci-performance-gate
fix/tent-rdma-unknown-topology-rail-pairing
ci/tent-e2e-dual-job
mc-patch-1
codex/p2p-merge-prerequisites
sima/store-hca-numa-placement
codex/p2p-merge-cli
codex/p2p-merge-ipv6
copilot/fix-ctests-job-failure
czh/pg-device-collectives-pr6
dependabot/cargo/mooncake-store-rs/pyo3-0.29.0
czh/pg-device-collectives-pr5
codex/segment-pool-master-cutover
sima/nightly-ci-recovery
ci/tent-params
stmatengss-patch-1
dependabot/cargo/mooncake-store-rs/rand-0.8.8
dependabot/cargo/mooncake-store-rs/opentelemetry_sdk-0.32.1
dependabot/cargo/mooncake-store-rs/xxhash-rust-0.8.16
codex/migrate-store-rest-service
fix/tone-wheel-arch-selection
czh/pg-device-collectives-pr4
czh/pg-device-collectives-pr3
ketong/fix_moe_tc
copilot/fix-test-wheel-ubuntu-job
fix/ci-wheel-torch-pg-pin
codex/migrate-structured-object-storage
codex/migrate-allocator-python
codex/migrate-vllm-python
codex/migrate-shared-segment
ci/fix_sgl_eval_install
codex/segment-pool-nof-placement-cleanup
feat/pip-native-release
copilot/bugfix-ha-standby-promotion
copilot/mooncake-store-transport-settings
copilot/fix-tensor-support-issues
copilot/optimize-rl-direction
copilot/mooncake-storeagentic-workflow
copilot/fix-client-reconnect-timeout
copilot/mooncake-error-debugging
copilot/mooncake-ipv6-support
feat/nccl-host-python-api
dependabot/go_modules/mooncake-p2p-store/src/p2pstore/google.golang.org/grpc-1.82.1
copilot/migrate-and-move-operations
copilot/migrate-and-move-operations-again
refactor/centralize-env-vars
idhanani/mooncake-retain-groups
copilot/codeownerreviewer
copilot/give-feedback-on-issue-2123
copilot/create-issue-rfc
copilot/break-cuda-monopoly-logic
copilot/compare-cudamemcpy-cu12-cu13
copilot/report-ci-error-issue
copilot/research-commit-history-analysis
copilot/explain-mooncake-store-service
copilot/analyze-project-functionality
copilot/sub-pr-1680
copilot/review-pr-1680
copilot/sub-pr-1597
copilot/fix-github-actions-job-failure
copilot/move-test-sglang-integration-job
v0.3.14
v0.3.14-pre-4
v0.3.14-pre-3
v0.3.14-pre-2
v0.3.14-pre
v0.3.14-rc1
v0.3.13.post1
v0.3.13-rc2
v0.3.13-rc1
v0.3.13
v0.3.12.post1
v0.3.12
v0.3.11-rc1
v0.3.11.post1
v0.3.11
v0.3.10.post2
v0.3.10.post1
v0.3.10
v0.3.9
v0.3.8.post1
v0.3.8
v0.3.7.post2
v0.3.7.post1
v0.3.7
v0.3.6.post1
v0.3.6
v0.3.5
v0.3.4.post2
v0.3.4.post1
v0.3.4
v0.3.3.post2
v0.3.3.post1
v0.3.3
v0.3.2.post1
v0.3.2
v0.3.1
v0.3.0
v0.3.0-beta
v0.3.0-alpha
v0.2.0
Labels
Clear labels   
ascend

   
auto-closed

   
bug

   
bug

   
CI/Build

   
Common

   
documentation

   
enhancement

   
feature request

   
good first issue

   
help wanted

   
high priority

   
MetaX

   
nightly-failure

   
pull-request
Mirrored from GitHub Pull Request

  
PyTorch Backend

   
question

   
RFC

   
Roadmap

   
stale

   
Store

   
Tests

   
Transfer Engine
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
kvcache-ai/Mooncake#39
No description provided.