mirror of
https://github.com/kvcache-ai/Mooncake.git
synced 2026-11-05 01:17:55 +00:00
[GH-ISSUE #2162] [Bug]: glm5+vllm0.21.0+mooncake error #597
Labels
No labels
ascend
auto-closed
bug
bug
CI/Build
Common
documentation
enhancement
feature request
good first issue
help wanted
high priority
MetaX
nightly-failure
pull-request
PyTorch Backend
question
RFC
Roadmap
stale
Store
Tests
Transfer Engine
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/Mooncake#597
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @yiminghub2024 on GitHub (May 21, 2026).
Original GitHub issue: https://github.com/kvcache-ai/Mooncake/issues/2162
Originally assigned to: @LujhCoconut on GitHub.
Bug Report
glm5-fp8+vllm0.21.0+mooncake error(8*h20)
start:
vllm serve /data/cache/model_scope/ZhipuAI/GLM-5-FP8
--host 10.133.72.178
--port 40036
--served-model-name glm-5-fp8-copy
--tensor-parallel-size 8
--gpu-memory-utilization 0.95
--max-model-len 200000
--performance-mode throughput
--speculative-config.method mtp
--speculative-config.num_speculative_tokens 3
--enable-expert-parallel
--tool-call-parser glm47
--reasoning-parser glm45
--enable-auto-tool-choice
--default-chat-template-kwargs '{"enable_thinking":false}'
--chat-template-content-format string
--trust-remote-code
--kv-transfer-config '{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both"}'
error:
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.runnable(*args, **kwargs)
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/compilation/piecewise_backend.py", line 380, in call
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return range_entry.runnable(*args)
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_inductor/standalone_compile.py", line 122, in call
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self._compiled_fn(*args)
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1263, in _fn
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return fn(*args, **kwargs)
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1200, in forward
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return compiled_fn(full_args)
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 580, in runtime_wrapper
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] all_outs = call_func_at_runtime_with_args(
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 138, in call_func_at_runtime_with_args
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] out = normalize_as_list(f(args))
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 2298, in call
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.compiled_fn(*args, **kwargs)
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 783, in wrapper
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return compiled_fn(runtime_args)
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 1011, in inner_fn
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] outs = compiled_fn(args)
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_inductor/output_code.py", line 656, in call
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.current_callable(inputs)
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_inductor/utils.py", line 3401, in run
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] out = model(new_inputs)
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.cache/vllm/torch_compile_cache/torch_aot_compile/7f96116d443cdb5d82062835e1f50f1e3ab2137fa18992e02839c43f3e56a536/inductor_cache/p5/cp5hbnwey632w5busaam2uzywiyeqzwh3ipzyqwhfm5qubvxlcww.py", line 2091, in call
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] buf8 = torch.ops.vllm.moe_forward_shared.default(buf5, buf6, buf7, None, arg6_1, 0)
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_ops.py", line 865, in call
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self._op(*args, **kwargs)
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_compile.py", line 54, in inner
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return disable_fn(*args, **kwargs)
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1263, in _fn
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return fn(*args, **kwargs)
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 409, in torch_dispatch
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] res = func(*args, **kwargs)
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_ops.py", line 865, in call
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self._op(*args, **kwargs)
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 134, in _moe_forward_shared
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return layer.runner._forward_impl(
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 795, in _forward_impl
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] shared_output, hidden_states = self._apply_quant_method(
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 538, in _apply_quant_method
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] fused_out = self._quant_method.apply(
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/quantization/fp8.py", line 907, in apply
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.moe_kernel.apply(
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1633, in apply
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.impl.apply(
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1410, in apply
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] fused_out = self.fused_experts(
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1264, in fused_experts
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] self.fused_experts.apply(
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/experts/flashinfer_cutlass_moe.py", line 369, in apply
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] _ = flashinfer_cutlass_fused_moe(
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/utils/flashinfer.py", line 99, in wrapper
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return impl(*args, **kwargs)
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/flashinfer/fused_moe/core.py", line 794, in cutlass_fused_moe
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return get_cutlass_fused_moe_module(device_arch).cutlass_fused_moe(
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/flashinfer/fused_moe/core.py", line 499, in cutlass_fused_moe
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] run_moe(
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "python/tvm_ffi/cython/function.pxi", line 929, in tvm_ffi.core.Function.call
(Worker_TP5_EP5 pid=214235) ERROR 05-21 02:40:18 [multiproc_executor.py:962] RuntimeError: Assertion failed: !cubin.empty() || isPathValid(path)
sh: 1: nvcc: not found
[TensorRT-LLM][ERROR] Warning: Failed to copy kernel files to cache: filesystem error: cannot rename: No such file or directory [/root/.tensorrt_llm/tmp/gemm_4096_6144_128_128_128_32_532_5_1_GroupedWithOffset_1779331218419_463457/nvcc_kernel.cubin] [/root/.tensorrt_llm/cache/gemm_4096_6144_128_128_128_32_532_5_1_GroupedWithOffset/nvcc_kernel.cubin]
Assertion failed: /root/.venv/lib/python3.10/site-packages/flashinfer/data/csrc/nv_internal/tensorrt_llm/deep_gemm/runtime.cuh:63, condition: !cubin.empty() || isPathValid(path)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] WorkerProc hit an exception.
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] Traceback (most recent call last):
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 957, in worker_busy_loop
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] output = func(*args, **kwargs)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return func(*args, **kwargs)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 392, in determine_available_memory
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] self.model_runner.profile_run()
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5948, in profile_run
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] hidden_states, last_hidden_states = self._dummy_run(
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return func(*args, **kwargs)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5616, in _dummy_run
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] outputs = self.model(
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/compilation/cuda_graph.py", line 254, in call
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.runnable(*args, **kwargs)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self._call_impl(*args, **kwargs)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return forward_call(*args, **kwargs)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/models/deepseek_v2.py", line 1692, in forward
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] hidden_states = self.model(
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 670, in call
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] output = self.aot_compiled_fn(self, *args, **kwargs)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_dynamo/aot_compile.py", line 224, in call
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.fn(*args, **kwargs)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/models/deepseek_v2.py", line 1271, in forward
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] def forward(
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/compilation/caching.py", line 217, in call
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.optimized_call(*args, **kwargs)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "", line 850, in execution_fn
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/compilation/cuda_graph.py", line 254, in call
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.runnable(*args, **kwargs)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/compilation/piecewise_backend.py", line 380, in call
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return range_entry.runnable(*args)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_inductor/standalone_compile.py", line 122, in call
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self._compiled_fn(*args)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1263, in _fn
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return fn(*args, **kwargs)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1200, in forward
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return compiled_fn(full_args)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 580, in runtime_wrapper
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] all_outs = call_func_at_runtime_with_args(
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 138, in call_func_at_runtime_with_args
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] out = normalize_as_list(f(args))
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 2298, in call
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.compiled_fn(*args, **kwargs)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 783, in wrapper
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return compiled_fn(runtime_args)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 1011, in inner_fn
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] outs = compiled_fn(args)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_inductor/output_code.py", line 656, in call
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.current_callable(inputs)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_inductor/utils.py", line 3401, in run
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] out = model(new_inputs)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.cache/vllm/torch_compile_cache/torch_aot_compile/7f96116d443cdb5d82062835e1f50f1e3ab2137fa18992e02839c43f3e56a536/inductor_cache/cx/ccxjd4cjkzfsucvgpinz2seuiwuz276n4tisz4iz7atj6xqqu2ux.py", line 2091, in call
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] buf8 = torch.ops.vllm.moe_forward_shared.default(buf5, buf6, buf7, None, arg6_1, 0)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_ops.py", line 865, in call
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self._op(*args, **kwargs)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_compile.py", line 54, in inner
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return disable_fn(*args, **kwargs)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1263, in _fn
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return fn(*args, **kwargs)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 409, in torch_dispatch
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] res = func(*args, **kwargs)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_ops.py", line 865, in call
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self._op(*args, **kwargs)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 134, in _moe_forward_shared
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return layer.runner._forward_impl(
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 795, in _forward_impl
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] shared_output, hidden_states = self._apply_quant_method(
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 538, in _apply_quant_method
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] fused_out = self._quant_method.apply(
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/quantization/fp8.py", line 907, in apply
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.moe_kernel.apply(
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1633, in apply
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.impl.apply(
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1410, in apply
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] fused_out = self.fused_experts(
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1264, in fused_experts
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] self.fused_experts.apply(
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/experts/flashinfer_cutlass_moe.py", line 369, in apply
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] _ = flashinfer_cutlass_fused_moe(
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/utils/flashinfer.py", line 99, in wrapper
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return impl(*args, **kwargs)
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/flashinfer/fused_moe/core.py", line 794, in cutlass_fused_moe
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return get_cutlass_fused_moe_module(device_arch).cutlass_fused_moe(
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/flashinfer/fused_moe/core.py", line 499, in cutlass_fused_moe
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] run_moe(
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "python/tvm_ffi/cython/function.pxi", line 929, in tvm_ffi.core.Function.call
(Worker_TP7_EP7 pid=214275) ERROR 05-21 02:40:18 [multiproc_executor.py:962] RuntimeError: Assertion failed: !cubin.empty() || isPathValid(path)
sh: 1: nvcc: not found
[TensorRT-LLM][ERROR] Warning: Failed to copy kernel files to cache: filesystem error: cannot rename: No such file or directory [/root/.tensorrt_llm/tmp/gemm_4096_6144_128_128_128_32_532_5_1_GroupedWithOffset_1779331218510_999614/nvcc_kernel.cubin] [/root/.tensorrt_llm/cache/gemm_4096_6144_128_128_128_32_532_5_1_GroupedWithOffset/nvcc_kernel.cubin]
Assertion failed: /root/.venv/lib/python3.10/site-packages/flashinfer/data/csrc/nv_internal/tensorrt_llm/deep_gemm/runtime.cuh:63, condition: !cubin.empty() || isPathValid(path)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] WorkerProc hit an exception.
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] Traceback (most recent call last):
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 957, in worker_busy_loop
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] output = func(*args, **kwargs)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return func(*args, **kwargs)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_worker.py", line 392, in determine_available_memory
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] self.model_runner.profile_run()
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5948, in profile_run
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] hidden_states, last_hidden_states = self._dummy_run(
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 124, in decorate_context
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return func(*args, **kwargs)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/v1/worker/gpu_model_runner.py", line 5616, in _dummy_run
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] outputs = self.model(
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/compilation/cuda_graph.py", line 254, in call
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.runnable(*args, **kwargs)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1779, in _wrapped_call_impl
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self._call_impl(*args, **kwargs)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1790, in _call_impl
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return forward_call(*args, **kwargs)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/models/deepseek_v2.py", line 1692, in forward
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] hidden_states = self.model(
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/compilation/decorators.py", line 670, in call
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] output = self.aot_compiled_fn(self, *args, **kwargs)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_dynamo/aot_compile.py", line 224, in call
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.fn(*args, **kwargs)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/models/deepseek_v2.py", line 1271, in forward
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] def forward(
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/compilation/caching.py", line 217, in call
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.optimized_call(*args, **kwargs)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "", line 850, in execution_fn
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/compilation/cuda_graph.py", line 254, in call
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.runnable(*args, **kwargs)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/compilation/piecewise_backend.py", line 380, in call
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return range_entry.runnable(*args)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_inductor/standalone_compile.py", line 122, in call
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self._compiled_fn(*args)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1263, in _fn
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return fn(*args, **kwargs)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/aot_autograd.py", line 1200, in forward
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return compiled_fn(full_args)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 580, in runtime_wrapper
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] all_outs = call_func_at_runtime_with_args(
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/utils.py", line 138, in call_func_at_runtime_with_args
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] out = normalize_as_list(f(args))
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 2298, in call
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.compiled_fn(*args, **kwargs)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 783, in wrapper
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return compiled_fn(runtime_args)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 1011, in inner_fn
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] outs = compiled_fn(args)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_inductor/output_code.py", line 656, in call
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.current_callable(inputs)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_inductor/utils.py", line 3401, in run
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] out = model(new_inputs)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.cache/vllm/torch_compile_cache/torch_aot_compile/7f96116d443cdb5d82062835e1f50f1e3ab2137fa18992e02839c43f3e56a536/inductor_cache/ws/cws6w65l23x6x24syyraynid3kpvoyju2jtfdbsh5jcswd4vlmw4.py", line 2091, in call
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] buf8 = torch.ops.vllm.moe_forward_shared.default(buf5, buf6, buf7, None, arg6_1, 0)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_ops.py", line 865, in call
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self._op(*args, **kwargs)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_compile.py", line 54, in inner
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return disable_fn(*args, **kwargs)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_dynamo/eval_frame.py", line 1263, in _fn
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return fn(*args, **kwargs)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_functorch/_aot_autograd/runtime_wrappers.py", line 409, in torch_dispatch
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] res = func(*args, **kwargs)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/torch/_ops.py", line 865, in call
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self._op(*args, **kwargs)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 134, in _moe_forward_shared
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return layer.runner._forward_impl(
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 795, in _forward_impl
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] shared_output, hidden_states = self._apply_quant_method(
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/runner/moe_runner.py", line 538, in _apply_quant_method
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] fused_out = self._quant_method.apply(
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/quantization/fp8.py", line 907, in apply
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.moe_kernel.apply(
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1633, in apply
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return self.impl.apply(
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1410, in apply
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] fused_out = self._fused_experts(
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/modular_kernel.py", line 1264, in fused_experts
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] self.fused_experts.apply(
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/model_executor/layers/fused_moe/experts/flashinfer_cutlass_moe.py", line 369, in apply
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] _ = flashinfer_cutlass_fused_moe(
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/vllm/utils/flashinfer.py", line 99, in wrapper
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return impl(*args, **kwargs)
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/flashinfer/fused_moe/core.py", line 794, in cutlass_fused_moe
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] return get_cutlass_fused_moe_module(device_arch).cutlass_fused_moe(
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "/root/.venv/lib/python3.10/site-packages/flashinfer/fused_moe/core.py", line 499, in cutlass_fused_moe
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] run_moe(
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] File "python/tvm_ffi/cython/function.pxi", line 929, in tvm_ffi.core.Function.call
(Worker_TP0_EP0 pid=214136) ERROR 05-21 02:40:18 [multiproc_executor.py:962] RuntimeError: Assertion failed: !cubin.empty() || isPathValid(path)
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] EngineCore failed to start.
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] Traceback (most recent call last):
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] File "/root/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1114, in run_engine_core
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] File "/root/.venv/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] File "/root/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 880, in init
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] super().init(
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] File "/root/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 128, in init
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] File "/root/.venv/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] return func(*args, **kwargs)
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] File "/root/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] File "/root/.venv/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] return self.collective_rpc("determine_available_memory")
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] File "/root/.venv/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 403, in collective_rpc
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] return future if non_block else future.result()
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] File "/root/.venv/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 90, in result
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] return super().result()
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] return self.__get_result()
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] raise self._exception
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] File "/root/.venv/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 94, in wait_for_response
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] response = self.aggregate(self.get_response())
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] File "/root/.venv/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 390, in get_response
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] raise RuntimeError(
(EngineCore pid=214116) ERROR 05-21 02:40:18 [core.py:1140] RuntimeError: Worker failed with error 'Assertion failed: !cubin.empty() || isPathValid(path)', please check the stack trace above for the root cause
(Worker_TP0_EP0 pid=214136) WARNING 05-21 02:40:18 [multiproc_executor.py:884] WorkerProc was terminated
(Worker_TP6_EP6 pid=214255) WARNING 05-21 02:40:18 [multiproc_executor.py:884] WorkerProc was terminated
(Worker_TP7_EP7 pid=214275) WARNING 05-21 02:40:18 [multiproc_executor.py:884] WorkerProc was terminated
(Worker_TP5_EP5 pid=214235) WARNING 05-21 02:40:18 [multiproc_executor.py:884] WorkerProc was terminated
(Worker_TP3_EP3 pid=214189) WARNING 05-21 02:40:18 [multiproc_executor.py:884] WorkerProc was terminated
(Worker_TP1_EP1 pid=214154) WARNING 05-21 02:40:18 [multiproc_executor.py:884] WorkerProc was terminated
(Worker_TP4_EP4 pid=214212) WARNING 05-21 02:40:18 [multiproc_executor.py:884] WorkerProc was terminated
(Worker_TP2_EP2 pid=214172) WARNING 05-21 02:40:18 [multiproc_executor.py:884] WorkerProc was terminated
(EngineCore pid=214116) ERROR 05-21 02:40:23 [multiproc_executor.py:283] Worker proc VllmWorker-2 died unexpectedly, shutting down executor.
(EngineCore pid=214116) Process EngineCore:
(EngineCore pid=214116) Traceback (most recent call last):
(EngineCore pid=214116) File "/usr/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=214116) self.run()
(EngineCore pid=214116) File "/usr/lib/python3.10/multiprocessing/process.py", line 108, in run
(EngineCore pid=214116) self._target(*self._args, **self._kwargs)
(EngineCore pid=214116) File "/root/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1144, in run_engine_core
(EngineCore pid=214116) raise e
(EngineCore pid=214116) File "/root/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 1114, in run_engine_core
(EngineCore pid=214116) engine_core = EngineCoreProc(*args, engine_index=dp_rank, **kwargs)
(EngineCore pid=214116) File "/root/.venv/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=214116) return func(*args, **kwargs)
(EngineCore pid=214116) File "/root/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 880, in init
(EngineCore pid=214116) super().init(
(EngineCore pid=214116) File "/root/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 128, in init
(EngineCore pid=214116) kv_cache_config = self._initialize_kv_caches(vllm_config)
(EngineCore pid=214116) File "/root/.venv/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(EngineCore pid=214116) return func(*args, **kwargs)
(EngineCore pid=214116) File "/root/.venv/lib/python3.10/site-packages/vllm/v1/engine/core.py", line 250, in _initialize_kv_caches
(EngineCore pid=214116) available_gpu_memory = self.model_executor.determine_available_memory()
(EngineCore pid=214116) File "/root/.venv/lib/python3.10/site-packages/vllm/v1/executor/abstract.py", line 147, in determine_available_memory
(EngineCore pid=214116) return self.collective_rpc("determine_available_memory")
(EngineCore pid=214116) File "/root/.venv/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 403, in collective_rpc
(EngineCore pid=214116) return future if non_block else future.result()
(EngineCore pid=214116) File "/root/.venv/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 90, in result
(EngineCore pid=214116) return super().result()
(EngineCore pid=214116) File "/usr/lib/python3.10/concurrent/futures/_base.py", line 451, in result
(EngineCore pid=214116) return self.__get_result()
(EngineCore pid=214116) File "/usr/lib/python3.10/concurrent/futures/_base.py", line 403, in __get_result
(EngineCore pid=214116) raise self._exception
(EngineCore pid=214116) File "/root/.venv/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 94, in wait_for_response
(EngineCore pid=214116) response = self.aggregate(self.get_response())
(EngineCore pid=214116) File "/root/.venv/lib/python3.10/site-packages/vllm/v1/executor/multiproc_executor.py", line 390, in get_response
(EngineCore pid=214116) raise RuntimeError(
(EngineCore pid=214116) RuntimeError: Worker failed with error 'Assertion failed: !cubin.empty() || isPathValid(path)', please check the stack trace above for the root cause
(APIServer pid=213847) Traceback (most recent call last):
(APIServer pid=213847) File "/root/.venv/bin/vllm", line 10, in
(APIServer pid=213847) sys.exit(main())
(APIServer pid=213847) File "/root/.venv/lib/python3.10/site-packages/vllm/entrypoints/cli/main.py", line 92, in main
(APIServer pid=213847) args.dispatch_function(args)
(APIServer pid=213847) File "/root/.venv/lib/python3.10/site-packages/vllm/entrypoints/cli/serve.py", line 122, in cmd
(APIServer pid=213847) uvloop.run(run_server(args))
(APIServer pid=213847) File "/root/.venv/lib/python3.10/site-packages/uvloop/init.py", line 69, in run
(APIServer pid=213847) return loop.run_until_complete(wrapper())
(APIServer pid=213847) File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
(APIServer pid=213847) File "/root/.venv/lib/python3.10/site-packages/uvloop/init.py", line 48, in wrapper
(APIServer pid=213847) return await main
(APIServer pid=213847) File "/root/.venv/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 693, in run_server
(APIServer pid=213847) await run_server_worker(listen_address, sock, args, **uvicorn_kwargs)
(APIServer pid=213847) File "/root/.venv/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 707, in run_server_worker
(APIServer pid=213847) async with build_async_engine_client(
(APIServer pid=213847) File "/usr/lib/python3.10/contextlib.py", line 199, in aenter
(APIServer pid=213847) return await anext(self.gen)
(APIServer pid=213847) File "/root/.venv/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 100, in build_async_engine_client
(APIServer pid=213847) async with build_async_engine_client_from_engine_args(
(APIServer pid=213847) File "/usr/lib/python3.10/contextlib.py", line 199, in aenter
(APIServer pid=213847) return await anext(self.gen)
(APIServer pid=213847) File "/root/.venv/lib/python3.10/site-packages/vllm/entrypoints/openai/api_server.py", line 136, in build_async_engine_client_from_engine_args
(APIServer pid=213847) async_llm = AsyncLLM.from_vllm_config(
(APIServer pid=213847) File "/root/.venv/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 217, in from_vllm_config
(APIServer pid=213847) return cls(
(APIServer pid=213847) File "/root/.venv/lib/python3.10/site-packages/vllm/v1/engine/async_llm.py", line 146, in init
(APIServer pid=213847) self.engine_core = EngineCoreClient.make_async_mp_client(
(APIServer pid=213847) File "/root/.venv/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=213847) return func(*args, **kwargs)
(APIServer pid=213847) File "/root/.venv/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 130, in make_async_mp_client
(APIServer pid=213847) return AsyncMPClient(*client_args)
(APIServer pid=213847) File "/root/.venv/lib/python3.10/site-packages/vllm/tracing/otel.py", line 178, in sync_wrapper
(APIServer pid=213847) return func(*args, **kwargs)
(APIServer pid=213847) File "/root/.venv/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 900, in init
(APIServer pid=213847) super().init(
(APIServer pid=213847) File "/root/.venv/lib/python3.10/site-packages/vllm/v1/engine/core_client.py", line 535, in init
(APIServer pid=213847) with launch_core_engines(
(APIServer pid=213847) File "/usr/lib/python3.10/contextlib.py", line 142, in exit
(APIServer pid=213847) next(self.gen)
(APIServer pid=213847) File "/root/.venv/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 1128, in launch_core_engines
(APIServer pid=213847) wait_for_engine_startup(
(APIServer pid=213847) File "/root/.venv/lib/python3.10/site-packages/vllm/v1/engine/utils.py", line 1187, in wait_for_engine_startup
(APIServer pid=213847) raise RuntimeError(
(APIServer pid=213847) RuntimeError: Engine core initialization failed. See root cause above. Failed core proc(s): {}
/usr/lib/python3.10/multiprocessing/resource_tracker.py:224: UserWarning: resource_tracker: There appear to be 8 leaked shared_memory objects to clean up at shutdown
warnings.warn('resource_tracker: There appear to be %d '
Before submitting...
@yiminghub2024 commented on GitHub (May 21, 2026):
@LujhCoconut commented on GitHub (May 21, 2026):
After reviewing your error logs, I believe this is not a Mooncake issue. The root cause is that
nvccis not found in your environment.Because nvcc failed to execute, the source file does not exist, which in turn caused the .cubin compilation step to fail.
The solution is to install the CUDA toolkit and ensure that nvcc is in your PATH. If you run into any further issues, feel free to follow up.
@yiminghub2024 commented on GitHub (May 21, 2026):
as your said , i add env path of nvcc ,vllm started success ,but when i use vllm bench serve test ,vllm clash , why ?
my test command :uv run vllm bench serve
--backend vllm
--base-url http://10.133.72.178:40036
--model /data/cache/model_scope/ZhipuAI/GLM-5-FP8
--served-model-name glm-5-fp8-copy
--dataset-name random
--random-input-len 16384
--random-output-len 16384
--random-range-ratio 0.0
--random-prefix-len 13107
--num-prompts 100
--request-rate inf
--max-concurrency 100
clash log
测试的时候 VLLM引擎挂了 (APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for prompt_idx, res in result_generator:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/utils/async_utils.py", line 283, in merge_async_iterators
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] async for item in iterators[0]:
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 579, in generate
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] out = q.get_nowait() or await q.get()
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] ^^^^^^^^^^^^^
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/output_processor.py", line 88, in get
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] raise output
(APIServer pid=4591) ERROR 05-21 03:40:21 [serving.py:468] File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/completion/serving.py", line 302, in completion_stream_generator
(APIServer pid=45
@LujhCoconut please help thanks
@yiminghub2024 commented on GitHub (May 21, 2026):
WARNING: vllm bench serve no longer sets temperature==0 (greedy) in requests by default. The default will be determined on the server side and can be model/API specific. For the old behavior, include --temperature=0.
Starting initial single prompt test run...
(Worker_TP0_EP0 pid=24134) WARNING 05-21 04:01:22 [jit_monitor.py:103] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(Worker_TP0_EP0 pid=24134) WARNING 05-21 04:01:22 [jit_monitor.py:103] Triton kernel JIT compilation during inference: _build_prefill_chunk_metadata_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(APIServer pid=23539) INFO: 10.133.72.178:58260 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=23539) INFO: 10.133.72.178:58270 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=23539) INFO: 10.133.72.178:58276 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=23539) INFO: 10.133.72.178:58286 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=23539) INFO: 10.133.72.178:58302 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=23539) INFO: 10.133.72.178:58314 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=23539) INFO: 10.133.72.178:58326 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=23539) INFO: 10.133.72.178:58268 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=23539) INFO: 10.133.72.178:58318 - "POST /v1/completions HTTP/1.1" 200 OK
(Worker_TP0_EP0 pid=24134) WARNING 05-21 04:01:27 [jit_monitor.py:103] Triton kernel JIT compilation during inference: eagle_prepare_next_token_padded_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
!!!!!!! Segfault encountered !!!!!!!
File "/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.9.linux-amd64/src/runtime/sys_linux_amd64.s", line 330, in runtime.sigfwd
!!!!!!! Segfault encountered !!!!!!!
File "/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.9.linux-amd64/src/runtime/sys_linux_amd64.s", line 330, in runtime.sigfwd
!!!!!!! Segfault encountered !!!!!!!
File "/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.9.linux-amd64/src/runtime/sys_linux_amd64.s", line 330, in runtime.sigfwd
!!!!!!! Segfault encountered !!!!!!!
File "/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.9.linux-amd64/src/runtime/sys_linux_amd64.s", line 330, in runtime.sigfwd
!!!!!!! Segfault encountered !!!!!!!
File "/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.9.linux-amd64/src/runtime/sys_linux_amd64.s", line 330, in runtime.sigfwd
!!!!!!! Segfault encountered !!!!!!!
File "/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.9.linux-amd64/src/runtime/sys_linux_amd64.s", line 330, in runtime.sigfwd
!!!!!!! Segfault encountered !!!!!!!
File "/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.9.linux-amd64/src/runtime/sys_linux_amd64.s", line 330, in runtime.sigfwd
!!!!!!! Segfault encountered !!!!!!!
File "/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.9.linux-amd64/src/runtime/sys_linux_amd64.s", line 330, in runtime.sigfwd
(EngineCore pid=23935) ERROR 05-21 04:01:29 [multiproc_executor.py:283] Worker proc VllmWorker-5 died unexpectedly, shutting down executor.
(EngineCore pid=23935) ERROR 05-21 04:01:32 [dump_input.py:72] Dumping input data for V1 LLM engine (v0.21.0) with config: model='/data/cache/model_scope/ZhipuAI/GLM-5-FP8', speculative_config=SpeculativeConfig(method='mtp', model='/data/cache/model_scope/ZhipuAI/GLM-5-FP8', num_spec_tokens=3), tokenizer='/data/cache/model_scope/ZhipuAI/GLM-5-FP8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=200000, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=fp8, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='glm45', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=glm-5-fp8-copy, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+quant_fp8', 'none', '+quant_fp8', '+quant_fp8', '+quant_fp8'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [42, 42, 16384], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=False, moe_backend='auto'),
(EngineCore pid=23935) ERROR 05-21 04:01:32 [dump_input.py:79] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=cmpl-bench-d351196a-0-0-9cce4f81,prompt_token_ids_len=29491,prefill_token_ids_len=None,mm_features=[],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=0.95, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], thinking_token_budget=None, include_stop_str_in_output=False, ignore_eos=True, max_tokens=16384, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, structured_outputs=None, extra_args=None),block_ids=([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256],),num_computed_tokens=0,lora_request=None,prompt_embeds_shape=None)], scheduled_cached_reqs=CachedRequestData(req_ids=[],resumed_req_ids=set(),new_token_ids_lens=[],all_token_ids_lens={},new_block_ids=[],num_computed_tokens=[],num_output_tokens=[]), num_scheduled_tokens={cmpl-bench-d351196a-0-0-9cce4f81: 16384}, total_num_scheduled_tokens=16384, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[256], finished_req_ids=[], free_encoder_mm_hashes=[], preempted_req_ids=[], has_structured_output_requests=false, pending_structured_output_tokens=false, num_invalid_spec_tokens=null, kv_connector_metadata=MooncakeStoreConnectorMetadata(requests=[ReqMeta(req_id='cmpl-bench-d351196a-0-0-9cce4f81', token_len_chunk=16384, block_ids=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201, 202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215, 216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229, 230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243, 244, 245, 246, 247, 248, 249, 250, 251, 252, 25@@@ @LujhCoconut
@LujhCoconut commented on GitHub (May 21, 2026):
@yiminghub2024 Could you help try reducing the concurrency pressure a bit? I suspect it might be a thread-safety issue under concurrent access.
such as
@yiminghub2024 commented on GitHub (May 21, 2026):
the same error after a very small request
(root) root@hzhindmc178x:~# uv run vllm bench serve
--backend vllm
--base-url http://10.133.72.178:40036
--model /data/cache/model_scope/ZhipuAI/GLM-5-FP8
--served-model-name glm-5-fp8-copy
--dataset-name random
--random-input-len 4096
--random-output-len 4096
--random-range-ratio 0.0
--random-prefix-len 2048
--num-prompts 5
--request-rate inf
--max-concurrency 5
Namespace(subparser='bench', bench_type='serve', dispatch_function=<function BenchmarkServingSubcommand.cmd at 0x7f12b1145b40>, trust_remote_code=False, seed=0, num_prompts=5, dataset_name='random', no_stream=False, dataset_path=None, no_oversample=False, skip_chat_template=False, enable_multimodal_chat=False, disable_shuffle=False, custom_output_len=256, spec_bench_output_len=256, spec_bench_category=None, sonnet_input_len=550, sonnet_output_len=150, sonnet_prefix_len=200, sharegpt_output_len=None, blazedit_min_distance=0.0, blazedit_max_distance=1.0, asr_max_audio_len_sec=inf, asr_min_audio_len_sec=0.0, random_input_len=4096, random_output_len=4096, random_range_ratio='0.0', random_prefix_len=2048, random_batch_size=1, no_reranker=False, random_mm_base_items_per_request=1, random_mm_num_mm_items_range_ratio=0.0, random_mm_limit_mm_per_prompt={'image': 255, 'video': 1}, random_mm_bucket_config={(256, 256, 1): 0.5, (720, 1280, 1): 0.5, (720, 1280, 16): 0.0}, hf_subset=None, hf_split=None, hf_name=None, hf_output_len=None, prefix_repetition_prefix_len=256, prefix_repetition_suffix_len=256, prefix_repetition_num_prefixes=10, prefix_repetition_output_len=128, speed_bench_dataset_subset='qualitative', speed_bench_output_len=4096, speed_bench_category=None, label=None, backend='vllm', base_url='http://10.133.72.178:40036', host='127.0.0.1', port=8000, endpoint='/v1/completions', header=None, max_concurrency=5, model='/data/cache/model_scope/ZhipuAI/GLM-5-FP8', input_len=None, output_len=None, tokenizer=None, tokenizer_mode='auto', use_beam_search=False, logprobs=None, request_rate=inf, burstiness=1.0, disable_tqdm=False, num_warmups=0, profile=False, save_result=False, save_detailed=False, append_result=False, metadata=None, result_dir=None, result_filename=None, ignore_eos=False, percentile_metrics=None, metric_percentiles='99', goodput=None, request_id_prefix='bench-23a07ded-', top_p=None, top_k=None, min_p=None, temperature=None, frequency_penalty=None, presence_penalty=None, repetition_penalty=None, served_model_name='glm-5-fp8-copy', lora_modules=None, lora_assignment='random', ramp_up_strategy=None, ramp_up_start_rps=None, ramp_up_end_rps=None, ready_check_timeout_sec=0, extra_body=None, skip_tokenizer_init=False, insecure=False, plot_timeline=False, timeline_itl_thresholds='25,50', plot_dataset_stats=False)
INFO 05-21 04:18:29 [utils.py:90] Sampling input_len from [4096, 4096] and output_len from [4096, 4096]
WARNING: vllm bench serve no longer sets temperature==0 (greedy) in requests by default. The default will be determined on the server side and can be model/API specific. For the old behavior, include --temperature=0.
Starting initial single prompt test run...
Skipping endpoint ready check.
Starting main benchmark run...
Traffic request rate: inf
Burstiness factor: 1.0 (Poisson process)
Maximum request concurrency: 5
100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 5/5 [00:07<00:00, 1.51s/it]
/root/.venv/lib/python3.10/site-packages/vllm/benchmarks/serve.py:917: UserWarning: All requests failed. This is likely due to a misconfiguration on the benchmark arguments.
metrics, actual_output_lens = calculate_metrics(
Failed requests during benchmark run detected (capping to 10):
Error 0: Never received a valid chunk to calculate TTFT.This response will be marked as failed!
Error 1: Never received a valid chunk to calculate TTFT.This response will be marked as failed!
Error 2: Never received a valid chunk to calculate TTFT.This response will be marked as failed!
Error 3: Never received a valid chunk to calculate TTFT.This response will be marked as failed!
Error 4: Never received a valid chunk to calculate TTFT.This response will be marked as failed!
============ Serving Benchmark Result ============
Successful requests: 0
Failed requests: 5
Maximum request concurrency: 5
Benchmark duration (s): 7.54
Total input tokens: 0
Total generated tokens: 0
Request throughput (req/s): 0.00
Output token throughput (tok/s): 0.00
Peak output token throughput (tok/s): 0.00
Peak concurrent requests: 0.00
Total token throughput (tok/s): 0.00
---------------Time to First Token----------------
Mean TTFT (ms): 0.00
Median TTFT (ms): 0.00
P99 TTFT (ms): 0.00
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms): 0.00
Median TPOT (ms): 0.00
P99 TPOT (ms): 0.00
---------------Inter-token Latency----------------
Mean ITL (ms): 0.00
Median ITL (ms): 0.00
P99 ITL (ms): 0.00
same error
(APIServer pid=26618) INFO 05-21 04:17:28 [launcher.py:46] Route: /v1/chat/completions/render, Methods: POST
(APIServer pid=26618) INFO 05-21 04:17:28 [launcher.py:46] Route: /v1/completions/render, Methods: POST
(APIServer pid=26618) INFO: Started server process [26618]
(APIServer pid=26618) INFO: Waiting for application startup.
(APIServer pid=26618) INFO: Application startup complete.
(APIServer pid=26618) INFO: 10.133.72.178:45986 - "GET /metrics HTTP/1.1" 200 OK
(APIServer pid=26618) INFO: 10.133.72.178:46000 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=26618) INFO: 10.133.72.178:46016 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=26618) INFO: 10.133.72.178:46030 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=26618) INFO: 10.133.72.178:45986 - "POST /v1/completions HTTP/1.1" 200 OK
(APIServer pid=26618) INFO: 10.133.72.178:46040 - "POST /v1/completions HTTP/1.1" 200 OK
(Worker_TP0_EP0 pid=27213) WARNING 05-21 04:18:29 [jit_monitor.py:103] Triton kernel JIT compilation during inference: _compute_slot_mapping_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(Worker_TP0_EP0 pid=27213) WARNING 05-21 04:18:29 [jit_monitor.py:103] Triton kernel JIT compilation during inference: _build_prefill_chunk_metadata_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(Worker_TP0_EP0 pid=27213) WARNING 05-21 04:18:31 [jit_monitor.py:103] Triton kernel JIT compilation during inference: eagle_prepare_next_token_padded_kernel. This causes a latency spike; consider extending warmup to cover this shape/config.
(Worker_TP0_EP0 pid=27213) WARNING 05-21 04:18:31 [op.py:290] Priority not set for op rms_norm, using native implementation.
!!!!!!! Segfault encountered !!!!!!!
File "/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.9.linux-amd64/src/runtime/sys_linux_amd64.s", line 330, in runtime.sigfwd
!!!!!!! Segfault encountered !!!!!!!
File "/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.9.linux-amd64/src/runtime/sys_linux_amd64.s", line 330, in runtime.sigfwd
!!!!!!! Segfault encountered !!!!!!!
File "/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.9.linux-amd64/src/runtime/sys_linux_amd64.s", line 330, in runtime.sigfwd
!!!!!!! Segfault encountered !!!!!!!
File "/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.9.linux-amd64/src/runtime/sys_linux_amd64.s", line 330, in runtime.sigfwd
!!!!!!! Segfault encountered !!!!!!!
File "/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.9.linux-amd64/src/runtime/sys_linux_amd64.s", line 330, in runtime.sigfwd
!!!!!!! Segfault encountered !!!!!!!
File "/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.9.linux-amd64/src/runtime/sys_linux_amd64.s", line 330, in runtime.sigfwd
!!!!!!! Segfault encountered !!!!!!!
File "/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.9.linux-amd64/src/runtime/sys_linux_amd64.s", line 330, in runtime.sigfwd
!!!!!!! Segfault encountered !!!!!!!
File "/home/runner/go/pkg/mod/golang.org/toolchain@v0.0.1-go1.25.9.linux-amd64/src/runtime/sys_linux_amd64.s", line 330, in runtime.sigfwd
(EngineCore pid=27014) ERROR 05-21 04:18:33 [multiproc_executor.py:283] Worker proc VllmWorker-4 died unexpectedly, shutting down executor.
(EngineCore pid=27014) ERROR 05-21 04:18:37 [dump_input.py:72] Dumping input data for V1 LLM engine (v0.21.0) with config: model='/data/cache/model_scope/ZhipuAI/GLM-5-FP8', speculative_config=SpeculativeConfig(method='mtp', model='/data/cache/model_scope/ZhipuAI/GLM-5-FP8', num_spec_tokens=3), tokenizer='/data/cache/model_scope/ZhipuAI/GLM-5-FP8', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=200000, download_dir=None, load_format=auto, tensor_parallel_size=8, pipeline_parallel_size=1, data_parallel_size=1, decode_context_parallel_size=1, dcp_comm_backend=ag_rs, disable_custom_all_reduce=False, quantization=fp8, quantization_config=None, enforce_eager=False, enable_return_routed_experts=False, kv_cache_dtype=auto, device_config=cuda, structured_outputs_config=StructuredOutputsConfig(backend='auto', disable_any_whitespace=False, disable_additional_properties=False, reasoning_parser='glm45', reasoning_parser_plugin='', enable_in_reasoning=False), observability_config=ObservabilityConfig(show_hidden_metrics_for_version=None, otlp_traces_endpoint=None, collect_detailed_traces=None, kv_cache_metrics=False, kv_cache_metrics_sample=0.01, cudagraph_metrics=False, enable_layerwise_nvtx_tracing=False, enable_mfu_metrics=False, enable_mm_processor_stats=False, enable_logging_iteration_details=False), seed=0, served_model_name=glm-5-fp8-copy, enable_prefix_caching=True, enable_chunked_prefill=True, pooler_config=None, compilation_config={'mode': <CompilationMode.VLLM_COMPILE: 3>, 'debug_dump_path': None, 'cache_dir': '', 'compile_cache_save_format': 'binary', 'backend': 'inductor', 'custom_ops': ['+quant_fp8', 'none', '+quant_fp8', '+quant_fp8', '+quant_fp8'], 'ir_enable_torch_wrap': True, 'splitting_ops': ['vllm::unified_attention_with_output', 'vllm::unified_mla_attention_with_output', 'vllm::mamba_mixer2', 'vllm::mamba_mixer', 'vllm::short_conv', 'vllm::linear_attention', 'vllm::plamo2_mamba_mixer', 'vllm::gdn_attention_core', 'vllm::gdn_attention_core_xpu', 'vllm::olmo_hybrid_gdn_full_forward', 'vllm::kda_attention', 'vllm::sparse_attn_indexer', 'vllm::rocm_aiter_sparse_attn_indexer', 'vllm::deepseek_v4_attention', 'vllm::unified_kv_cache_update', 'vllm::unified_mla_kv_cache_update'], 'compile_mm_encoder': False, 'cudagraph_mm_encoder': False, 'encoder_cudagraph_token_budgets': [], 'encoder_cudagraph_max_vision_items_per_batch': 0, 'encoder_cudagraph_max_frames_per_batch': None, 'compile_sizes': [], 'compile_ranges_endpoints': [42, 42, 16384], 'inductor_compile_config': {'enable_auto_functionalized_v2': False, 'size_asserts': False, 'alignment_asserts': False, 'scalar_asserts': False, 'combo_kernels': True, 'benchmark_combo_kernel': True}, 'inductor_passes': {}, 'cudagraph_mode': <CUDAGraphMode.FULL_AND_PIECEWISE: (2, 1)>, 'cudagraph_num_of_warmups': 1, 'cudagraph_capture_sizes': [1, 2, 4, 8, 16, 24, 32, 40, 48, 56, 64, 72, 80, 88, 96, 104, 112, 120, 128, 136, 144, 152, 160, 168, 176, 184, 192, 200, 208, 216, 224, 232, 240, 248, 256, 272, 288, 304, 320, 336, 352, 368, 384, 400, 416, 432, 448, 464, 480, 496, 512], 'cudagraph_copy_inputs': False, 'cudagraph_specialize_lora': True, 'use_inductor_graph_partition': False, 'pass_config': {'fuse_norm_quant': True, 'fuse_act_quant': True, 'fuse_attn_quant': False, 'enable_sp': False, 'fuse_gemm_comms': False, 'fuse_allreduce_rms': True, 'fuse_act_padding': False}, 'max_cudagraph_capture_size': 512, 'dynamic_shapes_config': {'type': <DynamicShapesType.BACKED: 'backed'>, 'evaluate_guards': False, 'assume_32_bit_indexing': False}, 'local_cache_dir': None, 'fast_moe_cold_start': False, 'static_all_moe_layers': []}, kernel_config=KernelConfig(ir_op_priority=IrOpPriorityConfig(rms_norm=['native'], fused_add_rms_norm=['native']), enable_flashinfer_autotune=False, moe_backend='auto'),
(EngineCore pid=27014) ERROR 05-21 04:18:37 [dump_input.py:79] Dumping scheduler output for model execution: SchedulerOutput(scheduled_new_reqs=[NewRequestData(req_id=cmpl-bench-23a07ded-1-0-be952c27,prompt_token_ids_len=6144,prefill_token_ids_len=None,mm_features=[],sampling_params=SamplingParams(n=1, presence_penalty=0.0, frequency_penalty=0.0, repetition_penalty=1.0, temperature=1.0, top_p=0.95, top_k=0, min_p=0.0, seed=None, stop=[], stop_token_ids=[], bad_words=[], thinking_token_budget=None, include_stop_str_in_output=False, ignore_eos=True, max_tokens=4096, min_tokens=0, logprobs=None, prompt_logprobs=None, skip_special_tokens=True, spaces_between_special_tokens=True, structured_outputs=None, extra_args=None),block_ids=([1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96],),num_computed_tokens=0,lora_request=None,prompt_embeds_shape=None)], scheduled_cached_reqs=CachedRequestData(req_ids=[],resumed_req_ids=set(),new_token_ids_lens=[],all_token_ids_lens={},new_block_ids=[],num_computed_tokens=[],num_output_tokens=[]), num_scheduled_tokens={cmpl-bench-23a07ded-1-0-be952c27: 6144}, total_num_scheduled_tokens=6144, scheduled_spec_decode_tokens={}, scheduled_encoder_inputs={}, num_common_prefix_blocks=[96], finished_req_ids=[], free_encoder_mm_hashes=[], preempted_req_ids=[], has_structured_output_requests=false, pending_structured_output_tokens=false, num_invalid_spec_tokens=null, kv_connector_metadata=MooncakeStoreConnectorMetadata(requests=[ReqMeta(req_id='cmpl-bench-23a07ded-1-0-be952c27', token_len_chunk=6144, block_ids=[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96], block_hashes=[b'}C\x81\xde?K\x7f\x0b7\xef\x0bw\xcf-\xb7\x94R\x89\xf8b\x84\xd7tR\xae1\xbdj5\xb4\xb7p', b'\xd5F\xbe\xc0\xd4\x02\xce6\xbc
\xb2\x17\xb9\xe1\x0e\x87\xb5\x94\xe1\x91\x91\xa9{V\x1cw\xfb\x1c\x1e\xf2\xf3~', b'\xdc\x86\xbc%_\xf6\x9b]=;\xb9\xe6@&G\x04\x1e\xa6#\xd5,;)\x1e\xd1\x1f\x0e3\x0b\xf7\xe0\xaf', b'(\xe7f\xd7fo|\xfa9\xb1l\xadu\xd7\x10eSQ)J\x91\xbf\x15\x83\xeeF\x10G\x8f\x9b\xce\xef', b"\\\xa4\x9b'[8\x07\xdf.\xd6v\xbd\xdeW\x80&v\xdch\x7f\x8f\xb5L\xef\xca\x1eD\x13\xcdw\xd3", b'\xbb\xfe\xb4{\xba\xabS\xa3\xe0\xe2;M\x90t\xf8l\xe9\xa3\xf6\r\x9c9-\x95\rt\xecmY^\xb6\xd6', b'\xfa\xd5\xcc3M\xfd\x8fk\xca\x0cIR\xa4\xab\x97fF\x08\xac\xc4L\n\xa2\x0b\xb2*\xa4\x06\x1b\\xc3\x1d', b'\xea\xcf\x11\r\x9b\x95g\x8d9\xd3\xd5sj\xb3[\x184\xb2\xc6 I\x92\xaa\xeeS\\x89t\x99\x88_\x00', b'A\x9dm\x0c\x03\xeb3\xcf\xd4\x92\x81\x07q\xffmi{\xbc\x86\xebI\xfc\x8a\xd5I~\xbf\x1fS?\xca\xc3', b'\xbduG\xa3\xe5#\xc3T\xccAS\xe0*\xd3hU\x85\x1e\xffWL\x8e\x86\x03@@@(EngineCore pid=27014) ERROR 05-21 04:18:37 [dump_input.py:81] Dumping scheduler stats: SchedulerStats(num_running_reqs=5, num_waiting_reqs=0, num_skipped_waiting_reqs=0, step_counter=0, current_wave=0, kv_cache_usage=0.08075955159002512, prefix_cache_stats=PrefixCacheStats(reset=False, requests=5, queries=30720, hits=7872, preempted_requests=0, preempted_queries=0, preempted_hits=0), connector_prefix_cache_stats=PrefixCacheStats(reset=False, requests=5, queries=22848, hits=0, preempted_requests=0, preempted_queries=0, preempted_hits=0), kv_cache_eviction_events=[], spec_decoding_stats=None, kv_connector_stats=None, waiting_lora_adapters={}, running_lora_adapters={}, cudagraph_stats=None, perf_stats=None)
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] EngineCore encountered a fatal error.^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] Traceback (most recent call last):^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1133, in run_engine_core^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] engine_core.run_busy_loop()^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1174, in run_busy_loop^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] self._process_engine_step()^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1213, in _process_engine_step^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] outputs, model_executed = self.step_fn()^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] ^^^^^^^^^^^^^^^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 525, in step_with_batch_queue^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] model_output = future.result()^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] ^^^^^^^^^^^^^^^^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 90, in result^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] return super().result()^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] ^^^^^^^^^^^^^^^^^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] return self.__get_result()^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] ^^^^^^^^^^^^^^^^^^^^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] raise self._exception^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 94, in _wait_for_response^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] response = self.aggregate(self.get_response())^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] ^^^^^^^^^^^^^^^^^^^^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 386, in get_response^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] status, result = mq.dequeue(timeout=dequeue_timeout)^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 755, in dequeue^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] with self.acquire_read(timeout, indefinite) as buf:^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] File "/usr/lib/python3.12/contextlib.py", line 137, in enter^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] return next(self.gen)^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] ^^^^^^^^^^^^^^^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] File "/usr/local/lib/python3.12/dist-packages/vllm/distributed/device_communicators/shm_broadcast.py", line 677, in acquire_read^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] raise RuntimeError("cancelled")^M
(EngineCore pid=27014) ERROR 05-21 04:18:37 [core.py:1142] RuntimeError: cancelled
(EngineCore pid=27014) Process EngineCore:
(EngineCore pid=27014) Traceback (most recent call last):
(APIServer pid=26618) ERROR 05-21 04:18:37 [async_llm.py:704] AsyncLLM output_handler failed.^M
(APIServer pid=26618) ERROR 05-21 04:18:37 [async_llm.py:704] Traceback (most recent call last):^M
(APIServer pid=26618) ERROR 05-21 04:18:37 [async_llm.py:704] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/async_llm.py", line 660, in output_handler^M
(APIServer pid=26618) ERROR 05-21 04:18:37 [async_llm.py:704] outputs = await engine_core.get_output_async()^M
(APIServer pid=26618) ERROR 05-21 04:18:37 [async_llm.py:704] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^M
(APIServer pid=26618) ERROR 05-21 04:18:37 [async_llm.py:704] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core_client.py", line 998, in get_output_async^M
(APIServer pid=26618) ERROR 05-21 04:18:37 [async_llm.py:704] raise self._format_exception(outputs) from None^M
(APIServer pid=26618) ERROR 05-21 04:18:37 [async_llm.py:704] vllm.v1.engine.exceptions.EngineDeadError: EngineCore encountered an issue. See stack trace (above) for the root cause.
(EngineCore pid=27014) File "/usr/lib/python3.12/multiprocessing/process.py", line 314, in _bootstrap
(EngineCore pid=27014) self.run()
(EngineCore pid=27014) File "/usr/lib/python3.12/multiprocessing/process.py", line 108, in run
(EngineCore pid=27014) self._target(*self._args, **self._kwargs)
(EngineCore pid=27014) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1144, in run_engine_core
(EngineCore pid=27014) raise e
(EngineCore pid=27014) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1133, in run_engine_core
(EngineCore pid=27014) engine_core.run_busy_loop()
(EngineCore pid=27014) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1174, in run_busy_loop
(EngineCore pid=27014) self._process_engine_step()
(EngineCore pid=27014) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 1213, in _process_engine_step
(EngineCore pid=27014) outputs, model_executed = self.step_fn()
(EngineCore pid=27014) ^^^^^^^^^^^^^^
(EngineCore pid=27014) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 525, in step_with_batch_queue
(EngineCore pid=27014) model_output = future.result()
(EngineCore pid=27014) ^^^^^^^^^^^^^^^
(EngineCore pid=27014) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 90, in result
(EngineCore pid=27014) return super().result()
(EngineCore pid=27014) ^^^^^^^^^^^^^^^^
(EngineCore pid=27014) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 449, in result
(EngineCore pid=27014) return self.__get_result()
(EngineCore pid=27014) ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=27014) File "/usr/lib/python3.12/concurrent/futures/_base.py", line 401, in __get_result
(EngineCore pid=27014) raise self._exception
(EngineCore pid=27014) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 94, in _wait_for_response
(EngineCore pid=27014) response = self.aggregate(self.get_response())
(EngineCore pid=27014) ^^^^^^^^^^^^^^^^^^^
(EngineCore pid=27014) File "/usr/local/lib/python3.12/dist-packages/vllm/v1/executor/multiproc_executor.py", line 386, in get_response
(EngineCore pid=27014) status, result = mq.dequeue(timeout=dequeue_timeout)
@LujhCoconut
@LujhCoconut commented on GitHub (May 21, 2026):
@yiminghub2024 Thanks, I'll dig deeper into this issue.
@yiminghub2024 commented on GitHub (May 21, 2026):
OK ,I WILL wait for your good news ,you can add my wechat , i am interesing in this project . my wechat:yiminghub2012 @LujhCoconut maybe we can have a talk :)
@LujhCoconut commented on GitHub (May 21, 2026):
ok