mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1269] [Question] benchmark with evalscope failed #859
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#859
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @ant-ob-hengtang on GitHub (May 7, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1269
I meet a CUDA error when running benchmark with evalscope QAQ~
cmd:
err logs:
ENV
docker images:
to use 2cpus and NUMA, I build ktransformers in container again by
@ant-ob-hengtang commented on GitHub (May 7, 2025):
and my start cmd is
ktransformers --model_path /data/modelscope/ds-r1 --gguf_path /data/modelscope/ds-r1-q4km/DeepSeek-R1-Q4_K_M --cpu_infer 65 --port 10002 --max_new_tokens 16384 --model_name deepseek-r1 --optimize_config_path ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-serve.yaml --cache_lens 65536 --chunk_size 256 --cache_q4 true --max_batch_size 4 --force_think --backend_type balance_serve
anyone else meet this err? 中文交流也很ok
@liuwenzhoa commented on GitHub (May 7, 2025):
一样的错误，我是用的是 unsloth/Qwen3-30B-A3B-128K-GGUF-IQ2_XXS，换成Q2_K_L就好了。
2025-05-07 13:07:33,117 INFO /root/autodl-tmp/ktransformers/ktransformers/server/main.py[23]: Creating SQL tables
2025-05-07 13:07:33,121 INFO /root/miniconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/server/api/openai/assistants/assistants.py[75]: Creating default assistant
INFO: Started server process [154315]
INFO: Waiting for application startup.
Queue Proxy Started
INFO: Application startup complete.
INFO: Uvicorn running on http://0.0.0.0:10002 (Press CTRL+C to quit)
/root/miniconda3/envs/kt311/lib/python3.11/site-packages/pydantic/main.py:519: UserWarning: Pydantic serializer warnings:
PydanticSerializationUnexpectedValue(Expected
list[dict[str, any]]- serialized value may not be as expected [input_value={}, input_type=dict])return self.pydantic_serializer.to_json(
2025-05-07 13:08:16,440 DEBUG /root/miniconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py[418]: get input ids of shape torch.Size([1, 16])
add query id: 1, batch.query_lengths: 16, batch_query_tokens: torch.Size([516]), batch.block_indexes: tensor([0, 1, 2], dtype=torch.int32)
prefill_batch_i: 16,
padded_batch_size 57 capture_padded_batch_size 57
Model execution time (GPU): 1249.500 ms, 0.800 tokens/s
2025-05-07 13:08:17,723 - INFO - flashinfer.jit: Loading JIT ops: sampling
2025-05-07 13:08:17,833 - INFO - flashinfer.jit: Finished loading JIT ops: sampling
59693
decode_batch_i: 1,
padded_batch_size 57 capture_padded_batch_size 57
Model execution time (GPU): 352.752 ms, 2.835 tokens/s
Process SpawnProcess-1:
Traceback (most recent call last):
File "/root/miniconda3/envs/kt311/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/root/miniconda3/envs/kt311/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 282, in run_engine
engine.loop()
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 260, in loop
generated_tokens, probs = self.sampling( self.model_runner.output)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 224, in sampling
generated_tokens, probs=self.sampler(logit, sample_options)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/server/balance_serve/inference/sampling/sampler.py", line 97, in forward
temperature_0_idx = torch.where(sampling_config.temperatures == 0)[0]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x98 (0x7f93edd785e8 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0xe0 (0x7f93edd0d4a2 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x7f93ee10b422 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
frame #3: + 0x32953 (0x7f93ee0e7953 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
frame #4: + 0x32b01 (0x7f93ee0e7b01 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
frame #5: + 0x66c7f2 (0x7f93e59bb7f2 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #6: + 0x457608 (0x7f93e57a6608 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #7: + 0x7b385 (0x7f93edd59385 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f93edd52f39 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #9: + 0x7182b8 (0x7f93e5a672b8 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #10: + 0x7186d1 (0x7f93e5a676d1 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #11: /root/miniconda3/envs/kt311/bin/python() [0x4f5eb7]
frame #12: /root/miniconda3/envs/kt311/bin/python() [0x5031bf]
frame #13: /root/miniconda3/envs/kt311/bin/python() [0x55f246]
frame #14: /root/miniconda3/envs/kt311/bin/python() [0x50302a]
frame #15: /root/miniconda3/envs/kt311/bin/python() [0x5031bf]
frame #16: /root/miniconda3/envs/kt311/bin/python() [0x55f246]
frame #17: /root/miniconda3/envs/kt311/bin/python() [0x55ef87]
frame #18: /root/miniconda3/envs/kt311/bin/python() [0x55f040]
frame #19: /root/miniconda3/envs/kt311/bin/python() [0x541355]
frame #20: /root/miniconda3/envs/kt311/bin/python() [0x53fd88]
frame #21: /root/miniconda3/envs/kt311/bin/python() [0x53fdc4]
frame #22: /root/miniconda3/envs/kt311/bin/python() [0x53fdc4]
frame #23: /root/miniconda3/envs/kt311/bin/python() [0x54e833]
frame #24: _PyEval_EvalFrameDefault + 0x46f9 (0x515ac9 in /root/miniconda3/envs/kt311/bin/python)
frame #25: /root/miniconda3/envs/kt311/bin/python() [0x5cc3aa]
frame #26: PyEval_EvalCode + 0x9f (0x5cba7f in /root/miniconda3/envs/kt311/bin/python)
frame #27: /root/miniconda3/envs/kt311/bin/python() [0x5ecba7]
frame #28: /root/miniconda3/envs/kt311/bin/python() [0x5e8740]
frame #29: PyRun_StringFlags + 0x5f (0x5db24f in /root/miniconda3/envs/kt311/bin/python)
frame #30: PyRun_SimpleStringFlags + 0x3b (0x5daffb in /root/miniconda3/envs/kt311/bin/python)
frame #31: Py_RunMain + 0x388 (0x5f7498 in /root/miniconda3/envs/kt311/bin/python)
frame #32: Py_BytesMain + 0x39 (0x5bc149 in /root/miniconda3/envs/kt311/bin/python)
frame #33: + 0x29d90 (0x7f93f0556d90 in /lib/x86_64-linux-gnu/libc.so.6)
frame #34: __libc_start_main + 0x80 (0x7f93f0556e40 in /lib/x86_64-linux-gnu/libc.so.6)
frame #35: /root/miniconda3/envs/kt311/bin/python() [0x5bbf93]
@jizhilong commented on GitHub (May 7, 2025):
Hi, please do a favor to wrap logs/codes/traceback texts in a proper markdown fenced code block, such that:
#34prevented from incorrectly rendered into a reference to other issues.@ant-ob-hengtang commented on GitHub (May 8, 2025):
oh sry for my bad format ~ i have edit my question by some markdown format. Thanks for your advices
@ant-ob-hengtang commented on GitHub (May 8, 2025):
@liuwenzhoa 这个模型的格式我有点一知半解，您这个前后两个模型是什么区别呢？
是模型大小变了，然后就避免了CUDA error: an illegal memory access was encountered ？
@Atream commented on GitHub (May 9, 2025):
Could you tell me whether this bug occurs occasionally or frequently? Could you attempt to send several requests via the API to see if it still causes an error?
请问这个bug是偶发性出现还是经常出现，能否尝试用api发送几个请求，看看是否仍然会报错。
@ant-ob-hengtang commented on GitHub (May 15, 2025):
你好，很抱歉才看到您的回复~我个人单独发送api是不会遇到这个问题的，之后没有再研究这个事情了 sry