mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1288] [Bug] #876
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#876
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @sxc510134974 on GitHub (May 13, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1288
Checklist
Describe the bug
帮我找找看 我用ktransformers 加载了模型 每次都是 cuda_graph这儿失败 我看别人是cuda_graph6个 为啥我的8个 还失败掉了
mbind: Operation not permitted
mbind: Operation not permitted
mbind: Operation not permitted
mbind: Operation not permitted
loading model.layers.47.input_layernorm.weight to cuda
loading model.layers.47.post_attention_layernorm.weight to cuda
loading model.norm.weight to cuda
Getting inference context from sched_client.
sched_rpc started with PID: 1646
Got inference context, sending it to subscribers.
Rebuilding kvcache
48
kv_cache loaded successfully.
capturing cuda graph 1 1
2025-05-13 18:15:42,601 - INFO - flashinfer.jit: Loading JIT ops: batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_128_head_dim_vo_128_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False
/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
2025-05-13 18:15:58,823 - INFO - flashinfer.jit: Finished loading JIT ops: batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_128_head_dim_vo_128_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False
2025-05-13 18:15:59,337 - INFO - flashinfer.jit: Loading JIT ops: norm
/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
2025-05-13 18:16:18,505 - INFO - flashinfer.jit: Finished loading JIT ops: norm
2025-05-13 18:16:18,685 - INFO - flashinfer.jit: Loading JIT ops: page
/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
2025-05-13 18:16:36,102 - INFO - flashinfer.jit: Finished loading JIT ops: page
cuda_graph: 1/8, warmup finished.
capturing cuda graph 2 2
cuda_graph: 2/8, warmup finished.
capturing cuda graph 3 3
cuda_graph: 3/8, warmup finished.
帮我找找看 到底啥问题
48
kv_cache loaded successfully.
capturing cuda graph 1 1
2025-05-13 18:15:42,601 - INFO - flashinfer.jit: Loading JIT ops: batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_128_head_dim_vo_128_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False
/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
2025-05-13 18:15:58,823 - INFO - flashinfer.jit: Finished loading JIT ops: batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_128_head_dim_vo_128_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False
2025-05-13 18:15:59,337 - INFO - flashinfer.jit: Loading JIT ops: norm
/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
2025-05-13 18:16:18,505 - INFO - flashinfer.jit: Finished loading JIT ops: norm
2025-05-13 18:16:18,685 - INFO - flashinfer.jit: Loading JIT ops: page
/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/utils/cpp_extension.py:2356: UserWarning: TORCH_CUDA_ARCH_LIST is not set, all archs for visible cards are included for compilation.
If this is not desired, please set os.environ['TORCH_CUDA_ARCH_LIST'].
warnings.warn(
2025-05-13 18:16:36,102 - INFO - flashinfer.jit: Finished loading JIT ops: page
cuda_graph: 1/8, warmup finished.
capturing cuda graph 2 2
cuda_graph: 2/8, warmup finished.
capturing cuda graph 3 3
cuda_graph: 3/8, warmup finished.
capturing cuda graph 4 4
Process SpawnProcess-1:
Traceback (most recent call last):
File "/root/miniconda3/envs/kt311/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/root/miniconda3/envs/kt311/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 279, in run_engine
engine.model_runner.warmup()
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/server/balance_serve/inference/model_runner.py", line 148, in warmup
self.outputs_buf[i] = self.model(self.input[i], self.features_buf[i], self.bsz_tensor_buf, self.num_tokens_tensor_buf, self.page_idx_buf[i], self.page_offset_buf[i], cuda_graph_idx=i)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/models/custom_modeling_qwen3_moe.py", line 109, in forward
hidden_states = decode_layer.mlp(hidden_states.unsqueeze(0), num_tokens_tensors, cuda_graph_idx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/operators/experts.py", line 1438, in forward
y = self.moe_on_cpuinfer(hidden_states, selected_experts, routing_weights, bsz_tensor, cuda_graph_idx).view(*orig_shape).to(device=hidden_states.device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/operators/experts.py", line 1459, in moe_on_cpuinfer
outs = self.experts(x, topk_ids, topk_weight, bsz_tensor, cuda_graph_idx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/operators/experts.py", line 1272, in forward
return self.generate_experts.forward(input_tensor, expert_ids, weights, bsz_tensor, cuda_graph_idx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/operators/experts.py", line 311, in forward
input_tensor = input_tensor.contiguous().cpu()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x98 (0x7fb5c1f785e8 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0xe0 (0x7fb5c1f0d4a2 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x7fb5c23cf422 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
frame #3: + 0x32953 (0x7fb5c23ab953 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
frame #4: + 0x32b01 (0x7fb5c23abb01 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
frame #5: + 0x66c7f2 (0x7fb5b9bbb7f2 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #6: + 0x457608 (0x7fb5b99a6608 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #7: + 0x7b385 (0x7fb5c1f59385 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x9 (0x7fb5c1f52f39 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #9: + 0x7182b8 (0x7fb5b9c672b8 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #10: + 0x7186d1 (0x7fb5b9c676d1 in /root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #11: /root/miniconda3/envs/kt311/bin/python() [0x4f5eb7]
frame #12: /root/miniconda3/envs/kt311/bin/python() [0x5031bf]
frame #13: /root/miniconda3/envs/kt311/bin/python() [0x55f246]
frame #14: /root/miniconda3/envs/kt311/bin/python() [0x50302a]
frame #15: /root/miniconda3/envs/kt311/bin/python() [0x5031bf]
frame #16: /root/miniconda3/envs/kt311/bin/python() [0x55f246]
frame #17: /root/miniconda3/envs/kt311/bin/python() [0x55ef87]
frame #18: /root/miniconda3/envs/kt311/bin/python() [0x55f040]
frame #19: /root/miniconda3/envs/kt311/bin/python() [0x541355]
frame #20: /root/miniconda3/envs/kt311/bin/python() [0x53fd88]
frame #21: /root/miniconda3/envs/kt311/bin/python() [0x53fdc4]
frame #22: /root/miniconda3/envs/kt311/bin/python() [0x53fdc4]
frame #23: /root/miniconda3/envs/kt311/bin/python() [0x54e833]
frame #24: _PyEval_EvalFrameDefault + 0x46f9 (0x515ac9 in /root/miniconda3/envs/kt311/bin/python)
frame #25: /root/miniconda3/envs/kt311/bin/python() [0x5cc3aa]
frame #26: PyEval_EvalCode + 0x9f (0x5cba7f in /root/miniconda3/envs/kt311/bin/python)
frame #27: /root/miniconda3/envs/kt311/bin/python() [0x5ecba7]
frame #28: /root/miniconda3/envs/kt311/bin/python() [0x5e8740]
frame #29: PyRun_StringFlags + 0x5f (0x5db24f in /root/miniconda3/envs/kt311/bin/python)
frame #30: PyRun_SimpleStringFlags + 0x3b (0x5daffb in /root/miniconda3/envs/kt311/bin/python)
frame #31: Py_RunMain + 0x388 (0x5f7498 in /root/miniconda3/envs/kt311/bin/python)
frame #32: Py_BytesMain + 0x39 (0x5bc149 in /root/miniconda3/envs/kt311/bin/python)
frame #33: + 0x29d90 (0x7fb5c4814d90 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #34: __libc_start_main + 0x80 (0x7fb5c4814e40 in /usr/lib/x86_64-linux-gnu/libc.so.6)
frame #35: /root/miniconda3/envs/kt311/bin/python() [0x5bbf93]
Reproduction
每次加载完，最有几步就报错 为什么
Environment
XX
@qiyuxinlin commented on GitHub (May 13, 2025):
@sxc510134974 请问您的启动命令是什么？
@BryceWinston commented on GitHub (May 13, 2025):
同样的问题，python ktransformers/server/main.py
--architectures Qwen3MoeForCausalLM
--model_path /root/autodl-tmp/Qwen3-235B-A22B
--gguf_path /root/autodl-tmp/Qwen3-235B-A22B-GGUF
--optimize_config_path ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml
--port 10002
--cpu_infer 54
--chunk_size 256
--max_new_tokens 4096
--max_batch_size 4
--cache_lens 32768
--backend_type balance_serve
--web True
要加载8个cuda_graph,每次加载到6个就报错
@qiyuxinlin commented on GitHub (May 13, 2025):
具体的报错和上面遇到的是一个吗？我使用 Qwen3-30B 模型运行了下您的启动命令 warmup 是成功的
@BryceWinston commented on GitHub (May 13, 2025):
是的，4卡4090d，一样的报错，请问你的模型文件和配置文件在哪里下的
@qiyuxinlin commented on GitHub (May 13, 2025):
最新代码更新了 model_path 和 gguf_path 都传入 safetensor，我是从 modelscope 下的，应该和模型权重没关系，你的代码是什么时候的？更新下最新版本试试呢？
@BryceWinston commented on GitHub (May 13, 2025):
使用的是最新版本，请问你在运行cuda_graph时是1/8吗，我在另一台电脑上运行deepseek-r1，是1/8且不报错能正常跑起来
@BryceWinston commented on GitHub (May 14, 2025):
@BryceWinston commented on GitHub (May 14, 2025):
报错如下：kv_cache loaded successfully.
capturing cuda graph 1 1
2025-05-14 14:00:56,212 - INFO - flashinfer.jit: Loading JIT ops: batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_128_head_dim_vo_128_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False
2025-05-14 14:01:13,285 - INFO - flashinfer.jit: Finished loading JIT ops: batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_128_head_dim_vo_128_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False
2025-05-14 14:01:13,804 - INFO - flashinfer.jit: Loading JIT ops: norm
2025-05-14 14:01:31,561 - INFO - flashinfer.jit: Finished loading JIT ops: norm
2025-05-14 14:01:31,742 - INFO - flashinfer.jit: Loading JIT ops: page
2025-05-14 14:01:48,189 - INFO - flashinfer.jit: Finished loading JIT ops: page
cuda_graph: 1/8, warmup finished.
capturing cuda graph 2 2
cuda_graph: 2/8, warmup finished.
capturing cuda graph 3 3
cuda_graph: 3/8, warmup finished.
capturing cuda graph 4 4
cuda_graph: 4/8, warmup finished.
capturing cuda graph 4 64
cuda_graph: 5/8, warmup finished.
capturing cuda graph 4 256
cuda_graph: 6/8, warmup finished.
capturing cuda graph 4 512
Process SpawnProcess-1:
Traceback (most recent call last):
File "/root/miniconda3/envs/kt311/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/root/miniconda3/envs/kt311/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 279, in run_engine
engine.model_runner.warmup()
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/server/balance_serve/inference/model_runner.py", line 133, in warmup
self.model_attn_plan(self.input[i], i)
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/server/balance_serve/inference/model_runner.py", line 97, in model_attn_plan
self.model.flash_infer_attn_plan(batch, self.bsz_tensor_buf, self.num_tokens_tensor_buf,
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/models/custom_modeling_qwen3_moe.py", line 131, in flash_infer_attn_plan
self.attn[cuda_graph_idx].plan(minibatch.q_indptr, minibatch.kv_indptr, minibatch.kv_indices,
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/operators/flashinfer_batch_prefill_wrapper.py", line 84, in plan
self.warpper.plan(
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/flashinfer/prefill.py", line 1425, in plan
self._plan_info = self._cached_module.plan(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/root/miniconda3/envs/kt311/lib/python3.11/site-packages/torch/_ops.py", line 756, in call
return self._op(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: Error in function 'aligned_alloc' at /root/miniconda3/envs/kt311/lib/python3.11/site-packages/flashinfer/data/include/flashinfer/attention/../allocator.h:48: Failed to allocate memory for batch_prefill_tmp_v with size 281018368 and alignment 16 in AlignedAllocator
是显存不够吗
@qiyuxinlin commented on GitHub (May 14, 2025):
我刚刚修复了这个问题，您拉下最新代码试试
@guanxingithub commented on GitHub (May 19, 2025):
Could you please let me know where is your new code? Thanks.