mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1286] [Bug] inference_context.v_cache[0][i] IndexError: list index out of range #871
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#871
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @coderjingluo on GitHub (May 13, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1286
检查清单
问题描述
Deploying qwen3-235-a22b-q4 with ktransformers reports an error
复现步骤
Use USE_BALANCE_SERVE=1 KTRANSFORMERS_FORCE_BUILD=TRUE MAX_JOBS=50 TORCH_USE_CUDA_DSA=1 TORCH_CUDA_ARCH_LIST="8.6" . After /install.sh has been compiled successfully
python ktransformers/server/main.py --architectures Qwen3MoeForCausalLm --model_path /home/user/aimodel/models/lmstudio-community/config --gguf_path /home/user/aimodel/models/lmstudio-community/Qwen3-235B-A22B-GGUF --optimize_config_path ktransformers/optimize/optimize_rules/Qwen3Moe-serve.yaml --port 10002 --use_cuda_graph --cpu_infer 54 --chunk_size 512 --max_new_tokens 10240 --max_batch_size 4 --cache_lens 4096 --cache_q4 true --backend_type balance_serve
环境信息
@qiyuxinlin commented on GitHub (May 13, 2025):
@coderjingluo 您好，这是最新代码吗？我知道您这个报错是因为 --architectures Qwen3MoeForCausaLlm 参数不对，但是最新的代码我删掉了这个参数
@coderjingluo commented on GitHub (May 13, 2025):
这个问题解决，不过好像又有新的问题
2025-05-13 12:41:06,232 - INFO - flashinfer.jit: Loading JIT ops: batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_128_head_dim_vo_128_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False
2025-05-13 12:41:06,246 - INFO - flashinfer.jit: Finished loading JIT ops: batch_prefill_with_kv_cache_dtype_q_bf16_dtype_kv_bf16_dtype_o_bf16_dtype_idx_i32_head_dim_qk_128_head_dim_vo_128_posenc_0_use_swa_False_use_logits_cap_False_f16qk_False
2025-05-13 12:41:06,629 - INFO - flashinfer.jit: Loading JIT ops: norm
2025-05-13 12:41:06,641 - INFO - flashinfer.jit: Finished loading JIT ops: norm
2025-05-13 12:41:06,748 - INFO - flashinfer.jit: Loading JIT ops: page
2025-05-13 12:41:06,759 - INFO - flashinfer.jit: Finished loading JIT ops: page
cuda_graph: 1/8, warmup finished.
capturing cuda graph 2 2
cuda_graph: 2/8, warmup finished.
capturing cuda graph 3 3
cuda_graph: 3/8, warmup finished.
capturing cuda graph 1 1
cuda_graph: 4/8, warmup finished.
capturing cuda graph 4 64
Process SpawnProcess-1:
Traceback (most recent call last):
File "/home/user/anaconda3/envs/kt311/lib/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/home/user/anaconda3/envs/kt311/lib/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 279, in run_engine
engine.model_runner.warmup()
File "/home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/server/balance_serve/inference/model_runner.py", line 148, in warmup
self.outputs_buf[i] = self.model(self.input[i], self.features_buf[i], self.bsz_tensor_buf, self.num_tokens_tensor_buf, self.page_idx_buf[i], self.page_offset_buf[i], cuda_graph_idx=i)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/models/custom_modeling_qwen3_moe.py", line 109, in forward
hidden_states = decode_layer.mlp(hidden_states.unsqueeze(0), num_tokens_tensors, cuda_graph_idx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/operators/experts.py", line 1438, in forward
y = self.moe_on_cpuinfer(hidden_states, selected_experts, routing_weights, bsz_tensor, cuda_graph_idx).view(*orig_shape).to(device=hidden_states.device)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/operators/experts.py", line 1459, in moe_on_cpuinfer
outs = self.experts(x, topk_ids, topk_weight, bsz_tensor, cuda_graph_idx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/operators/experts.py", line 1272, in forward
return self.generate_experts.forward(input_tensor, expert_ids, weights, bsz_tensor, cuda_graph_idx)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/ktransformers/operators/experts.py", line 311, in forward
input_tensor = input_tensor.contiguous().cpu()
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.terminate called after throwing an instance of 'c10::Error'
what(): CUDA error: an illegal memory access was encountered
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with
TORCH_USE_CUDA_DSAto enable device-side assertions.Exception raised from c10_cuda_check_implementation at /pytorch/c10/cuda/CUDAException.cpp:43 (most recent call first):
frame #0: c10::Error::Error(c10::SourceLocation, std::__cxx11::basic_string<char, std::char_traits, std::allocator >) + 0x98 (0x7f6acdd785e8 in /home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #1: c10::detail::torchCheckFail(char const*, char const*, unsigned int, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&) + 0xe0 (0x7f6acdd0d4a2 in /home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #2: c10::cuda::c10_cuda_check_implementation(int, char const*, char const*, int, bool) + 0x3c2 (0x7f6ace145422 in /home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
frame #3: + 0x32953 (0x7f6ace121953 in /home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
frame #4: + 0x32b01 (0x7f6ace121b01 in /home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10_cuda.so)
frame #5: + 0x66c7f2 (0x7f6ac59bb7f2 in /home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #6: + 0x457608 (0x7f6ac57a6608 in /home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #7: + 0x7b385 (0x7f6acdd59385 in /home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #8: c10::TensorImpl::~TensorImpl() + 0x9 (0x7f6acdd52f39 in /home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libc10.so)
frame #9: + 0x7182b8 (0x7f6ac5a672b8 in /home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #10: + 0x7186d1 (0x7f6ac5a676d1 in /home/user/anaconda3/envs/kt311/lib/python3.11/site-packages/torch/lib/libtorch_python.so)
frame #11: /home/user/anaconda3/envs/kt311/bin/python() [0x4f5eb7]
frame #12: /home/user/anaconda3/envs/kt311/bin/python() [0x5031bf]
frame #13: /home/user/anaconda3/envs/kt311/bin/python() [0x55f246]
frame #14: /home/user/anaconda3/envs/kt311/bin/python() [0x50302a]
frame #15: /home/user/anaconda3/envs/kt311/bin/python() [0x5031bf]
frame #16: /home/user/anaconda3/envs/kt311/bin/python() [0x55f246]
frame #17: /home/user/anaconda3/envs/kt311/bin/python() [0x55ef87]
frame #18: /home/user/anaconda3/envs/kt311/bin/python() [0x55f040]
frame #19: /home/user/anaconda3/envs/kt311/bin/python() [0x541355]
frame #20: /home/user/anaconda3/envs/kt311/bin/python() [0x53fd88]
frame #21: /home/user/anaconda3/envs/kt311/bin/python() [0x53fdc4]
frame #22: /home/user/anaconda3/envs/kt311/bin/python() [0x53fdc4]
frame #23: /home/user/anaconda3/envs/kt311/bin/python() [0x54e833]
frame #24: _PyEval_EvalFrameDefault + 0x46f9 (0x515ac9 in /home/user/anaconda3/envs/kt311/bin/python)
frame #25: /home/user/anaconda3/envs/kt311/bin/python() [0x5cc3aa]
frame #26: PyEval_EvalCode + 0x9f (0x5cba7f in /home/user/anaconda3/envs/kt311/bin/python)
frame #27: /home/user/anaconda3/envs/kt311/bin/python() [0x5ecba7]
frame #28: /home/user/anaconda3/envs/kt311/bin/python() [0x5e8740]
frame #29: PyRun_StringFlags + 0x5f (0x5db24f in /home/user/anaconda3/envs/kt311/bin/python)
frame #30: PyRun_SimpleStringFlags + 0x3b (0x5daffb in /home/user/anaconda3/envs/kt311/bin/python)
frame #31: Py_RunMain + 0x388 (0x5f7498 in /home/user/anaconda3/envs/kt311/bin/python)
frame #32: Py_BytesMain + 0x39 (0x5bc149 in /home/user/anaconda3/envs/kt311/bin/python)
frame #33: + 0x29d90 (0x7f6ad082ad90 in /lib/x86_64-linux-gnu/libc.so.6)
frame #34: __libc_start_main + 0x80 (0x7f6ad082ae40 in /lib/x86_64-linux-gnu/libc.so.6)
frame #35: /home/user/anaconda3/envs/kt311/bin/python() [0x5bbf9
@qiyuxinlin commented on GitHub (May 13, 2025):
我刚刚提交了一份新的代码，默认的 cache_lens 太小了，你改成 10k以上试试呢
@coderjingluo commented on GitHub (May 13, 2025):
嗯嗯，运行成功了，有个小问题想问下，92GB显存，256GB内存 Q4量化，使用ktransformers的话最多同时支持多少人访问呢
@qiyuxinlin commented on GitHub (May 15, 2025):
这个主要取决于设置的 max_batch_size 和 cache_lens，默认 max_batch_size=4 支持 4 并发，另外如果你当前运行的所有请求 cache 长度超过了 cache_lens 程序会降并发度