[GH-ISSUE #73] When the input token exceeds 4096, an error will occur.

bsakaguchi commented

2026-10-29 14:32:52 +00:00

Owner

Originally created by @fengyang95 on GitHub (Sep 2, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/73

               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/fastapi/routing.py", line 210, in run_endpoint_function
return await dependant.call(**values)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 32, in chat_completion
async for token in interface.inference(input_message,id):
File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 323, in inference
for t in self.prefill(input_ids,self.check_is_new(thread_id)):
File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context
response = gen.send(None)
^^^^^^^^^^^^^^
File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 272, in prefill
logits = self.model(
^^^^^^^^^^^
File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1731, in forward
outputs = self.model(
^^^^^^^^^^^
File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl
return forward_call(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/operators/models.py", line 651, in forward
causal_mask = self._update_causal_mask(
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1624, in _update_causal_mask
padding_mask = causal_mask[:, :, :, :mask_length] + attention_mask[:, None, None, :]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
RuntimeError: The size of tensor a (4096) must match the size of tensor b (8122) at non-singleton dimension 3

Originally created by @fengyang95 on GitHub (Sep 2, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/73 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/fastapi/routing.py", line 210, in run_endpoint_function return await dependant.call(**values) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 32, in chat_completion async for token in interface.inference(input_message,id): File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 323, in inference for t in self.prefill(input_ids,self.check_is_new(thread_id)): File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context response = gen.send(None) ^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 272, in prefill logits = self.model( ^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1731, in forward outputs = self.model( ^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1553, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1562, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/operators/models.py", line 651, in forward causal_mask = self._update_causal_mask( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek.py", line 1624, in _update_causal_mask padding_mask = causal_mask[:, :, :, :mask_length] + attention_mask[:, None, None, :] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ RuntimeError: The size of tensor a (4096) must match the size of tensor b (8122) at non-singleton dimension 3

bsakaguchi closed this issue

2026-10-29 14:32:52 +00:00

bsakaguchi commented

2026-10-29 14:32:53 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Sep 3, 2024):

Hi, I haven’t encountered this issue in our most recent code update. Could you please try using the latest version and let me know if the problem persists?

@Azure-Tang commented on GitHub (Sep 3, 2024): Hi, I haven’t encountered this issue in our most recent code update. Could you please try using the latest version and let me know if the problem persists?

bsakaguchi commented

2026-10-29 14:32:54 +00:00

Author

Owner

@fengyang95 commented on GitHub (Sep 3, 2024):

Hi, I haven’t encountered this issue in our most recent code update. Could you please try using the latest version and let me know if the problem persists?

I compiled it using the most recent code, and my configs are:

- match:
    class: ktransformers.models.modeling_deepseek.DeepseekV2YarnRotaryEmbedding
  replace:
    class: ktransformers.operators.RoPE.YarnRotaryEmbedding
    kwargs:
      generate_device: "cuda"
      prefill_device: "cuda"
- match:
    name: "^model\\.layers\\.(?!.*self_attn\\.kv_b_proj).*$"  # regular expression
    class: torch.nn.Linear  # only match modules matching name and class simultaneously
  replace:
    class: ktransformers.operators.linear.KTransformersLinear  # optimized Kernel on quantized data types
    kwargs:
      generate_device: "cuda"
      prefill_device: "cuda"
      generate_op: "KLinearMarlin"
      prefill_op: "KLinearTorch"
- match:
    name: "^model\\.layers\\..*\\.mlp$"
    class: ktransformers.models.modeling_deepseek.DeepseekV2MoE
  replace:
    class: ktransformers.operators.experts.KDeepseekV2MoE     # mlp module with custom forward function
    kwargs:
      generate_device: "cuda"
      prefill_device: "cuda"
- match:
    name: "^model\\.layers\\..*\\.mlp\\.experts$"
  replace:
    class: ktransformers.operators.experts.KTransformersExperts     # custom MoE Kernel with expert paralleism
    kwargs:
      prefill_device: "cuda"
      prefill_op: "KExpertsTorch"
      generate_device: "cpu"
      generate_op: "KExpertsCPU"
      out_device: "cuda"
  recursive: False # don't recursively inject submodules of this module
- match:
    name: "^model\\.layers\\..*\\.self_attn$"
  replace:
    class: ktransformers.operators.attention.KDeepseekV2Attention # optimized MLA implementation
    kwargs:
      generate_device: "cuda"
      prefill_device: "cuda"
- match:
    name: "^model$"
  replace:
    class: "ktransformers.operators.models.KDeepseekV2Model"
    kwargs:
      per_layer_prefill_intput_threshold: 0 # 0 is close layer wise prefill
- match:
    name: "^model.embed_tokens"
  replace:
    class: "default"
    kwargs:
      generate_device: "cpu"
      prefill_device: "cuda"

@fengyang95 commented on GitHub (Sep 3, 2024): > Hi, I haven’t encountered this issue in our most recent code update. Could you please try using the latest version and let me know if the problem persists? I compiled it using the most recent code, and my configs are: ``` yaml - match: class: ktransformers.models.modeling_deepseek.DeepseekV2YarnRotaryEmbedding replace: class: ktransformers.operators.RoPE.YarnRotaryEmbedding kwargs: generate_device: "cuda" prefill_device: "cuda" - match: name: "^model\\.layers\\.(?!.*self_attn\\.kv_b_proj).*$" # regular expression class: torch.nn.Linear # only match modules matching name and class simultaneously replace: class: ktransformers.operators.linear.KTransformersLinear # optimized Kernel on quantized data types kwargs: generate_device: "cuda" prefill_device: "cuda" generate_op: "KLinearMarlin" prefill_op: "KLinearTorch" - match: name: "^model\\.layers\\..*\\.mlp$" class: ktransformers.models.modeling_deepseek.DeepseekV2MoE replace: class: ktransformers.operators.experts.KDeepseekV2MoE # mlp module with custom forward function kwargs: generate_device: "cuda" prefill_device: "cuda" - match: name: "^model\\.layers\\..*\\.mlp\\.experts$" replace: class: ktransformers.operators.experts.KTransformersExperts # custom MoE Kernel with expert paralleism kwargs: prefill_device: "cuda" prefill_op: "KExpertsTorch" generate_device: "cpu" generate_op: "KExpertsCPU" out_device: "cuda" recursive: False # don't recursively inject submodules of this module - match: name: "^model\\.layers\\..*\\.self_attn$" replace: class: ktransformers.operators.attention.KDeepseekV2Attention # optimized MLA implementation kwargs: generate_device: "cuda" prefill_device: "cuda" - match: name: "^model$" replace: class: "ktransformers.operators.models.KDeepseekV2Model" kwargs: per_layer_prefill_intput_threshold: 0 # 0 is close layer wise prefill - match: name: "^model.embed_tokens" replace: class: "default" kwargs: generate_device: "cpu" prefill_device: "cuda" ```

bsakaguchi commented

2026-10-29 14:32:54 +00:00

Author

Owner

@UnicornChan commented on GitHub (Sep 3, 2024):

I'm very sorry. For the convenience of server testing, we set the cache_lens parameter to 4096, which has caused errors when the cache length exceeds this value. This configuration item is hardcoded as "cache_lens" in ktransformers/server/backend/args.py. We will make these configuration items configurable in the next release.
If you want to support more tokens now, I suggest to modify the config in /home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/server/backend/args.py line 93, cache_lens

@UnicornChan commented on GitHub (Sep 3, 2024): I'm very sorry. For the convenience of server testing, we set the cache_lens parameter to 4096, which has caused errors when the cache length exceeds this value. This configuration item is hardcoded as "cache_lens" in ktransformers/server/backend/args.py. We will make these configuration items configurable in the next release. If you want to support more tokens now, I suggest to modify the config in `/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/server/backend/args.py` line 93, cache_lens

bsakaguchi commented

2026-10-29 14:32:54 +00:00

Author

Owner

@fengyang95 commented on GitHub (Sep 3, 2024):

I'm very sorry. For the convenience of server testing, we set the cache_lens parameter to 4096, which has caused errors when the cache length exceeds this value. This configuration item is hardcoded as "cache_lens" in ktransformers/server/backend/args.py. We will make these configuration items configurable in the next release. If you want to support more tokens now, I suggest to modify the config in /home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/server/backend/args.py line 93, cache_lens

LGTM

@fengyang95 commented on GitHub (Sep 3, 2024): > I'm very sorry. For the convenience of server testing, we set the cache_lens parameter to 4096, which has caused errors when the cache length exceeds this value. This configuration item is hardcoded as "cache_lens" in ktransformers/server/backend/args.py. We will make these configuration items configurable in the next release. If you want to support more tokens now, I suggest to modify the config in `/home/tiger/.pyenv/versions/3.11.2/lib/python3.11/site-packages/ktransformers/server/backend/args.py` line 93, cache_lens LGTM

bsakaguchi referenced this issue

2026-10-29 14:45:29 +00:00

[GH-ISSUE #1269] [Question] benchmark with evalscope failed #859

bsakaguchi referenced this issue

2026-10-29 14:45:38 +00:00

[GH-ISSUE #1286] [Bug] inference_context.v_cache[0][i] IndexError: list index out of range #871

bsakaguchi referenced this issue

2026-10-29 14:45:39 +00:00

[GH-ISSUE #1288] [Bug] #876

bsakaguchi referenced this issue

2026-10-29 14:51:26 +00:00

[PR #35] [MERGED] [update] Update readme; Add tutorial #1324