kvcache-ai/ktransformers

Fork 0

mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00

[GH-ISSUE #1375] [Bug] DeekSeek-R1-0528 max_new_tokens limit? #922

New issue

Open

opened 2026-10-29 14:46:09 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 14:46:09 +00:00

Owner

Originally created by @createthis on GitHub (Jun 8, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1375

Checklist

1. I have searched related issues but cannot get the expected help.
2. The bug has not been fixed in the latest version.
3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback.
4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed.
5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed.

Describe the bug

DeepSeek-R1-0528's output is cut short when thinking sometimes.

Reproduction

I start R1 0528 with:

python ktransformers/server/main.py \
  --port 11434 \
  --model_path /data/DeepSeek-R1 \
  --model_name "DeepSeek-R1-0528:671b-q4_k_m" \
  --gguf_path /data/DeepSeek-R1-0528/Q4_K_M \
  --optimize_config_path /home/jesse/ktransformers/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml \
  --cpu_infer 30 \
  --temperature 0.6 \
  --top_p 0.95 \
  --cache_lens 65536 \
  --chunk_size 256 \
  --max_new_tokens 131072 \
  --backend_type ktransformers

However, R1's output is still cut off sometimes when thinking. It certainly does not appear to have outputted 128k tokens.

Environment

ubuntu 24.04.02
dual EPYC 9355
768gb ram
3090 24gb
latest ktransformers
using the ktransformers backend

Originally created by @createthis on GitHub (Jun 8, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1375 ### Checklist - [x] 1. I have searched related issues but cannot get the expected help. - [x] 2. The bug has not been fixed in the latest version. - [x] 3. Please note that if the bug-related issue you submitted lacks corresponding environment info and a minimal reproducible demo, it will be challenging for us to reproduce and resolve the issue, reducing the likelihood of receiving feedback. - [x] 4. If the issue you raised is not a bug but a question, please raise a discussion at https://github.com/kvcache-ai/ktransformers/discussions. Otherwise, it will be closed. - [x] 5. To help the community, I will use Chinese/English or attach an Chinese/English translation if using another language. Non-Chinese/English content without translation may be closed. ### Describe the bug DeepSeek-R1-0528's output is cut short when thinking sometimes. ### Reproduction I start R1 0528 with: ```bash python ktransformers/server/main.py \ --port 11434 \ --model_path /data/DeepSeek-R1 \ --model_name "DeepSeek-R1-0528:671b-q4_k_m" \ --gguf_path /data/DeepSeek-R1-0528/Q4_K_M \ --optimize_config_path /home/jesse/ktransformers/ktransformers/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat.yaml \ --cpu_infer 30 \ --temperature 0.6 \ --top_p 0.95 \ --cache_lens 65536 \ --chunk_size 256 \ --max_new_tokens 131072 \ --backend_type ktransformers ``` However, R1's output is still cut off sometimes when thinking. It certainly does not appear to have outputted 128k tokens. ### Environment ubuntu 24.04.02 dual EPYC 9355 768gb ram 3090 24gb latest ktransformers using the ktransformers backend

bsakaguchi commented

2026-10-29 14:46:10 +00:00

Author

Owner

@qiuqiu10 commented on GitHub (Jun 9, 2025):

Can you please provide more evidence, including the question you asked and the answer you received? How do you know that the answer is cut, maybe it is just that short?

@qiuqiu10 commented on GitHub (Jun 9, 2025): Can you please provide more evidence, including the question you asked and the answer you received? How do you know that the answer is cut, maybe it is just that short?

bsakaguchi commented

2026-10-29 14:46:10 +00:00

Author

Owner

@createthis commented on GitHub (Jun 9, 2025):

@qiuqiu10 we're in luck! This issue is 100% reproducible and not encumbered by IP, so I can share the entire session with you. It happened when I was asking R1-0528 to debug https://github.com/kvcache-ai/ktransformers/issues/1272 locally.

Note this may also be the same underlying issue as https://github.com/kvcache-ai/ktransformers/issues/1302

I used mitmdump in reverse proxy mode to record the entire session. Please read the README.md at https://github.com/createthis/ktransformers_r1_stops_before_closing_think_060925 for complete details.

I've been debugging this a LOT lately because I use ktransformers and DeepSeek-V3-0324 (and now probably DeepSeek-R1-0528 too) for paid work. When the system is unreliable, it impacts my work. I think the issue may be the custom version of flashinfer-python 0.2.3 that ships with balance_serve. My next step is to build without balance_serve and use the latest official flashinfer-python and see if the issue is still reproducible.

@createthis commented on GitHub (Jun 9, 2025): @qiuqiu10 we're in luck! This issue is 100% reproducible and not encumbered by IP, so I can share the entire session with you. It happened when I was asking R1-0528 to debug https://github.com/kvcache-ai/ktransformers/issues/1272 locally. Note this may also be the same underlying issue as https://github.com/kvcache-ai/ktransformers/issues/1302 I used `mitmdump` in reverse proxy mode to record the entire session. Please read the `README.md` at https://github.com/createthis/ktransformers_r1_stops_before_closing_think_060925 for complete details. I've been debugging this a **LOT** lately because I use `ktransformers` and `DeepSeek-V3-0324` (and now probably `DeepSeek-R1-0528` too) for paid work. When the system is unreliable, it impacts my work. I think the issue may be the custom version of `flashinfer-python` `0.2.3` that ships with `balance_serve`. My next step is to build without `balance_serve` and use the latest official `flashinfer-python` and see if the issue is still reproducible.

bsakaguchi commented

2026-10-29 14:46:11 +00:00

Author

Owner

@qiuqiu10 commented on GitHub (Jun 10, 2025):

Thanks for your response, we may need more time to look into it since the workflow in your case is a little complicated, and we would very much appreciate it if you could help narrowing down the problem to a simple case(for example, only one prompt without other restrictions).
Well after a quick look into https://github.com/createthis/ktransformers_r1_stops_before_closing_think_060925, I advise that you use serve yaml instead of chat yaml when using server mode(you should use DeepSeek-V3-Chat-serve.yaml instead of DeepSeek-V3-Chat.yaml), please change the yaml file and see if it works any differently

@qiuqiu10 commented on GitHub (Jun 10, 2025): Thanks for your response, we may need more time to look into it since the workflow in your case is a little complicated, and we would very much appreciate it if you could help narrowing down the problem to a simple case(for example, only one prompt without other restrictions). Well after a quick look into https://github.com/createthis/ktransformers_r1_stops_before_closing_think_060925, I advise that you use serve yaml instead of chat yaml when using server mode(you should use DeepSeek-V3-Chat-serve.yaml instead of DeepSeek-V3-Chat.yaml), please change the yaml file and see if it works any differently

bsakaguchi commented

2026-10-29 14:46:12 +00:00

Author

Owner

@createthis commented on GitHub (Jun 10, 2025):

@qiuqiu10 I'm using the ktransformers backend, so if I try to use DeepSeek-V3-Chat-serve.yaml I get this error:

TypeError: flashinfer_attn.forward() got an unexpected keyword argument 'attention_mask'

Traceback (most recent call last): File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__ return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__ raise exc File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__ await self.app(scope, receive, _send) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__ await self.app(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__ await self.middleware_stack(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/routing.py", line 734, in app await route.handle(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle await self.app(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/routing.py", line 76, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/routing.py", line 74, in app await response(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/responses.py", line 262, in __call__ with collapse_excgroups(): File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/contextlib.py", line 158, in __exit__ self.gen.throw(typ, value, traceback) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups raise exc File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/responses.py", line 266, in wrap await func() File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/responses.py", line 246, in stream_response async for chunk in self.body_iterator: File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link async for event in async_events: File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply async for event in async_events: File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done async for event in async_events: File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk async for event in async_events: File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 266, in inner async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens): File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 241, in inference async for v in super().inference(local_messages, thread_id, temperature, top_p, max_tokens, max_completion_tokens): File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 466, in inference for t in self.prefill(input_ids, self.check_is_new(thread_id), temperature, top_p, max_tokens, max_completion_tokens): File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context response = gen.send(None) ^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 224, in prefill logits = chunk_prefill(input_ids[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 207, in chunk_prefill logits = self.model( ^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1690, in forward outputs = self.model( ^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/operators/models.py", line 751, in forward layer_outputs = decoder_layer( ^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1207, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: flashinfer_attn.forward() got an unexpected keyword argument 'attention_mask'

@createthis commented on GitHub (Jun 10, 2025): @qiuqiu10 I'm using the `ktransformers` backend, so if I try to use `DeepSeek-V3-Chat-serve.yaml` I get this error: <details> <summary>TypeError: flashinfer_attn.forward() got an unexpected keyword argument 'attention_mask' </summary> Traceback (most recent call last): File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/uvicorn/protocols/http/h11_impl.py", line 403, in run_asgi result = await app( # type: ignore[func-returns-value] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/uvicorn/middleware/proxy_headers.py", line 60, in __call__ return await self.app(scope, receive, send) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/fastapi/applications.py", line 1054, in __call__ await super().__call__(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/applications.py", line 112, in __call__ await self.middleware_stack(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/middleware/errors.py", line 187, in __call__ raise exc File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/middleware/errors.py", line 165, in __call__ await self.app(scope, receive, _send) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/middleware/cors.py", line 85, in __call__ await self.app(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/middleware/exceptions.py", line 62, in __call__ await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/routing.py", line 714, in __call__ await self.middleware_stack(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/routing.py", line 734, in app await route.handle(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/routing.py", line 288, in handle await self.app(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/routing.py", line 76, in app await wrap_app_handling_exceptions(app, request)(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/_exception_handler.py", line 53, in wrapped_app raise exc File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/_exception_handler.py", line 42, in wrapped_app await app(scope, receive, sender) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/routing.py", line 74, in app await response(scope, receive, send) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/responses.py", line 262, in __call__ with collapse_excgroups(): File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/contextlib.py", line 158, in __exit__ self.gen.throw(typ, value, traceback) File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/_utils.py", line 82, in collapse_excgroups raise exc File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/responses.py", line 266, in wrap await func() File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/starlette/responses.py", line 246, in stream_response async for chunk in self.body_iterator: File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 80, in check_client_link async for event in async_events: File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 93, in to_stream_reply async for event in async_events: File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 87, in add_done async for event in async_events: File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/server/schemas/assistants/streaming.py", line 107, in filter_chat_chunk async for event in async_events: File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/server/api/openai/endpoints/chat.py", line 266, in inner async for res in interface.inference(input_message, id, create.temperature, create.top_p, create.max_tokens, create.max_completion_tokens): File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 241, in inference async for v in super().inference(local_messages, thread_id, temperature, top_p, max_tokens, max_completion_tokens): File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/transformers.py", line 466, in inference for t in self.prefill(input_ids, self.check_is_new(thread_id), temperature, top_p, max_tokens, max_completion_tokens): File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/torch/utils/_contextlib.py", line 36, in generator_context response = gen.send(None) ^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 224, in prefill logits = chunk_prefill(input_ids[:, chunk_start:chunk_end], cache_position[chunk_start:chunk_end]) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 207, in chunk_prefill logits = self.model( ^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1690, in forward outputs = self.model( ^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/operators/models.py", line 751, in forward layer_outputs = decoder_layer( ^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/ktransformers/models/modeling_deepseek_v3.py", line 1207, in forward hidden_states, self_attn_weights, present_key_value = self.self_attn( ^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1751, in _wrapped_call_impl return self._call_impl(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/jesse/anaconda3/envs/ktransformers_mc26/lib/python3.11/site-packages/torch/nn/modules/module.py", line 1762, in _call_impl return forward_call(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: flashinfer_attn.forward() got an unexpected keyword argument 'attention_mask' </details>

bsakaguchi commented

2026-10-29 14:46:12 +00:00

Author

Owner

@createthis commented on GitHub (Jun 13, 2025):

FYI, this issue where deepseek R1-0528 cuts off before finishing it's "thought" also happens with cuda 12.8 and a blackwell 6000 pro. Seems to be card and cuda agnostic.

@createthis commented on GitHub (Jun 13, 2025): FYI, this issue where deepseek R1-0528 cuts off before finishing it's "thought" also happens with cuda 12.8 and a blackwell 6000 pro. Seems to be card and cuda agnostic.