[GH-ISSUE #92] Specify MAX_NEW_TOKENS for ktransformers server #48

New issue

Closed

opened 2026-10-29 14:33:01 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-29 14:33:01 +00:00

Owner

Originally created by @arthurv on GitHub (Sep 19, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/92

max_new_tokens = 1000 by default, and this can be specified in ktransformers.local_chat through --max_new_tokens, but not the server.

Please add the --max_new_tokens option to the ktransformers server so we can specify longer output context lengths, and add more generation options (like input context, etc).

Originally created by @arthurv on GitHub (Sep 19, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/92 max_new_tokens = 1000 by default, and this can be specified in ktransformers.local_chat through --max_new_tokens, but not the server. Please add the --max_new_tokens option to the ktransformers server so we can specify longer output context lengths, and add more generation options (like input context, etc).

bsakaguchi

2026-10-29 14:33:01 +00:00

closed this issue
added the
stale
label

bsakaguchi commented

2026-10-29 14:33:01 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Sep 23, 2024):

Apologies for the inconvenience. If you’re building from source, you can modify the max_new_tokens parameter in ktransformers/server/backend/args.py. We will include this update in the next Docker release.

@Azure-Tang commented on GitHub (Sep 23, 2024): Apologies for the inconvenience. If you’re building from source, you can modify the max_new_tokens parameter in ktransformers/server/backend/args.py. We will include this update in the next Docker release.

bsakaguchi commented

2026-10-29 14:33:02 +00:00

Author

Owner

@bitbottrap commented on GitHub (Oct 14, 2024):

I just encountered this limitation. It would be even better if the REST API honored the maximum context length and maximum number of generation tokens.

@bitbottrap commented on GitHub (Oct 14, 2024): I just encountered this limitation. It would be even better if the REST API honored the maximum context length and maximum number of generation tokens.

bsakaguchi commented

2026-10-29 14:33:02 +00:00

Author

Owner

@ErvinXie commented on GitHub (Dec 11, 2025):

This issue has been automatically closed due to inactivity for 60 days.

If you believe this issue is still relevant, please feel free to reopen it with additional information or context.

Thank you for your contribution to KTransformers!

此 issue 因超过 60 天无活动而被自动关闭。

如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。

感谢您对 KTransformers 的贡献！

@ErvinXie commented on GitHub (Dec 11, 2025): This issue has been automatically closed due to inactivity for 60 days. If you believe this issue is still relevant, please feel free to reopen it with additional information or context. Thank you for your contribution to KTransformers! --- 此 issue 因超过 60 天无活动而被自动关闭。如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。感谢您对 KTransformers 的贡献！

bsakaguchi referenced this issue

2026-10-29 14:51:29 +00:00

[PR #48] [MERGED] [fix] Fix bugs about static cache and server param; #1332