kvcache-ai/ktransformers

Fork 0

mirror of https://github.com/kvcache-ai/ktransformers.git synced 2026-11-05 01:07:28 +00:00

[GH-ISSUE #89] Installation requirements #45

New issue

Closed

opened 2026-10-29 14:32:58 +00:00 by bsakaguchi · 5 comments

bsakaguchi commented

2026-10-29 14:32:58 +00:00

Owner

Originally created by @arthurv on GitHub (Sep 13, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/89

Hi,

I tried to install ktransformers on a clean install of Linux Mint 22 (based on Ubuntu 24.04), and there are a few things that I had to add:

pip instal numpy
pip install cpufeature
pip install flash_attn
conda install -c conda-forge libstdcxx-ng

Please update the pip dependencies.

Are there any plans to increase the number of quants supported for Deepseek-Coder-V2-Instruct-0724?

Originally created by @arthurv on GitHub (Sep 13, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/89 Hi, I tried to install ktransformers on a clean install of Linux Mint 22 (based on Ubuntu 24.04), and there are a few things that I had to add: ``` pip instal numpy pip install cpufeature pip install flash_attn conda install -c conda-forge libstdcxx-ng ``` Please update the pip dependencies. Are there any plans to increase the number of quants supported for Deepseek-Coder-V2-Instruct-0724?

bsakaguchi

2026-10-29 14:32:58 +00:00

closed this issue
added the
stale
label

bsakaguchi commented

2026-10-29 14:32:59 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Sep 14, 2024):

Okay, we will update these packages in our next release.

Regarding quantizations, we now support multiple quantization methods(Qx_k and IQ4_XS) . What else would you like?

@Azure-Tang commented on GitHub (Sep 14, 2024): Okay, we will update these packages in our next release. Regarding quantizations, we now support multiple quantization methods(Qx_k and IQ4_XS) . What else would you like?

bsakaguchi commented

2026-10-29 14:32:59 +00:00

Author

Owner

@arthurv commented on GitHub (Sep 15, 2024):

I have a system with 192GB DRAM and 48GB VRAM (2x 3090). Would it be able to handle 128k context with the specs? Would it be able to handle Q5_K_M or Q6_K_M?

Also I can only set max_new_tokens in local_chat, not the ktransformers server, and I can't set the total context size anywhere.

@arthurv commented on GitHub (Sep 15, 2024): I have a system with 192GB DRAM and 48GB VRAM (2x 3090). Would it be able to handle 128k context with the specs? Would it be able to handle Q5_K_M or Q6_K_M? Also I can only set max_new_tokens in local_chat, not the ktransformers server, and I can't set the total context size anywhere.

bsakaguchi commented

2026-10-29 14:32:59 +00:00

Author

Owner

@devprimed commented on GitHub (Sep 18, 2024):

Having this issue as well, not being able to set --max_new_tokens in the container breaks downstream projects that require longer output lengths.

@devprimed commented on GitHub (Sep 18, 2024): Having this issue as well, not being able to set --max_new_tokens in the container breaks downstream projects that require longer output lengths.

bsakaguchi commented

2026-10-29 14:32:59 +00:00

Author

Owner

@Azure-Tang commented on GitHub (Sep 23, 2024):

I have a system with 192GB DRAM and 48GB VRAM (2x 3090). Would it be able to handle 128k context with the specs? Would it be able to handle Q5_K_M or Q6_K_M?

Also I can only set max_new_tokens in local_chat, not the ktransformers server, and I can't set the total context size anywhere.

Yes, we supported Q5_K_M and Q6_K_M.

About setting max_new_tokens in container, sorry for the inconvenience that this is not supported now. If you’re building from source, you can modify the max_new_tokens parameter in ktransformers/server/backend/args.py. We will include this update in the next Docker release.

@Azure-Tang commented on GitHub (Sep 23, 2024): > I have a system with 192GB DRAM and 48GB VRAM (2x 3090). Would it be able to handle 128k context with the specs? Would it be able to handle Q5_K_M or Q6_K_M? > > Also I can only set max_new_tokens in local_chat, not the ktransformers server, and I can't set the total context size anywhere. Yes, we supported Q5_K_M and Q6_K_M. About setting max_new_tokens in container, sorry for the inconvenience that this is not supported now. If you’re building from source, you can modify the `max_new_tokens` parameter in `ktransformers/server/backend/args.py`. We will include this update in the next Docker release.

bsakaguchi commented

2026-10-29 14:33:00 +00:00

Author

Owner

@ErvinXie commented on GitHub (Dec 11, 2025):

This issue has been automatically closed due to inactivity for 60 days.

If you believe this issue is still relevant, please feel free to reopen it with additional information or context.

Thank you for your contribution to KTransformers!

此 issue 因超过 60 天无活动而被自动关闭。

如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。

感谢您对 KTransformers 的贡献！

@ErvinXie commented on GitHub (Dec 11, 2025): This issue has been automatically closed due to inactivity for 60 days. If you believe this issue is still relevant, please feel free to reopen it with additional information or context. Thank you for your contribution to KTransformers! --- 此 issue 因超过 60 天无活动而被自动关闭。如果您认为此问题仍然存在，请随时重新打开并提供更多信息或上下文。感谢您对 KTransformers 的贡献！