[GH-ISSUE #1676] CPU memory usage should be reduced during large chunk prefill. #1074

New issue

Closed

opened 2026-10-29 14:48:13 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:48:13 +00:00

Owner

Originally created by @chenht2022 on GitHub (Dec 6, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1676

Originally assigned to: @ouqingliang on GitHub.

Reminder

I have read the above rules and searched the existing issues.

System Info

Running K2T-RAWINT4 with --chunked-prefill-size 32000 consumes around 1000GB of CPU memory, causing an OOM on machines with 768GB of CPU memory. This needs optimization.

Reproduction

TP MOE layer 1, pool: 0x28f29710, expert num: 384, num_experts_per_tok: 8
Creating AMX_K2_MOE_TP 1 at numa 1
Creating AMX_K2_MOE_TP 0 at numa 0
cannot aligned alloc 963379200000 bytes (align=64)
posix_memalign: Cannot allocate memory

Others

No response

Originally created by @chenht2022 on GitHub (Dec 6, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1676 Originally assigned to: @ouqingliang on GitHub. ### Reminder - [x] I have read the above rules and searched the existing issues. ### System Info Running K2T-RAWINT4 with --chunked-prefill-size 32000 consumes around 1000GB of CPU memory, causing an OOM on machines with 768GB of CPU memory. This needs optimization. ### Reproduction ```text TP MOE layer 1, pool: 0x28f29710, expert num: 384, num_experts_per_tok: 8 Creating AMX_K2_MOE_TP 1 at numa 1 Creating AMX_K2_MOE_TP 0 at numa 0 cannot aligned alloc 963379200000 bytes (align=64) posix_memalign: Cannot allocate memory ``` ### Others _No response_

bsakaguchi

2026-10-29 14:48:13 +00:00