[GH-ISSUE #19] confused about the cpu memory. #10

New issue

Closed

opened 2026-10-29 14:32:38 +00:00 by bsakaguchi · 0 comments

bsakaguchi commented

2026-10-29 14:32:38 +00:00

Owner

Originally created by @Eutenacity on GitHub (Aug 3, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/19

It seems that all experts tensor loaded from gguf will be dequantized into float32. It will take large CPU memory, is it right? Is it possible to save CPU memory?

"ktransformers/ktransformers/util/custom_gguf.py" "line 274 def load_gguf_tensor ..."

Originally created by @Eutenacity on GitHub (Aug 3, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/19 It seems that all experts tensor loaded from gguf will be dequantized into float32. It will take large CPU memory, is it right? Is it possible to save CPU memory? "ktransformers/ktransformers/util/custom_gguf.py" "line 274 def load_gguf_tensor ..."

bsakaguchi closed this issue

2026-10-29 14:32:38 +00:00

bsakaguchi referenced this issue

2026-10-29 14:37:56 +00:00

[GH-ISSUE #465] /pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion probability tensor contains either inf, nan or element < 0 failed. #327

bsakaguchi referenced this issue

2026-10-29 14:42:22 +00:00

[GH-ISSUE #903] Rocm run deepseek r1 671b q4 failed #646

bsakaguchi referenced this issue

2026-10-29 14:45:29 +00:00

[GH-ISSUE #1269] [Question] benchmark with evalscope failed #859

bsakaguchi referenced this issue