[GH-ISSUE #22] error with ffn_down #14

New issue

Closed

opened 2026-10-29 14:32:39 +00:00 by bsakaguchi · 2 comments

bsakaguchi commented

2026-10-29 14:32:39 +00:00

Owner

Originally created by @Eutenacity on GitHub (Aug 4, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/22

I try to load ffn_weight and run it with cpuinfer_ext.linear. ffn_up and ffn_gate are good. But ffn_down result in nan. What is wrong
code is below


from ktransformers.util.custom_gguf import GGUFLoader
import cpuinfer_ext
import torch
import torch.nn.functional as F
import psutil
import os
import ctypes
gguf_path = "/VM/share/models/Mixtral-8x7b-q4k_m/"
gguf_loader=GGUFLoader(gguf_path)
# print(gguf_loader.tensor_info)
tensor = gguf_loader.get_mmap_tensor("blk.0.ffn_down.0.weight")
print(tensor)
a = torch.tensor(tensor,dtype= torch.uint8)
print(a.nbytes)
print(a.shape)
tensor_fp32 = gguf_loader.load_gguf_tensor("blk.0.ffn_down.0.weight").to(torch.bfloat16)
print(tensor_fp32.shape)
gate_ptr = ctypes.addressof(
            ctypes.cast(tensor.ctypes.data, ctypes.POINTER(ctypes.c_uint64)).contents
        )
output_size =  4096
input_size = 14336
stride = 16
proj_type=12
hidden_type=30
CPUInfer = cpuinfer_ext.CPUInfer(32)
config = cpuinfer_ext.linear.LinearConfig(input_size, output_size, stride, gate_ptr, proj_type, hidden_type)
linear = cpuinfer_ext.linear.Linear(config)

input = torch.randn((2, input_size), dtype=torch.bfloat16).contiguous()
output = torch.zeros((2, output_size), dtype=torch.bfloat16).contiguous()
for i in range(2):
    CPUInfer.submit(linear.forward, input[i,:].data_ptr(), output[i,:].data_ptr())
CPUInfer.sync()
print(output)
out = F.linear(input,tensor_fp32)
print(out)

Originally created by @Eutenacity on GitHub (Aug 4, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/22 I try to load ffn_weight and run it with cpuinfer_ext.linear. ffn_up and ffn_gate are good. But ffn_down result in nan. What is wrong code is below ``` from ktransformers.util.custom_gguf import GGUFLoader import cpuinfer_ext import torch import torch.nn.functional as F import psutil import os import ctypes gguf_path = "/VM/share/models/Mixtral-8x7b-q4k_m/" gguf_loader=GGUFLoader(gguf_path) # print(gguf_loader.tensor_info) tensor = gguf_loader.get_mmap_tensor("blk.0.ffn_down.0.weight") print(tensor) a = torch.tensor(tensor,dtype= torch.uint8) print(a.nbytes) print(a.shape) tensor_fp32 = gguf_loader.load_gguf_tensor("blk.0.ffn_down.0.weight").to(torch.bfloat16) print(tensor_fp32.shape) gate_ptr = ctypes.addressof( ctypes.cast(tensor.ctypes.data, ctypes.POINTER(ctypes.c_uint64)).contents ) output_size = 4096 input_size = 14336 stride = 16 proj_type=12 hidden_type=30 CPUInfer = cpuinfer_ext.CPUInfer(32) config = cpuinfer_ext.linear.LinearConfig(input_size, output_size, stride, gate_ptr, proj_type, hidden_type) linear = cpuinfer_ext.linear.Linear(config) input = torch.randn((2, input_size), dtype=torch.bfloat16).contiguous() output = torch.zeros((2, output_size), dtype=torch.bfloat16).contiguous() for i in range(2): CPUInfer.submit(linear.forward, input[i,:].data_ptr(), output[i,:].data_ptr()) CPUInfer.sync() print(output) out = F.linear(input,tensor_fp32) print(out) ```

bsakaguchi closed this issue

2026-10-29 14:32:40 +00:00

bsakaguchi commented

2026-10-29 14:32:40 +00:00

Author

Owner

@chenht2022 commented on GitHub (Aug 5, 2024):

I ran your script and got the following results, which seem to be fine.

(base) chenht@sapphire2:~/ktransformers-dev$ python test.py 
[151   3 227 ...   8 169 135]
33030144
torch.Size([33030144])
torch.Size([4096, 14336])
tensor([[ 0.2041, -3.1406, -0.9922,  ..., -1.1406,  0.3828,  0.4023],
        [-1.7188,  0.3164,  0.5352,  ...,  1.5000,  1.5234,  1.0547]],
       dtype=torch.bfloat16)
tensor([[ 0.2070, -3.1562, -0.9883,  ..., -1.1484,  0.3828,  0.3984],
        [-1.7109,  0.3262,  0.5469,  ...,  1.5000,  1.5156,  1.0547]],
       dtype=torch.bfloat16)

Maybe you should check if your gguf file is intact and if the ffn_down weights are indeed q4_k quantized.
If you confirm that these two aspects are fine, please provide more information (e.g., use lscpu to show your CPU instruction set) to help us fix it.

@chenht2022 commented on GitHub (Aug 5, 2024): I ran your script and got the following results, which seem to be fine. ``` (base) chenht@sapphire2:~/ktransformers-dev$ python test.py [151 3 227 ... 8 169 135] 33030144 torch.Size([33030144]) torch.Size([4096, 14336]) tensor([[ 0.2041, -3.1406, -0.9922, ..., -1.1406, 0.3828, 0.4023], [-1.7188, 0.3164, 0.5352, ..., 1.5000, 1.5234, 1.0547]], dtype=torch.bfloat16) tensor([[ 0.2070, -3.1562, -0.9883, ..., -1.1484, 0.3828, 0.3984], [-1.7109, 0.3262, 0.5469, ..., 1.5000, 1.5156, 1.0547]], dtype=torch.bfloat16) ``` Maybe you should check if your gguf file is intact and if the ffn_down weights are indeed q4_k quantized. If you confirm that these two aspects are fine, please provide more information (e.g., use `lscpu` to show your CPU instruction set) to help us fix it.

bsakaguchi commented

2026-10-29 14:32:40 +00:00

Author

Owner

@Eutenacity commented on GitHub (Aug 5, 2024):

Many thanks for your quick response. I found the reason. The gguf I download has a different ggml_type down ffn compared to up ffn.

@Eutenacity commented on GitHub (Aug 5, 2024): Many thanks for your quick response. I found the reason. The gguf I download has a different ggml_type down ffn compared to up ffn.

bsakaguchi referenced this issue

2026-10-29 14:37:56 +00:00

[GH-ISSUE #465] /pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion probability tensor contains either inf, nan or element < 0 failed. #327

bsakaguchi referenced this issue

2026-10-29 14:42:22 +00:00

[GH-ISSUE #903] Rocm run deepseek r1 671b q4 failed #646

bsakaguchi referenced this issue

2026-10-29 14:45:29 +00:00

[GH-ISSUE #1269] [Question] benchmark with evalscope failed #859

bsakaguchi referenced this issue

2026-10-29 14:45:38 +00:00

[GH-ISSUE #1286] [Bug] inference_context.v_cache[0][i] IndexError: list index out of range #871