[GH-ISSUE #37] Ubuntu 24.04 GLIBCXX version fail

bsakaguchi commented

2026-10-29 14:32:43 +00:00

Owner

Originally created by @ELigoP on GitHub (Aug 15, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/37

Originally assigned to: @UnicornChan, @chenht2022 on GitHub.

I follow README.md until section https://github.com/kvcache-ai/KTransformers?tab=readme-ov-file#-supported-model
If I run default command from section, I get

ERROR: The function received no value for the required argument: model_path
Usage: local_chat.py MODEL_PATH <flags>
  optional flags:        --optimize_rule_path | --gguf_path |
                         --max_new_tokens | --cpu_infer | --use_cuda_graph

For detailed information on this command, run:
  local_chat.py --help

I then run commands from from bartowski/DeepSeek-V2-Chat-0628-GGUF corresponding section, but with --model_path instead of --model_name:

(ktransformers) user@hostname:~/3rdparty/KTransformers$ python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-V2-Chat-0628 --gguf_path ./models/bartowski/DeepSeek-V2-Chat-0628-GGUF

and I get the following output:

using custom modeling_xxx.py.
using default_optimize_rule for DeepseekV2ForCausalLM
Injecting model as default
Injecting model.embed_tokens as default
Injecting model.layers as default
Injecting model.layers.0 as default
Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention
Injecting model.layers.0.self_attn.q_a_proj as default
Injecting model.layers.0.self_attn.q_a_layernorm as default
Injecting model.layers.0.self_attn.q_b_proj as default
Injecting model.layers.0.self_attn.kv_a_proj_with_mqa as default
Injecting model.layers.0.self_attn.kv_a_layernorm as default
Injecting model.layers.0.self_attn.kv_b_proj as default
Injecting model.layers.0.self_attn.o_proj as default
Injecting model.layers.0.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbedding
Injecting model.layers.0.mlp as default
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/user/3rdparty/KTransformers/ktransformers/local_chat.py", line 120, in <module>
    fire.Fire(local_chat)
  File "/home/user/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 143, in Fire
    component_trace = _Fire(component, args, parsed_flag_args, context, name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 477, in _Fire
    component, remaining_args = _CallAndUpdateTrace(
                                ^^^^^^^^^^^^^^^^^^^^
  File "/home/user/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace
    component = fn(*varargs, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/3rdparty/KTransformers/ktransformers/local_chat.py", line 92, in local_chat
    optimize_and_load_gguf(model, optimize_rule_path, gguf_path, config)
  File "/home/user/3rdparty/KTransformers/ktransformers/optimize/optimize.py", line 128, in optimize_and_load_gguf
    inject(module, optimize_config, model_config, gguf_loader)
  File "/home/user/3rdparty/KTransformers/ktransformers/optimize/optimize.py", line 42, in inject
    inject(child, child_optimization_dict, model_config, gguf_loader, child_prefix)
  File "/home/user/3rdparty/KTransformers/ktransformers/optimize/optimize.py", line 42, in inject
    inject(child, child_optimization_dict, model_config, gguf_loader, child_prefix)
  File "/home/user/3rdparty/KTransformers/ktransformers/optimize/optimize.py", line 42, in inject
    inject(child, child_optimization_dict, model_config, gguf_loader, child_prefix)
  [Previous line repeated 1 more time]
  File "/home/user/3rdparty/KTransformers/ktransformers/optimize/optimize.py", line 31, in inject
    module_cls=getattr(__import__(import_module_name, fromlist=[""]), import_class_name)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/user/3rdparty/KTransformers/ktransformers/operators/linear.py", line 33, in <module>
    import cpuinfer_ext
ImportError: /home/user/miniconda3/envs/ktransformers/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /home/user/miniconda3/envs/ktransformers/lib/python3.11/site-packages/cpuinfer_ext.cpython-311-x86_64-linux-gnu.so)

I guess this is because the code expects GLibc version from Ubuntu 22?

Is there a simple workaround?

Originally created by @ELigoP on GitHub (Aug 15, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/37 Originally assigned to: @UnicornChan, @chenht2022 on GitHub. I follow README.md until section https://github.com/kvcache-ai/KTransformers?tab=readme-ov-file#-supported-model If I run default command from section, I get ``` ERROR: The function received no value for the required argument: model_path Usage: local_chat.py MODEL_PATH <flags> optional flags: --optimize_rule_path | --gguf_path | --max_new_tokens | --cpu_infer | --use_cuda_graph For detailed information on this command, run: local_chat.py --help ``` I then run commands from from `bartowski/DeepSeek-V2-Chat-0628-GGUF` corresponding section, but with --model_path instead of --model_name: ``` (ktransformers) user@hostname:~/3rdparty/KTransformers$ python -m ktransformers.local_chat --model_path deepseek-ai/DeepSeek-V2-Chat-0628 --gguf_path ./models/bartowski/DeepSeek-V2-Chat-0628-GGUF ``` and I get the following output: ``` using custom modeling_xxx.py. using default_optimize_rule for DeepseekV2ForCausalLM Injecting model as default Injecting model.embed_tokens as default Injecting model.layers as default Injecting model.layers.0 as default Injecting model.layers.0.self_attn as ktransformers.operators.attention . KDeepseekV2Attention Injecting model.layers.0.self_attn.q_a_proj as default Injecting model.layers.0.self_attn.q_a_layernorm as default Injecting model.layers.0.self_attn.q_b_proj as default Injecting model.layers.0.self_attn.kv_a_proj_with_mqa as default Injecting model.layers.0.self_attn.kv_a_layernorm as default Injecting model.layers.0.self_attn.kv_b_proj as default Injecting model.layers.0.self_attn.o_proj as default Injecting model.layers.0.self_attn.rotary_emb as ktransformers.operators.RoPE . YarnRotaryEmbedding Injecting model.layers.0.mlp as default Traceback (most recent call last): File "<frozen runpy>", line 198, in _run_module_as_main File "<frozen runpy>", line 88, in _run_code File "/home/user/3rdparty/KTransformers/ktransformers/local_chat.py", line 120, in <module> fire.Fire(local_chat) File "/home/user/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 143, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 477, in _Fire component, remaining_args = _CallAndUpdateTrace( ^^^^^^^^^^^^^^^^^^^^ File "/home/user/miniconda3/envs/ktransformers/lib/python3.11/site-packages/fire/core.py", line 693, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/3rdparty/KTransformers/ktransformers/local_chat.py", line 92, in local_chat optimize_and_load_gguf(model, optimize_rule_path, gguf_path, config) File "/home/user/3rdparty/KTransformers/ktransformers/optimize/optimize.py", line 128, in optimize_and_load_gguf inject(module, optimize_config, model_config, gguf_loader) File "/home/user/3rdparty/KTransformers/ktransformers/optimize/optimize.py", line 42, in inject inject(child, child_optimization_dict, model_config, gguf_loader, child_prefix) File "/home/user/3rdparty/KTransformers/ktransformers/optimize/optimize.py", line 42, in inject inject(child, child_optimization_dict, model_config, gguf_loader, child_prefix) File "/home/user/3rdparty/KTransformers/ktransformers/optimize/optimize.py", line 42, in inject inject(child, child_optimization_dict, model_config, gguf_loader, child_prefix) [Previous line repeated 1 more time] File "/home/user/3rdparty/KTransformers/ktransformers/optimize/optimize.py", line 31, in inject module_cls=getattr(__import__(import_module_name, fromlist=[""]), import_class_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/user/3rdparty/KTransformers/ktransformers/operators/linear.py", line 33, in <module> import cpuinfer_ext ImportError: /home/user/miniconda3/envs/ktransformers/bin/../lib/libstdc++.so.6: version `GLIBCXX_3.4.32' not found (required by /home/user/miniconda3/envs/ktransformers/lib/python3.11/site-packages/cpuinfer_ext.cpython-311-x86_64-linux-gnu.so) ``` I guess this is because the code expects GLibc version from Ubuntu 22? Is there a simple workaround?

bsakaguchi closed this issue

2026-10-29 14:32:43 +00:00

bsakaguchi commented

2026-10-29 14:32:44 +00:00

Author

Owner

@UnicornChan commented on GitHub (Aug 15, 2024):

We will test and validate this on different versions of the Linux operating system. @chenht2022

@UnicornChan commented on GitHub (Aug 15, 2024): We will test and validate this on different versions of the Linux operating system. @chenht2022

bsakaguchi commented

2026-10-29 14:32:44 +00:00

Author

Owner

@chenht2022 commented on GitHub (Aug 19, 2024):

Hello, this might be because you are running on Ubuntu 24, and the libraries in our package are from Ubuntu 22 which is not up-to-date enough. You can try the following steps to resolve the issue:

Find the library in your environment:
sudo find /usr/ -name libstdc++.so.6
Check if it contains the required version:
strings /path/to/libstdc++.so.6 | grep GLIBCXX
Replace the library in KTransformers with the one that contains the required version:
cp /path/to/libstdc++.so.6 /home/user/miniconda3/envs/ktransformers/bin/../lib/libstdc++.so.6

@chenht2022 commented on GitHub (Aug 19, 2024): Hello, this might be because you are running on Ubuntu 24, and the libraries in our package are from Ubuntu 22 which is not up-to-date enough. You can try the following steps to resolve the issue: 1. Find the library in your environment: `sudo find /usr/ -name libstdc++.so.6` 2. Check if it contains the required version: `strings /path/to/libstdc++.so.6 | grep GLIBCXX` 3. Replace the library in KTransformers with the one that contains the required version: `cp /path/to/libstdc++.so.6 /home/user/miniconda3/envs/ktransformers/bin/../lib/libstdc++.so.6`

bsakaguchi commented

2026-10-29 14:32:44 +00:00

Author

Owner

@ELigoP commented on GitHub (Aug 20, 2024):

Yes, that worked. Now I get a different error which is subject to another issue...

@ELigoP commented on GitHub (Aug 20, 2024): Yes, that worked. Now I get a different error which is subject to another issue...

bsakaguchi referenced this issue

2026-10-29 14:37:56 +00:00

[GH-ISSUE #465] /pytorch/aten/src/ATen/native/cuda/TensorCompare.cu:110: _assert_async_cuda_kernel: block: [0,0,0], thread: [0,0,0] Assertion probability tensor contains either inf, nan or element < 0 failed. #327

bsakaguchi referenced this issue

2026-10-29 14:42:22 +00:00

[GH-ISSUE #903] Rocm run deepseek r1 671b q4 failed #646

bsakaguchi referenced this issue

2026-10-29 14:45:29 +00:00

[GH-ISSUE #1269] [Question] benchmark with evalscope failed #859

bsakaguchi referenced this issue

2026-10-29 14:45:38 +00:00

[GH-ISSUE #1286] [Bug] inference_context.v_cache[0][i] IndexError: list index out of range #871