[GH-ISSUE #28] using docker got errors

bsakaguchi commented

2026-10-29 14:32:41 +00:00

Owner

Originally created by @goldenquant on GitHub (Aug 11, 2024).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/28

从 https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct 下载了gguf
根据 Docker.md 启动docker，
命令：

docker run --gpus all -v /data/models_dir/gguf/:/models -p 10002:10002 approachingai/ktransformers:0.1.1 --port 10002 --gguf_path /models/DeepSeek-V2-Coder --model_path /models/DeepSeek-V2-Coder --web True

报如下错误

Traceback (most recent call last):
  File "/opt/conda/bin/ktransformers", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 134, in main
    create_interface(config=cfg, default_args=default_args)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface
    GlobalInterface.interface = BackendInterface(default_args)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 20, in __init__
    self.tokenizer = AutoTokenizer.from_pretrained(args.model_dir,device = args.device)
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 853, in from_pretrained
    config = AutoConfig.from_pretrained(
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 972, in from_pretrained
    config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/configuration_utils.py", line 632, in get_config_dict
    config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs)
  File "/opt/conda/lib/python3.10/site-packages/transformers/configuration_utils.py", line 689, in _get_config_dict
    resolved_config_file = cached_file(
  File "/opt/conda/lib/python3.10/site-packages/transformers/utils/hub.py", line 373, in cached_file
    raise EnvironmentError(
OSError: /models/DeepSeek-V2-Coder does not appear to have a file named config.json. Checkout 'https://huggingface.co//models/DeepSeek-V2-Coder/tree/None' for available files.

去这个链接下面下载了config.json后，报错如下：

Traceback (most recent call last):
  File "/opt/conda/bin/ktransformers", line 8, in <module>
    sys.exit(main())
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 134, in main
    create_interface(config=cfg, default_args=default_args)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface
    GlobalInterface.interface = BackendInterface(default_args)
  File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 20, in __init__
    self.tokenizer = AutoTokenizer.from_pretrained(args.model_dir,device = args.device)
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 853, in from_pretrained
    config = AutoConfig.from_pretrained(
  File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 975, in from_pretrained
    trust_remote_code = resolve_trust_remote_code(
  File "/opt/conda/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 626, in resolve_trust_remote_code
    raise ValueError(
ValueError: The repository for /models/DeepSeek-V2-Coder contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co//models/DeepSeek-V2-Coder.
Please pass the argument `trust_remote_code=True` to allow custom code to be run.

Originally created by @goldenquant on GitHub (Aug 11, 2024). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/28 从 https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct 下载了gguf 根据 [Docker.md](https://github.com/kvcache-ai/ktransformers/blob/main/doc/en/Docker.md) 启动docker，命令： ``` docker run --gpus all -v /data/models_dir/gguf/:/models -p 10002:10002 approachingai/ktransformers:0.1.1 --port 10002 --gguf_path /models/DeepSeek-V2-Coder --model_path /models/DeepSeek-V2-Coder --web True ``` 报如下错误 ``` Traceback (most recent call last): File "/opt/conda/bin/ktransformers", line 8, in <module> sys.exit(main()) File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 134, in main create_interface(config=cfg, default_args=default_args) File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface GlobalInterface.interface = BackendInterface(default_args) File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 20, in __init__ self.tokenizer = AutoTokenizer.from_pretrained(args.model_dir,device = args.device) File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 853, in from_pretrained config = AutoConfig.from_pretrained( File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 972, in from_pretrained config_dict, unused_kwargs = PretrainedConfig.get_config_dict(pretrained_model_name_or_path, **kwargs) File "/opt/conda/lib/python3.10/site-packages/transformers/configuration_utils.py", line 632, in get_config_dict config_dict, kwargs = cls._get_config_dict(pretrained_model_name_or_path, **kwargs) File "/opt/conda/lib/python3.10/site-packages/transformers/configuration_utils.py", line 689, in _get_config_dict resolved_config_file = cached_file( File "/opt/conda/lib/python3.10/site-packages/transformers/utils/hub.py", line 373, in cached_file raise EnvironmentError( OSError: /models/DeepSeek-V2-Coder does not appear to have a file named config.json. Checkout 'https://huggingface.co//models/DeepSeek-V2-Coder/tree/None' for available files. ``` 去这个链接下面下载了config.json后，报错如下： ``` Traceback (most recent call last): File "/opt/conda/bin/ktransformers", line 8, in <module> sys.exit(main()) File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/main.py", line 134, in main create_interface(config=cfg, default_args=default_args) File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/utils/create_interface.py", line 27, in create_interface GlobalInterface.interface = BackendInterface(default_args) File "/opt/conda/lib/python3.10/site-packages/ktransformers/server/backend/interfaces/ktransformers.py", line 20, in __init__ self.tokenizer = AutoTokenizer.from_pretrained(args.model_dir,device = args.device) File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/tokenization_auto.py", line 853, in from_pretrained config = AutoConfig.from_pretrained( File "/opt/conda/lib/python3.10/site-packages/transformers/models/auto/configuration_auto.py", line 975, in from_pretrained trust_remote_code = resolve_trust_remote_code( File "/opt/conda/lib/python3.10/site-packages/transformers/dynamic_module_utils.py", line 626, in resolve_trust_remote_code raise ValueError( ValueError: The repository for /models/DeepSeek-V2-Coder contains custom code which must be executed to correctly load the model. You can inspect the repository content at https://hf.co//models/DeepSeek-V2-Coder. Please pass the argument `trust_remote_code=True` to allow custom code to be run. ```

bsakaguchi closed this issue

2026-10-29 14:32:41 +00:00

bsakaguchi commented

2026-10-29 14:32:42 +00:00

Author

Owner

@UnicornChan commented on GitHub (Aug 12, 2024):

Perhaps you can download the GGUF file from this link: https://huggingface.co/LoneStriker/DeepSeek-Coder-V2-Instruct-GGUF/tree/main
On the https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct, there are only SafeTensor format files and the model's configs.
about input parameter --gguf_path, there is only one GGUF format file placed in the directory.

#if your gguf path input as : --gguf_path /data/model/qwen2-57b-a14b-instruct-q4_k_m/
ls /data/model/qwen2-57b-a14b-instruct-q4_k_m/
#output
qwen2-57b-a14b-instruct-q4_k_m.gguf

about input parameter --model_path, it is where all the config files, excluding the SafeTensor model file, are stored.

#your model path input as : --model_path /data/model/Qwen2-57B-A14B-Instruct/
ls /data/model/Qwen2-57B-A14B-Instruct/
# output
config.json  generation_config.json  special_tokens_map.json  tokenizer_config.json  tokenizer.json  vocab.json

you can download all configs of model "DeepSeek-Coder-V2-Instruct" with GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/deepseek-ai/DeepSeek-Coder-V2-Instruct.git

@UnicornChan commented on GitHub (Aug 12, 2024): Perhaps you can download the GGUF file from this link: https://huggingface.co/LoneStriker/DeepSeek-Coder-V2-Instruct-GGUF/tree/main On the https://huggingface.co/deepseek-ai/DeepSeek-Coder-V2-Instruct, there are only SafeTensor format files and the model's configs. about input parameter --gguf_path, there is only one GGUF format file placed in the directory. ``` #if your gguf path input as : --gguf_path /data/model/qwen2-57b-a14b-instruct-q4_k_m/ ls /data/model/qwen2-57b-a14b-instruct-q4_k_m/ #output qwen2-57b-a14b-instruct-q4_k_m.gguf ``` about input parameter --model_path, it is where all the config files, excluding the SafeTensor model file, are stored. ``` #your model path input as : --model_path /data/model/Qwen2-57B-A14B-Instruct/ ls /data/model/Qwen2-57B-A14B-Instruct/ # output config.json generation_config.json special_tokens_map.json tokenizer_config.json tokenizer.json vocab.json ``` you can download all configs of model "DeepSeek-Coder-V2-Instruct" with ```GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/deepseek-ai/DeepSeek-Coder-V2-Instruct.git```

bsakaguchi commented

2026-10-29 14:32:42 +00:00

Author

Owner

@UnicornChan commented on GitHub (Aug 12, 2024):

I recommend starting with a smaller model first:
gguf file in：https://huggingface.co/mzwing/DeepSeek-V2-Lite-Chat-GGUF/tree/main
model config file in: https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite/tree/main (you can download by GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/deepseek-ai/DeepSeek-V2-Lite-Chat.git)

@UnicornChan commented on GitHub (Aug 12, 2024): I recommend starting with a smaller model first: gguf file in：https://huggingface.co/mzwing/DeepSeek-V2-Lite-Chat-GGUF/tree/main model config file in: https://huggingface.co/deepseek-ai/DeepSeek-V2-Lite/tree/main (you can download by ```GIT_LFS_SKIP_SMUDGE=1 git clone https://www.modelscope.cn/deepseek-ai/DeepSeek-V2-Lite-Chat.git```)

bsakaguchi commented

2026-10-29 14:32:42 +00:00

Author

Owner

@goldenquant commented on GitHub (Aug 12, 2024):

solved!thx.

@goldenquant commented on GitHub (Aug 12, 2024): solved!thx.

bsakaguchi referenced this issue

2026-10-29 14:42:22 +00:00

[GH-ISSUE #903] Rocm run deepseek r1 671b q4 failed #646

bsakaguchi referenced this issue

2026-10-29 14:45:29 +00:00