[GH-ISSUE #1002] [Bug] cannot import name 'shard_checkpoint' from 'transformers.modeling_utils' #711

New issue

Closed

opened 2026-10-29 14:43:13 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented 2026-10-29 14:43:13 +00:00

Owner

Copy link

Originally created by @carcoonzyk on GitHub (Mar 31, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1002

检查清单

• 1. 我已经搜索过相关问题，但未能获得预期的帮助
• 2. 该问题在最新版本中尚未修复
• 3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性
• 4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭
• 5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭

问题描述

报错：

2025-03-31 22:14:59,434 xinference.core.worker 3156 ERROR    Failed to load model QwQ-32B-0
Traceback (most recent call last):
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\worker.py", line 926, in launch_builtin_model
    await model_ref.load()
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\context.py", line 231, in send
    return self._process_result_message(result)
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\context.py", line 102, in _process_result_message
    raise message.as_instanceof_cause()
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\pool.py", line 667, in send
    result = await self._run_coro(message.message_id, coro)
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\pool.py", line 370, in _run_coro
    return await coro
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\api.py", line 384, in __on_receive__
    return await super().__on_receive__(message)  # type: ignore
  File "xoscar\\core.pyx", line 558, in __on_receive__
    raise ex
  File "xoscar\\core.pyx", line 520, in xoscar.core._BaseActor.__on_receive__
    async with self._lock:
  File "xoscar\\core.pyx", line 521, in xoscar.core._BaseActor.__on_receive__
    with debug_async_timeout('actor_lock_timeout',
  File "xoscar\\core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
    result = await result
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\model.py", line 466, in load
    self._model.load()
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\model\llm\transformers\core.py", line 687, in load
    super().load()
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\model\llm\transformers\core.py", line 338, in load
    self._model, self._tokenizer = self._load_model(**kwargs)
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\model\llm\transformers\core.py", line 210, in _load_model
    model = AutoModelForCausalLM.from_pretrained(
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\models\auto\auto_factory.py", line 573, in from_pretrained
    return model_class.from_pretrained(
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py", line 272, in _wrapper
    return func(*args, **kwargs)
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py", line 4422, in from_pretrained
    hf_quantizer.preprocess_model(
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\quantizers\base.py", line 215, in preprocess_model
    return self._process_model_before_weight_loading(model, **kwargs)
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\quantizers\quantizer_awq.py", line 108, in _process_model_before_weight_loading
    model, has_been_replaced = replace_with_awq_linear(
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\integrations\awq.py", line 134, in replace_with_awq_linear
    from awq.modules.linear.gemm import WQLinear_GEMM
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\__init__.py", line 2, in <module>
    from awq.models.auto import AutoAWQForCausalLM
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\models\__init__.py", line 1, in <module>
    from .mpt import MptAWQForCausalLM
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\models\mpt.py", line 1, in <module>
    from .base import BaseAWQForCausalLM
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\models\base.py", line 13, in <module>
    from transformers.modeling_utils import shard_checkpoint
ImportError: [address=192.168.2.2:62688, pid=50444] cannot import name 'shard_checkpoint' from 'transformers.modeling_utils' (F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py)
2025-03-31 22:14:59,474 xinference.core.worker 3156 ERROR    [request 81517f6c-0e3a-11f0-9260-bcf4d4c164ad] Leave launch_builtin_model, error: [address=192.168.2.2:62688, pid=50444] cannot import name 'shard_checkpoint' from 'transformers.modeling_utils' (F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py), elapsed time: 10 s
Traceback (most recent call last):
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\utils.py", line 93, in wrapped
    ret = await func(*args, **kwargs)
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\worker.py", line 926, in launch_builtin_model
    await model_ref.load()
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\context.py", line 231, in send
    return self._process_result_message(result)
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\context.py", line 102, in _process_result_message
    raise message.as_instanceof_cause()
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\pool.py", line 667, in send
    result = await self._run_coro(message.message_id, coro)
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\pool.py", line 370, in _run_coro
    return await coro
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\api.py", line 384, in __on_receive__
    return await super().__on_receive__(message)  # type: ignore
  File "xoscar\\core.pyx", line 558, in __on_receive__
    raise ex
  File "xoscar\\core.pyx", line 520, in xoscar.core._BaseActor.__on_receive__
    async with self._lock:
  File "xoscar\\core.pyx", line 521, in xoscar.core._BaseActor.__on_receive__
    with debug_async_timeout('actor_lock_timeout',
  File "xoscar\\core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
    result = await result
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\model.py", line 466, in load
    self._model.load()
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\model\llm\transformers\core.py", line 687, in load
    super().load()
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\model\llm\transformers\core.py", line 338, in load
    self._model, self._tokenizer = self._load_model(**kwargs)
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\model\llm\transformers\core.py", line 210, in _load_model
    model = AutoModelForCausalLM.from_pretrained(
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\models\auto\auto_factory.py", line 573, in from_pretrained
    return model_class.from_pretrained(
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py", line 272, in _wrapper
    return func(*args, **kwargs)
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py", line 4422, in from_pretrained
    hf_quantizer.preprocess_model(
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\quantizers\base.py", line 215, in preprocess_model
    return self._process_model_before_weight_loading(model, **kwargs)
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\quantizers\quantizer_awq.py", line 108, in _process_model_before_weight_loading
    model, has_been_replaced = replace_with_awq_linear(
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\integrations\awq.py", line 134, in replace_with_awq_linear
    from awq.modules.linear.gemm import WQLinear_GEMM
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\__init__.py", line 2, in <module>
    from awq.models.auto import AutoAWQForCausalLM
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\models\__init__.py", line 1, in <module>
    from .mpt import MptAWQForCausalLM
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\models\mpt.py", line 1, in <module>
    from .base import BaseAWQForCausalLM
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\models\base.py", line 13, in <module>
    from transformers.modeling_utils import shard_checkpoint
ImportError: [address=192.168.2.2:62688, pid=50444] cannot import name 'shard_checkpoint' from 'transformers.modeling_utils' (F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py)
2025-03-31 22:14:59,478 xinference.api.restful_api 50240 ERROR    [address=192.168.2.2:62688, pid=50444] cannot import name 'shard_checkpoint' from 'transformers.modeling_utils' (F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py)
Traceback (most recent call last):
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\api\restful_api.py", line 1002, in launch_model
    model_uid = await (await self._get_supervisor_ref()).launch_builtin_model(
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\context.py", line 231, in send
    return self._process_result_message(result)
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\context.py", line 102, in _process_result_message
    raise message.as_instanceof_cause()
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\pool.py", line 667, in send
    result = await self._run_coro(message.message_id, coro)
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\pool.py", line 370, in _run_coro
    return await coro
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\api.py", line 384, in __on_receive__
    return await super().__on_receive__(message)  # type: ignore
  File "xoscar\\core.pyx", line 558, in __on_receive__
    raise ex
  File "xoscar\\core.pyx", line 520, in xoscar.core._BaseActor.__on_receive__
    async with self._lock:
  File "xoscar\\core.pyx", line 521, in xoscar.core._BaseActor.__on_receive__
    with debug_async_timeout('actor_lock_timeout',
  File "xoscar\\core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
    result = await result
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\supervisor.py", line 1190, in launch_builtin_model
    await _launch_model()
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\supervisor.py", line 1125, in _launch_model
    subpool_address = await _launch_one_model(
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\supervisor.py", line 1083, in _launch_one_model
    subpool_address = await worker_ref.launch_builtin_model(
  File "xoscar\\core.pyx", line 284, in __pyx_actor_method_wrapper
    async with lock:
  File "xoscar\\core.pyx", line 287, in xoscar.core.__pyx_actor_method_wrapper
    result = await result
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\utils.py", line 93, in wrapped
    ret = await func(*args, **kwargs)
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\worker.py", line 926, in launch_builtin_model
    await model_ref.load()
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\context.py", line 231, in send
    return self._process_result_message(result)
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\context.py", line 102, in _process_result_message
    raise message.as_instanceof_cause()
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\pool.py", line 667, in send
    result = await self._run_coro(message.message_id, coro)
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\pool.py", line 370, in _run_coro
    return await coro
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\api.py", line 384, in __on_receive__
    return await super().__on_receive__(message)  # type: ignore
  File "xoscar\\core.pyx", line 558, in __on_receive__
    raise ex
  File "xoscar\\core.pyx", line 520, in xoscar.core._BaseActor.__on_receive__
    async with self._lock:
  File "xoscar\\core.pyx", line 521, in xoscar.core._BaseActor.__on_receive__
    with debug_async_timeout('actor_lock_timeout',
  File "xoscar\\core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__
    result = await result
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\model.py", line 466, in load
    self._model.load()
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\model\llm\transformers\core.py", line 687, in load
    super().load()
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\model\llm\transformers\core.py", line 338, in load
    self._model, self._tokenizer = self._load_model(**kwargs)
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\model\llm\transformers\core.py", line 210, in _load_model
    model = AutoModelForCausalLM.from_pretrained(
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\models\auto\auto_factory.py", line 573, in from_pretrained
    return model_class.from_pretrained(
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py", line 272, in _wrapper
    return func(*args, **kwargs)
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py", line 4422, in from_pretrained
    hf_quantizer.preprocess_model(
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\quantizers\base.py", line 215, in preprocess_model
    return self._process_model_before_weight_loading(model, **kwargs)
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\quantizers\quantizer_awq.py", line 108, in _process_model_before_weight_loading
    model, has_been_replaced = replace_with_awq_linear(
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\integrations\awq.py", line 134, in replace_with_awq_linear
    from awq.modules.linear.gemm import WQLinear_GEMM
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\__init__.py", line 2, in <module>
    from awq.models.auto import AutoAWQForCausalLM
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\models\__init__.py", line 1, in <module>
    from .mpt import MptAWQForCausalLM
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\models\mpt.py", line 1, in <module>
    from .base import BaseAWQForCausalLM
  File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\models\base.py", line 13, in <module>
    from transformers.modeling_utils import shard_checkpoint
ImportError: [address=192.168.2.2:62688, pid=50444] cannot import name 'shard_checkpoint' from 'transformers.modeling_utils' (F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py)

复现步骤

在gui界面：选择qwq-32B
使用transformers awq int4 gpux2

环境信息

winserer2025
gpux2 t4
mamba create -n xinference python=3.9
pip install "xinference[transformers]"
pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu126

Originally created by @carcoonzyk on GitHub (Mar 31, 2025). Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1002 ### 检查清单 - [x] 1. 我已经搜索过相关问题，但未能获得预期的帮助 - [x] 2. 该问题在最新版本中尚未修复 - [x] 3. 请注意，如果您提交的BUG相关 issue 缺少对应环境信息和最小可复现示例，我们将难以复现和定位问题，降低获得反馈的可能性 - [x] 4. 如果您提出的不是bug而是问题，请在讨论区发起讨论 https://github.com/kvcache-ai/ktransformers/discussions。否则该 issue 将被关闭 - [x] 5. 为方便社区交流，我将使用中文/英文或附上中文/英文翻译（如使用其他语言）。未附带翻译的非中文/英语内容可能会被关闭 ### 问题描述 报错： ``` 2025-03-31 22:14:59,434 xinference.core.worker 3156 ERROR Failed to load model QwQ-32B-0 Traceback (most recent call last): File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\worker.py", line 926, in launch_builtin_model await model_ref.load() File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\context.py", line 231, in send return self._process_result_message(result) File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\context.py", line 102, in _process_result_message raise message.as_instanceof_cause() File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\pool.py", line 667, in send result = await self._run_coro(message.message_id, coro) File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\pool.py", line 370, in _run_coro return await coro File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\api.py", line 384, in __on_receive__ return await super().__on_receive__(message) # type: ignore File "xoscar\\core.pyx", line 558, in __on_receive__ raise ex File "xoscar\\core.pyx", line 520, in xoscar.core._BaseActor.__on_receive__ async with self._lock: File "xoscar\\core.pyx", line 521, in xoscar.core._BaseActor.__on_receive__ with debug_async_timeout('actor_lock_timeout', File "xoscar\\core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__ result = await result File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\model.py", line 466, in load self._model.load() File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\model\llm\transformers\core.py", line 687, in load super().load() File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\model\llm\transformers\core.py", line 338, in load self._model, self._tokenizer = self._load_model(**kwargs) File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\model\llm\transformers\core.py", line 210, in _load_model model = AutoModelForCausalLM.from_pretrained( File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\models\auto\auto_factory.py", line 573, in from_pretrained return model_class.from_pretrained( File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py", line 272, in _wrapper return func(*args, **kwargs) File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py", line 4422, in from_pretrained hf_quantizer.preprocess_model( File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\quantizers\base.py", line 215, in preprocess_model return self._process_model_before_weight_loading(model, **kwargs) File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\quantizers\quantizer_awq.py", line 108, in _process_model_before_weight_loading model, has_been_replaced = replace_with_awq_linear( File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\integrations\awq.py", line 134, in replace_with_awq_linear from awq.modules.linear.gemm import WQLinear_GEMM File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\__init__.py", line 2, in <module> from awq.models.auto import AutoAWQForCausalLM File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\models\__init__.py", line 1, in <module> from .mpt import MptAWQForCausalLM File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\models\mpt.py", line 1, in <module> from .base import BaseAWQForCausalLM File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\models\base.py", line 13, in <module> from transformers.modeling_utils import shard_checkpoint ImportError: [address=192.168.2.2:62688, pid=50444] cannot import name 'shard_checkpoint' from 'transformers.modeling_utils' (F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py) 2025-03-31 22:14:59,474 xinference.core.worker 3156 ERROR [request 81517f6c-0e3a-11f0-9260-bcf4d4c164ad] Leave launch_builtin_model, error: [address=192.168.2.2:62688, pid=50444] cannot import name 'shard_checkpoint' from 'transformers.modeling_utils' (F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py), elapsed time: 10 s Traceback (most recent call last): File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\utils.py", line 93, in wrapped ret = await func(*args, **kwargs) File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\worker.py", line 926, in launch_builtin_model await model_ref.load() File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\context.py", line 231, in send return self._process_result_message(result) File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\context.py", line 102, in _process_result_message raise message.as_instanceof_cause() File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\pool.py", line 667, in send result = await self._run_coro(message.message_id, coro) File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\pool.py", line 370, in _run_coro return await coro File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\api.py", line 384, in __on_receive__ return await super().__on_receive__(message) # type: ignore File "xoscar\\core.pyx", line 558, in __on_receive__ raise ex File "xoscar\\core.pyx", line 520, in xoscar.core._BaseActor.__on_receive__ async with self._lock: File "xoscar\\core.pyx", line 521, in xoscar.core._BaseActor.__on_receive__ with debug_async_timeout('actor_lock_timeout', File "xoscar\\core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__ result = await result File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\model.py", line 466, in load self._model.load() File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\model\llm\transformers\core.py", line 687, in load super().load() File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\model\llm\transformers\core.py", line 338, in load self._model, self._tokenizer = self._load_model(**kwargs) File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\model\llm\transformers\core.py", line 210, in _load_model model = AutoModelForCausalLM.from_pretrained( File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\models\auto\auto_factory.py", line 573, in from_pretrained return model_class.from_pretrained( File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py", line 272, in _wrapper return func(*args, **kwargs) File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py", line 4422, in from_pretrained hf_quantizer.preprocess_model( File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\quantizers\base.py", line 215, in preprocess_model return self._process_model_before_weight_loading(model, **kwargs) File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\quantizers\quantizer_awq.py", line 108, in _process_model_before_weight_loading model, has_been_replaced = replace_with_awq_linear( File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\integrations\awq.py", line 134, in replace_with_awq_linear from awq.modules.linear.gemm import WQLinear_GEMM File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\__init__.py", line 2, in <module> from awq.models.auto import AutoAWQForCausalLM File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\models\__init__.py", line 1, in <module> from .mpt import MptAWQForCausalLM File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\models\mpt.py", line 1, in <module> from .base import BaseAWQForCausalLM File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\models\base.py", line 13, in <module> from transformers.modeling_utils import shard_checkpoint ImportError: [address=192.168.2.2:62688, pid=50444] cannot import name 'shard_checkpoint' from 'transformers.modeling_utils' (F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py) 2025-03-31 22:14:59,478 xinference.api.restful_api 50240 ERROR [address=192.168.2.2:62688, pid=50444] cannot import name 'shard_checkpoint' from 'transformers.modeling_utils' (F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py) Traceback (most recent call last): File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\api\restful_api.py", line 1002, in launch_model model_uid = await (await self._get_supervisor_ref()).launch_builtin_model( File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\context.py", line 231, in send return self._process_result_message(result) File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\context.py", line 102, in _process_result_message raise message.as_instanceof_cause() File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\pool.py", line 667, in send result = await self._run_coro(message.message_id, coro) File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\pool.py", line 370, in _run_coro return await coro File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\api.py", line 384, in __on_receive__ return await super().__on_receive__(message) # type: ignore File "xoscar\\core.pyx", line 558, in __on_receive__ raise ex File "xoscar\\core.pyx", line 520, in xoscar.core._BaseActor.__on_receive__ async with self._lock: File "xoscar\\core.pyx", line 521, in xoscar.core._BaseActor.__on_receive__ with debug_async_timeout('actor_lock_timeout', File "xoscar\\core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__ result = await result File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\supervisor.py", line 1190, in launch_builtin_model await _launch_model() File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\supervisor.py", line 1125, in _launch_model subpool_address = await _launch_one_model( File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\supervisor.py", line 1083, in _launch_one_model subpool_address = await worker_ref.launch_builtin_model( File "xoscar\\core.pyx", line 284, in __pyx_actor_method_wrapper async with lock: File "xoscar\\core.pyx", line 287, in xoscar.core.__pyx_actor_method_wrapper result = await result File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\utils.py", line 93, in wrapped ret = await func(*args, **kwargs) File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\worker.py", line 926, in launch_builtin_model await model_ref.load() File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\context.py", line 231, in send return self._process_result_message(result) File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\context.py", line 102, in _process_result_message raise message.as_instanceof_cause() File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\pool.py", line 667, in send result = await self._run_coro(message.message_id, coro) File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\backends\pool.py", line 370, in _run_coro return await coro File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xoscar\api.py", line 384, in __on_receive__ return await super().__on_receive__(message) # type: ignore File "xoscar\\core.pyx", line 558, in __on_receive__ raise ex File "xoscar\\core.pyx", line 520, in xoscar.core._BaseActor.__on_receive__ async with self._lock: File "xoscar\\core.pyx", line 521, in xoscar.core._BaseActor.__on_receive__ with debug_async_timeout('actor_lock_timeout', File "xoscar\\core.pyx", line 526, in xoscar.core._BaseActor.__on_receive__ result = await result File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\core\model.py", line 466, in load self._model.load() File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\model\llm\transformers\core.py", line 687, in load super().load() File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\model\llm\transformers\core.py", line 338, in load self._model, self._tokenizer = self._load_model(**kwargs) File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\xinference\model\llm\transformers\core.py", line 210, in _load_model model = AutoModelForCausalLM.from_pretrained( File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\models\auto\auto_factory.py", line 573, in from_pretrained return model_class.from_pretrained( File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py", line 272, in _wrapper return func(*args, **kwargs) File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py", line 4422, in from_pretrained hf_quantizer.preprocess_model( File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\quantizers\base.py", line 215, in preprocess_model return self._process_model_before_weight_loading(model, **kwargs) File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\quantizers\quantizer_awq.py", line 108, in _process_model_before_weight_loading model, has_been_replaced = replace_with_awq_linear( File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\integrations\awq.py", line 134, in replace_with_awq_linear from awq.modules.linear.gemm import WQLinear_GEMM File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\__init__.py", line 2, in <module> from awq.models.auto import AutoAWQForCausalLM File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\models\__init__.py", line 1, in <module> from .mpt import MptAWQForCausalLM File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\models\mpt.py", line 1, in <module> from .base import BaseAWQForCausalLM File "F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\awq\models\base.py", line 13, in <module> from transformers.modeling_utils import shard_checkpoint ImportError: [address=192.168.2.2:62688, pid=50444] cannot import name 'shard_checkpoint' from 'transformers.modeling_utils' (F:\ProgramData\miniconda3\envs\xinference\lib\site-packages\transformers\modeling_utils.py) ``` ### 复现步骤 在gui界面：选择qwq-32B 使用transformers awq int4 gpux2 ### 环境信息 winserer2025 gpux2 t4 mamba create -n xinference python=3.9 pip install "xinference[transformers]" pip install torch torchvision torchaudio --extra-index-url https://download.pytorch.org/whl/cu126

bsakaguchi closed this issue 2026-10-29 14:43:13 +00:00

bsakaguchi commented 2026-10-29 14:43:14 +00:00

Author

Owner

Copy link

@Atream commented on GitHub (Apr 2, 2025):

This seems to be Xinference rather than KTransformers.

<!-- gh-comment-id:2771501459 --> @Atream commented on GitHub (Apr 2, 2025): This seems to be Xinference rather than KTransformers.

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

auto/sync-sglang

gh-pages

docs/kt-ft-figure2-single-column-20261010

perf/bf16-grad-norm-20261010

docs/kt-ft-paper-20261008

release/20260911-carrier-repair

release/20260911-ci-python

release/20260911-pypi

ci/four-main-pypi-release

ci/model-e2e-acceptance

release/0.7.0.post3-frozen-pypi

fix/glm53-index-share-sync

release/0.7.0.post2-frozen-pypi

hotfix/0.7.0.post2-pypi-runtime

fix/avx2-wheel-feature-isolation

fix/release-self-hosted-runner

release/automate-kt-sft-pypi

support-glm5-next-djw

docs/deepseek-v4-flash-docker

ovowei-patch-3

support-glm52-djw

doc-reorg-entrypoints-support-matrix

bump-sglang-pr31

docs-v061-refresh

fp4-moe-amx

flatten-ktransformers-shim

archive-kt-sft

kt-kernel-torch-range-v061

sft-whl

sft

docs/gosim-2026-and-roadmap-link

ovowei-patch-2

ovowei-patch-1

revert-1918-fix/rocm-cpuinfer-stream-bridge

chore/kt-layerwise-prefill-main

chore/kt-layerwise-prefill-label

sft-rel

support-glm51-djw

release/v0.5.3

chore/kt-run-numa-nodes-unified

mtp

ksft-sglang

avx2

fix/sglang-kt-detection

layerwise-prefill-opt

feat/mistral-loader-support

fix-docker-build

minimax-m2.5

qwen3.5

add-maintainers

oql-doc

cli-dev-ljq

ljq-cli-dev-new

JimmyPeilinLi-patch-2

kimi_k2.5

oql-dev

experts_sched

JimmyPeilinLi-patch-1

ErvinXie-patch-1

ksft-llama

glm47_fp8

debug_llama

refactor_sft

ljq-cli-dev

refactor_sft_int4_debug

lut-nvfp4

fix-import

fix-deploy-djw

cpu-fix

ljq-dev-1

route-moe

sosp25-ae

v0.7.2

paper-ktft-v1.1

paper-ktft-v1

v0.7.1

v0.7.0.post4

v0.7.0.post1

v0.7.0

v0.6.4

v0.6.3.post1

v0.6.3

v0.6.2.post3

v0.6.2.post2

v0.6.2.post1

v0.6.2

v0.6.1

v0.6.1.post1

v0.5.3

v0.5.2.post1

v0.5.1

v0.5.0.post1

v0.5.0

v0.4.4

v0.4.3

v0.4.2

v0.4.1

v0.3.2

v0.3.1

v0.3

v0.2.4post1

v0.2.4

v0.2.3post2

v0.2.3post1

v0.2.3

v0.2.2

v0.2.2rc2

v0.2.2rc1

v0.2.1.post1

v0.2.1

v0.2.0

v0.1.4

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

bug

  

duplicate

  

enhancement

  

good first issue

  

help wanted

  

pull-request

Mirrored from GitHub Pull Request

  

stale

No labels

bug

duplicate

enhancement

good first issue

help wanted

pull-request

stale

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

kvcache-ai/ktransformers#711

No description provided.