mirror of
https://github.com/kvcache-ai/ktransformers.git
synced 2026-11-05 01:07:28 +00:00
[GH-ISSUE #1536] [Bug] 使用ascend 310p基于deepseek v3 q4km量化模型推理，报错call hccl api failed，Failed to allocate memory #1026
Labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
kvcache-ai/ktransformers#1026
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @fanzetian on GitHub (Oct 29, 2025).
Original GitHub issue: https://github.com/kvcache-ai/ktransformers/issues/1536
Checklist
Describe the bug
执行python /home/ktransformers-main/ktransformers/server/main.py，使用deepseek-v3 q4km gguf模型，报错：
2025-10-29 03:01:27,790 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
found flashinfer
/usr/local/lib64/python3.11/site-packages/torch_npu/contrib/transfer_to_npu.py:298: ImportWarning:
The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
The backend in torch.distributed.init_process_group set to hccl now..
The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
The device parameters have been replaced with npu in the function below:
torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.autocast, torch.load, torch.set_default_device, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.Tensor.pin_memory, torch.nn.Module.to, torch.nn.Module.to_empty
warnings.warn(msg, ImportWarning)
/usr/local/lib64/python3.11/site-packages/torch_npu/contrib/transfer_to_npu.py:255: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
warnings.warn(msg, RuntimeWarning)
/usr/local/lib64/python3.11/site-packages/ktransformers/server/api/ollama/completions.py:257: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/
class OllamaShowResponse(BaseModel):
The module name (originally ) is not a valid Python identifier. Please rename the original module to avoid import issues.
The module name (originally ) is not a valid Python identifier. Please rename the original module to avoid import issues.
flash_attn not found, flashinfer unit test needed it. If you are using balance serve, ignore this.
set start method
Connected to server at tcp://localhost:41617
2025-10-29 03:01:36,526 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
found flashinfer
/usr/local/lib64/python3.11/site-packages/torch_npu/contrib/transfer_to_npu.py:298: ImportWarning:
The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
The backend in torch.distributed.init_process_group set to hccl now..
The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
The device parameters have been replaced with npu in the function below:
torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.autocast, torch.load, torch.set_default_device, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.Tensor.pin_memory, torch.nn.Module.to, torch.nn.Module.to_empty
warnings.warn(msg, ImportWarning)
/usr/local/lib64/python3.11/site-packages/torch_npu/contrib/transfer_to_npu.py:255: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
warnings.warn(msg, RuntimeWarning)
/usr/local/lib64/python3.11/site-packages/ktransformers/server/api/ollama/completions.py:257: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/
class OllamaShowResponse(BaseModel):
flash_attn not found, flashinfer unit test needed it. If you are using balance serve, ignore this.
start method already set to spawn
2025-10-29 03:01:48,232 - INFO - flashinfer.jit: Prebuilt kernels not found, using JIT backend
found flashinfer
/usr/local/lib64/python3.11/site-packages/torch_npu/contrib/transfer_to_npu.py:298: ImportWarning:
The torch.Tensor.cuda and torch.nn.Module.cuda are replaced with torch.Tensor.npu and torch.nn.Module.npu now..
The torch.cuda.DoubleTensor is replaced with torch.npu.FloatTensor cause the double type is not supported now..
The backend in torch.distributed.init_process_group set to hccl now..
The torch.cuda.* and torch.cuda.amp.* are replaced with torch.npu.* and torch.npu.amp.* now..
The device parameters have been replaced with npu in the function below:
torch.logspace, torch.randint, torch.hann_window, torch.rand, torch.full_like, torch.ones_like, torch.rand_like, torch.randperm, torch.arange, torch.frombuffer, torch.normal, torch._empty_per_channel_affine_quantized, torch.empty_strided, torch.empty_like, torch.scalar_tensor, torch.tril_indices, torch.bartlett_window, torch.ones, torch.sparse_coo_tensor, torch.randn, torch.kaiser_window, torch.tensor, torch.triu_indices, torch.as_tensor, torch.zeros, torch.randint_like, torch.full, torch.eye, torch._sparse_csr_tensor_unsafe, torch.empty, torch._sparse_coo_tensor_unsafe, torch.blackman_window, torch.zeros_like, torch.range, torch.sparse_csr_tensor, torch.randn_like, torch.from_file, torch._cudnn_init_dropout_state, torch._empty_affine_quantized, torch.linspace, torch.hamming_window, torch.empty_quantized, torch._pin_memory, torch.autocast, torch.load, torch.set_default_device, torch.Tensor.new_empty, torch.Tensor.new_empty_strided, torch.Tensor.new_full, torch.Tensor.new_ones, torch.Tensor.new_tensor, torch.Tensor.new_zeros, torch.Tensor.to, torch.Tensor.pin_memory, torch.nn.Module.to, torch.nn.Module.to_empty
warnings.warn(msg, ImportWarning)
/usr/local/lib64/python3.11/site-packages/torch_npu/contrib/transfer_to_npu.py:255: RuntimeWarning: torch.jit.script and torch.jit.script_method will be disabled by transfer_to_npu, which currently does not support them, if you need to enable them, please do not use transfer_to_npu.
warnings.warn(msg, RuntimeWarning)
/usr/local/lib64/python3.11/site-packages/ktransformers/server/api/ollama/completions.py:257: PydanticDeprecatedSince20: Support for class-based config is deprecated, use ConfigDict instead. Deprecated in Pydantic V2.0 to be removed in V3.0. See Pydantic V2 Migration Guide at https://errors.pydantic.dev/2.12/migration/
class OllamaShowResponse(BaseModel):
start to init process group ------rank is 0, world_size is 1
[W1029 03:01:51.812052080 socket.cpp:752] [c10d] The client socket cannot be initialized to connect to [localhost]:31777 (errno: 97 - Address family not supported by protocol).
init process group success ------rank is 0, world_size is 1
Connected to server at tcp://localhost:41617
args.architectures: DeepSeek-Coder-V2-Instruct
The module name (originally ) is not a valid Python identifier. Please rename the original module to avoid import issues.
The module name (originally ) is not a valid Python identifier. Please rename the original module to avoid import issues.
/usr/lib64/python3.11/tempfile.py:904: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpf2t8y3jt'>
_warnings.warn(warn_message, ResourceWarning)
sys:1: DeprecationWarning: builtin type swigvarlink has no module attribute
Process SpawnProcess-1:
Traceback (most recent call last):
File "/usr/lib64/python3.11/multiprocessing/process.py", line 314, in _bootstrap
self.run()
File "/usr/lib64/python3.11/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self.kwargs)
File "/usr/local/lib64/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 403, in run_engine
engine = Engine(args, token_queue, broadcast_endpoint, kvcache_event)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/ktransformers/server/backend/interfaces/balance_serve.py", line 255, in init
torch.distributed.barrier(group=tp_group)
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/c10d_logger.py", line 83, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib64/python3.11/site-packages/torch/distributed/distributed_c10d.py", line 4159, in barrier
work = group.barrier(opts=opts)
^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: create_config:build/CMakeFiles/torch_npu.dir/compiler_depend.ts:148 HCCL function error: hcclCommInitRootInfoConfig(numRanks, &rootInfo, rank, config, &(comm->hcclComm)), error code is 2
[ERROR] 2025-10-29-03:02:35 (PID:501951, Device:0, RankID:0) ERR02200 DIST call hccl api failed.
EL0004: [PID: 501951] 2025-10-29-03:02:35.297.548 Failed to allocate memory.
Possible Cause: Available memory is insufficient.
Solution: Close applications not in use.
TraceBack (most recent call last):
Failed to allocate resource[DeviceMemory] with info [size:32]. Reason: Memory resources are exhausted.
/usr/lib64/python3.11/multiprocessing/process.py:330: ResourceWarning: Unclosed socket <zmq.Socket(zmq.PUB) at 0xfffd27733460>
traceback.print_exc()
/usr/lib64/python3.11/multiprocessing/process.py:330: ResourceWarning: Unclosed context <zmq.Context() at 0xfffcea209c10>
traceback.print_exc()
sys:1: DeprecationWarning: builtin type swigvarlink has no module attribute
Reproduction
python /home/ktransformers-main/ktransformers/server/main.py --model_path /models/deepseek/deepseek-v3-config/ --gguf_path /models/deepseek/deepseek-v3-gguf/ --cpu_infer 120 --optimize_config_path /home/ktransformers-main/ktransformers/optimize/optimize_rules/DeepSeek-V3-Chat-npu.yaml --backend_type balance_serve --port 31444 --architectures KDeepseekV3ForCausalLM --max_new_tokens 128 --max_batch_size 4 --use_cuda_graph --tp 1
Environment
安装rpm：
Ascend-cann-toolkit-8.2.RC1-linux.aarch64
Ascend-cann-nnal-8.2.RC1-linux.aarch64
Ascend-cann-kernels-310p-8.2.RC1-linux.aarch64
pip安装的关键包版本：
ktransformers 0.3.2+npu2.5.1.post1torch25aarch64
torch 2.5.1
torch-npu 2.5.1.post1
torchaudio 2.5.1
torchvision 0.20.1
transformers 4.57.1
npu及驱动信息：
npu-smi info
+--------------------------------------------------------------------------------------------------------+
| npu-smi 25.2.2 Version: 25.2.2 |
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Name | Health | Power(W) Temp(C) Hugepages-Usage(page) |
| Chip Device | Bus-Id | AICore(%) Memory-Usage(MB) |
+===============================+=================+======================================================+
| 1 310P3 | OK | NA 47 0 / 0 |
| 0 0 | 0000:01:00.0 | 0 1872 / 23047 |
+===============================+=================+======================================================+
+-------------------------------+-----------------+------------------------------------------------------+
| NPU Chip | Process id | Process name | Process memory(MB) |
+===============================+=================+======================================================+
| No running processes found in NPU 1 |
+===============================+=================+======================================================+
@ShiyaNiu commented on GitHub (Oct 30, 2025):
您好，readme中Atlas 300IA2推理卡使用的是Ascend910B4 32G的芯片，目前Ascend 310P硬件暂时不支持
@fanzetian commented on GitHub (Oct 30, 2025):
您好，感谢回复，如果我想自己适配的话，有哪些工作需要做呢
@xwqtju commented on GitHub (Nov 6, 2025):
你好，请问最后解决了吗？
@fanzetian commented on GitHub (Nov 6, 2025):
还没有；我又试了cann内置的hccl_test进行性能测试看看，依然报错内存申请失败。反馈给ascend等回复