deepseek-ai/DeepSeek-R1
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepSeek-R1.git synced 2026-11-05 00:47:42 +00:00
Code Issues 45 Projects Packages Wiki Activity

[GH-ISSUE #588] Deploy DeepSeek R1 671B ,ValueError("type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')") #472

New issue
Closed
opened 2026-10-26 21:21:53 +00:00 by bsakaguchi · 2 comments
bsakaguchi commented 2026-10-26 21:21:53 +00:00
Owner
Copy link

Originally created by @gongshaojie12 on GitHub (Mar 23, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/588

System environment:

INFO 03-23 00:23:55 [__init__.py:256] Automatically detected platform cuda.
Collecting environment information...
PyTorch version: 2.6.0+cu124
Is debug build: False
CUDA used to build PyTorch: 12.4
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.4 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: version 3.22.1
Libc version: glibc-2.35

Python version: 3.10.12 (main, Feb  4 2025, 14:57:36) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.15.0-134-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: Could not collect
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration:
GPU 0: NVIDIA A800-SXM4-80GB
GPU 1: NVIDIA A800-SXM4-80GB
GPU 2: NVIDIA A800-SXM4-80GB
GPU 3: NVIDIA A800-SXM4-80GB
GPU 4: NVIDIA A800-SXM4-80GB
GPU 5: NVIDIA A800-SXM4-80GB
GPU 6: NVIDIA A800-SXM4-80GB
GPU 7: NVIDIA A800-SXM4-80GB

Nvidia driver version: 560.35.03
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                         x86_64
CPU op-mode(s):                       32-bit, 64-bit
Address sizes:                        52 bits physical, 57 bits virtual
Byte Order:                           Little Endian
CPU(s):                               128
On-line CPU(s) list:                  0-127
Vendor ID:                            GenuineIntel
Model name:                           Intel(R) Xeon(R) Platinum 8358P CPU @ 2.60GHz
CPU family:                           6
Model:                                106
Thread(s) per core:                   2
Core(s) per socket:                   32
Socket(s):                            2
Stepping:                             6
CPU max MHz:                          3400.0000
CPU min MHz:                          800.0000
BogoMIPS:                             5200.00
Flags:                                fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 ds_cpl smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid fsrm md_clear pconfig flush_l1d arch_capabilities
L1d cache:                            3 MiB (64 instances)
L1i cache:                            2 MiB (64 instances)
L2 cache:                             80 MiB (64 instances)
L3 cache:                             96 MiB (2 instances)
NUMA node(s):                         4
NUMA node0 CPU(s):                    0-15,64-79
NUMA node1 CPU(s):                    16-31,80-95
NUMA node2 CPU(s):                    32-47,96-111
NUMA node3 CPU(s):                    48-63,112-127
Vulnerability Gather data sampling:   Mitigation; Microcode
Vulnerability Itlb multihit:          Not affected
Vulnerability L1tf:                   Not affected
Vulnerability Mds:                    Not affected
Vulnerability Meltdown:               Not affected
Vulnerability Mmio stale data:        Mitigation; Clear CPU buffers; SMT vulnerable
Vulnerability Reg file data sampling: Not affected
Vulnerability Retbleed:               Not affected
Vulnerability Spec rstack overflow:   Not affected
Vulnerability Spec store bypass:      Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:             Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:             Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop
Vulnerability Srbds:                  Not affected
Vulnerability Tsx async abort:        Not affected

Versions of relevant libraries:
[pip3] numpy==1.26.4
[pip3] nvidia-cublas-cu12==12.4.5.8
[pip3] nvidia-cuda-cupti-cu12==12.4.127
[pip3] nvidia-cuda-nvrtc-cu12==12.4.127
[pip3] nvidia-cuda-runtime-cu12==12.4.127
[pip3] nvidia-cudnn-cu12==9.1.0.70
[pip3] nvidia-cufft-cu12==11.2.1.3
[pip3] nvidia-curand-cu12==10.3.5.147
[pip3] nvidia-cusolver-cu12==11.6.1.9
[pip3] nvidia-cusparse-cu12==12.3.1.170
[pip3] nvidia-cusparselt-cu12==0.6.2
[pip3] nvidia-nccl-cu12==2.21.5
[pip3] nvidia-nvjitlink-cu12==12.4.127
[pip3] nvidia-nvtx-cu12==12.4.127
[pip3] pyzmq==26.3.0
[pip3] torch==2.6.0
[pip3] torchaudio==2.6.0
[pip3] torchvision==0.21.0
[pip3] transformers==4.50.0
[pip3] triton==3.2.0
[conda] Could not collect
ROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: 0.8.1
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      PXB     NODE    NODE    SYS     SYS     SYS     SYS     PXB     PXB     PXB     NODE    NODE    NODE    SYS     SYS     16-31,80-95     1        N/A
GPU1    PXB      X      NODE    NODE    SYS     SYS     SYS     SYS     PXB     PXB     PXB     NODE    NODE    NODE    SYS     SYS     16-31,80-95     1        N/A
GPU2    NODE    NODE     X      PXB     SYS     SYS     SYS     SYS     NODE    NODE    NODE    PXB     PXB     PXB     SYS     SYS     16-31,80-95     1        N/A
GPU3    NODE    NODE    PXB      X      SYS     SYS     SYS     SYS     NODE    NODE    NODE    PXB     PXB     PXB     SYS     SYS     16-31,80-95     1        N/A
GPU4    SYS     SYS     SYS     SYS      X      PXB     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     PXB     SYS     32-47,96-111    2        N/A
GPU5    SYS     SYS     SYS     SYS     PXB      X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS     PXB     SYS     32-47,96-111    2        N/A
GPU6    SYS     SYS     SYS     SYS     SYS     SYS      X      PXB     SYS     SYS     SYS     SYS     SYS     SYS     SYS     PXB     48-63,112-127   3        N/A
GPU7    SYS     SYS     SYS     SYS     SYS     SYS     PXB      X      SYS     SYS     SYS     SYS     SYS     SYS     SYS     PXB     48-63,112-127   3        N/A
NIC0    PXB     PXB     NODE    NODE    SYS     SYS     SYS     SYS      X      PIX     PIX     NODE    NODE    NODE    SYS     SYS
NIC1    PXB     PXB     NODE    NODE    SYS     SYS     SYS     SYS     PIX      X      PIX     NODE    NODE    NODE    SYS     SYS
NIC2    PXB     PXB     NODE    NODE    SYS     SYS     SYS     SYS     PIX     PIX      X      NODE    NODE    NODE    SYS     SYS
NIC3    NODE    NODE    PXB     PXB     SYS     SYS     SYS     SYS     NODE    NODE    NODE     X      PIX     PIX     SYS     SYS
NIC4    NODE    NODE    PXB     PXB     SYS     SYS     SYS     SYS     NODE    NODE    NODE    PIX      X      PIX     SYS     SYS
NIC5    NODE    NODE    PXB     PXB     SYS     SYS     SYS     SYS     NODE    NODE    NODE    PIX     PIX      X      SYS     SYS
NIC6    SYS     SYS     SYS     SYS     PXB     PXB     SYS     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X      SYS
NIC7    SYS     SYS     SYS     SYS     SYS     SYS     PXB     PXB     SYS     SYS     SYS     SYS     SYS     SYS     SYS      X

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_0
  NIC1: mlx5_1
  NIC2: mlx5_2
  NIC3: mlx5_3
  NIC4: mlx5_4
  NIC5: mlx5_5
  NIC6: mlx5_6
  NIC7: mlx5_7

NCCL_CUMEM_ENABLE=0
TORCHINDUCTOR_COMPILE_THREADS=1
CUDA_MODULE_LOADING=LAZY

When I run the following command, I encounter an error.

python3 -m vllm.entrypoints.openai.api_server \
        --model=/data/deepseek-r1 \
        --dtype=auto \
        --block-size 32 \
        --tokenizer-mode=slow \
        --max-model-len 32768 \
        --max-num-batched-tokens 2048 \
        --tensor-parallel-size 8 \
        --pipeline-parallel-size 2 \
        --gpu-memory-utilization 0.90 \
        --max-num-seqs 128 \
        --trust-remote-code \
        --enable-prefix-caching \
        --enable-chunked-prefill=True \
        --disable-custom-all-reduce \
        --port 8888

The error is as follows:

INFO 03-22 06:48:13 [loader.py:429] Loading weights took 19.04 seconds
ERROR 03-22 06:48:14 [worker_base.py:620] Error executing method 'load_model'. This might cause deadlock in distributed execution.
ERROR 03-22 06:48:14 [worker_base.py:620] Traceback (most recent call last):
ERROR 03-22 06:48:14 [worker_base.py:620]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 612, in execute_                      method
ERROR 03-22 06:48:14 [worker_base.py:620]     return run_method(self, method, args, kwargs)
ERROR 03-22 06:48:14 [worker_base.py:620]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-22 06:48:14 [worker_base.py:620]   File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2221, in run_method
ERROR 03-22 06:48:14 [worker_base.py:620]     return func(*args, **kwargs)
ERROR 03-22 06:48:14 [worker_base.py:620]            ^^^^^^^^^^^^^^^^^^^^^
ERROR 03-22 06:48:14 [worker_base.py:620]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 183, in load_model
ERROR 03-22 06:48:14 [worker_base.py:620]     self.model_runner.load_model()
ERROR 03-22 06:48:14 [worker_base.py:620]   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1113, in load_m                      odel
ERROR 03-22 06:48:14 [worker_base.py:620]     self.model = get_model(vllm_config=self.vllm_config)
ERROR 03-22 06:48:14 [worker_base.py:620]                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-22 06:48:14 [worker_base.py:620]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/__init__.py", lin                      e 14, in get_model
ERROR 03-22 06:48:14 [worker_base.py:620]     return loader.load_model(vllm_config=vllm_config)
ERROR 03-22 06:48:14 [worker_base.py:620]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-22 06:48:14 [worker_base.py:620]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line                       442, in load_model
ERROR 03-22 06:48:14 [worker_base.py:620]     _process_weights_after_loading(model, model_config, target_device)
ERROR 03-22 06:48:14 [worker_base.py:620]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line                       178, in _process_weights_after_loading
ERROR 03-22 06:48:14 [worker_base.py:620]     module.process_weights_after_loading(model_config.dtype)
ERROR 03-22 06:48:14 [worker_base.py:620]   File "/usr/local/lib/python3.12/dist-packages/vllm/attention/layer.py", line 251, in process_wei                      ghts_after_loading
ERROR 03-22 06:48:14 [worker_base.py:620]     self.impl.process_weights_after_loading(act_dtype)
ERROR 03-22 06:48:14 [worker_base.py:620]   File "/usr/local/lib/python3.12/dist-packages/vllm/attention/backends/mla/common.py", line 1101,                       in process_weights_after_loading
ERROR 03-22 06:48:14 [worker_base.py:620]     kv_b_proj_weight = get_and_maybe_dequant_weights(self.kv_b_proj).T
ERROR 03-22 06:48:14 [worker_base.py:620]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-22 06:48:14 [worker_base.py:620]   File "/usr/local/lib/python3.12/dist-packages/vllm/attention/backends/mla/common.py", line 1090,                       in get_and_maybe_dequant_weights
ERROR 03-22 06:48:14 [worker_base.py:620]     dequant_weights = layer.quant_method.apply(layer,
ERROR 03-22 06:48:14 [worker_base.py:620]                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-22 06:48:14 [worker_base.py:620]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/fp8.py", l                      ine 376, in apply
ERROR 03-22 06:48:14 [worker_base.py:620]     return torch.ops.vllm.apply_w8a8_block_fp8_linear(
ERROR 03-22 06:48:14 [worker_base.py:620]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-22 06:48:14 [worker_base.py:620]   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1123, in __call__
ERROR 03-22 06:48:14 [worker_base.py:620]     return self._op(*args, **(kwargs or {}))
ERROR 03-22 06:48:14 [worker_base.py:620]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-22 06:48:14 [worker_base.py:620]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_                      utils.py", line 70, in apply_w8a8_block_fp8_linear
ERROR 03-22 06:48:14 [worker_base.py:620]     q_input, x_scale = per_token_group_quant_fp8(input_2d,
ERROR 03-22 06:48:14 [worker_base.py:620]                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-22 06:48:14 [worker_base.py:620]   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_                      utils.py", line 295, in per_token_group_quant_fp8
ERROR 03-22 06:48:14 [worker_base.py:620]     _per_token_group_quant_fp8[(M, )](
ERROR 03-22 06:48:14 [worker_base.py:620]   File "/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py", line 330, in <lambda>
ERROR 03-22 06:48:14 [worker_base.py:620]     return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
ERROR 03-22 06:48:14 [worker_base.py:620]                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-22 06:48:14 [worker_base.py:620]   File "/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py", line 623, in run
ERROR 03-22 06:48:14 [worker_base.py:620]     kernel = self.compile(
ERROR 03-22 06:48:14 [worker_base.py:620]              ^^^^^^^^^^^^^
ERROR 03-22 06:48:14 [worker_base.py:620]   File "/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py", line 273, in compile
ERROR 03-22 06:48:14 [worker_base.py:620]     module = src.make_ir(options, codegen_fns, module_map, context)
ERROR 03-22 06:48:14 [worker_base.py:620]              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-22 06:48:14 [worker_base.py:620]   File "/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py", line 100, in make_ir
ERROR 03-22 06:48:14 [worker_base.py:620]     return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns,
ERROR 03-22 06:48:14 [worker_base.py:620]            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ERROR 03-22 06:48:14 [worker_base.py:620] triton.compiler.errors.CompilationError: at 1:0:
ERROR 03-22 06:48:14 [worker_base.py:620] def _per_token_group_quant_fp8(
ERROR 03-22 06:48:14 [worker_base.py:620] ^
ERROR 03-22 06:48:14 [worker_base.py:620] ValueError("type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b                      15', 'fp8e5')")
[rank0]: Traceback (most recent call last):
[rank0]:   File "<frozen runpy>", line 198, in _run_module_as_main
[rank0]:   File "<frozen runpy>", line 88, in _run_code
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1066, in <module>
[rank0]:     uvloop.run(run_server(args))
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 109, in run
[rank0]:     return __asyncio.run(
[rank0]:            ^^^^^^^^^^^^^^
[rank0]:   File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run
[rank0]:     return runner.run(main)
[rank0]:            ^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run
[rank0]:     return self._loop.run_until_complete(task)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 61, in wrapper
[rank0]:     return await main
[rank0]:            ^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1016, in run_server
[rank0]:     async with build_async_engine_client(args) as engine_client:
[rank0]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
[rank0]:     return await anext(self.gen)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 141, in build_async_engine_client
[rank0]:     async with build_async_engine_client_from_engine_args(
[rank0]:                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__
[rank0]:     return await anext(self.gen)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 189, in build_async_engine_client_from                      _engine_args
[rank0]:     engine_client = AsyncLLMEngine.from_vllm_config(
[rank0]:                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/async_llm_engine.py", line 651, in from_vllm_config
[rank0]:     return cls(
[rank0]:            ^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/async_llm_engine.py", line 606, in __init__
[rank0]:     self.engine = self._engine_class(*args, **kwargs)
[rank0]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/async_llm_engine.py", line 267, in __init__
[rank0]:     super().__init__(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 280, in __init__
[rank0]:     self.model_executor = executor_class(vllm_config=vllm_config, )
[rank0]:                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 271, in __init__
[rank0]:     super().__init__(*args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 52, in __init__
[rank0]:     self._init_executor()
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/ray_distributed_executor.py", line 114, in _init_executor
[rank0]:     self._init_workers_ray(placement_group)
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/ray_distributed_executor.py", line 396, in _init_workers_ray
[rank0]:     self._run_workers("load_model",
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/executor/ray_distributed_executor.py", line 516, in _run_workers
[rank0]:     self.driver_worker.execute_method(sent_method, *args, **kwargs)
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 621, in execute_method
[rank0]:     raise e
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 612, in execute_method
[rank0]:     return run_method(self, method, args, kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2221, in run_method
[rank0]:     return func(*args, **kwargs)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 183, in load_model
[rank0]:     self.model_runner.load_model()
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1113, in load_model
[rank0]:     self.model = get_model(vllm_config=self.vllm_config)
[rank0]:                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/__init__.py", line 14, in get_model
[rank0]:     return loader.load_model(vllm_config=vllm_config)
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 442, in load_model
[rank0]:     _process_weights_after_loading(model, model_config, target_device)
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 178, in _process_weights_after_lo                      ading
[rank0]:     module.process_weights_after_loading(model_config.dtype)
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/attention/layer.py", line 251, in process_weights_after_loading
[rank0]:     self.impl.process_weights_after_loading(act_dtype)
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/attention/backends/mla/common.py", line 1101, in process_weights_after_loading
[rank0]:     kv_b_proj_weight = get_and_maybe_dequant_weights(self.kv_b_proj).T
[rank0]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/attention/backends/mla/common.py", line 1090, in get_and_maybe_dequant_weights
[rank0]:     dequant_weights = layer.quant_method.apply(layer,
[rank0]:                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/fp8.py", line 376, in apply
[rank0]:     return torch.ops.vllm.apply_w8a8_block_fp8_linear(
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1123, in __call__
[rank0]:     return self._op(*args, **(kwargs or {}))
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 70, in apply_w8a8                      _block_fp8_linear
[rank0]:     q_input, x_scale = per_token_group_quant_fp8(input_2d,
[rank0]:                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 295, in per_token                      _group_quant_fp8
[rank0]:     _per_token_group_quant_fp8[(M, )](
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py", line 330, in <lambda>
[rank0]:     return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs)
[rank0]:                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py", line 623, in run
[rank0]:     kernel = self.compile(
[rank0]:              ^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py", line 273, in compile
[rank0]:     module = src.make_ir(options, codegen_fns, module_map, context)
[rank0]:              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]:   File "/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py", line 100, in make_ir
[rank0]:     return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns,
[rank0]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
[rank0]: triton.compiler.errors.CompilationError: at 1:0:
[rank0]: def _per_token_group_quant_fp8(
[rank0]: ^
[rank0]: ValueError("type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')")
(RayWorkerWrapper pid=9143) INFO 03-22 06:47:54 [utils.py:930] Found nccl from library libnccl.so.2 [repeated 29x across cluster]
(RayWorkerWrapper pid=9143) INFO 03-22 06:47:54 [pynccl.py:69] vLLM is using nccl==2.26.2 [repeated 29x across cluster]
(RayWorkerWrapper pid=1343, ip=172.29.20.44) INFO 03-22 06:47:54 [parallel_state.py:967] rank 15 in world size 16 is assigned as DP rank 0,                       PP rank 1, TP rank 7 [repeated 14x across cluster]
(RayWorkerWrapper pid=9143) INFO 03-22 06:47:54 [model_runner.py:1110] Starting to load model /workspace/deepseek-r1... [repeated 14x across                       cluster]
(RayWorkerWrapper pid=1337, ip=172.29.20.44) WARNING 03-22 06:47:54 [utils.py:169] The model class DeepseekV3ForCausalLM has not defined `pa                      cked_modules_mapping`, this may lead to incorrect mapping of quantized or ignored modules [repeated 14x across cluster]
(RayWorkerWrapper pid=1337, ip=172.29.20.44) INFO 03-22 06:47:54 [utils.py:104] Hidden layers were unevenly partitioned: 31,30 [repeated 14x                       across cluster]
(RayWorkerWrapper pid=1337, ip=172.29.20.44) INFO 03-22 06:47:54 [utils.py:106] This can be manually overridden using the VLLM_PP_LAYER_PART                      ITION environment variable [repeated 14x across cluster]
INFO 03-22 06:48:15 [ray_distributed_executor.py:127] Shutting down Ray distributed executor. If you see error log from logging.cc regarding                       SIGTERM received, please ignore because this is the expected termination process in Ray.
[rank0]:[W322 06:48:15.833449907 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, wh                      ich can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator())
/usr/lib/python3.12/multiprocessing/resource_tracker.py:255: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory object                      s to clean up at shutdown
  warnings.warn('resource_tracker: There appear to be %d '

How can I resolve this issue? Is it not possible to deploy Deepseek R1 671B on the NVIDIA A800?

Originally created by @gongshaojie12 on GitHub (Mar 23, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-R1/issues/588 System environment: ``` INFO 03-23 00:23:55 [__init__.py:256] Automatically detected platform cuda. Collecting environment information... PyTorch version: 2.6.0+cu124 Is debug build: False CUDA used to build PyTorch: 12.4 ROCM used to build PyTorch: N/A OS: Ubuntu 22.04.4 LTS (x86_64) GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0 Clang version: Could not collect CMake version: version 3.22.1 Libc version: glibc-2.35 Python version: 3.10.12 (main, Feb 4 2025, 14:57:36) [GCC 11.4.0] (64-bit runtime) Python platform: Linux-5.15.0-134-generic-x86_64-with-glibc2.35 Is CUDA available: True CUDA runtime version: Could not collect CUDA_MODULE_LOADING set to: LAZY GPU models and configuration: GPU 0: NVIDIA A800-SXM4-80GB GPU 1: NVIDIA A800-SXM4-80GB GPU 2: NVIDIA A800-SXM4-80GB GPU 3: NVIDIA A800-SXM4-80GB GPU 4: NVIDIA A800-SXM4-80GB GPU 5: NVIDIA A800-SXM4-80GB GPU 6: NVIDIA A800-SXM4-80GB GPU 7: NVIDIA A800-SXM4-80GB Nvidia driver version: 560.35.03 cuDNN version: Could not collect HIP runtime version: N/A MIOpen runtime version: N/A Is XNNPACK available: True CPU: Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Address sizes: 52 bits physical, 57 bits virtual Byte Order: Little Endian CPU(s): 128 On-line CPU(s) list: 0-127 Vendor ID: GenuineIntel Model name: Intel(R) Xeon(R) Platinum 8358P CPU @ 2.60GHz CPU family: 6 Model: 106 Thread(s) per core: 2 Core(s) per socket: 32 Socket(s): 2 Stepping: 6 CPU max MHz: 3400.0000 CPU min MHz: 800.0000 BogoMIPS: 5200.00 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfmperf pni pclmulqdq dtes64 ds_cpl smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid dca sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm 3dnowprefetch cpuid_fault epb cat_l3 invpcid_single intel_ppin ssbd mba ibrs ibpb stibp ibrs_enhanced fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid cqm rdt_a avx512f avx512dq rdseed adx smap avx512ifma clflushopt clwb intel_pt avx512cd sha_ni avx512bw avx512vl xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local split_lock_detect wbnoinvd dtherm ida arat pln pts avx512vbmi umip pku ospke avx512_vbmi2 gfni vaes vpclmulqdq avx512_vnni avx512_bitalg tme avx512_vpopcntdq la57 rdpid fsrm md_clear pconfig flush_l1d arch_capabilities L1d cache: 3 MiB (64 instances) L1i cache: 2 MiB (64 instances) L2 cache: 80 MiB (64 instances) L3 cache: 96 MiB (2 instances) NUMA node(s): 4 NUMA node0 CPU(s): 0-15,64-79 NUMA node1 CPU(s): 16-31,80-95 NUMA node2 CPU(s): 32-47,96-111 NUMA node3 CPU(s): 48-63,112-127 Vulnerability Gather data sampling: Mitigation; Microcode Vulnerability Itlb multihit: Not affected Vulnerability L1tf: Not affected Vulnerability Mds: Not affected Vulnerability Meltdown: Not affected Vulnerability Mmio stale data: Mitigation; Clear CPU buffers; SMT vulnerable Vulnerability Reg file data sampling: Not affected Vulnerability Retbleed: Not affected Vulnerability Spec rstack overflow: Not affected Vulnerability Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp Vulnerability Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization Vulnerability Spectre v2: Mitigation; Enhanced / Automatic IBRS; IBPB conditional; RSB filling; PBRSB-eIBRS SW sequence; BHI SW loop, KVM SW loop Vulnerability Srbds: Not affected Vulnerability Tsx async abort: Not affected Versions of relevant libraries: [pip3] numpy==1.26.4 [pip3] nvidia-cublas-cu12==12.4.5.8 [pip3] nvidia-cuda-cupti-cu12==12.4.127 [pip3] nvidia-cuda-nvrtc-cu12==12.4.127 [pip3] nvidia-cuda-runtime-cu12==12.4.127 [pip3] nvidia-cudnn-cu12==9.1.0.70 [pip3] nvidia-cufft-cu12==11.2.1.3 [pip3] nvidia-curand-cu12==10.3.5.147 [pip3] nvidia-cusolver-cu12==11.6.1.9 [pip3] nvidia-cusparse-cu12==12.3.1.170 [pip3] nvidia-cusparselt-cu12==0.6.2 [pip3] nvidia-nccl-cu12==2.21.5 [pip3] nvidia-nvjitlink-cu12==12.4.127 [pip3] nvidia-nvtx-cu12==12.4.127 [pip3] pyzmq==26.3.0 [pip3] torch==2.6.0 [pip3] torchaudio==2.6.0 [pip3] torchvision==0.21.0 [pip3] transformers==4.50.0 [pip3] triton==3.2.0 [conda] Could not collect ROCM Version: Could not collect Neuron SDK Version: N/A vLLM Version: 0.8.1 vLLM Build Flags: CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled GPU Topology: GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X PXB NODE NODE SYS SYS SYS SYS PXB PXB PXB NODE NODE NODE SYS SYS 16-31,80-95 1 N/A GPU1 PXB X NODE NODE SYS SYS SYS SYS PXB PXB PXB NODE NODE NODE SYS SYS 16-31,80-95 1 N/A GPU2 NODE NODE X PXB SYS SYS SYS SYS NODE NODE NODE PXB PXB PXB SYS SYS 16-31,80-95 1 N/A GPU3 NODE NODE PXB X SYS SYS SYS SYS NODE NODE NODE PXB PXB PXB SYS SYS 16-31,80-95 1 N/A GPU4 SYS SYS SYS SYS X PXB SYS SYS SYS SYS SYS SYS SYS SYS PXB SYS 32-47,96-111 2 N/A GPU5 SYS SYS SYS SYS PXB X SYS SYS SYS SYS SYS SYS SYS SYS PXB SYS 32-47,96-111 2 N/A GPU6 SYS SYS SYS SYS SYS SYS X PXB SYS SYS SYS SYS SYS SYS SYS PXB 48-63,112-127 3 N/A GPU7 SYS SYS SYS SYS SYS SYS PXB X SYS SYS SYS SYS SYS SYS SYS PXB 48-63,112-127 3 N/A NIC0 PXB PXB NODE NODE SYS SYS SYS SYS X PIX PIX NODE NODE NODE SYS SYS NIC1 PXB PXB NODE NODE SYS SYS SYS SYS PIX X PIX NODE NODE NODE SYS SYS NIC2 PXB PXB NODE NODE SYS SYS SYS SYS PIX PIX X NODE NODE NODE SYS SYS NIC3 NODE NODE PXB PXB SYS SYS SYS SYS NODE NODE NODE X PIX PIX SYS SYS NIC4 NODE NODE PXB PXB SYS SYS SYS SYS NODE NODE NODE PIX X PIX SYS SYS NIC5 NODE NODE PXB PXB SYS SYS SYS SYS NODE NODE NODE PIX PIX X SYS SYS NIC6 SYS SYS SYS SYS PXB PXB SYS SYS SYS SYS SYS SYS SYS SYS X SYS NIC7 SYS SYS SYS SYS SYS SYS PXB PXB SYS SYS SYS SYS SYS SYS SYS X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_0 NIC1: mlx5_1 NIC2: mlx5_2 NIC3: mlx5_3 NIC4: mlx5_4 NIC5: mlx5_5 NIC6: mlx5_6 NIC7: mlx5_7 NCCL_CUMEM_ENABLE=0 TORCHINDUCTOR_COMPILE_THREADS=1 CUDA_MODULE_LOADING=LAZY ``` When I run the following command, I encounter an error. ``` python3 -m vllm.entrypoints.openai.api_server \ --model=/data/deepseek-r1 \ --dtype=auto \ --block-size 32 \ --tokenizer-mode=slow \ --max-model-len 32768 \ --max-num-batched-tokens 2048 \ --tensor-parallel-size 8 \ --pipeline-parallel-size 2 \ --gpu-memory-utilization 0.90 \ --max-num-seqs 128 \ --trust-remote-code \ --enable-prefix-caching \ --enable-chunked-prefill=True \ --disable-custom-all-reduce \ --port 8888 ``` The error is as follows: ``` INFO 03-22 06:48:13 [loader.py:429] Loading weights took 19.04 seconds ERROR 03-22 06:48:14 [worker_base.py:620] Error executing method 'load_model'. This might cause deadlock in distributed execution. ERROR 03-22 06:48:14 [worker_base.py:620] Traceback (most recent call last): ERROR 03-22 06:48:14 [worker_base.py:620] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 612, in execute_ method ERROR 03-22 06:48:14 [worker_base.py:620] return run_method(self, method, args, kwargs) ERROR 03-22 06:48:14 [worker_base.py:620] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 03-22 06:48:14 [worker_base.py:620] File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2221, in run_method ERROR 03-22 06:48:14 [worker_base.py:620] return func(*args, **kwargs) ERROR 03-22 06:48:14 [worker_base.py:620] ^^^^^^^^^^^^^^^^^^^^^ ERROR 03-22 06:48:14 [worker_base.py:620] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 183, in load_model ERROR 03-22 06:48:14 [worker_base.py:620] self.model_runner.load_model() ERROR 03-22 06:48:14 [worker_base.py:620] File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1113, in load_m odel ERROR 03-22 06:48:14 [worker_base.py:620] self.model = get_model(vllm_config=self.vllm_config) ERROR 03-22 06:48:14 [worker_base.py:620] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 03-22 06:48:14 [worker_base.py:620] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/__init__.py", lin e 14, in get_model ERROR 03-22 06:48:14 [worker_base.py:620] return loader.load_model(vllm_config=vllm_config) ERROR 03-22 06:48:14 [worker_base.py:620] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 03-22 06:48:14 [worker_base.py:620] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 442, in load_model ERROR 03-22 06:48:14 [worker_base.py:620] _process_weights_after_loading(model, model_config, target_device) ERROR 03-22 06:48:14 [worker_base.py:620] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 178, in _process_weights_after_loading ERROR 03-22 06:48:14 [worker_base.py:620] module.process_weights_after_loading(model_config.dtype) ERROR 03-22 06:48:14 [worker_base.py:620] File "/usr/local/lib/python3.12/dist-packages/vllm/attention/layer.py", line 251, in process_wei ghts_after_loading ERROR 03-22 06:48:14 [worker_base.py:620] self.impl.process_weights_after_loading(act_dtype) ERROR 03-22 06:48:14 [worker_base.py:620] File "/usr/local/lib/python3.12/dist-packages/vllm/attention/backends/mla/common.py", line 1101, in process_weights_after_loading ERROR 03-22 06:48:14 [worker_base.py:620] kv_b_proj_weight = get_and_maybe_dequant_weights(self.kv_b_proj).T ERROR 03-22 06:48:14 [worker_base.py:620] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 03-22 06:48:14 [worker_base.py:620] File "/usr/local/lib/python3.12/dist-packages/vllm/attention/backends/mla/common.py", line 1090, in get_and_maybe_dequant_weights ERROR 03-22 06:48:14 [worker_base.py:620] dequant_weights = layer.quant_method.apply(layer, ERROR 03-22 06:48:14 [worker_base.py:620] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 03-22 06:48:14 [worker_base.py:620] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/fp8.py", l ine 376, in apply ERROR 03-22 06:48:14 [worker_base.py:620] return torch.ops.vllm.apply_w8a8_block_fp8_linear( ERROR 03-22 06:48:14 [worker_base.py:620] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 03-22 06:48:14 [worker_base.py:620] File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1123, in __call__ ERROR 03-22 06:48:14 [worker_base.py:620] return self._op(*args, **(kwargs or {})) ERROR 03-22 06:48:14 [worker_base.py:620] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 03-22 06:48:14 [worker_base.py:620] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_ utils.py", line 70, in apply_w8a8_block_fp8_linear ERROR 03-22 06:48:14 [worker_base.py:620] q_input, x_scale = per_token_group_quant_fp8(input_2d, ERROR 03-22 06:48:14 [worker_base.py:620] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 03-22 06:48:14 [worker_base.py:620] File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_ utils.py", line 295, in per_token_group_quant_fp8 ERROR 03-22 06:48:14 [worker_base.py:620] _per_token_group_quant_fp8[(M, )]( ERROR 03-22 06:48:14 [worker_base.py:620] File "/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py", line 330, in <lambda> ERROR 03-22 06:48:14 [worker_base.py:620] return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) ERROR 03-22 06:48:14 [worker_base.py:620] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 03-22 06:48:14 [worker_base.py:620] File "/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py", line 623, in run ERROR 03-22 06:48:14 [worker_base.py:620] kernel = self.compile( ERROR 03-22 06:48:14 [worker_base.py:620] ^^^^^^^^^^^^^ ERROR 03-22 06:48:14 [worker_base.py:620] File "/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py", line 273, in compile ERROR 03-22 06:48:14 [worker_base.py:620] module = src.make_ir(options, codegen_fns, module_map, context) ERROR 03-22 06:48:14 [worker_base.py:620] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 03-22 06:48:14 [worker_base.py:620] File "/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py", line 100, in make_ir ERROR 03-22 06:48:14 [worker_base.py:620] return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns, ERROR 03-22 06:48:14 [worker_base.py:620] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ ERROR 03-22 06:48:14 [worker_base.py:620] triton.compiler.errors.CompilationError: at 1:0: ERROR 03-22 06:48:14 [worker_base.py:620] def _per_token_group_quant_fp8( ERROR 03-22 06:48:14 [worker_base.py:620] ^ ERROR 03-22 06:48:14 [worker_base.py:620] ValueError("type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b 15', 'fp8e5')") [rank0]: Traceback (most recent call last): [rank0]: File "<frozen runpy>", line 198, in _run_module_as_main [rank0]: File "<frozen runpy>", line 88, in _run_code [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1066, in <module> [rank0]: uvloop.run(run_server(args)) [rank0]: File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 109, in run [rank0]: return __asyncio.run( [rank0]: ^^^^^^^^^^^^^^ [rank0]: File "/usr/lib/python3.12/asyncio/runners.py", line 195, in run [rank0]: return runner.run(main) [rank0]: ^^^^^^^^^^^^^^^^ [rank0]: File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run [rank0]: return self._loop.run_until_complete(task) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "uvloop/loop.pyx", line 1518, in uvloop.loop.Loop.run_until_complete [rank0]: File "/usr/local/lib/python3.12/dist-packages/uvloop/__init__.py", line 61, in wrapper [rank0]: return await main [rank0]: ^^^^^^^^^^ [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 1016, in run_server [rank0]: async with build_async_engine_client(args) as engine_client: [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ [rank0]: return await anext(self.gen) [rank0]: ^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 141, in build_async_engine_client [rank0]: async with build_async_engine_client_from_engine_args( [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/usr/lib/python3.12/contextlib.py", line 210, in __aenter__ [rank0]: return await anext(self.gen) [rank0]: ^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/api_server.py", line 189, in build_async_engine_client_from _engine_args [rank0]: engine_client = AsyncLLMEngine.from_vllm_config( [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/engine/async_llm_engine.py", line 651, in from_vllm_config [rank0]: return cls( [rank0]: ^^^^ [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/engine/async_llm_engine.py", line 606, in __init__ [rank0]: self.engine = self._engine_class(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/engine/async_llm_engine.py", line 267, in __init__ [rank0]: super().__init__(*args, **kwargs) [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/engine/llm_engine.py", line 280, in __init__ [rank0]: self.model_executor = executor_class(vllm_config=vllm_config, ) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 271, in __init__ [rank0]: super().__init__(*args, **kwargs) [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/executor/executor_base.py", line 52, in __init__ [rank0]: self._init_executor() [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/executor/ray_distributed_executor.py", line 114, in _init_executor [rank0]: self._init_workers_ray(placement_group) [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/executor/ray_distributed_executor.py", line 396, in _init_workers_ray [rank0]: self._run_workers("load_model", [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/executor/ray_distributed_executor.py", line 516, in _run_workers [rank0]: self.driver_worker.execute_method(sent_method, *args, **kwargs) [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 621, in execute_method [rank0]: raise e [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker_base.py", line 612, in execute_method [rank0]: return run_method(self, method, args, kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/utils.py", line 2221, in run_method [rank0]: return func(*args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/worker/worker.py", line 183, in load_model [rank0]: self.model_runner.load_model() [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/worker/model_runner.py", line 1113, in load_model [rank0]: self.model = get_model(vllm_config=self.vllm_config) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/__init__.py", line 14, in get_model [rank0]: return loader.load_model(vllm_config=vllm_config) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 442, in load_model [rank0]: _process_weights_after_loading(model, model_config, target_device) [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/model_loader/loader.py", line 178, in _process_weights_after_lo ading [rank0]: module.process_weights_after_loading(model_config.dtype) [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/attention/layer.py", line 251, in process_weights_after_loading [rank0]: self.impl.process_weights_after_loading(act_dtype) [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/attention/backends/mla/common.py", line 1101, in process_weights_after_loading [rank0]: kv_b_proj_weight = get_and_maybe_dequant_weights(self.kv_b_proj).T [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/attention/backends/mla/common.py", line 1090, in get_and_maybe_dequant_weights [rank0]: dequant_weights = layer.quant_method.apply(layer, [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/fp8.py", line 376, in apply [rank0]: return torch.ops.vllm.apply_w8a8_block_fp8_linear( [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/usr/local/lib/python3.12/dist-packages/torch/_ops.py", line 1123, in __call__ [rank0]: return self._op(*args, **(kwargs or {})) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 70, in apply_w8a8 _block_fp8_linear [rank0]: q_input, x_scale = per_token_group_quant_fp8(input_2d, [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/usr/local/lib/python3.12/dist-packages/vllm/model_executor/layers/quantization/utils/fp8_utils.py", line 295, in per_token _group_quant_fp8 [rank0]: _per_token_group_quant_fp8[(M, )]( [rank0]: File "/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py", line 330, in <lambda> [rank0]: return lambda *args, **kwargs: self.run(grid=grid, warmup=False, *args, **kwargs) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/usr/local/lib/python3.12/dist-packages/triton/runtime/jit.py", line 623, in run [rank0]: kernel = self.compile( [rank0]: ^^^^^^^^^^^^^ [rank0]: File "/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py", line 273, in compile [rank0]: module = src.make_ir(options, codegen_fns, module_map, context) [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: File "/usr/local/lib/python3.12/dist-packages/triton/compiler/compiler.py", line 100, in make_ir [rank0]: return ast_to_ttir(self.fn, self, context=context, options=options, codegen_fns=codegen_fns, [rank0]: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ [rank0]: triton.compiler.errors.CompilationError: at 1:0: [rank0]: def _per_token_group_quant_fp8( [rank0]: ^ [rank0]: ValueError("type fp8e4nv not supported in this architecture. The supported fp8 dtypes are ('fp8e4b15', 'fp8e5')") (RayWorkerWrapper pid=9143) INFO 03-22 06:47:54 [utils.py:930] Found nccl from library libnccl.so.2 [repeated 29x across cluster] (RayWorkerWrapper pid=9143) INFO 03-22 06:47:54 [pynccl.py:69] vLLM is using nccl==2.26.2 [repeated 29x across cluster] (RayWorkerWrapper pid=1343, ip=172.29.20.44) INFO 03-22 06:47:54 [parallel_state.py:967] rank 15 in world size 16 is assigned as DP rank 0, PP rank 1, TP rank 7 [repeated 14x across cluster] (RayWorkerWrapper pid=9143) INFO 03-22 06:47:54 [model_runner.py:1110] Starting to load model /workspace/deepseek-r1... [repeated 14x across cluster] (RayWorkerWrapper pid=1337, ip=172.29.20.44) WARNING 03-22 06:47:54 [utils.py:169] The model class DeepseekV3ForCausalLM has not defined `pa cked_modules_mapping`, this may lead to incorrect mapping of quantized or ignored modules [repeated 14x across cluster] (RayWorkerWrapper pid=1337, ip=172.29.20.44) INFO 03-22 06:47:54 [utils.py:104] Hidden layers were unevenly partitioned: 31,30 [repeated 14x across cluster] (RayWorkerWrapper pid=1337, ip=172.29.20.44) INFO 03-22 06:47:54 [utils.py:106] This can be manually overridden using the VLLM_PP_LAYER_PART ITION environment variable [repeated 14x across cluster] INFO 03-22 06:48:15 [ray_distributed_executor.py:127] Shutting down Ray distributed executor. If you see error log from logging.cc regarding SIGTERM received, please ignore because this is the expected termination process in Ray. [rank0]:[W322 06:48:15.833449907 ProcessGroupNCCL.cpp:1496] Warning: WARNING: destroy_process_group() was not called before program exit, wh ich can leak resources. For more info, please see https://pytorch.org/docs/stable/distributed.html#shutdown (function operator()) /usr/lib/python3.12/multiprocessing/resource_tracker.py:255: UserWarning: resource_tracker: There appear to be 1 leaked shared_memory object s to clean up at shutdown warnings.warn('resource_tracker: There appear to be %d ' ``` How can I resolve this issue? Is it not possible to deploy Deepseek R1 671B on the NVIDIA A800?
bsakaguchi 2026-10-26 21:21:53 +00:00
bsakaguchi commented 2026-10-26 21:21:55 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (May 14, 2025):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

<!-- gh-comment-id:2878231427 --> @github-actions[bot] commented on GitHub (May 14, 2025): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!
bsakaguchi commented 2026-10-26 21:21:55 +00:00
Author
Owner
Copy link

@github-actions[bot] commented on GitHub (Jun 10, 2025):

false

<!-- gh-comment-id:2957338367 --> @github-actions[bot] commented on GitHub (Jun 10, 2025): false
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
v1.0.0
Labels
Clear labels   
bug

   
closed-as-stale

   
closed-as-stale

   
enhancement

   
pull-request
Mirrored from GitHub Pull Request

  
stale

   
stale
No labels
bug
closed-as-stale
closed-as-stale
enhancement
pull-request
stale
stale
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepSeek-R1#472
No description provided.