deepseek-ai/DeepEP
1
0
Fork 0
mirror of https://github.com/deepseek-ai/DeepEP.git synced 2026-11-05 00:47:32 +00:00
Code Issues 308 Projects Packages Wiki Activity

[GH-ISSUE #402] Combine precision loss is serious. Stable reproduce #272

New issue
Closed
opened 2026-10-26 09:57:45 +00:00 by bsakaguchi · 3 comments
bsakaguchi commented 2026-10-26 09:57:45 +00:00
Owner
Copy link

Originally created by @alpha-baby on GitHub (Sep 11, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/402

run test_low_latency test on 6*H200 failed.

Machine information

#nvidia-smi topo -m
        GPU0    GPU1    GPU2    GPU3    GPU4    GPU5    GPU6    GPU7    NIC0    NIC1    NIC2    NIC3    NIC4    NIC5    NIC6    NIC7    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      NV18    NV18    NV18    NV18    NV18    NV18    NV18    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS     0-47,96-143     0               N/A
GPU1    NV18     X      NV18    NV18    NV18    NV18    NV18    NV18    NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS     0-47,96-143     0               N/A
GPU2    NV18    NV18     X      NV18    NV18    NV18    NV18    NV18    NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS     0-47,96-143     0               N/A
GPU3    NV18    NV18    NV18     X      NV18    NV18    NV18    NV18    NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     0-47,96-143     0               N/A
GPU4    NV18    NV18    NV18    NV18     X      NV18    NV18    NV18    SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    48-95,144-191   1               N/A
GPU5    NV18    NV18    NV18    NV18    NV18     X      NV18    NV18    SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE    48-95,144-191   1               N/A
GPU6    NV18    NV18    NV18    NV18    NV18    NV18     X      NV18    SYS     SYS     SYS     SYS     NODE    NODE    PIX     NODE    48-95,144-191   1               N/A
GPU7    NV18    NV18    NV18    NV18    NV18    NV18    NV18     X      SYS     SYS     SYS     SYS     NODE    NODE    NODE    PIX     48-95,144-191   1               N/A
NIC0    PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS      X      NODE    NODE    NODE    SYS     SYS     SYS     SYS
NIC1    NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS     NODE     X      NODE    NODE    SYS     SYS     SYS     SYS
NIC2    NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS     NODE    NODE     X      NODE    SYS     SYS     SYS     SYS
NIC3    NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     NODE    NODE    NODE     X      SYS     SYS     SYS     SYS
NIC4    SYS     SYS     SYS     SYS     PIX     NODE    NODE    NODE    SYS     SYS     SYS     SYS      X      NODE    NODE    NODE
NIC5    SYS     SYS     SYS     SYS     NODE    PIX     NODE    NODE    SYS     SYS     SYS     SYS     NODE     X      NODE    NODE
NIC6    SYS     SYS     SYS     SYS     NODE    NODE    PIX     NODE    SYS     SYS     SYS     SYS     NODE    NODE     X      NODE
NIC7    SYS     SYS     SYS     SYS     NODE    NODE    NODE    PIX     SYS     SYS     SYS     SYS     NODE    NODE    NODE     X 

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

NIC Legend:

  NIC0: mlx5_bond_0
  NIC1: mlx5_bond_1
  NIC2: mlx5_bond_2
  NIC3: mlx5_bond_3
  NIC4: mlx5_bond_4
  NIC5: mlx5_bond_5
  NIC6: mlx5_bond_6
  NIC7: mlx5_bond_7

start command:

WORLD_SIZE=6 RANK=0 python ./tests/test_low_latency.py
WORLD_SIZE=6 RANK=1 python ./tests/test_low_latency.py
WORLD_SIZE=6 RANK=2 python ./tests/test_low_latency.py
WORLD_SIZE=6 RANK=3 python ./tests/test_low_latency.py
WORLD_SIZE=6 RANK=4 python ./tests/test_low_latency.py
WORLD_SIZE=6 RANK=5 python ./tests/test_low_latency.py

log: node-rank6

/tmp/origin-DeepEP/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 46 nranks 48 tag 6 - DONE
/tmp/origin-DeepEP/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 41 nranks 48 tag 6 - DONE
/tmp/origin-DeepEP/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 45 nranks 48 tag 6 - DONE
/tmp/origin-DeepEP/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 42 nranks 48 tag 6 - DONE
/tmp/origin-DeepEP/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 40 nranks 48 tag 6 - DONE
/tmp/origin-DeepEP/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 47 nranks 48 tag 6 - DONE
/tmp/origin-DeepEP/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 43 nranks 48 tag 6 - DONE
WARNING: destroy() was not called before DeepEP buffer destruction, which can leak resources.
W0911 19:30:43.860000 232900 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232965 via signal SIGTERM
W0911 19:30:43.860000 232900 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232966 via signal SIGTERM
W0911 19:30:43.861000 232900 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232967 via signal SIGTERM
W0911 19:30:43.862000 232900 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232968 via signal SIGTERM
W0911 19:30:43.862000 232900 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232970 via signal SIGTERM
W0911 19:30:43.862000 232900 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232971 via signal SIGTERM
W0911 19:30:43.863000 232900 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232972 via signal SIGTERM
Traceback (most recent call last):
  File "/home/cpfs/fujianhao.fjh/origin-DeepEP/tests/test_low_latency.py", line 223, in <module>
    torch.multiprocessing.spawn(test_loop, args=(num_processes, args), nprocs=num_processes)
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
    while not context.join():
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 215, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 4 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap
    fn(i, *args)
  File "/home/cpfs/fujianhao.fjh/origin-DeepEP/tests/test_low_latency.py", line 179, in test_loop
    test_main(num_tokens, hidden, num_experts, num_topk, rank, num_ranks, group, buffer,
  File "/home/cpfs/fujianhao.fjh/origin-DeepEP/tests/test_low_latency.py", line 116, in test_main
    assert diff < (9e-4 if dispatch_use_fp8 else 1e-5), f'Error: {diff=}, {dispatch_use_fp8=}, {zero_copy=}'
AssertionError: Error: diff=0.0011682080943252737, dispatch_use_fp8=True, zero_copy=False

log node-rank5:

WARNING: destroy() was not called before DeepEP buffer destruction, which can leak resources.
W0911 19:30:43.907000 232826 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232891 via signal SIGTERM
W0911 19:30:43.907000 232826 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232892 via signal SIGTERM
W0911 19:30:43.908000 232826 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232893 via signal SIGTERM
W0911 19:30:43.909000 232826 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232894 via signal SIGTERM
W0911 19:30:43.909000 232826 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232896 via signal SIGTERM
W0911 19:30:43.910000 232826 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232897 via signal SIGTERM
W0911 19:30:43.910000 232826 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232898 via signal SIGTERM
Traceback (most recent call last):
  File "/home/cpfs/fujianhao.fjh/origin-DeepEP/tests/test_low_latency.py", line 223, in <module>
    torch.multiprocessing.spawn(test_loop, args=(num_processes, args), nprocs=num_processes)
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
    while not context.join():
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 215, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 4 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap
    fn(i, *args)
  File "/home/cpfs/fujianhao.fjh/origin-DeepEP/tests/test_low_latency.py", line 179, in test_loop
    test_main(num_tokens, hidden, num_experts, num_topk, rank, num_ranks, group, buffer,
  File "/home/cpfs/fujianhao.fjh/origin-DeepEP/tests/test_low_latency.py", line 116, in test_main
    assert diff < (9e-4 if dispatch_use_fp8 else 1e-5), f'Error: {diff=}, {dispatch_use_fp8=}, {zero_copy=}'
AssertionError: Error: diff=0.0009134306187068564, dispatch_use_fp8=True, zero_copy=False

othen ranks no failed log.

Originally created by @alpha-baby on GitHub (Sep 11, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/402 run test_low_latency test on 6*H200 failed. Machine information ``` #nvidia-smi topo -m GPU0 GPU1 GPU2 GPU3 GPU4 GPU5 GPU6 GPU7 NIC0 NIC1 NIC2 NIC3 NIC4 NIC5 NIC6 NIC7 CPU Affinity NUMA Affinity GPU NUMA ID GPU0 X NV18 NV18 NV18 NV18 NV18 NV18 NV18 PIX NODE NODE NODE SYS SYS SYS SYS 0-47,96-143 0 N/A GPU1 NV18 X NV18 NV18 NV18 NV18 NV18 NV18 NODE PIX NODE NODE SYS SYS SYS SYS 0-47,96-143 0 N/A GPU2 NV18 NV18 X NV18 NV18 NV18 NV18 NV18 NODE NODE PIX NODE SYS SYS SYS SYS 0-47,96-143 0 N/A GPU3 NV18 NV18 NV18 X NV18 NV18 NV18 NV18 NODE NODE NODE PIX SYS SYS SYS SYS 0-47,96-143 0 N/A GPU4 NV18 NV18 NV18 NV18 X NV18 NV18 NV18 SYS SYS SYS SYS PIX NODE NODE NODE 48-95,144-191 1 N/A GPU5 NV18 NV18 NV18 NV18 NV18 X NV18 NV18 SYS SYS SYS SYS NODE PIX NODE NODE 48-95,144-191 1 N/A GPU6 NV18 NV18 NV18 NV18 NV18 NV18 X NV18 SYS SYS SYS SYS NODE NODE PIX NODE 48-95,144-191 1 N/A GPU7 NV18 NV18 NV18 NV18 NV18 NV18 NV18 X SYS SYS SYS SYS NODE NODE NODE PIX 48-95,144-191 1 N/A NIC0 PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE SYS SYS SYS SYS NIC1 NODE PIX NODE NODE SYS SYS SYS SYS NODE X NODE NODE SYS SYS SYS SYS NIC2 NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE X NODE SYS SYS SYS SYS NIC3 NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE X SYS SYS SYS SYS NIC4 SYS SYS SYS SYS PIX NODE NODE NODE SYS SYS SYS SYS X NODE NODE NODE NIC5 SYS SYS SYS SYS NODE PIX NODE NODE SYS SYS SYS SYS NODE X NODE NODE NIC6 SYS SYS SYS SYS NODE NODE PIX NODE SYS SYS SYS SYS NODE NODE X NODE NIC7 SYS SYS SYS SYS NODE NODE NODE PIX SYS SYS SYS SYS NODE NODE NODE X Legend: X = Self SYS = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI) NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node PHB = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU) PXB = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge) PIX = Connection traversing at most a single PCIe bridge NV# = Connection traversing a bonded set of # NVLinks NIC Legend: NIC0: mlx5_bond_0 NIC1: mlx5_bond_1 NIC2: mlx5_bond_2 NIC3: mlx5_bond_3 NIC4: mlx5_bond_4 NIC5: mlx5_bond_5 NIC6: mlx5_bond_6 NIC7: mlx5_bond_7 ``` start command: ``` WORLD_SIZE=6 RANK=0 python ./tests/test_low_latency.py WORLD_SIZE=6 RANK=1 python ./tests/test_low_latency.py WORLD_SIZE=6 RANK=2 python ./tests/test_low_latency.py WORLD_SIZE=6 RANK=3 python ./tests/test_low_latency.py WORLD_SIZE=6 RANK=4 python ./tests/test_low_latency.py WORLD_SIZE=6 RANK=5 python ./tests/test_low_latency.py ``` log: node-rank6 ``` /tmp/origin-DeepEP/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 46 nranks 48 tag 6 - DONE /tmp/origin-DeepEP/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 41 nranks 48 tag 6 - DONE /tmp/origin-DeepEP/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 45 nranks 48 tag 6 - DONE /tmp/origin-DeepEP/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 42 nranks 48 tag 6 - DONE /tmp/origin-DeepEP/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 40 nranks 48 tag 6 - DONE /tmp/origin-DeepEP/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 47 nranks 48 tag 6 - DONE /tmp/origin-DeepEP/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 43 nranks 48 tag 6 - DONE WARNING: destroy() was not called before DeepEP buffer destruction, which can leak resources. W0911 19:30:43.860000 232900 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232965 via signal SIGTERM W0911 19:30:43.860000 232900 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232966 via signal SIGTERM W0911 19:30:43.861000 232900 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232967 via signal SIGTERM W0911 19:30:43.862000 232900 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232968 via signal SIGTERM W0911 19:30:43.862000 232900 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232970 via signal SIGTERM W0911 19:30:43.862000 232900 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232971 via signal SIGTERM W0911 19:30:43.863000 232900 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232972 via signal SIGTERM Traceback (most recent call last): File "/home/cpfs/fujianhao.fjh/origin-DeepEP/tests/test_low_latency.py", line 223, in <module> torch.multiprocessing.spawn(test_loop, args=(num_processes, args), nprocs=num_processes) File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes while not context.join(): File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 215, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException: -- Process 4 terminated with the following error: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap fn(i, *args) File "/home/cpfs/fujianhao.fjh/origin-DeepEP/tests/test_low_latency.py", line 179, in test_loop test_main(num_tokens, hidden, num_experts, num_topk, rank, num_ranks, group, buffer, File "/home/cpfs/fujianhao.fjh/origin-DeepEP/tests/test_low_latency.py", line 116, in test_main assert diff < (9e-4 if dispatch_use_fp8 else 1e-5), f'Error: {diff=}, {dispatch_use_fp8=}, {zero_copy=}' AssertionError: Error: diff=0.0011682080943252737, dispatch_use_fp8=True, zero_copy=False ``` log node-rank5: ``` WARNING: destroy() was not called before DeepEP buffer destruction, which can leak resources. W0911 19:30:43.907000 232826 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232891 via signal SIGTERM W0911 19:30:43.907000 232826 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232892 via signal SIGTERM W0911 19:30:43.908000 232826 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232893 via signal SIGTERM W0911 19:30:43.909000 232826 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232894 via signal SIGTERM W0911 19:30:43.909000 232826 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232896 via signal SIGTERM W0911 19:30:43.910000 232826 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232897 via signal SIGTERM W0911 19:30:43.910000 232826 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 232898 via signal SIGTERM Traceback (most recent call last): File "/home/cpfs/fujianhao.fjh/origin-DeepEP/tests/test_low_latency.py", line 223, in <module> torch.multiprocessing.spawn(test_loop, args=(num_processes, args), nprocs=num_processes) File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes while not context.join(): File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 215, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException: -- Process 4 terminated with the following error: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap fn(i, *args) File "/home/cpfs/fujianhao.fjh/origin-DeepEP/tests/test_low_latency.py", line 179, in test_loop test_main(num_tokens, hidden, num_experts, num_topk, rank, num_ranks, group, buffer, File "/home/cpfs/fujianhao.fjh/origin-DeepEP/tests/test_low_latency.py", line 116, in test_main assert diff < (9e-4 if dispatch_use_fp8 else 1e-5), f'Error: {diff=}, {dispatch_use_fp8=}, {zero_copy=}' AssertionError: Error: diff=0.0009134306187068564, dispatch_use_fp8=True, zero_copy=False ``` othen ranks no failed log.
bsakaguchi commented 2026-10-26 09:57:47 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Sep 23, 2025):

Did this error occur when round_scale = True? If so, this is a known issue — round scale can indeed lead to precision loss, so the accuracy tolerance should be relaxed during testing.

<!-- gh-comment-id:3323282060 --> @sphish commented on GitHub (Sep 23, 2025): Did this error occur when round_scale = True? If so, this is a known issue — round scale can indeed lead to precision loss, so the accuracy tolerance should be relaxed during testing.
bsakaguchi commented 2026-10-26 09:57:47 +00:00
Author
Owner
Copy link

@alpha-baby commented on GitHub (Sep 24, 2025):

Did this error occur when round_scale = True? If so, this is a known issue — round scale can indeed lead to precision loss, so the accuracy tolerance should be relaxed during testing.

yes

print value: round_scale , my log:

/tmp/debug-combine-AntDeepEP/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 47 nranks 48 tag 6 - DONE
WARNING: destroy() was not called before DeepEP buffer destruction, which can leak resources.
W0924 14:37:09.301000 47426 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 47491 via signal SIGTERM
W0924 14:37:09.302000 47426 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 47492 via signal SIGTERM
W0924 14:37:09.302000 47426 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 47493 via signal SIGTERM
W0924 14:37:09.303000 47426 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 47494 via signal SIGTERM
W0924 14:37:09.303000 47426 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 47496 via signal SIGTERM
W0924 14:37:09.304000 47426 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 47497 via signal SIGTERM
W0924 14:37:09.305000 47426 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 47498 via signal SIGTERM
Traceback (most recent call last):
  File "/home/cpfs/fujianhao.fjh/origin-DeepEP/tests/test_low_latency.py", line 224, in <module>
    torch.multiprocessing.spawn(test_loop, args=(num_processes, args), nprocs=num_processes)
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn
    return start_processes(fn, args, nprocs, join, daemon, start_method="spawn")
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes
    while not context.join():
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 215, in join
    raise ProcessRaisedException(msg, error_index, failed_process.pid)
torch.multiprocessing.spawn.ProcessRaisedException: 

-- Process 4 terminated with the following error:
Traceback (most recent call last):
  File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap
    fn(i, *args)
  File "/home/cpfs/fujianhao.fjh/origin-DeepEP/tests/test_low_latency.py", line 180, in test_loop
    test_main(num_tokens, hidden, num_experts, num_topk, rank, num_ranks, group, buffer,
  File "/home/cpfs/fujianhao.fjh/origin-DeepEP/tests/test_low_latency.py", line 116, in test_main
    assert diff < (9e-4 if dispatch_use_fp8 else 1e-5), f'Error: {diff=}, {dispatch_use_fp8=}, {zero_copy=}, {round_scale=}'
AssertionError: Error: diff=0.0011682080943252737, dispatch_use_fp8=True, zero_copy=False, round_scale=True
<!-- gh-comment-id:3326808024 --> @alpha-baby commented on GitHub (Sep 24, 2025): > Did this error occur when round_scale = True? If so, this is a known issue — round scale can indeed lead to precision loss, so the accuracy tolerance should be relaxed during testing. yes print value: `round_scale` , my log: ``` /tmp/debug-combine-AntDeepEP/nvshmem_src/src/modules/bootstrap/uid/bootstrap_uid.cpp:bootstrap_uid_barrier:575: rank 47 nranks 48 tag 6 - DONE WARNING: destroy() was not called before DeepEP buffer destruction, which can leak resources. W0924 14:37:09.301000 47426 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 47491 via signal SIGTERM W0924 14:37:09.302000 47426 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 47492 via signal SIGTERM W0924 14:37:09.302000 47426 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 47493 via signal SIGTERM W0924 14:37:09.303000 47426 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 47494 via signal SIGTERM W0924 14:37:09.303000 47426 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 47496 via signal SIGTERM W0924 14:37:09.304000 47426 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 47497 via signal SIGTERM W0924 14:37:09.305000 47426 site-packages/torch/multiprocessing/spawn.py:169] Terminating process 47498 via signal SIGTERM Traceback (most recent call last): File "/home/cpfs/fujianhao.fjh/origin-DeepEP/tests/test_low_latency.py", line 224, in <module> torch.multiprocessing.spawn(test_loop, args=(num_processes, args), nprocs=num_processes) File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 340, in spawn return start_processes(fn, args, nprocs, join, daemon, start_method="spawn") File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 296, in start_processes while not context.join(): File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 215, in join raise ProcessRaisedException(msg, error_index, failed_process.pid) torch.multiprocessing.spawn.ProcessRaisedException: -- Process 4 terminated with the following error: Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/torch/multiprocessing/spawn.py", line 90, in _wrap fn(i, *args) File "/home/cpfs/fujianhao.fjh/origin-DeepEP/tests/test_low_latency.py", line 180, in test_loop test_main(num_tokens, hidden, num_experts, num_topk, rank, num_ranks, group, buffer, File "/home/cpfs/fujianhao.fjh/origin-DeepEP/tests/test_low_latency.py", line 116, in test_main assert diff < (9e-4 if dispatch_use_fp8 else 1e-5), f'Error: {diff=}, {dispatch_use_fp8=}, {zero_copy=}, {round_scale=}' AssertionError: Error: diff=0.0011682080943252737, dispatch_use_fp8=True, zero_copy=False, round_scale=True ```
bsakaguchi commented 2026-10-26 09:57:48 +00:00
Author
Owner
Copy link

@sphish commented on GitHub (Oct 11, 2025):

The round_scale check has already been removed in a recent commit.

<!-- gh-comment-id:3392729017 --> @sphish commented on GitHub (Oct 11, 2025): The `round_scale` check has already been removed in a recent commit.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
hybrid-ep
chore/remove-clang-format-yapf
v2.5
nvDev
try-fix-cu128
mori-ep
antgroup-opt
compat-old-torch
add-linter
eager-rdma
v1.2.1
Labels
Clear labels   
pull-request
Mirrored from GitHub Pull Request

No labels
pull-request
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
deepseek-ai/DeepEP#272
No description provided.