[GH-ISSUE #1148] [Bug]: FP8 mixed precision training produces inconsistent results across different GPU architectures #725

New issue

Closed

opened 2026-10-26 21:27:33 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-26 21:27:33 +00:00

Owner

Originally created by @RewFa-CEO on GitHub (Mar 11, 2026).
Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V3/issues/1148

环境信息

操作系统：Ubuntu 22.04
编程语言：Python 3.11
CUDA版本：12.4
GPU：H800 (SXM 80GB)
框架版本：DeepSeek-V3 base

问题描述

使用FP8混合精度训练时，不同GPU架构产生不一致的结果。H800和H100上的训练loss曲线存在显著差异。

复现步骤

使用官方提供的训练脚本启动FP8混合精度训练
在H800上训练1000步，记录loss
在H100上使用相同配置训练1000步，记录loss
对比两个平台的loss曲线

预期行为

相同配置下，不同GPU架构应该产生相近的训练结果。

实际行为

H800和H100的loss曲线存在5-10%的差异，且收敛速度不同。

报错日志

# H800 FP8 training
Step 1000: loss=1.234
Step 2000: loss=0.876

# H100 FP8 training  
Step 1000: loss=1.312
Step 2000: loss=0.912

最小复现代码

# training script
import torch
import torch.distributed as dist

# FP8 training config
torch.set_float32_matmul_precision('high')
model = your_model()

# Train on H800 vs H100
# Observe different loss trajectories

初步分析

FP8格式在不同Tensor Core架构上的实现存在差异。NVIDIA H800使用的FP8 Tensor Core与H100可能存在精度差异，导致累积误差不同。

建议解决方案

在文档中明确说明不同GPU架构的训练结果可能存在差异
提供FP8精度的校准指南
考虑添加环境变量控制FP8精度行为
在跨架构训练时使用BF16作为中间精度

Reported by Financier-Nuri from Unum AI

Originally created by @RewFa-CEO on GitHub (Mar 11, 2026). Original GitHub issue: https://github.com/deepseek-ai/DeepSeek-V3/issues/1148 ## 环境信息 - 操作系统：Ubuntu 22.04 - 编程语言：Python 3.11 - CUDA版本：12.4 - GPU：H800 (SXM 80GB) - 框架版本：DeepSeek-V3 base ## 问题描述使用FP8混合精度训练时，不同GPU架构产生不一致的结果。H800和H100上的训练loss曲线存在显著差异。 ## 复现步骤 1. 使用官方提供的训练脚本启动FP8混合精度训练 2. 在H800上训练1000步，记录loss 3. 在H100上使用相同配置训练1000步，记录loss 4. 对比两个平台的loss曲线 ## 预期行为相同配置下，不同GPU架构应该产生相近的训练结果。 ## 实际行为 H800和H100的loss曲线存在5-10%的差异，且收敛速度不同。 ## 报错日志 ``` # H800 FP8 training Step 1000: loss=1.234 Step 2000: loss=0.876 # H100 FP8 training Step 1000: loss=1.312 Step 2000: loss=0.912 ``` ## 最小复现代码 ```python # training script import torch import torch.distributed as dist # FP8 training config torch.set_float32_matmul_precision('high') model = your_model() # Train on H800 vs H100 # Observe different loss trajectories ``` ## 初步分析 FP8格式在不同Tensor Core架构上的实现存在差异。NVIDIA H800使用的FP8 Tensor Core与H100可能存在精度差异，导致累积误差不同。 ## 建议解决方案 1. 在文档中明确说明不同GPU架构的训练结果可能存在差异 2. 提供FP8精度的校准指南 3. 考虑添加环境变量控制FP8精度行为 4. 在跨架构训练时使用BF16作为中间精度 --- *Reported by Financier-Nuri from Unum AI*

bsakaguchi

2026-10-26 21:27:33 +00:00

closed this issue
added the
stale

closed-as-stale
labels

bsakaguchi commented

2026-10-26 21:27:34 +00:00

Author

Owner

@qingkong66 commented on GitHub (Mar 19, 2026):

Thanks for the detailed report — this kind of cross‑architecture consistency is exactly what makes large‑scale training reproducible. Hope the team picks this up.

@qingkong66 commented on GitHub (Mar 19, 2026): Thanks for the detailed report — this kind of cross‑architecture consistency is exactly what makes large‑scale training reproducible. Hope the team picks this up.

bsakaguchi commented

2026-10-26 21:27:34 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (Apr 21, 2026):

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

@github-actions[bot] commented on GitHub (Apr 21, 2026): This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. If you believe this issue is still relevant, please leave a comment to keep it open. Thank you for your contributions!

bsakaguchi commented

2026-10-26 21:27:34 +00:00

Author

Owner

@github-actions[bot] commented on GitHub (May 9, 2026):

false

@github-actions[bot] commented on GitHub (May 9, 2026): false