[GH-ISSUE #176] Why is it necessary to convert BF16 to FP32 for addition in reduce operations? #151

New issue

Closed

opened 2026-10-26 09:55:16 +00:00 by bsakaguchi · 1 comment

bsakaguchi commented

2026-10-26 09:55:16 +00:00

Owner

Originally created by @lixingyii on GitHub (May 24, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/176

We noticed that in the code for reduce operations, the hidden data needs to be read from the buffer, converted from BF16 to float (FP32) for summation, and then converted back to BF16 for storage. Is this data format conversion necessary? Will directly using BF16 for addition affect numerical precision?

Originally created by @lixingyii on GitHub (May 24, 2025). Original GitHub issue: https://github.com/deepseek-ai/DeepEP/issues/176 We noticed that in the code for reduce operations, the hidden data needs to be read from the buffer, converted from BF16 to float (FP32) for summation, and then converted back to BF16 for storage. Is this data format conversion necessary? Will directly using BF16 for addition affect numerical precision? ![Image](https://github.com/user-attachments/assets/cbe81b86-754b-4213-a66a-cc2324ba15ef)

bsakaguchi closed this issue

2026-10-26 09:55:17 +00:00

bsakaguchi commented

2026-10-26 09:55:17 +00:00

Author

Owner

@LyricZhao commented on GitHub (May 27, 2025):

Is this data format conversion necessary? Will directly using BF16 for addition affect numerical precision?

Yes and yes. BF16 addition has low precision due to its number of mantissa, which will leads to lower model performance (tested with experiments). Generally, the whole model training process cannot have even one BF16 accumulator. FP32 accumulator is a must.

@LyricZhao commented on GitHub (May 27, 2025): > Is this data format conversion necessary? Will directly using BF16 for addition affect numerical precision? Yes and yes. BF16 addition has low precision due to its number of mantissa, which will leads to lower model performance (tested with experiments). Generally, the whole model training process cannot have even one BF16 accumulator. FP32 accumulator is a must.

bsakaguchi referenced this issue

2026-10-26 10:00:48 +00:00

[PR #151] [MERGED] Feat: enhance nvidia peer memory detection #443