mirror of
https://github.com/deepseek-ai/DeepGEMM.git
synced 2026-11-05 00:47:26 +00:00
[GH-ISSUE #109] [Question] The clamp value in per_token_cast_to_fp8 #62
Labels
No labels
pull-request
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
deepseek-ai/DeepGEMM#62
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Originally created by @chenyu-jiang on GitHub (Jun 9, 2025).
Original GitHub issue: https://github.com/deepseek-ai/DeepGEMM/issues/109
Hi,
Thank you for the amazing work! May I ask a question about the fp8 format used:
In your test cases, e.g., per_token_cast_to_fp8, the amax value is clamped to 1e-4. May I know what is the purpose of such clamping? Is it just to avoid dividing by zero? Why is the value set to 1e-4 instead of lower (e.g., 1e-12)?
@LyricZhao commented on GitHub (Jun 9, 2025):
Nothing special, just a heuristic number to avoid dividing by zero.
But there is a kindly reminder: this value can ensure the scaling factor
>= 1e-4 / 448, which is greater than the FP32's max subnormal value. So that you can directly use bit operations to do round-to-power-of-2-scaling casts.@chenyu-jiang commented on GitHub (Jun 9, 2025):
Thanks for the prompt response and the enlightening reminder!
I tried to use such clamping in training, but found that 1e-4 clamping would result in larger quantization error for some gradients during backward. May I know if this optimization is only targeted for inference? Should we use a lower value during training?
@LyricZhao commented on GitHub (Jun 10, 2025):
1e-4is for both training and inference.@chenyu-jiang commented on GitHub (Jun 10, 2025):
Thanks for the clarification!
@chenyu-jiang commented on GitHub (Jun 24, 2025):
Hi, sorry for bothering again. I noticed that:
1e-4 / 448 = 2.23214286e-7, while the smallest normal value of FP32 is 2^(-126) ~= 1.1754944e-38.
So it seems that we can use a much smaller clamping value (e.g., 1e-12) while still making sure the scale stays in the range of normal values.
Could you let me know if the understanding is correct? Thanks.
@weiyx16 commented on GitHub (Aug 14, 2025):
We don't have enough experiments on this value. It's free to adjust it to your case and I'll appreciate your results~
@weiyx16 commented on GitHub (Aug 18, 2025):
We tried to change the amax clamp value from 1e-4 to 1e-12 on a pre-training task using a 9B MoE (total parameters) and around 400 Billion tokens. We didn't observe a huge difference in this new setting (around 0.001-0.002 loss difference).
A hypothesis is that clamping at 1e-4 leads to values lower than 1e-4/(448/2^-9) ~ 4.4e-10 underflow, and it's enough for most cases. And clamping at 1e-4 makes the max value imprecise: e.g. 1e-5 will be mapped to 1e-5*448/1e-4 = 44.8 ~ 44, but this error is smaller than the micro-scaling case with E8M0 Scaling Factor.
Anyway, it's free to clamp lower for your case.
@chenyu-jiang commented on GitHub (Aug 18, 2025):
Thanks for the insights!
In our case, clamping to 1e-4 leads to loss divergence during initial training phases, while 1e-12 clamping seems to work fine. We also hypothesize that this is caused by the underflow of certain values. I guess this could be specific to certain model or data?
@weiyx16 commented on GitHub (Aug 19, 2025):
Thanks for sharing your results! It may be highly entangled with the training cases. Could you share the model size and training task, like pre-training, sft or rl? Really appreciate it.
@chenyu-jiang commented on GitHub (Aug 19, 2025):
It was small MoE model with 2.7B active parameters trained from scratch.
@weiyx16 commented on GitHub (Aug 21, 2025):
Thanks for the information~