@guoday Hi, I tried to fine-tune the model in fp16 mode which results in large training loss (>4.0) and NaN eval loss. Turning off the fp16 mode solves the loss problem but the training speed is unbearably slow. It seems that the loss problem is due to the original model training in bf16 mode. How could I fine-tune the model in fp16 mode to obtain normal results, i.e. without the large and NaN loss problem? Great thanks.

@II-Matto commented on GitHub (Nov 22, 2023): @guoday Hi, I tried to fine-tune the model in fp16 mode which results in large training loss (>4.0) and NaN eval loss. Turning off the fp16 mode solves the loss problem but the training speed is unbearably slow. It seems that the loss problem is due to the original model training in bf16 mode. How could I fine-tune the model in fp16 mode to obtain normal results, i.e. without the large and NaN loss problem? Great thanks.

bsakaguchi commented

2026-10-26 21:02:23 +00:00

Author

Owner

@guoday commented on GitHub (Nov 22, 2023):

@guoday Hi, I tried to fine-tune the model in fp16 mode which results in large training loss (>4.0) and NaN eval loss. Turning off the fp16 mode solves the loss problem but the training speed is unbearably slow. It seems that the loss problem is due to the original model training in bf16 mode. How could I fine-tune the model in fp16 mode to obtain normal results, i.e. without the large and NaN loss problem? Great thanks.

Do you use apex when you fine-tune the model in fp16?

@guoday commented on GitHub (Nov 22, 2023): > @guoday Hi, I tried to fine-tune the model in fp16 mode which results in large training loss (>4.0) and NaN eval loss. Turning off the fp16 mode solves the loss problem but the training speed is unbearably slow. It seems that the loss problem is due to the original model training in bf16 mode. How could I fine-tune the model in fp16 mode to obtain normal results, i.e. without the large and NaN loss problem? Great thanks. Do you use apex when you fine-tune the model in fp16?

bsakaguchi commented

2026-10-26 21:02:24 +00:00

Author

Owner

@II-Matto commented on GitHub (Nov 22, 2023):

@guoday I used the Llama-X for fine-tuning, which is quite similar to the DeepSeek-Coder official fine-tuning script. I have not found any explicit clue that apex is used. Could you give some more specific guidance or links to references?

@II-Matto commented on GitHub (Nov 22, 2023): @guoday I used the [Llama-X](https://github.com/AetherCortex/Llama-X) for fine-tuning, which is quite similar to the DeepSeek-Coder official [fine-tuning script](https://github.com/deepseek-ai/DeepSeek-Coder/blob/main/finetune/finetune_deepseekcoder.py). I have not found any explicit clue that apex is used. Could you give some more specific guidance or links to references?

bsakaguchi commented

2026-10-26 21:02:24 +00:00

Author

Owner

@askxiaozhang commented on GitHub (Nov 22, 2023):

如果是fp16的话，大概需要14G～15G的显存

好的，非常感谢！

@askxiaozhang commented on GitHub (Nov 22, 2023): > 如果是fp16的话，大概需要14G～15G的显存好的，非常感谢！

bsakaguchi commented

2026-10-26 21:02:24 +00:00

Author

Owner

@soloice commented on GitHub (Nov 28, 2023):

@guoday I used the Llama-X for fine-tuning, which is quite similar to the DeepSeek-Coder official fine-tuning script. I have not found any explicit clue that apex is used. Could you give some more specific guidance or links to references?

There's no apex configuration in our released code and we do not use fp16/amp when training our model. If you want to train it with fp16, I guess you'll need to find a training framework that supports amp (Maybe DeepSpeed?)
Closed this issue because the original question has been addressed. Feel free to open another issue if you still have questions.

@soloice commented on GitHub (Nov 28, 2023): > @guoday I used the [Llama-X](https://github.com/AetherCortex/Llama-X) for fine-tuning, which is quite similar to the DeepSeek-Coder official [fine-tuning script](https://github.com/deepseek-ai/DeepSeek-Coder/blob/main/finetune/finetune_deepseekcoder.py). I have not found any explicit clue that apex is used. Could you give some more specific guidance or links to references? There's no apex configuration in our released code and we do not use fp16/amp when training our model. If you want to train it with fp16, I guess you'll need to find a training framework that supports amp (Maybe DeepSpeed?) Closed this issue because the original question has been addressed. Feel free to open another issue if you still have questions.