[GH-ISSUE #688] [Feature Request] Save the last n checkpoint #413

New issue

Open

opened 2026-10-20 09:59:06 +00:00 by bsakaguchi · 3 comments

bsakaguchi commented

2026-10-20 09:59:06 +00:00

Owner

Originally created by @nikita488 on GitHub (Jul 4, 2023).
Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/688

While saving only the latest '.ckpt' file to save disk space is a great idea, there are several situations where something unexpected happens like some error in Colab that is corrupting latest D/G_2333333.pth file (not saving it fully), or saving only D while not having time to save G checkpoint (or vise versa) because it was interrupted.

I trained a few models using Colab and a lot of them had corrupted D or G checkpoint because some error happened while it was saving, so i lost hours of trained data and need to start from scratch which is very annoying.

My idea is to have some option inside UI like "Save last n checkpoints". As an example, if you set this option to 10 and the 'Save frequency (save_every_epoch)' set to 10, it will save 10, 20, 30 ... 100 epoch D and G checkpoints, then it will delete oldest checkpoints (D_10.pth and G_10.pth in this example), and save latest one (save 110, then delete 20 epoch checkpoints and save 120 and continue). That way we will have some backups and can continue to train model from a older checkpoint if something bad happened with latest checkpoint.

Originally created by @nikita488 on GitHub (Jul 4, 2023). Original GitHub issue: https://github.com/RVC-Project/Retrieval-based-Voice-Conversion-WebUI/issues/688 While saving only the latest '.ckpt' file to save disk space is a great idea, there are several situations where something unexpected happens like some error in Colab that is corrupting latest D/G_2333333.pth file (not saving it fully), or saving only D while not having time to save G checkpoint (or vise versa) because it was interrupted. I trained a few models using Colab and a lot of them had corrupted D or G checkpoint because some error happened while it was saving, so i lost hours of trained data and need to start from scratch which is very annoying. My idea is to have some option inside UI like "Save last n checkpoints". As an example, if you set this option to 10 and the 'Save frequency (save_every_epoch)' set to 10, it will save 10, 20, 30 ... 100 epoch D and G checkpoints, then it will delete oldest checkpoints (D_10.pth and G_10.pth in this example), and save latest one (save 110, then delete 20 epoch checkpoints and save 120 and continue). That way we will have some backups and can continue to train model from a older checkpoint if something bad happened with latest checkpoint.

bsakaguchi added the

following up

todo list

labels

2026-10-20 09:59:06 +00:00

bsakaguchi commented

2026-10-20 09:59:07 +00:00

Author

Owner

@RVC-Boss commented on GitHub (Jul 5, 2023):

ok, in the next version I will modify it to save the latestest 2 ckpt.

@RVC-Boss commented on GitHub (Jul 5, 2023): ok, in the next version I will modify it to save the latestest 2 ckpt.

bsakaguchi commented

2026-10-20 09:59:07 +00:00

Author

Owner

@nikita488 commented on GitHub (Jul 5, 2023):

Great!

@nikita488 commented on GitHub (Jul 5, 2023): Great!

bsakaguchi commented

2026-10-20 09:59:07 +00:00

Author

Owner

@nikita488 commented on GitHub (Jul 5, 2023):

Renaming existing checkpoints to G/D_*.pth.bak and saving new ones probably will do

@nikita488 commented on GitHub (Jul 5, 2023): Renaming existing checkpoints to G/D_*.pth.bak and saving new ones probably will do