Steps for model fine-tuning or resuming training

[cosalexelle]

Hi!

Great work with the fork!

I have two models trained on some dataset G_1200 and D_1200 and want to continue training them using the same speaker and dataset with these as a base. How would we do that?

I no longer have the tensorboard outputs, only the two G/D models.

Would the correct method be to run svc pre-resample svc pre-config svc pre-hubert on dataset_raw as usual, then before training, place the two G/D models in logs/44k/ and rename them to G_0 and D_0?

That's how I am running it now.

The metric "loss/g/total" seems to start slightly higher from where the previous model training left off: starting around 36.0 and after a few minutes reduces to approximately 31.5. This seems higher than expected, but could be due the dataset being re-preprocessed, so different items may be in train/test/val.

Edit: tensorboard also shows starting at approx step 7000, which I assume is embedded in the models?

Are there any other recommendations on how to fine-tune the models?

(on an unrelated note, I am creating a new Google Colab notebook with some enhancements which I may submit a pull request for later)

[Citramones]

Hello, did you get something?

[brandostrong]

Yes, this is what you would do. However, on pre-config, so-vits-svc will choose entirely new train/test/val datasets, so it makes sense you would see a spike in loss. I've actually had some success shuffling around the train/test/val, just wait it out and maybe it'll make your model more accurate in the long run. This happens because loss is calculated by comparing what the model generates at that step to the train/val files selected during pre-config. If you run it for a thousand epochs it'll get really good at approximating those original train/val, and now they've changed it has train for the new ones.

[venturaEffect]

I'm following the steps here. Tryin to fine-tune the speaker with new data. Use same speaker, added the new dataset on raw_dataset. Took out the old raw_dataset, I have run: svc pre-resample, svc pre-config, svc pre-hubert. And then I changed the G_10000 and the D_10000 to G_0 and D_0. Just left this files on logs/44k and run svc train -t. But it want super fast, in less than a minute and I think it should take way longer. Am I doing something wrong? Maybe increasing the epochs? Appreciate any suggestion.

[34j]

As the current epoch is derived from the model and not from the file name due to the compatibility, you need to modify the value to change it, or you can just simply continue training with the model unchanged with a different dataset.

[venturaEffect]

Got it! Thanks!