Model Evaluation

[ionut-grigore99]

I have a question regarding model evaluation. I do not see any validation loss or validation metrics in the training script. What is the recommended way to determine whether the model is overfitting or underfitting during FLUX.2 Inpainting + ControlNet training? Is there a standard validation setup, evaluation metric, or early-stopping strategy that you recommend?

[Artiprocher]

@ionut-grigore99 This is a somewhat controversial topic.

• In our earlier versions, we supported displaying the training loss. However, the loss values fluctuated dramatically during training, making it nearly impossible to draw any meaningful conclusions from the loss curves—a source of considerable confusion.
• Therefore, we recommend adopting an "evaluate-while-training" approach. Multiple checkpoints are saved throughout the training process, allowing you to evaluate each checkpoint at any time and subjectively assess model performance.
• Currently, if you still wish to monitor the loss, you can enable logging via enable_tensorboard_log, --enable_swanlab_log, or --enable_wandb_log.
• Additionally, we have integrated a variety of image quality assessment metrics. For more details, please refer to https://diffsynth-studio-doc.readthedocs.io/en/latest/Model_Details/Image-Quality-Metrics.html