Improve model generalization

[SarahAlidoost]

The current model after merging #91 is able to learn the train set (without validation dataset in training workflow). In other words, it can overfit the train data which shows the good model capacity. We have seen this in the example notebooks when data covers only one month and train_test split is done spatially on a small subset.

But for real scenario, we are interested in using the model on a different temporal coverage. So we changed the train_test split as: train 202001, validation 202101, test 202201 (same months different years). After running the training workflow including validation, we see underfitting which is normal considering the amount of data and the architecture of the model. Still the model performs better than baseline.

There are several approaches in improving the generalization:

• using 3 years of data for training and 1 year for validation --> in progress after Bug in dataset when calculating geo_pos_embedding_patch #91
• using other loss metrics
• using anomalies as input to the model rather than daily/hourly SST field
• adding other climate co-variate to the encoder channel
• adding a spatial mixing layer in decoder to improve the spatial correlation
• ....

These can be explored one by one.

[SarahAlidoost]

@meiertgrootes and @rogerkuou this is related to our discussion on model underfitting issue.