Allow grouping of input files

README.md

@@ -20,6 +20,30 @@ If there are ancillary fields that should only be present in only some of the ou
 
 Example of these variables are the `latitude_longitude` found in atmosphere files or the `uarea`, `tmask`, `tarea`, `VGRDb`, `VGRDi`, `VGRDs` variables from ice files.
 
+## Grouping Input Files
+
+Input file paths can be grouped together using the `--input-group-regex INPUT_GROUP_REGEX` command line option.
+This can be used for example to group together 12 monthly input files into one yearly output file per variable.
+This option uses a regex with a named capture group "wild" to identify the portion of the files that varies across the group.
+For example, to group togther these two sets of monthly files into yearly output files:
+```
+aiihca.pa-234501_mon.nc  aiihca.pe-234501_dai.nc
+aiihca.pa-234502_mon.nc  aiihca.pe-234502_dai.nc
+aiihca.pa-234503_mon.nc  aiihca.pe-234503_dai.nc
+aiihca.pa-234504_mon.nc  aiihca.pe-234504_dai.nc
+aiihca.pa-234505_mon.nc  aiihca.pe-234505_dai.nc
+aiihca.pa-234506_mon.nc  aiihca.pe-234506_dai.nc
+aiihca.pa-234507_mon.nc  aiihca.pe-234507_dai.nc
+aiihca.pa-234508_mon.nc  aiihca.pe-234508_dai.nc
+aiihca.pa-234509_mon.nc  aiihca.pe-234509_dai.nc
+aiihca.pa-234510_mon.nc  aiihca.pe-234510_dai.nc
+aiihca.pa-234511_mon.nc  aiihca.pe-234511_dai.nc
+aiihca.pa-234512_mon.nc  aiihca.pe-234512_dai.nc
+```
+use the regex `aiihca\.p[ae]-\d{4}(?P<wild>\d{2})_(mon|dai)\.nc` which uses the `wild` capture group to match the month.
+
+Any file paths that don't match the regex will be processed individually.
+
 ## Config File
 
 The `-c`/`--command-line-file` option can be used to supply a filepath to a file that contains command line options.
@@ -73,13 +97,19 @@ options:
                         access-esm1p6.{component}.{dimensions}.{field}.{freq}.{time_cell_method}.{datestamp}.nc
                         splitnc will attempt to deduce all the components of the filename. If this option is not given
                         {field}_{original_filename} will be used.
+                        Note: This option also enables special preprocessing for ESM1.6 files.
   --fix-cell-methods    Correct cell_methods by adding 'time: point' to cell_methods for variables that have 'time' but
                         not 'time_bnds' and no other 'time' cell_methods.
   --file-freq FILE_FREQ
                         Specify the frequency of the files (not the data), e.g. if each file contains a month of data
                         then the file-frequency is '1mon'. Used to determine the resolution of the timestamp for ESM1.6
                         filenames. Follows the ACCESS frequency vocabulary (e.g. '1yr', '1mon', '1day', '1hr'), any
                         unrecognised frequency will use the full timestamp. Defaults to '1yr'.
+  --input-group-regex INPUT_GROUP_REGEX
+                        Specify a regex that will be used to group a subset of the input into a single set. E.g. group together 12
+                        input monthly files to a single year of output. Use a named capture group "wild" to specify the portion of
+                        the filename that varies. E.g. to group monthly files with this pattern - "aiihca.pa-YYYYMM_mon.nc" - use
+                        the regex \"aiihca\.pa-\d{4}(?P<wild>\d{2})_mon\.nc\".
   --output-dir OUTPUT_DIR
                         Output directory for the processed files. If not given output files will be placed in the same
                         directory as the original file.

pyproject.toml

@@ -17,6 +17,7 @@ requires-python = ">=3.9"
 dependencies = [
     "netcdf4",
     "xarray",
+    "dask",
 ]
 
 [project.optional-dependencies]

src/splitnc/esm1p6.py

@@ -131,7 +131,8 @@ def _build_datestamp(ds, field_name, file_freq):
 
     # Calculate the middle point
     first, last = time_arr.min(), time_arr.max()
-    datestamp_dt = (first + (last - first) / 2).dt
+    datestamp_dt = (first + (last - first) / 2).compute().dt
+        # Need to .compute when using open_mfdataset
 
     return "." + datestamp_dt.strftime(fmt).data.flatten()[0]
 
@@ -156,3 +157,17 @@ def build_esm1p6_filename(ds, field_name, input_filepath, esm1p6_filename=False,
         raise
 
     return template.format(**d)
+
+
+def preprocess_esm1p6_files(ds):
+    """
+    Prepare ESM1.6 files before grouping.
+    - Round off surface_altitude in atmos files to remove variation due to numerical issues
+    """
+    if 'surface_altitude' in ds:
+        # Surface altitude is measured in meters
+        # surface_altitude in different files can vary down around 10^-10
+        # Round it off to 4 decimal places
+        ds['surface_altitude'] = ds['surface_altitude'].round(4)
+
+    return ds