Skip to content

Allow grouping of input files - #4

Merged
joshuatorrance merged 10 commits into
mainfrom
3-yearly-output
Jul 31, 2026
Merged

Allow grouping of input files#4
joshuatorrance merged 10 commits into
mainfrom
3-yearly-output

Conversation

@joshuatorrance

@joshuatorrance joshuatorrance commented Jul 21, 2026

Copy link
Copy Markdown
Collaborator

This PR adds a new feature to allow for the grouping of input files into output files. E.g. 12 monthly files into 1 yearly file per variable.

  • Code
  • Tests
  • Docs

Closes #3
Closes #7

@joshuatorrance joshuatorrance self-assigned this Jul 21, 2026
@joshuatorrance
joshuatorrance marked this pull request as ready for review July 21, 2026 05:58
@joshuatorrance
joshuatorrance requested a review from blimlim July 21, 2026 05:58
@blimlim

blimlim commented Jul 23, 2026

Copy link
Copy Markdown
Collaborator

Thanks @joshuatorrance for putting this together. The implementation with the wild group is really neat. I haven't looked through the code changes/tests in detail yet, but have started by running the new code on a year of ESM1.6 output in /g/data/tm70/sw6175/development/esm1p6/test_splitnc/hist-r1-output000. This has brought up a couple small catches related to the data:

Encodings
Combining daily outputs, and combining subdaily outputs fails for both the atmosphere and sea ice. Because of the different month lengths, the original_shape, chunksizes and preferred_chunks values in the encodings aren't always the same across the files. E.g comparing encodings in two of the 3hrly output files from the atmosphere:

fld_s03i237 original_shape
file1: (248, 145, 192) file2: (224, 145, 192)
--------
fld_s05i215 chunksizes
file1: (124, 73, 96) file2: (112, 73, 96)
--------
fld_s05i215 preferred_chunks
file1: {'time_0': 124, 'lat': 73, 'lon': 96} file2: {'time_0': 112, 'lat': 73, 'lon': 96}
--------

I think we could drop the original_shape value without issue. What would the best way be to deal with the chunksizes and preferred_chunks values?

Inconsistent surface altitude
The 3D atmosphere variables come with a 2D surface_altitude coordinate attached. Due to how it's been calculated in the monthly mean files, it has rounding level differences in different files:
download-65

This causes an issue with xr.open_mfdataset:

 xarray.structure.merge.MergeError: conflicting values for variable 'surface_altitude' on objects to be combined. You can skip this check by specifying compat='override'.

Not sure what the best way around this is, perhaps we could catch this error and use compat='override' when it occurs, or have a preprocessing step that rounds surface_altitude when it is present?

Dropping surface_altitude from the datasets, no other issues come up when merging.

@joshuatorrance

Copy link
Copy Markdown
Collaborator Author

Inconsistent surface altitude

This change should handle the inconsistent surface_altitude fields.
74ec035

It adds an extra preprocessing step to just round off 'surface_altitude' to 4 decimal places. I've piggybacked this behaviour off of the --use-esm1p6-filenames to keep things simple.

@joshuatorrance

joshuatorrance commented Jul 24, 2026

Copy link
Copy Markdown
Collaborator Author

Encodings

original_shape, chunksizes and preferred_chunks are now also being excluded from the saved encodings:
0547716

None of these are important for splitnc. If we end up with more encodings causing problems we can consider only including the encodings we need, probably just coordinates but I'd have to confirm.

@blimlim blimlim left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still testing running it on full model output, but thought I'd put in my comments on the code in the meantime. I really like the implementation, and just had a couple of small comments

Comment thread src/splitnc/splitnc.py Outdated
Comment thread src/splitnc/splitnc.py Outdated

@blimlim blimlim left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @joshuatorrance, the outputs from this look good when testing it on a years worth of ESM1.6 data. I compared the results to um2nc's single variable output on yearly files produced natively by the model, and everything I checked was identical up to cosmetic differences.

joshuatorrance and others added 4 commits July 31, 2026 10:07
Co-authored-by: Spencer Wong <88933912+blimlim@users.noreply.github.com>
…tcdf. Added option to skip existing files.

Combined these options allow interrupted runs to recover faster, i.e. skip writing out to files that have already been finished
@joshuatorrance

Copy link
Copy Markdown
Collaborator Author

I just added to option to --skip-existing files.

splitnc now writes to file.temp then moves to file after to_netcdf

Combined these two allow for a interrupted run to recover without having to re-write completed files.

@joshuatorrance joshuatorrance changed the title Allow grouping of input files into output files Allow grouping of input files Jul 31, 2026
@joshuatorrance
joshuatorrance merged commit b1563de into main Jul 31, 2026
4 checks passed
@joshuatorrance
joshuatorrance deleted the 3-yearly-output branch July 31, 2026 04:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Record dimensions in output files Generating yearly output with splitnc

2 participants