Add preliminary support for ISO-8601 timestamps via date: archive match pattern (#8715) by c-herz · Pull Request #8776 · borgbackup/borg

c-herz · 2025-04-19T22:55:28Z

This PR adds preliminary support for matching ISO 8601 timestamps with the date: archive filter, and intends to begin addressing the requirements of #8715.

Timestamps (except for Unix epoch forms) are currently assumed to be in the user's local timezone and converted to UTC internally. The following formats are currently supported:

YYYY
YYYY-MM
YYYY-MM-DD
YYYY-MM-DDTHH -> matches a 1-hour interval
YYYY-MM-DDTHH:MM -> matches a 1-minute interval
YYYY-MM-DDTHH:MM:SS -> matches a 1-second interval
YYYY-MM-DDTHH:MM:SS.ffff -> matches an exact timestamp, including fractional seconds
@123456789 -> Unix epoch (interpreted as UTC)

(Is the fractional-second exact match useful in practice? Feedback welcome on this.)

Still in progress:

Support for wildcard-style matching (date:2025-*-01)
User-specified timezones and offsets
Keywords like now, oldest, newest, etc
Explicit duration intervals
RFC 3339 / RFC 9557 format support
Test/documentation coverage

…moment)

PhrozenByte · 2025-04-20T13:42:57Z

Thanks for picking this up! ❤️

YYYY-MM-DDTHH:MM:SS.ffff -> matches an exact timestamp, including fractional seconds

@123456789 -> Unix epoch (interpreted as UTC)

(Is the fractional-second exact match useful in practice? Feedback welcome on this.)

What's the precision of an archive's creation time? From the code I assume it's with fractional seconds, right? I absolutely agree with you then: There should be a variant that is guaranteed to match a single archive. I feel like that Unix timestamps should also optionally support fractional seconds then.

YYYY-MM-DDTHH -> matches a 1-hour interval

From reading the code I derive that the 1-hour interval is matched exclusively, i.e. it's actually matching any archive within 00:59:59.9999… hours, correct? Perfect 👍

# Year/Year-month/Year-month-day
parts = expr.split("-")
try:
    if len(parts) == 1:                    # YYYY
        year = int(parts[0])

Even though I like the simplicity, I feel like that Borg should be pretty strict about the format, because being less strict easily leads to ambiguity. For example, is date:0 supposed to match any archive created in year 0? Probably, but it gets way less clear with (now deprecated) truncated ISO8601 dates: What does the pattern date:25-1 describe? January of the year 25, January 1925, or January 2025?

If there's no library that can be used, I always imagined that the code would basically revolve around a single, rather strict regex with bottom-up optional groups for year, month, day, hour, minutes, seconds, and fractal seconds, or * as wildcard, supplemented by another regex to match periods, and simple matchers for Unix timestamps and keywords. I'm not saying that this is the best approach, that's just what I imagined while writing #8715.

In general I like to encourage creating extensive unit tests as early as possible. It's elegant and simple code now (🚀 👍), but complexity will increase greatly when adding more and more features.

Note: I can read the code, but can't do an actual code review - for that I just don't known enough of Borg's code.

ThomasWaldmann

Thanks for the PR!

Some minor stuff I found...

…d. Working on testing now

ThomasWaldmann · 2025-04-21T12:37:48Z

BTW, if you install the pre-commit hook, you can have your commits automatically formatted.

https://borgbackup.readthedocs.io/en/stable/development.html#building-a-development-environment

Refactored timestamp parsing to use `groupdict` and verbose regex, as recommended. # Conflicts: # src/borg/helpers/time.py

…us testing later today

…mat to pass style checks

…d timezones; added mutual exclusion enforcement to wildcard+epoch

…maintainer recommendation (all tests passing)

…onstruction and interval parsing

…ith basic tests)

… preserve time-of-day in offset_n_months

…g; include tests for oldest/newest scenarios

…ive patterns; include tests

…N formats

ThomasWaldmann · 2025-04-25T08:44:59Z

+                 (?:
+                     :(?P<minute>\d{2}|\*)     # minute (MM or *)
+                     (?:
+                         :(?P<second>\d{2}(?:\.\d+)?|\*)  # second (SS or SS.fff or *)


Yay, that is a nice regex now. :-)

You could also deal with fractional seconds as with all other components: it is a optional component, so you can also match it with a named group and later check the groupdict.

ThomasWaldmann · 2025-04-25T08:47:08Z

+    r"(?:(?P<years>\d+)Y)?"
+    r"(?:(?P<months>\d+)M)?"
+    r"(?:(?P<weeks>\d+)W)?"
+    r"(?:(?P<days>\d+)D)?"
+    r"(?:(?P<hours>\d+)h)?"
+    r"(?:(?P<minutes>\d+)m)?"
+    r"(?:(?P<seconds>\d+)s)?"


didn't we use to have lowercase ymwd and uppercase HMS?

@c-herz, is that a custom format, or did I miss an ISO 8601 or RFC update? I know ISO-8601's P3Y6M4DT12H30M5S (P designator, 3 years, 6 months, 4 days, T time separator, 12 hours, 30 minutes, 5 seconds, all being optional) format. I honestly don't like that "official" format very much (it's so cumbersome…), especially in regards to using M for both months and minutes (it's still unambiguous due to the T separator), but I feel like we should support it, because it's an official part of ISO 8601.

However, I kinda like the idea of additionally supporting our own format. Like, why not support 12:34:56 (or similar, just a quick idea) to specify a 12 hours, 34 minutes, 56 seconds duration? Why not also support (space) as alternative to T to separate times? The designators could ignore case (i.e. also supporting 7d for 7 days), and we could allow a space after each term. This might go too far though. In general, I'm not sure whether we should put things into the "official" P designator, or rather additionally support our own (like the suggested D, e.g. D 3y 6m 4d 12:30:05?). Is there maybe another common or even formalized/standardized (like another RFC; I did some research, unfortunately I didn't find anything official or quasi-official) format? WDYT?

PhrozenByte · 2025-04-25T10:30:13Z

+     # ISO week date: YYYY-Www or YYYY-Www-D
+     (?P<isoweek_year>\d{4})-W(?P<isoweek_week>\d{2})(?:-(?P<isoweek_day>\d))?
+   | # Ordinal date: YYYY-DDD
+     (?P<ordinal_year>\d{4})-(?P<ordinal_day>\d{3})


This might be a bit silly, but it just got me thinking: Could we support ordinal days with wildcard years? 🤔 Same for ISO weeks, possibly even *-W*-5 (wow, that looks crazy 😆) to match all archives created on a Friday? Not sure whether users would use it, but if it's possible? WDYT?

PhrozenByte · 2025-04-25T10:31:24Z

+   | # Ordinal date: YYYY-DDD
+     (?P<ordinal_year>\d{4})-(?P<ordinal_day>\d{3})
+   | # Unix epoch
+     @(?P<epoch>\d+)


Also supporting fractal seconds here would be amazing! ❤️

Side note: I might read the regex wrong, but this also means @1745577106[Europe/Berlin] (or any other TZ format) is supported? AFAIK Unix timestamps are UTC per definition, right? Or is the TZ info used later for something else?

fractional :-)

PhrozenByte · 2025-04-25T11:27:08Z

+    Also supports:
+      TIMESTAMP/TIMESTAMP
+      TIMESTAMP/DURATION
+      DURATION/TIMESTAMP.


Great work! ❤️

Just DURATION (i.e. without a TIMESTAMP) isn't supported yet, right? Just specifying a duration is helpful to match the latest archives relative to "now" (which could be another useful keyword).

codecov · 2025-04-25T19:48:18Z

Codecov Report

❌ Patch coverage is 90.71038% with 17 lines in your changes missing coverage. Please review.
✅ Project coverage is 81.62%. Comparing base (d1899c1) to head (9cb5e5f).
⚠️ Report is 584 commits behind head on master.

Files with missing lines	Patch %	Lines
src/borg/helpers/time.py	88.66%	11 Missing and 6 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #8776      +/-   ##
==========================================
- Coverage   81.89%   81.62%   -0.28%     
==========================================
  Files          74       74              
  Lines       13324    13517     +193     
  Branches     1968     2008      +40     
==========================================
+ Hits        10912    11033     +121     
- Misses       1750     1802      +52     
- Partials      662      682      +20

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

ThomasWaldmann · 2025-05-09T20:12:07Z

before doing further changes, please rebase on current master branch to get the cython workaround. otherwise, builds will fail.

ThomasWaldmann · 2025-05-22T17:25:55Z

Guess I'ld like to merge this for next beta, can we finish this until then?

ThomasWaldmann · 2025-06-03T17:57:57Z

ping?

c-herz · 2025-06-05T18:13:17Z

My apologies, I have been caught up with finals and starting a new internship. I should be able to look into @PhrozenByte's suggestions by this weekend. Apologies for the delays and the rather poor communication!

ThomasWaldmann · 2025-08-05T14:32:17Z

ping? (no hurry, but if you find some time, it would be nice to finish this)

ThomasWaldmann · 2026-05-08T13:35:05Z

Note: this PR can't even run the tests anymore, but it's not the fault of the code here, but current Cython requires some fixes not yet present here, but fixed in current master branch.

Please rebase on current master.

ThomasWaldmann · 2026-05-08T13:41:33Z

Also, I would appreciate if we could get this into a mergable state ASAP.

We will soon have bigger changes and movements in the code, so there could be lots of merge conflicts coming.

ThomasWaldmann · 2026-06-22T17:12:43Z

ping

PhrozenByte · 2026-06-22T19:44:35Z

What do you think about reducing the scope of this PR? It might help move things forward and get the core functionality merged sooner.

A recent example is #8775, which was opened just one day before this PR and was successfully merged yesterday after its scope was reduced.

In particular, support for durations, from/to matching, and special keywords (e.g. "oldest") seems to be more involved than originally anticipated. Would it make sense to focus on date support first, perhaps with only very basic or no wildcard support, and leave the more complex parts for follow-up PRs?

c-herz · 2026-06-22T20:12:40Z

Re: @PhrozenByte's suggestion--that definitely would make it a bit easier for me to get things up to speed after a year of neglect.

My sincere apologies for the radio silence on my end. Over the past year life really got in the way for me and I just didn't have time to work on hobby projects. However, I certainly should've communicated that better.

In any case, I should have some time this weekend and possibly here and there throughout the week to clean things up and at least get the basic feature I worked on last year rebased onto master and integrated with the major changes since then (which to be honest, I need to catch up on). I can shelve any nice-to-haves for now and focus on core functionality, and certainly keep everything as a reference if these are desired in the future and/or I have time to extend. Please let me know what should be the highest priority moving forward!

PhrozenByte · 2026-06-22T20:57:06Z

There's really no need to apologize. We're all contributing in our spare time, and there are certainly more important things in life than getting a PR merged quickly 🙂

In my opinion, it should ultimately be up to you to decide which features you want to work on and where you want to draw the line for the initial implementation.

That said, since I originally suggested this in #8715, my personal motivation was simply being able to match archives by static dates. I create daily backups, and while the date used to be part of the archive name in Borg 1.x, archive series in Borg 2.0 work differently. Because of that, something as simple as a static match pattern like date:2026-06-22 would already cover my personal use case. Everything else suggested in #8715 mostly came from considering how the feature could be extended in useful ways. For example, the idea of even more coarse-grained dates such as date:2026-06 or date:2026-W26 came from considering users that create monthly or weekly backups.

But again, you're the one doing the actual work here, so feel free to draw the line wherever it makes sense to you, both in terms of implementation effort and what you actually feel like working on.

c-herz force-pushed the datefilter branch from 04c5569 to 9e1c910 Compare April 19, 2025 23:01

c-herz changed the title ~~Add preliminary support for ISO-8601 timestamps via date: archive match pattern~~ Add preliminary support for ISO-8601 timestamps via date: archive match pattern (#8715) Apr 19, 2025

Add preliminary support for ISO-8601 timestamps (no timezones at the …

282d70c

…moment)

c-herz force-pushed the datefilter branch from 9e1c910 to 282d70c Compare April 19, 2025 23:16

ThomasWaldmann requested changes Apr 20, 2025

View reviewed changes

Comment thread src/borg/helpers/time.py Outdated

Comment thread src/borg/helpers/time.py Outdated

Comment thread src/borg/helpers/time.py Outdated

degabe and others added 3 commits April 20, 2025 21:49

reformatted to pass style checks

db46cdb

Applied recommended changes from ThomasWald, still working as intende…

4363bf7

…d. Working on testing now

fix bug with local timezone attachment not correctly respecting DST

69e8608

PhrozenByte mentioned this pull request Apr 21, 2025

Prune by either int or interval for all retention policies #8775

Merged

c-herz added 5 commits April 22, 2025 01:39

Reformatted for consistency with code style guide

5c20d8f

Added basic test suite for ISO-8601 and Unix timestamp matching

6f1bcd4

Merge remote-tracking branch 'origin/dateFilterImprov' into datefilter

4060e94

Refactored timestamp parsing to use `groupdict` and verbose regex, as recommended. # Conflicts: # src/borg/helpers/time.py

add day-precision filter test for date:YYYY-MM-DD

e9a8c5f

support timezone suffixes in date: patterns and add tests

470758d

ThomasWaldmann reviewed Apr 22, 2025

View reviewed changes

Comment thread src/borg/helpers/time.py Outdated

degabe and others added 11 commits April 23, 2025 01:42

Wildcard working. Done some manual testing, will focus on more rigoro…

df2d33d

…us testing later today

add tests for wildcard support in date: archive match patterns; refor…

870bf7a

…mat to pass style checks

fix bug with wildcards in date: match patterns not respecting supplie…

461df75

…d timezones; added mutual exclusion enforcement to wildcard+epoch

remove stray testfile.txt

9553c35

refactor date: pattern parser to use structured bottom-up regex, per …

409733b

…maintainer recommendation (all tests passing)

refactor date: pattern parsing to use helper functions for datetime c…

de03806

…onstruction and interval parsing

add explicit time interval matching in date: archive match pattern (w…

796981c

…ith basic tests)

add duration-based interval support for date: archive match patterns;…

7b8a194

… preserve time-of-day in offset_n_months

add support for keyword-based date intervals in archive date: matchin…

8e3f1e4

…g; include tests for oldest/newest scenarios

refactor time.py: rename internal functions for clarity and consistency

904853d

add support for ISO week-date and ordinal-date matching in date: arch…

6032c4a

…ive patterns; include tests

c-herz force-pushed the datefilter branch from 899ee35 to 6032c4a Compare April 25, 2025 08:31

enhance compile_date_pattern docstring: clarify TIMESTAMP and DURATIO…

9cb5e5f

…N formats

ThomasWaldmann reviewed Apr 25, 2025

View reviewed changes

PhrozenByte reviewed Apr 25, 2025

View reviewed changes

c-herz marked this pull request as ready for review April 26, 2025 00:45

Uh oh!

Conversation

c-herz commented Apr 19, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

PhrozenByte commented Apr 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ThomasWaldmann left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ThomasWaldmann commented Apr 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ThomasWaldmann Apr 25, 2025

Choose a reason for hiding this comment

Uh oh!

ThomasWaldmann Apr 25, 2025

Choose a reason for hiding this comment

Uh oh!

PhrozenByte Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

PhrozenByte Apr 25, 2025

Choose a reason for hiding this comment

Uh oh!

PhrozenByte Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ThomasWaldmann May 8, 2026

Choose a reason for hiding this comment

Uh oh!

PhrozenByte Apr 25, 2025

Choose a reason for hiding this comment

Uh oh!

codecov Bot commented Apr 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ThomasWaldmann commented May 9, 2025

Uh oh!

ThomasWaldmann commented May 22, 2025

Uh oh!

ThomasWaldmann commented Jun 3, 2025

Uh oh!

c-herz commented Jun 5, 2025

Uh oh!

ThomasWaldmann commented Aug 5, 2025

Uh oh!

ThomasWaldmann commented May 8, 2026

Uh oh!

ThomasWaldmann commented May 8, 2026

Uh oh!

ThomasWaldmann commented Jun 22, 2026

Uh oh!

PhrozenByte commented Jun 22, 2026

Uh oh!

c-herz commented Jun 22, 2026

Uh oh!

PhrozenByte commented Jun 22, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

c-herz commented Apr 19, 2025 •

edited

Loading

PhrozenByte commented Apr 20, 2025 •

edited

Loading

ThomasWaldmann commented Apr 21, 2025 •

edited

Loading

PhrozenByte Apr 25, 2025 •

edited

Loading

PhrozenByte Apr 25, 2025 •

edited

Loading

codecov Bot commented Apr 25, 2025 •

edited

Loading