Repository navigation
What's new in 2.3.4 - #289
Merged
Merged
Conversation
Documents the hidden `_prov` attribute added in datajoint-python 2.3.4. The feature's entire author-facing surface is configuration — insert takes no argument and no author can write the attribute — so the documentation carries what the API does not. New: - reference/specs/boundary-provenance.md — the specification: the attribute, which tiers carry it and why Imported does not, the three content sources, the framework-ownership invariant, configuration, retrofitting through deploy.add_prov_column, and querying a hidden JSON attribute - how-to/record-data-origin.md — configure the source, confirm rows are carrying an origin, find rows that are not, and what to model instead when a row needs something only it can say - examples/fan_out_provenance.py — the fan-out shape with the origin recorded Revised: - reference/configuration.md — the config.provenance settings - explanation/fan-out-ingestion.md — the section that told authors to record each row's origin by hand now separates the two jobs: the framework records the audit trail, and the author models the link the pipeline itself queries. The worked example keeps its source_file column, since hidden attributes are excluded from query composition and cannot replace a modelled one. - explanation/comparison-to-provenance-systems.md — the claim that entry-point tables record source identity now has a standard shape behind it - tutorials/basics/03-data-entry.ipynb — entry is the boundary, so the data entry tutorial says what is recorded there and how to read it back - mkdocs.yaml — two nav entries The tutorial addition is a markdown cell only; no code cells or outputs change, so the notebook needs no re-execution.
Adds the 2.3.4 section to the 2.3 release notes, covering the hidden `_prov` attribute and the two provenance settings, and names the release's shape in the page intro. Also corrects a forward reference the 2.3.3 notes carried: the codec `context=` parameter that replaces the `key["_config"]` threading was said to be scheduled for 2.3.4, but #1550 now sits on the 2.3.5 milestone.
#1550 moves onto the 2.3.4 milestone, so the 2.3.3 forward reference is correct again and the release gains a bullet of its own. The bullet leads on what the argument is for and states plainly that codecs written before 2.3.4 keep working, since that is the first question a codec author will have.
dimitri-yatsenko
requested review from
MilagrosMarin,
lum-agilyti and
ttngu207
September 30, 2026 22:17
The slot is granted by matching the Manual tier rather than by excluding the other tiers' prefixes, so job queues and lineage tables are excluded by construction. Worth stating: the first implementation enumerated prefixes and job tables slipped through, which is the kind of thing a reader should be able to check against the spec.
Three pages advertised a mapping restriction on `_prov`
(`& {"_prov.system": "PyRat"}`). That form is silently ignored on a hidden
attribute: no WHERE clause is emitted and every row comes back. Documenting it
was worse than documenting nothing, since the natural uses are audit questions
and the wrong answer looks like a right one. Replaced with the string form,
which works, plus a warning pointing at datajoint-python#1561.
Reading a hidden attribute back needs SQL until 2.4. job-metadata.md had
advertised `to_arrays('_job_start_time', ...)` as the access path since the
feature shipped; that call has never worked, because the heading excludes
hidden names and every accessor derives from it. The spec now says so and shows
the SQL, pointing at datajoint-python#1562 for the supported accessor.
Covers reference/specs/boundary-provenance.md, how-to/record-data-origin.md,
the data-entry tutorial, and reference/specs/job-metadata.md — the last being
the original claim behind datajoint-docs#1553.
Tutorial edit is a markdown cell; no code cells or outputs change.
The previous commit framed that ignoring as a defect. It is not: a mapping restriction drops attributes it cannot match so that `Session & key` works when `key` carries attributes from a more detailed downstream table. Passing a full key dict down the graph is the normal idiom and `make()` depends on it. The narrow point stands — a hidden attribute is invisible to that matching, so naming one in a mapping drops the predicate and returns every row — but the reason is the design working as intended on an attribute it cannot see, not a bug in the ignoring. Reworded across all four pages so a reader does not come away thinking unmatched attributes should raise. The guidance is unchanged: write the condition as a string.
The recommended workaround used MySQL's JSON_VALUE, which fails on PostgreSQL with UndefinedFunction — verified on postgres:15. Half the supported backends were given a condition that does not run. Both spellings are now shown, with the reason stated: DataJoint's own JSON-path translation is what the mapping form would have provided, and that is exactly the path a hidden attribute cannot take. IS NULL / IS NOT NULL remain portable.
The previous wording read as a general claim that filtering inside a JSON
attribute has no portable spelling. It does: {"data.system": "PyRat"} on an
ordinary json attribute returns the right rows on MySQL 8.0 and postgres:15
alike, because translate_attribute hands the path to adapter.json_path_expr.
The backend-specific SQL is needed here only because _prov is hidden and the
mapping form cannot reach a hidden attribute — which is datajoint-python#1561,
not a JSON defect. Reworded so a reader does not conclude JSON paths are
generally unportable.
Audited every page that mentions `_prov`, `_job_*` or `_singleton` against
MySQL 8.0 and postgres:15. Four claims were wrong.
- `autopopulate.md` advertised `to_arrays('result', '_job_duration')`, which
raises: the heading excludes hidden names. Replaced with the condition
string that does work and the SQL read that is needed until 2.4.
- `_job_duration` was documented as `float64` in two reference tables. It is
`float32` -- `float` on MySQL, `real` on PostgreSQL.
- `table-declaration.md`'s platform-attribute table omitted `_prov`, and said
these columns are written by raw SQL during populate. That is true of
`_job_*` only; `_prov` is written on the insert path.
- `job-metadata.md` showed the declaration building hand-written MySQL
strings. That code is gone -- the columns are DataJoint notation compiled
through the type system, which is what makes them correct on PostgreSQL.
The query rules now have their own section in `job-metadata.md`, which the
other pages point at instead of restating. Every row of it is verified on both
backends: the string restriction works, the mapping form returns every row,
`to_arrays` and `proj` raise, `Top(order_by=...)` passes through.
Also noted that the job-metadata retrofit never ran on PostgreSQL before
2.3.4, and dropped the reference to `provenance_columns()`, which no longer
exists.
Two user-visible changes in the milestone were missing from the notes. `dj.Entry`, `dj.Ingest` and `dj.Compute` are permanent aliases, verified as the same classes rather than subclasses, so nothing about `describe()` or the diagram changes. The platform-column refactor is a fix, not only tidying: on PostgreSQL `_job_start_time` had microsecond precision where MySQL had millisecond, and the job-metadata retrofit had never run there at all.
The bullet read as though the rename had happened. What ships in 2.3.4 is the second set of names (#1558); making them primary is #1546, slated for 2.4, and the docs keep the original names until then. Also dropped "which already say what they are" of Lookup and Part -- a comment on the other names by implication, and the release notes have no reason to make it.
Follows datajoint/datajoint-python#1568. Capture changes the DDL of every Manual table declared after it is enabled, so it is a deployment's decision rather than a library default -- `jobs.add_job_metadata` defaults off for the same reason. - The spec's "Capture defaults on" section argued the opposite case. It now states the one the default rests on, and keeps the property a deployment gets once it enables capture. - Both settings tables and the config example read `False`. - The how-to told readers to configure a source first and turn capture off later, which meant following it from the top recorded nothing. "Turn capture on" is now the first step. - The tutorial said DataJoint records the origin of every Manual insert. It does where a deployment has enabled it.
Follows datajoint/datajoint-python#1568. The notes promised a column on every Manual table and argued the case for defaulting on, both of which are now wrong. Capture is a deployment's decision, and the upgrade note is the stronger line: an unchanged schema declares under 2.3.4 exactly what it declared under 2.3.3.
ttngu207
previously approved these changes
Oct 4, 2026
dimitri-yatsenko
dismissed
ttngu207’s stale review
October 4, 2026 19:59
The base branch was changed.
Member
Author
|
Re-approval needed, @ttngu207 — sorry, this one is on me. Your approval was dismissed by a push, and I merged All four checks pass. Worth knowing when you re-read: |
ttngu207
approved these changes
Oct 4, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Draft release notes for 2.3.4.
Stacked on #288 — it links the two new pages that PR adds, so the base is
docs/entry-provenancerather thanmain. Merging #288 first retargets this automatically. Onmainalone,mkdocs build --strictfails on the three unresolved links.What it says
Two bullets in a new Changes in 2.3.4 section, matching the per-patch format the page already uses for 2.3.1–2.3.3:
_provattribute — what the boundary problem is, that no author writes it, where the content comes from, what makes fan-out traceable, and that tables declared earlier record nothing untildj.deploy.add_prov_columnadds the column.provenance.*settings, including why capture defaults to on.The page intro gains the release's shape alongside the diagram and S3 entries already there.
A correction it carries
The 2.3.3 notes say the codec
context=parameter replacing thekey["_config"]threading is "scheduled ... in 2.3.4", referencing #1550. That issue is now on the v2.3.5 milestone, so the published sentence points at the wrong release. Corrected here, since it is the same page.Scope caveat
2.3.4's milestone holds two issues. Only #1547 has an implementation (datajoint-python#1555, open). #1546 — the
dj.Entry/dj.Ingest/dj.Computealiases — has no PR yet, so it is deliberately not described here; the aliases are mentioned only in the new spec'sversion-addednote, which #288 carries. Add a bullet for them when that lands, or move the issue off the milestone.Nothing here should merge before datajoint-python#1555 does — these notes describe behavior that is not on
masteryet.Verification
mkdocs build --strictclean andcheck_links.pypasses across 144 pages on the stacked base.