Generate proto messages for non-indexed sourced_from source types - #1361
Open
jwils wants to merge 1 commit into
Open
Generate proto messages for non-indexed sourced_from source types#1361jwils wants to merge 1 commit into
jwils wants to merge 1 commit into
Conversation
`schema.proto` only had messages for the indexed types and the types they reference, so a publisher could not send an event for a pure-source type: one that feeds a `sourced_from` field but has no index of its own. #1346 widened the JSON schema event envelope to accept these events, but proto ingestion never got the same treatment. The proto traversal now starts from the ingestible types -- the indexed types plus the `sourced_from` source types -- instead of the indexed types alone. The source-type names come from `Results#sourced_update_targets_by_source_type_name`, the same memoized resolver pass the event envelope uses, so the two formats cannot disagree about which types are sources. The messages themselves needed no change: `proto_fields` builds from `indexing_fields_by_name_in_index`, which does not depend on the type having an index. Derived indexing types are now excluded from the traversal seed. ElasticGraph builds their documents from the events of other types, so a publisher never sends one, and the JSON schema artifact has never described them either. This removes their messages from `schema.proto`, which Buf reports as a breaking change. No published schema depends on them yet. An abstract source type keeps its `oneof` wrapper message, as an indexed abstract type does. Note that the JSON event envelope does not accept an abstract type by name; that gap is not addressed here. Closes #1360
jwils
requested review from
BrianSigafoos-SQ,
bsorbo,
ellisandrews-toast,
jwondrusch,
marcdaniels-toast,
myronmarston and
rossroberts-toast
as code owners
August 19, 2026 21:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #1360. Stacked on #1358, so review that one first.
schema.protoonly had messages for the indexed types and the types they reference, so a publisher could not send an event for a pure-source type: one that feeds asourced_fromfield but has no index of its own. #1346 widened the JSON schema event envelope to accept these events, but proto ingestion never got the same treatment. Nothing in an indexed type's proto representation refers to a pure-source type, so the traversal never reached it.The traversal now starts from the ingestible types -- the indexed types plus the
sourced_fromsource types -- instead of the indexed types alone. The source-type names come fromResults#sourced_update_targets_by_source_type_name, the same memoized resolver pass the event envelope uses, so the two formats cannot disagree about which types are sources.The messages themselves needed no change, as #1360 predicted.
proto_fieldsbuilds fromindexing_fields_by_name_in_index, which does not depend on the type having an index:Derived indexing types (a Buf-breaking change)
Derived indexing types are now excluded from the seed. ElasticGraph builds their documents from the events of other types, so a publisher never sends one, and the JSON schema artifact has never described them either. Proto ingestion described them only because a derived type has its own index, which put it in the old seed.
Removing a message is a breaking change under the Buf check that #1352 adds. No published schema depends on these messages yet, so this is the moment to align the two formats.
Abstract source types
A
sourced_fromsource type can be an interface or a union. Such a type keeps theoneofwrapper message it would get as an indexed abstract type, and its subtypes get messages too.Note the JSON event envelope does not accept an abstract type by name, and it does not accept the concrete subtypes either, because they are neither indexed nor source types. So an abstract source type is not publishable over JSON today. That gap is not addressed here; it belongs on the JSON side.
Notes
Rakefiledoes not register the proto extension, and this repo commits noschema.proto.sourced_fromresolver, so it can raise relationship andsourced_fromvalidation errors that it previously skipped. No existing proto spec defines a relationship, so no fixture needed a repair.