Skip to content

Generate proto messages for non-indexed sourced_from source types - #1361

Open
jwils wants to merge 1 commit into
joshuaw/proto-spec-organizationfrom
joshuaw/proto-sourced-from-source-types
Open

Generate proto messages for non-indexed sourced_from source types#1361
jwils wants to merge 1 commit into
joshuaw/proto-spec-organizationfrom
joshuaw/proto-sourced-from-source-types

Conversation

@jwils

@jwils jwils commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Summary

Closes #1360. Stacked on #1358, so review that one first.

schema.proto only had messages for the indexed types and the types they reference, so a publisher could not send an event for a pure-source type: one that feeds a sourced_from field but has no index of its own. #1346 widened the JSON schema event envelope to accept these events, but proto ingestion never got the same treatment. Nothing in an indexed type's proto representation refers to a pure-source type, so the traversal never reached it.

The traversal now starts from the ingestible types -- the indexed types plus the sourced_from source types -- instead of the indexed types alone. The source-type names come from Results#sourced_update_targets_by_source_type_name, the same memoized resolver pass the event envelope uses, so the two formats cannot disagree about which types are sources.

The messages themselves needed no change, as #1360 predicted. proto_fields builds from indexing_fields_by_name_in_index, which does not depend on the type having an index:

message ComponentDesign {
  string id = 1;
  string component_id = 2;
  string designer_name = 3;
  // Next field number: 4
}

Derived indexing types (a Buf-breaking change)

Derived indexing types are now excluded from the seed. ElasticGraph builds their documents from the events of other types, so a publisher never sends one, and the JSON schema artifact has never described them either. Proto ingestion described them only because a derived type has its own index, which put it in the old seed.

Removing a message is a breaking change under the Buf check that #1352 adds. No published schema depends on these messages yet, so this is the moment to align the two formats.

Abstract source types

A sourced_from source type can be an interface or a union. Such a type keeps the oneof wrapper message it would get as an indexed abstract type, and its subtypes get messages too.

Note the JSON event envelope does not accept an abstract type by name, and it does not accept the concrete subtypes either, because they are neither indexed nor source types. So an abstract source type is not publishable over JSON today. That gap is not addressed here; it belongs on the JSON side.

Notes

  • No schema artifact churn. The root Rakefile does not register the proto extension, and this repo commits no schema.proto.
  • Proto schema generation now runs the sourced_from resolver, so it can raise relationship and sourced_from validation errors that it previously skipped. No existing proto spec defines a relationship, so no fixture needed a repair.
  • Coverage stays at 100% for line and branch.

`schema.proto` only had messages for the indexed types and the types they
reference, so a publisher could not send an event for a pure-source type: one
that feeds a `sourced_from` field but has no index of its own. #1346 widened
the JSON schema event envelope to accept these events, but proto ingestion
never got the same treatment.

The proto traversal now starts from the ingestible types -- the indexed types
plus the `sourced_from` source types -- instead of the indexed types alone. The
source-type names come from `Results#sourced_update_targets_by_source_type_name`,
the same memoized resolver pass the event envelope uses, so the two formats
cannot disagree about which types are sources. The messages themselves needed no
change: `proto_fields` builds from `indexing_fields_by_name_in_index`, which does
not depend on the type having an index.

Derived indexing types are now excluded from the traversal seed. ElasticGraph
builds their documents from the events of other types, so a publisher never
sends one, and the JSON schema artifact has never described them either. This
removes their messages from `schema.proto`, which Buf reports as a breaking
change. No published schema depends on them yet.

An abstract source type keeps its `oneof` wrapper message, as an indexed
abstract type does. Note that the JSON event envelope does not accept an
abstract type by name; that gap is not addressed here.

Closes #1360
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant