Extract indexing field metadata behind a format-neutral value object - #1351
Open
jwils wants to merge 1 commit into
Open
Extract indexing field metadata behind a format-neutral value object#1351jwils wants to merge 1 commit into
jwils wants to merge 1 commit into
Conversation
`Indexer::RecordPreparer` read its per-field metadata straight out of the
JSON schema artifacts: it received raw `{"type" => ..., "nameInIndex" =>
...}` hashes and called `eg_meta.fetch("nameInIndex")`. That made the
JSON schema representation part of the indexer's record-preparation
contract, so an ingestion format that has no JSON schemas could only
supply metadata by synthesizing JSON-schema-shaped hashes.
The metadata is now a `RecordPreparer::FieldMetadata` value object with
`type` and `name_in_index`, and `RecordPreparer::TypeMetadata` exposes
`fields_by_name` instead of `eg_meta_by_field_name`. The JSON schema key
names stay inside `elasticgraph-json_ingestion`, which converts them when
it builds its type metadata. Behavior is unchanged.
jwils
requested review from
BrianSigafoos-SQ,
bsorbo,
ellisandrews-toast,
jwondrusch,
marcdaniels-toast,
myronmarston and
rossroberts-toast
as code owners
August 15, 2026 21:58
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Indexer::RecordPreparerread its per-field indexing metadata straight out of the JSON schema artifacts. It received raw hashes shaped like{"type" => "ID!", "nameInIndex" => "id"}and calledeg_meta.fetch("nameInIndex")directly.That made the JSON schema representation part of the indexer's record-preparation contract.
elasticgraph-indexeris meant to stay ingestion-format-neutral, but an ingestion format with no JSON schemas (such as the plannedelasticgraph-proto_ingestionruntime) could only supply metadata by synthesizing JSON-schema-shaped hashes with camelCase keys.What
RecordPreparer::FieldMetadata, a value object withtypeandname_in_index.RecordPreparer::TypeMetadatanow exposesfields_by_name(aHash<String, FieldMetadata>) instead ofeg_meta_by_field_name.RecordPreparerusesfield.name_in_indexandfield.typeinstead of fetching camelCase JSON schema keys.JSONIngestion::RecordPreparerFactoryconverts the JSON schemaElasticGraphmetadata intoFieldMetadata, so the JSON schema key names stay insideelasticgraph-json_ingestion.RecordPreparerdirectly fromTypeMetadataandFieldMetadata, with no JSON schema involved. This is the path a non-JSON ingestion format will use.Behavior is unchanged. This is a pure refactor of the contract between the indexer and the ingestion format gems.
Follow-up
This removes the shape coupling. A later PR still needs to give non-JSON formats a source for this metadata. Today
name_in_indexand the indexing type are recorded only in the versioned JSON schema artifacts.Runtime metadata is not a sufficient substitute. Its
graphql_fields_by_nameentries recordname_in_index, but they carry no fieldtype, whichRecordPreparerneeds to select a scalar indexing preparer and to recurse into nested objects. The entries are also unversioned and describe the GraphQL field set, not the ingestion field set.Verification
elasticgraph-indexer,elasticgraph-json_ingestion,elasticgraph-indexer_lambda,elasticgraph-warehouse_lambda,elasticgraph-lambda_support,elasticgraph-warehouse,elasticgraph-localandelasticgraph-apollounit suites: 673 examples, 0 failuresscript/lint,script/spellcheck,script/type_check: all greenbundle exec rake schema_artifacts:check: artifacts up to dateStack
Current PR is marked with
->.