Add skip_record_validation_for indexer config for sampled backfill validation

[vermatron]

Add skip_record_validation_for indexer config for sampled backfill validation

Why

During large backfills of already-validated data, per-record JSON schema validation is wasted work. Every record walks the full schema (regex, enum, min/max, format, abstract-type discriminators) even though the source has already been validated upstream. Today there's no way to trade that cost for throughput.

This adds a config option that skips per-record validation for a chosen fraction of records, per GraphQL type, while keeping a sampled slice validated as a canary so schema drift still surfaces. It's a sibling of the existing skip_derived_indexing_type_updates backfill knob and follows the same shape.

Design notes:

• skip_record_validation_for maps a type name to a fraction in [0.0, 1.0]. 0.0 (or an absent key) validates everything, 1.0 skips everything, and values in between sample. The value is the fraction skipped, so 0.9 skips 90% and validates 10%.

• The skip decision is a Zlib.crc32 of the event id (type:id@vversion) bucketed into [0.0, 1.0) and compared against the rate. Same event id, same decision, so a retry never flips a record between validated and skipped, even across pods. String#hash won't do here: its seed is per-process, so two pods would disagree.

• The event envelope is always validated. Only the per-record schema walk gets sampled.

• Skipping isn't silent. Processor counts skipped records per batch and logs one RecordValidationSkipped line with per-type counts, the same way it logs ElasticGraphIndexingLatencies. Logging per record would drown a 1.0 backfill in log lines, so the count is aggregated per batch.

• Safety net for skipped abstract-type records: once validation is off, a record with a missing or unknown __typename can reach RecordPreparer. It now raises a typed RecordPreparer::UnknownTypeError, and Factory#build rescues it into a FailedEventError, so that one failure mode degrades a single event instead of the batch. The rescue is narrow (a typed error, not a bare rescue KeyError) so it can't hide schema-artifact bugs, and its message drops the offending value to avoid leaking record data. A pre-walk check was considered and dropped: it would re-implement prepare_for_index's recursion over nested fields and drift out of sync with it.

• That safety net is the only per-record isolation guarantee, and the config documentation says so explicitly rather than implying a broader one. With validation off, other malformed data that the schema walk would have caught can still raise an unhandled error and fail the whole batch: a value IndexingPreparers::Integer cannot coerce, a missing id_source path, or - after build has already returned success - the rollover index suffix and routing value computed lazily in Update#metadata during router.bulk. Since such a batch produces no partial-failure response, the queue redelivers all of its events and the malformed record fails them again on each retry until it drains to the DLQ. These paths are not new code, but this config is what makes them reachable, so the caveat ships with it.

The field defaults to {}, so nothing changes unless you set it. Additive and minor-release-safe.

What

Config:

indexer:
  skip_record_validation_for:
    Widget: 0.9      # skip validation for 90% of Widget records, validate 10%
    Component: 1.0   # skip validation for all Component records

• config.rb: new skip_record_validation_for JSON schema property (object, per-type number in [0, 1], additionalProperties: false, default {}); convert_values coerces rates to Float. The description: states which failure mode is isolated and which ones can still fail a batch.
• operation/factory.rb: new skip_validation?(type, event) helper; build skips record validation when sampled, records the skipped type on the result, and rescues UnknownTypeError. BuildResult gains validation_skipped_for.
• processor.rb: aggregate RecordValidationSkipped log per batch when any record was skipped.
• record_preparer.rb: new RecordPreparer::UnknownTypeError; the abstract-type fetch raises it with a value-free message.
• indexer.rb: wire config.skip_record_validation_for into the factory.
• RBS signatures updated for all of the above.
• elasticgraph-local config_schema.yaml: the new property added to the shared config validation schema.

13 files changed, 393 insertions(+), 18 deletions(-).

Verification

• script/run_specs (COVERAGE=1, real Elasticsearch): 5222 examples, 0 failures. Coverage holds at the project's expected level (the one non-100% file, gem_spec.rb, is pre-existing and untouched here).
• script/type_check (Steep): no type errors.
• script/lint (Standard Ruby): 890 files, no offenses.
• script/spellcheck (codespell): clean.
• bundle exec rake schema_artifacts:check: up to date (runtime config only, no artifact changes).
• bundle exec rake site:validate: 148 runs, 0 failures.

New tests:

• config_spec.rb: integer YAML rates coerce to Float (1 to 1.0), and out-of-range rates (1.5, -0.1) are rejected at config load.
• operation/factory_spec.rb: a skipped type builds operations without record validation; non-skipped types still fail on bad records; envelope validation still runs for skipped types; fractional sampling (stubbed Zlib.crc32 for both branches); retry stability; the derived-index path under skip; and the unknown-__typename case returning a FailedEventError instead of raising.
• processor_spec.rb: a batch with skips logs one RecordValidationSkipped with the right count/counts_by_type; a batch with no skips logs none.

[CLAassistant]

All committers have signed the CLA.

[marcdaniels-toast]

@vermatron I'm digging into this PR. Meanwhile can you merge the latest origin/main into it?

[marcdaniels-toast]

This looks good to me. I'll ask @myronmarston to take a look in case there are bigger picture things I didn't think of or catch.

[myronmarston]

Nice work @vermatron! I left some suggestions.

[myronmarston]

skip_record_validation_for reads a bit odd as a config setting:

• The name doesn't hint that it's a map keyed by event type
• The name doesn't hint that the map values are percents.
• The _for suffix, as a preposition, reads oddly without something coming after it. (I do like the _for suffix on the end of method names when it reads together with the argument, such as cluster_for(record), but it doesn't read the same as a config setting).

Can we call it skip_record_validation_percents_by_type?

• The x_by_type naming implies its a map keyed by type
• skip_record_validation_percents describes what the values are

[myronmarston]

Can we make the max value 100 and treat it as a percent? IMO that's more natural than making it a value between 0 and 1. It can still be a floating point value.

[myronmarston]

It's a bit odd to start the description with the caveat. Can we instead start with a description of what the config option does, and then put the Note: caveat later?

Maybe something like:

A map, keyed by event type, which controls how often records are validated during ingestion. A value of 100 skips all record validation for the named type; a value of 0 (the default) validates all records of the named type.

With large schemas, the record validation consumes a significant chunk of CPU, and skipping validation can optimize your indexing performance. This setting allows you to make a tradeoff based on your needs.

When validation is skipped, ElasticGraph's usual guarantees (e.g. that all valid records in a batch that has invalid records still get successfully indexed) may no longer apply.

[myronmarston]

Can you move this definition to here?

https://github.com/block/elasticgraph/blob/main/elasticgraph-support/lib/elastic_graph/errors.rb

Then it aligns with how other EG exception classes are defined. Also, please subclass ElasticGraph::Errors::Error as that's intended to be a common supertype of all exceptions raised by EG.

...although I left another suggestion (see the next comment) which I think remotes the need for this type entirely.

[myronmarston]

Curious on your thinking here. There's an unbounded number of ways things can go wrong when the record hasn't been validated. Why is it worth handling the unknown type case but not worth handling any other case?

An alternate approach worth considering:

• When validation is skipped, wrap the entire BuildResult.success(...) expression in a begin/rescue, and do rescue => exception so that you rescue any kind of error.
• In the rescue block, re-run the validator (so the caller gets a failure indicating what was malformed like they would have if validation was used).
• If the validator returns no failed_result, re-raise the exception--it means something went wrong that's not detectable via record validation and we shouldn't hide the error.

The hope with this approach:

• Don't just handle one specific error you thought of (UnknownTypeError)--instead handle anything that goes wrong.
• Give the caller the same error they would have gotten w/ validation when validation has been skipped and that skipping lead to an exception. The validation error usually pinpoints the problem.

[myronmarston]

Can we extract a constant for 2**32 so we don't re-compute it each time?

[myronmarston]

Similar to my feedback on config.rb: I like _for as a suffix on a method name that takes an argument (e.g. payroll.salary_for(employee)) but I don't think it reads well when there's no argument that follows, as is the case here since it's just a property. Let's come up with a better name.

[myronmarston]

I don't love the _for suffix on the attribute here, either.