Skip to content

event_queue: stop poison-record stalls, await broker acks, cap retries - #295

Merged
extreme4all merged 3 commits into
developfrom
t3/tiger-component-event_queue
Nov 14, 2026
Merged

extreme4all merged 3 commits into
developfrom
t3/tiger-component-event_queue

Conversation

@extreme4all

Copy link
Copy Markdown
Contributor

Tiger-style adversarial review of components/bot_detector/event_queue found one critical and two high-severity defects in the queue layer every worker base depends on. This fixes them plus the medium/low findings, in the smallest change per bug.

Problem

  1. Poison records stalled partitions forever (critical). One malformed payload made get_many return a ValidationError for the whole batch. With commit-after-dequeue semantics the offset never advanced, so the same record was re-fetched in a 1s retry loop — a silent, permanent stall with no DLQ and no metric.
  2. Silent data loss on produce (high). put() awaited send(), which only confirms buffering in aiokafka. The delivery future was dropped, so broker-ack failures (broker loss, record-too-large, auth) vanished while MESSAGES_PRODUCED counted them.
  3. Retry loop could run forever (high). The cap compared with == after incrementing, so MAX_PRODUCE_RETRIES=0 never matched; non-timeout producer errors also raised out of put(), breaking the errors-as-values contract.

Also fixed: start() not retry-safe after a failure, lag probe reporting lag=0 when the broker was unreachable (false-healthy monitoring) with unbounded awaits and an assert stripped under -O, memory commit() raising ValueError on unpaired commits, silent validation drops on the memory backend diverging from the kafka contract, InMemoryConfig(maxsize=0) silently creating an unbounded queue, broad except Exception inside the component, and the factory mixing raise/return error styles.

Solution

Lazy per bug — no DLQ infrastructure, no new deps:

  • get_many skips invalid records loudly (warn log + event_queue_messages_skipped_total); with commit-after-dequeue, skipping is how the consumer advances.
  • send_and_wait for real broker acks; retry cap uses >= before the sleep; KafkaError returns as a value; Field(ge=1) blocks the infinite-loop config at the border.
  • Adapters assign clients only after a successful start, so a failed start retries.
  • Lag probe raises LagProbeError on unavailable metadata (both callers verified to tolerate raises), bounds all broker calls with a 30s timeout, cleans up on partial-start failure.
  • Memory backend returns validation errors for parity with kafka; commit() drops the pointless task_done(); maxsize validates ge=1; factory returns InvalidConfig everywhere.

Package README added per docs standards.

Tests

  • Unit: 23 new/updated — poison skip keeps valid records, retry progression, zero-retries rejected at construction, failed-start recovery, first-ever KafkaLagProbe coverage (offset math, truncation, metadata failure), memory commit-without-get, error-value parity.
  • DST: new kafka_poison_advance scenario drives the real adapter get_many/commit loop over poison-laced pages and pins the system property the fix exists for: the consumer advances (calls bounded at pages + slack), loses no valid record, and the skip counter matches the poison total. Registered in the naming gate; replays byte-identically.

Full gates: ruff, mypy, pytest 651 passed / sim 249 passed.

Created with glm-5.3-flash in T3 Code.

extreme4all added 3 commits November 14, 2026 18:13
- kafka get_many skips invalid/poison records (warn + skipped counter)
  instead of aborting the batch: returning the error discarded valid
  records and, with no commit, re-fetched the same record forever
- put() uses send_and_wait: send() resolves on buffering, not broker
  ack; the dropped delivery future silently lost messages
- retry cap uses >= before sleep (retries=0 no longer loops forever),
  catch KafkaError so producer errors return as values, not raises
- producer/consumer start() assign after success, so a failed start
  retries instead of no-oping forever
- lag probe raises on unavailable metadata instead of reporting lag=0,
  bounds all broker calls with a 30s timeout, cleans up admin if
  consumer start fails
- memory commit() drops task_done() (raised ValueError on unpaired
  commits); get_one returns validation errors for parity with kafka;
  InMemoryConfig validates maxsize >= 1 (Queue(0) is unbounded)
- factory returns InvalidConfig instead of raising: one error contract
- README per docs standards; tests for every changed path
kafka_poison_advance drives the real adapter get_many/commit path in
the bases' commit-after-dequeue loop over poison-laced pages (unparseable
bytes every 7th record, schema drift every 11th). Invariants: the loop
advances (get_many calls bounded at pages + slack), no valid record is
lost, no error results, and the skip counter matches the poison total.
Registered in the scenario naming gate. Runs replay byte-identically.
Adds runemetrics_scraper, worker_prediction, job_backfill_banned,
job_hs_migration_v3 to docker-compose-dev.yml with explicit
bootstrap_servers (the .env.example name gap) and root DATABASE_URL
(no per-service grants exist for these).

Running them in the stack exposed a real dependency bug:
job_backfill_banned and job_hs_migration_v3 import prometheus_client
via their metrics module but never declared it - both images crashed
at boot with ModuleNotFoundError. Pinned prometheus-client==0.22.0
matching worker_report/worker_prediction and relocked.

Verified in the stack: job_backfill_banned published 9 ban events,
worker_ban_migration consumed all 9 with lag 0 and committed offsets.
runemetrics_scraper fail-fasts on empty PROXY_API_KEY by design;
worker_prediction boots and registers prediction_worker on
predictions.to_insert.
@extreme4all
extreme4all merged commit 0ec90ca into develop Nov 14, 2026
2 checks passed
@extreme4all
extreme4all deleted the t3/tiger-component-event_queue branch November 14, 2026 18:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant