Repository navigation
event_queue: stop poison-record stalls, await broker acks, cap retries - #295
Merged
Merged
Conversation
added 3 commits
November 14, 2026 18:13
- kafka get_many skips invalid/poison records (warn + skipped counter) instead of aborting the batch: returning the error discarded valid records and, with no commit, re-fetched the same record forever - put() uses send_and_wait: send() resolves on buffering, not broker ack; the dropped delivery future silently lost messages - retry cap uses >= before sleep (retries=0 no longer loops forever), catch KafkaError so producer errors return as values, not raises - producer/consumer start() assign after success, so a failed start retries instead of no-oping forever - lag probe raises on unavailable metadata instead of reporting lag=0, bounds all broker calls with a 30s timeout, cleans up admin if consumer start fails - memory commit() drops task_done() (raised ValueError on unpaired commits); get_one returns validation errors for parity with kafka; InMemoryConfig validates maxsize >= 1 (Queue(0) is unbounded) - factory returns InvalidConfig instead of raising: one error contract - README per docs standards; tests for every changed path
kafka_poison_advance drives the real adapter get_many/commit path in the bases' commit-after-dequeue loop over poison-laced pages (unparseable bytes every 7th record, schema drift every 11th). Invariants: the loop advances (get_many calls bounded at pages + slack), no valid record is lost, no error results, and the skip counter matches the poison total. Registered in the scenario naming gate. Runs replay byte-identically.
Adds runemetrics_scraper, worker_prediction, job_backfill_banned, job_hs_migration_v3 to docker-compose-dev.yml with explicit bootstrap_servers (the .env.example name gap) and root DATABASE_URL (no per-service grants exist for these). Running them in the stack exposed a real dependency bug: job_backfill_banned and job_hs_migration_v3 import prometheus_client via their metrics module but never declared it - both images crashed at boot with ModuleNotFoundError. Pinned prometheus-client==0.22.0 matching worker_report/worker_prediction and relocked. Verified in the stack: job_backfill_banned published 9 ban events, worker_ban_migration consumed all 9 with lag 0 and committed offsets. runemetrics_scraper fail-fasts on empty PROXY_API_KEY by design; worker_prediction boots and registers prediction_worker on predictions.to_insert.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tiger-style adversarial review of
components/bot_detector/event_queuefound one critical and two high-severity defects in the queue layer every worker base depends on. This fixes them plus the medium/low findings, in the smallest change per bug.Problem
get_manyreturn aValidationErrorfor the whole batch. With commit-after-dequeue semantics the offset never advanced, so the same record was re-fetched in a 1s retry loop — a silent, permanent stall with no DLQ and no metric.put()awaitedsend(), which only confirms buffering in aiokafka. The delivery future was dropped, so broker-ack failures (broker loss, record-too-large, auth) vanished whileMESSAGES_PRODUCEDcounted them.==after incrementing, soMAX_PRODUCE_RETRIES=0never matched; non-timeout producer errors also raised out ofput(), breaking the errors-as-values contract.Also fixed:
start()not retry-safe after a failure, lag probe reportinglag=0when the broker was unreachable (false-healthy monitoring) with unbounded awaits and an assert stripped under-O, memorycommit()raisingValueErroron unpaired commits, silent validation drops on the memory backend diverging from the kafka contract,InMemoryConfig(maxsize=0)silently creating an unbounded queue, broadexcept Exceptioninside the component, and the factory mixing raise/return error styles.Solution
Lazy per bug — no DLQ infrastructure, no new deps:
get_manyskips invalid records loudly (warn log +event_queue_messages_skipped_total); with commit-after-dequeue, skipping is how the consumer advances.send_and_waitfor real broker acks; retry cap uses>=before the sleep;KafkaErrorreturns as a value;Field(ge=1)blocks the infinite-loop config at the border.LagProbeErroron unavailable metadata (both callers verified to tolerate raises), bounds all broker calls with a 30s timeout, cleans up on partial-start failure.commit()drops the pointlesstask_done();maxsizevalidatesge=1; factory returnsInvalidConfigeverywhere.Package README added per docs standards.
Tests
KafkaLagProbecoverage (offset math, truncation, metadata failure), memory commit-without-get, error-value parity.kafka_poison_advancescenario drives the real adapterget_many/commitloop over poison-laced pages and pins the system property the fix exists for: the consumer advances (calls bounded at pages + slack), loses no valid record, and the skip counter matches the poison total. Registered in the naming gate; replays byte-identically.Full gates: ruff, mypy, pytest 651 passed / sim 249 passed.
Created with glm-5.3-flash in T3 Code.