Skip to content

job_broadcaster: dst storm scenario pins webhook retry time budget - #291

Merged
extreme4all merged 5 commits into
developfrom
t3/webhook-storm-dst
Nov 14, 2026
Merged

extreme4all merged 5 commits into
developfrom
t3/webhook-storm-dst

Conversation

@extreme4all

@extreme4all extreme4all commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Unit tests scripted the webhook's 429 responses but never pinned the time the retry policy costs: RATE_LIMIT_RETRIES sleeps, chunk pacing, and a hostile retry_after compose into wall-clock time a cron-window job can't afford, and no test could catch a regression there without sleeping in CI. The tiger review of the brick also found a set of correctness and contract gaps: malformed 429 bodies crashing out of post(), an unclamped retry_after (Discord's own shared-limit example is 1336.57s — 22+ minutes), dead embed-limit constants, a docstring contradicting the code on which retry signal wins, naive embed timestamps silently shifted to local time, payloads replayed to redirect targets, and stdlib-json serialization where the repo mandates orjson. A docs pass on Discord's embed limits then revealed the 6000-char sum binds per message across all embeds — our check was per single embed.

This PR closes the full findings list for components/bot_detector/discord_webhook:

Transport (core.py):

  • retry_after clamped before sleeping: min(max(v, 0.0), MAX_RETRY_AFTER_SECONDS = 60) (constant, cron-window bound). The log line shows the value actually slept.
  • _rate_limit_seconds prefers the body retry_after — per the docs it's fractional (64.57) while the Retry-After header is its integer ceiling (65), so body-first always waits the exact time; header fills in only when the body omits or garbles the value, sane default last. The docstring now matches the code. Malformed bodies (non-object JSON, retry_after: null, HTML from a proxy) fall back instead of raising — post() keeps its errors-as-values contract.
  • Message-level guards, all refusing locally as error values with no transport round-trip: >10 embeds (docs limit), combined embed chars >6000 across the message (the documented per-message bound), and empty payloads.
  • Payload serialized with orjson (data=orjson.dumps(...) + Content-Type header) per repo guidelines.
  • allow_redirects=False — a redirect must never get the embed payload replayed to another host.

Structs (structs.py):

  • Naive Embed timestamps rejected at construction — astimezone() would have silently treated them as system-local time.
  • color bounded to its 24-bit RGB domain (0–0xFFFFFF) and thumbnail_url required to be http(s) — the docs' image-url contract; anything else was a guaranteed 400 at send time.
  • Embed/EmbedField are frozen and fields is a tuple: validators bind only at construction, so post-construction mutation (.fields.append(...), embed.title = ...) was a silent bypass of every size limit; now it's a loud error.

Consumer note (bases/bot_detector/job_broadcaster/embeds.py): 3 mechanical bracket changes — the builders now pass fields=() tuples to satisfy the component's type. Zero behavior change.

Sim scenario (test/sim/test_broadcaster_rate_limit_storm.py): the deadline-tripwire test is flipped as documented: retry_after=86_400 now asserts broadcast completes — bans applied, all 12 post attempts, inside the same [480s, 600s] virtual budget as the honest 60s storm. One hostile 429 costs exactly what an honest one costs.

Tests: 16 new unit tests (cap, negative clamp, body-over-header precedence, header-on-null-body, HTML-body fallback, request timeout, embed-count/empty-payload/message-total guards incl. the exact-6000 boundary, naive timestamp, color range + boundaries, thumbnail scheme, frozen immutability) plus a rename of the misleading "malformed 429" test to what it actually tests.

Gates: full suite 634 passed / 4 skipped, ruff check, ruff format, mypy all clean.

Still open (design-level objections, deliberately not defects): no transport-error retry for idempotent webhook POSTs, errors-as-values carries no typed variant (callers string-match), and the component-wraps-HTTP tension in AGENTS.md's component definition.

Created with GLM-5.3-Flash in T3 Code.

@extreme4all
extreme4all merged commit a66a4b0 into develop Nov 14, 2026
9 checks passed
@extreme4all
extreme4all deleted the t3/webhook-storm-dst branch November 14, 2026 16:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant