Skip to content

osrs hs api: bound requests, validate payloads, pin result contracts - #290

Merged
extreme4all merged 4 commits into
developfrom
t3/osrs-hs-api-fixes
Nov 13, 2026
Merged

extreme4all merged 4 commits into
developfrom
t3/osrs-hs-api-fixes

Conversation

@extreme4all

@extreme4all extreme4all commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

The tiger-style review of components/bot_detector/osrs_hs_api found the fetch contract had holes the automated gates can't see: requests ran on aiohttp's 300s default timeout (one stalled proxy pins a fetcher slot for minutes), a validation failure reported latency 0.0 so outcome metrics silently lost every such scrape, and a truncated hiscore payload validated as a full successful scrape — corrupting ML training data without any error.

Fixes

  • requests are bounded (ClientTimeout(total=10, connect=5) default, injectable) — prod p99 is ~6s, so draws past 10s fail anyway; no fetch hangs on aiohttp's 300s default
  • PlayerStats requires every known skill id (unknown extras tolerated, so a Jagex schema addition doesn't break scrapes) and non-empty unique activities — truncated payloads fail loudly instead of scraping as success
  • transform errors keep the latency the fetch paid, keeping fetch_outcome_latency accounting complete
  • fetch error handling catches only operational failures (ClientError, timeouts, JSON errors); programming bugs now surface, and cancellation still propagates
  • redirect detection covers 301/302/303/307/308, not just 302
  • url and timeout are constructor args; Result is Generic[T] instead of Any

Testing: simulated unit tests + live compose smoke

Unit tests now run with no sockets at all: a FakeSession mirrors the aiohttp client surface (status/history/json/raise_for_status), so the mapping logic is tested in 0.45s. What a fake cannot prove — real redirect following, real timeout timers, the live payload shape — moved to a new fetch-level smoke test:

make smoke-scraper

It runs the real brick (packaged hiscore_scraper image) against the live Jagex hiscores on the compose network: default-timeout check, a real player through the full-skill-set validator, a missing player → Err(PlayerDoesNotExist), and a blackhole-address fetch proving the timeout bound under real timers. No kafka, mysql, or proxies needed (a full pipeline smoke would require a real Webshare PROXY_API_KEY).

The smoke earned its keep immediately: its first run caught that the live API now ships 25 skills (ids 0–24), which the initially hardcoded 24-id validator rejected. The validator was corrected to subset semantics and the smoke now guards against that drift. Current run: 7 passed, 0 failed.

Gates: ruff check/format clean, mypy clean, 638 unit tests pass (+4 skipped), scraper_throughput sim passes, make smoke-scraper passes against the live API.

Created with GLM-5.3-Flash in T3 Code.

Drift logging

Schema drift is also visible in production logs: tolerated unknown skill ids produce one WARNING per distinct id-set (deduped per process, so 1800 fetchers scraping the same drifted payload emit a single line), pointing at EXPECTED_SKILL_IDS for the one-line update. Pinned by test: a drifted payload scraped twice logs exactly once.

Created with GLM-5.3-Flash in T3 Code.

  • duplicate skill ids are rejected — a set-membership check alone would silently pass a payload with two rows for one skill

@extreme4all
extreme4all merged commit d23233a into develop Nov 13, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant