Repository navigation
osrs hs api: bound requests, validate payloads, pin result contracts - #290
Merged
Merged
Conversation
extreme4all
force-pushed
the
t3/osrs-hs-api-fixes
branch
from
November 13, 2026 22:08
1842cc9 to
1eb7793
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The tiger-style review of
components/bot_detector/osrs_hs_apifound the fetch contract had holes the automated gates can't see: requests ran on aiohttp's 300s default timeout (one stalled proxy pins a fetcher slot for minutes), a validation failure reported latency 0.0 so outcome metrics silently lost every such scrape, and a truncated hiscore payload validated as a full successful scrape — corrupting ML training data without any error.Fixes
ClientTimeout(total=10, connect=5)default, injectable) — prod p99 is ~6s, so draws past 10s fail anyway; no fetch hangs on aiohttp's 300s defaultPlayerStatsrequires every known skill id (unknown extras tolerated, so a Jagex schema addition doesn't break scrapes) and non-empty unique activities — truncated payloads fail loudly instead of scraping as successfetch_outcome_latencyaccounting completeClientError, timeouts, JSON errors); programming bugs now surface, and cancellation still propagatesurlandtimeoutare constructor args;ResultisGeneric[T]instead ofAnyTesting: simulated unit tests + live compose smoke
Unit tests now run with no sockets at all: a
FakeSessionmirrors the aiohttp client surface (status/history/json/raise_for_status), so the mapping logic is tested in 0.45s. What a fake cannot prove — real redirect following, real timeout timers, the live payload shape — moved to a new fetch-level smoke test:It runs the real brick (packaged
hiscore_scraperimage) against the live Jagex hiscores on the compose network: default-timeout check, a real player through the full-skill-set validator, a missing player →Err(PlayerDoesNotExist), and a blackhole-address fetch proving the timeout bound under real timers. No kafka, mysql, or proxies needed (a full pipeline smoke would require a real WebsharePROXY_API_KEY).The smoke earned its keep immediately: its first run caught that the live API now ships 25 skills (ids 0–24), which the initially hardcoded 24-id validator rejected. The validator was corrected to subset semantics and the smoke now guards against that drift. Current run: 7 passed, 0 failed.
Gates: ruff check/format clean, mypy clean, 638 unit tests pass (+4 skipped), scraper_throughput sim passes,
make smoke-scraperpasses against the live API.Created with GLM-5.3-Flash in T3 Code.
Drift logging
Schema drift is also visible in production logs: tolerated unknown skill ids produce one WARNING per distinct id-set (deduped per process, so 1800 fetchers scraping the same drifted payload emit a single line), pointing at
EXPECTED_SKILL_IDSfor the one-line update. Pinned by test: a drifted payload scraped twice logs exactly once.Created with GLM-5.3-Flash in T3 Code.