Skip to content

database: atomic batch writes, bounded temp tables, dead hiscore repos removed - #288

Merged
extreme4all merged 5 commits into
developfrom
t3/tiger-component-database
Nov 13, 2026
Merged

extreme4all merged 5 commits into
developfrom
t3/tiger-component-database

Conversation

@extreme4all

@extreme4all extreme4all commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

Problem

Tiger-style review of components/bot_detector/database found hazards in the write paths that the gates can't catch:

  • Retried batches duplicated or lost rows. The prediction repos committed mid-batch, so a failure on the second insert left the first committed — and the worker's requeue then re-inserted everything (the prediction table has no unique key). Worse, the report repo used DROP TABLE instead of DROP TEMPORARY, whose implicit commit silently broke the rollback contract the worker relies on.
  • Temp tables had a built-in poison-pill. ENGINE=MEMORY with a VARCHAR(255) column allocates ~1 KB/row fixed-width; ~16k rows blows the 16 MB default max_heap_table_size → "table is full" → whole batch requeued → retry storm on the hottest write path (worker_hiscore).
  • Dead code was a landmine. Four unused single-row hiscore repos insert into STORED generated columns — guaranteed runtime error masked by INSERT IGNORE into silent data loss for the first future caller.
  • Smaller hits: get_report_score bound an IN list as a bare tuple (worked only via an asyncmy escaping accident; empty list = invalid IN () → 500), set_primary_rsn could permanently wipe all of a user's primary-RSN flags on a miss, get_user could 500 auth on a duplicate username (column has no unique constraint), and POOL_RECYCLE=25s recycled every connection every 25 seconds across every service.

Solution

  • Prediction inserts no longer commit inside the repos; batch transactions belong to the caller. This must land together with the matching worker_prediction session.begin() wrap (owned by a separate branch) — the commit message says so too.
  • Temp tables moved to ENGINE=InnoDB (player, report, hiscore); report's drops now use DROP TEMPORARY.
  • Deleted the 4 dead single-row hiscore repos + 3 unused interfaces (net −196 lines).
  • get_report_score uses bindparam(expanding=True); set_primary_rsn bails before the clear when the target row is missing and the clear now excludes the target row (the smoke tier caught that the reordered sequence was resetting the just-set primary); get_user takes LIMIT 1; POOL_RECYCLE 25s → 3600s; select_player guards player_id=0 truthiness.

Smoke tier

_infra/smoke/db_smoke.py — standalone script (not part of pytest, so normal test runs carry zero smoke noise and no DB-writing footgun), 7 checks that prove the raw-SQL surfaces against real MySQL: prediction batch atomicity under FK failure, 20k-row player update through the temp-table path (proves the MEMORY → InnoDB change past the 16 MB ceiling), hiscore upserts across all 4 partitions, sighting dedupe, expanding IN list including empty tuple, primary-RSN miss safety, engine connectivity. Idempotent (purges its own rows before and after).

Run against the compose mysql service (port 3307):

docker compose up -d --wait mysql
uv run python _infra/smoke/db_smoke.py

Exits non-zero on any failure. This tier caught the set_primary_rsn regression described above.

Verification

  • ruff check + format clean; database component tests + worker_prediction tests: 43 passed
  • Full workspace suite ran green on this content before the rebase (655 passed)
  • Tuple-bind vs expanding behavior verified empirically against SQLAlchemy 2.0.44 + asyncmy; generated-column/DDL semantics checked against the actual 01_tables.sql

Created with GLM-5.3-Flash in T3 Code.

extreme4all added 5 commits November 13, 2026 22:46
- prediction: remove repo-level commits; caller owns the tx. MUST land
  together with the worker_prediction begin() wrap (claimed by another
  branch) or retried batches roll back silently
- report/player/hiscore: temp tables on InnoDB (MEMORY had a 16MB
  ceiling -> batch-full -> retry storm); report DROP TABLE -> DROP
  TEMPORARY (non-temp DDL implicitly commits the caller's tx)
- hiscore: delete 4 dead single-row repos + 3 unused interfaces; they
  insert into STORED generated columns and are masked by INSERT IGNORE
- api_public: get_report_score uses expanding bindparam (bare tuple bind
  only worked via asyncmy accident; empty tuple = invalid IN ())
- discord: set_primary_rsn bails before the clear when the target row is
  missing (was committing a wipe of all primary flags)
- api: get_user limit(1); username has no unique constraint, a dup
  raised MultipleResultsFound -> auth 500
- core: POOL_RECYCLE 25s -> 3600s (constant reconnect churn)
- player: select_player player_id=0 truthiness guard
- smoke: 7 tests against the compose mysql (skip unless SMOKE_DATABASE_URL
  is set): prediction batch atomicity, 20k-row player temp table (beyond
  the MEMORY ceiling), hiscore upserts across all 4 partitions, sighting
  dedupe, expanding IN list incl. empty, primary-rsn miss safety, engine
  connectivity. Idempotent across runs (autouse purge, disjoint id ranges)
- smoke caught a real bug in the previous commit: after reordering
  set-first, the clear still reset the just-set row - the clear now
  excludes Player_id = :player_id
- the pytest file skipped in normal runs but became a footgun if
  SMOKE_DATABASE_URL was ever exported: every plain pytest run would
  write to and purge rows in that database
- now a standalone script, same 7 checks, same idempotent purge:
  uv run python _infra/smoke/db_smoke.py [--url DSN]
- defaults to the compose mysql DSN; exits non-zero on any failure
the database branch removed the prediction repos' self-commits; without
this caller-owned begin() the runner's session close would roll back
every batch. also fixes both test mocks to model begin() as an async CM.
landed together with the database changes so PR #288 is self-contained.
@extreme4all
extreme4all merged commit 18a4378 into develop Nov 13, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant