Skip to content

logfmt: fall back to raw text when orjson rejects dict keys - #292

Merged
extreme4all merged 3 commits into
developfrom
t3/fix-logfmt-dict-key-error
Nov 14, 2026
Merged

extreme4all merged 3 commits into
developfrom
t3/fix-logfmt-dict-key-error

Conversation

@extreme4all

Copy link
Copy Markdown
Contributor

Summary

  • Fix a --- Logging error --- crash in the scraper: dump_error() built header dicts from aiohttp's CIMultiDict, whose istr keys are str subclasses that orjson rejects (TypeError: Dict key must be str), killing the whole log line.
  • errors.py: coerce response-header keys with str(key) at the boundary so dumped errors are always JSON-safe.
  • logfmt/core.py: JsonFormatter.format() now catches TypeError from orjson and emits record.getMessage() as raw text, so a JSON-unsafe payload at a log site still lands instead of being lost.
  • Known trade-off: the fallback line drops the JSON envelope (no ts/lvl fields); log consumers should tolerate a rare non-JSON line.

Testing

  • uv run pytest test/components/bot_detector/logfmt/test_core.py test/bases/bot_detector/hiscore_scraper/test_errors.py -q — 15 passed
  • New regression test: dump_error() header keys are exactly str and orjson.dumps(dumped) succeeds on a simulated 502 ClientResponseError
  • New regression test: a dict log message with nested non-str keys formats without raising and its content appears in the output
  • uv run ruff check and uv run ruff format --check on the four changed files — clean
  • uv run mypy on the two changed source files — clean
  • Full workspace test suite: Not run
  • Live verification against a real 502 from the hiscore endpoint: Not run (unit tests simulate the aiohttp error shape)

extreme4all added 3 commits November 14, 2026 01:44
- hiscore_scraper: str() aiohttp istr header keys; orjson rejects str subclasses
- JsonFormatter: catch TypeError from orjson, emit record.getMessage()
- add regression tests for both paths
- prefix fallback output with logfmt_json_fallback instead of silent raw text
- append exception traceback to fallback lines
- scraper: sample info wide events at 1/1000 (~600 req/s)
- scraper: assert dumped header keys are str for orjson
@extreme4all
extreme4all merged commit 864c6e9 into develop Nov 14, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant