[remigiusz-antczak]

Summary

Perun's deterministic security layer added nothing on top of raw semgrep. security_verify triages findings rather than detecting them, and review_checks ran semgrep only when given a local config. This PR bundles taint rules, runs them by default and measures them on a blind held-out set.

On the held-out set, the bundled rules plus semgrep's p/default find 19 of 20 planted vulnerabilities, against 15 of 20 for p/default alone. The cost is more false positives: 21 against 14.

• templates/semgrep-taint/{py,js}.yml: request-sourced taint rules for SQL injection, command injection, path traversal, SSRF, XSS and template injection. Guards and sanitizers are bound to the sink argument and must exit (raise, return or throw); host allowlists apply to SSRF only.
• review_checks.sh: semgrep uses the bundled rules by default. REVIEW_SEMGREP_CONFIG still overrides them, and a missing bundle now prints its own message.
• security_verify.py: reads native semgrep JSON. OSS semgrep emits no dataflow trace (only the Pro engines do), so these rows stay unverified with a note to trace the source by hand.
• docs/bench/sast-sca-1027: the in-sample SAST and SCA bench. The rules were tuned on it.
• docs/bench/sast-sca-heldout1-1027: a blind fixture with 20 planted vulnerabilities and 20 decoys. It was preregistered as prereg/sast-heldout1 before any scoring and is checked against SHA256 sums.

Results (TP / FP / FN)

Arm	In-sample (tuned)	Held-out, pre-review rules	Held-out, post-review rules
semgrep p/default	8 / 3 / 5	15 / 14 / 5	15 / 14 / 5 (same run)
bundled rules only	13 / 0 / 0	13 / 6 / 7	14 / 8 / 6
bundled + p/default	13 / 3 / 0	19 / 19 / 1	19 / 21 / 1

Caveats:

• n = 20, so a one-item change is not significant.
• A finding counts as a TP only on an exact sink-line match. Rows that map to no planted item count as FP.
• p/default is unpinned; the post-review arms reuse the first run's p/default output.
• The bundled rules missed all 3 unsafe-deserialization items.
• The rule author never saw the held-out set. The post-review rules came from code review, not from held-out results. Because the set had already been scored once, the second score is less blind than the first.

Verification

Run on head a51ce46:

• scripts/ci-tests.d/*.sh: 169/169 pass. bench_check.py --changed origin/main is clean for both bench dirs. The in-sample bench's ledger.jsonl states in its own text that the freeze is self-attested and not blind. This includes test_semgrep_taint.py, with negative cases for self-method, subscript source, unrelated guard, Map.get, document.body, non-exiting guard and braced guard.
• test-review-checks-semgrep-default.sh is fail-closed: set -euo pipefail, a guarded mktemp, and no git commit outside its temp dir.
• The secret-pattern grep over changed files is clean. land-release.sh --regen makes no diff, and git diff --check is clean.
• Independent review: the bench had 1 red and 7 yellow, all folded. The rules had 1 red and 5 yellow, all folded. One gap remains open: there is no test for the missing-bundle message.

🤖 Generated with Claude Code

https://claude.ai/code/session_01Hi1ChtXVVrE8YEA2Ano93i