fix(cache): record audit scope and reject mismatched cache reads (#542)

[Marc-cn]

Fixes #542 (partially, two bullets deferred, see below)

Problem

Remediation read the audit cache without checking that the cached audit covered what it was about to remediate. read_audit_cache validated TTL, HEAD commit and dirty state, but never compared the framework or level the envelope records and the tag filter wasn't recorded at all.

So a level-filtered run, a --tags domain=VM run or an audit of a different framework in the same checkout left a cache that a later Baseline remediation treated as a full Baseline audit, and remediated only the subset it found.

What changed

• The envelope records the tag filter alongside framework and level.
• read_audit_cache takes expected_framework, expected_level and expected_tags. A mismatch is a miss, checked alongside the existing staleness rules. All three default to None, so existing callers are unaffected.
• The CLI filters controls itself and passes the narrowed list down (cli.py:238, apply_user_config=False), so run_sieve_audit never saw the filter and a filtered run wrote tags: "". It now passes cache_tags explicitly for the envelope.
• remediate_audit_findings asks for an unfiltered openssf-baseline cache and re-audits on a mismatch.

expected_level is deliberately not passed from remediation: that function has no level of its own, so there'd be nothing truthful to compare against.

Verification

Before — a tag-filtered audit writes a cache indistinguishable from a full one:

darnit audit --framework openssf-baseline -t domain=VM <repo>   # 11 controls
envelope: {'framework': 'openssf-baseline', 'level': 3, 'tags': ''}

After:

envelope: {'framework': 'openssf-baseline', 'level': 3, 'tags': 'domain=VM'}
read with expected_tags=""        -> miss
full audit, read expected_tags="" -> hit
full audit, expected_framework="amber" -> miss

4 new tests in tests/darnit/core/test_audit_cache.py::TestScopeMismatch. tests/darnit (minus parity) 2047 passed; tests/darnit_baseline 900 passed.

Type of Change

• Bug fix (non-breaking change fixing an issue)

Testing

• Tests pass locally
• Added tests for new functionality
• Linting passes

AI assistance

• No AI assistance was used
• AI assistance was used

Not included

Two bullets from the issue, both need their decision:

1. Store routing. The orchestrator still reads via the default store, ignoring [stores.cache]. Fixing it means pulling _load_merged_stores + resolve_stores into remediation/orchestrator.py, mirroring tools/audit.py:567. This is the follow-up flagged in feat(stores): route audit-cache through AuditCacheStore protocol (feature 035) #413 and seemed better separate.
2. Whether a filtered run should avoid overwriting the full-audit cache. Recording the filter already stops a full-audit consumer from reusing a filtered cache, which is the harmful case. Writing filtered runs under a different key would also preserve the full-audit entry, but that changes the on-disk layout feature 035 just settled.

[mlieberman85]

Nice start. A few gaps before this fully closes #542:

1. Remediation reads without expected_level, but on a miss it re-audits at level 3, so a level-1 cache is still a hit. Please pass expected_level=3.
2. --include/--exclude and profile= also produce filtered audits that currently look unfiltered. Please record the full scope (a filtered flag or the control-ID set).
3. A filtered audit still replaces an unfiltered cache, and the remediation read ignores [stores.cache]. Fix both here, or change "Fixes #542" to "Refs".
4. Bump CACHE_VERSION, or treat a missing tags key as a miss, so old cache files aren't read as unfiltered.
5. Spec-first: describe the envelope change in framework-design.md 10.4.
6. Add the model version to Assisted-by and name the tool in the PR body.

Drafted with Claude Code; reviewed and posted by me.

[Marc-cn]

Addressed.

1. expected_level=3 passed. _run_baseline_checks defaults to level=3, so accepting a level-1 cache would have remediated against a narrower audit than the one it would otherwise run
2. Added a filtered boolean to the envelope rather than the control-ID set, --include/--exclude can't be expressed as a tag string, and a set would make the envelope large and the comparison brittle. run_sieve_audit has no profile parameter (that filtering happens in the MCP tool), so profile isn't covered here
3. Both done. A filtered audit now writes under its own cache key, so it can't replace the unfiltered entry but repeat filtered runs still hit. Remediation resolves [stores.cache] via _load_merged_stores + resolve_stores and reproduces the same key derivation as tools/audit.py. Kept "Fixes Remediation trusts the audit cache regardless of framework, level, or tag filter #542".
4. CACHE_VERSION 1 → 2, and the check is now != CACHE_VERSION rather than > — the old upper-bound form would still have read a v1 envelope, which has no tags/filtered keys and would present a possibly filtered audit as unfiltered.
5. framework-design.md 10.4 gains two requirements: the envelope's scope fields plus the distinct key for filtered runs, and the scoped-read semantics including the version rule. The remediation requirement now also states the store it reads through.
6. Trailer is `Assisted-by: Claude:claude-opus-4-5.

Tests: 38 in test_audit_cache.py (7 new across two classes), tests/darnit_baseline 900 passed, tests/darnit minus parity 2053 passed.

Three failures in the long tests/darnit run — test_mcp_equals_direct_equals_cli, TestProgressLines::test_progress_lines_use_n_over_m_counter, TestSecretRedaction::test_leaked_exception_message_does_not_reach_report, all pass individually and fail only under a full-suite run. I checked test_mcp_equals_direct_equals_cli against the unmodified lockfile and it fails identically there, so they're pre-existing. Probably worth their own issue.

I also left ruff format alone on cli.py and orchestrator.py: running it there reformats ~180 lines I didn't touch (171 files in the repo currently fail format --check), which seemed like a separate PR.

[mlieberman85]

Thanks, this covers everything I asked. Two things before it can merge:

1. Rebase onto main. It conflicts in orchestrator.py, cli.py and audit.py, so CI hasn't run. #552 changed _load_merged_stores to (framework_name, operator). After the rebase, _load_merged_stores(local_path, "openssf-baseline") would raise inside the try, and every remediation would quietly re-run the audit. Please resolve the operator configuration the way _run_baseline_checks does. Ideally, also move the cache-key derivation into one helper that both tools/audit.py and the orchestrator call, so the two can't drift apart again.
2. Profile-filtered audits. audit_openssf_baseline(profile=...) narrows controls itself and passes tags=None, so its cache entry still says filtered=False, and remediation reads it as a full audit. Please fail closed: when a caller passes controls without saying cache_filtered, treat the audit as filtered. Callers that pass the full set (the MCP tool without a profile, the CLI without filters) then pass cache_filtered=False explicitly. Please add a test for the profile case.

On the three tests that fail in a full-suite run: main now passes the full suite (5210 passed after #548 made the gh calls offline). If they still fail after the rebase, please open an issue with the output.

Drafted with Claude Code; reviewed and posted by me.

[Marc-cn]

Rebuilt on current main and force-pushed (one commit). Main had rewritten the three conflicting files, so I re-applied the change instead of resolving the conflicts in place.

1. Store loading and the cache key. Remediation now reads through read_full_audit_cache() in tools/audit.py. It resolves the operator configuration the way run_sieve_audit does, calls _load_merged_stores(framework_name, operator) with the new signature, and closes the bundle. The key comes from one helper, audit_cache_key(), used by the writer and the reader.
2. Profile-filtered audits fail closed. run_sieve_audit records an audit as filtered when it has tags, when the caller says so, or when the caller passed controls without saying. The Baseline tool passes cache_filtered=bool(profile), the CLI computes it from --tags/--include/--exclude, and the generic MCP audit tool states cache_filtered=False. Profile test: tests/darnit_baseline/test_audit_cache_scope.py.

Checked through the CLI on a committed one-file repository:

Sequence	Remediation read
main: audit -t level=1	HIT, 26 results, as if it were a full audit
this branch: audit -t level=1	MISS
this branch: audit --include OSPS-QA-05.01	MISS
this branch: full audit	HIT, 66 results
this branch: full audit, then -t level=1	HIT, still 66 results
this branch, reading the cache main wrote	MISS (old envelope version)

One existing test changed: test_run_sieve_audit_without_cache_write passes its own controls, so it now states cache_filtered=False.

Full suite: 5167 passed, 19 skipped. The three tests that failed for me before did not fail.

Note for #576: it wires --profile into darnit audit. Whichever of the two merges second should add args.profile to the CLI's cache_filtered.

Drafted with Claude; reviewed and posted by me.