Skip to content

Add dry_run option to the accessibility scanner#232

Merged
taarikashenafi merged 9 commits into
mainfrom
dry-run-option
Jun 19, 2026
Merged

Add dry_run option to the accessibility scanner#232
taarikashenafi merged 9 commits into
mainfrom
dry-run-option

Conversation

@taarikashenafi

Copy link
Copy Markdown
Contributor

Tracking issue with context: https://github.com/github/accessibility/issues/10757

Based on feedback from @joseinthearena: with many open scanner issues there was no way to preview what a scan would do without actually filing, closing, or reopening issues — and no way to do so without mutating the gh-cache branch. This adds a dry_run input that runs a normal scan and logs the issues that would be filed, while making no changes: no issues are opened, closed, or reopened, no Copilot assignment, and the cache is not written. Because dry runs don't update the cache, the next real run behaves exactly as if the dry run never happened.

Changes

  • action.yml (root) — New dry_run input (default false); forwarded to the file step; skips the Fix, Save screenshots, Copy results to cache path, and Save cached results steps when dry_run is true
  • .github/actions/file/action.yml — New dry_run input declared
  • file/src/index.ts — When dry_run is set, each filing is classified via the existing isNewFiling / isRepeatedFiling / isResolvedFiling guards and the intended action is logged (early continue, so no Octokit calls are made); grouped/tracking-issue creation is skipped; a summary count is logged; filings_file is still written so the output contract is unchanged
  • README.mddry_run input documented in the inputs table and getting-started example
  • FAQ.md — New entry explaining how to preview a scan and noting that dry runs don't write the cache

Test Updates

  • Added file/tests/dryRun.test.ts — asserts no issues are opened/closed/reopened in dry run, the correct [dry run] Would … lines and summary are logged, and filings_file is still written; includes a regression guard that mutations still occur when dry_run is false

Usage

- uses: github/accessibility-scanner@v3
  with:
    urls: |
      https://example.com
    repository: owner/repo
    token: ${{ secrets.GH_TOKEN }}
    cache_key: cached_results.json
    dry_run: true # Scan and log what would be filed without creating/closing issues or writing the cache

When dry_run is not provided, the scanner behaves exactly as before.

Notes

  • Design decision: cache restore is kept (read-only) so dry-run logs can accurately distinguish "would open (new)" from "would reopen (existing)"; only the cache save is skipped.
  • The file-action logic is covered by the new tests. The action.yml step gating was verified by inspection but not run end-to-end — flagging for confirmation in CI.
  • Per the tracking issue's second acceptance criterion, this will need a new minor release (and the v3 tag re-pointed) cut by a maintainer.

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a dry_run mode to the accessibility-scanner GitHub Action so users can preview what a scan would do (open/reopen/close issues) via logs, without mutating issues, running Copilot fix/assignment, or writing updates to the gh-cache branch.

Changes:

  • Introduces a new dry_run input in the root action.yml and forwards it into the file sub-action; skips cache-write and other mutating steps when enabled.
  • Implements dry-run behavior in .github/actions/file/src/index.ts to classify filings and log the actions that would be taken, while still writing filings_file.
  • Adds dedicated dry-run test coverage and updates documentation (README + FAQ) to explain usage and behavior.
Show a summary per file
File Description
README.md Documents the new dry_run input in the example and inputs table.
FAQ.md Adds an FAQ entry describing how to preview scanner behavior using dry_run.
action.yml Adds dry_run input, forwards it to the file step, and gates mutating/cache-write steps when enabled.
.github/actions/file/action.yml Declares the dry_run input for the file sub-action.
.github/actions/file/src/index.ts Implements dry-run logging/classification and skips Octokit issue mutations and grouped/tracking issue creation.
.github/actions/file/tests/dryRun.test.ts Adds tests asserting no issue mutations happen in dry run and that expected logs/summary/output are produced.

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 6/6 changed files
  • Comments generated: 1

Comment thread .github/actions/file/src/index.ts Outdated
@taarikashenafi taarikashenafi requested a review from JoyceZhu June 18, 2026 16:37
GitHub Advanced Security started work on behalf of taarikashenafi June 18, 2026 16:42 View session
GitHub Advanced Security finished work on behalf of taarikashenafi June 18, 2026 16:43
Comment thread FAQ.md Outdated
Comment thread .github/actions/file/src/index.ts Outdated
dryRunCounts.reopen++
core.info(`[dry run] Would REOPEN issue: ${filing.issue.url}`)
}
continue

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To make the logic easier to follow, I'd suggest deleting the continue, then wrapping the not-dry-run code in an else block; this way, a reader will immediately see this is code which definitely doesn't execute for a dry run

@taarikashenafi taarikashenafi Jun 18, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Refactored to if/else and dropped the continue, reads much clearer now

Co-authored-by: Joyce Zhu <joycezhu@github.com>
GitHub Advanced Security started work on behalf of taarikashenafi June 18, 2026 20:41 View session
GitHub Advanced Security finished work on behalf of taarikashenafi June 18, 2026 20:43
GitHub Advanced Security started work on behalf of taarikashenafi June 18, 2026 21:21 View session
GitHub Advanced Security finished work on behalf of taarikashenafi June 18, 2026 21:23
@taarikashenafi taarikashenafi merged commit ce2f468 into main Jun 19, 2026
7 checks passed
@taarikashenafi taarikashenafi deleted the dry-run-option branch June 19, 2026 20:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants