From 9bcba1280feab22fd35bf094e2153e283c550023 Mon Sep 17 00:00:00 2001 From: wenyutang-ms Date: Wed, 1 Jul 2026 16:59:19 +0800 Subject: [PATCH] Add Copilot UI test skill --- .github/copilot-instructions.md | 9 +++ .../instructions/uitest-plan.instructions.md | 46 +++++++++++ .github/skills/uitest/SKILL.md | 79 +++++++++++++++++++ 3 files changed, 134 insertions(+) create mode 100644 .github/copilot-instructions.md create mode 100644 .github/instructions/uitest-plan.instructions.md create mode 100644 .github/skills/uitest/SKILL.md diff --git a/.github/copilot-instructions.md b/.github/copilot-instructions.md new file mode 100644 index 00000000..e9d786dc --- /dev/null +++ b/.github/copilot-instructions.md @@ -0,0 +1,9 @@ +# Copilot instructions for vscode-java-pack + +## UI and E2E tests + +- When asked to add, update, run, or debug UI/E2E coverage, prefer the AutoTest YAML workflow under `test-plans/`. +- Use the `uitest` skill for UI test work. It should create or update `test-plans/*.yaml`, validate the plan, package the extension when needed, run AutoTest, and inspect `test-results/`. +- Do not create legacy VS Code extension tests for UI coverage unless the user explicitly asks for that format. +- Prefer deterministic AutoTest verifiers (`verifyWebview`, `verifyTreeItem`, `verifyFile`, `verifyProblems`, `verifyCompletion`, `verifyEditorTab`, `verifyTerminal`, `verifyOutputChannel`) over screenshot-only checks. + diff --git a/.github/instructions/uitest-plan.instructions.md b/.github/instructions/uitest-plan.instructions.md new file mode 100644 index 00000000..bf0f90ec --- /dev/null +++ b/.github/instructions/uitest-plan.instructions.md @@ -0,0 +1,46 @@ +--- +applyTo: "test-plans/**/*.yaml" +description: "Authoring rules for vscode-java-pack AutoTest UI/E2E YAML test plans" +--- + +# AutoTest UI/E2E test plan instructions + +Test plans under `test-plans/` are executable YAML files consumed by `@vscjava/vscode-autotest`. They should describe stable user scenarios, not raw implementation details. + +## Setup rules + +- Use `setup.extension: "redhat.java"` plus `setup.extensions: ["vscjava.vscode-java-pack"]` for most Java extension pack scenarios. +- Use a local VSIX at runtime with `--vsix ` when validating current-branch changes. +- Keep setup paths relative to the test plan file. Use `~/` in verification paths to refer to the runtime workspace root. +- Prefer existing lightweight fixtures in `test-fixtures/` or external sample projects already used by CI (`../vscode-java`, `../eclipse.jdt.ls`). Do not add large binary fixtures. +- Disable noisy startup surfaces with settings when relevant, for example `java.help.showReleaseNotes: false` and `java.help.firstView: "none"`. + +## Action rules + +- Prefer stable commands and command IDs (`run command`, `executeVSCodeCommand`) before UI locators. +- Use `clickInWebview` and `verifyWebview` for webview scenarios. +- Use `insertLineInFile` for Java edits that the language server must analyze. Use `typeInEditor` only for text that does not require language-server analysis. +- Use `waitForLanguageServer`, `waitForTestDiscovery`, or verifier polling before static waits. Short static waits are acceptable only for UI rendering settle time. +- Quote action arguments that contain spaces: + +```yaml +action: 'contextMenu "Maven Dependencies" "Add JAR"' +``` + +## Verification rules + +- Add deterministic verification to every meaningful step. The natural-language `verify` field is context for humans and failure analysis; it is not pass/fail authority by itself. +- Prefer `verifyFile` after language-server edits such as code actions, organize imports, or rename. VS Code can open duplicate editor tabs with stale buffers. +- Use `verifyProblems` for compiler/problem expectations and `verifyCompletion` for IntelliSense expectations. +- Use screenshots only as diagnostics produced by AutoTest; do not make screenshots the only evidence of pass/fail. + +## Local validation commands + +```powershell +npx -y @vscjava/vscode-autotest validate test-plans\.yaml +npm ci # first time only; on later iterations run just the three commands below +npm run build +npx @vscode/vsce@latest package -o vscode-java-pack-pr.vsix +npx -y @vscjava/vscode-autotest run test-plans\.yaml --vsix vscode-java-pack-pr.vsix --no-llm +``` + diff --git a/.github/skills/uitest/SKILL.md b/.github/skills/uitest/SKILL.md new file mode 100644 index 00000000..2746e160 --- /dev/null +++ b/.github/skills/uitest/SKILL.md @@ -0,0 +1,79 @@ +--- +name: uitest +description: Write, update, run, or debug vscode-java-pack UI/E2E tests using AutoTest YAML plans. Use when the user asks for a UI test, E2E test, VS Code UI validation, webview test, Java extension pack workflow test, or autotest plan. +--- + +# UI/E2E tests with AutoTest + +Use this skill to add or update UI/E2E coverage for `vscode-java-pack`. + +The repository uses `@vscjava/vscode-autotest`: YAML plans in `test-plans/*.yaml` launch VS Code, install Java extensions or local VSIX files, execute user-facing actions, capture screenshots, and write `test-results//results.json`. + +## Prerequisites (local) + +- Node.js >= 18 and JDK 21+ installed and on `PATH`. +- Close any running VS Code instance before running a plan locally; a running instance can block AutoTest from launching its own VS Code. +- When a plan references external sample projects, clone them as siblings of this repo first (CI does this automatically): + +```powershell +git clone --depth 1 https://github.com/redhat-developer/vscode-java.git ../vscode-java +git clone --depth 1 https://github.com/eclipse-jdtls/eclipse.jdt.ls.git ../eclipse.jdt.ls +``` + +## Workflow + +1. Identify the scenario and search `test-plans/*.yaml` for an existing plan that already covers the area. +2. Update the existing plan when possible. Create a new `test-plans/.yaml` only when no existing plan fits. +3. Use stable AutoTest actions and deterministic verifiers. Do not add raw Playwright tests or screenshot-only checks. +4. Validate the plan: + +```powershell +npx -y @vscjava/vscode-autotest validate test-plans\.yaml +``` + +5. If validating the current branch, build and package the extension: + +```powershell +npm ci # first time only; on later iterations run just the two commands below +npm run build +npx @vscode/vsce@latest package -o vscode-java-pack-pr.vsix +``` + +6. Run the plan against the packaged VSIX: + +```powershell +npx -y @vscjava/vscode-autotest run test-plans\.yaml --vsix vscode-java-pack-pr.vsix --no-llm +``` + +7. Inspect `test-results//results.json` and `test-results//screenshots/`. +8. Iterate based on the failure cause: + - **Incorrect plan**: fix the YAML and rerun step 6. No rebuild is needed. + - **Product code fix**: after editing extension source, re-run step 5 (build + repackage the VSIX) before rerunning step 6. Never rerun against a stale VSIX. + - **Product bug (report only)**: report the observed behavior and cite the failing step, screenshot, and result reason. + +## Authoring rules + +- For most plans, use: + +```yaml +setup: + extension: "redhat.java" + extensions: + - "vscjava.vscode-java-pack" + vscodeVersion: "stable" +``` + +- Use `--vsix vscode-java-pack-pr.vsix` to test current-branch changes. +- Prefer `executeVSCodeCommand` or `run command` for command-driven UI. +- Prefer `verifyWebview` for webview content, `verifyTreeItem` for tree views, `verifyFile` for generated or modified files, `verifyProblems` for diagnostics, and `verifyCompletion` for IntelliSense. +- Use `insertLineInFile` for Java source edits that JDT LS must observe. +- Use `verifyFile` instead of `verifyEditor` after code actions, organize imports, or rename. +- Keep step IDs unique, descriptive, and kebab-case. +- Avoid hard-coded coordinates and brittle DOM structure assumptions. + +## CI + +The repository workflow `.github/workflows/e2e-autotest.yml` builds a branch VSIX, discovers `test-plans/*.yaml`, runs plans on Windows/Linux/macOS, and uploads `test-results/` artifacts. + +Use workflow dispatch when the user asks to validate in CI or across platforms. The `test_plan` input can target a single plan; leave it empty to run all plans. +