Skip to content

Commit 82fe9cd

Browse files
wenytang-mswenytang-ms
authored
chore: Add Copilot UI test skill (#1682)
1 parent e4f40ea commit 82fe9cd

3 files changed

Lines changed: 134 additions & 0 deletions

File tree

‎.github/copilot-instructions.md‎

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
# Copilot instructions for vscode-java-pack
2+
3+
## UI and E2E tests
4+
5+
- When asked to add, update, run, or debug UI/E2E coverage, prefer the AutoTest YAML workflow under `test-plans/`.
6+
- Use the `uitest` skill for UI test work. It should create or update `test-plans/*.yaml`, validate the plan, package the extension when needed, run AutoTest, and inspect `test-results/`.
7+
- Do not create legacy VS Code extension tests for UI coverage unless the user explicitly asks for that format.
8+
- Prefer deterministic AutoTest verifiers (`verifyWebview`, `verifyTreeItem`, `verifyFile`, `verifyProblems`, `verifyCompletion`, `verifyEditorTab`, `verifyTerminal`, `verifyOutputChannel`) over screenshot-only checks.
9+

‎.github/instructions/uitest-plan.instructions.md‎

Lines changed: 46 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
---
2+
applyTo: "test-plans/**/*.yaml"
3+
description: "Authoring rules for vscode-java-pack AutoTest UI/E2E YAML test plans"
4+
---
5+
6+
# AutoTest UI/E2E test plan instructions
7+
8+
Test plans under `test-plans/` are executable YAML files consumed by `@vscjava/vscode-autotest`. They should describe stable user scenarios, not raw implementation details.
9+
10+
## Setup rules
11+
12+
- Use `setup.extension: "redhat.java"` plus `setup.extensions: ["vscjava.vscode-java-pack"]` for most Java extension pack scenarios.
13+
- Use a local VSIX at runtime with `--vsix <path>` when validating current-branch changes.
14+
- Keep setup paths relative to the test plan file. Use `~/` in verification paths to refer to the runtime workspace root.
15+
- Prefer existing lightweight fixtures in `test-fixtures/` or external sample projects already used by CI (`../vscode-java`, `../eclipse.jdt.ls`). Do not add large binary fixtures.
16+
- Disable noisy startup surfaces with settings when relevant, for example `java.help.showReleaseNotes: false` and `java.help.firstView: "none"`.
17+
18+
## Action rules
19+
20+
- Prefer stable commands and command IDs (`run command`, `executeVSCodeCommand`) before UI locators.
21+
- Use `clickInWebview` and `verifyWebview` for webview scenarios.
22+
- Use `insertLineInFile` for Java edits that the language server must analyze. Use `typeInEditor` only for text that does not require language-server analysis.
23+
- Use `waitForLanguageServer`, `waitForTestDiscovery`, or verifier polling before static waits. Short static waits are acceptable only for UI rendering settle time.
24+
- Quote action arguments that contain spaces:
25+
26+
```yaml
27+
action: 'contextMenu "Maven Dependencies" "Add JAR"'
28+
```
29+
30+
## Verification rules
31+
32+
- Add deterministic verification to every meaningful step. The natural-language `verify` field is context for humans and failure analysis; it is not pass/fail authority by itself.
33+
- Prefer `verifyFile` after language-server edits such as code actions, organize imports, or rename. VS Code can open duplicate editor tabs with stale buffers.
34+
- Use `verifyProblems` for compiler/problem expectations and `verifyCompletion` for IntelliSense expectations.
35+
- Use screenshots only as diagnostics produced by AutoTest; do not make screenshots the only evidence of pass/fail.
36+
37+
## Local validation commands
38+
39+
```powershell
40+
npx -y @vscjava/vscode-autotest validate test-plans\<name>.yaml
41+
npm ci # first time only; on later iterations run just the three commands below
42+
npm run build
43+
npx @vscode/vsce@latest package -o vscode-java-pack-pr.vsix
44+
npx -y @vscjava/vscode-autotest run test-plans\<name>.yaml --vsix vscode-java-pack-pr.vsix --no-llm
45+
```
46+

‎.github/skills/uitest/SKILL.md‎

Lines changed: 79 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,79 @@
1+
---
2+
name: uitest
3+
description: Write, update, run, or debug vscode-java-pack UI/E2E tests using AutoTest YAML plans. Use when the user asks for a UI test, E2E test, VS Code UI validation, webview test, Java extension pack workflow test, or autotest plan.
4+
---
5+
6+
# UI/E2E tests with AutoTest
7+
8+
Use this skill to add or update UI/E2E coverage for `vscode-java-pack`.
9+
10+
The repository uses `@vscjava/vscode-autotest`: YAML plans in `test-plans/*.yaml` launch VS Code, install Java extensions or local VSIX files, execute user-facing actions, capture screenshots, and write `test-results/<plan>/results.json`.
11+
12+
## Prerequisites (local)
13+
14+
- Node.js >= 18 and JDK 21+ installed and on `PATH`.
15+
- Close any running VS Code instance before running a plan locally; a running instance can block AutoTest from launching its own VS Code.
16+
- When a plan references external sample projects, clone them as siblings of this repo first (CI does this automatically):
17+
18+
```powershell
19+
git clone --depth 1 https://github.com/redhat-developer/vscode-java.git ../vscode-java
20+
git clone --depth 1 https://github.com/eclipse-jdtls/eclipse.jdt.ls.git ../eclipse.jdt.ls
21+
```
22+
23+
## Workflow
24+
25+
1. Identify the scenario and search `test-plans/*.yaml` for an existing plan that already covers the area.
26+
2. Update the existing plan when possible. Create a new `test-plans/<scenario>.yaml` only when no existing plan fits.
27+
3. Use stable AutoTest actions and deterministic verifiers. Do not add raw Playwright tests or screenshot-only checks.
28+
4. Validate the plan:
29+
30+
```powershell
31+
npx -y @vscjava/vscode-autotest validate test-plans\<name>.yaml
32+
```
33+
34+
5. If validating the current branch, build and package the extension:
35+
36+
```powershell
37+
npm ci # first time only; on later iterations run just the two commands below
38+
npm run build
39+
npx @vscode/vsce@latest package -o vscode-java-pack-pr.vsix
40+
```
41+
42+
6. Run the plan against the packaged VSIX:
43+
44+
```powershell
45+
npx -y @vscjava/vscode-autotest run test-plans\<name>.yaml --vsix vscode-java-pack-pr.vsix --no-llm
46+
```
47+
48+
7. Inspect `test-results/<name>/results.json` and `test-results/<name>/screenshots/`.
49+
8. Iterate based on the failure cause:
50+
- **Incorrect plan**: fix the YAML and rerun step 6. No rebuild is needed.
51+
- **Product code fix**: after editing extension source, re-run step 5 (build + repackage the VSIX) before rerunning step 6. Never rerun against a stale VSIX.
52+
- **Product bug (report only)**: report the observed behavior and cite the failing step, screenshot, and result reason.
53+
54+
## Authoring rules
55+
56+
- For most plans, use:
57+
58+
```yaml
59+
setup:
60+
extension: "redhat.java"
61+
extensions:
62+
- "vscjava.vscode-java-pack"
63+
vscodeVersion: "stable"
64+
```
65+
66+
- Use `--vsix vscode-java-pack-pr.vsix` to test current-branch changes.
67+
- Prefer `executeVSCodeCommand` or `run command` for command-driven UI.
68+
- Prefer `verifyWebview` for webview content, `verifyTreeItem` for tree views, `verifyFile` for generated or modified files, `verifyProblems` for diagnostics, and `verifyCompletion` for IntelliSense.
69+
- Use `insertLineInFile` for Java source edits that JDT LS must observe.
70+
- Use `verifyFile` instead of `verifyEditor` after code actions, organize imports, or rename.
71+
- Keep step IDs unique, descriptive, and kebab-case.
72+
- Avoid hard-coded coordinates and brittle DOM structure assumptions.
73+
74+
## CI
75+
76+
The repository workflow `.github/workflows/e2e-autotest.yml` builds a branch VSIX, discovers `test-plans/*.yaml`, runs plans on Windows/Linux/macOS, and uploads `test-results/` artifacts.
77+
78+
Use workflow dispatch when the user asks to validate in CI or across platforms. The `test_plan` input can target a single plan; leave it empty to run all plans.
79+

0 commit comments

Comments
 (0)