Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions .github/copilot-instructions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# Copilot instructions for vscode-java-pack

## UI and E2E tests

- When asked to add, update, run, or debug UI/E2E coverage, prefer the AutoTest YAML workflow under `test-plans/`.
- Use the `uitest` skill for UI test work. It should create or update `test-plans/*.yaml`, validate the plan, package the extension when needed, run AutoTest, and inspect `test-results/`.
- Do not create legacy VS Code extension tests for UI coverage unless the user explicitly asks for that format.
- Prefer deterministic AutoTest verifiers (`verifyWebview`, `verifyTreeItem`, `verifyFile`, `verifyProblems`, `verifyCompletion`, `verifyEditorTab`, `verifyTerminal`, `verifyOutputChannel`) over screenshot-only checks.

46 changes: 46 additions & 0 deletions .github/instructions/uitest-plan.instructions.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
---
applyTo: "test-plans/**/*.yaml"
description: "Authoring rules for vscode-java-pack AutoTest UI/E2E YAML test plans"
---

# AutoTest UI/E2E test plan instructions

Test plans under `test-plans/` are executable YAML files consumed by `@vscjava/vscode-autotest`. They should describe stable user scenarios, not raw implementation details.

## Setup rules

- Use `setup.extension: "redhat.java"` plus `setup.extensions: ["vscjava.vscode-java-pack"]` for most Java extension pack scenarios.
- Use a local VSIX at runtime with `--vsix <path>` when validating current-branch changes.
- Keep setup paths relative to the test plan file. Use `~/` in verification paths to refer to the runtime workspace root.
- Prefer existing lightweight fixtures in `test-fixtures/` or external sample projects already used by CI (`../vscode-java`, `../eclipse.jdt.ls`). Do not add large binary fixtures.
- Disable noisy startup surfaces with settings when relevant, for example `java.help.showReleaseNotes: false` and `java.help.firstView: "none"`.

## Action rules

- Prefer stable commands and command IDs (`run command`, `executeVSCodeCommand`) before UI locators.
- Use `clickInWebview` and `verifyWebview` for webview scenarios.
- Use `insertLineInFile` for Java edits that the language server must analyze. Use `typeInEditor` only for text that does not require language-server analysis.
- Use `waitForLanguageServer`, `waitForTestDiscovery`, or verifier polling before static waits. Short static waits are acceptable only for UI rendering settle time.
- Quote action arguments that contain spaces:

```yaml
action: 'contextMenu "Maven Dependencies" "Add JAR"'
```

## Verification rules

- Add deterministic verification to every meaningful step. The natural-language `verify` field is context for humans and failure analysis; it is not pass/fail authority by itself.
- Prefer `verifyFile` after language-server edits such as code actions, organize imports, or rename. VS Code can open duplicate editor tabs with stale buffers.
- Use `verifyProblems` for compiler/problem expectations and `verifyCompletion` for IntelliSense expectations.
- Use screenshots only as diagnostics produced by AutoTest; do not make screenshots the only evidence of pass/fail.

## Local validation commands

```powershell
npx -y @vscjava/vscode-autotest validate test-plans\<name>.yaml
npm ci # first time only; on later iterations run just the three commands below
npm run build
npx @vscode/vsce@latest package -o vscode-java-pack-pr.vsix
npx -y @vscjava/vscode-autotest run test-plans\<name>.yaml --vsix vscode-java-pack-pr.vsix --no-llm
```

79 changes: 79 additions & 0 deletions .github/skills/uitest/SKILL.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
---
name: uitest
description: Write, update, run, or debug vscode-java-pack UI/E2E tests using AutoTest YAML plans. Use when the user asks for a UI test, E2E test, VS Code UI validation, webview test, Java extension pack workflow test, or autotest plan.
---

# UI/E2E tests with AutoTest

Use this skill to add or update UI/E2E coverage for `vscode-java-pack`.

The repository uses `@vscjava/vscode-autotest`: YAML plans in `test-plans/*.yaml` launch VS Code, install Java extensions or local VSIX files, execute user-facing actions, capture screenshots, and write `test-results/<plan>/results.json`.

## Prerequisites (local)

- Node.js >= 18 and JDK 21+ installed and on `PATH`.
- Close any running VS Code instance before running a plan locally; a running instance can block AutoTest from launching its own VS Code.
- When a plan references external sample projects, clone them as siblings of this repo first (CI does this automatically):

```powershell
git clone --depth 1 https://github.com/redhat-developer/vscode-java.git ../vscode-java
git clone --depth 1 https://github.com/eclipse-jdtls/eclipse.jdt.ls.git ../eclipse.jdt.ls
```

## Workflow

1. Identify the scenario and search `test-plans/*.yaml` for an existing plan that already covers the area.
2. Update the existing plan when possible. Create a new `test-plans/<scenario>.yaml` only when no existing plan fits.
3. Use stable AutoTest actions and deterministic verifiers. Do not add raw Playwright tests or screenshot-only checks.
4. Validate the plan:

```powershell
npx -y @vscjava/vscode-autotest validate test-plans\<name>.yaml
```

5. If validating the current branch, build and package the extension:

```powershell
npm ci # first time only; on later iterations run just the two commands below
npm run build
npx @vscode/vsce@latest package -o vscode-java-pack-pr.vsix
```

6. Run the plan against the packaged VSIX:

```powershell
npx -y @vscjava/vscode-autotest run test-plans\<name>.yaml --vsix vscode-java-pack-pr.vsix --no-llm
```

7. Inspect `test-results/<name>/results.json` and `test-results/<name>/screenshots/`.
8. Iterate based on the failure cause:
- **Incorrect plan**: fix the YAML and rerun step 6. No rebuild is needed.
- **Product code fix**: after editing extension source, re-run step 5 (build + repackage the VSIX) before rerunning step 6. Never rerun against a stale VSIX.
- **Product bug (report only)**: report the observed behavior and cite the failing step, screenshot, and result reason.

## Authoring rules

- For most plans, use:

```yaml
setup:
extension: "redhat.java"
extensions:
- "vscjava.vscode-java-pack"
vscodeVersion: "stable"
```

- Use `--vsix vscode-java-pack-pr.vsix` to test current-branch changes.
- Prefer `executeVSCodeCommand` or `run command` for command-driven UI.
- Prefer `verifyWebview` for webview content, `verifyTreeItem` for tree views, `verifyFile` for generated or modified files, `verifyProblems` for diagnostics, and `verifyCompletion` for IntelliSense.
- Use `insertLineInFile` for Java source edits that JDT LS must observe.
- Use `verifyFile` instead of `verifyEditor` after code actions, organize imports, or rename.
- Keep step IDs unique, descriptive, and kebab-case.
- Avoid hard-coded coordinates and brittle DOM structure assumptions.

## CI

The repository workflow `.github/workflows/e2e-autotest.yml` builds a branch VSIX, discovers `test-plans/*.yaml`, runs plans on Windows/Linux/macOS, and uploads `test-results/` artifacts.

Use workflow dispatch when the user asks to validate in CI or across platforms. The `test_plan` input can target a single plan; leave it empty to run all plans.

Loading