Move banners, spinners, and run progress to stderr (output contract, Phase 0)

AGENTS.md

@@ -37,6 +37,22 @@ If an older command does not follow this yet, do not spread the inconsistency to
 new commands. Either follow the canonical shape above or explain the deviation
 in the PR.
 
+## Output Streams
+
+stdout carries the command's result document only; stderr carries progress,
+spinners, banners, warnings, and hints. The full contract, including envelope
+shapes and the exit-code table, lives in `docs/output-contract.md`.
+
+- Print results, JSON documents, and envelopes via `this.log` / `style` /
+  `printLn` (stdout).
+- Print spinners via `style.actionStart` (it renders on stderr), warnings via
+  `style.longWarning` / `style.shortWarning` (stderr), and run-progress via
+  `printLnErr` (stderr). Do not call `ux.action.start(..., { stdout: true })`.
+- Never print anything to stdout around a `--output json` document; a piped
+  `| jq` must always parse.
+- Terminal users see both streams, so moving chatter to stderr does not change
+  the interactive experience — that is the default for new output.
+
 ## Construct Conventions
 
 - Every property of a construct is synthesized with an explicit value, the

docs/output-contract.md

@@ -0,0 +1,74 @@
+# CLI Output Contract
+
+How the `checkly` CLI shapes its output. This contract is normative for new
+commands and for changes to existing output: follow it, or call out the
+deviation explicitly in the PR.
+
+## Streams: stdout is data, stderr is chatter
+
+A terminal shows both streams interleaved, but scripts, CI jobs, and AI agents
+read them separately — most often they pipe stdout into a parser. That only
+works if stdout is predictable.
+
+- **stdout carries the command's result document only**: tables, JSON output,
+  check results and final summaries, session IDs and links, confirmation and
+  `dry_run` envelopes, API pass-through bodies.
+- **stderr carries everything else**: spinners, progress lines ("Running N
+  checks…", live check summaries), banners and warnings (engine compatibility,
+  stale skill), hints, streamed logs, oclif usage errors.
+
+Consequences:
+
+- `checkly <cmd> --output json | jq` must always parse: nothing may print to
+  stdout ahead of or after the JSON document.
+- Human-facing progress still shows in an interactive terminal (both streams
+  display), so redirecting chatter to stderr changes nothing for a person
+  watching the terminal.
+- `2>/dev/null` silences the chatter without hiding results; `>file` captures
+  a clean result document.
+
+### What renders where today
+
+| Output | Stream |
+|---|---|
+| `--output json` documents, list tables, API pass-through bodies | stdout |
+| Agent envelopes (`confirmation_required`, `dry_run`, login status lines) | stdout |
+| Check results, retry/failure details, final run summaries, session links | stdout |
+| Spinners (`Parsing your project`, `Bundling project resources`, …) | stderr |
+| Engine-compatibility and stale-skill banners, other warnings | stderr |
+| `checkly test` live progress: "Running N checks…", live summary redraws, streamed logs | stderr |
+| oclif usage errors, `login` JSON lines started by another command | stderr |
+
+## Envelopes
+
+One JSON envelope per surface, versioned by convention:
+
+- **List commands** (`--output json`): `{ data, pagination }` where
+  `pagination` is `{ page, limit, total, totalPages }` for offset paging and
+  `{ nextId, length }` for cursor paging. `data` is the array of entries —
+  never the raw API response. See `AGENTS.md` for details.
+- **Error output in JSON mode**: `{ error, detail, hint?, searchedAccount? }`.
+- **Agent envelopes** (auth, confirmation, `--dry-run`): documents built
+  around a `status` field (`success`, `action_required`,
+  `confirmation_required`, `dry_run`, `error`) plus a `confirmCommand` for
+  approval flows.
+
+Do not invent new envelope shapes: extend the closest one above or propose the
+change in the PR.
+
+## Exit codes
+
+| Code | Meaning |
+|---|---|
+| 0 | Success. Also a user-cancelled run (`Ctrl-C`, declined confirmation) and `--dry-run` previews. |
+| 1 | Failure: failed or degraded check runs, API errors, invalid configuration. |
+| 2 | Usage error (unknown flags/args, oclif parse errors) **or** agent-mode `confirmation_required`. These two share a code today — parse the stdout envelope (present only for `confirmation_required`) to tell them apart. |
+| 127 | Unknown command. |
+
+## Notes
+
+- Non-TTY runs do **not** switch output to JSON: CI logs stay human-readable;
+  only spinners and live redraws degrade to plain lines.
+- Spinners are suppressed by `CHECKLY_E2E_DISABLE_FANCY_OUTPUT` and disabled
+  per command when the output format would not use them (for example
+  `checks run --output json`).

eslint.config.mjs

@@ -46,6 +46,9 @@ export default defineConfig([
     // Holds git worktrees, each a full checkout of the repo with its own
     // in-progress code that must not be linted as part of this checkout.
     '**/.claude/',
+    // Scratch space for coding agents (session transcripts, demo tooling),
+    // never part of the shipped code.
+    '**/.context/',
     '**/dist/',
     `**/gen/`,
     '**/__checks__/**/*.spec.{js,ts,mjs}',

packages/cli/e2e/__tests__/deploy.spec.ts

@@ -172,6 +172,15 @@ async function runDeploy (fixt: FixtureSandbox, args: string[], options?: RunOpt
   return result
 }
 
+// A successful deploy may print warnings (construct diagnostics) to stderr —
+// that is their stream under the CLI's output contract — but never an error
+// report. log-symbols renders the error glyph as ✖ on POSIX and × on Windows.
+function expectStderrWithoutErrors (stderr: string): void {
+  // eslint-disable-next-line no-control-regex
+  const plain = stderr.replace(/[\u001b\u009b][[()#;?]*(?:[0-9]{1,4}(?:;[0-9]{0,4})*)?[0-9A-ORZcf-nqry=><]/g, '')
+  expect(plain).not.toMatch(/^[✖×]/m)
+}
+
 /**
  * A regex for consecutive rows of the plan's overview table, one per
  * `[marker, type, logicalId]`, whatever the column padding. A row ends with
@@ -223,7 +232,7 @@ describe('deploy', { timeout: 45_000 }, () => {
         },
       })
 
-      expect(stderr).toBe('')
+      expectStderrWithoutErrors(stderr)
       // expect not to change version since the version is specified
       expect(stdout).not.toContain('Notice: replacing version')
 
@@ -246,7 +255,7 @@ describe('deploy', { timeout: 45_000 }, () => {
         CHECKLY_E2E_CLI_VERSION: undefined,
       }
       const first = await runDeploy(fixt, ['--force'], { env })
-      expect(first.stderr).toBe('')
+      expectStderrWithoutErrors(first.stderr)
       const api = axios.create({
         baseURL: config.get('baseURL'),
         headers: { 'x-checkly-account': config.get('accountId'), 'Authorization': `Bearer ${config.get('apiKey')}` },
@@ -274,7 +283,7 @@ describe('deploy', { timeout: 45_000 }, () => {
       // Unchanged code: the row moved since the last deploy, so the plan writes
       // the check and the code's unset values win.
       const second = await runDeploy(fixt, ['--force'], { env })
-      expect(second.stderr).toBe('')
+      expectStderrWithoutErrors(second.stderr)
       const { data: redeployed } = await api.get(`/v1/checks/${homePage.id}`)
       expect(redeployed.intent).toBeNull()
       expect(redeployed.aiAutoRepairEnabled).toBeNull()
@@ -288,7 +297,7 @@ describe('deploy', { timeout: 45_000 }, () => {
           CHECKLY_E2E_CLI_VERSION: undefined,
         },
       })
-      expect(stderr).toBe('')
+      expectStderrWithoutErrors(stderr)
       // Without a plan the deploy writes every resource, whether or not it
       // changed, so a deploy of what is already deployed is never "up to date".
       expect(stdout).not.toContain('No changes.')
@@ -303,7 +312,7 @@ describe('deploy', { timeout: 45_000 }, () => {
           CHECKLY_E2E_CLI_VERSION: undefined,
         },
       })
-      expect(stderr).toBe('')
+      expectStderrWithoutErrors(stderr)
       // Non-interactive runs should no longer emit the local-dev version notice.
       expect(stdout).not.toContain('Notice: replacing version')
 
@@ -408,7 +417,7 @@ describe('deploy', { timeout: 45_000 }, () => {
         },
       })
 
-      expect(stderr).toBe('')
+      expectStderrWithoutErrors(stderr)
     })
   })
 

packages/cli/src/ai-context/skill.md

@@ -85,6 +85,24 @@ The `confirmCommand` is the approved command, ready to run verbatim: it starts w
 
 Run `npx checkly skills communicate` for the full protocol details, or `npx checkly skills configure` for what `deploy` confirms.
 
+## Output Streams and Exit Codes
+
+The CLI keeps data separate from chatter, so you can pipe and parse stdout without surprises:
+
+- **stdout carries data only**: command results, `--output json` documents, API pass-through bodies, and the confirmation/dry-run envelopes above.
+- **stderr carries everything else**: spinners and progress lines ("Running N checks…", live check summaries), warnings and banners (e.g. a stale installed skill), hints, and streamed check logs.
+
+`npx checkly <command> --output json | jq` is therefore always safe — nothing else can appear on stdout around the JSON. If you need the human-facing progress while capturing data, read stderr separately.
+
+Exit codes:
+
+| Code | Meaning |
+|---|---|
+| 0 | Success (also a user-cancelled run and `--dry-run` previews) |
+| 1 | Failure: failed checks, API errors, invalid configuration |
+| 2 | Usage error (bad flags) **or** `confirmation_required` — the envelope on stdout only appears for `confirmation_required` |
+| 127 | Unknown command |
+
 ## API Pass-Through (fallback for any endpoint)
 
 When no dedicated CLI command exists for an endpoint, use `npx checkly api` to make authenticated requests directly. The CLI handles auth headers and base URL automatically.

packages/cli/src/commands/__tests__/login.spec.ts

@@ -1238,15 +1238,15 @@ describe('checkly login', () => {
         const cmd = createCommand()
         await expect(cmd.run()).rejects.toThrow('EXIT_0')
 
-        expect(ux.action.start).toHaveBeenCalledWith('Waiting for you to finish in the browser', undefined, { stdout: true })
+        expect(ux.action.start).toHaveBeenCalledWith('Waiting for you to finish in the browser')
         expect(ux.action.stop).toHaveBeenCalledWith('✅')
         expect(loggedLines(cmd)).not.toContain('Waiting for you to finish in the browser…')
-        // The spinner line itself is drawn by oclif, between the two empty
-        // lines; actionSuccess's empty line separates the result, no second one.
+        // The spinner and its trailing empty line are drawn by oclif on
+        // stderr, so stdout carries only the result lines; actionSuccess's
+        // empty line separates the result on the terminal, not on stdout.
         expect(loggedLines(cmd)).toEqual([
           'Visit auth.checklyhq.com/activate and enter ABCD-EFGH',
           '',
-          '',
           'Logged in as Ada Lovelace to Acme.',
           '',
           'To create checks for a project, run `pnpm checkly init`.',
@@ -1265,7 +1265,7 @@ describe('checkly login', () => {
         expect(lines[lines.indexOf('Logged in as Ada Lovelace to Globex.') - 1]).toBe('')
       })
 
-      it('keeps a login another command started off the spinner, which writes to stdout', async () => {
+      it('keeps a login another command started off the spinner, which writes to stderr', async () => {
         const cmd = createCommand()
         await expect(cmd.login({ inline: true })).resolves.toBe(true)
 

packages/cli/src/commands/login.ts

@@ -899,7 +899,7 @@ export default class Login extends BaseCommand {
    * command's own output on a terminal; otherwise prints `message` as a plain
    * line, so piped output gets no animation frames and a login another
    * command started keeps its output on stderr (the spinner writes to
-   * stdout). `report` shows a note while waiting.
+   * stderr, too). `report` shows a note while waiting.
    */
   async #whileWaiting<T> (message: string, wait: (report: (note: string) => void) => Promise<T>): Promise<T> {
     if (this.#mode !== 'interactive' || this.#inline || !this.fancy || !process.stdout.isTTY) {

packages/cli/src/commands/sync-playwright.ts

@@ -15,7 +15,8 @@ export default class SyncPlaywright extends BaseCommand {
 
   async run (): Promise<void> {
     if (this.fancy) {
-      ux.action.start('Syncing Playwright config to the Checkly config file', undefined, { stdout: true })
+      // oclif renders the spinner on stderr by default, keeping stdout clean.
+      ux.action.start('Syncing Playwright config to the Checkly config file')
     }
 
     const configFile = await getChecklyConfigFile()

packages/cli/src/commands/test.ts

@@ -19,7 +19,7 @@ import { splitConfigFilePath, getGitInformation, getCiInformation, getEnvs } fro
 import { createReporters, ReporterType } from '../reporters/reporter.js'
 import commonMessages from '../messages/common-messages.js'
 import { TestResultsShortLinks } from '../rest/test-sessions.js'
-import { printLn, formatCheckTitle, CheckStatus } from '../reporters/util.js'
+import { printLn, printLnErr, formatCheckTitle, CheckStatus } from '../reporters/util.js'
 import { uploadSnapshots } from '../services/snapshot-service.js'
 import { isEntrypoint } from '../constructs/construct.js'
 import { BrowserCheckBundle } from '../constructs/browser-check-bundle.js'
@@ -428,7 +428,7 @@ export default class Test extends AuthCommand {
   prepareTestRetryStrategy (retries?: number, configRetries?: number) {
     const numRetries = retries ?? configRetries ?? 0
     if (numRetries > MAX_RETRIES) {
-      printLn(`Defaulting to the maximum of ${MAX_RETRIES} retries.`)
+      printLnErr(`Defaulting to the maximum of ${MAX_RETRIES} retries.`)
     }
     return numRetries
       ? RetryStrategyBuilder.fixedStrategy({

packages/cli/src/commands/trigger.ts

@@ -18,7 +18,7 @@ import {
 } from '../services/abstract-check-runner.js'
 import config from '../services/config.js'
 import { createReporters, ReporterType } from '../reporters/reporter.js'
-import { printLn } from '../reporters/util.js'
+import { printLnErr } from '../reporters/util.js'
 import { NoMatchingChecksError, TestResultsShortLinks } from '../rest/test-sessions.js'
 import { Session, RetryStrategyBuilder } from '../constructs/index.js'
 import { DEFAULT_REGION } from '../helpers/constants.js'
@@ -284,7 +284,7 @@ export default class Trigger extends AuthCommand {
   prepareTestRetryStrategy (retries?: number, configRetries?: number) {
     const numRetries = retries ?? configRetries ?? 0
     if (numRetries > MAX_RETRIES) {
-      printLn(`Defaulting to the maximum of ${MAX_RETRIES} retries.`)
+      printLnErr(`Defaulting to the maximum of ${MAX_RETRIES} retries.`)
     }
     return numRetries
       ? RetryStrategyBuilder.fixedStrategy({

packages/cli/src/helpers/__tests__/command-style.spec.ts

@@ -1,18 +1,21 @@
-import { describe, expect, it, vi } from 'vitest'
+import { describe, expect, it, vi, afterEach } from 'vitest'
+import { ux } from '@oclif/core'
 
 import { CommandStyle } from '../command-style.js'
 import { NotFoundError, ServerError } from '../../rest/errors.js'
 
 function createStyle (hint: string | undefined, outputFormat?: string) {
   const command = {
     log: vi.fn(),
+    logToStderr: vi.fn(),
     notFoundHint: vi.fn(() => hint),
     searchedAccount: vi.fn(() => hint ? { id: 'acc-1', name: 'Acme' } : undefined),
   }
   const style = new CommandStyle(command as any)
   style.outputFormat = outputFormat
   const lines = () => command.log.mock.calls.map(([line]) => String(line ?? ''))
-  return { style, lines }
+  const errLines = () => command.logToStderr.mock.calls.map(([line]) => String(line ?? ''))
+  return { style, command, lines, errLines }
 }
 
 const notFound = () => new NotFoundError({ statusCode: 404, error: 'Not Found', message: 'No such test session.' })
@@ -55,3 +58,68 @@ describe('CommandStyle.longError', () => {
     expect(lines().join('\n')).not.toContain('Searched account')
   })
 })
+
+describe('CommandStyle output streams', () => {
+  // Mocked implementations: a real oclif spinner would write to the test
+  // runner's stderr.
+  const startSpy = vi.spyOn(ux.action, 'start').mockImplementation(() => {})
+  const stopSpy = vi.spyOn(ux.action, 'stop').mockImplementation(() => {})
+
+  afterEach(() => {
+    startSpy.mockClear()
+    stopSpy.mockClear()
+  })
+
+  it('renders the spinner and its separator line on stderr, not stdout', () => {
+    const { style, command } = createStyle(undefined)
+    command.fancy = true
+
+    style.actionStart('Parsing your project')
+    expect(startSpy).toHaveBeenCalledWith('Parsing your project')
+    expect(startSpy).not.toHaveBeenCalledWith('Parsing your project', undefined, { stdout: true })
+
+    style.actionSuccess()
+    expect(stopSpy).toHaveBeenCalledWith('✅')
+
+    style.actionStart('Deploying project')
+    style.actionFailure()
+    expect(stopSpy).toHaveBeenCalledWith('❌')
+
+    // The blank separator line follows the spinner to stderr.
+    expect(command.logToStderr).toHaveBeenCalled()
+    expect(command.log).not.toHaveBeenCalled()
+  })
+
+  it('does not start a spinner when fancy output is off', () => {
+    const { style, command } = createStyle(undefined)
+    command.fancy = false
+
+    style.actionStart('Parsing your project')
+
+    expect(startSpy).not.toHaveBeenCalled()
+  })
+
+  it('prints warnings to stderr only', () => {
+    const { style, command, lines, errLines } = createStyle(undefined)
+
+    style.longWarning('Checkly skill is out of date', 'Re-install to update.')
+    style.shortWarning('No Checkly config file found')
+
+    const stderr = errLines().join('\n')
+    expect(stderr).toContain('Checkly skill is out of date')
+    expect(stderr).toContain('Re-install to update.')
+    expect(stderr).toContain('No Checkly config file found')
+    expect(lines()).toEqual([])
+    expect(command.log).not.toHaveBeenCalled()
+  })
+
+  it('keeps JSON error documents on stdout', () => {
+    const { style, lines } = createStyle(undefined, 'json')
+
+    style.shortError('Your Checkly configuration file is not valid.')
+
+    expect(lines().map(line => JSON.parse(line))).toEqual([{
+      error: 'Your Checkly configuration file is not valid.',
+    }])
+  })
+})