Apify Test Tools

Build, test and release Apify Actors from a Git repository or local source code. The package has three parts:

1. A vitest library for writing platform tests that run your Actors on Apify and check the results:

   testActor('myteam/web-scraper', 'scrapes the start URL', async ({ expect, run }) => {
       const runResult = await run({ input: { startUrls: [{ url: 'https://apify.com' }] } });
       await expect(runResult).toFinishWith({ datasetItemCount: { min: 1, max: 10 } });
   });

2. A CLI that finds the Actors a change affects, builds them, and reports the results. It runs anywhere, so you can build your own pipeline on it in any CI:

   npx apify-test-tools get-affected-actors --base origin/master \
     | jq '{ actors: .functionallyChangedActors }' \
     | npx apify-test-tools build --version-number 0.99

3. Ready-made GitHub Actions workflows built on the CLI and the library, so a GitHub repo needs only a few lines per workflow:

   jobs:
       buildDevelAndTest:
           uses: apify/apify-test-tools/.github/workflows/public_pr-build-test.yaml@workflows-v0
           secrets:
               TESTER_APIFY_TOKEN: ${{ secrets.TESTER_APIFY_TOKEN }}
               # One entry per secret apify-test-tools.config.json names, today each `tokenEnvVar`
               CONFIG_SECRETS: |
                   { "APIFY_TOKEN_MYTEAM": "${{ secrets.APIFY_TOKEN_MYTEAM }}" }

Most repos only need the library and the workflows: the workflows call the CLI for you. Use the CLI directly only to build your own pipeline, e.g. on a CI platform other than GitHub, or to build and test locally.

Only what changed is retested

Builds and platform tests cost time and money, so the tool skips everything a change can't affect:

• Only affected Actors. A changed file only counts for the Actors whose build context it falls in, minus their .dockerignore. In a monorepo, a change to one Actor doesn't build or test the others.
• Cosmetic changes skip tests. README, CHANGELOG and formatting-only schema changes don't trigger tests on a PR; they're only rebuilt when released.
• Only new changes on a PR. Each push to a PR retests only the Actors that changed since its last green run, not the whole PR again.
• Only their tests. Only the tests of the Actors that were built run.

See Change detection for the exact rules.

This README covers what you need to get a repo running. The full reference lives in docs/:

• Vitest library API — testActor, testStandbyActor, RunTestResult, matchers, running tests locally
• CLI API — every command, change detection, releasing, building locally
• Workflow API — every GitHub workflow, its inputs and secrets, versioning

Contributing

Getting started

Your Actors can be written in any programming language, e.g. Python: anything the platform builds from a Dockerfile works. Only the tests and this tooling run on Node.js, so all a repo needs is a package.json at the root with apify-test-tools and vitest in its devDependencies, and tests written in TypeScript or JavaScript. If you use the PR workflow, also define the build-check, lint and test scripts its static checks run, e.g. with your language's own linter and unit tests.

1. Install the package

npm i -D apify-test-tools

• Requires vitest version 3.2.4 or later (uses annotate)
• Make sure target and module in your tsconfig.json's compilerOptions are set to ES2022
• Every Actor must already have at least one build under its default build tag (usually latest), built from a Git repository version whose URL points to this repository. Build it once manually on the platform before CI takes over. See Repository check.

2. Create the config file

Every repo must have an apify-test-tools.config.json file at the root that lists its Actors. See Config file below.

3. Set up Actor folders

Each Actor in the config must have a .actor/actor.json file. The dockerContextDir field in actor.json defines the build context boundary — this is what the tool uses to decide which files can affect the Actor's build (see Change detection).

This example has two Actors that share the code at the repo root, and one standalone Actor with its own code:

my-repo
├── apify-test-tools.config.json
├── Dockerfile                      <- shared by web-scraper and email-sender
├── package.json
├── src/                            <- shared by web-scraper and email-sender
├── actors
│   ├── web-scraper
│   │   └── .actor
│   │       └── actor.json          <- points to the repo root
│   └── email-sender
│       └── .actor
│           └── actor.json          <- points to the repo root
├── standalone-actors
│   └── orchestrator
│       ├── .actor
│       │   └── actor.json          <- default context: its own folder
│       ├── Dockerfile
│       ├── package.json
│       └── src/
└── test
    └── platform
        ├── web-scraper.test.ts
        ├── email-sender.test.ts
        └── orchestrator
            └── orchestrator.test.ts

The Actors that share code point their dockerfile and dockerContextDir at the repo root. Both paths are relative to the .actor folder:

{
    "actorSpecification": 1,
    "name": "web-scraper",
    "version": "0.1",
    "dockerfile": "../../../Dockerfile",
    "dockerContextDir": "../../.."
}

A change to the shared src/ then builds and tests both of them, while a change in standalone-actors/orchestrator counts only for the orchestrator, even though it lies inside the others' context. The orchestrator leaves dockerContextDir out, so its context is its own folder.

For a single-Actor repo, set "folder": "." in the config and place .actor/actor.json at the repo root.

4. Write tests

Put platform tests in test/platform. How you organize them inside is up to you. See Writing tests below.

5. Set up CI

On GitHub, add the GitHub workflows below. On another CI platform, call the CLI from your own pipeline; the workflows are a good template for the steps.

Config file

apify-test-tools.config.json lives at the repo root and tells the tool which Actors live in the repo, where they are, and which token builds them. Actors are organized into groups, and each group sets the token once for all of its Actors:

{
    "mode": "grouped",
    "groups": {
        "myteam": {
            "tokenEnvVar": "APIFY_TOKEN_MYTEAM",
            "actors": [
                { "folder": "actors/web-scraper", "actorFullName": "myteam/web-scraper" },
                { "folder": "actors/email-sender", "actorFullName": "myteam/email-sender" },
                {
                    "folder": "standalone-actors/orchestrator",
                    "actorFullName": "myteam/orchestrator",
                    "overrideActorContext": ["standalone-actors/orchestrator", "test/platform/orchestrator"]
                }
            ]
        }
    }
}

The group key (myteam above) is only a label for readability; it doesn't need to match any folder or username. Each group needs a tokenEnvVar and at least one Actor. Add another group for Actors that need a different token, e.g. Actors owned by another account.

The orchestrator sets overrideActorContext because its tests live outside its own folder: without it, a change to only test/platform/orchestrator wouldn't build and test it. The other two already have the whole repo, tests included, as their context.

Field	Where	Required	Description
folder	Actor	yes	Relative path from repo root to the Actor's own project directory — the folder that directly contains .actor/actor.json (i.e. <folder>/.actor/actor.json), the Actor's README/CHANGELOG, and its source. Use "." for a single-Actor repo where .actor/ is at the root.
actorFullName	Actor	yes	Full Actor identifier in owner/name format (e.g. "apify/web-scraper"). This is the source of truth for the Actor name — the name field in actor.json is not used.
tokenEnvVar	group or Actor	yes	Name of the environment variable holding the Apify API token that builds the group's Actors. No fallback — if the env var is not set at build time, the build fails.
overrideActorContext	group or Actor	no	Array of paths (relative to repo root) that define which files are relevant to the Actor. When set, replaces the dockerContextDir from actor.json for change detection. Useful when an Actor depends on shared packages outside its Docker build context. Entries must not be prefixes of one another (e.g. ["", "code"] or ["actors", "actors/foo"] are rejected). The Actor's own folder is always part of its context — if none of the listed entries reach it, it's added automatically.

A value set on an Actor replaces the group's for that Actor only. Values are replaced, not merged: if the myteam group also set overrideActorContext, the orchestrator would still get only the two paths it lists.

No two Actors may share a folder, and no two entries may point at the same actorFullName.

GitHub workflows

The workflows take every change straight to production, with tests in between:

• Every push to a PR builds the Actors the PR changes into a test version (0.99) that users never run, and runs their platform tests against those builds. Only the Actors whose files changed are built and tested, and a later push only re-tests what changed since the last green run.
• Every merge to master builds the changed Actors into their default version (latest), which is what users run, and announces the release on Slack. There is no separate release step.
• Scheduled runs (optional) run the tests against the released builds to catch what breaks without a code change, like a website changing its layout.

These are GitHub Actions workflows. On another CI platform, build the same flow with the CLI.

Add these files to .github/workflows in your repo. They call the reusable workflows in this repo, pinned at the @workflows-v0 major tag — never at @master. Every input, the optional Claude workflows, and how the tag moves are in the Workflow API.

pr-build-devel-test.yaml

name: PR Test

on:
    pull_request:
        branches: [master]

jobs:
    buildDevelAndTest:
        uses: apify/apify-test-tools/.github/workflows/public_pr-build-test.yaml@workflows-v0
        secrets:
            TESTER_APIFY_TOKEN: ${{ secrets.TESTER_APIFY_TOKEN }}
            # One entry per secret apify-test-tools.config.json names, today each `tokenEnvVar`
            CONFIG_SECRETS: |
                {
                    "APIFY_TOKEN_MYTEAM": "${{ secrets.APIFY_TOKEN_MYTEAM }}",
                    "APIFY_TOKEN_PARTNER": "${{ secrets.APIFY_TOKEN_PARTNER }}"
                }

release-latest.yaml

name: Release latest

on:
    push:
        branches: [master]

jobs:
    buildLatest:
        uses: apify/apify-test-tools/.github/workflows/public_push-build-latest.yaml@workflows-v0
        secrets:
            SLACK_TOKEN_RELEASES_BOT: ${{ secrets.SLACK_TOKEN_RELEASES_BOT }}
            # One entry per secret apify-test-tools.config.json names, today each `tokenEnvVar`
            CONFIG_SECRETS: |
                {
                    "APIFY_TOKEN_MYTEAM": "${{ secrets.APIFY_TOKEN_MYTEAM }}",
                    "APIFY_TOKEN_PARTNER": "${{ secrets.APIFY_TOKEN_PARTNER }}"
                }

Scheduled tests (optional)

How often to run which tests is up to you. For example, you can run every test once a day:

name: Platform tests - Daily

on:
    schedule:
        # Runs at 00:00 UTC every day
        - cron: '0 0 * * *'
    workflow_dispatch:

jobs:
    platformTestsDaily:
        uses: apify/apify-test-tools/.github/workflows/public_platform-tests.yaml@workflows-v0
        secrets:
            TESTER_APIFY_TOKEN: ${{ secrets.TESTER_APIFY_TOKEN }}
            SLACK_TOKEN_TESTS_BOT: ${{ secrets.SLACK_TOKEN_TESTS_BOT }}

and a few critical tests, kept in e.g. test/platform/hourly, every hour:

name: Platform tests - Hourly

on:
    schedule:
        # Runs at the start of every hour
        - cron: '0 * * * *'
    workflow_dispatch:

jobs:
    platformTestsHourly:
        uses: apify/apify-test-tools/.github/workflows/public_platform-tests.yaml@workflows-v0
        with:
            test-files-glob: hourly
        secrets:
            TESTER_APIFY_TOKEN: ${{ secrets.TESTER_APIFY_TOKEN }}
            SLACK_TOKEN_TESTS_BOT: ${{ secrets.SLACK_TOKEN_TESTS_BOT }}

Secrets

Set these as repository or organization secrets and pass them in each job's secrets: block, as in the examples above. The Actor tokens go packed in one JSON object, CONFIG_SECRETS, because their names differ per repo:

Secret	Needed for
the Actor tokens named by tokenEnvVar in apify-test-tools.config.json	building Actors and deleting their old builds
TESTER_APIFY_TOKEN	running the platform tests
SLACK_TOKEN_RELEASES_BOT	announcing releases on Slack
SLACK_TOKEN_TESTS_BOT	reporting scheduled test results to Slack
NPM_TOKEN	optional, installing private npm packages. Use a read-only token

Repos in the apify org can pass secrets: inherit instead. Each secret only reaches the step that needs it. See Secrets for both, and for passing CONFIG_SECRETS without listing the tokens.

Writing tests

Platform tests use describe and testActor from apify-test-tools instead of the ones from vitest. testActor registers a test that only runs when that Actor was built for it (or when you ask for all tests), and gives the callback an extended expect and a run function that starts the Actor.

import { describe, testActor } from 'apify-test-tools';

describe('web-scraper', () => {
    testActor('myteam/web-scraper', 'scrapes the start URL', async ({ expect, run }) => {
        const runResult = await run({ input: { startUrls: [{ url: 'https://apify.com' }] } });

        // Status SUCCEEDED, a sane duration, no failed requests, no errors in the log, and the item count
        await expect(runResult).toFinishWith({ datasetItemCount: { min: 1, max: 10 } });

        const { items } = await runResult.getDataset();
        expect(items[0].title, 'Page title').toBeNonEmptyString();
    });
});

Platform tests are skipped unless TESTER_APIFY_TOKEN is set, so your regular unit test run stays unaffected.

To run them locally against the Actors' current default builds:

RUN_ALL_PLATFORM_TESTS=1 TESTER_APIFY_TOKEN=<token> npx vitest --run test/platform

The Vitest library API covers everything toFinishWith checks, the other matchers, standby Actors, reading run data, and running tests locally against builds of your own changes.