# Last passing run per environment
Source: https://empirical.run/docs/api-reference/analytics/last-passing-run-per-environment

/api-reference/openapi.generated.json post /api/v2/analytics/test-case-last-pass-per-env
Returns the most recent passing run for a test case, grouped by environment.



# Test-case analytics
Source: https://empirical.run/docs/api-reference/analytics/test-case-analytics

/api-reference/openapi.generated.json get /api/v2/analytics/test-cases
Returns aggregated test-case metrics for the scoped project (without per-test history).



# Test-case execution history (batch)
Source: https://empirical.run/docs/api-reference/analytics/test-case-execution-history-batch

/api-reference/openapi.generated.json post /api/v2/analytics/test-case-history/batch
Returns per-test-case execution history for the requested test cases, plus echoed query metadata.



# Test-count analytics
Source: https://empirical.run/docs/api-reference/analytics/test-count-analytics

/api-reference/openapi.generated.json get /api/v2/analytics/test-count
Returns the number of distinct test cases run per day for the scoped project.



# Test-run analytics
Source: https://empirical.run/docs/api-reference/analytics/test-run-analytics

/api-reference/openapi.generated.json get /api/v2/analytics/test-runs
Returns daily test-run statistics and a period summary for the scoped project.

Analytics uses path-based versioning; the current recommended version is **v2** (`/api/v2/analytics/`). The **v1** endpoints (`/api/v1/analytics/*`) are deprecated but remain available for backward compatibility. In v2, the v1 `test-cases` endpoint is split: `GET /api/v2/analytics/test-cases` returns metrics only, and `POST /api/v2/analytics/test-case-history/batch` fetches history for one or many test cases in a single request.

Percentage fields (`pass_rate`, `fail_rate`, `flaky_rate`) are integers in the **0-100** range, not decimals. All endpoints support date filtering via `days` (1-90), or a custom `start_date`/`end_date` range (ISO 8601).


# Get an environment status badge
Source: https://empirical.run/docs/api-reference/badges/get-an-environment-status-badge

/api-reference/openapi.generated.json get /api/badges/projects/{project_slug}/environments/{environment_slug}
Renders an SVG status badge (passing/failing/running/etc.) for the latest test run of a project environment. Served unauthenticated for embedding in READMEs.



# Create an environment
Source: https://empirical.run/docs/api-reference/environments/create-an-environment

/api-reference/openapi.generated.json post /api/environments
Creates an environment in the scoped project. Rejected for projects managed by environments.yaml.



# Delete an environment
Source: https://empirical.run/docs/api-reference/environments/delete-an-environment

/api-reference/openapi.generated.json delete /api/environments/{id}
Deletes an environment by ID and prunes environment-specific subscriptions. The project is derived from the entity.



# List environments
Source: https://empirical.run/docs/api-reference/environments/list-environments

/api-reference/openapi.generated.json get /api/environments/list
Returns the scoped project's environments, optionally filtered by slug and disabled state. The run worker uses this to resolve env vars by slug.



# Sync environments from YAML
Source: https://empirical.run/docs/api-reference/environments/sync-environments-from-yaml

/api-reference/openapi.generated.json post /api/environments/sync
Upserts the scoped project's environments from its environments.yaml file and flags the project as YAML-managed.



# Update an environment
Source: https://empirical.run/docs/api-reference/environments/update-an-environment

/api-reference/openapi.generated.json patch /api/environments/{id}
Updates an environment by ID. The project is derived from the entity. Returns the bare updated row (no `data` envelope), matching the dashboard settings UI it serves.



# Attach a failing test to a failure group
Source: https://empirical.run/docs/api-reference/failure-groups/attach-a-failing-test-to-a-failure-group

/api-reference/openapi.generated.json post /api/failure-groups/{id}/members
Records that a run's failing test belongs to this group, refreshing the group's last-seen rollups, and returns the derived member entry. Idempotent: re-attaching an existing member refreshes its sighting (the entry's sighting_count and last_seen reflect it). An agent attach will not resurrect a membership a human detached.



# Create a failure group
Source: https://empirical.run/docs/api-reference/failure-groups/create-a-failure-group

/api-reference/openapi.generated.json post /api/failure-groups
Creates a durable failure group for a new failure mode. Labels are unique per project: if the label already exists the request fails with 409 and returns the existing group unchanged — inspect it (note its status) and either reuse it via PATCH/attach or create under a different label.



# Delete a failure group
Source: https://empirical.run/docs/api-reference/failure-groups/delete-a-failure-group

/api-reference/openapi.generated.json delete /api/failure-groups/{id}
Deletes a failure group and its members. Its edit-history revisions survive.



# Get a failure group
Source: https://empirical.run/docs/api-reference/failure-groups/get-a-failure-group

/api-reference/openapi.generated.json get /api/failure-groups/{id}
Retrieves a single failure group. Its members are served by `GET /api/failure-groups/{id}/members`.



# Get a failure group's activity
Source: https://empirical.run/docs/api-reference/failure-groups/get-a-failure-groups-activity

/api-reference/openapi.generated.json get /api/failure-groups/{id}/activity
Returns the group's activity timeline (newest first): creation, authored edits, status changes, merges, member joins, repeat sightings, and removals. Derived from the revision trail and the membership-event log.



# List a failure group's members
Source: https://empirical.run/docs/api-reference/failure-groups/list-a-failure-groups-members

/api-reference/openapi.generated.json get /api/failure-groups/{id}/members
Returns the group's members (one entry per test, newest sighting first), derived from the membership-event log: state, provenance of the latest attach, first/last sighting, sighting count and environments.



# List a run's failure-group members
Source: https://empirical.run/docs/api-reference/failure-groups/list-a-runs-failure-group-members

/api-reference/openapi.generated.json get /api/failure-groups/members
Returns one entry per failed test in the run that is attached to a failure group, with the group's content hydrated. Read from the canonical membership-event log; when several sources attached the same test, human wins over agent over machine. Dedupe on group_id for the run's distinct groups.



# List failure groups
Source: https://empirical.run/docs/api-reference/failure-groups/list-failure-groups

/api-reference/openapi.generated.json get /api/failure-groups
Returns a paginated list of failure groups for the scoped project, newest sighting first. Use this to match a failure against an existing group before creating a new one.



# Merge a failure group into another
Source: https://empirical.run/docs/api-reference/failure-groups/merge-a-failure-group-into-another

/api-reference/openapi.generated.json post /api/failure-groups/{id}/merge
Moves this group's members into the target, unions fingerprints, and deletes this group. Returns the target.



# Update a failure group
Source: https://empirical.run/docs/api-reference/failure-groups/update-a-failure-group

/api-reference/openapi.generated.json patch /api/failure-groups/{id}
Updates a group's authored fields. `fingerprint`, when present, replaces the stored one; omit it to keep it. Latest write wins. Renaming to a label another group holds fails with 409 and returns that group — if the two are the same failure mode, merge them instead.



# Introduction
Source: https://empirical.run/docs/api-reference/introduction

Base URL and authentication for the Empirical API

The Empirical API lets you trigger and inspect test runs, manage snoozes, browse test cases and environments, and pull analytics.

## Base URL

All endpoints are served from:

```
https://api.empirical.run
```

## Authentication

Generate a project API key under [Settings → API keys](https://empirical.run/settings/api-keys), then send it as a Bearer token:

```bash theme={null}
curl https://api.empirical.run/api/test-runs \
  -H "Authorization: Bearer $EMPIRICALRUN_KEY"
```

Project API keys are scoped to their project. Keep them secret and use a separate key for each automation context so it can be rotated independently.

If you are signed in with the [Empirical CLI](/docs/cli), you can call the same endpoints without handling a key locally:

```sh theme={null}
empirical api api/test-runs --project <project-slug>
```

## Versioning

Endpoint paths may be unversioned or include a version such as `/api/v2/`. Use the path shown in this reference rather than adding a version yourself.

Deprecated endpoints are described in [openapi.deprecated.json](https://api.empirical.run/openapi.deprecated.json), including their replacements where available.

The current public API is available as an [OpenAPI specification](https://api.empirical.run/openapi.json).


# Create a resource
Source: https://empirical.run/docs/api-reference/resources/create-a-resource

/api-reference/openapi.generated.json post /api/resources
Creates a resource. With `url`, creates a link-type file; otherwise creates a pending upload and returns a signed `upload_url` to stream the file into R2.



# Delete a resource
Source: https://empirical.run/docs/api-reference/resources/delete-a-resource

/api-reference/openapi.generated.json delete /api/resources/{id}
Deletes a resource, removing its R2 object and stopping any active Drive watch.



# Download resource content
Source: https://empirical.run/docs/api-reference/resources/download-resource-content

/api-reference/openapi.generated.json get /api/resources/{id}/content
Streams an uploaded resource's content from R2. Authorized by the signed capability URL (HMAC in the query string), not bearer auth.



# Get a resource
Source: https://empirical.run/docs/api-reference/resources/get-a-resource

/api-reference/openapi.generated.json get /api/resources/{id}
Retrieves a single resource by ID, with a fresh download URL.



# List resources
Source: https://empirical.run/docs/api-reference/resources/list-resources

/api-reference/openapi.generated.json get /api/resources
Returns a paginated list of uploaded resources for the project.



# Update a resource
Source: https://empirical.run/docs/api-reference/resources/update-a-resource

/api-reference/openapi.generated.json patch /api/resources/{id}
Updates a resource's name/description, or confirms a pending upload with `status: "uploaded"`.



# Upload resource content
Source: https://empirical.run/docs/api-reference/resources/upload-resource-content

/api-reference/openapi.generated.json put /api/resources/{id}/content
Streams the request body into R2 for a pending resource. Authorized by the signed capability URL (HMAC in the query string), not bearer auth.



# Add tests to a snooze
Source: https://empirical.run/docs/api-reference/snoozes/add-tests-to-a-snooze

/api-reference/openapi.generated.json post /api/snoozes/{id}/add-test-cases
Adds Playwright tests to an existing snooze.



# Create a snooze
Source: https://empirical.run/docs/api-reference/snoozes/create-a-snooze

/api-reference/openapi.generated.json post /api/snoozes
Creates a snooze to temporarily suppress failures for the given Playwright tests until `snooze_until`.



# Delete a snooze
Source: https://empirical.run/docs/api-reference/snoozes/delete-a-snooze

/api-reference/openapi.generated.json delete /api/snoozes/{id}
Permanently deletes a snooze.



# Get a snooze
Source: https://empirical.run/docs/api-reference/snoozes/get-a-snooze

/api-reference/openapi.generated.json get /api/snoozes/{id}
Retrieves a single snooze by ID.



# List snoozes
Source: https://empirical.run/docs/api-reference/snoozes/list-snoozes

/api-reference/openapi.generated.json get /api/snoozes
Returns a paginated list of snoozes for the scoped project, with optional filtering by environment and status.



# Update a snooze
Source: https://empirical.run/docs/api-reference/snoozes/update-a-snooze

/api-reference/openapi.generated.json patch /api/snoozes/{id}
Updates a snooze — extend it, change its tests/description, or expire it immediately with `expire_now`.



# Get a test case
Source: https://empirical.run/docs/api-reference/test-cases/get-a-test-case

/api-reference/openapi.generated.json get /api/v2/test-cases/{id}
Retrieves a single test case by its Playwright test ID. Access is derived from the test case's project, returning 403 when the caller cannot access it.



# List test case tags
Source: https://empirical.run/docs/api-reference/test-cases/list-test-case-tags

/api-reference/openapi.generated.json get /api/v2/test-cases/tags
Returns the distinct set of tags across the scoped project's active test cases (those present at the last synced commit; all synced test cases if the project has never synced). The response is a bare `{ tags }` object (no `data` envelope) and is cached for 5 minutes via a `Cache-Control` header.



# List test cases
Source: https://empirical.run/docs/api-reference/test-cases/list-test-cases

/api-reference/openapi.generated.json get /api/v2/test-cases
Returns a paginated list of the scoped project's test cases with optional filtering, along with the project's latest test-cases sync error.



# Update test case tags
Source: https://empirical.run/docs/api-reference/test-cases/update-test-case-tags

/api-reference/openapi.generated.json patch /api/v2/test-cases/{id}
Replaces the tags on a test case and attempts to update the source file. Access is derived from the test case's project, returning 403 when the caller cannot access it.



# Cancel a test run
Source: https://empirical.run/docs/api-reference/test-runs/cancel-a-test-run

/api-reference/openapi.generated.json post /api/test-runs/{id}/cancel
Cancels an in-progress test run, recording an optional reason and the cancelling user. Returns 412 when the run is already in a terminal state.



# Get a test run
Source: https://empirical.run/docs/api-reference/test-runs/get-a-test-run

/api-reference/openapi.generated.json get /api/test-runs/{id}
Retrieves a single test run by ID, including its project and per-test summary details. Access is derived from the run's project; inaccessible or unknown runs return 404.



# Get test run metadata
Source: https://empirical.run/docs/api-reference/test-runs/get-test-run-metadata

/api-reference/openapi.generated.json get /api/v2/test-runs/{id}
Returns Postgres-backed metadata for a test run. This endpoint does not read the run ledger or compose per-test results.



# List test run artifacts
Source: https://empirical.run/docs/api-reference/test-runs/list-test-run-artifacts

/api-reference/openapi.generated.json get /api/test-runs/{id}/artifacts
Lists downloadable artifacts (traces, reports, screenshots, etc.) stored for a test run. Provide `key` for an exact lookup, or `prefix`/`suffix` to filter the listing. Returns an empty list when nothing matches.



# List test runs
Source: https://empirical.run/docs/api-reference/test-runs/list-test-runs

/api-reference/openapi.generated.json get /api/test-runs
Returns a paginated list of test runs for the scoped project, with optional filtering by id, branch, state, result, environment, and time window.

Test runs transition through these states:

| State | Description |
| - | - |
| `pending` | Run is queued behind other runs (concurrency limit reached) |
| `queued` | Run is queued and will start soon |
| `started` | Run is currently executing |
| `ended` | Run finished executing; inspect its result counts to determine whether tests passed |
| `error` | Run failed with an error |
| `cancelling` | Run is being cancelled |
| `cancelled` | Run was cancelled |

A failed run may carry one of these error codes:

| Error code | Description |
| - | - |
| `branch_missing` | Explicitly requested test branch does not exist |
| `provider_error` | Test-code provider failed during branch resolution |
| `merge_conflict` | Merge conflict detected when merging the test branch |
| `dispatch_failed` | Failed to dispatch the test run to workers |
| `report_not_generated` | Test report was not generated |
| `run_failed_with_error` | Test run failed with an error during execution |
| `worker_interrupted` | Test run worker was interrupted |
| `run_timed_out` | Test run exceeded the maximum duration |
| `merge_reports_failed` | Failed to merge sharded reports |
| `empty_selection` | The run's selection (tags or test case ids) matched no tests |


# Re-run a test run
Source: https://empirical.run/docs/api-reference/test-runs/re-run-a-test-run

/api-reference/openapi.generated.json post /api/test-runs/{id}/re-run
Re-triggers a test run from a source run's branch, build, and overrides. Optionally re-runs only the failed tests of a completed run.



# Skip a test run
Source: https://empirical.run/docs/api-reference/test-runs/skip-a-test-run

/api-reference/openapi.generated.json post /api/v1/test-runs/skip
Marks the Empirical check run on a commit as skipped. Callers authenticate with a project API key.



# Trigger a test run
Source: https://empirical.run/docs/api-reference/test-runs/trigger-a-test-run

/api-reference/openapi.generated.json put /api/test-runs
Triggers a new end-to-end test run for the scoped project against the given environment, resolving the branch/build and dispatching the run.



# CLI
Source: https://empirical.run/docs/cli

Install and configure the Empirical command line tool

The Empirical CLI lets you - or your local coding agents - work with Empirical from the terminal. The CLI also bundles an agent skill that can be configured for common coding agents (Claude Code and Codex).

## Install

In your terminal, run:

```sh theme={null}
# Install the standalone binary
curl -fsSL https://cli.empirical.run/install | sh

# Check installed version
empirical version

# Authenticate as yourself
empirical login
```

The installer adds `~/.empirical/bin` to your shell `PATH`. If `empirical` is not available in a new terminal, add it manually:

```sh theme={null}
export PATH="$HOME/.empirical/bin:$PATH"
```

Verify the login and see the projects available to you:

```sh theme={null}
empirical whoami
empirical api api/projects
```

## Agent skill

The Empirical skill teaches compatible local coding agents how to use the `empirical` command.

The installer suggests this as a next step after install. Run it with exactly one of `--global` or `--project`:

```sh theme={null}
# Install in the global agent directories (Claude Code, plus Codex and other agents)
empirical skill install --global

# Install in your project at .agents/skills
empirical skill install --project
```

Run `empirical skill install --help` for all options.

The `SKILL.md` file is managed by the CLI: installs and upgrades overwrite it, so don't edit it directly. To customize the skill for your project, add a `CUSTOM.md` file in the same directory — the skill instructs agents to read it, and the CLI never touches it.

## Commands reference

### User auth

* `empirical login` authorizes the CLI with Empirical.
* `empirical logout` clears local CLI authorization.
* `empirical whoami` prints the current authenticated account.

### Version and upgrade

* `empirical version` prints the installed CLI version.
* `empirical upgrade` upgrades to the latest published CLI version.

### Agent skill

* `empirical skill install` installs the Empirical `SKILL.md` for local coding agents, overwriting any existing copy. Requires exactly one of `--global` or `--project`.

### Sessions

* `empirical session "<prompt>"` creates a session and sends a prompt.
* `empirical session --id <id> "<prompt>"` sends a prompt to an existing session.
* `empirical session -x "<prompt>"` creates a session, waits for the agent response, and prints only the final assistant text.
* `empirical session -x --id <id> "<prompt>"` sends to an existing session, waits for the agent response, and prints only the final assistant text.
* `echo "<prompt>" | empirical session --execute --id <id>` reads the prompt from stdin.
* `empirical session list` lists recent sessions.
* `empirical session status <id>` shows the current agent and sandbox state (with the error reason when the sandbox failed), queued messages, latest assistant response, branches, and pull requests.
* `empirical session logs <id>` prints the session history in chronological order with RFC 3339 UTC timestamps. Add `--since <time>` and `--before <time>` to select an interval; both accept relative durations or RFC 3339 timestamps. Add `--json` for raw NDJSON.
* `empirical session logs <id> -f` prints the history, then follows live events.
* `empirical session logs <id> -f --since 0s --until idle --timeout 300` waits until the agent is idle and no messages remain queued.
* Add `--project <slug>` only when you need to choose a specific project.

### Git

* `empirical git setup` configures git to authenticate to `git.empirical.run` with your CLI login, via a credential helper scoped to that host. Run it once per machine.
* `empirical clone [directory]` clones a project's test repository from `git.empirical.run`, running the setup above first if needed. Without a directory, git creates `./<project-slug>`.
* Add `--project <slug>` only when you need to choose a specific project; with several projects the error lists the available slugs.
* After setup, plain `git clone https://git.empirical.run/<project-slug>.git` works too.
* With your login you can fetch and push; pushes to the project's default branch are rejected. Project API keys are read-only. Without the CLI (for example in CI), use a project API key as the HTTP password.

### API

* `empirical api <endpoint>` calls any Empirical API endpoint using your CLI login token.
* Use `--project <slug>` to select a project without supplying an `x-project-slug` header. Do not combine it with `x-project-id` or `x-project-slug` headers.
* Add repeatable `-q, --query <key=value>` options for URL query parameters. Values are URL-encoded automatically, and repeated keys append to any parameters already in the endpoint. Query parameters alone keep the request method as GET.
* Example: `empirical api api/slack/threads --project flash -q entity_type=test_run -q entity_id=131825`.
* Use `--jq <expression>` to filter JSON output (requires jq). The former `-q` shorthand for jq filtering now means `--query`; update existing commands to use `--jq`.
* `empirical api api/endpoints` lists the endpoints available to your authentication method.

## Telemetry

The CLI collects usage telemetry to help improve the product. Command arguments and flag values are never collected. To opt out, set the [`DO_NOT_TRACK`](https://consoledonottrack.com/) environment variable.


# Analytics
Source: https://empirical.run/docs/features/analytics

Track test health and performance over time

[Analytics](https://empirical.run/analytics) page provides insights into your test suite's
health, including pass rates, flaky rates, and historical trends.

<img alt="Analytics" />

## Search Syntax

Filter test cases using the search bar with these patterns:

### Text Search

| Syntax | Example | Description |
| - | - | - |
| Term | `login` | Matches name or file path containing "login" |
| Multiple terms | `login auth` | Must contain both terms (AND) |
| Quoted phrase | `"user login"` | Treats multi-word text as a single term |
| Negation | `-admin` | Excludes tests matching "admin" |
| Negated phrase | `-"slow test"` | Excludes tests matching "slow test" |
| Combined | `checkout -guest` | Contains "checkout", excludes "guest" |

### Field Filters

| Syntax | Example | Description |
| - | - | - |
| `project:` | `project:chromium` | Filter by Playwright project name |
| `tag:` | `tag:critical` | Filter by tag (multiple tags use OR) |
| `tag:__none__` | `tag:__none__` | Filter tests with no tags |
| `flaky_rate:` | `flaky_rate:>10` | Filter by flaky rate percentage |
| `fail_rate:` | `fail_rate:>=25` | Filter by fail rate percentage |
| `flaky_count:` | `flaky_count:>2` | Filter by flaky count (absolute number) |
| `fail_count:` | `fail_count:>=5` | Filter by fail count (absolute number) |

#### Metric Filter Operators

For `flaky_rate`, `fail_rate`, `flaky_count`, and `fail_count`, you can use `>`, `>=`, `<`, `<=`, `=`. If no operator is specified, `>=` is used by default.

### Examples

| Query | Description |
| - | - |
| `login flaky_rate:>10` | Login tests with flaky rate above 10% |
| `tag:smoke tag:critical` | Tests with either smoke OR critical tag |
| `tag:__none__` | Tests that have no tags assigned |
| `project:firefox -skip` | Firefox project tests, excluding "skip" |
| `fail_rate:>=20 -tag:known-flaky` | High failure tests, excluding known flaky |

### Using Search Syntax via the API

The same search syntax can be used with the [Test Cases Analytics API](/docs/api-reference/analytics/test-case-analytics) by passing the query as the `search` parameter. Remember to URL-encode the value.

```bash theme={null}
curl -X GET \
  "https://api.empirical.run/api/v2/analytics/test-cases?search=login%20flaky_rate%3A%3E10%20-tag%3Aknown-flaky" \
  -H "Authorization: Bearer $EMPIRICALRUN_KEY"
```

The example above is equivalent to the search query `login flaky_rate:>10 -tag:known-flaky` — login tests with a flaky rate above 10%, excluding any tagged `known-flaky`.


# Branching
Source: https://empirical.run/docs/features/branching

Align test changes with application feature branches

Tests written by Empirical are saved in a git repo, and by default, all test runs
use test code from the default branch of the repo (usually called "main"). However, this is
sub-optimal in two scenarios.

### Feature branches

If feature changes introduce UI changes, tests need to be updated
for that particular branch. Test runs from other branches will fail, because they don't have
corresponding UI changes.

<img alt="Feature testing - without branching" />

Empirical can select an existing branch in the test repository for each run. This keeps test
changes aligned with feature branches in your application repository without making branch
selection a project-wide setting.

## Get started

### Feature branches

These are short-lived branches that represent feature branches with preview deployments. These
branches are expected to be merged into the default branch.

<img alt="Feature testing - with branching" />

### Configure default branch name

For new projects, the default branch is `main`. You can change it under **Settings → General**.

Choose the branch where new test changes should land. For example, if features are merged to a `staging`
branch in your app, it is a good idea for the Empirical default branch to be called `staging`.

### Select test code for a run

* For a pull request in your application repository, send a `pull_request` trigger. The run uses
  the test branch with the same name as the pull request's head branch, merged into the default
  branch. If no such test branch exists, the run uses the default branch. See
  [Pull request runs](#pull-request-runs).
* To request a particular existing test branch, pass `branch` when triggering the run. The branch
  runs as is, without a merge. A missing explicit branch produces an errored run; Empirical does
  not create it. `branch` takes precedence over a `pull_request` trigger.
* If you send neither, the run uses the project's default test-repository branch.
* Add the [merge branch step to your CI workflows](#merge-branch-workflow). This informs Empirical when a feature branch is merged.
  * Feature branches are automatically deleted after they are merged

<Accordion title="Lifecycle of feature branches">
  A durable feature branch must already exist in the test repository. Test-run triggers never create it.

  * Runs that pass `branch` use the test code on that branch.
  * When the feature is merged in your application repository, the [merge branch workflow](#merge-branch-workflow)
    merges the test branch into the default branch.
</Accordion>

### Pull request runs

A test run triggered for an application pull request tests the test code as it will look once
that pull request merges. Empirical merges the matching test branch into the default branch on
a temporary `test-run/merge/*` ref and runs the merge commit. It cleans these refs up after the
test branch is merged. If the test branch conflicts with the default branch, the run errors with
a merge conflict. Rebase the test branch, or pass `branch` to run it without a merge.

The [dispatch-action](/docs/triggers/ci-cd#github-actions) sends the trigger automatically for
`pull_request` workflow events. With the API, send `trigger` together with `build`:

```json theme={null}
{
  "environment": "preview",
  "build": { "repo": "acme/web", "branch": "feat-x", "commit": "<sha>" },
  "trigger": {
    "event": "pull_request",
    "pull_request": { "number": 123, "head": "feat-x", "base": "main" }
  }
}
```

## Merge branch workflow

To make branching work, Empirical needs to know when a branch in your repo gets merged and where. Empirical
will do a corresponding merge in the test repo.

Note that this continues to treat your app as a black-box: we do not need the contents of your repo or app source code. This
only captures a branch lifecycle event.

### GitHub Actions

Add this to `.github/workflows/merge-branches.yaml`. Which branch to merge and where is automatically
picked up from the `pull_request` event that triggers this workflow.

```yml theme={null}
name: Merge branches in Empirical

on:
  pull_request:
    types: [closed]

jobs:
  auto-merge:
    if: github.event.pull_request.merged == true
    runs-on: ubuntu-latest
    steps:
      - name: Merge branches in Empirical
        uses: empirical-run/merge-branches-action@v1
        with:
          auth-key: ${{ secrets.EMPIRICALRUN_KEY }}
```

The `v1` reference receives backwards-compatible fixes while keeping breaking
major releases opt-in. You can instead pin an exact release such as `v1.0.0`.

### Other CI/CD providers

For other CI/CD providers, you can use our API endpoint to merge branches. Note that you
need to pass the correct values for head and base branch. Head branch is what gets
merged into the base branch.

```
curl -X POST https://api.empirical.run/api/projects/merge-branches \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${{ secrets.EMPIRICALRUN_KEY }}" \
  -d '{
    "head_branch": "${{ REPLACE_WITH_VALUE }}",
    "base_branch": "${{ REPLACE_WITH_VALUE }}"
  }'
```


# Code Coverage
Source: https://empirical.run/docs/features/code-coverage

Measure code coverage for your test runs

Empirical runs Playwright tests that treat your application as a black box. This makes it harder to measure code
coverage for these tests.

We can combine some work on the app code with test side changes to fix this. Specifically, setting up code coverage requires three steps:

1. **App bundler changes** — instrument your app with Istanbul (requires app source code, done by app team)
2. **Test repo changes** — enable coverage collection in your test fixtures (can be done by Empirical agents)
3. **Build final report** — download coverage artifacts and generate an HTML report (requires app source code, done by app team)

## App repo changes

The example below is for Vite applications. Install the instrumentation plugin in your application repository:

```sh theme={null}
npm install --save-dev vite-plugin-istanbul
```

Add the plugin to your Vite config to instrument your source code with Istanbul coverage hooks. Other bundlers need equivalent Istanbul instrumentation.

If you are running tests against a deployed URL (not localhost), also add `forceBuildInstrument: true` to ensure instrumentation is
included in production builds.

```diff theme={null}
diff --git a/vite.config.ts b/vite.config.ts
index cfc922f..f2e8295 100644
--- a/vite.config.ts
+++ b/vite.config.ts
@@ -2,6 +2,7 @@ import { defineConfig } from "vite";
 import react from "@vitejs/plugin-react";
 import tailwindcss from "@tailwindcss/vite";
 import { tanstackRouter } from "@tanstack/router-plugin/vite";
+import istanbul from "vite-plugin-istanbul";
 
 function resolvePath(relativePath: string): string {
@@ -23,6 +24,12 @@ export default defineConfig({
     }),
     react(),
     tailwindcss(),
+    istanbul({
+      include: "src/*",
+      exclude: ["node_modules", "test/"],
+      extension: [".js", ".ts", ".tsx"],
+      requireEnv: false,
+      forceBuildInstrument: true,
+    }),
   ],
   resolve: {
     alias: {
```

## Tests repo changes

`@empiricalrun/playwright-utils` 0.43.0+ includes coverage collection utilities. In your test fixture file, enable
coverage by passing `{ collectCoverage: true }` to `baseTestFixture`.

```ts theme={null}
import { test as base, expect as baseExpect } from "@playwright/test";
import { baseTestFixture, extendExpect } from "@empiricalrun/playwright-utils/test";

export const test = baseTestFixture(base, { collectCoverage: true });
export const expect = extendExpect(baseExpect);
```

This automatically collects `window.__coverage__` data from every page before it closes and attaches
the Istanbul coverage JSON files to the test report. These files are uploaded like other artifacts (e.g. videos).

## Build final report

To build the final coverage report with `nyc`, we will first download the coverage JSON files from Empirical's
storage buckets, and then build the report.

First, fetch the test run details using the [Get Test Run](/docs/api-reference/test-runs/get-a-test-run) API, then download the
Playwright JSON report from the `summary_url` property.

```bash theme={null}
# Fetch the test run to get the report URL
REPORT_URL=$(curl -s "https://api.empirical.run/api/test-runs/<TEST_RUN_ID>" \
  -H "Authorization: Bearer $EMPIRICALRUN_KEY" \
  | jq -r '.data.test_run.testRun.summary_url')

# Download the report
curl -s "$REPORT_URL" -o report.json
```

Then, build a mapping of test case IDs to their coverage filenames:

```bash theme={null}
jq '
  [.suites[] | recurse(.suites[]?) | .specs[]? |
    {id: .id} + {
      coverage: [.tests[]? | .results[]? | .attachments[]?
        | select(.contentType == "application/json" and (.name | test("coverage")))
        | (.path | split("/") | last)
      ]
    }
  ] | map(select(.coverage | length > 0))
' report.json > coverage-map.json
```

Download all coverage files into `.nyc_output`:

```bash theme={null}
mkdir -p .nyc_output
jq -r '
  [.suites[] | recurse(.suites[]?) | .specs[]? | .tests[]? | .results[]? | .attachments[]?
   | select(.contentType == "application/json" and (.name | test("coverage"))) | .path
  ] | .[]
' report.json | while read -r url; do
  curl -s "$url" -o ".nyc_output/$(basename "$url")"
done
```

Finally, generate the HTML report using `nyc` from your app source directory:

```bash theme={null}
cd /path/to/your-app-code
npm install --save-dev nyc

npx nyc report --reporter=html \
               --temp-dir=/path/to/final-output-dir/.nyc_output \
               --report-dir=/path/to/final-output-dir
```


# Concurrency
Source: https://empirical.run/docs/features/concurrency-limits

Control what happens when a test run is triggered while another one is still going

Every test run belongs to a **concurrency group**. Runs in the same group (within a project)
execute one at a time. When a run is triggered and its group already has an active run, the
new run's `on_conflict` policy decides what happens.

Both are set on the trigger request, so each caller says what it needs: a CI pipeline can
cancel the previous run for the same branch, a nightly schedule can wait its turn, and a
one-off run from the dashboard can opt out.

## The `concurrency` object

```json theme={null}
{
  "environment": "preview",
  "build": { "branch": "feat-x", "repo": "acme/web" },
  "concurrency": {
    "group": "preview/acme/web@feat-x",
    "on_conflict": "cancel"
  }
}
```

| Field | What it means |
| - | - |
| `group` | Free-form key, scoped to the project. Runs with the same group run one at a time. Max 512 chars |
| `on_conflict` | What the new run does when the group already has an active run (below) |

Omit `concurrency` to use the project's default (see [Legacy project setting](#legacy-project-setting)).
Send `"concurrency": null` for no limit at all.

### `on_conflict`

| Value | What happens |
| - | - |
| `cancel` | The group's in-progress and waiting runs are cancelled and the new run starts immediately |
| `wait` | The new run enters **Pending** until the in-progress run finishes. If another `wait` run arrives, it replaces the older pending run, so at most one run waits per group |

The policy belongs to the new run. A `wait` run never cancels a running one; a `cancel` run
always does, even if the running one was itself triggered with `wait`.

Only runs created in the last 6 hours count as active, so a stuck run cannot block a group
forever.

### Choosing a group

| You want | Group | `on_conflict` |
| - | - | - |
| One run per PR; a new push supersedes the previous run | `preview/acme/web@<branch>` | `cancel` |
| One run per deployment target that cannot take two runs | the deployment URL | `wait` |
| Nightly runs on production never overlap | `production-nightly` | `wait` |
| Everything in the project strictly one at a time | any fixed string, e.g. `all` | `wait` |

Groups are plain strings compared exactly. Include whatever makes two runs "the same thing"
for you, and nothing else.

## Scheduled runs

A schedule carries its own block in `ENVIRONMENTS.yaml`:

```yaml theme={null}
environments:
  - slug: production
    name: Production
    scheduled_trigger:
      cron: "0 2 * * *"
      concurrency:
        group: production-nightly
        on_conflict: wait
```

A plain `scheduled_trigger: "0 2 * * *"` still works and uses the project default. Configure
scheduled concurrency in `ENVIRONMENTS.yaml` alongside the cron expression.

The change takes effect only after its pull request is merged into the test repository's default branch.

## What a run records

Every run stores the group and policy it was created with, so the decision stays accurate
even if settings change later. It comes back as `concurrency` on the run in the API and shows
under **Concurrency** in the run's details on the dashboard, where the group links to the list
of all runs in that group. `GET /api/test-runs?concurrency_group=<group>` filters the list.

```json theme={null}
"concurrency": { "group": "production-nightly", "on_conflict": "wait" }
```

`concurrency` is `null` when no limit applied.

Reruns reuse the source run's group and policy.

## Legacy project setting

Projects created before per-run concurrency have a project-level setting
(`test_run_concurrency_limit` and `test_run_concurrency_behavior`) that still applies to any
trigger that omits `concurrency`. It is **frozen**: it cannot be changed any more (the API
rejects writes and the dashboard no longer shows it), and it will be removed once callers set
their own blocks. To change how a project's runs are limited, set `concurrency` on the trigger
request or `scheduled_concurrency` on the environment. The frozen scope maps to a group as follows:

| Scope | Group |
| - | - |
| `none` | No group |
| `project` | The project slug |
| `environment` | `<environment slug>[/<repo>]` |
| `branch_environment` | `<environment slug>[/<repo>]@<build branch>` |

`/<repo>` is present when the run was triggered with a `build` that names a repository. For `branch_environment`, a run without `build.branch` has no group and is not limited.

The legacy behavior independently maps to the conflict policy:

| Behavior | `on_conflict` |
| - | - |
| `cancel_previous` | `cancel` |
| `queue_new` | `wait` |


# Email Testing
Source: https://empirical.run/docs/features/email-testing

Test email workflows like OTPs, invitations, and notifications

Empirical provides an `EmailClient` that lets you test email workflows in your
application. It generates unique email addresses and waits for incoming emails,
making it easy to test OTPs, invite flows, and notification emails.

All test repositories managed by Empirical have this built-in — no additional
setup is required.

## Usage

### Dynamic email

Generates a random email address that can be used for the test (e.g. invite a
new user).

```ts theme={null}
import { EmailClient } from "@empiricalrun/playwright-utils";
import { expect } from "@playwright/test";

const client = new EmailClient();
const address = client.getAddress();

// Input the `address` in the application
// that sends the email.

// Wait for the email to arrive
const email = await client.waitForEmail();
expect(
  email.links.find((l) => l.text === "Join your team")
).toBeTruthy();
```

### Static email

Uses a known (static) email address that can be used to login into an
application. The email id is appended with a managed domain to get the full
email address.

```ts theme={null}
import { EmailClient } from "@empiricalrun/playwright-utils";

const emailId = "test-login-user";

const client = new EmailClient({ emailId });
const address = client.getAddress(); // Returns full address with domain

// Wait for the email to arrive
const email = await client.waitForEmail();

// Get login OTP
const loginCode = email.codes[0];
```

### Filtering emails

You can filter emails by sender, subject, or body content. The `subject` and
`body` filters use substring matching, so you don't need to provide the full
value. The `from` filter requires the full email address.

```ts theme={null}
const email = await client.waitForEmail({
  from: "no-reply@example.com",
  subject: "Verify your email",
  body: "verification code",
});
```

## Email properties

The received email object has the following properties:

| Property | Type | Description |
| - | - | - |
| `subject` | `string` | Email subject line |
| `text` | `string` | Plain text body |
| `html` | `string` | HTML body |
| `links` | `{ text?: string; href?: string }[]` | Links found in the email |
| `codes` | `string[]` | Verification codes extracted from the email |

## Client options

| Option | Type | Default | Description |
| - | - | - | - |
| `emailId` | `string` | Random | A specific email id to use instead of generating a random one |
| `timeout` | `number` | `30000` | Timeout in milliseconds to wait for the email |

## Wait options

Pass these values to `waitForEmail` to select the message you need:

| Option | Type | Description |
| - | - | - |
| `from` | `string` | Exact sender email address |
| `subject` | `string` | Substring to match in the subject |
| `body` | `string` | Substring to match in the text or HTML body |
| `receivedAfter` | `Date` | Ignore messages received before this time; defaults to five seconds before the call |


# Slack Notifications
Source: https://empirical.run/docs/features/slack-notifications

Tag test owners in Slack when tests fail

When tests fail, you can tag owners in the Slack notification. Ownership rules are defined in a `.empiricalrun/OWNERS.yaml` file in your test repo.

## Setup

Create `.empiricalrun/OWNERS.yaml` in the root of your test repository.

## Format

```yaml theme={null}
rules:
  - match: { tag: "@payments" }
    owners:
      - slack: "U04ABCDEF"
```

Each rule has a `match` condition and a list of `owners` to mention.

## Matching

### By tag (exact match)

```yaml theme={null}
- match: { tag: "@payments" }
  owners:
    - slack: "U04ABCDEF"       # Alice
    - slack: "S07XYZ"          # #payments-team group
    - slack: "A08APPID"        # Slack app bot
```

### By file path (exact)

```yaml theme={null}
- match: { file: "tests/checkout.spec.ts" }
  owners:
    - slack: "U04ABCDEF"
```

### By directory (glob)

```yaml theme={null}
# All tests under tests/checkout/
- match: { file: "tests/checkout/**" }
  owners:
    - slack: "U04ABCDEF"
```

### By filename pattern (wildcard)

```yaml theme={null}
# All spec files directly in tests/api/
- match: { file: "tests/api/*.spec.ts" }
  owners:
    - slack: "U04ABCDEF"
```

<Note>`*` matches within a single directory, `**` matches across directories.</Note>

### All tests (catch-all)

```yaml theme={null}
- match: { file: "**" }
  owners:
    - slack: "U04ONCALL"
```

## Owner format

Use the Slack member ID, enterprise user ID, user group ID, or app ID directly:

* **User**: `U04ABCDEF` (find via Slack profile → Copy member ID)
* **Enterprise user**: `W04ABCDEF`
* **User group**: `S07XYZ` (find via Slack admin → User groups)
* **App**: `A08APPID` (find in the Slack app details URL)

The full mention syntax (`<@U04ABCDEF>`, `<!subteam^S07XYZ>`) is also supported.

App IDs mention the app's bot user, which must be in the channel.

## Behavior

* Rules are evaluated top-down; a test can match multiple rules
* All matched owners are mentioned (deduplicated)
* If `OWNERS.yaml` is missing or invalid, notifications are sent without mentions


# Webhook Testing
Source: https://empirical.run/docs/features/webhook-testing

Assert that your application sends the correct webhooks

Empirical lets you verify your application sends the correct webhooks. It
integrates with [webhook.site](https://webhook.site) to capture and query
webhook payloads, and includes a custom Playwright assertion for use in tests.

All test repositories managed by Empirical have this built-in — no additional
setup is required.

## Usage

### Create a webhook URL

Use `getWebhookUrl` to generate a unique webhook.site URL for your test.
Configure your application to send webhooks to this URL.

```ts theme={null}
import { getWebhookUrl } from "@empiricalrun/playwright-utils";

const webhookUrl = await getWebhookUrl();
// e.g. "https://webhook.site/9981f9f4-657a-4ebf-be7c-1915bedd4775"

// Optionally set expiry in seconds (default: 604800 = 7 days)
const webhookUrl = await getWebhookUrl({ expiry: 3600 }); // expires in 1 hour
const webhookUrl = await getWebhookUrl({ expiry: null }); // never expires
```

### Custom assertion

Use the `toHaveReceivedWebhook` assertion to verify that a webhook was received.
Pass any webhook.site URL to `expect()` — either one generated with `getWebhookUrl`
or an existing URL — and specify content to match against the body.

```ts theme={null}
import { getWebhookUrl } from "@empiricalrun/playwright-utils";
import { expect } from "@empiricalrun/playwright-utils/test";

// Works with a newly generated URL
const webhookUrl = await getWebhookUrl();
await expect(webhookUrl).toHaveReceivedWebhook({
  content: orderId,
});

// Or with an existing webhook.site URL
await expect("https://webhook.site/your-token-id").toHaveReceivedWebhook({
  content: [roomId, "recording.started"],
});
```

The assertion polls webhook.site until a matching webhook is found or the
timeout is reached. When multiple content strings are provided, all of them
must be present in the webhook body.

### Querying webhooks

Use `queryWebhookRequests` to fetch matching webhook requests directly. This
is useful when you need to inspect the payload for further assertions.

```ts theme={null}
import { getWebhookUrl, queryWebhookRequests } from "@empiricalrun/playwright-utils";

const webhookUrl = await getWebhookUrl();
const requests = await queryWebhookRequests(webhookUrl, [roomId, "recording.started"]);

// Access the parsed payload
const body = JSON.parse(requests[0].content);
expect(body.status).toBe("completed");
```

Each request in the returned array is a `WebhookRequest` with the following
properties:

| Property | Type | Description |
| - | - | - |
| `content` | `string` | Raw request body (parse with `JSON.parse` for JSON payloads) |
| `method` | `string` | HTTP method (e.g. `POST`) |
| `headers` | `Record<string, string[]>` | Request headers |
| `url` | `string` | Full request URL |
| `created_at` | `string` | Timestamp of when the request was received |

## Assertion options

| Option | Type | Default | Description |
| - | - | - | - |
| `content` | `string \| string[]` | Required | One or more terms to match against the webhook body. Multiple terms are AND'd together. |
| `timeout` | `number` | `300000` | Maximum time in milliseconds to wait for the webhook (default: 5 minutes) |


# Introduction
Source: https://empirical.run/docs/index

Understand how Empirical creates, runs, and maintains end-to-end tests

Empirical combines AI agents and managed test infrastructure to create and maintain end-to-end Playwright coverage for your product.

Your test suite lives in a git repository you can review and clone. Tests run against deployed environments such as staging or production, so Empirical can validate real user flows without access to your application source code.

## Start here

1. [Create a request](/docs/platform/requests) when you want new or updated coverage.
2. [Configure environments](/docs/platform/environments) for each deployment you test.
3. Trigger a run from the [dashboard, CLI, or API](/docs/triggers/ad-hoc), on a [schedule](/docs/triggers/scheduled), or after a [deployment](/docs/triggers/ci-cd).
4. Review the run report, artifacts, and history in the dashboard. Use [snoozes](/docs/platform/snoozes) to suppress a known failure temporarily.

You can also [clone the test repository](/docs/platform/repository), use the [CLI](/docs/cli) to work with Empirical agents, and connect [Slack](/docs/integrations/slack), [GitHub](/docs/integrations/github), [Linear](/docs/integrations/linear), or [Jira](/docs/integrations/jira).

## How work moves through Empirical

When you submit an open request, an agent session starts with your description. The agent works in the test repository and produces changes for review. Once those changes are merged, test runs execute them on Empirical's managed infrastructure. Empirical engineers monitor results, investigate failures, and keep the suite current as your product changes.

When a run finds a problem, you can inspect the report in the dashboard and receive notifications through configured reporters or Slack channels.

Watch the demo video for a full walkthrough:

<iframe title="YouTube video player" />


# GitHub Reporter
Source: https://empirical.run/docs/integrations/github

Install and configure our GitHub Reporter app

Use our GitHub Reporter app to see test run results in your GitHub: as comments on pull requests and
commit status checks.

### Pull request comments

Empirical posts a summary of test results as a comment on your pull requests.

<img alt="PR comments screenshot" />

### Status checks

Empirical reports the status of test runs in commit
[status checks](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/collaborating-on-repositories-with-code-quality-features/about-status-checks).

<img alt="Status checks screenshot" />

## Get started

Pre-requisites:

The GitHub Reporter requires test runs to be triggered from your CI/CD pipelines. Each test run trigger
needs to have commit and branch info, so that it can be used for reporting.

* If you are using our [GitHub Actions](/docs/triggers/ci-cd#github-actions), this is already done and there's no action required
* If you are using an [API call](/docs/triggers/ci-cd#metadata-for-reporting), make sure to send the `build` object in the request payload

To install the GitHub Reporter:

1. Navigate to the [Integrations](https://empirical.run/settings/integrations) page
2. Click "Install" on the GitHub app
3. Choose the repository where you want to see the reports (this will be the repo where test runs are triggered from, either using the GitHub Action or API calls)

## Permissions

The GitHub Reporter relies on the following two permissions only, and it does not need or have access to your app's source code.

| Scope | Purpose |
| - | - |
| Read and write access to checks | Create and update status checks in GitHub |
| Read and write access to pull requests | Create and update a comment for a pull request in GitHub |


# Jira
Source: https://empirical.run/docs/integrations/jira

Create coverage requests from Jira tickets

With our Jira integration, you can create coverage requests from your issue trackers.

## User flow

* In your Jira workspace, create a new ticket or use an existing ticket
  * The title and description are used to create a prompt, and should cover the user scenario that needs to be covered
  * Attachments are also supported: e.g., images in the ticket are also provided to the agent
* Add "empirical" as a label on the ticket
* This creates a [request](https://empirical.run/requests) in Empirical
  * A link to the request is sent as a comment on the Jira ticket
  * Empirical AI agents start working on the request right away
* When the request is marked **Resolved**, Empirical posts another Jira comment linking to the completed request

## Routing to projects

If your organization has one project, all Jira tickets are routed to it.

With multiple projects, map each Jira project key to a project under **Settings → Requests → Project associations**. Tickets are then routed by their Jira project.

## Get started

To install the Jira integration:

1. Navigate to the [Integrations](https://empirical.run/settings/integrations) page
2. Click "Connect" on the Jira integration
3. Once you accept the permissions, the integration is configured

## Permissions

The integration requests for the following permissions.

| Scope | Purpose |
| - | - |
| Manage jira-webhook | Create a webhook that receives Jira events when label is added |
| View and update jira-work | Get contents of the ticket, fetch attachments, post comments with status updates |
| View jira-user | Get details of the user who assigned the label |


# Linear
Source: https://empirical.run/docs/integrations/linear

Create coverage requests from Linear issues

With our Linear integration, you can create coverage requests directly from your issue tracker.

## User flow

* In your Linear workspace, create a new issue or use an existing one
  * The title and description are used to create a prompt, and should cover the user scenario that needs to be tested
  * Attachments are also supported: e.g., images in the issue are provided to the agent
* Assign the issue to the **Empirical** agent
* This creates a request in Empirical
  * A link to the request is posted as a comment on the Linear issue
  * Empirical AI agents start working on the request right away
* Once the request is completed, the integration posts a status update back on the issue

## Routing to projects

If your organization has one project, all Linear issues are routed to it.

With multiple projects, map each Linear team to a project under **Settings → Requests → Project associations**. Issues are then routed by their team.

## Get started

To install the Linear integration:

<Steps>
  <Step title="Navigate to Integrations">
    Go to the **Integrations** page
  </Step>

  <Step title="Connect">
    Click **"Connect"** on the Linear integration
  </Step>

  <Step title="Accept permissions">
    Accept the permissions — the integration is configured instantly
  </Step>
</Steps>

## Permissions

The integration requests the following permissions:

| Scope | Purpose |
| - | - |
| Create webhooks | Create a webhook that receives Linear events when the agent is assigned an issue |
| Read issues | Get the contents of the issue, including title, description, and attachments |
| Write comments | Post comments with request links and status updates |
| Read user | Get details of the user who assigned the issue |


# Slack
Source: https://empirical.run/docs/integrations/slack

Install and configure our Slack app

Our Slack app adds the `@Empirical` bot to your Slack workspace. This bot
shares test reports in the channel, and tagging the bot will start agent sessions.

## Installation

1. Open **Settings** > **Reporters** for your project
2. Click **Install** on the Slack integration to start the OAuth flow

## Add a channel

After installation, mention `@Empirical` in any Slack channel you want to use.
Slack will prompt you to add the bot to the channel if needed. The first mention
registers the channel for your Empirical organization and starts an agent session.

If your organization has more than one project, start the message with the project
slug in square brackets:

```text theme={null}
@Empirical [project-slug] investigate this failure
```

With one active project, mentioning `@Empirical` is enough—no project slug is
required. View registered channels under **Settings** > **Slack channels**.

## Configure test run notifications

Registering a channel does not automatically enable test run notifications. Open
**Settings** > **Environments**, edit an environment's **Slack Notifications**, and
select the registered channels that should receive its run updates.

Repeat this for each environment that should send notifications. Removing a channel
from an environment stops that environment's notifications without disconnecting the
channel from the organization.

## Permissions

The Slack app can send and read messages only in channels where the bot has been added. Specifically, it requests for these permission scopes:

| Scope | Purpose |
| - | - |
| `chat:write`, `chat:write.customize` | Used to share test run reports and AI agent responses |
| `app_mentions:read` | Bot listens to `@Empirical` mentions to start AI chat sessions |
| `reactions:write` | Add reactions to messages to acknowledge that agent is working |
| `channels:history`, `groups:history` | Read message threads to get context in threads where `@Empirical` is mentioned, in public and private channels |
| `files:read`, `files:write` | Download image attachments shared in Slack threads for AI agent context, and upload reports and screenshots |
| `links:read`, `links:write` | Show a preview card when an `empirical.run` link is shared in a channel |
| `im:history`, `im:write` | Hold direct-message conversations with the bot |
| `assistant:write` | Reply in the Slack assistant pane |
| `team:read` | Resolve Slack Connect users' home workspace domains for message permalinks |
| `users:read`, `users:read.email` | Resolve Slack users to their Empirical account by email |


# Environments
Source: https://empirical.run/docs/platform/environments

Configure deployment targets, test selection, variables, and schedules

An environment is a named deployment or customer configuration that tests run against, such as staging or production. It controls which Playwright projects run, which variables they receive, and whether the suite runs on a schedule.

Each environment has:

* A **name** shown in the dashboard.
* A unique **slug** used by the API, CLI, and `ENV_SLUG` at runtime. Slugs contain lowercase letters, numbers, and hyphens.
* **Match** and **ignore** patterns that select Playwright projects.
* Optional variables and a scheduled trigger.

## Manage environments as code

Environment configuration lives in `.empiricalrun/ENVIRONMENTS.yaml` in the test repository. Add an entry for each deployment or customer configuration:

```yaml theme={null}
environments:
  - slug: staging
    name: Staging
    playwright_projects_match:
      - "*"
    playwright_projects_ignore:
      - "*-production"
    env_files:
      - .env.example
      - .env.example.staging
    variables:
      - name: BASE_URL
        value: https://staging.example.com

  - slug: production
    name: Production
    playwright_projects_match:
      - "*-production"
```

Open a pull request for the change. Empirical does not read `ENVIRONMENTS.yaml` from a feature branch or an open pull request; the configuration is picked up only after the pull request is merged into the test repository's default branch. Empirical then syncs the environments automatically.

To retry the sync or force Empirical to reread the file from the default branch, go to **Settings → Environments** and click **Sync**.

The dashboard shows the synced configuration but does not create or edit environments. When an entry is removed from YAML, Empirical disables an existing environment that has run history and deletes one that has never been used.

`playwright_projects_match` and `playwright_projects_ignore` accept glob patterns. Put exclusions in the ignore list; patterns beginning with `!` are rejected.

## Environment variables

There are three ways to provide values. Later sources override earlier ones:

1. `env_files` from the test repository, merged in listed order.
2. Inline `variables` in `ENVIRONMENTS.yaml`.
3. Dashboard variables under **Settings → Environment variables**.

Use tracked template files such as `.env.example`; do not add `.env` itself to `env_files`. Keep secrets in the dashboard, where a variable can apply to all environments or only selected ones.

At runtime Empirical writes `.env.<slug>` and sets `ENV_SLUG`. If the test repo loads dotenv, it can select the matching file:

```ts theme={null}
import * as dotenv from "dotenv";

const envSlug = process.env.ENV_SLUG?.toLowerCase();
dotenv.config({ path: envSlug ? `.env.${envSlug}` : ".env" });
```

For a deployment-specific URL supplied by CI, set `BUILD_URL` as a run-level environment variable override. Run-level overrides take precedence over the environment's configured values.

## Scheduled runs

Add a five-field cron expression to the environment in `.empiricalrun/ENVIRONMENTS.yaml`. Schedules are evaluated in UTC and cannot be created or edited in the dashboard.

```yaml theme={null}
scheduled_trigger:
  cron: "0 2 * * *"
  concurrency:
    group: production-nightly
    on_conflict: wait
```

The `concurrency` block is optional. Use `wait` to prevent scheduled runs from overlapping, or `cancel` to replace an active run in the same group. See [Scheduled runs](/docs/triggers/scheduled) and [Concurrency](/docs/features/concurrency-limits).


# Memories
Source: https://empirical.run/docs/platform/memories

Durable knowledge about your product that guides test generation agents

Memories are durable notes about your product that guide our agents when writing tests. The more context an agent has, the better the generated tests.

Each memory has a name, a short description, and a markdown body. Your agent writes them as it learns your app — after exploring a flow, or when you tell it something worth keeping — and reads them back on later requests. Every memory index entry is visible to the agent from the start; it loads the full body of the ones that look relevant to the task at hand.

## What gets remembered

| Topic | Examples |
| - | - |
| User flows | Sign up, checkout, onboarding steps |
| Business rules & edge cases | Discount logic, permission boundaries, rate limits |
| Auth patterns | Login methods, session handling, protected routes |
| Domain terminology | Product-specific terms the agent should understand |

## Viewing memories

Go to the **Memories** page to browse everything the agent has saved for your project, search across names and content, and open a memory's version history to see how it changed over time.

Memories are read-only in the dashboard — the agent owns them. To add or correct one, tell the agent in a chat session (for example, "remember that staging seeds a fresh cart for each test user") and it will save or update the memory for you.

<Note>
  Memories replace the older **App Knowledge** files, which lived as markdown in your `.empiricalrun` directory. Existing App Knowledge was migrated automatically — nothing to do on your side.
</Note>


# Private Runner
Source: https://empirical.run/docs/platform/private-runner

Run test runs on infrastructure in your own AWS account

A Private Runner runs your test runs in your own AWS account instead of on Empirical's hosted
infrastructure. You install it with a CloudFormation template from the dashboard. Empirical still
schedules runs, and reports still show up in the dashboard as usual.

## Prerequisites

* **AWS account ID** (12 digits). The setup link only works in that account. One runner per account per region.
* **A supported region:** `us-east-1` or `ap-south-1`. Projects that use the runner move to its region.
* **Permission to create a CloudFormation stack with named IAM roles.** `AdministratorAccess` is enough.
* **A VPC with subnets that reach the internet:** public subnets (`0.0.0.0/0 → igw-…`, as in the default VPC), or
  private subnets with `0.0.0.0/0 → nat-…`.
  * Pick subnets in two or more Availability Zones, all in the same VPC.
  * Keep one free IP per shard that runs at the same time.
  * No inbound rules are needed. Workers only make outbound connections.
* **Fargate vCPU quota** (Service Quotas, On-Demand and Spot) of about vCPU per shard × shards at once.
  Shards use 4–16 vCPU each, depending on machine size.

## Set up a private runner

<Tabs>
  <Tab title="CLI">
    Sign in with `empirical login` (these endpoints need a signed-in user, not a project API key).
    If you have access to more than one project, add `-p <project-slug>` to each command.

    1. Create the runner and a setup link:
       ```bash theme={null}
       empirical api api/runner-backends/project/setup \
         -f aws_account_id=123456789012 -f region=us-east-1 \
         --jq '.data | {id: .runner_backend.id, url: .registration.quick_create_url, expires_at: .registration.expires_at}'
       ```
    2. Open `url` while signed in to that AWS account. Choose the VPC and subnets, acknowledge IAM resources, and create the stack.
    3. Wait for the runner to become `active` (about 5 minutes):
       ```bash theme={null}
       empirical api api/runner-backends/project/available \
         --jq '.data.runner_backends[] | {id, name, region, status}'
       ```
       The project that started the setup switches to the runner on its own.
    4. Use the runner for another project in the organization, or switch back with `null`:
       ```bash theme={null}
       empirical api api/runner-backends/project/runner-backend -X PUT -p other-project -F runner_backend_id=<id>
       empirical api api/runner-backends/project/runner-backend -X PUT -p other-project -F runner_backend_id=null
       ```

    * **Link expired or the stack rolled back:** `empirical api api/runner-backends/project/<id>/registration -X POST` gives a new link.
    * **Re-run the checks:** `empirical api api/runner-backends/project/verify -X POST`.
  </Tab>

  <Tab title="Dashboard">
    1. Open **Settings → Infra** for the project and click **Set up private runner**.
    2. Pick the **AWS region**, enter the **AWS account ID**, and click **Create setup link**.
    3. Sign in to that AWS account and click **Open AWS console**. The link works once and expires in 7 days.
    4. In CloudFormation, choose the VPC and subnets, acknowledge that the stack creates IAM resources, and click **Create stack**.
    5. Wait about 5 minutes. The card shows **Waiting for your stack**, then the runner once it connects.
       The project that started the setup switches to the runner on its own.

    * **Other projects in the organization:** open their **Settings → Infra** and click **Use for this project**.
    * **Switch a project back:** click **Use for this project** on the **Empirical hosted** row.
  </Tab>
</Tabs>

If the stack fails a check, it rolls back. The reason is in the stack's **Events** tab in CloudFormation.

## Reach private apps

Workers run in your subnets, so they can reach anything those subnets can route to, using the VPC's DNS.

* **App in the same VPC:** allow inbound traffic from the runner's security group (the `SecurityGroupId` stack output).
* **App in another VPC or on-premises:** use VPC peering, Transit Gateway or Site-to-Site VPN, and add routes in the runner subnets.
* **App behind NetBird or Tailscale:** you can't run a VPN client inside the workers (Fargate), so use a routing peer:
  * Run an EC2 instance in the runner VPC as a routing peer
    ([NetBird network route](https://docs.netbird.io/use-cases/remote-access/site-to-vpn) with masquerade, or a
    [Tailscale subnet router](https://tailscale.com/kb/1019/subnets)).
  * Turn off its source/destination check.
  * In each runner subnet's route table, route the app's private CIDR to the instance.
  * Make the app's hostname resolve in the VPC (a Route 53 private hosted zone or a Resolver forwarding rule), or use IPs.

## Limitations

* **A project runs everything in one place.** Once a project uses a private runner, all its test runs go there, from every
  trigger and environment. You can't send only some runs to it. To split runs, use a separate project.


# Test repository
Source: https://empirical.run/docs/platform/repository

Clone and work with the Playwright tests Empirical maintains

Every Empirical project has a git repository containing its Playwright tests. This is separate from your application repository: Empirical tests your deployed application as a black box and does not need your application source code.

## Clone the repository

Install and sign in to the [Empirical CLI](/docs/cli), then run:

```sh theme={null}
empirical clone
```

If your account has access to more than one project, select one by slug:

```sh theme={null}
empirical clone --project <project-slug>
```

Pass a directory as the first argument to choose where the repository is cloned:

```sh theme={null}
empirical clone my-tests --project <project-slug>
```

The clone URL uses Empirical's git endpoint at `git.empirical.run`. The CLI configures a credential helper automatically so git can use your Empirical login.

## Use git directly

To configure the credential helper without cloning, run this once on each machine:

```sh theme={null}
empirical git setup
```

You can then use standard git commands:

```sh theme={null}
git clone https://git.empirical.run/<project-slug>.git
git fetch origin
git push -u origin <branch-name>
```

Your user login can fetch and push feature branches. Pushes to the project's default branch are rejected; changes should be reviewed and merged through the normal pull request workflow. Project API keys, including keys used by CI and test-run containers, have read-only repository access.

## Ownership

The test code written for your project belongs to your organization. Contact [hey@empirical.run](mailto:hey@empirical.run) if you need repository access or an export.


# Requests
Source: https://empirical.run/docs/platform/requests

Ask Empirical agents to add or update test coverage

A request is a work item for adding or updating test coverage. It keeps the original brief, status, and every agent session associated with the work in one place.

## Creating a request

<Steps>
  <Step title="Navigate to Requests">
    Open **Requests** in your project.
  </Step>

  <Step title="Create a new request">
    Click **New Request**.
  </Step>

  <Step title="Describe the coverage">
    Add a clear title and description. Leave **Create as draft** off to start an agent session immediately, or turn it on to save the request without starting work.
  </Step>

  <Step title="Track progress">
    Open the request and select the **Sessions** tab to follow the agent. You can start another conversation with **New Session**.
  </Step>
</Steps>

A clear request leads to a better test. Cover these four things:

* **What to test** — the feature or user flow
* **Steps** — the actions the user takes, in order
* **Expected outcome** — what a passing test looks like
* **Edge cases** — variations or special conditions to account for

To provide screenshots, videos, or other files, attach them to the prompt when you start a session from the request.

### Example Prompt

```
Test the checkout flow for a returning customer:
1. User logs in with existing credentials
2. Adds product to cart from the search results
3. Navigates to checkout
4. Selects saved payment method
5. Completes purchase
6. Verify order confirmation appears with correct order details
```

## Drafts and statuses

New requests are **Open** by default and immediately create a session for the agent. A **Draft** does not start a session; changing it from Draft to Open starts one.

Use the status selector on the request to move it between **Draft**, **Open**, **In Review**, **Resolved**, and **Closed**. Editing the title or description does not change existing session messages, so send important follow-up context in the relevant session as well.

Requests created from [Linear](/docs/integrations/linear) or [Jira](/docs/integrations/jira) also link back to their source issue.


# Snoozes
Source: https://empirical.run/docs/platform/snoozes

Suppress known failures while you work on a fix

Snooze known failures so they don't clutter your reports while you work on a fix. Snoozed tests are still visible in runs — just clearly marked as suppressed, not ignored.

## Creating a snooze

From any test run, select one or more failing tests and click **"Snooze"**. Configure:

* **Description** — why you're snoozing (optional but recommended)
* **Duration** — set an expiration date or snooze indefinitely
* **Scope** — snooze in specific environments or across all of them

## Managing snoozes

Go to the **Snoozes** page for a full view of active and expired snoozes.

**Active tab** shows each snooze with its creator, creation time, description, expiration, environment scope, and linked test cases.

**Expired tab** shows historical snoozes. You can reactivate any of them if needed.

### Actions

| Action | What it does |
| - | - |
| Edit | Update description, expiration, or environment scope |
| Expire Now | End the snooze immediately |
| Activate | Reactivate an expired snooze |

Permanent deletion is available through the [Delete a snooze API](/docs/api-reference/snoozes/delete-a-snooze), not from the dashboard.


# Test Run Configuration
Source: https://empirical.run/docs/platform/test-run-configuration

Understand execution regions, sharding, and report notifications

## Region

Each project runs in one AWS region. Empirical currently supports:

| Region | Code | Best for |
| - | - | - |
| US East (N. Virginia) | `us-east-1` | North America |
| Asia (Mumbai) | `ap-south-1` | Asia & Europe |

Region selection is managed by Empirical. Contact [hey@empirical.run](mailto:hey@empirical.run) if your tests should run in a different supported region.

## Shards

Sharding splits a suite across parallel workers to reduce run time. A project's configured shard count is a maximum from 1 to 50; smaller filtered runs may use fewer shards so workers are not started with too little work.

The run report shows each shard's progress and logs. The public trigger API does not expose a shard override, and regular project members cannot change the project maximum in the dashboard. Contact [hey@empirical.run](mailto:hey@empirical.run) to adjust it.

## Report notifications

Use **Settings → Reporters** to enable email reports and configure GitHub reporting. Use **Settings → Slack channels** to choose which connected channels receive reports and for which environments.

When starting or rerunning a test from the dashboard, use the **Notifications** tab to choose whether that individual run sends report notifications. This setting does not change the project's reporter configuration.

See [Slack notifications](/docs/features/slack-notifications) for notification behavior and [Slack setup](/docs/integrations/slack) for channel setup.


# Ad-hoc runs
Source: https://empirical.run/docs/triggers/ad-hoc

Trigger a one-off test run from the dashboard, CLI, or API

Use an ad-hoc run when you want to execute a suite now rather than wait for a schedule or deployment trigger.

## Dashboard

Open **Test runs** in your project, click **New Test Run**, and select an environment. The dialog also lets you:

* Run specific tags or test cases.
* Override environment variables for this run.
* Run a test-repository branch.
* Choose whether to send report notifications.
* Set a [concurrency](/docs/features/concurrency-limits) group and conflict policy.

## CLI

The CLI can call the test-runs API using your login:

```sh theme={null}
empirical api api/test-runs \
  --project <project-slug> \
  --method PUT \
  --field environment=production
```

## API

Use a project API key to trigger a run from a script:

```sh theme={null}
curl -X PUT https://api.empirical.run/api/test-runs \
  -H "Authorization: Bearer $EMPIRICALRUN_KEY" \
  -H "Content-Type: application/json" \
  -d '{"environment":"production"}'
```

See [From CI/CD pipelines](/docs/triggers/ci-cd) for build metadata, variable overrides, and concurrency examples, or the [Trigger a test run API](/docs/api-reference/test-runs/trigger-a-test-run) for every field.


# From CI/CD pipelines
Source: https://empirical.run/docs/triggers/ci-cd

Run tests after production or pre-prod deployments

## Authentication

All of the methods below use an API key to authenticate. Generate one under [Settings → API keys](https://empirical.run/settings/api-keys).

## GitHub Actions

Use our [dispatch-action](https://github.com/empirical-run/dispatch-action) if you are using GitHub Actions for CI/CD.

```yml theme={null}
- name: Deploy step
  id: deploy-step
  run: ...

- name: Dispatch for tests
  uses: empirical-run/dispatch-action@v1
  with:
    auth-key: ${{ secrets.EMPIRICALRUN_KEY }}
    environment: production
    environment-variables: |
      BUILD_URL: ${{ steps.deploy-step.outputs.url }}
    concurrency: |
      group: production/acme/web
      on-conflict: wait
```

The `v1` reference receives backwards-compatible fixes while keeping breaking
major releases opt-in. You can instead pin an exact release such as `v1.0.0`.

Use the `environment-variables` input to set environment variables for the test
run, one per line in `NAME: value` format. These override the environment's
configured variables. The URL of the build under test is passed to tests as the
`BUILD_URL` environment variable.

Set `group` and `on-conflict` in the `concurrency` input to control what happens
when the group already has an active run. For preview deployments, a group such
as `preview/acme/web@${{ github.head_ref }}` with `cancel` lets a new push
supersede the previous run for that branch. For production, use an
environment/repository group with `wait` to prevent deployments from testing
concurrently. See [Concurrency](/docs/features/concurrency-limits).

<Note>
  The `build-url` input is deprecated. It is sent to tests as the `BUILD_URL`
  environment variable, so set `BUILD_URL` via `environment-variables` instead.
</Note>

### Vercel deployments

For Vercel deployments, trigger tests on the `deployment_status` event. You can
copy this workflow into `.github/workflows/trigger-e2e.yaml`:

```yml theme={null}
name: Trigger e2e tests

on:
  workflow_dispatch:
  deployment_status:

jobs:
  trigger-e2e-tests-production:
    if: github.event_name == 'workflow_dispatch' ||
        (github.event_name == 'deployment_status' &&
          github.event.deployment_status.state == 'success' &&
          github.event.deployment_status.environment == 'Production')
    runs-on: ubuntu-latest
    steps:
      - name: Dispatch for tests
        uses: empirical-run/dispatch-action@v1
        with:
          auth-key: ${{ secrets.EMPIRICALRUN_KEY }}
          environment: production
          environment-variables: |
            BUILD_URL: ${{ github.event.deployment_status.target_url }}
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
```

The `GITHUB_TOKEN` environment variable lets the action pull branch info for
the deployment, which is attached to the test run report.

## Other providers

For other CI/CD providers, you can trigger a test run with an API call.

### Minimal snippet

```sh theme={null}
curl -X PUT \
  https://api.empirical.run/api/test-runs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $EMPIRICALRUN_KEY" \
  -d '{ "environment": "production" }'
```

### Metadata for reporting

You can add metadata that will be associated with the test run report.

* Repo where the trigger was sent from
* Details of the commit or branch
* Other metadata (key-value pairs)

```sh theme={null}
jq -n \
  --arg repo "$REPOSITORY" \
  --arg commit "$COMMIT" \
  --arg branch "$BRANCH" \
  --arg build_url "$BUILD_URL" \
  '{
    environment: "production",
    build: {
      repo: $repo,
      commit: $commit,
      branch: $branch
    },
    metadata: {
      pairs: "of",
      key: "value"
    },
    environment_variables_overrides: [
      {name: "BUILD_URL", value: $build_url},
      {name: "MY_VAR", value: "my_value"}
    ]
  }' | curl -X PUT \
  https://api.empirical.run/api/test-runs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $EMPIRICALRUN_KEY" \
  --data-binary @-
```

### Pull request runs

For a pre-merge run on an application pull request, send `trigger` with the pull request. The
run uses the matching test branch, merged into the default branch. See
[Branching](/docs/features/branching#pull-request-runs).

```sh theme={null}
jq -n \
  --arg commit "$COMMIT" \
  --arg head "$PR_HEAD_BRANCH" \
  --arg base "$PR_BASE_BRANCH" \
  --argjson number "$PR_NUMBER" \
  '{
    environment: "preview",
    build: { repo: "acme/web", branch: $head, commit: $commit },
    trigger: {
      event: "pull_request",
      pull_request: { number: $number, head: $head, base: $base }
    }
  }' | curl -X PUT \
  https://api.empirical.run/api/test-runs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $EMPIRICALRUN_KEY" \
  --data-binary @-
```

The [dispatch-action](#github-actions) sends `trigger` automatically on `pull_request` workflow events.

### Concurrency

Tell Empirical what to do when a run for the same thing is already going. Runs that share a
`concurrency.group` (within the project) execute one at a time; `on_conflict` is what the
new run does when the group is busy. See [Concurrency](/docs/features/concurrency-limits).

```sh theme={null}
jq -n \
  --arg branch "$BRANCH" \
  --arg build_url "$BUILD_URL" \
  '{
    environment: "preview",
    build: { branch: $branch, repo: "acme/web" },
    environment_variables_overrides: [{name: "BUILD_URL", value: $build_url}],
    concurrency: {
      group: ("preview/acme/web@" + $branch),
      on_conflict: "cancel"
    }
  }' | curl -X PUT \
  https://api.empirical.run/api/test-runs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $EMPIRICALRUN_KEY" \
  --data-binary @-
```

A new push to the same branch cancels the run still going for the previous push. Use
`"on_conflict": "wait"` to queue behind it instead, or the deployment URL as the group when
one server can only take one run at a time.


# Scheduled runs
Source: https://empirical.run/docs/triggers/scheduled

Run an environment automatically using a UTC cron schedule

Each environment can trigger its full selected suite on a recurring schedule.

Schedules are configured in the test repository's `.empiricalrun/ENVIRONMENTS.yaml` file. They cannot be created or edited in the dashboard.

## Configure a schedule

In `.empiricalrun/ENVIRONMENTS.yaml`, `scheduled_trigger` can be a cron string:

```yaml theme={null}
environments:
  - slug: production
    name: Production
    scheduled_trigger: "0 2 * * *"
```

Or an object with an explicit [concurrency](/docs/features/concurrency-limits) policy:

```yaml theme={null}
environments:
  - slug: production
    name: Production
    scheduled_trigger:
      cron: "0 2 * * *"
      concurrency:
        group: production-nightly
        on_conflict: wait
```

Open a pull request for the change. Empirical does not read the schedule from a feature branch or an open pull request; it is picked up only after the pull request is merged into the test repository's default branch. Empirical then syncs the file automatically.

**Settings → Environments → Sync** retries the sync or forces Empirical to reread the file from the default branch. Invalid cron syntax or unknown YAML fields cause the sync to fail with a validation message.

## Cron format

Empirical accepts standard five-field cron syntax:

```text theme={null}
minute hour day-of-month month day-of-week
```

| Schedule | Expression |
| - | - |
| Every four hours | `0 */4 * * *` |
| Every day at 02:00 UTC | `0 2 * * *` |
| Weekdays at 09:00 UTC | `0 9 * * 1-5` |

To pause a schedule, remove `scheduled_trigger` in a pull request and merge it into the default branch. Disabling an environment also keeps it out of new-run selectors and scheduled execution.