From CI/CD pipelines

Authentication

All of the methods below use an API key to authenticate. Generate one under Settings → API keys.

GitHub Actions

Use our dispatch-action if you are using GitHub Actions for CI/CD.

- name: Deploy step
  id: deploy-step
  run: ...

- name: Dispatch for tests
  uses: empirical-run/dispatch-action@v1
  with:
    auth-key: ${{ secrets.EMPIRICALRUN_KEY }}
    environment: production
    environment-variables: |
      BUILD_URL: ${{ steps.deploy-step.outputs.url }}
    concurrency: |
      group: production/acme/web
      on-conflict: wait

The v1 reference receives backwards-compatible fixes while keeping breaking major releases opt-in. You can instead pin an exact release such as v1.0.0. Use the environment-variables input to set environment variables for the test run, one per line in NAME: value format. These override the environment’s configured variables. The URL of the build under test is passed to tests as the BUILD_URL environment variable. Set group and on-conflict in the concurrency input to control what happens when the group already has an active run. For preview deployments, a group such as preview/acme/web@${{ github.head_ref }} with cancel lets a new push supersede the previous run for that branch. For production, use an environment/repository group with wait to prevent deployments from testing concurrently. See Concurrency.

The build-url input is deprecated. It is sent to tests as the BUILD_URL environment variable, so set BUILD_URL via environment-variables instead.

Vercel deployments

For Vercel deployments, trigger tests on the deployment_status event. You can copy this workflow into .github/workflows/trigger-e2e.yaml:

name: Trigger e2e tests

on:
  workflow_dispatch:
  deployment_status:

jobs:
  trigger-e2e-tests-production:
    if: github.event_name == 'workflow_dispatch' ||
        (github.event_name == 'deployment_status' &&
          github.event.deployment_status.state == 'success' &&
          github.event.deployment_status.environment == 'Production')
    runs-on: ubuntu-latest
    steps:
      - name: Dispatch for tests
        uses: empirical-run/dispatch-action@v1
        with:
          auth-key: ${{ secrets.EMPIRICALRUN_KEY }}
          environment: production
          environment-variables: |
            BUILD_URL: ${{ github.event.deployment_status.target_url }}
        env:
          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}

The GITHUB_TOKEN environment variable lets the action pull branch info for the deployment, which is attached to the test run report.

Other providers

For other CI/CD providers, you can trigger a test run with an API call.

Minimal snippet

curl -X PUT \
  https://api.empirical.run/api/test-runs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $EMPIRICALRUN_KEY" \
  -d '{ "environment": "production" }'

Metadata for reporting

You can add metadata that will be associated with the test run report.

Repo where the trigger was sent from
Details of the commit or branch
Other metadata (key-value pairs)

jq -n \
  --arg repo "$REPOSITORY" \
  --arg commit "$COMMIT" \
  --arg branch "$BRANCH" \
  --arg build_url "$BUILD_URL" \
  '{
    environment: "production",
    build: {
      repo: $repo,
      commit: $commit,
      branch: $branch
    },
    metadata: {
      pairs: "of",
      key: "value"
    },
    environment_variables_overrides: [
      {name: "BUILD_URL", value: $build_url},
      {name: "MY_VAR", value: "my_value"}
    ]
  }' | curl -X PUT \
  https://api.empirical.run/api/test-runs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $EMPIRICALRUN_KEY" \
  --data-binary @-

Pull request runs

For a pre-merge run on an application pull request, send trigger with the pull request. The run uses the matching test branch, merged into the default branch. See Branching.

jq -n \
  --arg commit "$COMMIT" \
  --arg head "$PR_HEAD_BRANCH" \
  --arg base "$PR_BASE_BRANCH" \
  --argjson number "$PR_NUMBER" \
  '{
    environment: "preview",
    build: { repo: "acme/web", branch: $head, commit: $commit },
    trigger: {
      event: "pull_request",
      pull_request: { number: $number, head: $head, base: $base }
    }
  }' | curl -X PUT \
  https://api.empirical.run/api/test-runs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $EMPIRICALRUN_KEY" \
  --data-binary @-

The dispatch-action sends trigger automatically on pull_request workflow events.

Concurrency

Tell Empirical what to do when a run for the same thing is already going. Runs that share a concurrency.group (within the project) execute one at a time; on_conflict is what the new run does when the group is busy. See Concurrency.

jq -n \
  --arg branch "$BRANCH" \
  --arg build_url "$BUILD_URL" \
  '{
    environment: "preview",
    build: { branch: $branch, repo: "acme/web" },
    environment_variables_overrides: [{name: "BUILD_URL", value: $build_url}],
    concurrency: {
      group: ("preview/acme/web@" + $branch),
      on_conflict: "cancel"
    }
  }' | curl -X PUT \
  https://api.empirical.run/api/test-runs \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $EMPIRICALRUN_KEY" \
  --data-binary @-

A new push to the same branch cancels the run still going for the previous push. Use "on_conflict": "wait" to queue behind it instead, or the deployment URL as the group when one server can only take one run at a time.

Getting Started

Platform

Features

Integrations

Trigger a test run

From CI/CD pipelines

Authentication

GitHub Actions

Vercel deployments

Other providers

Minimal snippet

Metadata for reporting

Pull request runs

Concurrency

​Authentication

​GitHub Actions

​Vercel deployments

​Other providers

​Minimal snippet

​Metadata for reporting

​Pull request runs

​Concurrency

Authentication

GitHub Actions

Vercel deployments

Other providers

Minimal snippet

Metadata for reporting

Pull request runs

Concurrency