Select tests by tag
Tag your tests in Playwright:exclude_tags to skip tests, for example --exclude-tags @destructive on an environment that points at production.
Empirical adds a missing @ to each tag, so smoke and @smoke select the same tests. A tag matches as a substring of the test’s tags: @smoke also selects a test tagged @smoke-nightly.
tags does not filter by test title. A title such as checkout with a saved card becomes @checkout with a saved card, matches no test, and the run ends with empty_selection. To run specific tests, pass their ids in test_case_ids (--tests in the CLI).
Selection fields
The trigger request and the environment use the same field names with the same meaning.
Most selections need only
tags and exclude_tags. Use the project fields when your Playwright config splits tests into projects you want to run separately, such as one project per browser.
How Empirical combines them
A test runs when it passes every filter:- If
tagsis set, it carries one of the tags. - If
test_case_idsis set, its id is in the list. - Its Playwright project matches
playwright_projects_matchand noplaywright_projects_ignorepattern. - It carries none of the
exclude_tags.
exclude_tags applies last, so it only removes tests. Playwright setup projects that the selected tests depend on run too.
When nothing matches, the run ends with the empty_selection error code.
Environment defaults and request overrides
For each field, Empirical takes the first value it finds:- The trigger request.
- For a rerun, the run it reruns.
- The run’s environment.
- Empty.
"tags": [] runs the environment’s selection without its tags.
The environment’s tags and exclude_tags apply to a run that sends test_case_ids. To run a test the environment’s tags don’t select, send "tags": [] with the ids.
Empirical stores the resolved selection on the run. Changing the environment later doesn’t change runs that already exist.
Save a selection on an environment
Add the fields to the environment’s entry in.empiricalrun/ENVIRONMENTS.yaml:
Select tests for one run
With the CLI:Rerun failed tests
A rerun keeps its source run’s selection, including Re-run all tests. To rerun only the tests that failed, open Re-run on a completed run and click Re-run failed tests, or send"only_failed": true to the rerun API. When the source run has no failed tests, an API rerun with only_failed runs the source run’s whole selection.