Appwright: A new test framework for mobile apps

The cost of shipping buggy software is higher than ever before: software is mission-critical for billions of people to accomplish their goals. Software also increasingly depends on third-party code, which increases the chances that customers run into bugs. At the same time, customers demand more: new features needed to be shipped yesterday. This pressure forces software teams to balance velocity and quality, and they can often be at odds with each other.

In theory, automated software testing helps — especially when it replicates how customers interact with the application. This is achievable through end-to-end tests that treat the application as a black box, and focus on testing end-user scenarios.

But in practice, we see software teams struggle to setup an automated testing suite, and once setup, stumble upon unreliable, flaky tests that are hard to write and maintain. Software builders need simpler solutions that get them a reliable end-to-end test suite.

To fulfill this, we built Appwright: a test framework for end-to-end testing of mobile apps. Appwright is open source on GitHub (opens in a new tab), and v0.1.0 is available today on NPM (opens in a new tab).

npm install --save-dev appwright

Lessons from Playwright

We are big fans of the impact that Playwright (opens in a new tab) has had on the end-to-end testing landscape for web apps. Playwright was launched as an automation driver for web browsers, and since v1.12 (opens in a new tab), it has shipped with an integrated test framework. An automation driver exposes APIs to automate user actions (e.g. click) on an application running inside a web browser or a mobile device. A test framework provides primitives for testing: a CLI to run tests, setup/teardown, and assertions.

This integration ensured that Playwright users got everything they needed to get a modern end-to-end testing suite. Playwright's continued focus on this problem, and a monthly release cadence since May 2020, has made Playwright the best tool for this job.

The mobile ecosytem lacks an equivalent experience.

Appwright

Our goal with Appwright is to fill that gap. Appwright is a single package that exposes a test framework and automation APIs to write end-to-end, UI-level tests for mobile apps.

Automation

Internally, Appwright is built on top of Playwright's test framework, and glues that to Appium (opens in a new tab), the industry standard for native mobile automation. Building on top of Appium enables Appwright users to leverage Appium's large ecosystem of device farm vendors (e.g. BrowserStack (opens in a new tab)) and open source implementations. This eliminates concerns of vendor lock-in.

Test framework

While Appium solves mobile device automation, it lacks a purpose-built test framework. Yes, you can use it with any test framework (like Mocha or TestNG), but this flexibility comes at a cost: users need to invest time in setting things up. Popular test frameworks also juggle different types of testing: unit and end-to-end testing, and the popularity of unit testing tends to influence their defaults. For example,

• Test timeouts: Default timeouts are set for unit tests, which are faster than end-to-end tests. You can bump it up to something larger – say, 30 secs – but what happens when a new test takes 45 secs? Increasing timeout further causes runs to slow down (since tests that fail wait for longer). We believe end-to-end tests are better off without test timeouts entirely: it's more useful to enforce timeouts at the action-level (e.g. timeout if the next click takes >10 secs). Popular test frameworks don't support this.
• Flakes and retries: Another key difference is that end-to-end tests can be flaky, and can pass on retries. Test runners that optimize for unit testing do not support retrying a single test, or even reporting "flaky" as a test status.
• Visual reporting: You don't need a visual report for unit tests: few lines of text in the terminal are enough, and so no popular test framework ships with one. Visual reporting is table-stakes for end-to-end tests: you can't debug a failing test without inspecting screenshots or videos from the run.

These differences add up. Just "setting up end-to-end tests" becomes a long-standing ticket on your project tracker, when you could be using that time to increase test coverage instead.

Appwright fills this gap by building on top of Playwright's test framework (opens in a new tab). The test framework has matured well for the end-to-end testing workflow, and it exposes powerful primitives (see fixtures (opens in a new tab)) that made it possible for us to build Appwright.

Usage highlights

Readable tests

Each test written with Appwright gets a device that loads a freshly installed instance of the application under test. This ensures all tests are ready for parallel execution, by design, and this behavior can be modified to run in serial, like in Playwright (opens in a new tab).

Our example project (opens in a new tab) runs Appwright on the Wikipedia's mobile apps, and the following test is lifted from there.

import { test, expect } from "appwright";
 
test("Search for playwright and verify contents", async ({ device }) => {
  await device.getByText("Search Wikipedia").fill("playwright");
  await device.getByText("Playwright (software)").tap();
  await expect(device.getByText("Microsoft")).toBeVisible();
});

Simple and reliable automation APIs

End-to-end tests can be frustrating when a test fails to do an action that a manual tester would accomplish successfully. This is because manual testers do many checks implicitly: they will patiently wait for a loader to disappear, instead of tapping on it.

Appwright solves this by auto-waiting for the underlying element to be visible. Appwright also delays resolving to the element as late as possible, which prevents stale element references. Additionally, APIs that find element prefer user-facing attributes (like text content). This ensures implementation changes in the UI don't break the test.

const button = device.getByText("Submit");
await button.tap();

Built with TypeScript

Appwright is written in TypeScript, and can also be used in JavaScript. This should feel at home to React Native developers. We understand that Swift, Kotlin, or Flutter developers might have to stretch to find it comfortable. This was a conscious trade-off, for we want to get it right for at least one ecosystem.

Appwright ships with types, to enable a high-quality developer experience inside VS Code. Types also assist LLM copilots and agents, which we believe will play a big role in writing tests going forward.

Get started with Appwright

Appwright v0.1.0 (opens in a new tab) ships today, and is ready for early adopters to try. We're eager to hear your feedback: file issues (opens in a new tab), and expect a fast response. We're committed to making Appwright the new standard for end-to-end testing, and enable mobile developers to ship high quality apps that improve lives of their end-users.

Some links to get you started:

• Appwright on NPM (opens in a new tab)
• Appwright on GitHub (opens in a new tab)
• Documentation (opens in a new tab)

About us

Our team has extensive experience with this problem: Arjun (opens in a new tab) worked on Playwright at Microsoft, and Saikat (opens in a new tab) led client engineering teams that built India's largest consumer apps. We're applying our experience to now build LLM agents that write and maintain end-to-end tests for web and mobile apps.

Our agents on mobile are built on top of Appwright, and they power hundreds of tests for our early customers. We're grateful to the mobile engineering team at Leap (opens in a new tab) for being early adopters. Writing and running their end-to-end tests has helped mature the Appwright API and given us confidence to make this release.