We use Percy via Github actions to run visual regression tests. Percy provides pull-request-based workflow, handles diff review and approval flow conveniently. In addition to that, It integrates with Cypress, which allows us to use all power of our custom helpers and commands. We run
How to run visual tests on CI
When your PR is ready for review, you can run visual tests (handled by Percy) in one of two ways:
- Add a
visuallabel to the pull request.
- Comment on the pull request with
@metabase-bot run visual tests
Only run these visual tests when your pull request is ready for review. Percy charges by the screenshot, so running the visual tests on every commit gets expensive.
By default, Cypress tests will run to ensure that the underlying tests are valid, but Cypress won’t submit screenshots to the Percy servers. When the PR is labeled or commented to run the visual tests, the Cypress run command is prefixed by
percy exec -- and a valid
PERCY_TOKEN environment variable, and Cypress will submit the screenshots to the Percy servers.
visual label to your pull request
2. If there are some visual changes, it shows a failed Percy check in the PR
3. Once you review and approve the changes, the PR check becomes green
How to create a visual test
We use Cypress to write Percy tests so we can fully use all existing helpers and custom commands.
Visual regression tests live inside the
e2e/test/visual directory. Writing a Percy test consists of creating a desired page state and executing
Each visual test should cover as many as possible different elements, variants on the same screenshot. For instance, when we are writing E2E test that checks a chart on a dashboard we add just one card and run assertions. In opposite to that, a visual test can contain every type of chart on the same dashboard because it significantly reduces the number of screenshots we produce which reduces the cost of using Percy.
- Run Metabase in the dev mode locally (
yarn devor similar commands).
yarn test-visual-opento open Cypress locally. You do not need to export any
- Create a spec inside
e2e/test/visualand run it via Cypress runner.
At this step, if you added
percySnapshot command somewhere in your test, you will see
percyHealthCheck step in your test:
Consider the page state at
percyHealthCheck step as the one that will be captured.
- You don’t need to export
PERCY_TOKENfor running tests. If a token is exported Percy will send snapshots from your local machine to their servers so that you will be able to see your local run in their interface.
- When the application code uses
Date.now(), you can freeze date/time in Cypress.
Math.randomwhen you deal with randomization.
- When testing a page that renders any dates coming from the server, in order to avoid unwanted visual changes, add
data-server-dateattribute to all DOM nodes that render dates. The custom
createPercySnapshotcommand replaces content with a constant placeholder before capturing a snapshot.