9d318dc4cd
Workers no longer run `npm ci` — `node_modules` and framework binaries are restored from actions/cache populated once by a new `prep-deps` job. This closes the intermittent EEXIST/ENOENT failure inside npm's own cacache writer that occasionally fails `npm ci` on a runner. Removing `npm ci` from workers also cuts ~5 min of duplicated install work per worker. dispatch-begin now runs as its own job after prep-deps so it fires once the per-worker test-server setup is the only remaining work before dispatch-run.
22 KiB
E2E Test Pipelines
Three automated E2E test pipelines cover different stages of the development lifecycle.
Pipelines
| Pipeline | Trigger | Editions Tested | Image Source |
|---|---|---|---|
PR (e2e-tests-ci.yml) |
Argo Events on Enterprise CI/docker-image status |
enterprise | mattermostdevelopment/** |
Merge to master/release (e2e-tests-on-merge.yml) |
Platform delivery after docker build (delivery-platform/.github/workflows/mattermost-platform-delivery.yaml) |
enterprise, fips | mattermostdevelopment/** |
Release cut (e2e-tests-on-release.yml) |
Platform release after docker build (delivery-platform/.github/workflows/release-mattermost-platform.yml) |
enterprise, fips, team (future) | mattermost/** |
All pipelines follow the smoke-then-full pattern: smoke tests run first, full tests only run if smoke passes.
Workflow Files
.github/workflows/
├── e2e-tests-ci.yml # PR orchestrator
├── e2e-tests-on-merge.yml # Merge orchestrator (master/release branches)
├── e2e-tests-on-release.yml # Release cut orchestrator
├── e2e-tests-cypress.yml # Shared wrapper: routes to v1 or v2 template
├── e2e-tests-playwright.yml # Shared wrapper: routes to v1 or v2 template
├── e2e-tests-cypress-template-v2.yml # Active: cypress + test-system-io dispatch
├── e2e-tests-playwright-template-v2.yml # Active: playwright + test-system-io dispatch
├── e2e-tests-cypress-template.yml # Deprecated v1 (legacy in-job execution)
└── e2e-tests-playwright-template.yml # Deprecated v1 (legacy in-job execution)
v1 templates are deprecated. They remain available behind a feature flag during cutover but receive no further changes. New work targets the v2 templates exclusively. The wrappers route by
vars.E2E_USE_TEST_IO_DISPATCH—'true'selects v2, anything else falls back to v1.
Call hierarchy
e2e-tests-ci.yml ─────────────────┐
e2e-tests-on-merge.yml ───────────┤──► e2e-tests-cypress.yml ─────┐
e2e-tests-on-release.yml ─────────┘ e2e-tests-playwright.yml ──┤
│
┌──────────────────────────┘
│ routes on E2E_USE_TEST_IO_DISPATCH
▼
v2 (active) ──► e2e-tests-{cypress,playwright}-template-v2.yml
v1 (legacy) ──► e2e-tests-{cypress,playwright}-template.yml
Workflow Architecture (v2)
v2 splits the template into five jobs — prepare-run, prep-deps, dispatch-begin, workers (matrix), and report — and pushes spec-level execution to Test System IO so workers stay thin and identical.
┌──────────────────────────────────────────────────────────────────────────┐
│ Template v2: e2e-tests-{cypress,playwright}-template-v2.yml │
│ │
│ ┌───────────────────┐ ┌──────────────────────────────┐ │
│ │ prepare-run │ │ prep-deps │ │
│ │ (1 runner) │ parallel │ (1 runner) │ │
│ │ │ ◄────────────► │ │ │
│ │ • build workers │ │ Cypress: │ │
│ │ matrix [1..N] │ │ • cypress/node_modules │ │
│ │ • compute commit │ │ • ~/.cache/Cypress (binary)│ │
│ │ status context │ │ │ │
│ │ • emit composite │ │ Playwright: │ │
│ │ identity │ │ • webapp/platform/{client, │ │
│ │ │ │ types}/{lib,node_mod} │ │
│ │ │ │ • playwright/node_modules │ │
│ │ │ │ • playwright/lib/dist │ │
│ │ │ │ • ~/.cache/ms-playwright │ │
│ │ │ │ (chromium only) │ │
│ │ │ │ │ │
│ │ │ │ → saved to actions/cache │ │
│ └─────────┬─────────┘ └───────────────┬──────────────┘ │
│ │ │ │
│ │ ▼ │
│ │ ┌──────────────────────────────┐ │
│ │ │ dispatch-begin │ │
│ │ │ • register run with │ │
│ │ │ Test System IO │ │
│ │ │ • runs immediately before │ │
│ │ │ workers to minimise the │ │
│ │ │ inactivity-timeout window │ │
│ │ └───────────────┬──────────────┘ │
│ │ │ │
│ └────────────────────┬─────────────────────┘ │
│ ▼ │
│ ┌─────────────────────────────────────────────────────────────────┐ │
│ │ workers (matrix, fail-fast: false) │ │
│ │ Cypress full: N=40 | Playwright full: N=10 │ │
│ │ │ │
│ │ each worker, in parallel: │ │
│ │ 1. sparse-checkout actions + full checkout-repo │ │
│ │ 2. setup-node │ │
│ │ 3. restore caches ◄─── actions/cache (from prep-deps) │ │
│ │ (fail-on-cache-miss: true) │ │
│ │ 4. cloud-init + start-server (docker compose stack) │ │
│ │ 5. prepare-cypress | prepare-playwright (run setup project) │ │
│ │ 6. dispatch-run ──────────────────────────────────┐ │ │
│ │ (pulls specs from Test System IO, runs locally, │ │ │
│ │ posts result, loops until queue is empty) │ │ │
│ │ 7. cloud-teardown │ │ │
│ └────────────────────┬────────────────────────────────────┼───────┘ │
│ │ │ │
│ ▼ │ │
│ ┌─────────────────────────────────────────────────┐ │ │
│ │ report │ │ │
│ │ • pull aggregated results from Test System IO │ ◄────┘ │
│ │ • post commit status │ │
│ │ • send webhook notification │ │
│ └─────────────────────────────────────────────────┘ │
└────────────────────────────────────────────────────────────────────────┬─┘
│
┌───────────────────────────▼───┐
│ Test System IO (external) │
│ • spec-level dispatch │
│ • result aggregation │
│ • retry orchestration │
└───────────────────────────────┘
Key properties
- Spec-level vs. job-level parallelism. The matrix sizes the runner pool; Test System IO does the spec assignment. Slow specs don't block a worker — fast workers keep pulling the next spec from the queue.
- Cache-only workers.
prep-depsinstalls once per workflow run and saves toactions/cache. Every worker restores withfail-on-cache-miss: trueand runs zeronpm ci. Eliminates the 40-wayEEXIST/ENOENTrace in npm's shared cacache writer. - dispatch-begin runs late. It depends on
prep-depsso the gap between Test System IO run registration and the first worker callingdispatch-runis just per-worker setup (~3–5 min). Registering earlier risks the run timing out before any worker checks in, bulk-failing every spec. - Playwright slim slice. Playwright only consumes
@mattermost/clientand@mattermost/typesfrom webapp, so prep-deps caches just those two packages' builtlib/andnode_modules(~10–30 MB) instead of the fullwebapp/node_modulestree (~1–2 GB). - Browser/binary caches. Cypress caches
~/.cache/Cypress(cypress binary lives outside node_modules); playwright caches~/.cache/ms-playwright(chromium only). Both keyed on the framework's lockfile so they invalidate on version bumps. - No retry plumbing in the template. Test System IO handles per-spec retries; the workflow only sees aggregated results.
Pipeline 1: PR (e2e-tests-ci.yml)
Runs E2E tests for every PR commit after the enterprise docker image is built. Fails if the commit is not associated with an open PR.
Trigger chain:
PR commit ─► Enterprise CI builds docker image
─► Argo Events detects "Enterprise CI/docker-image" status
─► dispatches e2e-tests-ci.yml
For PRs from forks, body.branches may be empty so the workflow falls back to master for workflow files (trusted code), while commit_sha still points to the fork's commit.
Jobs: 2 (cypress + playwright), each does smoke -> full
Commit statuses (4 total):
| Context | Description (pending) | Description (result) |
|---|---|---|
e2e-test/cypress-smoke|enterprise |
tests running, image_tag:abc1234 |
100% passed (1313), 440 specs, image_tag:abc1234 |
e2e-test/cypress-full|enterprise |
tests running, image_tag:abc1234 |
100% passed (1313), 440 specs, image_tag:abc1234 |
e2e-test/playwright-smoke|enterprise |
tests running, image_tag:abc1234 |
100% passed (200), 50 specs, image_tag:abc1234 |
e2e-test/playwright-full|enterprise |
tests running, image_tag:abc1234 |
99.5% passed (199/200), 1 failed, 50 specs, image_tag:abc1234 |
Manual trigger (CLI):
gh workflow run e2e-tests-ci.yml \
--repo mattermost/mattermost \
--field pr_number="35171"
Manual trigger (GitHub UI):
- Go to Actions > E2E Tests (smoke-then-full)
- Click Run workflow
- Fill in
pr_number(e.g.,35171) - Click Run workflow
On-demand testing
For on-demand E2E testing, the existing triggers still work:
- Comment triggers:
/e2e-test,/e2e-test fips, or withMM_ENVparameters - Label trigger:
E2E/Run
These are separate from the automated workflow and can be used for custom test configurations or re-runs.
Pipeline 2: Merge (e2e-tests-on-merge.yml)
Runs E2E tests after every push/merge to master or release-* branches.
Trigger chain:
Push to master/release-*
─► Argo Events (mattermost-platform-package sensor)
─► delivery-platform/.github/workflows/mattermost-platform-delivery.yaml
─► builds docker images (enterprise + fips)
─► trigger-e2e-tests job dispatches e2e-tests-on-merge.yml
Jobs: 4 (cypress + playwright) x (enterprise + fips), smoke skipped, full tests only
Commit statuses (4 total):
| Context | Description example |
|---|---|
e2e-test/cypress-full|enterprise |
100% passed (1313), 440 specs, image_tag:abc1234_def5678 |
e2e-test/cypress-full|fips |
100% passed (1313), 440 specs, image_tag:abc1234_def5678 |
e2e-test/playwright-full|enterprise |
100% passed (200), 50 specs, image_tag:abc1234_def5678 |
e2e-test/playwright-full|fips |
100% passed (200), 50 specs, image_tag:abc1234_def5678 |
Manual trigger (CLI):
# For master
gh workflow run e2e-tests-on-merge.yml \
--repo mattermost/mattermost \
--field branch="master" \
--field commit_sha="<full_commit_sha>" \
--field server_image_tag="<image_tag>"
# For release branch
gh workflow run e2e-tests-on-merge.yml \
--repo mattermost/mattermost \
--field branch="release-11.4" \
--field commit_sha="<full_commit_sha>" \
--field server_image_tag="<image_tag>"
Manual trigger (GitHub UI):
- Go to Actions > E2E Tests (master/release - merge)
- Click Run workflow
- Fill in:
branch:masterorrelease-11.4commit_sha: full 40-char SHAserver_image_tag: e.g.,abc1234_def5678
- Click Run workflow
Pipeline 3: Release Cut (e2e-tests-on-release.yml)
Runs E2E tests after a release cut against the published release images.
Trigger chain:
Manual release cut
─► delivery-platform/.github/workflows/release-mattermost-platform.yml
─► builds and publishes release docker images
─► trigger-e2e-tests job dispatches e2e-tests-on-release.yml
Jobs: 4 (cypress + playwright) x (enterprise + fips), smoke skipped, full tests only. Team edition planned for future.
Commit statuses (4 total, 6 when team is enabled):
Descriptions include alias tags showing which rolling docker tags point to the same image.
RC example (11.4.0-rc3):
| Context | Description example |
|---|---|
e2e-test/cypress-full|enterprise |
100% passed (1313), 440 specs, image_tag:11.4.0-rc3 (release-11.4, release-11) |
e2e-test/cypress-full|fips |
100% passed (1313), 440 specs, image_tag:11.4.0-rc3 (release-11.4, release-11) |
e2e-test/cypress-full|team (future) |
100% passed (1313), 440 specs, image_tag:11.4.0-rc3 (release-11.4, release-11) |
Stable example (11.4.0) — includes MAJOR.MINOR alias:
| Context | Description example |
|---|---|
e2e-test/cypress-full|enterprise |
100% passed (1313), 440 specs, image_tag:11.4.0 (release-11.4, release-11, 11.4) |
e2e-test/cypress-full|fips |
100% passed (1313), 440 specs, image_tag:11.4.0 (release-11.4, release-11, 11.4) |
e2e-test/cypress-full|team (future) |
100% passed (1313), 440 specs, image_tag:11.4.0 (release-11.4, release-11, 11.4) |
Manual trigger (CLI):
gh workflow run e2e-tests-on-release.yml \
--repo mattermost/mattermost \
--field branch="release-11.4" \
--field commit_sha="<full_commit_sha>" \
--field server_image_tag="11.4.0" \
--field server_image_aliases="release-11.4, release-11, 11.4"
Manual trigger (GitHub UI):
- Go to Actions > E2E Tests (release cut)
- Click Run workflow
- Fill in:
branch:release-11.4commit_sha: full 40-char SHAserver_image_tag: e.g.,11.4.0or11.4.0-rc3server_image_aliases: e.g.,release-11.4, release-11, 11.4(optional)
- Click Run workflow
Commit Status Format
Context name: e2e-test/<phase>|<edition>
Where <phase> is cypress-smoke, cypress-full, playwright-smoke, or playwright-full.
Description format:
-
All passed:
100% passed (<count>), <specs> specs, image_tag:<tag>[ (<aliases>)] -
With failures:
<rate>% passed (<passed>/<total>), <failed> failed, <specs> specs, image_tag:<tag>[ (<aliases>)] -
Pending:
tests running, image_tag:<tag>[ (<aliases>)] -
Pass rate:
100%if all pass, otherwise one decimal (e.g.,99.5%) -
Aliases only present for release cuts
Failure behavior
- Smoke test fails: Full tests are skipped, only smoke commit status shows failure
- Full test fails: Full commit status shows failure with pass rate
- Both pass: Both smoke and full commit statuses show success
- No PR found (PR pipeline only): Workflow fails immediately
Smoke-then-Full Pattern
Each wrapper (Cypress/Playwright) follows this flow:
generate-build-variables (branch, build_id, server_image)
─► smoke tests (1 worker, minimal docker services)
─► if smoke passes ─► full tests (20 workers cypress / 1 worker playwright, all docker services)
─► report (aggregate results, update commit status)
Test filtering
| Framework | Smoke | Full |
|---|---|---|
| Cypress | --stage=@prod --group=@smoke |
--stage="@prod" --excludeGroup="@te_only,@cloud_only,@high_availability" --sortFirst=... --sortLast=... |
| Playwright | --grep @smoke |
--grep-invert "@smoke|@visual" |
Worker configuration
| Framework | Smoke Workers | Full Workers |
|---|---|---|
| Cypress | 1 | 20 |
| Playwright | 1 | 1 (uses internal parallelism via PW_WORKERS) |
Docker services
| Test Phase | Docker Services |
|---|---|
| Smoke | postgres inbucket |
| Full | postgres inbucket minio openldap elasticsearch keycloak |
Tagging Smoke Tests
Cypress
Add @smoke to the Group comment at the top of spec files:
// Stage: @prod
// Group: @channels @messaging @smoke
Playwright
Add @smoke to the test tag option:
test('critical login flow', {tag: ['@smoke', '@login']}, async ({pw}) => {
// ...
});
Shared Wrapper Inputs
The wrappers (e2e-tests-cypress.yml, e2e-tests-playwright.yml) accept these inputs:
| Input | Default | Description |
|---|---|---|
server_edition |
enterprise |
Edition: enterprise, fips, or team |
server_image_repo |
mattermostdevelopment |
Docker namespace: mattermostdevelopment or mattermost |
server_image_tag |
derived from commit_sha |
Docker image tag |
server_image_aliases |
(empty) | Alias tags shown in commit status description |
ref_branch |
(empty) | Source branch name for webhook messages (e.g., master or release-11.4) |
The automation dashboard branch name is derived from context:
- PR:
pr-<pr_number>(e.g.,pr-35205) - Master merge:
master - Release merge:
release-<version>(e.g.,release-11.7) - Fallback:
commit-<short_sha>(e.g.,commit-abc1234)
Master and release runs use the real branch verbatim so the
/reports/{repo}/{branch} view aggregates every build on that branch
into one history. PR runs use a pr-<n> token because there's no real
branch to use; the commit-only fallback uses a commit-<short_sha>
token (defensive — no current caller hits this path).
The test type suffix (-smoke or -full) is appended by the template.
The server image is derived as:
{server_image_repo}/{edition_image_name}:{server_image_tag}
Where edition_image_name maps to:
enterprise->mattermost-enterprise-editionfips->mattermost-enterprise-fips-editionteam->mattermost-team-edition
Webhook Message Format
After full tests complete, a webhook notification is sent to the configured REPORT_WEBHOOK_URL. The results line uses the same commit_status_message as the GitHub commit status. The source line varies by pipeline using report_type and ref_branch.
Report types: PR, MASTER, RELEASE, RELEASE_CUT
PR
:open-pull-request: mattermost-pr-35205
:docker: mattermostdevelopment/mattermost-enterprise-edition:abc1234
100% passed (1313), 440 specs | full report
Merge to master
:git_merge: abc1234 on master
:docker: mattermostdevelopment/mattermost-enterprise-edition:abc1234_def5678
100% passed (1313), 440 specs | full report
Merge to release branch
:git_merge: abc1234 on release-11.4
:docker: mattermostdevelopment/mattermost-enterprise-edition:abc1234_def5678
100% passed (1313), 440 specs | full report
Release cut
:github_round: abc1234 on release-11.4
:docker: mattermost/mattermost-enterprise-edition:11.4.0-rc3
100% passed (1313), 440 specs | full report
The commit short SHA links to the commit on GitHub. The PR number links to the pull request.
Related Files
e2e-tests/cypress/- Cypress test suitee2e-tests/playwright/- Playwright test suitee2e-tests/.ci/- CI configuration and environment filese2e-tests/Makefile- Makefile with targets for running tests, generating cycles, and reporting