Improve Visual Regression Testing

Status: In Progress · Proposal

August 2026¶

Summary of Work Completed¶

No major new developments - the budget for this proposal was exhausted. We continued the work on flaky tests supporting the development of custom linting rules that encourage the use of galata helpers to prevent the introduction of flaky tests in the future - see the report for Custom Linting for JupyterLab for full details; here we only summarize the flaky tests fixed and the work supporting them.

Of note:

• the flakiness reports are posted weekly on jupyterlab/jupyterlab#19153 and we continue monitoring them

• we moved the UI test report comment workflow that we described in the previous report to a shared maintainer-tools repository to encourage reuse (jupyterlab/maintainer-tools#301, jupyterlab/jupyterlab#19290)

• to help understand the impact of UI test changes, the video is now preserved for tests modified in the PR (jupyterlab/jupyterlab#18865)

We continued fixing flaky screenshot tests:

• Wait for launcher icons before screenshots (jupyterlab/jupyterlab#19076)

• Stabilize notebook scroll anchoring Galata test (jupyterlab/jupyterlab#19151)

• Fix active cell prompt and source blink in notebook tools & prevent a memory leak (jupyterlab/jupyterlab#19168)

• Stabilize Open in Terminal Galata test on Firefox (jupyterlab/jupyterlab#19152)

• Stabilize debugger Galata setup (jupyterlab/jupyterlab#19169)

We track most of the issues and pull requests associated with this proposal in this public GitHub project.

Deliverables/Milestones¶

Completed as discussed in the previous reports.

Challenges or Risks¶

We exhausted the budget, however some flaky snapshots remain and maintenance is still needed to address newly discovered edge cases around the new snapshot update workflows.

Budget Update¶

As of 25th August 2026 we used 16,182 USD out of 16,182 USD budget.

Next Steps¶

While we exhausted the budget, we continue making the visual regression tests more reliable.

In particular, we are carrying over the lessons learned from this work and aim to codify the newly established best practices via custom linting rules, as tracked in the report for Custom Linting for JupyterLab.

June 2026¶

Summary of Work Completed¶

We completed the logistically challenging transition of galata to use fixed fonts (final PR jupyterlab/jupyterlab#18535). This required coordination across a number of PRs and updating a significant set of snapshots. This change:

• allowed perfect reproduction of snapshots on local machines, unblocking local iteration on tests, including shortening the loop for debugging flaky tests which enables both contributors and AI agents to get a tight feedback loop

• lessened the pain of future runner updates which change system fonts with major OS updates

We integrated the example test snapshots with the new workflow for updating snapshots (jupyterlab/jupyterlab#19007). The workflows were further improved:

• the PR adding a badge with the count of failing/flaky tests was merged jupyterlab/jupyterlab#18798

• the re-runs handling was improved jupyterlab/jupyterlab#18801

• a trigger to run a subset of UI test (once or more) from GitHub Actions UI was added (jupyterlab/jupyterlab#18800)

• documentation was updated jupyterlab/jupyterlab#18886

We fixed multiple flaky screenshots:

• Stabilize subpixel editor line shift on Welcome screenshot jupyterlab/jupyterlab#19039

• Avoid debugger toolbar button flicker during kernel startup jupyterlab/jupyterlab#19006

• Fix scrolling to heading in ToC, workspace opening and flaky UI tests jupyterlab/jupyterlab#18961

• Fix flaky Notebook No Kernel › Should maintain no kernel state when adding cells test jupyterlab/jupyterlab#18856

• Remove explicit timeouts in terminal playwright tests jupyterlab/jupyterlab#18610

We track most of the issues and pull requests associated with this proposal in this public GitHub project.

Deliverables/Milestones¶

Configure sharding for playwright and its test reporters in JupyterLab and Notebook

Done

Create a GitHub bot workflow generating PRs or commits for snapshot updates and comments with snapshot comparisons

Done.

Improving reliability of snapshots across setups (fonts, etc)

Done

Improve the default reporting configuration

Done

Challenges or Risks¶

We exhausted the budget, however some flaky snapshots remain and maintenance is still needed to address newly discovered edge cases around the new snapshot update workflows.

Budget Update¶

As of 23rd June 2026 we used 16,182 USD out of 16,182 USD budget.

Next Steps¶

While we exhausted the budget, we continue making the visual regression tests more reliable. In particular, we are carrying over the lessons learned from this work and aim to codify the newly established best practices via custom linting rules, as tracked in the report for Custom Linting for JupyterLab.

April 2026¶

Summary of Work Completed¶

• Visual regression test reports now can be previewed directly from browser, thanks to

  • a reusable action to inline playwright reports (jupyterlab/maintainer-tools#279)

  • integration with JupyterLab CI (jupyterlab/jupyterlab#18628)

  • new workflow to expose a badge with count of failing/flaky tests in the first bot comment (jupyterlab/jupyterlab#18798, in review)

• Documentation regression tests job was merged into the updated visual regression workflow (jupyterlab/jupyterlab#18629), simplifying the setup and allowing for sharding support (faster execution) and inclusion of all documentation snapshots in the new comment-driven snapshot update workflow

• We improved the coverage of the new snapshot update workflow to cover JSON snapshots (jupyterlab/jupyterlab#18658) and documentation debugger snapshots (jupyterlab/jupyterlab#18688)

• We continued to improve performance of the regression testing (e.g. jupyterlab/jupyterlab#18568) and bring down the number of flaky tests

• We enforced usage of fonts from pre-specified variables for both the benefit of the user and consistency of snapshots in visual testing when ran on developer machines

  • Use a font size and family specified in CSS for DataGrids jupyterlab/jupyterlab#18542

  • Fix font-family application across UI components jupyterlab/jupyterlab#18541

  • Enforce that certain CSS rules use variables jupyterlab/jupyterlab#18614

We track some of the issues and pull requests associated with this proposal in this public GitHub project.

Deliverables/Milestones¶

Configure sharding for playwright and its test reporters in JupyterLab and Notebook

Done

Create a GitHub bot workflow generating PRs or commits for snapshot updates and comments with snapshot comparisons

Done, final improvements awaiting

Improving reliability of snapshots across setups (fonts, etc)

In progress

Improve the default reporting configuration

Done

Challenges or Risks¶

The transition to fixed fonts on CI requires updating a large number of snapshots, which will introduce a temporary pain for authors of other PRs that modify screenshots.

To ensure the impact on other PR authors is minimal we delayed this transition while:

• working on further improving the snapshot update workflow to ensure that the path to recovery is smooth

• trying to assist with review and merge of any PRs that modify the screenshots to reduce the number of conflicts (in our own time)

With very little remaining budget it will be challenging, but we are determined to make this transition happen before the 4.6.0 beta release.

Budget Update¶

As of 26th April 2026 we used 15,547.60 USD out of 16,182 USD budget.

Next Steps¶

• Completion of fixing the fonts to ensure cross-platform reproducibility

February 2026¶

Summary of Work Completed¶

• Sharding for faster visual regression/integration tests (merged)

• Fixes to many flaky tests and tests which were failing locally (many PRs merged), but more work to be done

• Workflow for updating visual regression snapshots from artifacts (merged)

We track some of the issues and pull requests associated with this proposal in this public GitHub project.

Deliverables/Milestones¶

Metric 1. The time to run playwright tests on the JupyterLab repo should reduce from current 45 minutes down to ~20 minutes.

Achieved: the runtime is now on the order of 15 minutes.

Metric 2. The snapshot updates should no longer require a multi-step manual process that can be performed only by maintainers but instead allow every contributor to trigger an update of the snapshots. It should be near instantaneous (<5 minutes) once the regression testing is completed by reusing the published artifacts.

Achieved, although:

• the documentation snapshots are not yet covered

  • the workflow runs require approval while it matures

Metric 3. >90% of tests should pass when run on a non-Ubuntu machine

Work in progress. >95% tests pass on CI between different Ubuntu versions.

Challenges or Risks¶

We did not find a way to make snapshots identical without requiring an update to the existing ones. This means that while the goal can be achieved, there will be an initial cost to migrate (update all snapshots). This should not be too difficult and would be required when migrating to new Ubuntu runners anyways.

Budget Update¶

As of 20th Feb 2026 we used 7,507.50 USD out of 16,182 USD budget.

Next Steps¶

• Fixing the fonts to ensure cross-platform reproducibility

• Further fixes to snapshot differences caused by local installation differences

• Merging snapshot jobs for documentation with the sharded jobs for other packages