When was the last time you found a design issue in your application that you know had been fine before?

Traditional testing methodologies, while important, can sometimes be too limited in scope to catch issues with the user interface. They often focus on functionality, but can miss subtle visual regressions that negatively impact user experience. Imagine a misplaced button, misaligned text, or an unexpected color change – these seemingly small issues can erode user trust,create frustration, and make users second-guess the reliability of your product. In a competitive landscape, where seamlessness is non-negotiable, overlooking these details is a risk you can’t afford to take.

That's where visual testing comes in. By automatically comparing screenshots of our application across different screen sizes and browsers, we can catch these visual regressions before they reach our users. This proactive approach drastically reduces the risk of deploying faulty software and maintains the high quality our users expect. How do we ensure this process runs efficiently? Let’s break it down.

Playwright: robust end-to-end testing

Our end-to-end (E2E) testing strategy relies heavily on Playwright. Its cross-browser compatibility and powerful automation capabilities allow us to thoroughly test critical user flows and ensure core functionalities are working as expected. This provides a solid foundation for our overall testing strategy.

Most importantly, our team’s dedication to fix any flaky tests, paired with Playwright’s powerful assertions that automatically recheck and await status changes, has significantly improved the reliability of our test suite. The combination didn’t just improve resilience – it transforms our tests into a reliable backbone for consistent performance. This leads to a faster and more trustworthy development process that we can count on.

After all, no developer will listen to the boy who cried wolf too often, thus making it crucial for us that our E2E tests execute quickly – we are currently striving for under 5 minutes for the entire E2E test suite – and reliably.

Happo: supercharging visual testing

To complement our Playwright E2E tests, we've integrated Happo for visual regression testing. Happo seamlessly integrates into our existing workflow, capturing screenshots at various points during our E2E tests. It then compares these screenshots against a baseline, highlighting any visual discrepancies. This allows us to quickly identify and address unintended visual changes, ensuring a consistent and polished user experience across all platforms.

Happo doesn't constantly compare feature branches directly against the always-moving HEAD of the main branch. Instead, it uses a commit-based baseline system. As changes are merged into your default branch, baseline reports are generated. The report that was created at the git merge base of the feature branch is then used as the baseline for feature branch comparisons, highlighting any immediate visual differences. Subsequent builds on the feature branch continue to compare against the merge base baseline, allowing developers to isolate the visual impact of their changes without the noise of unrelated updates merged into the main branch in the meantime.

As the feature branch progresses, the screenshots for that branch evolve with each build. Upon merging into the main branch, the merged branch's screenshots become a new baseline for main. 

This approach ensures that comparisons are always made against the most relevant point in the code's history, providing clear and accurate visual diffs. It avoids the confusion and noise that would arise from constantly comparing a feature branch against a rapidly changing main branch. Each feature branch effectively has its own visual history tracked by Happo, simplifying the identification and management of visual regressions.

Tightly integrated into GitHub

Happo checks are tightly integrated into GitHub as status checks. At re:cap, they are configured to not block any deployments – while failing E2E tests would do exactly that – but visual enough to stop a developer from merging their pull request if Happo shows us an unexpected difference between our main branch and the newly developed feature.

A real-world example: the case of the overflowing notification

Recently, during the development of a new feature, Happo alerted us to a wrongly colored text and an issue with an overflowing user notification. Our Playwright E2E tests passed successfully, as the app’s functionality remained intact. However, Happo detected the unwanted changes in the text’s color and positioning, which would have been noticeable and disruptive to users.

Thanks to Happo's visual comparison, we were able to identify and fix the issue before it reached production, preventing a potentially negative user experience. This is just one example of how Happo has helped us maintain a high level of quality while continuing to ship features at a rapid pace.

"Happo gives us the confidence that we are not introducing any unwanted visual changes in our pull requests. Something that pure end-to-end tests cannot always cover." Adrian Burkhart, Fullstack Engineer at re:cap.

By combining the power of Playwright for functional testing with Happo's visual testing capabilities, we've created a robust testing strategy that empowers us to ship new features quickly and confidently. This not only benefits our users but also allows our engineering team to focus on building great features, knowing they have a safety net to catch visual regressions before they impact our users.