Vizregress: Visual Regression Detection with Selenium WebDriver

Vizregress: Visual Regression Detection with Selenium WebDriver

23 November 2013

On github you'll find a project I created called Vizregress. It contains my experimentation code in an attempt to solve something quite complex, Visual Regression Testing.

Browser Automation Today

I am proud of our automation system at numero. We have a VM infrastructure that will deploy our flagship product called numero interactive, configure it and run automated browser tests against it.

The beauty of automation tests is that we can ensure that features we implement will continue to function throughout our release cycle in different versions of numero interactive.
When comparing it with manual test scripts, we also help alleviate human error at following scripts.

The browser automations are also video recorded so that we can visually review failures.

Visual Regression

What isn't covered is the detection of visual regression. Selenium can continue to input values and click buttons based on CSS expressions ignorant that visually they look appalling.

Example

A simple user details screen has a layout like so:

Login

But a simple CSS change to another screen could cause this:

Login Wrong

Selenium will still address the textboxes and button because their selectors are the same. However; visually; this would be a regression. So, how do we spot this?

Vizregress tries to help

This is where Vizregress tries to help.
It's an experimentation where I attempt to use the browser to take a screenshot and compare it against an approved expected image. I use AForge Imaging that performs a pixel by pixel analysis of the images, but, this is of course not without it's challenges.

Highly fragile

  • Different browsers render differently.
  • Browser updates: A challenge with capturing browser images is that their rendering engines are changed over time with slight optimisations. This means captured images can change change subtlety resulting in image comparison failures.
  • Browser window size & desktop resolution: The window size affects the size of the images. Comparison has to be made with a constant resolution and window size.
  • Operating System: Browsers behave differently between operating systems.
  • Responsive design & different media representations: A web page is rendered differently on a mobile device than a tablet.
  • Globalisation: Text is translated and therefore invalidate capture images.

These key differences mean that captured images are needed per browser, per resolution, per operating system and per locale? This is where it gets complex for me and this area is in constant review because I am wondering whether or not using Phantomjs might solve most of my requirements.
A first step might be just to remove the complexities of different browsers and start by using Phantomjs...

.NET Automation CSharp Selenium Testing