Website Quality Assurance Testing – Our Approach

Testing is a crucial stage in the development of a website. On every project we set aside dedicated time in the schedule and budget for carrying out Quality Assurance (QA).

QA is not just about testing whether something works, but also that it meets certain standards in aspects such as performance and accessibility. Through our testing process we are trying to ensure that the user experience is the best it can be and that any issues are identified by us before they become a problem for real users.

We employ a combination of manual and automated tests as part of a methodical and repeatable process that helps ensure consistency in the websites we create. This happens in two phases, the first prior to launch on a staging copy of a website and second immediately after launch on the live site itself.

Pre-launch tests

Ideally we want to catch any avoidable issues before a website goes live. So our pre-launch testing covers everything from browser testing to content checks and performance benchmarking. There are seven main types of testing that we run including…

Browser Testing

Modern browsers are a lot more consistent in how they render webpages, but there are still some quirks. Using a combination of manual evaluation and automated screenshot comparisons we ensure that a website works well in the most recent releases of Chrome, Safari, Firefox, Edge and Internet Explorer. Where relevant we will also test back to older browser versions (we’re looking at you Internet Explorer) following the principle of Progressive Enhancement.

Mobile Device Testing

We use a combination of real devices and emulators to test on a range of mobile operating systems and browsers. We also run ‘mobile-friendliness’ checks on each page type which identify common issues such as touch targets that are too close together or font sizes that are too small. However The most useful insights usually come from a walk-through of he main user journeys using a real mobile device. This can quickly identify any pain points or interactions that don’t feel natural.

Content Testing

This is one area where automated testing tools really come into their own, allowing us to quickly scan an entire site for common content problems such as broken links, missing images or alt text and spelling errors. Because we use set Lorem Ipsum text for any test/placeholder content we can also scan specifically for that to avoid those pages making it into the live site.

Search Engine Optimisation

Making sure that a website is in the best shape to be received positively and indexed accurately by the search engines is crucial to a successful launch. We use scanning tools that mimic the way a search engine spider will crawl a site to identify any pages that aren’t being reached or where content is being blocked. At the same time we can quickly identify and resolve common issues such as duplicate, missing or malformed page titles and descriptions. Manual checks are carried out to ensure that the robots.txt file is configured properly and that a valid XML sitemap is present.

With the re-launch of an existing website the redirection of old urls is usually a factor to consider and one that is important to get right to best-protect your site’s existing ranking. We use a combination of automated and manual methods to create a list of all the valid URLs in the old site so that we can check that all those are still directly reachable or present an appropriate 301 redirect if they have been intentionally removed or moved and avoid issues like redirection chains.

Performance Benchmarking

The performance of a site has a significant impact on the overall user experience. That’s why we carry out site-wide performance tests to identify any anomalies in page size or load time. A sample of each page type is also benchmarked in more detail manually. This aids comparison between the development and live hosting environments and with/without optimisations such as page caching enabled. This is an example of an area where our Design Standards come into play with specific targets for page size, load time and the maximum number of http requests.

Security

It isn’t possible to completely secure a site against attack, but there are some common-sense precautions you can take to make a compromise less likely and to ensure that you can recover from any incident quickly. A lot of ‘hardening’ needs to take place on the hosting environment, but for sites which run a content management system like WordPress there are some things to tick off prior to launch. These include an audit of the users who have access and the strength/uniqueness of their passwords. Ensuring that the WordPress core and any associated plug-ins are up to date is another simple step towards better security, as is removing any plug-ins that aren’t essential, as each represents a potential route to compromise a site. Including some form of threat protection solution to block brute-force attacks and identify potential intrusions can offer additional protection and these systems should be tested in the development environment.

The other half of pragmatic security is to have a plan for recovery, which usually starts with a good backup. Here we want to test both that backups are running, but also that they include everything that is needed to perform the restoration of a site.

Accessibility

We’ve written in more depth before about the website accessibility principles that inform our design work and help ensure that best practice gets baked in early on. When it comes to testing there are variety of things that we look for including the colour contrast on key interface elements, missing or duplicate alt text on images and the spacing/sizing of touch targets. Seeing how a page will render to screen readers and ensuring that content is still accessible without images, CSS or javascript being loaded is also important.

Post-launch tests

As soon as a new site goes live our post-launch testing immediately kicks in. It combines a whole raft of one-off checks with some selection on-going monitoring. The initial focus is on ensuring that the site has successfully migrated to the live hosting environment and then we shift to collecting useful performance data that will help quickly identify any potential user experience issues.

Performance Tests

One of the sets of tests which is repeated from pre-launch is performance benchmarking. Here we are mainly looking for any differences between the development and live environment, but also how the site holds up under the weight of real traffic. Then there are usually two types of on-going test that we run, one is an automated performance benchmark on a sample page, which will run every 30 minutes over the course of a week. The other is what is called Real User Monitoring (RUM) which will keep track of the load times real users experience and provide some insights into how those are affected by device type, geography and the content being accessed.

Content

The same batch of automated content tests are run at the point of launch and data such as word-counts for individual pages can be used to help identify any differences between the pre and post-launch versions of a site. On-going automated testing is also used across the first week to run hourly screenshot comparisons to check that pages are rendering consistently over time.

Search Engine Optimisation

At launch our main concerns are ensuring that search engines can successfully crawl the stuff we want them to see and are blocked from anything we don’t want indexed. This includes testing the robots.txt file is configured correctly, validating and submitting an XML sitemap and testing any 301 redirects again. Over the first few days it is a good idea to keep an eye on the 404 logs to see if there are any recurring problems accessing certain content. Google’s Search Console provides some useful additional insights into any crawl errors there search engine spiders have come across and helpful HTML suggestions. Also flag-up issues around things like mobile friendliness and the content in any Accelerated Mobile Pages.

Analytics

Reporting tools like Google Analytics usually aren’t switched in until launch to avoid muddying the data, but at that point they become essential for helping to report back on how a new site is performing. Most services provide some form of validation tool to confirm that they have been implemented correctly. Google’s Tag Assistant for example is a good one-stop-shop for checking that various Analytics, AdWords and other Google services are firing correctly and common issues like duplicate tags are avoided.

SSL Validation & DNS Health

Some technical aspects can only be fully tested in a live environment. Those include ensuring that a site’s SSL certificate has been deployed correctly – both at the server level and for mixed-content issues within the site itself – and verifying that the domain name and associated DNS records aren’t throwing up any errors.

Session Recording

A thorough walk-through of all the main user journey’s in a website can be invaluable for identifying any bugs that have slipped through from previous rounds of testing, but we can enlist the help of real users too. During the first weeks of a new site being live we like to capture a good sample of real user sessions to see how people are moving between pages and whether they are hitting any obvious road-blocks. It’s a bit like remote usability testing, but with the added benefit that users aren’t altering their behaviour under test conditions. The same session recording tool also reports back useful information about ‘dead’ clicks (where users are attempting to interact with part of the user interface which isn’t clickable) and ‘rage’ clicks (where a lot of clicks are recorded in quick succession).

Test, test and test again

The length of time spent on quality assurance will depend on the size of a website and its complexity. For a very small site with just a handful of pages it could be as little as three hours. For large sites or those with more app-like functionality testing can easily stretch into days. Our approach to quality assurance is pragmatic, we know that we can’t necessarily ensure that no bugs slip through, but we can significantly reduce the likelihood and scale of those bugs affecting the user experience of a site. So although our 30-point testing plan is an effective tool in ensuring the quality of what goes live we also include three months complimentary bug support as part of the post-launch input we provide.