A/B split testing agency advice
How to save money, keep it simple & avoid the pitfalls when hiring an a/b split testing agency
Get our free best buy guide
Let us send you our regularly updated list of top digital agencies in your local area
A/B testing can be one of the most rewarding digital marketing practices. The ability to plan, design, build and run live tests on your website and to see positive results happening right in front of you has brought A/B testing very much into vogue.
But it doesn’t have to be the preserve of big business.
Who’s this guide for? Anyone who needs to understand their digital strategy options and be aware of the common issues so they can make informed choices.
What is A/B split testing?
A/B testing is the practice of showing one or more variations of a webpage to your website visitors. While these variations are being shown, data is tracked against the “Control” (or original version of the page) and the “Variant” (or new version of the page) to prove which delivers the greatest business benefit.
When is A/B split testing used and why?
Have you ever had a debate with a colleague about the kind of changes that should be made to your website? Has it been difficult to come to a compromise on the action that should be taken? This is a perfect time to employ A/B testing.
Using A/B testing, you can try out multiple ideas to determine which is best for your company. This is not only useful for delivering business benefit but also helps to engage your colleagues in the overall business improvement process, without sacrificing the need to make decisions based on data.
Why is choosing the right A/B split testing agency/freelancer so difficult?
The difficulty in choosing the right supplier is that A/B testing requires three very different skill types, as well as the software to deliver it. With these four variables in play, there can be a lot of decisions to make.
Skill 1: Knowing what to test
Understanding the right areas to test is essential in making your A/B testing programme successful. This involves data analysis to identify opportunity areas, but also the creativity to assess the situation from a visitor’s perspective in order to make strong recommendations. This can be tested during the pitching process, by insisting on a bespoke demonstration.
Skill 2: Building a test
This is often taken for granted, undervalued and one of the biggest causes of problems. Good web developers can be difficult to find, and because they are very rarely involved in pitches, it can be difficult to gauge their abilities before you sign. Wherever possible, try to arrange a “techy-on-techy” discussion to provide some background. Test builds often cause friction as client and agency disagree on the amount of time it should take to get something delivered, so during the pitching process, try to get an understanding of lead times.
Skill 3: Analysing a test
Reading or copying numbers from a screen is not analysis; true analysis relies on this structure:
DATA > INSIGHT > RECOMMENDATION
DATA: What do the numbers say?
INSIGHT: What do the numbers mean?
RECOMMENDATION: What can we do with that insight?
So during the pitching process, you should expect any supplier to be able to demonstrate this level of understanding. And remember, a test is only as good as its analysis, so if they cannot analyse correctly, your business value is likely to be greatly diminished.
Software: Which is right for you?
If you are looking for a recommendation from a supplier on which tool(s) to use, then you have another variable to consider. Make sure you set out your requirements as clearly as you can, and when a recommendation is presented, ensure you scrutinise it carefully. Many agencies receive commissions as resellers of products and are therefore not necessarily unbiased! Plus, most SME-focused tools allow for a free trial period, so make sure you utilise that before you commit to anything.
Why is A/B split testing so important for a SME?
A/B testing is an opportunity to safeguard against potentially risky decisions. For businesses (and especially SMEs) to survive, they need to be adaptable to market conditions and be constantly striving to improve. That means change is a necessity. But with change comes risk, so any opportunity to mitigate that risk should be grasped with both hands.
On the other side of the coin, there’s a money-making opportunity too. By trying something new, based on solid, data-backed hypotheses, you should be able to start turning higher revenues from your site too. And that is something that every SME would like!
A/B split testing questions that can help
What are the best ways to save money?
Take advantage of free trial periods with any paid A/B testing tools and keep an eye out for the release of Google Optimize. The product is currently in beta to a select audience but is widely touted for general release this year. Similar to Google Analytics, the Optimize product will be free and designed to be used alongside GA, which will be a big bonus for the many thousands of SMEs already using the Analytics product.
Outside of tools, using proof of concept (or POC) periods with agencies or freelancers can be very useful. You agree on a short-term contract together (usually three months) and in that time, they need to prove themselves worthy of an annual deal.
Which areas should I be aware of that take the most amount of time?
Building and correctly analysing A/B tests takes longer than most clients realise. If you want to do it properly and introduce less risk to the business, it takes time. If you cut corners, then you can expect an expensive mistake further down the line. With that in mind, be wary of “super-quick” turnaround times and insist on getting a breakdown of what QA analysis has been carried out; accuracy is more important than speed.
What should I do first to make sure I pick the right digital partner?
For the software itself, it comes down to what you want to do with it. Most of the tools available are flexible enough to deliver both simplistic tests quickly without code knowledge and more technically complex ones through writing code. Depending on which test types are the most common for your business, some tools will be better than others. Most were built with one of those test types in mind and then expanded to include the other.
From a service perspective, think back to the skills section above. Any partner needs to be analytical, creative and technically able, as well as able to see the bigger strategic picture of your business. Test each of these areas as you go through the process of appointing a new supplier, and that should serve you well.
How much should I spend and on what?
Many of the software providers will charge based on the number of tests/total amount of traffic your website has, so make sure you factor that in when considering budgets. The tool should suit your needs (as mentioned above). Many of them are similar in their capabilities, so cost may well become a factor.
In most cases, A/B testing services are much more expensive than the cost of the software itself. It’s for a good reason; no tool is intelligent enough to stop you testing the wrong things. Therefore the human element of A/B testing is the most important aspect. Screen any potential suppliers well and maybe even look to introduce some performance-related-pay (PRP) if you can. You may well encounter resistance on that point, but doesn’t mean they won’t compromise if they want the business! However you structure your services agreement, just be ready for that to take 70%+ of your total budget.
What can I do myself, if anything?
Most of the tools available to SMEs can build A/B tests without code. If you do purchase a tool like that, you can get straight into building tests yourself in a visual (rather than code-based) editor. An element of technical knowledge is helpful of course, but not essential. Just ask someone more technically-minded to check it before you launch it on your site!
But while building your tests seems like something you could do yourself, there’s probably more value in honing the skills of what to test and why to test it – because therein lies the real business value.
What are the signs that my digital partner is ripping me off?
If you allow your partner to provide the A/B testing tool for you, you must check what it would cost if you paid the software company directly. Often agencies can get “agency rates” from the tool vendors because they bring in a certain level of business so they can offer you a good deal. Of course, they will need to mark that up to make a profit on it, so best to confirm with the vendor themselves what it would cost.
From the services side, you just need to keep an eye on test delivery times from the idea phase to the completion of a test. This allows you to understand the total number of tests you can expect to complete in a year/quarter. This is vitally important in estimating your return-on-investment (and therefore if they are ripping you off!). Assume that 2 out of 3 should be successful, and if they aren’t, you might want to look for a new consultancy team!
Pitfalls and problems – before you hire an A/B split testing consultancy
Not knowing your own data
The number of visitors/sales/registrations your website gets is essential in determining the viability of A/B testing for your business. There’s no hard and fast rule on what that volume is. Generally speaking, if you don’t get at least 200-250 “conversions” (whatever the main goal of the test is, often a sale) in a four-week period, you’re going to struggle to reach a mathematically meaningful outcome for a test.
It’s a discipline, not a journey
Done correctly, A/B testing should be an ongoing initiative, not a “project” with a defined end-date. If you want to improve your site, you must be willing to commit to a continual process of improvement. That means iterating on what you have as much as possible.
This is probably the most heinous of the pitfalls, as getting this wrong will result in the same amount of effort being put in, but very little in the way of value coming out!
To be effective, a test must be measured in the right way. If the measurement strategy is wrong, the “results” of the test will be meaningless. What is going to be measured and how it is going to be measured is generally worked out in the planning phase of the test. This is of paramount importance.
If it goes wrong, the best-case scenario is that you have “results” that should be discarded. This assumes that you have realised that they cannot be trusted. The worst-case scenario is that you make key business decisions based on unreliable data, and that can be catastrophic.
The golden rules
These Golden Rules apply to running A/B tests
Write a proper hypothesis
If your test hypothesis (the statement you are trying to prove or disprove) contains phrases like “what happens if…”, or “how it will impact…”, then it’s not a proper hypothesis. In fact, it’s a guess.
A hypothesis should state the change you are testing, what impact you expect that change to have on a visitor (and why), and what the resulting change in conversion/sales/registrations/key success metric is.
Are they affable, humorous and sociable? Do you think that you could have a relationship with them?
Know how you will measure & define success
Accurate and appropriate measurement is essential in A/B testing and shouldn’t be taken lightly. The results of your tests could have a substantial impact on your business in the future. Realistically defining what success looks like is also important, especially if you need to convince management that it’s the right thing to do. They will likely be focused on outputs, rather than inputs.
Produce detailed documentation
Document what you plan to test, why you plan to test it and how you’re going to track and define success. This is useful for ensuring that you think through your test in a logical fashion (and help you to isolate areas of uncertainty!). Also, your documentation should explain to a previously-uninvolved party what you are doing and why. This is particularly important if there’s a team involved in A/B testing within the business, or a previous practitioner leaves and their work is taken up by someone else.
If you’re going to test it, be ready to keep the changes permanently
You put something into a test because you believe it will have a positive impact. And if it does, you should be ready and willing to have whatever “it” is as a permanent feature of your website. If you aren’t willing to do that, then the test is irrelevant.
Also, consider any potential technical constraints to making it permanent. Often A/B testing tools will allow a level of manipulation and flexibility that CMS-powered websites do not. Always establish with your technical employees what is possible before you commit.
Launch tests in the morning, and not on Fridays
This is just common sense! Any A/B test delivers new code onto your website, and if something was missed in QA, it could cause problems with certain pages or processes within the site. It can even bring it down entirely in very rare cases.
So, by launching tests in the morning, and not on Fridays, you maximise the likelihood of spotting any issues as well as having the right staff available to react to them as well. Plus, management is likely to be grateful that they aren’t receiving calls on Saturday mornings telling them the site is down!
Make sure the test is monitored after going live
Just because an issue didn’t happen straight away, doesn’t mean that it won’t, so try to check in on the test data and the test page(s) itself at least once a day. Checking the data should ensure your test isn’t having a major negative impact (for too long unnoticed at least!). Checking the pages themselves should verify that everything looks right for your visitors.
This is something to be particularly aware of around the time that regular updates are made to your site. A/B tests rely on the underlying code of your webpages to function correctly. If that code changes, it can have a disastrous effect on your test. If you know a change will be made to a page you’re testing, make sure you pause the test in good time to avoid any potential issues.
Know how to stop a test in case of emergency
if something terrible does happen, make sure you know how to stop any live tests on the site. This knowledge should be shared by a number of individuals with the right authority to make decisions, but certainly not resting solely in the hands of one team member. If you only have one colleague who can log into your tool and stop something, you could be in real trouble if they’re on holiday or taken ill and your order process is down!
Quarterly Appraisals & KPIs
Most A/B testing tools include their own calculations to help you understand how your tests are progressing and whether they are successful or not. They also provide statistical analysis around the percentage likelihood that a repetition of the same test would lead to the same result. This can never be 100% of course (there’s no such thing as certainty in statistics!), but if it says 50%, that isn’t a “result” you should take a chance on. Look for 90-95%+ before you make any decisions. Don’t be tempted to stop the test early because you like the variation that’s winning right now!
Have a process and stick to it
This helps everyone involved. A/B testing tends to involve a wide variety of people due to the skill-sets needed. Sticking to a process will help everyone to understand their responsibilities and place within the team.
If you work with an agency or freelancer, they may well have their own processes that they want you to follow. If you don’t understand why they do something a certain way, you should ask them. Most experienced practitioners will have good reasons why they do things a different way.
From every test should come more tests
The analysis of a test hypothesis should result in at least one more hypothesis, perhaps multiple hypotheses. The continual process of test-and-learn is what drives the improvements necessary to stay on top in this ever-evolving digital world.
It also means that any result of a test (positive or negative), is a lesson learned and therefore shouldn’t be seen as a waste of time or effort.