A/B testing is a great way to compare digital marketing tools or strategies to see which one is working better.
It’s a cost effective way of testing everything from web pages to email campaigns against each other so you get the best ROI. Using this method, you’ll get the numbers and statistics to see what needs to be changed out, tweaked or left alone.
How A/B Testing Works
A/B testing allows you to keep your emotions in check and use empirical data to make a decision. You’ll need to start by splitting what do you want to test between the two candidates.
Then, once the data starts flowing in, you can start to see who is performing best. With this example, you’ll also need to herd the numbers toward certain corrals that matter to you. For example, you might use metrics like the conversion rate and the bounce rate before making a final decision.
The first thing you need to do before you A/B test anything is determine what your expected outcome is. Do you want an increase in overall traffic, net new customers, or just an increase in return rates for existing customers? Or do you simply want to increase conversion rates? It’s of the utmost importance to determine the desired outcome, otherwise you won’t really know when to stop the experiment and evaluate the data you managed to collect.
You need to have an adequate tracking system in place so that you can understand how your users are interacting with the app. Depending on the experiment, you may want to track things like clicks, page views, initiated purchases, completed purchases, abandoned purchases and so on. This data, besides giving you relevant info on how your experiment is behaving, might also provide insight into other experiments you may want to run to further improve other parts of your app.
Another thing to factor in is the rollout percentage of the experiment. If you’re A/B testing a risky change which could potentially break the app or is an aggressive rebranding and you’re not sure how your users will receive it, you may not want to expose your entire customer base to it. In this case, it’s generally best to do a staged rollout. Start with 1% of your customers, let that roll for a few days, then increase to 5%, then 10%, then 25% and so on. Sometimes it’s better to be more conservative, especially with complex changes which touch a lot of code, and A/B testing provides both the medium and the appropriate safety net so you can get away unnoticed if something goes bad.
Downsides of A/B testing
It’s important to know that are cases when the A/B Testing is not recommended:
- When being first is more important than being optimized
- When you don’t have enough users (and you’re unable to get a reliable result from running an A/B test)
- When what you’re testing contradicts your brand or design principles (this could mean any number of things — from rebranding to expanding or contracting the inventory you’re offering, to changing button sizes and so on)
- When you are not certain of your hypothesis (though, ideally, this shouldn’t ever be the case, with sufficient planning beforehand)
Conclusion
A/B Testing, while it may be difficult to implement effectively, is an incredibly powerful tool, which can really help drive your business. It doesn’t come without its fair share of challenges and risks, but knowing how and when and how long to use it has the ability to make your good app, great.