What is A/B Testing?
Is an experiment where you give two versions of your service to different groups of people and measure each variation performance.
This can be as simple as changing the color of a button in a website and measuring which one gets more clicks.
Other names for A/B testing are bucket tests and split-run testing.
Why should I care?
A/B testing offers a scientific, not biased, method to improve the service you give. You can formulate an hypothesis, test it with real users and drive your business decisions based on real data and not driven by the HiPPO (highest paid person’s opinion).
The powerfulness of this technique is when you find counterintuitive results, this results are very difficult to find without using the scientific approach.
Who is using it?
Almost every tech company is using it in some way or another. Some famous example are Google, Facebook, and Spotify. It’s very probable that you were already a participant in a experiment without even knowing it, for example when in Facebook you see a different kind of UI than your friend.
How do I implement it?
In the simplest way A/B only needs two things, one way to create a uniform variation and a way to measure the results.
We can for example have a flag that when the user launches the app for the first time randomly decides if the user is on the A group or the B group and then change the behaviour based on this flag. When the user interacts with the application we measure their actions and using analytics we send the result along with the group the user is in.
However if you are starting small using a commercial platform can have some advantages and simplify the work you do.
In this blog post we will explore how A/B testing can be done using Taplytics.
Integrating the SDK is a very standard process that you can follow once you sign up in Taplytics.
After that you can start to create A/B test using the Experiments tab.
The experiment we devise was called “Emojis or Images” to find out if our users prefer an interface with emoji or instead images.
Our original UI was:
And we wanted to test if by replacing the images with emojis we could increase the overall usage of the application. One test and one metric that we can measure.
The Visual Editor of the service enables you to tap anything in the app and edit them for the experiment without coding.
This feature while promising didn’t work as expected, associating wrong images in the UI. Even giving the view different tags and IBOutlets didn’t change the situation. We contacted the Taplytics team to understand better the problem.
So we use a Boolean variable to control the UI and change it to replace the images with emojis.
Then we can create a goal of our experiment:
Then you can create user segments to which expose the test, and filter by specific parameters:
You can even select the percentage of users to expose the experiment and a percentage of users to do the rollout:
After you start the experiment you can get a summary and results to measure your hypothesis.
After some time you can declare a Winning variation and deploy that versions to all your users.
Our sample project is available at our GitHub.
Note: Don’t forget to change the API key and URL scheme for your account.
More tools and services