A/B Testing Mobile Apps

What is A/B Testing?

Is an experiment where you give two versions of your service to different groups of people and measure each variation performance.

This can be as simple as changing the color of a button in a website and measuring which one gets more clicks.


Other names for A/B testing are bucket tests and split-run testing.

Why should I care?

A/B testing offers a scientific, not biased, method to improve the service you give. You can formulate an hypothesis, test it with real users and drive your business decisions based on real data and not driven by the HiPPO (highest paid person’s opinion).9c1f992d775ea74726aa59e384f125e5-jpg

The powerfulness of this technique is when you find counterintuitive results, this results are very difficult to find without using the scientific approach.

Who is using it?

Almost every tech company is using it in some way or another. Some famous example are Google, Facebook, and Spotify. It’s very probable that you were already a participant in a experiment without even knowing it, for example when in Facebook you see a different kind of UI than your friend.

How do I implement it?

In the simplest way A/B only needs two things, one way to create a uniform variation and a way to measure the results.

We can for example have a flag that when the user launches the app for the first time randomly decides if the user is on the A group or the B group and then change the behaviour based on this flag. When the user interacts with the application we measure their actions and using analytics we send the result along with the group the user is in.

However if you are starting small using a commercial platform can have some advantages and simplify the work you do.

In this blog post we will explore how A/B testing can be done using Taplytics.


Integrating the SDK is a very standard process that you can follow once you sign up in Taplytics.

After that you can start to create A/B test using the Experiments tab.

Screen Shot 2017-01-15 at 12.29.42.png

The experiment we devise was called “Emojis or Images” to find out if our users prefer an interface with emoji or instead images.


Our original UI was:

Screen Shot 2017-01-15 at 12.54.51.png

And we wanted to test if by replacing the images with emojis we could increase the overall usage of the application. One test and one metric that we can measure.

The Visual Editor of the service enables you to tap anything in the app and edit them for the experiment without coding.

This feature while promising didn’t work as expected, associating wrong images in the UI. Even giving the view different tags and IBOutlets didn’t change the situation. We contacted the Taplytics team to understand better the problem.

Screen Shot 2017-01-15 at 13.01.59.png

So we use a Boolean variable to control the UI and change it to replace the images with emojis.

Screen Shot 2017-01-15 at 13.22.19.png

Then we can create a goal of our experiment:

Screen Shot 2017-01-15 at 13.24.41.png

Then you can create user segments to which expose the test, and filter by specific parameters:

Screen Shot 2017-01-15 at 13.27.28.png

You can even select the percentage of users to expose the experiment and a percentage of users to do the rollout:

Screen Shot 2017-01-15 at 13.28.04.png

After you start the experiment you can get a summary and results to measure your hypothesis.

After some time you can declare a Winning variation and deploy that versions to all your users.

Sample Project

Our sample project is available at our GitHub.

Note: Don’t forget to change the API key and URL scheme for your account.

More tools and services

A list of other tools and services are available here. Such as OptimizelyApptimizeLifecycleLeanplum and Localytics.



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s