App Store Testing: 6 Steps to Experiment

App Store Testing

App store testing or mobile A/B testing is the perfect ally for your App Store Optimization strategy. Why? Because it helps you to build the most converting app store listing.

The idea behind app store testing is to create one or more variations of your current store listing, split your audience to visit these variations, and see which one has the best performance. You can experiment with several aspects of your store listing.

In case you want to know in detail how to test Android app elements, make sure to check our guide to Store listing experiments in Google Play. And to read more about A/B testing for iOS apps, take a look at our guide to A/B testing with Product Page Optimization in Apple's App Store.

So let's take a look at the steps to do A/B testing!

Mobile App Ab Testing
App store testing helps you create a good performing app store listing.

6 Steps to Run App Store Testing

App store tests need to be done correctly in order to present results. You can focus on two types of experiments:

  • App Store Listing: discover which variation helps you get more downloads.
  • Search results: turn the search impressions into visitors and stand out from the competition.

Now that you know there are two ways to do mobile A/B testing , here are 6 steps for you to follow to run a successful experiment.

Step 1: Research

In order to do app store testing, you need to know what you should test, and for that, you need to research. Maybe you already have an idea about what to test, but the research phase is essential to prioritize your testing ideas.

Here are some tips to get started:

  • Spying on your competitors is a great source of inspiration. The idea is to understand what is working for them so that you can also apply it to your store listing. It's also good for figuring out what you can do to be different. With App Radar, you can keep track of all your competitors’ changes to their store listing.
App Timeline Competitor Tracking
Try the App Timeline in our free 7-day trial and check what your competitors are changing in their app store listing.
  • Take a look at the apps within your category. This way, you can understand the industry standard and get some ideas on how to stand out from the competition.
  • Consider what is happening outside of app stores. Remember that your potential user is consuming also other media besides app stores. So get inspired by taking a look at them! For example, if your target audience likes science-fiction books, take a look at covers of books like that and get inspiration for your screenshots!

Step 2: Creating Hypothesis

A hypothesis is an assumption, an idea that is proposed for the sake of argument so that it can be tested to see if it might be true. There are two questions that help you to create a good hypothesis:

  • What do you want to change in your store listing?
  • What do you want to achieve with these changes?

For example, this would be a good hypothesis:

I want to optimize my screenshots to increase conversion. Since my mobile game is played in landscape mode, if I change the orientation of my screenshots to landscape, I'll get more downloads.

What should the hypothesis be about?

There are three main aspects that you should consider experimenting with.

  • Screenshots
    These are great elements to test; you can experiment with orientation, colors, background, caption, order, and even social proof.
  • App Icon
    According to SplitMetrics, the app icon can uplift conversions up to 22%. The icon is one of the main elements that make your app stand out in a crowded marketplace. You can for example test a female or male character, background color, logo or no logo, and elements that represent your USP.
  • App Video
    "To have or not to have an app video" is already a good experiment. Why? Well, not every app store listing benefits from a video, some even experience a decrease in downloads. You can experiment with two videos presenting different features to test which feature showcase brings you more downloads. Another idea is to see if having more than one video increases conversions (only possible in the Apple App Store).
App Store Testing Creatives
App store testing is a great way to experiment with your app icon, screenshots, and video.

Note! Test only one hypothesis at a time so you can get precise results. Remember that testing too small details won't help you that much. For example, changing the fifth screenshot wouldn't give you enough insights since people usually don't scroll that far. Choose your hypothesis well to get results that are statistically significant.

Step 3: Traffic

To have a truthful result, you need to have enough traffic in all of your variations. This way, you can reach a realistic "Confidence Level," which is based on your traffic sample and shows you how accurate your result is. For example, if your variation A had 15-36% more installs, and its confidence level is 95%, this means that this result will prove to be confirmed 95 times out of 100. In this case, this is an evident winner variation.

Step 4: Creating your Variation

Now it's time to put into practice your research together with your hypothesis. You can create one variation to run an A/B test or more than one and run a split test. When you're experimenting with screenshots, we've put together a list of tools that you can use to speed up the process of creating app screenshots.

Step 5: Running the Experiment

In this step, there is not much for you to do. Just make sure you’re running your experiment for at least 7 days. This way, you can test on weekdays and weekends. Ideally, your experiment should run between 7 to 14 days.

Note! Even if you see results before the 7th day, let the experiment run until the end!

Step 6: Analysis and Implementation

Based on the results, you can make your decision. Check the Confidence Level, but also take into account the following KPIs:

  • Traffic volume per variation: did you have enough traffic to have an accurate result?
  • Conversion number per variation: how many visitors downloaded your app in each variation?
  • Time to click to install: which of the variations takes the least time to convert?
  • Engagement rate in the store listing: how many visitors scrolled through your screenshots, watched your video, read your description, and tapped on the install button?
  • Improvement metric: this metric shows you if the variation increased or decreased your conversion rate.
Mobile Ab Testing Kpis
Check the Confidence Level, but also these KPIs to decide which variation was the winner.

Bear in mind that a variation that wins by less than 2% isn't really a winner. You need to have at least a 3% of difference between your variations to have a winner. Don't be upset if your new variation loses. That's the idea of app store testing. It's a way to avoid putting live app store listing versions that would decrease conversions.

After implementing the results, allow it one to two months to see improvement. In the meantime, you should think about a new experiment. Just like App Store Optimization is an ongoing process, so is app store testing. There is always something to experiment, and there is always room for improvement! Hence, implement app store testing as part of your App Store Optimization strategy.

Summary of App Store Testing: 6 Steps Experiment

Now that you know all the 6 steps to run app store tests, don't forget also to do localized experiments. Your winner variation for Spain won't have the same performance in Belgium. And, of course, don't apply the same variation from the Google Play Store to the Apple App Store. Consumer behavior is different on both platforms, and the way the apps are presented is also different.

Do you want to learn more about app store testing? Check out our App Marketing Academy.

Get valuable insights
right in your inbox.
Sign up