A/B test your store listing: what to test, how to read it
Opinions about icons and screenshots are cheap; data is better. Both major stores let you test variants of your listing on real visitors. Used carefully, this is the cheapest conversion work you can do. Used carelessly, it produces confident answers from noise.
What the stores offer
- Google Play: store listing experiments in the Play Console let you test elements such as the icon, feature graphic, screenshots and descriptions, with traffic split between your current listing and variants.
- App Store: product page optimization in App Store Connect lets you test icon, screenshots and preview videos against your default page. Custom product pages are different: they give different audiences different pages, and are not a test by themselves.
Features, names and limits change, so check the current documentation in each console for what you can test and how many variants are allowed.
What to test first
Order by likely impact:
- First screenshots. Different first image, caption or ordering.
- Icon. Big changes (colour, symbol) rather than tiny tweaks.
- Short description / subtitle where the store allows testing it.
- Preview video versus none, or a different first frame.
Test one element at a time. If you change icon and screenshots together and conversion rises, you will not know why.
Start with a hypothesis
A test is a question with a prediction. Compare:
- Weak: "Let's try a blue icon."
- Better: "Our current screenshots show the settings screen first. Showing the result screen first will make the benefit clearer and raise installs per visitor."
Write the hypothesis, the metric (installs per page view) and the minimum change you care about before you start. That stops you from finding a "win" afterwards.
How long and how big
- Small apps get little traffic, so tests take weeks. Plan on at least one to two weeks and until the console shows a clear result, not a day-by-day fluctuation.
- Run across whole weeks so weekday and weekend behaviour both count.
- Do not stop the moment a variant looks ahead. Early leads often vanish.
- Avoid running a test during unusual traffic: a launch, a feature in the store, a holiday sale.
If your traffic is very low, consider fewer, bigger changes and rely on the five-second stranger test described in our screenshot guide instead.
Reading the result
Look at the console's confidence information and the size of the difference. A small, uncertain gain may not be worth the change. A large, consistent one is. Two cautions:
- Conversion is not the whole story. A screenshot that promises something the app does not do raises installs and then raises refunds and bad reviews. Check ratings and retention after applying a winner.
- One test is one data point. Re-test important winners later, because audiences and seasons change.
Keep a log
For each test record: date, hypothesis, variants (with screenshots saved), traffic, result and what you decided. After a year this becomes the most useful ASO document you own.
Common mistakes
- Testing several elements at once.
- Ending early on a lucky streak.
- Testing tiny variations that cannot move the result.
- Never applying the winner.
- Ignoring the effect on rating and refunds.
Checklist
- Choose one element and write a hypothesis.
- Create one clearly different variant.
- Run for full weeks, avoid unusual events.
- Decide using confidence and size of difference.
- Apply the winner, watch ratings and retention.
- Log everything.
Need testers for your app? Open a test group on AppReviewX. Start free