How to test your paywall without fooling yourself
Paywall tests are tempting because small changes can have large effects on revenue. They are also easy to get wrong, especially for small apps with limited traffic. This guide shows a method that keeps you honest. It builds on paywall design and paywall placement.
Decide what question you are asking
Start with one question, written as a prediction:
- "Showing the yearly plan first will raise yearly purchases without lowering total conversion."
- "Adding a three-day trial will increase paid conversions after the trial."
- "Moving the paywall after the first result will raise conversion without raising day-7 churn."
If you cannot state the prediction, you are not ready to test.
Choose the right metrics
Primary metric: the thing the question is about, normally paying users per paywall view or per new user.
Guardrail metrics: things that must not get worse.
- Retention at day 1, day 7 and day 30.
- Refund or cancellation rate.
- Ratings and review sentiment.
- Support requests.
A variant that increases first purchases but doubles refunds or one-star reviews is a loss. For subscriptions with trials, the real outcome often appears days later (when the trial converts), so measure the final result, not the trial start.
Randomize properly
- Split new users randomly into groups and keep each user in the same group for the whole test.
- Change one thing per test.
- Run the variants at the same time. Comparing this week to last week mixes in seasonality, promotions and store features.
- Exclude yourself, testers and anyone with a special offer.
Most subscription tools and remote-configuration services support this, or you can build a small version yourself: assign the group at first launch, store it, and log it with every event.
How long and how many
Small apps need patience. Rules of thumb:
- Run for at least a full week, ideally two or more, so weekday and weekend behaviour both count.
- For a trial-based subscription, keep the test going until the trial results are in.
- Do not stop on the first day that one group looks better. Early leads are usually noise.
- If you have very few paywall views per week, big changes are testable and small tweaks are not. A change that moves conversion from 2% to 2.2% cannot be detected with a few hundred views.
Use any online sample-size calculator for proportions to see how many views per group you would need to detect the difference you care about. If the number is far beyond your traffic, test bigger changes, or use qualitative feedback instead.
Combine numbers with people
Numbers tell you what happened, not why. Ask a handful of people to walk through the paywall and think aloud: what do they believe they are buying, for how much, and when will they be charged? Misunderstandings found this way usually explain a weak result better than another round of tests. Fellow developers are quick at this; AppReviewX lets you request a private review of a paywall flow.
Beware of these traps
- Peeking and stopping early.
- Testing many things at once on too little traffic.
- Counting trial starts as revenue.
- Ignoring refunds and retention.
- Dark patterns that win the test and lose the customer: hidden close buttons, confusing auto-renew terms, fake urgency. These can also breach store rules and consumer law.
- Different store territories and prices mixed together without noticing.
- Changing the product during the test.
Keep a log
Record date, hypothesis, variants with screenshots, traffic, outcome and decision. Next year you will not remember why the yearly plan is first.
Checklist
- One question, written as a prediction.
- Primary metric plus guardrails.
- Random assignment, same time, one change.
- At least one to two weeks; wait for trial results.
- Check required sample size.
- Add a few qualitative walkthroughs.
- Log the result and the decision.
Need testers for your app? Open a test group on AppReviewX. Start free