Most mobile app developers get their testing numbers completely wrong. They either test with far too few users and miss critical problems, or they waste thousands testing with way too many people. The magic number isn't what you think it is—and it changes depending on what you're actually trying to find out.
After years of working with companies who've made both mistakes, I can tell you that getting your sample size right is one of the most misunderstood parts of mobile app development. Some teams think five users is enough for everything. Others believe they need hundreds before they can make any decisions. Both approaches can kill your project.
The difference between testing with the right number of users and the wrong number isn't just about data quality—it's about whether your app succeeds or fails in the real world
This guide will show you exactly how many users you need for different types of testing, from early UX research through to final validation. We'll cover the science behind the numbers, why user behaviour changes everything, and how your testing methodology affects your sample size. Most importantly, we'll help you avoid the expensive mistakes that waste your budget and delay your launch. No more guessing—just the right numbers for your specific situation.
Understanding the Numbers Game in App Testing
Let's get straight to the point—there's no magic number that works for every app testing scenario. I've seen developers panic when they can't get hundreds of users for testing, and I've watched others waste thousands testing with far more people than they actually needed. The truth is, effective app testing is less about hitting some arbitrary target and more about understanding what you're trying to achieve.
Most people think bigger samples always mean better results, but that's not necessarily true. A well-designed test with 30 users can give you more valuable insights than a poorly planned test with 300. It all depends on what type of testing you're doing and what questions you're trying to answer.
The Basic Numbers You Need to Know
Different testing scenarios require different sample sizes, and here's where it gets interesting. For basic usability testing, you might only need 5-8 users to spot the most obvious problems. But for A/B testing or conversion optimisation, you could need thousands to get statistically significant results.
- Usability testing: 5-8 users per user group
- A/B testing: 100-1000+ users per variant
- Beta testing: 50-200 users depending on app complexity
- Accessibility testing: 3-5 users with specific needs
The key is matching your sample size to your testing goals—not just picking a number that sounds impressive. Quality data from the right users will always beat quantity from random participants.
Different Types of Mobile App Testing Need Different Sample Sizes
Here's what I've learnt after years of running mobile app testing sessions—there isn't a magic number that works for everything. The type of testing you're doing completely changes how many users you need, and getting this wrong can either waste your budget or give you useless results.
Usability testing is probably the most common type we do at Glance, and the good news is you don't need hundreds of people. Five to eight users will usually uncover about 80% of your major usability issues. That's because people tend to struggle with the same things—if three users can't find your checkout button, chances are most users won't either.
A/B Testing Needs Much Bigger Numbers
A/B testing is a completely different beast. You're looking for statistical significance here, which means you need much larger sample sizes—we're talking hundreds or even thousands of users depending on what you're testing. Testing a button colour change? You might need 500 users per variation. Testing a complete redesign? Even more.
Accessibility testing sits somewhere in between. You'll want at least 10-15 users with different disabilities to get meaningful insights about how your mobile app performs for people with various needs.
Start with smaller usability tests to fix obvious problems before investing in large-scale A/B testing—you'll save money and get better results.
The Science Behind UX Research Sample Sizes
Let's talk about the numbers that actually matter in UX research—and trust me, they're not as big as you might think! Jacob Nielsen's famous research back in the day showed that you only need 5 users to catch about 85% of usability problems. That's right, just 5 people. Now, I know what you're thinking: "Surely that can't be enough?" But here's the thing—this rule works brilliantly for qualitative testing where you're looking for problems, not measuring how many people have them.
The science gets more interesting when we look at quantitative research. If you want to measure conversion rates or user satisfaction scores, you'll need bigger numbers. The exact amount depends on what you're measuring and how precise you need to be.
Sample Size Guidelines by Research Type
- Usability testing: 5-8 users per user group
- Card sorting: 15-20 participants
- A/B testing: 100+ users per variant
- Survey research: 30-50 for basic insights
- Eye tracking studies: 8-12 participants
Here's what I've learned from years of running these tests: diminishing returns kick in fast. Going from 5 to 10 users in usability testing might give you a few extra insights, but going from 10 to 20 rarely adds much value. The key is understanding what type of research you're doing and what decisions you need to make with the data.
How User Behaviour Affects Your Testing Numbers
Your users aren't robots—they're real people with unpredictable habits, preferences, and ways of interacting with your mobile app. This reality has a massive impact on how many people you need for effective UX research and testing methodology.
When users behave consistently, you can get away with smaller sample sizes. But when they're all over the place? You'll need more participants to spot meaningful patterns. I've seen apps where users followed almost identical paths through the interface, making our testing super efficient. Then there are apps where every single person seemed to use it differently—those projects required much larger test groups.
Predictable vs Unpredictable Users
Apps with clear, single-purpose functions tend to produce more consistent user behaviour. Think about a simple calculator app versus a complex social media platform. The calculator users will likely tap numbers and operators in predictable sequences, whilst social media users might scroll, swipe, tap, comment, or share in countless different ways.
The more varied your user behaviour, the more participants you need to capture the full spectrum of how people actually use your app
Your target audience also plays a role here. Teenagers might interact with your app completely differently than adults—and if you're targeting both groups, your testing methodology needs to account for these behavioural differences by including more diverse participants in your research.
Testing Methodology Changes Everything
The way you conduct your app testing will completely change how many users you need. A/B testing where you're comparing two button colours? You'll need hundreds of participants to get reliable results. But if you're doing usability testing to find basic navigation problems, five users might be enough to spot the biggest issues.
Remote testing typically requires more participants than in-person sessions because you lose some of the nuanced feedback you get from face-to-face interaction. When users are sitting in front of you, you can pick up on hesitation, confusion, or frustration that might not come through in a remote session.
Quantitative vs Qualitative Testing Requirements
The type of data you're collecting makes a massive difference to your sample size needs:
- Quantitative testing (conversion rates, completion times) needs larger samples for statistical significance
- Qualitative testing (user feedback, pain points) can work with smaller, more focused groups
- Mixed methods often give you the best of both worlds but require careful planning
Moderated testing sessions where you're guiding users through tasks will give you richer insights with fewer people. Unmoderated testing—where users complete tasks independently—often needs more participants to compensate for the lack of real-time clarification and follow-up questions.
Common Mistakes That Waste Your Testing Budget
After years of working with clients on mobile app testing, I've seen the same costly mistakes repeated over and over again. The biggest one? Testing with far too many users when you're still in the early stages. I've watched companies burn through thousands of pounds testing with 100+ users when they could have found the same critical issues with just 8-10 people.
Another budget killer is mixing up your testing methodology halfway through. You start with moderated sessions, then switch to unmoderated, then throw in some A/B testing for good measure. Each method needs different sample sizes and approaches—jumping between them means you're essentially starting fresh each time.
The "More Users = Better Results" Trap
Here's what I tell every client: UX research follows the law of diminishing returns. Those first five users will uncover about 85% of your usability problems. Users six through fifteen might find another 10%. After that, you're paying premium prices for minimal insights.
The smartest approach? Start small, fix the obvious problems, then test again. This iterative cycle gives you much better value than one massive testing session that tries to solve everything at once.
Always define your testing goals before you recruit participants. Knowing whether you're testing navigation, feature discovery, or conversion flows will help you choose the right sample size and save money.
Conclusion
After working with hundreds of apps over the years, I can tell you that getting your testing numbers right isn't just about following a formula—it's about understanding what you're actually trying to achieve. You don't need massive user groups for every type of testing; you need the right number of people for the specific insights you're after.
The key takeaway? Start small and build up. Five users can catch major usability problems, but you'll need closer to 30 for reliable statistical data. Different testing methods require different approaches, and your app's complexity will influence these numbers too. Don't fall into the trap of thinking more users always means better results—I've seen teams waste thousands testing with 100 users when 15 would have given them the same insights.
Most importantly, remember that testing is an ongoing process, not a one-time event. Your user base will grow and change, new features will be added, and user expectations will evolve. The testing approach that works for your MVP might not work for your version 2.0. Keep testing, keep learning, and don't be afraid to adjust your sample sizes based on what you discover along the way.
