Which Statistical Test Should I Use? A Decision Guide

Choose the right statistical test from your study design: what you measured, how many groups, and whether observations are paired. A decision guide with worked rules, common mistakes, and a calculator for every test.

Almost every "which test?" question is answered by three features of your study design, not by your data:

  1. What did you measure? A number, a category, or a time-until-something-happened.
  2. How many groups are you comparing? One, two, or three or more.
  3. Are the observations independent, or paired? Different people in each group, or the same people measured twice.

Answer those three and the field narrows to one or two candidates. Everything below is organised that way. Every test named here links to a free calculator you can run in your browser.

One warning first: choose your test from the design, before you look at the results. Picking the test that gives the smaller p-value is p-hacking even when it feels like diligence, and it is the most common way honest people produce unreliable findings.

Comparing a numeric outcome between groups

Use Welch's, not Student's, as your default for two groups. Student's t-test assumes the two groups have equal variances; Welch's does not, costs almost nothing when they do, and protects you when they don't. Why Welch should be your default →

Skewed data does not automatically mean a rank test. With large, balanced groups, ANOVA is robust to mild non-normality. Switch when the skew is genuine and severe, when the outcome is ordinal, or when outliers would dominate the means. Kruskal-Wallis vs ANOVA →

Comparing categories or proportions

The commonest error here is running a chi-square on paired data. If the same subjects appear in both the "before" and "after" columns, the observations are not independent and chi-square is the wrong test — McNemar's exists precisely for that design.

Relationships and prediction

Time-to-event (survival) data

Use these whenever the outcome is how long until something happened and some subjects haven't had it yet. Censoring is the reason a t-test cannot answer these questions.

If subjects can experience an event that makes yours impossible — dying before transplant, say — standard Kaplan-Meier will systematically overstate your risk. Competing risks explained →

Classification and pattern-finding

ROC / AUC for how well a marker discriminates, and DeLong's test to compare two AUCs on the same cases. For unlabelled structure: k-means, DBSCAN, Gaussian mixture models, PCA, and outlier detection via local outlier factor or Mahalanobis distance. Which clustering algorithm? →

Before you run anything

Three checks that change the answer more often than the choice of test does.

Missing data. Dropping incomplete rows is only safe if the missingness is unrelated to the outcome — and it usually isn't. MCAR, MAR, MNAR and multiple imputation →

Assumptions. Shapiro-Wilk for normality, Levene's test for equal variances. Treat both as evidence, not verdicts: with a large sample almost everything fails a normality test, and with a small one almost nothing does.

Power. Work out the sample size before collecting, with the power calculator. A non-significant result from an underpowered study says nothing at all.

After you run it

A p-value tells you how surprising your data would be if nothing were going on. It does not tell you how big the effect is, and with a large enough sample a trivial difference will be "significant". Always report an effect size and a confidence interval alongside it — Hedges' g or Cohen's d for two means, η² or ε² for three or more groups, an odds ratio or hazard ratio for regression. P-value vs effect size →

Not sure? Let the app choose

Inference Stats has a Guided Analysis flow that asks the three questions at the top of this page in plain language, inspects the shape of your data, and recommends exactly one test — with its reasoning written out, so you can check the logic rather than trust it. It runs the assumption checks, applies the right post-hoc corrections, reports effect sizes, and drafts your Methods and Results paragraphs.

Every method is validated to numerical precision against an independent reference implementation — SciPy, statsmodels, lifelines, scikit-learn or R — and the validation page shows how.

Upload your data and let Guided Analysis pick the test — free →

FAQ

Should I test for normality before choosing? Not as a gate. Normality tests answer "is there evidence of non-normality?", which depends mostly on sample size. Judge the shape of the distribution and the size of the sample together; a histogram tells you more than a p-value.

Parametric or non-parametric? Parametric tests compare means and are more powerful when their assumptions roughly hold. Rank-based tests ask whether one group's values tend to be larger, which is often the question you actually mean for skewed or ordinal outcomes.

My groups are different sizes — is that a problem? Not in itself. Unequal sizes combined with unequal variances is the problem, and that is exactly what Welch's t-test and Welch's ANOVA handle.

Can I just run several tests and report the best one? No. Every extra test inflates the chance of a false positive. Decide the analysis from the design first; if you genuinely need multiple comparisons, correct for them (Tukey, Games-Howell, or Dunn with Holm).

What if none of these fit my design? Repeated measures nested in groups, clustered data, or multiple outcomes per subject need mixed-effects or multivariate models. Ask in the app's Guided flow — it will tell you when your design is outside what a single test can answer.

Written by Dr Hoong Sern Lim MB ChB MD FRCP, Consultant Cardiologist, Queen Elizabeth Hospital Birmingham; Honorary Senior Lecturer, University of Birmingham. ORCID 0000-0002-6569-1805

All guides · About the author · How we validate