P-Hacking: What It Is and How to Avoid It in Your Research

What Is P-Hacking?

Imagine you run a study and your result comes back with a p-value of 0.08 — just above the usual 0.05 cut-off for “statistical significance.” Nothing is significant, and your hypothesis looks unsupported. So you try a few things: you drop a couple of unusual responses, you test a different subgroup, you try a slightly different analysis. Suddenly, p = 0.04. You report that result and move on.

That is p-hacking: running analyses, tweaking data, or selectively reporting results until you get a statistically significant p-value, then presenting that result as if it were the plan all along. The term is sometimes written as “p hacking” or “p-hacking,” and it is also called data dredging or data fishing when the searching is broad and unfocused.

P-hacking is not the same as making an honest mistake. It is a set of choices — often small, often made under pressure — that push a non-significant result over the significance line. The researcher may not even realise they are doing it, which is part of what makes it so common and so hard to catch.

It matters because a p-value only means what it claims to mean when the analysis was planned in advance. Once you run many tests and report only the one that “worked,” the reported p-value no longer reflects the real probability of a false positive. The finding looks solid on paper, but it may be nothing more than chance dressed up as discovery.

Why Does P-Hacking Happen?

Most p-hacking is not driven by dishonesty. It is driven by the structure of academic life.

First, journals and examiners reward significant results. A study that finds “no effect” is harder to publish, harder to get excited about, and harder to defend in a viva — even though null results are often just as informative. When only significant findings get published, researchers feel pressure to produce them.

Second, data analysis is full of small decisions, and every decision is a fork in the road. Which participants count as outliers? Which questionnaire items go into the total score? Should you control for age, gender, or both? Each choice is defensible on its own, but if you only keep the combination that gives p < 0.05, you have quietly run dozens of hidden tests and reported one.

Third, many students are never taught that these choices matter. If nobody explains that trying ten analyses and reporting the best one inflates your false-positive rate, you can p-hack with a clear conscience. That is why understanding the practice matters as much as condemning it.

6 Common P-Hacking Practices (With Examples)

Here are the six forms students and researchers run into most often. Each one is easy to slip into, and each one distorts the result in the same direction — toward a “significant” finding that may not be real.

1. Optional stopping: collecting data until the result turns significant

You plan to collect data from 100 participants, but after 60 responses the p-value is 0.09. You keep collecting, checking the result after every batch, and stop the moment it dips below 0.05. Checking repeatedly and stopping at the first significant moment is like rolling a die until you get a six and then claiming sixes are common. If you peek at the data as it comes in, you need a plan that accounts for it — or you should decide your sample size in advance and stick to it.

2. Cherry-picking outcomes: reporting only the measures that “worked”

Your questionnaire measured five outcomes — anxiety, stress, sleep quality, mood, and self-esteem. Only sleep quality shows a significant difference. You write up the sleep result in detail and quietly drop the other four from the paper. Readers see one clean significant finding; they never learn that four other tests were run. The honest version reports all five outcomes, significant or not.

3. Subgroup fishing: slicing the sample until something is significant

The overall result is not significant, so you split the sample: men only, women only, under-25s only, first-year students only. One subgroup shows a significant effect, and that becomes the headline. With enough subgroups, one will almost always look significant by chance alone. Subgroup analyses are fine when they are planned in advance and labelled clearly — the problem is presenting a post-hoc subgroup finding as if it were the main result.

4. Dropping “outliers” until the result changes

A few participants gave extreme responses, so you remove them and re-run the analysis. The p-value drops below 0.05. Removing genuine outliers can be legitimate, but the decision must be based on a rule set before you see the results — for example, “scores more than three standard deviations from the mean will be excluded” — not on whether removing them helps your p-value. If the result only becomes significant after you delete data, that is a warning sign, not a finding.

5. Trying every analysis until one works

You run a t-test — not significant. You try an ANOVA — not significant. You try a regression with different predictors, a non-parametric test, a different way of scoring the scale. Eventually something gives p = 0.03, and that is the analysis you report. Each test you run is another chance for a false positive. If you run twenty tests on the same data, you should expect about one of them to be “significant” by chance alone at the 0.05 level, even when there is no real effect at all.

6. HARKing: hypothesising after the results are known

HARKing means writing your hypothesis after you have seen the data, then presenting it as if you predicted the result from the start. For example, you expected a difference between two teaching methods, found none, but noticed that students in the morning class scored higher — so you rewrite the paper around a “time of day” hypothesis you never had. HARKing turns exploration into fake confirmation. Exploring your data is good science; pretending the exploration was a prediction is not.

Why Is P-Hacking a Problem?

The damage is not abstract. It shows up in three concrete ways.

First, it fills the published literature with false positives. If many researchers nudge their analyses toward significance, a large share of published “significant” findings will be effects that do not really exist. Later researchers build on those findings, design interventions around them, or cite them in policy — and the whole chain rests on shaky ground.

Second, it wastes other people’s time and money. A false-positive finding invites replication attempts that fail, follow-up studies that go nowhere, and treatments or programmes built on an effect that was never real. Every hour spent chasing a phantom result is an hour not spent on real questions.

Third, it erodes trust in research itself. When well-known findings fail to replicate — as happened in psychology’s much-discussed replication debates — the public and policymakers start to doubt the whole enterprise. P-hacking is one of the reasons those replication failures happen, and it is one of the easiest to prevent.

P-Hacking vs Legitimate Exploratory Analysis

Not all data exploration is p-hacking. Looking around in your data is normal, useful, and often where the best ideas come from. The difference is honesty about what you did. This table draws the line:

The rule of thumb is simple: if a choice was made after seeing the results, say so. Transparency turns a questionable practice into an honest one.

How to Avoid P-Hacking in Your Own Research

Here is a practical routine you can follow in any study, from a class assignment to a doctoral thesis.

1. Preregister your study

Before you collect data, write down your hypothesis, your sample size, your measures, and your planned analysis — and register it somewhere with a timestamp, such as the Open Science Framework. Preregistration does not lock you out of exploration; it simply makes clear which analyses were planned and which were discovered later. When your examiners or reviewers can see the plan, the temptation to quietly change it disappears.

2. Decide your sample size before you start

Run a power analysis before data collection and commit to the sample size it suggests. Do not peek at the results halfway and stop early, and do not keep adding participants until the p-value cooperates. If you genuinely need to check data quality as you go, check it without looking at the hypothesis test.

3. Write your analysis plan in advance

Decide, before you see the data, how you will handle missing values, what counts as an outlier, which variables you will control for, and which test you will run. Write these decisions down. When the analysis is planned, there is nothing left to tweak in the direction of significance.

4. Correct for multiple comparisons

If you run several tests on the same data, adjust your threshold for significance. The simplest correction is the Bonferroni correction: divide your alpha level by the number of tests. With five tests at an overall 0.05 level, each test needs p < 0.01 to count as significant. Other methods, such as the Holm procedure, are less strict while still controlling the false-positive rate. The point is not which correction you choose — it is that you do one, and say which one.

5. Report everything, not just what “worked”

Report all the outcomes you measured, all the analyses you ran, and all the results you found — including the non-significant ones. A paper that says “we tested five outcomes; one was significant and four were not” is far more trustworthy than a paper that mentions only the one. Journals are increasingly welcoming null results, and examiners respect a student who reports honestly.

6. Focus on effect sizes and confidence intervals

A p-value tells you whether a result is unlikely under the null hypothesis, but it says nothing about how large or important the effect is. Report the effect size (for example, Cohen’s d or the correlation coefficient) and a confidence interval alongside every p-value. A tiny effect with a small p-value is often less interesting than a large effect with a wide interval — and readers who see the size of the effect are harder to mislead.

7. Share your data and analysis code

When your data and analysis script are openly available, anyone can check exactly what you did. That transparency is the strongest single protection against p-hacking, including your own unconscious version of it. Even if nobody ever checks, knowing that someone could keeps your decisions honest.

What If You Suspect P-Hacking in a Published Paper?

You will not always be able to prove it, but there are signs worth noting. A paper that reports a p-value just below 0.05 — say, 0.049 — with no effect size, no mention of other outcomes, and a hypothesis that fits the results suspiciously well deserves a careful read. Look for whether the authors preregistered, whether they report all the measures they collected, and whether the sample size looks planned or improvised.

If you are writing a literature review, treat such papers with caution rather than ignoring them entirely. Note the limitation, look for independent replication, and weigh the finding against the rest of the evidence. One shaky study does not sink a field — but a field built on shaky studies needs to know about it.

Frequently Asked Questions

Is p-hacking the same as fraud?

Not exactly. Fraud means inventing or falsifying data. P-hacking uses real data but analyses it selectively until a significant result appears. The result is still misleading, but the intent is often different: many researchers p-hack without realising it, under pressure to publish significant results. That said, deliberate p-hacking — running analyses you know are misleading to force a significant result — is a form of research misconduct, and journals and universities treat it seriously.

Can I explore my data at all, or is that always p-hacking?

You can and should explore your data. The problem is not exploration; it is presenting exploratory results as if they were planned tests. Explore freely, find interesting patterns, and then either test them on fresh data or label them clearly as exploratory findings that need confirmation. Honest labelling is the difference between good science and p-hacking.

What is the difference between p-hacking and HARKing?

P-hacking is about the analysis: trying different tests, samples, or exclusions until the p-value becomes significant. HARKing is about the story: writing your hypothesis after you have seen the results and presenting it as a prediction. They often go together — a researcher who fishes for a significant result may also rewrite the hypothesis to match it — but they are different moves, and both distort what the evidence actually shows.

Does a larger sample size prevent p-hacking?

No. A larger sample makes your estimates more precise, but it does not stop someone from cherry-picking outcomes, fishing through subgroups, or trying analyses until one works. The protections against p-hacking are planning, preregistration, transparency, and reporting everything — not sample size alone.

Key Takeaways

P-hacking is the practice of analysing data selectively — running extra tests, dropping inconvenient cases, or stopping early — until a p-value crosses the 0.05 line, then reporting that result as though it were the plan all along. It is common, often unintentional, and one of the main reasons published findings fail to replicate.

The fix is not complicated: plan your study before you collect data, preregister that plan, decide your sample size in advance, correct for multiple tests, and report all your results. Exploration is welcome — just label it as exploration. Research that is transparent about what was planned and what was discovered is research other people can trust, build on, and use. That is the standard your own work should meet, and it is a standard your examiners will respect.