What Is Inter-Rater Reliability?
Inter-rater reliability is the degree to which two or more independent raters agree when they judge, score, or classify the same thing. If two examiners mark the same essays, two coders code the same interview transcripts, or two observers rate the same classroom behaviour, inter-rater reliability tells you whether their judgements are consistent — or whether the result depends too much on who happened to be doing the rating.
It matters because research findings are only useful if someone else would reach the same conclusion from the same evidence. A study that reports “80% of participants showed improvement” is hard to trust if a second rater, looking at the same participants, would have said 40%. High inter-rater reliability does not prove a measure is valid — raters can agree and still both be wrong — but low inter-rater reliability is a warning sign that the rating instructions, the categories, or the training need work before any conclusions are drawn.
You will meet this concept under a few other names. “Inter-observer reliability” is common in observational research, “inter-coder reliability” in qualitative coding, and “inter-rater agreement” when people are talking loosely about the amount of agreement. They all point to the same idea: consistency between independent judges.
Why Percent Agreement Alone Is Not Enough
The simplest way to report agreement is percent agreement: the number of cases on which raters agreed, divided by the total number of cases, multiplied by 100. It is easy to understand, and it is a useful first check. But on its own it can flatter you, because some agreement happens by chance alone.
Imagine two raters classifying patients as “at risk” or “not at risk” in a sample where almost everyone is “not at risk”. Even if both raters guessed blindly, they would agree on most cases simply because “not at risk” is the common answer. Percent agreement would look impressive while telling you almost nothing about the raters’ real consistency. That is why researchers usually report a chance-corrected statistic alongside percent agreement: a measure such as Cohen’s kappa, which subtracts the agreement you would expect by chance and then scales the result.
A good rule of thumb is to report both. Percent agreement shows the raw picture in plain language; the chance-corrected statistic shows whether that picture survives a stricter test.
Which Measure Should You Use?
The right statistic depends on how many raters you have and what kind of data they produce. This table covers the choices most student projects need.
| Measure | When to use it | What it handles |
|---|---|---|
| Percent agreement | A quick, plain-language first check, or when categories are simple and chance agreement is low | Any number of raters, but does not correct for chance |
| Cohen’s kappa | Two raters, categorical judgements (yes/no, pass/fail, present/absent) | Corrects for chance agreement between two raters |
| Fleiss’ kappa | Three or more raters, categorical judgements | Extends the kappa idea to multiple raters |
| Intraclass correlation coefficient (ICC) | Continuous scores (marks out of 100, rating-scale scores, measurements) | Agreement and consistency of numerical ratings |
| Krippendorff’s alpha | Flexible designs, including missing data and different measurement levels | A versatile alternative when kappa or ICC does not fit |
For most undergraduate and master’s projects, the decision is simple: two raters and categories means Cohen’s kappa; numerical scores mean ICC; more than two raters and categories means Fleiss’ kappa. Do not force a statistic onto the wrong kind of data — a kappa calculated on continuous scores, or an ICC on yes/no categories, will mislead rather than inform.
Inter-Rater Reliability vs Related Concepts
Students often confuse inter-rater reliability with three nearby ideas. Keeping them apart will save you marks in a methods exam and awkward questions in a viva.
| Concept | What it means | How it differs |
|---|---|---|
| Inter-rater reliability | Different raters agree with each other | Between people, at the same time |
| Intra-rater reliability | The same rater agrees with themselves when rating the same material twice | Within one person, over time |
| Test-retest reliability | The same measure gives similar results when repeated on the same people | Stability of the instrument over time, not agreement between judges |
| Inter-coder reliability | Coders agree when coding qualitative data into themes or categories | The qualitative-research name for inter-rater reliability |
If your supervisor asks about inter-rater reliability, they are asking about consistency between judges — not about whether one judge is stable, and not about whether a questionnaire is stable over weeks.
How to Calculate Inter-Rater Reliability, Step by Step
Step 1: Decide what will be rated, and by whom. Choose a clear unit — one essay, one interview transcript, one observation session — and make sure every rater judges the same units independently, without discussing their answers first. Independence is essential: agreement reached after discussion is not reliability.
Step 2: Write clear rating instructions. Define every category in plain language, give an example of each, and state what to do with borderline cases. Most low reliability comes from vague instructions, not from careless raters. Pilot the instructions on a small set of cases first.
Step 3: Collect the ratings. Have each rater work through the full set (or a representative subset, if the full set is very large) and record their judgements separately.
Step 4: Calculate percent agreement. Count the cases where all raters agreed, divide by the total number of cases, and multiply by 100.
Step 5: Calculate a chance-corrected statistic. For two raters and categorical data, use Cohen’s kappa. The formula is: kappa = (observed agreement − expected agreement) / (1 − expected agreement). In plain words, it asks: of the agreement that chance cannot explain, how much did we actually get?
Worked Example: Cohen’s Kappa in Action
Suppose two examiners independently grade 100 short answers as Pass or Fail. Their results look like this:
Both raters said Pass: 50 answers. Rater A said Pass and Rater B said Fail: 10 answers. Rater A said Fail and Rater B said Pass: 5 answers. Both raters said Fail: 35 answers.
First, observed agreement: the raters agreed on 50 + 35 = 85 out of 100 answers, so observed agreement is 0.85, or 85%.
Next, expected agreement by chance. Rater A said Pass for 60 answers (0.60 of the total) and Fail for 40 (0.40). Rater B said Pass for 55 (0.55) and Fail for 45 (0.45). The chance of both saying Pass is 0.60 × 0.55 = 0.33, and the chance of both saying Fail is 0.40 × 0.45 = 0.18. Expected agreement is therefore 0.33 + 0.18 = 0.51.
Now apply the formula: kappa = (0.85 − 0.51) / (1 − 0.51) = 0.34 / 0.49 = 0.69 (rounded).
Notice the lesson: percent agreement was a healthy-looking 85%, but kappa is 0.69. The gap is the agreement chance alone would have produced. Reporting only the 85% would have overstated how consistent the examiners really were.
How to Interpret Your Result
For Cohen’s kappa, the widely used Landis and Koch benchmarks are: below 0, poor; 0.00 to 0.20, slight; 0.21 to 0.40, fair; 0.41 to 0.60, moderate; 0.61 to 0.80, substantial; 0.81 to 1.00, almost perfect. Treat these as a guide, not a pass mark. A kappa of 0.69, as in the example above, is usually described as substantial, but whether it is good enough depends on the stakes: a classroom exercise can live with moderate agreement, while a high-stakes exam or a clinical diagnosis needs much stronger consistency.
For ICC values, higher is again better, and many researchers look for values above 0.75 for good reliability and above 0.90 for excellent reliability, while recognising that the right threshold depends on the field and the use of the scores. Whatever statistic you use, report the value, the statistic, the number of raters, and the number of cases rated, so readers can judge the result for themselves. A sentence such as “Two independent raters coded 50 transcripts; Cohen’s kappa was 0.69, indicating substantial agreement” gives a reader everything they need.
How to Improve Low Inter-Rater Reliability
If your first reliability check comes back low, do not hide it — fix it. First, look at where raters disagreed. If disagreements cluster on one category, that category is probably defined too vaguely. Rewrite its definition, add examples, and clarify borderline cases. Second, train together: have raters practise on a small set, compare answers, and discuss disagreements before the real rating begins. Third, simplify where you can: fewer, clearer categories are easier to apply consistently than many overlapping ones. Fourth, rate independently and blind where possible, so one rater’s judgement cannot influence another’s. Finally, re-check after the fix on a fresh set of cases, and report the final figure honestly.
Common Mistakes to Avoid
Reporting only percent agreement and calling the job done. Always pair it with a chance-corrected statistic when categories are involved.
Letting raters discuss cases before rating. That measures consensus, not reliability.
Using kappa on continuous scores, or ICC on simple yes/no categories. Match the statistic to the data type.
Rating too few cases for the check. Reliability estimated from ten cases is shaky; use a reasonable subset of your full data.
Ignoring disagreement. A low figure is information, not embarrassment. Report it, explain what you changed, and re-check.
Frequently Asked Questions
What is a good inter-rater reliability score? There is no universal pass mark. As a rough guide, a Cohen’s kappa above 0.60 is often called substantial and above 0.80 almost perfect, but high-stakes decisions demand higher values than low-stakes classroom work.
How many raters do I need? Two independent raters are the minimum for a meaningful check. More raters strengthen the evidence, but each additional rater adds training and coordination work, so balance rigour against practicality.
Can I use percent agreement on its own? You can report it, and you should, but do not rely on it alone. When one category is very common, chance agreement inflates it. Pair it with kappa, ICC, or another chance-corrected statistic.
What is the difference between inter-rater and intra-rater reliability? Inter-rater reliability compares different raters with each other. Intra-rater reliability checks whether the same rater gives the same judgement when they rate the same material a second time.
Do I need inter-rater reliability for a questionnaire? Not in the same way. Questionnaires are usually checked with internal consistency (for example, Cronbach’s alpha) and test-retest reliability. Inter-rater reliability matters when human judgement does the scoring, coding, or observing — essays, interviews, observations, and clinical ratings.
Conclusion
Inter-rater reliability answers a simple, important question: would someone else, judging the same evidence, reach the same conclusion? Percent agreement gives you a first, plain-language answer, and a chance-corrected statistic such as Cohen’s kappa or the ICC gives you the stricter test that chance agreement cannot pass. Define your categories clearly, train your raters, rate independently, and report your figure honestly — including what you did when it came back lower than you hoped. That habit is what turns subjective judgement into evidence a reader can trust.