What Does Validity Actually Mean?
Learning how to establish validity of a questionnaire is one of the most important skills in research — because a questionnaire can be perfectly reliable and still completely useless. Reliability only tells you that your instrument gives consistent results. Validity tells you whether it measures what it claims to measure. A bathroom scale that always reads 5 kg too heavy is reliable, but it is not valid.
Validity is not a single property you either have or don’t have. It is a body of evidence you build up, piece by piece, showing that the scores from your instrument can be interpreted the way you intend. Examiners look for that evidence — and will ask when it is missing. This guide walks you through the process step by step, with a worked example you can adapt to your own instrument.
The Main Types of Validity You Need to Know
Before you start collecting evidence, you need to know what kind to collect. Validity comes in several forms, and a good instrument usually needs more than one:
- Face validity: Does the questionnaire look, on the surface, like it measures what it claims to? This is the weakest form of evidence, but it matters — confused respondents answer carelessly.
- Content validity: Do the items cover the full range of the concept? This is judged by experts, not by statistics.
- Construct validity: Does the instrument really capture the abstract concept (the “construct”) behind it? This is the most important type for most theses, and it is established through factor analysis and through expected relationships with other measures.
- Criterion validity: Do the scores predict or agree with some external standard? This splits into concurrent validity (agreement with a measure taken at the same time) and predictive validity (scores predicting a future outcome).
Face and content validity come first, while you are building the instrument. Construct and criterion validity come later, once you have data.
Step 1: Define Your Construct Clearly
Every validity argument starts with a definition. Before you write a single item, write down — in one or two sentences — exactly what your instrument is supposed to measure, and what it is not supposed to measure. For example, “academic procrastination” might be defined as “the voluntary delay of academic tasks despite expecting negative consequences” — a definition that tells you what belongs in the questionnaire (delaying assignments, putting off revision) and what does not (being busy, or working slowly because the task is genuinely hard).
Note the construct’s dimensions from the literature. Procrastination, for instance, involves delaying the start, delaying completion, and the emotional discomfort that follows. Your questionnaire should cover all of them — a vague construct produces vague items, and vague items cannot be validated.
Step 2: Build a Generous Item Pool
Write far more items than you will finally need — roughly twice as many. If you expect a final scale of 12 items, draft 20 to 25. Items should be short, clear, and written in plain language, each expressing one idea only — avoid double-barreled questions such as “I delay my assignments and feel guilty about it.” Mix positively and negatively worded items where appropriate, so respondents cannot simply agree with everything.
Step 3: Get Expert Review and Calculate the Content Validity Index
This is the step most students skip, and it is the one examiners ask about most often. Ask 5 to 10 experts — supervisors, lecturers, or researchers familiar with your topic — to rate each item for relevance on a 4-point scale (1 = not relevant, 4 = highly relevant).
From these ratings, calculate the Content Validity Index (CVI). For each item, the item-level CVI (I-CVI) is the proportion of experts who rated it 3 or 4. The widely used guideline (Lynn, 1986; Polit & Beck, 2006) is that an item should reach an I-CVI of at least 0.78 when six or more experts are involved, and the scale-level CVI (the average of all I-CVIs) should be 0.90 or higher.
Drop or rewrite any item below the threshold. This is not optional polishing — it is your documented evidence of content validity, and you should report the numbers in your thesis: how many experts you used, the I-CVI range, and the final S-CVI/Ave.
Step 4: Pilot Test with a Small Sample
Administer the revised questionnaire to a small pilot sample of 30 to 50 people from your target population.
Use the pilot data to check the basics: blank items, items with almost no variation, and internal consistency. A Cronbach’s alpha of 0.70 or above is generally acceptable for a new instrument, and items with item-total correlations below about 0.30 are candidates for removal.
Step 5: Test the Structure with Factor Analysis
Factor analysis is the main statistical tool for construct validity. With your full sample (ideally at least 100 respondents, and preferably 5 to 10 respondents per item), run an exploratory factor analysis (EFA) first. Check that your data are suitable — the Kaiser-Meyer-Olkin (KMO) measure should be above 0.60 and Bartlett’s test of sphericity should be significant — then look at how the items group together.
Do the items cluster into the dimensions you expected in Step 1? Items should load strongly (usually 0.40 or above) on one factor and weakly on the others. Items that load on the wrong factor, or on no factor at all, should be revised or removed. If your sample is large enough and you have a clear theoretical model, a confirmatory factor analysis (CFA) can then test whether that model actually fits the data. If the items do not group as theory predicts, that is strong evidence something is wrong with the instrument — and no later statistics will fix it.
Step 6: Check Convergent and Discriminant Validity
Two checks matter:
- Convergent validity: Your scores should correlate positively with an established measure of the same or a closely related construct. If you built a new procrastination scale, it should correlate moderately to strongly with an existing procrastination measure.
- Discriminant validity: Your scores should not correlate strongly with measures of unrelated constructs. Your procrastination scale should not be nearly identical to a measure of conscientiousness — if it is, you may be measuring the wrong thing.
Report the correlations and interpret them honestly. A correlation of around 0.50 to 0.70 with a related measure supports convergent validity; a correlation near zero with an unrelated measure supports discriminant validity. Correlations above 0.85 with another measure suggest your instrument adds nothing new.
Step 7: Establish Criterion Validity Where Possible
If there is an external standard your scores should predict or match, test it. Students who score high on a procrastination questionnaire should, on average, submit assignments later and earn lower grades than students who score low — that is predictive validity. Comparing your new scale with a well-established scale administered at the same time is concurrent validity. Not every study can do this step: if no good external criterion exists, say so in your thesis and rely on the evidence from the earlier steps. Examiners respect an honest limitation far more than a forced or meaningless comparison.
Step 8: Document Everything as Your Validity Evidence
Validity is an argument, and arguments need to be written down. In your thesis, report each piece of evidence in the methodology chapter: the construct definition and its literature base, the expert panel and the CVI values, the pilot test results, the factor analysis findings, the reliability coefficients, and any convergent, discriminant, or criterion correlations. Then summarise the overall judgement in a short paragraph: what the evidence supports, and what its limits are. Keep every version of the questionnaire, the expert rating sheets, and the analysis outputs.
Worked Example: Validating an Academic Procrastination Questionnaire
Imagine you are a master’s student building a short scale to measure academic procrastination among undergraduates — defined as voluntarily delaying academic tasks despite expecting to be worse off for the delay, with three dimensions: delaying starting, delaying finishing, and emotional discomfort about the delay. Here is how the steps above would look in practice:
Item pool. You draft 24 items, eight per dimension, on a 5-point Likert scale.
Expert review. Six lecturers rate the items; four score below 0.78 and are dropped, two are reworded. The remaining 20 items give an S-CVI/Ave of 0.93.
Pilot test. You run the 20-item version with 40 students. Three items show item-total correlations below 0.30 and are removed, leaving 17 items with a Cronbach’s alpha of 0.84.
Factor analysis. With 210 students, an EFA gives a KMO of 0.87 and a significant Bartlett’s test. Three factors emerge, matching your dimensions. Two cross-loading items are removed, leaving a clean 15-item, three-factor scale.
Convergent and discriminant validity. Your scale correlates 0.62 with an established procrastination measure (good convergent evidence) and only 0.18 with a measure of general life satisfaction (good discriminant evidence). Criterion check. Students in the top quarter of procrastination scores submitted, on average, two days later than students in the bottom quarter — the expected pattern.
The result is a 15-item scale with documented evidence at every step — shorter, clearer, and better supported than the first draft.
Common Mistakes to Avoid
- Confusing reliability with validity. A high Cronbach’s alpha is necessary but not sufficient. It tells you the items hang together — not that they measure the right thing.
- Skipping expert review. Statistical checks cannot rescue items that do not cover the construct. Content validity must be built in, not tested in afterwards.
- Running factor analysis on a tiny sample. Factor analysis with 40 respondents produces unstable, misleading results. Collect enough data first — at least 100 respondents, or 5 to 10 per item.
- Reporting only the evidence that looks good. If an item failed or a correlation was weak, report it and explain what you did about it.
- Claiming your instrument is “valid” in general. Validity is always validity for a purpose and a population. A scale validated with university students is not automatically valid for school pupils or for employees.
FAQ
How many experts do I need for content validity?
Most guidelines recommend 5 to 10 experts. With fewer than five, a single disagreement can sink an item’s I-CVI; with more than ten, the panel becomes hard to manage. Six is a common choice, and with six or more experts an I-CVI of 0.78 is the accepted cut-off.
Can I validate a questionnaire I adapted from another study?
Yes — but adaptation still needs evidence. If you changed the wording, translated the items, or used the scale with a new population, re-check content validity with experts and re-run reliability and factor analysis on your own data. A scale that was valid in its original study is not automatically valid in yours.
What sample size do I need for factor analysis?
Aim for at least 100 respondents and 5 to 10 respondents per item, whichever is larger — so a 20-item questionnaire needs roughly 100 to 200 respondents. Larger samples give more stable factor solutions, so aim high when you can.
Is a Cronbach’s alpha of 0.70 good enough?
For a new or adapted instrument, 0.70 is generally accepted as the minimum, and 0.80 or above is preferable. But alpha measures internal consistency, not validity.
Do I need all four types of validity for my thesis?
No. Most theses present content validity, construct validity, and reliability — a strong package. Add criterion validity when a sensible external standard exists, and be honest about what you could not test.
The Bottom Line
Establishing validity is not a single test or a box to tick. It is a step-by-step argument: define the construct, write good items, let experts judge them, pilot the instrument, test its structure, and check that it behaves the way the construct should. Build that evidence as you go, report it honestly, and your questionnaire — and your thesis — will stand up to any examiner’s questions.