Introduction
You have spent weeks designing your questionnaire. The questions look clear to you, the order feels logical, and the response scales seem obvious. Then you send it out to 300 people — and the problems start pouring in. Respondents email you to ask what a question means. Half of them abandon the survey halfway through. And when the data comes back, you realise that one badly worded item has made a whole section useless. Learning how to pilot test a questionnaire before you collect your real data is the single best way to catch these problems while they are still cheap to fix.
A pilot test is a small trial run of your questionnaire with a handful of people who resemble your target respondents. You are not collecting data to analyse — you are testing the instrument itself. Does every question make sense? Does the survey flow naturally? How long does it take? Where do people get stuck, confused, or bored? A one-hour pilot test can save you weeks of regret, and in this guide you will learn exactly how to run one, step by step.
What Is a Pilot Test?
A pilot test (sometimes called a pre-test) is a rehearsal. You give your near-final questionnaire to a small group of people, watch how they respond, and ask them what was confusing, annoying, or unclear. Then you fix the problems before the real data collection begins.
It is important to understand what a pilot test is not. It is not a small version of your main study, and pilot responses should not be mixed into your final dataset. The pilot group is too small and too specially treated to represent your population, and — as we will see in Step 2 — pilot participants have usually seen the questions before, which changes how they answer. Think of the pilot as quality control for your instrument, not as early data collection.
Every questionnaire benefits from a pilot test, but it is especially important when you have written your own items (rather than using a published scale), when your respondents include children, patients, or people answering in a second language, and when your survey will be distributed online where nobody can ask you for clarification.
Why Pilot Testing Matters
Researchers sometimes skip the pilot test because they are short on time. That is a false economy. Here is what a pilot test protects you from:
- Unclear wording. Words that seem obvious to you may mean something different to your respondents. A pilot catches jargon, double meanings, and culturally loaded phrases.
- Bad timing. If your survey takes 25 minutes but you told respondents it takes 10, they will quit halfway — and you will lose exactly the busy people you most need.
- Broken flow. Questions in the wrong order, confusing skip logic, and missing response options only show up when real people try to complete the survey.
- Technical glitches. Online surveys break in ways you cannot predict: a question that does not display on a phone, a “next” button hidden below the fold, a required question with no valid answer for some respondents.
- Response-scale problems. A scale with no “not applicable” option, or one where every item is worded in the same direction, invites careless or forced answers.
How to Pilot Test a Questionnaire in 7 Steps
Step 1: Decide What You Want to Learn
Before you recruit anyone, write down the questions your pilot test must answer. A good list usually includes: Is every item clear? Is any item embarrassing, sensitive, or likely to be skipped? Does the survey flow logically? How long does it take? Do the instructions make sense? Are the response options complete — is there an answer for everyone? Having this list keeps the pilot focused and makes the debrief (Step 5) much easier.
Step 2: Choose Your Pilot Participants
Aim for a small group of about 5 to 10 people who resemble your target respondents as closely as possible. If your study targets first-year nursing students, pilot with first-year nursing students — not your supervisor, not your classmates from another programme, and not your family. People too close to the research tend to read what you meant rather than what you actually wrote, so they miss exactly the problems you need to find.
As a general rule, do not include pilot participants in your main sample later. They have already seen the questions, thought about them, and discussed them with you, so their second answers will not be fresh. Recruiting 5 to 10 fresh people for the main study instead is a small price for clean data.
Step 3: Brief Them Properly
Explain that you are testing the questionnaire, not them. Ask participants to be brutally honest: if a question is confusing, silly, or annoying, you want to hear about it. The most useful instruction you can give is “think aloud” — ask them to say whatever comes into their head as they read each question. You will be amazed at what you learn from hearing someone hesitate over a word you never questioned.
Also tell them you will be timing the survey, so they should complete it at their normal pace rather than rushing. And reassure them that their answers are confidential and will not be used in the study.
Step 4: Run the Test Under Real Conditions
The pilot should mimic the real data collection as closely as possible. If respondents will complete the survey on their phones, pilot it on a phone. If it will be a paper questionnaire handed out in a classroom, pilot it on paper in a classroom. Do not sit beside participants explaining items as they go — in the real study you will not be there, so any explanation you give during the pilot hides a problem you need to see. Sit nearby, stay quiet, take notes, and record how long each person takes from the first question to the last.
Step 5: Ask Structured Debrief Questions
When each person finishes, interview them briefly. Do not just ask “was it okay?” — that invites a polite yes. Ask specific questions instead:
- Was any question difficult to understand? Which one, and what did you think it meant?
- Was any question uncomfortable or intrusive?
- Did any question feel repetitive?
- Was there a question you wanted to answer differently but could not find the right option?
- Did the survey feel too long? At what point did you start losing interest?
- Did the instructions at the start make sense?
- Was anything unclear about the response scales?
- Is there anything important about this topic that we did not ask?
Write down the answers, including the exact words participants use. If two or more people stumble on the same item, that item needs rewriting — it is not a coincidence.
Step 6: Analyse What You Found
Look at the pilot results question by question. Check the completion times: if the average is much longer than you planned, you need to cut items. Look at the answers themselves: an item where everyone picks the same response may be leading or trivially easy, and an item with a strange spread of answers may be interpreted in different ways. Note where people hesitated, skipped, or asked for help — those are your problem spots. Also check for patterns in missing answers: a question many people leave blank is usually confusing, sensitive, or badly placed.
Step 7: Revise — and Re-Test If You Changed a Lot
Fix what the pilot revealed: rewrite confusing items, reorder questions, add missing response options, and shorten the survey if it ran long. Small fixes — a word changed here, an option added there — are usually safe to accept without another pilot. But if you rewrote several items, restructured the survey, or changed the response scales, run a second, smaller pilot with two or three fresh people. It takes an hour and confirms that your fixes actually fixed the problems instead of creating new ones.
A Worked Example: Piloting a 15-Item Student Wellbeing Questionnaire
Imagine you have designed a 15-item questionnaire about student wellbeing for final-year undergraduates. You recruit 8 final-year students who will not be in your main sample and ask them to complete it on their phones, thinking aloud.
The results are instructive. The median completion time is 6 minutes — close to the 5 minutes you promised, so no cuts are needed. But three problems emerge. First, two participants hesitate over the item “I feel in control of my academic workload,” because “in control” feels vague; you rewrite it as “I can manage my academic workload without feeling overwhelmed.” Second, one participant points out that the item “How often do you exercise?” offers options from “never” to “daily” but no option for “several times a week” that fits their routine — you add a missing middle option. Third, three participants say the two reverse-worded items (“I often feel hopeless about my studies”) feel jarring after a run of positive items; you keep the items but add a short instruction line explaining that some statements are worded differently on purpose.
You make the three fixes and run a quick second pilot with two fresh students. Both complete the survey without hesitation, and the timing holds at 6 minutes. Your questionnaire is ready — and you know it works, instead of hoping it does.
5 Common Pilot Testing Mistakes
1. Piloting on the wrong people. Your supervisor and your research group already know the topic too well to spot unclear wording. Pilot with people who match your actual respondents.
2. Explaining items as people answer. Every explanation you give hides a problem. Stay quiet, observe, and let the confusion show itself.
3. Ignoring the timing. If you never measure completion time, you will promise respondents 10 minutes and deliver 25 — and your response rate will pay for it.
4. Treating pilot data as real data. Pilot responses come from a tiny, specially treated group. Including them in your final sample can distort your results, so keep them separate and report the pilot honestly in your methodology chapter.
5. Skipping the re-test after major changes. If the pilot forced you to rewrite several items, those new items are untested. A quick second pilot with two or three fresh people is cheap insurance.
Frequently Asked Questions
How many people do I need for a pilot test?
For most student questionnaires, 5 to 10 people is a good starting point — enough to reveal the common problems without turning the pilot into a second study. If your questionnaire is very long, targets a diverse population, or will be translated, aim for the higher end of that range and include people from each subgroup.
Can I include pilot responses in my final sample?
As a general rule, no. Pilot participants have seen the questions before, discussed them with you, and may answer differently the second time. Most methodology textbooks and examiners expect pilot participants to be excluded from the main sample, so plan your recruitment numbers with that in mind.
How long should a pilot test take?
The pilot itself usually takes 30 to 60 minutes per participant, including the debrief interview. Allow a day or two afterwards to analyse the feedback and revise the questionnaire. If your survey is long or your revisions are extensive, budget time for a second round — it is far quicker than fixing a broken dataset later.
Do I need ethical approval for a pilot test?
In most institutions, yes — pilot testing with human participants falls under the same ethical approval as the main study, because you are still collecting information from people. Check your institution’s rules and your approved ethics application before you start, and if your pilot was not covered in the original approval, ask your supervisor or ethics committee whether an amendment is needed.
Final Checklist Before You Launch
Before your questionnaire goes out to real respondents, run through this checklist. Have you piloted it with 5 to 10 people like your target respondents? Have you timed it? Have you fixed every item that two or more people stumbled on? Have you re-tested after any major changes? And have you kept the pilot responses out of your final dataset? If you can answer yes to all five, your questionnaire is ready.
Pilot testing is not a bureaucratic hurdle — it is the difference between data you can trust and data you have to explain away in your limitations section. A single afternoon spent piloting can save you from the one sentence no researcher wants to write: “the questionnaire was not pilot tested.”
Next, learn how to design a research questionnaire so the items you pilot are strong from the start — and if your survey uses rating scales, read our guide to Likert scales before you finalise your response options.