Pilot survey items should not be removed simply because they attract missing answers, unusual response patterns or criticism in a nursing dissertation pretest. First identify what the evidence means and what deletion would change. This guide explains how to choose between retaining, revising, removing or deferring an item while protecting conceptual coverage and the integrity of an established scoring method.
This guide examines item-retention decisions after a pilot. Its practical output is an item decision register showing retain, revise, retest or remove, with the consequences for construct coverage and scoring. Begin with the how to pilot a nursing research questionnaire and record changes if you first need to establish the broader project approach. Use the detailed decisions below once that foundation is clear.
Review pilot survey items against their purpose
Write down what each item is meant to contribute before examining whether it “performed well”. An item may represent an essential aspect of the concept, a screening route, a contextual variable or a repeated check. The reason for including it determines which problems matter.
A useful review asks whether the item is relevant, understood as intended and necessary for the proposed use. COSMIN’s content-validity guidance for patient-reported outcome measures distinguishes relevance, comprehensiveness and comprehensibility (COSMIN, 2019). Apply that framework within its scope rather than claiming that a simple deletion exercise establishes validity for every kind of questionnaire.
Separate evidence from a preferred shorter form
Researchers may want a shorter questionnaire because collection time is limited. That is a practical concern, but it should not be disguised as evidence that particular items are defective. Record burden and content decisions separately.
If shortening is a project requirement, explain what information can be lost and whether an authorised shorter version exists. Removing the most difficult questions may leave only easy but less informative items. A neat final survey is not necessarily a better measure of the question your dissertation originally set out to address.
Identify the instrument before changing it
Check whether you are reviewing a newly developed questionnaire, an adapted form or an established instrument with defined scoring. The implications differ. Deleting an optional contextual question is not the same as removing a component of a scored subscale.
Verify any adaptation conditions through the permissions process. Then consider the measurement consequences separately. Even where deletion is permitted, the modified form should not be presented as unchanged or supported by every finding reported for the original instrument.
Investigate missing answers before deleting
A blank response may reflect unclear wording, a sensitive topic, an inapplicable experience, a technical problem or an overlooked page. Those explanations call for different actions. Deletion without investigation can conceal a solvable administration issue.
Check whether the missingness follows a legitimate skip route and whether the item was actually displayed. Where approved qualitative feedback is available, use it to examine the problem. Do not assume that every blank answer means confusion or refusal. Record the evidence and any uncertainty about why respondents did not provide an answer.
Check relevance and eligibility
An item about an experience that many respondents have not had may need a clear eligibility route. The relevant decision could be to ask it only of an appropriate group, rather than removing the concept from the questionnaire entirely.
For example, an invented item about follow-up contact may not apply to people whose episode ended without planned follow-up. Before adding a “not applicable” option or a skip rule, consider permissions, scoring and the intended analysis. Make sure the final report distinguishes inapplicability from missing information and from an unfavourable experience.
Use comprehension evidence to choose a revision
If respondents interpret an item differently from its intended meaning, identify the source of the mismatch. It may concern a term, timeframe, assumed event or response option. A precise diagnosis supports a more focused revision.
The cognitive-interviewing guide explains how to explore the response process. Do not remove a concept merely because the first wording was weak. Retaining the concept with a justified, permitted revision may preserve important coverage, although the revision still needs appropriate testing.
Worked example: a difficult but important item
This fictional example uses original wording. A questionnaire asks whether patients knew whom to contact when plans changed. Some pilot respondents leave the item blank. The student initially proposes deleting it to reduce missing answers.
Follow-up evidence within the approved pretest shows two different situations. Some respondents had no change of plan and could not identify a relevant event. Others experienced a change but were unsure which contact route the question meant. Deletion would remove an important aspect of the evaluation rather than solve those two problems.
The review records a possible eligibility instruction and a clearer definition of the contact route, subject to permissions and approval. The next version needs testing. The pilot does not establish that either revision will work, but it supports a more specific decision than “delete because several people skipped it”.
Do not overinterpret floor or ceiling patterns
If most pilot respondents choose the lowest or highest option, consider the item’s purpose and the pilot group. A highly favourable response may reflect a genuinely common experience in that small group. It may also reflect recruitment, wording or social conditions.
Do not declare the item useless from a small distribution alone. Ask whether the measure needs to distinguish people at that end of the scale and whether the pilot includes the intended range of experiences. Report the limited evidence without inventing a universal percentage threshold for deletion.
Protect rare but consequential content
An infrequently endorsed item can still concern an important experience. Frequency is not the same as relevance. Removing every uncommon concern can produce a questionnaire that describes routine satisfaction while missing the issue the service most needs to understand.
Explain why the content belongs in the project and whether the item is an appropriate way to ask about it. Sensitive or consequential topics may require particular attention to wording and local procedures. Do not retain a poorly handled item merely because the subject matters; decide how the concept can be examined responsibly within the approved design.
Examine apparent duplication conceptually
Two items may look similar while addressing different stages or perspectives. Conversely, differently worded items may ask essentially the same thing. Compare their intended contribution before deciding that one is redundant.
The guide on mapping questionnaire items to objectives can help show the content role. Use that map to explain the consequence of removal rather than repeating the entire mapping process. After a proposed deletion, check whether a domain or objective has lost its only relevant item.
Do not optimise a small pilot around one statistic
A reliability coefficient or item statistic can be useful within an appropriate design, but it should not become an automatic deletion instruction. A small pilot may produce unstable patterns, and numerical consistency does not establish that the questionnaire covers the intended concept.
If your project includes formal psychometric analysis, follow the justified analytical plan and appropriate guidance. This article does not offer a factor-analysis recipe or a universal coefficient threshold. The decision here concerns how pilot evidence, content and practical consequences should be brought together transparently.
Consider scoring and comparability
Removing an item can change a total score, subscale composition, missing-data rule or comparison with published work. Check the authorised instructions before calculating a familiar score from an altered item set. A similar-looking number may have a different meaning.
If the final instrument is modified, identify the modification and limit claims accordingly. Do not silently prorate scores or substitute a new denominator unless that procedure is justified for the exact use. Where the project cannot support the intended score interpretation, discuss a narrower analysis rather than presenting false comparability.
Use a retain–revise–remove–defer table
| Decision | Possible rationale | What still needs checking |
|---|---|---|
| Retain | Relevant content with no identified problem in available evidence | Limited pilot coverage and untested uses |
| Revise | Specific wording or response-task concern | Permission, construct meaning and retesting |
| Remove | Unnecessary or unsuitable content with a defensible rationale | Coverage, scoring and comparability consequences |
| Defer | Insufficient evidence or unresolved conditions | What information or decision is needed next |
This is an original planning aid, not a validated item-selection instrument. Use it to expose the reasoning rather than to generate an automatic answer.
Worked example: a genuinely unnecessary question
In a fictional service-experience questionnaire, an item asks for a detailed employment category that has no role in the evaluation question or planned analysis. Pilot feedback indicates that respondents find the question intrusive. No scoring system depends on it, and the team cannot identify a justified purpose.
Removal may be reasonable, subject to the agreed process. The rationale is unnecessary collection and lack of relevance, supported by feedback about burden. It is not simply that participants disliked answering. The review also checks that no approved analysis or eligibility decision depends on the field.
The change log identifies the removed item, evidence, decision date and version. The final methods explain the relevant design change without reproducing sensitive participant comments. This provides a clearer rationale than a generic statement that the questionnaire was shortened after piloting.
Record unresolved disagreement in the team
A supervisor, service partner and student may disagree about whether an item is essential. Record the competing reasons and the evidence each depends on. Do not represent a provisional discussion as formal consensus.
If the decision depends on a missing permission response or a scoring rule that has not been verified, label that dependency. A deferred decision can be more defensible than an unsupported final choice. Clarify who has responsibility for the decision and what must happen before the instrument is administered.
Retest changes rather than assuming success
A revised item can introduce a new ambiguity, and deletion can disrupt the flow of instructions or skip routes. Check the final questionnaire as a whole, not only the line that changed. Version control is especially important when several edits are made together.
The parent guide on piloting and recording changes explains the wider process. Keep evidence from each version separate. Do not write that participants approved a final form when they only saw an earlier draft.
Report the decision with appropriate limits
A good methods account explains what was reviewed, which evidence informed the decisions and what changed. It also states what the pilot could not establish. Avoid claiming that removing several items made the questionnaire valid or eliminated response bias.
In the discussion, identify consequences for coverage and interpretation. If the final form differs from an established instrument, readers need to know that before comparing results with published scores. Transparent limitations help the reader judge the contribution; they do not need to be hidden behind a claim that the pilot “confirmed reliability”.
Check the effect on the questionnaire’s flow
Deletion can alter numbering, instructions, skip routes and the context in which neighbouring items are read. Review the final form from a participant’s perspective. A removed question may leave a later instruction referring to an answer that no longer exists.
Where the questionnaire is digital, check the approved routing logic and exported variable labels. These practical checks do not establish measurement validity, but they can prevent a justified content decision from creating a new administration problem. Keep the version used for this review identifiable in the change log.
Distinguish a collection decision from an analysis exclusion
Removing an item before the main study changes what participants are asked. Excluding an already collected variable from an analysis is a different decision and requires its own rationale. Do not describe the latter as though the item was never administered.
If an item is collected but not analysed as originally planned, explain the reason and any effect on the reported score or coverage. Follow the relevant approved plan and institutional expectations. Avoid selectively omitting an awkward finding while claiming that the questionnaire was improved during a pilot.
This distinction matters when the pilot and main collection are separate stages. Evidence from the pilot may justify a documented revision before the main stage. It does not provide a retrospective reason to erase data from the main stage without explanation. A clear chronology helps readers assess the decision fairly.
Use a final coverage review
After all proposed changes, return to the question and check what remains represented. Identify any domain now covered by only one ambiguous item or by no item at all. The review should consider the combined consequences of several small deletions. Individually reasonable changes can collectively narrow the instrument beyond the purpose originally agreed.
Pilot survey items review checklist
- Identify the intended contribution of each item.
- Check the exact instrument and adaptation conditions.
- Investigate missingness and eligibility before proposing deletion.
- Use comprehension evidence to locate a specific problem.
- Protect relevant content even when it is uncommon.
- Consider scoring, coverage and comparability consequences.
- Record retain, revise, remove or defer decisions with reasons.
- Retest relevant changes and report the pilot’s limits.
Check the abstract’s account of revision
Use the same decision language in the abstract and methods. If items were revised after limited pretesting, do not shorten that account to “a validated questionnaire was developed”. State the actual stage completed and the important limitation. Accurate summary wording protects a careful item-review process from becoming an unsupported measurement claim.
Retain enough detail to explain why the final form differs from the starting version and what that difference means for interpretation.
Frequently asked questions
Should I remove every item with missing answers?
No. Investigate whether the item was applicable, displayed correctly and understood. Missing answers can arise for different reasons. The right response may be a clearer instruction, an approved skip route or a narrower interpretation rather than deletion.
Can I delete an item to improve a reliability coefficient?
Do not use one statistic as an automatic instruction. Consider the design, sample, content coverage and scoring consequences. If formal psychometric analysis is planned, follow an appropriate justified method and avoid overinterpreting unstable pilot results.
Is a rarely endorsed item unimportant?
Not necessarily. It may concern an uncommon but relevant experience. Assess its contribution to the question and the intended population. Frequency alone does not determine relevance, and a small pilot cannot establish how common an experience is in the wider service.
Can I still use the original questionnaire’s name?
Identify the original source, but disclose meaningful modifications and follow its conditions. Do not imply that an altered item set is the unchanged instrument or that all original scoring evidence applies automatically. Explain exactly which version was administered.
What if there is not enough evidence to decide?
Record a deferred decision and identify the missing information. Discuss the dependency with your supervisor before collection. A transparent unresolved issue is preferable to an invented rationale or a claim of approval that no responsible party has actually given.
Explain a defensible revision decision
Our nursing dissertation methods support can help you organise an anonymised item-review rationale. Ask about the specific revision decision rather than sending restricted instrument text or participant records. Use feedback to develop your own explanation and follow the institution’s approved process for changes.
Reference
COSMIN (2019) Study design checklist for patient-reported outcome measurement instruments. Available at: COSMIN (Accessed: 16 October 2026).
