Cohen’s kappa helps nursing researchers describe agreement between two assessors who classify the same cases. For example, two nurses might independently record whether a checklist item is present. Their ratings can agree often, but the pattern of categories also matters.
Start by asking what the assessments represent. Agreement between two observers is different from accuracy against a reference standard. Likewise, agreement on categories is different from correlation between numerical scores.
This guide explains the formula through invented nursing examples. Next, it shows how to prepare paired ratings, interpret the result and report limitations. All examples are teaching data, rather than findings about a real instrument or clinical service.
When is Cohen’s kappa useful in nursing research?
Use ordinary Cohen’s kappa when two raters assign each case to the same set of categories. The categories might be “present” and “absent”, or several unordered classifications. Each case must have a rating from both observers for the chosen complete-pair analysis.
For example, a nursing education study could ask two assessors to classify 50 simulated observations using one checklist. The question is whether they apply its categories consistently. It is not whether one group of students performs better than another.
The unit of analysis could be a patient, record, simulation or observation. However, name that unit explicitly. Fifty ratings from each nurse do not automatically mean that you have 100 independent cases.
Before choosing the statistic, check the measurement scale. Ordinary kappa treats all disagreements as category mismatches. By contrast, an ordered scale may require a justified weighted kappa, while numerical measurements need another agreement method.
Our statistical-test selection guide explains broader method choices. Here, the main question is agreement on paired categorical assessments.
Separate agreement, association and accuracy
Agreement asks whether observers give the same classification to the same case. Association asks whether variables show a relationship. Accuracy asks whether an assessment matches a defensible reference classification. These questions can require different methods.
For instance, two observers could use a numerical scale in a closely related way while one consistently awards higher scores. Their correlation might be strong even though their scores do not match. Therefore, a correlation coefficient alone does not establish agreement.
Similarly, two nurses could agree on every classification while both misunderstand the instrument. Perfect agreement would describe their consistency with each other. It would not prove that their classifications are correct.
If the research question concerns detection of a condition against a reference standard, consider diagnostic-accuracy measures. Our sensitivity and specificity guide explains that different question.
A well-written methods section connects the statistic to the question. Instead of saying “kappa was used because the data were categorical”, explain that two raters independently classified the same observations using identical categories.
Build a paired agreement table
Place the first rater’s categories in the rows and the second rater’s categories in the columns. Then count cases in each combination. In a square table, the diagonal cells represent exact agreement.
Imagine two nurses assessing 50 simulated records for the presence of a specified checklist feature. Nurse A records “present” in 25 cases and “absent” in 25. Nurse B records “present” in 21 and “absent” in 29.
| Nurse A rating | Nurse B: present | Nurse B: absent | Row total |
|---|---|---|---|
| Present | 18 | 7 | 25 |
| Absent | 3 | 22 | 25 |
| Column total | 21 | 29 | 50 |
The observers agree in 18 present cases and 22 absent cases. Consequently, observed agreement is 40 out of 50, or 80%. They disagree on the remaining ten cases.
Notice that disagreement has a direction. Nurse A marks the feature present while Nurse B marks it absent seven times. The opposite pattern occurs three times. Therefore, the full table reveals more than the overall percentage.
Keep the category labels consistent across both dimensions. If one column uses “yes” and another uses “present”, document the equivalence before analysing. Otherwise, software may treat coding differences as substantive categories.
The Cohen’s kappa formula explained
The usual formula is κ = (Po − Pe) ÷ (1 − Pe). Here, Po is the observed agreement proportion. Pe is the agreement expected under the model based on the two raters’ marginal category proportions.
The marginal proportions come from the row and column totals. For each category, multiply the proportion assigned by Nurse A by the proportion assigned by Nurse B. Then add those products across categories.
In the example, Nurse A assigns half the cases to each category. Nurse B assigns 21/50, or 0.42, to present and 29/50, or 0.58, to absent. Therefore, Pe = (0.50 × 0.42) + (0.50 × 0.58) = 0.50.
Now substitute the observed agreement: κ = (0.80 − 0.50) ÷ (1 − 0.50) = 0.30 ÷ 0.50 = 0.60. The calculation describes agreement beyond that model’s expected agreement, relative to the remaining possible improvement.
MedCalc’s documentation sets out this formula and distinguishes ordinary from weighted kappa (MedCalc, n.d.). However, do not describe Pe as proof that the nurses actually guessed. It is a model-based comparison using their category proportions.
Interpret the number without overstating it
A value of one indicates perfect agreement when the statistic is defined. Zero indicates agreement equal to the model’s expected level. Negative values indicate agreement below that expected level.
However, kappa is not simply the percentage of cases on which observers agreed. In our example, observed agreement is 80%, while kappa is 0.60. Writing “the nurses agreed on 60% of cases” would therefore misreport the result.
Some papers attach verbal labels to numerical bands. Such labels can help describe a result, but they are conventions rather than universal clinical acceptance thresholds. A value near a boundary should not produce a dramatic change in your conclusion.
Instead, report the numerical estimate, observed agreement, uncertainty and context. Consider which disagreements occurred and what they would mean in the proposed application. A mismatch on a low-stakes coding task may have different consequences from a disagreement affecting urgent care.
For the hypothetical data, a cautious sentence is: “The two assessors agreed on 40 of 50 classifications (80%); Cohen’s kappa was 0.60.” A real report should also provide a suitable confidence interval and cite the actual study or approved dataset.
Why category prevalence affects the result
Kappa depends partly on how often each category is used. Therefore, two datasets with similar observed agreement can produce different kappa values. Do not interpret a change in kappa without checking the marginal totals.
Consider a second invented sample of 100 observations. Both raters say “absent” for 90 cases and “present” for five. Nurse A alone says present for three, while Nurse B alone says present for two.
Observed agreement is 95/100 = 95%. Nurse A uses present in eight cases and Nurse B in seven. Consequently, expected agreement is (0.08 × 0.07) + (0.92 × 0.93) = 0.8612.
Kappa is therefore (0.95 − 0.8612) ÷ (1 − 0.8612), approximately 0.64. Despite 95% exact agreement, the coefficient is much lower than 0.95. The common absent category creates a high expected agreement in the model.
This is an illustration of the calculation, not a claim that the raters performed poorly. Sim and Wright (2005) discuss prevalence, differences between raters and non-independent ratings as influences on interpretation. Accordingly, examine the table before applying a verbal label.
Do not change the sample after seeing an inconvenient result. Instead, describe its category distribution and assess whether that distribution reflects the population relevant to your research question.
Check independence and assessment procedures
Each pair of ratings belongs to the same case, so the two ratings within a pair are deliberately linked. The study should nevertheless prevent one observer’s assessment from influencing the other when independent ratings are intended.
For example, one nurse should not copy a colleague’s classification and then present it as an independent assessment. Shared discussion may be appropriate for training or consensus, but it changes the meaning of the agreement exercise.
Also distinguish ratings before and after adjudication. Consensus reached through discussion can be useful for resolving cases. However, reporting only the consensus as evidence of initial inter-rater agreement conceals the original disagreements.
Across cases, repeated observations from the same patient or closely linked records may require additional statistical consideration. Treating all those observations as independent can affect standard errors and confidence intervals. Explain clustering to your supervisor or analyst before choosing the analysis.
Timing also matters. If assessors observe a changing condition on different days, disagreement may reflect genuine change rather than inconsistent use of the tool. Therefore, describe the interval and whether both assessed the same information.
Prepare the dataset before using software
Use one row per case for a simple paired assessment dataset. Include a case identifier and a separate rating variable for each observer. The identifier helps verify pairing; it should not expose personal information in a submitted report.
| Teaching case | Nurse A | Nurse B |
|---|---|---|
| 001 | Present | Present |
| 002 | Present | Absent |
| 003 | Absent | Absent |
Next, check duplicate identifiers, missing ratings and unexpected codes. A value such as 9 might mean “not assessed” rather than a third substantive category. Document that coding and apply the approved missing-data rule.
Do not sort the two rating columns separately. Doing so would pair different cases and destroy the study’s structure. Likewise, do not match records merely because they happen to appear in the same row of separate exports.
Before calculating kappa, create the cross-tabulation and compare totals with the number of usable pairs. Our missing-data guide can support transparent reporting of excluded observations.
Finally, preserve an unedited source dataset and a record of cleaning decisions. For broader software preparation, see our SPSS data-analysis guide. Verify software settings against the method specified in your protocol.
Handle missing ratings transparently
A complete-pair analysis includes cases with a rating from both observers. If a study starts with 60 cases but only 50 have two usable ratings, state both numbers. The smaller paired denominator belongs in the agreement calculation.
However, removing incomplete pairs can affect applicability. Suppose difficult observations are more likely to lack a second rating. The retained sample may then make agreement look better than it would across all eligible cases.
Explain why ratings were missing where that information exists. Separate administrative omissions from observations that could not be classified. An explicit “uncertain” category may be meaningful if the instrument defines it in advance.
Do not automatically convert every missing value to “absent”. That would manufacture a rating and could change both observed and expected agreement. Similarly, filling missing ratings with the other assessor’s value artificially creates agreement.
If a paper does not explain exclusions, describe the reporting gap. You cannot infer that an unspecified procedure was definitely wrong. Nevertheless, the missing information limits your ability to judge the estimate.
Ordinary and weighted kappa answer different questions
Ordinary kappa counts any category mismatch as disagreement. This is suitable when the categories have no meaningful order or when exact matching is the intended criterion.
For ordered categories, the size of a disagreement may matter. For example, adjacent levels on a three-level teaching rubric differ from the lowest and highest levels. Weighted kappa allows a specified weighting scheme to reflect that distinction.
However, the weights change what the coefficient measures. Linear and quadratic schemes can produce different results from the same ratings. Therefore, report the scheme and justify it before interpreting the number.
Do not describe a weighted value as ordinary kappa. Likewise, avoid comparing studies as if their coefficients are equivalent when one uses exact agreement and another discounts small disagreements.
For three or more assessors, a single ordinary two-rater coefficient is not automatically the appropriate summary. Other methods may be suitable depending on the design. Seek methodological advice rather than pooling ratings into invented pairs.
Read the confidence interval and sample-size justification
A kappa estimate from a sample has uncertainty. The confidence interval helps describe the precision of that estimate under the method used. A point estimate alone can conceal substantial uncertainty.
For example, a small study might produce a seemingly encouraging coefficient with a wide interval. A larger study could produce the same coefficient with greater precision. Therefore, read the interval alongside the number of paired observations.
Do not construct a confidence interval by adding and subtracting an arbitrary amount. Use an appropriate software procedure and explain it where your reporting requirements call for that detail.
Sample-size planning should reflect the design, expected category proportions and the precision or hypothesis of interest. A generic rule based only on the number of questionnaire items is not a sufficient justification for every agreement study.
Our sample-size guide introduces planning principles. Likewise, our confidence intervals article helps with interpretation. An agreement study may still need a tailored calculation.
Report Cohen’s kappa clearly in a dissertation
Begin with the assessors, observations and classification task. Then describe training, assessment timing and whether observers rated independently. State the category definitions and the number of complete pairs.
Next, present the cross-tabulation or enough counts to show the main disagreement pattern. Report observed agreement and the kappa estimate, together with its confidence interval where available. Specify weighting if used.
A hypothetical results paragraph could read: “Two nurses independently classified 50 simulated records as feature present or absent. Exact agreement occurred in 40 records (80%). Cohen’s kappa was 0.60; disagreements occurred in ten records, with seven classified present only by Nurse A.”
This paragraph deliberately reports only quantities calculated above. For a real study, add a verified confidence interval and the relevant study citation. Do not present the teaching paragraph as a clinical finding.
Kottner et al. (2011) developed GRRAS to improve reporting of reliability and agreement studies. EQUATOR lists that guideline for the whole report. Use it to check reporting completeness, rather than turning it into an invented numerical quality score.
For tables and statistical prose, consult our nursing results chapter guide. Keep conclusions tied to what the ratings and study design actually support.
Common mistakes to correct before submission
First, check whether you confused kappa with a percentage. Then verify pairing and category coding. These basic checks can prevent a technically correct calculation from answering the wrong question.
- Report observed agreement separately from kappa.
- Keep both raters’ category labels consistent.
- Explain missing and excluded pairs.
- Describe independent ratings before consensus.
- Check marginal totals and category frequency.
- State any weighting scheme.
- Include uncertainty where available.
- Avoid treating agreement as proof of validity.
Finally, read the conclusion against the methods. If the study assessed simulated records with two trained nurses, do not generalise automatically to every clinical setting and assessor. Explain the scope of the evidence.
Frequently asked questions
Is Cohen’s kappa the same as percentage agreement?
No. Percentage agreement counts exact matches. Kappa also uses expected agreement based on the raters’ marginal category proportions. Therefore, report both when possible.
Does a kappa of 0.60 mean agreement in 60% of cases?
No. In our invented example, kappa is 0.60 while exact agreement is 80%. The coefficient and the percentage describe different quantities.
Can kappa prove that an assessment tool is valid?
No. Observers can agree while both apply an incorrect classification. Validity needs evidence that the tool measures or classifies what it is intended to assess.
When should weighted kappa be considered?
Consider it for ordered categories when the size of disagreement matters. However, justify the weights and report the scheme because it affects interpretation.
Why can high agreement produce a lower kappa?
A common category can produce high expected agreement from the marginal totals. Consequently, examine the full table and category distribution alongside the coefficient.
Conclusion
Use Cohen’s kappa to answer a defined question about paired categorical ratings. Check the table, explain the expected-agreement calculation and report the coefficient alongside exact agreement and uncertainty. Then assess what the disagreements mean in context.
Get feedback on your agreement analysis
If your paired dataset or interpretation needs checking, request focused statistical-methods feedback. Share your protocol, anonymised table and own draft so you can understand the method and explain each reporting decision.
References
- MedCalc (n.d.) ‘Inter-rater agreement’, Biostatistics in Practice. Available at: MedCalc agreement documentation (Accessed: 2 October 2026).
- Sim, J. and Wright, C.C. (2005) ‘The kappa statistic in reliability studies: use, interpretation, and sample size requirements’, Physical Therapy, 85(3), pp. 257–268. Available at: PubMed article record (Accessed: 2 October 2026).
- Kottner, J. et al. (2011) ‘Guidelines for Reporting Reliability and Agreement Studies (GRRAS) were proposed’, Journal of Clinical Epidemiology, 64(1), pp. 96–106. doi: 10.1016/j.jclinepi.2010.03.002. Available at: GRRAS article record (Accessed: 2 October 2026).
- EQUATOR Network (n.d.) ‘Guidelines for reporting reliability and agreement studies (GRRAS) were proposed’. Available at: GRRAS reporting guideline (Accessed: 2 October 2026).
