Inter-Annotator Agreement: Choose a Statistic That Fits the Task

Define the decision before the label

Agreement analysis is useful only when it makes a downstream decision more consistent. Interface precision is not conceptual precision: a reviewer can place an exact boundary, box or class while applying the wrong rule.

Here the practical lens is choose a statistic that fits the task. Read the primary source for the authors' definitions and results; then translate the idea into an observable instruction, a nearest confounder and a cost for getting the decision wrong.

Write an evidence contract

A production rule should tell qualified reviewers what they can observe, what they must not infer, and what to do when the evidence is insufficient. At minimum, preserve:

• The observable rule that distinguishes each label and its nearest confounder.
• Positive, negative, boundary and insufficient-evidence examples.
• Reviewer disagreement by class, slice and severity.
• Versioned instructions, adjudications and acceptance criteria.

Examples should include clean positives, clean negatives and difficult boundaries. Without all three, agreement can look healthy simply because the calibration set avoided the decision that the ontology must resolve.

Use disagreement as specification evidence

The main risk is that apparent precision conceals missing evidence or disagreement about the concept. When two careful reviewers disagree, first ask whether they saw different evidence or applied different definitions. Averaging their answers may hide the defect rather than resolve it.

Separate isolated execution mistakes from systematic ambiguity. Repeated disagreement by class, visibility, sequence phase or reviewer role is a signal to change the instruction, context or label set. Only after the rule is stable does individual quality measurement become meaningful.

A production-ready calibration sequence

1. Frame the decision. Tie the technique to the downstream decision and cost of a wrong label.
2. Calibrate on ambiguity. Test the rule on representative ambiguity before production.
3. Diagnose disagreement. Separate individual mistakes from missing evidence and specification defects.
4. Set the release gate. Release only after the rule survives annotation, review and customer sampling.

Run the sequence on a deliberately mixed slice: ordinary cases, costly errors, near-confounders and incomplete evidence. Record every adjudication that changes the rule, then relabel the slice. The residual disagreement should reflect genuine uncertainty, not private interpretations.

Questions for the decision meeting

Before approving production work, the product, data and domain owners should be able to answer four questions in the same language:

• What changes if this works? Name the model behavior, evaluation decision or operational risk this evidence is meant to improve.
• Which conditions are still unrepresented? List the environments, sensors, actors, embodiments or failure modes that remain outside the claim.
• Where can qualified reviewers still disagree? Decide whether the remedy is more context, a clearer rule, an uncertainty label or domain adjudication.
• What result would stop or redirect the program? Define that threshold before scale, while the team can still change the collection and ontology inexpensively.

Write the answers into a one-page decision record and attach the calibration evidence. That record is more useful than a broad claim that the data is “high quality”: it identifies the intended use, the boundary of the evidence, the unresolved risks and the person accountable for accepting them. Revisit it when the model, ontology, collection hardware or operating environment changes.

What we would do next

We would select a compact, representative bridge set from the target environment and review every disagreement that could alter the operating decision. The resulting record becomes the first version of the ontology, calibration examples, escalation policy and acceptance test—not a polished demo disconnected from production.

The pilot should produce evidence even when the recommendation is to stop. A useful outcome may be a narrower ontology, a missing sensor requirement, a revised evaluation slice or proof that the proposed signal does not justify its cost. That is preferable to scaling a workflow whose assumptions have never been tested.