Annotation techniques

Temporal Annotation for Video: Events, States, and Failure Boundaries

How to design temporal labels that distinguish what happened, what remained true, and exactly where a failure became observable.

9 minute readGrasp editorial
Robot grasp sequence aligned with event segments, state bands and a highlighted failure boundary

A boundary is a claim about evidence

Temporal annotation often starts with a deceptively simple instruction: mark when the action begins and ends. Two careful annotators then disagree by half a second—not because one is inattentive, but because “begins” was never made observable.

Consider a robot picking up a cup. The episode contains several defensible starts: motion toward the cup, first contact, gripper closure, or the moment the cup leaves the surface. Each boundary answers a different question. A control model, an action recognizer and a failure-analysis system should not necessarily share one label.

Reliable temporal ground truth separates three objects:

  • an event, which is a change or occurrence;
  • a state, which remains true over an interval;
  • a boundary, which records the transition under an explicit evidence rule.

Events and states carry different information

An event might be first contact, release, collision or object begins to slip. It is localized to a moment or narrow transition window. A state might be gripper holding object, drawer open, person occluded or vehicle stopped. It spans time.

Trying to represent both as one generic segment creates avoidable ambiguity. If “pick up cup” runs from first movement to stable hold, the label combines approach, grasp, lift and stabilization. That may be appropriate for coarse retrieval, but it is poor supervision for diagnosing where a manipulation failed.

A stronger schema can include:

  1. atomic events: contact, closure, lift-off, slip onset, impact;
  2. interval states: in contact, grasp secured, object airborne, failure active;
  3. composite task phases derived from those primitives.

The primitives let teams change downstream phase definitions without relabeling every frame.

Write boundary rules with observable predicates

“When the grasp starts” is interpretive. “First frame in which either gripper surface visibly contacts the target” is observable. The second rule can still encounter uncertainty, but the uncertainty now has a location and a cause.

For each boundary, specify:

  • positive evidence: what must be visible or measurable;
  • negative evidence: what similar situation does not qualify;
  • temporal convention: first qualifying frame, last non-qualifying frame, or midpoint of an uncertainty interval;
  • sensor precedence: which signal wins when video and telemetry disagree;
  • occlusion policy: mark unknown, infer under constraints, or defer for review;
  • tolerance: whether evaluation accepts a frame window rather than an exact instant.

Generic Event Boundary Detection research is instructive here: human observers can perceive meaningful transitions at multiple granularities, and multiple annotations help capture genuine ambiguity. The lesson for production is not to collect endless opinions. It is to decide which granularity serves the model and where disagreement itself should be preserved.

Failure needs two boundaries, not one label

In many robotics episodes, a failure has at least three relevant moments:

  • the causal precursor, such as an off-center grasp;
  • the first observable deviation, such as object rotation beyond tolerance;
  • the terminal consequence, such as impact with the floor.

If the ontology marks only “failure” at impact, it is suitable for outcome counting but weak for learning recovery or early intervention. If it marks the precursor as failure without distinguishing hindsight from observable evidence, it can leak future knowledge into the label.

A practical schema records separate fields:

  • outcome: success, partial success, recoverable failure, terminal failure;
  • failure_observed_at: first point where the defined evidence threshold is met;
  • failure_type: slip, collision, miss, timeout, wrong object, safety stop;
  • recovery_window: interval during which the policy could still recover;
  • precursor_notes: causal analysis for review, not necessarily a training target.

That separation supports both policy training and post-hoc diagnosis without confusing them.

Annotate uncertainty as data

Frame-exact tools can create the illusion of frame-exact truth. Motion blur, asynchronous sensors and occlusion make some boundaries inherently uncertain. Forcing an annotator to choose one frame hides that uncertainty.

Useful options include:

  • a start and end for the plausible boundary window;
  • a confidence field tied to evidence quality;
  • an unobservable reason code;
  • independent labels followed by adjudication;
  • a tolerance-aware metric for model evaluation.

The choice depends on consequence. A two-frame disagreement at 30 fps may be irrelevant for video retrieval and important for high-frequency control. Resolution should follow use, not tool capability.

A calibration exercise that pays for itself

Choose 20 to 40 clips covering easy cases, edge cases and expected failures. Have annotators work independently, then compare:

  1. category disagreement;
  2. boundary distance in frames or milliseconds;
  3. missing versus extra events;
  4. uncertainty usage;
  5. disagreement by visibility and failure type.

Review the largest or most systematic disagreements with a domain owner. Update examples and rules, relabel the calibration set, and repeat until the residual disagreement represents acceptable ambiguity rather than inconsistent interpretation.

Do not optimize only for average boundary distance. A team can agree closely while applying the same wrong rule. Include customer acceptance checks tied to the model task.

The deliverable is a temporal contract

A mature temporal ontology is a contract between domain experts, annotators, reviewers and model developers. It states what the labels mean, what the evidence can support and how uncertainty travels into evaluation.

When that contract is explicit, annotators move faster because fewer cases require invention. Reviewers focus on real exceptions. Model teams understand whether a near-boundary error is a learning problem or a labeling convention.

Sources and further reading

Continue reading