Physical AI Needs More Than Frames: Designing Multimodal Ground Truth

A correct box can describe the wrong behavior

In a robotics episode, the camera may show a gripper neatly surrounding an object. The box is accurate. Yet the grasp may have almost no contact force, the commanded closure may not match the measured joint position, and the object may begin slipping before a human can see it.

That is the central ground-truth problem for Physical AI: pixels describe appearance, while behavior emerges from appearance, embodiment, action, timing and outcome together.

Large cross-embodiment efforts such as Open X-Embodiment demonstrate the value of representing robot experience as episodes rather than independent images. DROID pushes collection into diverse real-world environments. These datasets are useful not simply because they contain many frames, but because frames remain attached to actions and robot state.

Begin with an episode contract

Before choosing labels, define what makes an episode interpretable. A useful contract normally records:

• task or language goal;
• embodiment and end-effector configuration;
• synchronized visual streams and their calibration;
• commanded actions and measured robot state;
• relevant force, torque, tactile or contact signals;
• environment and object state needed to judge the task;
• outcome, failure and intervention information;
• known gaps, dropped samples and clock uncertainty.

This is not a demand that every project use every sensor. It is a demand that the dataset make absence explicit. A missing force signal is different from a measured zero. An unknown calibration is different from identity calibration.

Synchronization is part of the label

Suppose video is recorded at 30 frames per second, joint state at 100 hertz and force at 500 hertz. “Nearest timestamp” may be adequate for coarse behavior recognition and misleading for contact transitions. Clock offset, buffering and sensor exposure time can shift the apparent order of cause and effect.

The dataset should document:

1. clock source for every stream;
2. timestamp meaning—capture start, exposure midpoint, receipt or write time;
3. measured or estimated offsets;
4. interpolation and resampling policy;
5. maximum accepted gap;
6. whether annotators view raw or synchronized playback.

When an annotator marks first contact from video, retain the frame timestamp and the mapping used to retrieve force or proprioceptive context. The annotation is then reproducible when synchronization improves.

Separate observation, action and interpretation

A robust schema avoids collapsing three levels of information.

Observation

What the sensors directly support: contact visible, gripper aperture, object pose, measured wrench, person enters workspace.

Action

What the robot or operator commanded: end-effector delta, joint target, gripper close, policy mode, teleoperation input.

Interpretation

What a domain reviewer concludes: grasp secured, unsafe approach, recoverable slip, task success, wrong strategy.

This separation protects the dataset from circular reasoning. “Grasp secured” should not be inferred only because the episode later succeeds unless the schema explicitly allows retrospective outcome labels. A policy intended for online recovery cannot learn from information that is unavailable at inference time.

Outcomes require counterfactual discipline

Outcome labels look simple—success or failure—but they often hide task assumptions. Did the robot succeed if it placed the object in the correct bin but released it in an unstable pose? If a human corrected the object after placement, is the episode a partial success or an intervention?

Write acceptance predicates that can be checked:

• final object region and pose tolerance;
• required dwell time;
• prohibited contacts;
• whether external intervention occurred;
• allowed intermediate errors;
• timeout and abort rules.

Record task outcome separately from trajectory quality. A lucky success can include unsafe or inefficient behavior; a failure can contain a useful recovery attempt. Both matter to training and evaluation.

Model the embodiment, not just the task name

“Pick up mug” does not describe the same control problem for a parallel gripper, a dexterous hand and a suction cup. Cross-embodiment learning benefits from shared semantics, but only if embodiment-specific constraints remain available.

A practical representation combines:

• a shared task and object vocabulary;
• a canonical action description where defensible;
• original embodiment-native actions;
• kinematic and calibration metadata;
• explicit mappings, with limits, between the two.

Avoid inventing a universal action space that silently discards control modes, coordinate frames or gripper semantics. Retain the original signal even when producing a normalized view.

Quality control must cross modalities

Reviewing each stream independently misses contradictions. Multimodal checks should include:

• visible motion without corresponding state change;
• commanded movement with no measured response;
• force spikes outside any plausible contact window;
• outcome labels inconsistent with final state;
• language goals that do not match the demonstrated task;
• repeated timestamps, missing frames or calibration discontinuities.

Some checks can be automated; their failures still need human interpretation. A force spike may be a collision, sensor saturation or a clock shift. The reviewer should classify the cause rather than merely reject the episode.

Design the smallest sufficient truth

Multimodal does not mean maximal. Every field creates collection, annotation and maintenance cost. Start from a model decision:

• For contact prediction, prioritize synchronized force and visual contact events.
• For language-conditioned policy evaluation, preserve instruction, action, outcome and failure semantics.
• For trajectory imitation, retain high-quality action and state streams with embodiment metadata.
• For safety review, annotate people, zones, interventions and prohibited contacts across time.

Then run ablations. If a modality or derived label does not improve diagnosis or model behavior, it may not belong in production.

Sources and further reading

• Open X-Embodiment Collaboration, “Open X-Embodiment: Robotic Learning Datasets and RT-X Models”, 2023.
• Khazatsky et al., “DROID: A Large-Scale In-the-Wild Robot Manipulation Dataset”, 2024.
• The RLDS episode format, used to represent sequential decision-making datasets.