Why robot evaluation starts with the reset

Consistent starting conditions make policy comparisons easier to interpret. Record the intended setup, check the scene and track changes between trials.

How setup changes affect results

A mug returns to the table, almost where it was. The handle turns a little. A cable lies closer to the gripper. Nothing about the model changed, but its next observation did.

That does not make variation a mistake. A capable robot must operate through variation. The problem is variation that enters the evaluation without being chosen, measured or described.

When a team compares two policies, it needs to understand whether the policies faced comparable conditions. Otherwise a changed result may reflect a changed scene, a changed policy or both.

Define, observe and verify the setup

Intended condition: The scene the protocol asks for: objects, target positions and other declared conditions.
Observed condition: What the camera or operator can actually establish about the prepared scene.
Acceptance decision: Whether the observed scene is suitable for this trial under the task's tolerances, including any unresolved uncertainty.

Collapsing these into one “ready” flag makes the interface simple at the expense of the experiment. A useful system lets a reviewer recover the reason the trial was accepted.

Use projected setup guides

Prism projects placement guidance onto the work surface so an operator can prepare a target condition without referring back and forth to a separate drawing. Camera registration connects the projected instruction to the observed workspace.

The Prism technical brief describes this as an initial-condition layer: stage, verify, log and re-project a saved condition. It also identifies limits. Planar placement is easier to express than a complete three-dimensional configuration; deformable objects and lighting need additional treatment.

Visible projected marks may themselves affect a vision policy. The recording protocol must specify when guidance is shown and when it is removed. “No tape on the table” is helpful only if the replacement does not introduce an unexamined cue.

Repeat the reset consistently

Start by naming the objects and the target arrangement. Check calibration against the actual workspace. Stage the scene, inspect the observation and record the acceptance decision. Only then begin the policy episode.

When a reset cannot meet the requirement, preserve that fact. Do not quietly substitute a nearby scene while retaining the original condition label. When the table, camera or projector moves, re-check the relationship before using old targets.

This is not a claim that every source of variability can be eliminated. It is a way to make the setup part of the experiment instead of an invisible assumption.

Track each variable

Once a baseline is understood, move beyond it deliberately. Change one axis and record it. Then test combinations where interactions matter. The resulting capability map can show where a policy's performance changes across the conditions tested.

For background on richer physical evaluation reporting, see Robot Learning as an Empirical Science: Best Practices for Policy Evaluation. For the product workflow, see how Prism works.