How to plan a real-world robot evaluation

Are you choosing between two checkpoints, testing whether a new object breaks a skill, or deciding whether a workflow is ready for a specific setting? These are different decisions. They deserve different experiments.

Write the decision in one sentence. Then describe the evidence that would change your mind. Use that decision to set the test scope and required evidence.

A practical example: “We want to know whether the new checkpoint improves placing this mug in this tray across the positions used at this workcell.” That question names the policy change, task, objects and scope. It leaves room to report what was not tested.

Define success in observable terms. Specify the time limit, allowed contact, acceptable final state and any permitted assistance. Decide how to handle interruptions, sensor loss, a bad reset or a stopped robot.

Keep three outcomes separate: a valid success, a valid failure and an invalid or unresolved episode. A reset mistake should not silently count against the policy, and removing a difficult valid failure would make the report misleading.

When a score is disputed, keep the disagreement and resolve it against the rubric. Prism's scoring model judges every trial against the same rubric and records its reason, so any score can be checked against the recording.

List the conditions that matter: object identity, position, orientation, clutter, lighting, camera arrangement, robot start state and relevant surface properties. Choose which to hold fixed and which to vary deliberately.

Begin with a baseline that can be described and restored. Then choose a schedule of conditions that answers the question. Repeating an easy scene many times does not establish performance across a wider set of scenes.

Give every intended condition an identifier. Record what was actually staged and whether it met the acceptance rule. A projected target is useful, but the target is not proof that the object arrived there.

At minimum, connect each episode to the task and rubric versions, policy or checkpoint identity, scene condition, robot configuration, outcome and any intervention. Where recordings are collected, preserve their relationship to the episode and handle access according to the team's data agreement.

Here is a downloadable evaluation checklist.

• What was the intended setup, and how was it checked?
• Which policy ran, with which robot and sensor configuration?
• Was projected guidance visible during the recording?
• What happened, and what evidence supports the score?
• Did anyone intervene, discard a trial or change a rule?
• Which conditions remain untested?

Give the number of trials and successes, the conditions covered, the uncertainty appropriate to the experiment and the reasons for exclusions. Explain the important failure modes in words. A single average can hide the condition that matters most to the team using the robot.

Compare policies on a matched set of conditions where possible. Consider trial order and wear: running one candidate first all day and another after the objects or hardware have changed can introduce another difference.

The Robot Learning as an Empirical Science: Best Practices for Policy Evaluation paper calls for more informative reporting than a bare success rate. This guide applies that principle to a workcell planning conversation.

Next: why initial conditions belong in the result.