Capability maps for robot evaluation

Break down policy performance by position, orientation, clutter, objects and lighting. See which conditions cause failures and where more trials are needed.

Look beyond aggregate success

Imagine two policies with similar overall results. One works across the table but struggles with a particular object. The other handles that object well but fails near the edge of the workspace. A single number can make them look interchangeable.

A capability map organizes outcomes by the condition that produced them. It lets a team ask a practical question: does this policy work in the part of the world we actually care about?

The map is an account of tested conditions. It does not establish capability in every untested scene, and it is not a safety certification.

Five variables to test

Position: What happens when the target moves across the reachable surface? Keep the camera and robot configuration recorded.
Orientation: What happens when the object turns? State which orientation is being varied and what can actually be verified.
Clutter: What happens when another object enters the scene? Record its identity and placement rather than calling everything “cluttered.”
Objects: What happens when the target changes? Size, material, shape and familiarity may change together; a substitution is rarely one perfectly isolated property.
Lighting: What happens when illumination changes? Record the lighting arrangement and relevant camera settings. Projected staging does not automatically control ambient light.

Test individual variables and combinations

Begin with one axis at a time. It makes a failure easier to attribute and gives the team a readable first map. If the cup moves while the camera and light also change, the result cannot isolate the effect of position.

After the first pass, test combinations that matter. Clutter and orientation may interact. A policy that tolerates either change alone may fail when both occur. One-variable tests are a useful beginning, not a complete account of the world.

Colosseum is a simulation research example of evaluating environmental perturbations systematically. Its findings are not Prism measurements; the useful lesson here is to make the axes of variation explicit.

Distinguish untested conditions

Show trial counts alongside outcomes. Mark untested conditions clearly. Preserve invalid episodes and review disagreements separately. If one cell contains many trials and another contains only a few, the visual treatment should not suggest equal certainty.

Keep the task, rubric, policy and condition versions attached to the map. Comparing maps made under different rules can create a persuasive picture of a change that was never actually measured.

Compare model scores task by task in the Physical AI Index.

Choose the next experiment

A useful map changes what the team does. It might reveal a narrow position boundary, motivate a new object test or identify a scene worth replaying against the next checkpoint. Preserve the conditions that led to that decision.

Prism's role begins with making physical placement more explicit through projected guidance and camera checks. Scene history connects those conditions to the run record and the resulting comparison. Read about physical version control or bring us a task to discuss.