Ambiguous instructions break good robot policies. We put a wrist on the loop.

LIBERO benchmark, open π0.5 checkpoint — all results in simulation.

TL;DR

• On an open π0.5 checkpoint, an ambiguous instruction ("pick up the can", "the bottle", …) succeeded in 11 of 20 paired runs. When a memory taught from the wrist resolved the target — asking the person only when unsure — and the instruction named it exactly, 18 of 20. Same scenes, same seeds.
• With a scripted arm controller, a small memory taught from wrist-view images plus a "ask when unsure" step cut wrong-object picks from 4 of 60 runs to 0 of 90.
• Nothing was retrained. The memory is a few kilobytes.

Read this first — what these numbers are not. Everything below ran in simulation (MuJoCo, via robosuite/LIBERO). The simulator told us where each object was in the image; on a real robot, a segmentation model has to do that job, and we have not measured that yet. The "person" answering questions was an oracle that always answers correctly. Real-robot validation is the next step.

The failure mode

Instruction-following policies are now very good when the instruction is exact. The π0.5 LIBERO checkpoint published by Physical Intelligence reports 98% on LIBERO-Object, where every instruction names one grocery item. People don't talk like benchmark prompts. They say "the can", "the bottle", "that one". When two objects fit the description, a policy has to guess — and in our runs it mostly didn't fail loudly. It wandered until the episode timed out.

We wanted to know how much of that failure a person can remove, cheaply, without touching the policy.

What we added

A thin layer beside the policy. It does two things.

1. Remember what you showed it. A person shows each object to a wrist-worn camera: three views from above, at an angle, and from the side, the way you'd turn something over in your hand. Each view becomes a feature vector (84 numbers: color histogram, a few shape terms, a texture histogram). That's the memory — a few KB of JSON.
2. Ask when unsure. At run time, each object the robot sees is matched against the memory. If the best match is weak or too close to the second best, the layer asks the person to confirm on their wrist (👍 / 👎, or say the name). Answers are stored as positive or negative examples, so the same confusion doesn't come back.

The layer never sends motor commands. It decides what; the policy or controller decides how.

Setup

• Benchmark: LIBERO-Object, all 10 tasks ("pick up the {item} and place it in the basket"), Franka Panda with a wrist camera.
• Policies: (a) the open π0.5 LIBERO checkpoint from openpi, served unchanged; (b) a scripted pick-and-place controller, used to isolate the choosing step from the grasping step.
• Ambiguity test: we grouped the ten items by kind — cans, bottles, boxes, cartons. Condition A gives π0.5 the ambiguous prompt ("pick up the can"). Condition B lets the layer recognize the target from its memory — asking the (oracle) person only when unsure — and hands π0.5 the exact prompt. Both conditions run on the same scenes with the same seeds: 10 tasks × 2 episodes each.
• Compute: one A100 for an afternoon, setup included. No training.

Results

1. Ambiguous instructions on π0.5

Most condition-A failures were timeouts: the arm hovered between candidates. In condition B the layer resolved the target without asking in 18 of 20 runs. The other two — both ketchup — are generous to us: the layer asked the person three times in each, still could not match the bottle, fell back to the ambiguous prompt, and π0.5 happened to succeed. Counting those as failures gives 16 / 18 for the runs where the layer actually chose.

2. Choosing the right object (scripted controller, 30 runs per batch)

Across the three confirmation batches: 0 wrong objects in 90 runs (an intermediate variant we discarded for a question-budget bug also had 0 in 30). The remaining failures were grasps, not choices — all three failures in the last batch were the same flat box slipping out of the gripper.

3. How you show matters more than how much you show

The teaching views, the correction examples and the test crops never come from the same layouts. Rows 1–2 are measured on layout set A. The corrections are the robot's first two wrong claims on set A, and rows 3–4 are measured on a separate set B, where the same three-view memory scores 81% before any correction. The objects are the same ten throughout — the point is recognizing things you taught, in placements it hasn't seen.

From above, three different cans are three silver lids. From the side, they have labels. Turning an object in your hand is the whole trick. A bare 👎 helps less than naming the object; we now treat naming as the default correction and 👎 as the quick one.

Why a wrist

The person who can resolve "which can" is usually standing there with their hands busy. A phone means putting things down. A watch camera also sees objects from roughly where a robot's wrist camera does, which is why three views from a wrist were enough for the memory to match what the robot's camera saw. We make that watch (Vibe-Watch), so we are not neutral on this point — which is why the numbers are here in full, with the limits above.

The same teaching code, unchanged, also runs a fetch task on a simulated biped (MicroDuck, from the official open-source simulator): 90% of 30 runs placed the object where the person showed. Not affiliated with Hugging Face or Pollen Robotics.

What's next

• Real objects, real camera. Replace the simulator's object locations with an open segmentation model and the handcrafted features with a pretrained encoder, and measure on photos first.
• Real arm. The same experiments on an SO-101 with SmolVLA in place of π0.5.
• We're looking for a small number of teams to pilot this on LeRobot SO-100/101. Details: inhandplus.com/teach. Method notes and run logs on request — biz@inhandplus.com.

InHandPlus · Korea