Evaluate every robot policy to improve the data loop.
Task success, sim-to-real gap, intervention rate, and subtask failures.
- 5
- policy types
- 10
- embodiments
- 5
- simulators
from openbot import Client
ob = Client() # reads OPENBOT_API_KEY
# 4-step kitchen handover on a Franka Panda
run = ob.bench.rollout(
policy="openvla-7b",
embodiment="franka_panda",
task="open_drawer → pick_mug → pour → handover",
rollouts=200,
seeds=10,
sim="isaac_sim",
real_hw=True,
edge_target="jetson_orin",
)
result = run.wait() # poll until the run finishes
print(result.task_success) # e.g. 0.73
print(result.subtask["handover"]) # e.g. 0.60 ← bottleneck
print(result.sim_to_real_gap) # e.g. -0.29
print(result.intervention_rate) # e.g. 0.14The metrics behind the next data decision.
Turn failures into collection, curation, and replay tasks.
- 01
Long-horizon task success
Per-subtask survival and recovery rate. Know which step cracks first.
- 02
Sim-to-real gap, quantified
Paired sim and real rollouts, measured per subtask.
- 03
Across seeds, across embodiments
Variance and worst case, not one lucky run.
- 04
Edge deploy metrics
Latency, FPS, memory on Jetson Orin / Thor. Know if it runs before you ship.
- 05
Acceptance reports
PDF / Markdown scorecards with failure videos clustered by root cause.
- 06
Webhooks & integrations
Send results to Slack, Linear, or CI gates.
Failure signals for the next dataset.
Metrics that tell you what to collect, replay, or synthesize next.
A report built for sign-off.
Task success, failure point, and next action in one view.
kitchen_handover · open_drawer → pick_mug → pour → handover
Subtask success
200 rollouts, real Franka- open_drawer98%
- pick_mug91%
- pour82%
- handover60%
Success across 10 seeds
73% ± 5.2
Ship for drawer + pick + pour. Send failed handovers back to Data, then re-run Bench.
Bring your policy, your robot, your stack.
Bring a checkpoint, get a verdict.
Read the spec, run the examples, or request managed rollouts.