ArmnetBench v0.1 Dataset

has-success-labels · ArmnetBench v0.1 is a real-world manipulation *benchmark*: 7 policies run on a fleet of low-cost SO-101 cells, with every episode human-scored. It is released as evaluation data rather than a training corpus, which makes it unusual here — most of the catalog is demonstrations, and this is what happens when policies actually try. **What is in it.** 3,718 episodes across 12 tasks and two hardware configurations: 2,499 episodes on a single SO-101 arm (8 tasks) and 1,219 on a bimanual SO-101 cell (4 tasks). Three cameras per cell at 20 Hz — front/top/wrist for the single arm, top/left-wrist/right-wrist for the bimanual. LeRobot v3.0 format. **Three-way quality labels.** Every episode carries `success_class` ∈ {successful, suboptimal, failure} plus a `policy_type` and `policy_repo_id`. 600 episodes are teleoperated reference demonstrations (successful by construction, 50 per task); the remaining 3,118 are autonomous policy rollouts, human-scored. Scoring is strict: only `successful` counts, so the 106 `suboptimal` episodes are recorded as failures. Note that the suboptimal label is only actually populated on 4 of the 12 tasks, so it is not a benchmark-wide partial-credit signal. Success rates counted from the release, superseded checkpoints excluded (see below): | policy | episodes | success | task-balanced | |---|---|---|---| | teleoperated (reference) | 600 | 100.0% | 100.0% | | π0.5 | 359 | 47.6% | 47.7% | | π0 | 359 | 35.1% | 35.2% | | GR00T N1.7 | 360 | 29.4% | 29.4% | | Diffusion Policy | 360 | 26.7% | 26.7% | | MolmoAct2 | 360 | 18.9% | 18.9% | | ACT | 420 | 16.4% | 18.7% | | SmolVLA | 480 | 13.1% | 14.1% | "Task-balanced" weights each of the 12 tasks equally; the pooled column weights each episode equally, which over-counts the tasks that were run twice. Only ACT and SmolVLA move materially between the two. **Two checkpoint families were excluded, and you should know which.** MolmoAct2 was evaluated twice per task. The first family (`molmoact2_<task>_20k`, 20k steps, MEAN_STD normalization) scored 6.7% over 360 episodes; the second (`molmoact2_<task>_quantile_norm_fix_25k`) scored 18.9%. The table above uses only the second. Two caveats: that run changes two variables at once — quantile normalization *and* 50% more training — so the gain is not attributable to the normalization fix alone; and despite its name its `train_config.json` reports 30,000 steps, not 25,000. ACT similarly has two families, but there they are one lineage rather than two attempts: `act_<task>_20k` declares `pretrained_path: pravsels/act_<task>`, so the un-suffixed repos are the predecessor checkpoint it was warm-started from. Their 60 rollouts (1.7%, 2 tasks only) are excluded rather than averaged in. Pooling both families is what produces the 12.8% and 14.6% figures quoted elsewhere for MolmoAct2 and ACT. **The rollout horizon was not held constant across policies, and this is a real confound.** Maximum episode length varies by up to 3.0× between policies on the *same task*. The sharpest case is GR00T N1.7 on "insert the missing tool": its longest of 30 rollouts is 361 frames (18s at 20 Hz), while the median human demonstration of that task runs 385 frames and the longest runs 690 — 62% of the reference demos would not have fit inside the budget GR00T was given. Its episode lengths cluster at that ceiling (21 of 30 within 5% of the maximum), which is what truncation looks like rather than early failure; π0.5 on the same task spreads from 218 to 667 frames. Averaged across the 12 tasks, the share of human demonstrations longer than the policy's own horizon is 9.5% for GR00T and 5.0% for Diffusion Policy, against 0.8% for π0.5. The two policies with the tightest budgets are penalised most, and the leader was the least constrained. **Read this table as checkpoint scores, not a policy ranking.** Each entry is one specific released checkpoint, trained on 50 demonstrations per task and evaluated on 30 rollouts per task (45–60 on the handful run twice), under a single harness, by three different authors using three different training pipelines. The 50-demo budget is shared and deliberately small. At n=30 per cell the binomial standard error is 6–9pp depending on the rate, so a 95% interval on any single per-task number spans roughly ±13–18pp: per-task figures separate the extremes, not neighbours. None of these figures should be read as what the policy family achieves when tuned. **Tasks.** Single-arm: insert a ring onto a peg, insert/remove a tool from a toolbox, clip and unclip a DisplayPort cable, move eye drops to a basket and to a shelf, stack blocks. Bimanual: fold a tea towel, hold a lamp steady with one arm while opening a door with the other, insert a candle into a lantern and close the door, transfer a cube between arms into a basket. **A note on the numbers.** The paper reports 3,118 core episodes (2,518 policy rollouts + 600 demonstrations). The released repos contain 3,718 episodes — 3,118 rollouts plus the 600 demonstrations. The extra rollouts are largely a second MolmoAct2 evaluation. The figures above are counted from the release, so they are what you will actually download. **Companion repo.** `armnet/armnetbench_v01_robometer` carries the same episodes in RoboMeter format: per-episode quality labels, partial-success scores, and language embeddings, without the trajectories. Useful for training reward or preference models against the labels.

ArmnetBench v0.1 is a real-world manipulation *benchmark*: 7 policies run on a fleet of low-cost SO-101 cells, with every episode human-scored. It is released as evaluation data rather than a training corpus, which makes it unusual here — most of the catalog is demonstrations, and this is what happens when policies actually try. **What is in it.** 3,718 episodes across 12 tasks and two hardware configurations: 2,499 episodes on a single SO-101 arm (8 tasks) and 1,219 on a bimanual SO-101 cell (4 tasks). Three cameras per cell at 20 Hz — front/top/wrist for the single arm, top/left-wrist/right-wrist for the bimanual. LeRobot v3.0 format. **Three-way quality labels.** Every episode carries `success_class` ∈ {successful, suboptimal, failure} plus a `policy_type` and `policy_repo_id`. 600 episodes are teleoperated reference demonstrations (successful by construction, 50 per task); the remaining 3,118 are autonomous policy rollouts, human-scored. Scoring is strict: only `successful` counts, so the 106 `suboptimal` episodes are recorded as failures. Note that the suboptimal label is only actually populated on 4 of the 12 tasks, so it is not a benchmark-wide partial-credit signal. Success rates counted from the release, superseded checkpoints excluded (see below): | policy | episodes | success | task-balanced | |---|---|---|---| | teleoperated (reference) | 600 | 100.0% | 100.0% | | π0.5 | 359 | 47.6% | 47.7% | | π0 | 359 | 35.1% | 35.2% | | GR00T N1.7 | 360 | 29.4% | 29.4% | | Diffusion Policy | 360 | 26.7% | 26.7% | | MolmoAct2 | 360 | 18.9% | 18.9% | | ACT | 420 | 16.4% | 18.7% | | SmolVLA | 480 | 13.1% | 14.1% | "Task-balanced" weights each of the 12 tasks equally; the pooled column weights each episode equally, which over-counts the tasks that were run twice. Only ACT and SmolVLA move materially between the two. **Two checkpoint families were excluded, and you should know which.** MolmoAct2 was evaluated twice per task. The first family (`molmoact2_<task>_20k`, 20k steps, MEAN_STD normalization) scored 6.7% over 360 episodes; the second (`molmoact2_<task>_quantile_norm_fix_25k`) scored 18.9%. The table above uses only the second. Two caveats: that run changes two variables at once — quantile normalization *and* 50% more training — so the gain is not attributable to the normalization fix alone; and despite its name its `train_config.json` reports 30,000 steps, not 25,000. ACT similarly has two families, but there they are one lineage rather than two attempts: `act_<task>_20k` declares `pretrained_path: pravsels/act_<task>`, so the un-suffixed repos are the predecessor checkpoint it was warm-started from. Their 60 rollouts (1.7%, 2 tasks only) are excluded rather than averaged in. Pooling both families is what produces the 12.8% and 14.6% figures quoted elsewhere for MolmoAct2 and ACT. **The rollout horizon was not held constant across policies, and this is a real confound.** Maximum episode length varies by up to 3.0× between policies on the *same task*. The sharpest case is GR00T N1.7 on "insert the missing tool": its longest of 30 rollouts is 361 frames (18s at 20 Hz), while the median human demonstration of that task runs 385 frames and the longest runs 690 — 62% of the reference demos would not have fit inside the budget GR00T was given. Its episode lengths cluster at that ceiling (21 of 30 within 5% of the maximum), which is what truncation looks like rather than early failure; π0.5 on the same task spreads from 218 to 667 frames. Averaged across the 12 tasks, the share of human demonstrations longer than the policy's own horizon is 9.5% for GR00T and 5.0% for Diffusion Policy, against 0.8% for π0.5. The two policies with the tightest budgets are penalised most, and the leader was the least constrained. **Read this table as checkpoint scores, not a policy ranking.** Each entry is one specific released checkpoint, trained on 50 demonstrations per task and evaluated on 30 rollouts per task (45–60 on the handful run twice), under a single harness, by three different authors using three different training pipelines. The 50-demo budget is shared and deliberately small. At n=30 per cell the binomial standard error is 6–9pp depending on the rate, so a 95% interval on any single per-task number spans roughly ±13–18pp: per-task figures separate the extremes, not neighbours. None of these figures should be read as what the policy family achieves when tuned. **Tasks.** Single-arm: insert a ring onto a peg, insert/remove a tool from a toolbox, clip and unclip a DisplayPort cable, move eye drops to a basket and to a shelf, stack blocks. Bimanual: fold a tea towel, hold a lamp steady with one arm while opening a door with the other, insert a candle into a lantern and close the door, transfer a cube between arms into a basket. **A note on the numbers.** The paper reports 3,118 core episodes (2,518 policy rollouts + 600 demonstrations). The released repos contain 3,718 episodes — 3,118 rollouts plus the 600 demonstrations. The extra rollouts are largely a second MolmoAct2 evaluation. The figures above are counted from the release, so they are what you will actually download. **Companion repo.** `armnet/armnetbench_v01_robometer` carries the same episodes in RoboMeter format: per-episode quality labels, partial-success scores, and language embeddings, without the trajectories. Useful for training reward or preference models against the labels.

ArmnetBench v0.1 is a real-world manipulation *benchmark*: 7 policies run on a fleet of low-cost SO-101 cells, with every episode human-scored. It is released as evaluation data rather than a training corpus, which makes it unusual here — most of the catalog is demonstrations, and this is what happens when policies actually try. **What is in it.** 3,718 episodes across 12 tasks and two hardware configurations: 2,499 episodes on a single SO-101 arm (8 tasks) and 1,219 on a bimanual SO-101 cell (4 tasks). Three cameras per cell at 20 Hz — front/top/wrist for the single arm, top/left-wrist/right-wrist for the bimanual. LeRobot v3.0 format. **Three-way quality labels.** Every episode carries `success_class` ∈ {successful, suboptimal, failure} plus a `policy_type` and `policy_repo_id`. 600 episodes are teleoperated reference demonstrations (successful by construction, 50 per task); the remaining 3,118 are autonomous policy rollouts, human-scored. Scoring is strict: only `successful` counts, so the 106 `suboptimal` episodes are recorded as failures. Note that the suboptimal label is only actually populated on 4 of the 12 tasks, so it is not a benchmark-wide partial-credit signal. Success rates counted from the release, superseded checkpoints excluded (see below): | policy | episodes | success | task-balanced | |---|---|---|---| | teleoperated (reference) | 600 | 100.0% | 100.0% | | π0.5 | 359 | 47.6% | 47.7% | | π0 | 359 | 35.1% | 35.2% | | GR00T N1.7 | 360 | 29.4% | 29.4% | | Diffusion Policy | 360 | 26.7% | 26.7% | | MolmoAct2 | 360 | 18.9% | 18.9% | | ACT | 420 | 16.4% | 18.7% | | SmolVLA | 480 | 13.1% | 14.1% | "Task-balanced" weights each of the 12 tasks equally; the pooled column weights each episode equally, which over-counts the tasks that were run twice. Only ACT and SmolVLA move materially between the two. **Two checkpoint families were excluded, and you should know which.** MolmoAct2 was evaluated twice per task. The first family (`molmoact2_<task>_20k`, 20k steps, MEAN_STD normalization) scored 6.7% over 360 episodes; the second (`molmoact2_<task>_quantile_norm_fix_25k`) scored 18.9%. The table above uses only the second. Two caveats: that run changes two variables at once — quantile normalization *and* 50% more training — so the gain is not attributable to the normalization fix alone; and despite its name its `train_config.json` reports 30,000 steps, not 25,000. ACT similarly has two families, but there they are one lineage rather than two attempts: `act_<task>_20k` declares `pretrained_path: pravsels/act_<task>`, so the un-suffixed repos are the predecessor checkpoint it was warm-started from. Their 60 rollouts (1.7%, 2 tasks only) are excluded rather than averaged in. Pooling both families is what produces the 12.8% and 14.6% figures quoted elsewhere for MolmoAct2 and ACT. **The rollout horizon was not held constant across policies, and this is a real confound.** Maximum episode length varies by up to 3.0× between policies on the *same task*. The sharpest case is GR00T N1.7 on "insert the missing tool": its longest of 30 rollouts is 361 frames (18s at 20 Hz), while the median human demonstration of that task runs 385 frames and the longest runs 690 — 62% of the reference demos would not have fit inside the budget GR00T was given. Its episode lengths cluster at that ceiling (21 of 30 within 5% of the maximum), which is what truncation looks like rather than early failure; π0.5 on the same task spreads from 218 to 667 frames. Averaged across the 12 tasks, the share of human demonstrations longer than the policy's own horizon is 9.5% for GR00T and 5.0% for Diffusion Policy, against 0.8% for π0.5. The two policies with the tightest budgets are penalised most, and the leader was the least constrained. **Read this table as checkpoint scores, not a policy ranking.** Each entry is one specific released checkpoint, trained on 50 demonstrations per task and evaluated on 30 rollouts per task (45–60 on the handful run twice), under a single harness, by three different authors using three different training pipelines. The 50-demo budget is shared and deliberately small. At n=30 per cell the binomial standard error is 6–9pp depending on the rate, so a 95% interval on any single per-task number spans roughly ±13–18pp: per-task figures separate the extremes, not neighbours. None of these figures should be read as what the policy family achieves when tuned. **Tasks.** Single-arm: insert a ring onto a peg, insert/remove a tool from a toolbox, clip and unclip a DisplayPort cable, move eye drops to a basket and to a shelf, stack blocks. Bimanual: fold a tea towel, hold a lamp steady with one arm while opening a door with the other, insert a candle into a lantern and close the door, transfer a cube between arms into a basket. **A note on the numbers.** The paper reports 3,118 core episodes (2,518 policy rollouts + 600 demonstrations). The released repos contain 3,718 episodes — 3,118 rollouts plus the 600 demonstrations. The extra rollouts are largely a second MolmoAct2 evaluation. The figures above are counted from the release, so they are what you will actually download. **Companion repo.** `armnet/armnetbench_v01_robometer` carries the same episodes in RoboMeter format: per-episode quality labels, partial-success scores, and language embeddings, without the trajectories. Useful for training reward or preference models against the labels.

Open · Apache-2.0 · lerobot