LIBERO-Plus

LIBERO-Plus takes the position that headline success rates on [LIBERO](/datasets/libero) hide how brittle vision-language-action policies actually are, and it tests that by perturbing the benchmark rather than extending it. The tasks are the same; what changes is everything around them. **Seven perturbation factors**, decomposed into 21 low-level components: object layout, camera viewpoints, robot initial states, language instructions, lighting conditions, background textures, and sensor noise. Tasks are stratified into five difficulty levels (L1–L5) based on how a panel of models performs, and the mapping from task ID to perturbation category and level ships with the code. **What the perturbations expose.** The authors report success rates falling from about 95% to below 30% under modest changes, with camera viewpoint and robot initial state the most damaging axes. The finding that has attracted the most attention is the opposite one: models are largely *insensitive* to language perturbation, which the paper attributes to policies substantially ignoring the instruction rather than being robust to how it is phrased. Combined perturbations interact rather than compose independently. **What ships.** Two distinct artefacts, which is worth keeping straight: - **10,030 evaluation tasks** — environments and task definitions, not demonstrations. This is the number quoted as the benchmark's size, and it is what `num_tasks` records here. Evaluation runs one trial per task rather than LIBERO's 50. - **A training set of 14,347 episodes / 2,238,036 frames** (31.1 h at 20 Hz) spanning 40 task instructions, released in both RLDS (70.4 GiB) and LeRobot v2.1 form. Front and wrist RGB at 256×256, Franka Panda, 20 Hz. Note the paper describes the fine-tuning corpus as "over 20,000 successful trajectories" while the released LeRobot and RLDS sets contain 14,347 episodes; the released figure is the one recorded here. **Practical note.** The evaluation harness is a drop-in replacement — the authors state you can `pip install -e .` over your existing `libero` package without changing evaluation code, adjusting only `num_trials_per_task` from 50 to 1.

language · LIBERO-Plus takes the position that headline success rates on [LIBERO](/datasets/libero) hide how brittle vision-language-action policies actually are, and it tests that by perturbing the benchmark rather than extending it. The tasks are the same; what changes is everything around them. **Seven perturbation factors**, decomposed into 21 low-level components: object layout, camera viewpoints, robot initial states, language instructions, lighting conditions, background textures, and sensor noise. Tasks are stratified into five difficulty levels (L1–L5) based on how a panel of models performs, and the mapping from task ID to perturbation category and level ships with the code. **What the perturbations expose.** The authors report success rates falling from about 95% to below 30% under modest changes, with camera viewpoint and robot initial state the most damaging axes. The finding that has attracted the most attention is the opposite one: models are largely *insensitive* to language perturbation, which the paper attributes to policies substantially ignoring the instruction rather than being robust to how it is phrased. Combined perturbations interact rather than compose independently. **What ships.** Two distinct artefacts, which is worth keeping straight: - **10,030 evaluation tasks** — environments and task definitions, not demonstrations. This is the number quoted as the benchmark's size, and it is what `num_tasks` records here. Evaluation runs one trial per task rather than LIBERO's 50. - **A training set of 14,347 episodes / 2,238,036 frames** (31.1 h at 20 Hz) spanning 40 task instructions, released in both RLDS (70.4 GiB) and LeRobot v2.1 form. Front and wrist RGB at 256×256, Franka Panda, 20 Hz. Note the paper describes the fine-tuning corpus as "over 20,000 successful trajectories" while the released LeRobot and RLDS sets contain 14,347 episodes; the released figure is the one recorded here. **Practical note.** The evaluation harness is a drop-in replacement — the authors state you can `pip install -e .` over your existing `libero` package without changing evaluation code, adjusting only `num_trials_per_task` from 50 to 1.

LIBERO-Plus takes the position that headline success rates on [LIBERO](/datasets/libero) hide how brittle vision-language-action policies actually are, and it tests that by perturbing the benchmark rather than extending it. The tasks are the same; what changes is everything around them. **Seven perturbation factors**, decomposed into 21 low-level components: object layout, camera viewpoints, robot initial states, language instructions, lighting conditions, background textures, and sensor noise. Tasks are stratified into five difficulty levels (L1–L5) based on how a panel of models performs, and the mapping from task ID to perturbation category and level ships with the code. **What the perturbations expose.** The authors report success rates falling from about 95% to below 30% under modest changes, with camera viewpoint and robot initial state the most damaging axes. The finding that has attracted the most attention is the opposite one: models are largely *insensitive* to language perturbation, which the paper attributes to policies substantially ignoring the instruction rather than being robust to how it is phrased. Combined perturbations interact rather than compose independently. **What ships.** Two distinct artefacts, which is worth keeping straight: - **10,030 evaluation tasks** — environments and task definitions, not demonstrations. This is the number quoted as the benchmark's size, and it is what `num_tasks` records here. Evaluation runs one trial per task rather than LIBERO's 50. - **A training set of 14,347 episodes / 2,238,036 frames** (31.1 h at 20 Hz) spanning 40 task instructions, released in both RLDS (70.4 GiB) and LeRobot v2.1 form. Front and wrist RGB at 256×256, Franka Panda, 20 Hz. Note the paper describes the fine-tuning corpus as "over 20,000 successful trajectories" while the released LeRobot and RLDS sets contain 14,347 episodes; the released figure is the one recorded here. **Practical note.** The evaluation harness is a drop-in replacement — the authors state you can `pip install -e .` over your existing `libero` package without changing evaluation code, adjusting only `num_trials_per_task` from 50 to 1.

simulation

Open · MIT · lerobot · rlds