EgoInfinity

depth · video · **EgoInfinity** is not a static dataset but a **modular, upgradable data engine**: it takes ordinary internet video of people manipulating objects and automatically recovers metric 4D hand–object interaction, then retargets that motion into executable robot joint trajectories — no manual annotation and no depth sensors. - Runs over Meta FAIR's **Action100M** corpus — **14.6 years (~127,000 h)** of footage across **~142M action clips**. The publicly released artifact is a curated **preview subset of 1,150 rows / 104 fully retargeted clips (5.49 GB)**. - **Eight-stage pipeline**: metric depth + focal estimation (MoGe-2), gravity calibration (GeoCalib), hand detection and MANO reconstruction (YOLO + WiLoR), optical flow and depth stabilisation (MEMFOF), motion infilling (HaWoR), text-prompted detection and tracking (SAM 3.1 + SAM 2), single-image 3D object reconstruction, and 6-DoF mesh pose tracking. - Each processed clip yields per-frame **21-joint hand poses**, MANO meshes, **per-object 6-DoF pose trajectories**, Gaussian-splat object meshes, optical flow, metric depth, and segmentation masks. - **Retargets to four embodiments**: Franka Panda (7-DoF + gripper), Unitree G1 (7-DoF arm + 7-DoF hand), NASA Robonaut 2 (7-DoF arm + 18-DoF hand), and XLeRobot (5-DoF + gripper), with per-frame IK-success and manipulability metrics stored alongside. - Supports **arbitrary camera viewpoints and shot sizes**, including partial body visibility — unlike prior pipelines that assume a fixed egocentric rig. - Validated on a real robot for **grasping, cutting, wiping, and pouring** learned directly from internet video. - Because it is an engine rather than a fixed corpus, data quality improves automatically as its component models are swapped for better ones. Licensing is **non-commercial research only** (FAIR Noncommercial Research License v1 for data; the code carries mixed MIT / CC-BY-NC-ND / AGPL components).

ee_pose · **EgoInfinity** is not a static dataset but a **modular, upgradable data engine**: it takes ordinary internet video of people manipulating objects and automatically recovers metric 4D hand–object interaction, then retargets that motion into executable robot joint trajectories — no manual annotation and no depth sensors. - Runs over Meta FAIR's **Action100M** corpus — **14.6 years (~127,000 h)** of footage across **~142M action clips**. The publicly released artifact is a curated **preview subset of 1,150 rows / 104 fully retargeted clips (5.49 GB)**. - **Eight-stage pipeline**: metric depth + focal estimation (MoGe-2), gravity calibration (GeoCalib), hand detection and MANO reconstruction (YOLO + WiLoR), optical flow and depth stabilisation (MEMFOF), motion infilling (HaWoR), text-prompted detection and tracking (SAM 3.1 + SAM 2), single-image 3D object reconstruction, and 6-DoF mesh pose tracking. - Each processed clip yields per-frame **21-joint hand poses**, MANO meshes, **per-object 6-DoF pose trajectories**, Gaussian-splat object meshes, optical flow, metric depth, and segmentation masks. - **Retargets to four embodiments**: Franka Panda (7-DoF + gripper), Unitree G1 (7-DoF arm + 7-DoF hand), NASA Robonaut 2 (7-DoF arm + 18-DoF hand), and XLeRobot (5-DoF + gripper), with per-frame IK-success and manipulability metrics stored alongside. - Supports **arbitrary camera viewpoints and shot sizes**, including partial body visibility — unlike prior pipelines that assume a fixed egocentric rig. - Validated on a real robot for **grasping, cutting, wiping, and pouring** learned directly from internet video. - Because it is an engine rather than a fixed corpus, data quality improves automatically as its component models are swapped for better ones. Licensing is **non-commercial research only** (FAIR Noncommercial Research License v1 for data; the code carries mixed MIT / CC-BY-NC-ND / AGPL components).

language · **EgoInfinity** is not a static dataset but a **modular, upgradable data engine**: it takes ordinary internet video of people manipulating objects and automatically recovers metric 4D hand–object interaction, then retargets that motion into executable robot joint trajectories — no manual annotation and no depth sensors. - Runs over Meta FAIR's **Action100M** corpus — **14.6 years (~127,000 h)** of footage across **~142M action clips**. The publicly released artifact is a curated **preview subset of 1,150 rows / 104 fully retargeted clips (5.49 GB)**. - **Eight-stage pipeline**: metric depth + focal estimation (MoGe-2), gravity calibration (GeoCalib), hand detection and MANO reconstruction (YOLO + WiLoR), optical flow and depth stabilisation (MEMFOF), motion infilling (HaWoR), text-prompted detection and tracking (SAM 3.1 + SAM 2), single-image 3D object reconstruction, and 6-DoF mesh pose tracking. - Each processed clip yields per-frame **21-joint hand poses**, MANO meshes, **per-object 6-DoF pose trajectories**, Gaussian-splat object meshes, optical flow, metric depth, and segmentation masks. - **Retargets to four embodiments**: Franka Panda (7-DoF + gripper), Unitree G1 (7-DoF arm + 7-DoF hand), NASA Robonaut 2 (7-DoF arm + 18-DoF hand), and XLeRobot (5-DoF + gripper), with per-frame IK-success and manipulability metrics stored alongside. - Supports **arbitrary camera viewpoints and shot sizes**, including partial body visibility — unlike prior pipelines that assume a fixed egocentric rig. - Validated on a real robot for **grasping, cutting, wiping, and pouring** learned directly from internet video. - Because it is an engine rather than a fixed corpus, data quality improves automatically as its component models are swapped for better ones. Licensing is **non-commercial research only** (FAIR Noncommercial Research License v1 for data; the code carries mixed MIT / CC-BY-NC-ND / AGPL components).

**EgoInfinity** is not a static dataset but a **modular, upgradable data engine**: it takes ordinary internet video of people manipulating objects and automatically recovers metric 4D hand–object interaction, then retargets that motion into executable robot joint trajectories — no manual annotation and no depth sensors. - Runs over Meta FAIR's **Action100M** corpus — **14.6 years (~127,000 h)** of footage across **~142M action clips**. The publicly released artifact is a curated **preview subset of 1,150 rows / 104 fully retargeted clips (5.49 GB)**. - **Eight-stage pipeline**: metric depth + focal estimation (MoGe-2), gravity calibration (GeoCalib), hand detection and MANO reconstruction (YOLO + WiLoR), optical flow and depth stabilisation (MEMFOF), motion infilling (HaWoR), text-prompted detection and tracking (SAM 3.1 + SAM 2), single-image 3D object reconstruction, and 6-DoF mesh pose tracking. - Each processed clip yields per-frame **21-joint hand poses**, MANO meshes, **per-object 6-DoF pose trajectories**, Gaussian-splat object meshes, optical flow, metric depth, and segmentation masks. - **Retargets to four embodiments**: Franka Panda (7-DoF + gripper), Unitree G1 (7-DoF arm + 7-DoF hand), NASA Robonaut 2 (7-DoF arm + 18-DoF hand), and XLeRobot (5-DoF + gripper), with per-frame IK-success and manipulability metrics stored alongside. - Supports **arbitrary camera viewpoints and shot sizes**, including partial body visibility — unlike prior pipelines that assume a fixed egocentric rig. - Validated on a real robot for **grasping, cutting, wiping, and pouring** learned directly from internet video. - Because it is an engine rather than a fixed corpus, data quality improves automatically as its component models are swapped for better ones. Licensing is **non-commercial research only** (FAIR Noncommercial Research License v1 for data; the code carries mixed MIT / CC-BY-NC-ND / AGPL components).

depth · **EgoInfinity** is not a static dataset but a **modular, upgradable data engine**: it takes ordinary internet video of people manipulating objects and automatically recovers metric 4D hand–object interaction, then retargets that motion into executable robot joint trajectories — no manual annotation and no depth sensors. - Runs over Meta FAIR's **Action100M** corpus — **14.6 years (~127,000 h)** of footage across **~142M action clips**. The publicly released artifact is a curated **preview subset of 1,150 rows / 104 fully retargeted clips (5.49 GB)**. - **Eight-stage pipeline**: metric depth + focal estimation (MoGe-2), gravity calibration (GeoCalib), hand detection and MANO reconstruction (YOLO + WiLoR), optical flow and depth stabilisation (MEMFOF), motion infilling (HaWoR), text-prompted detection and tracking (SAM 3.1 + SAM 2), single-image 3D object reconstruction, and 6-DoF mesh pose tracking. - Each processed clip yields per-frame **21-joint hand poses**, MANO meshes, **per-object 6-DoF pose trajectories**, Gaussian-splat object meshes, optical flow, metric depth, and segmentation masks. - **Retargets to four embodiments**: Franka Panda (7-DoF + gripper), Unitree G1 (7-DoF arm + 7-DoF hand), NASA Robonaut 2 (7-DoF arm + 18-DoF hand), and XLeRobot (5-DoF + gripper), with per-frame IK-success and manipulability metrics stored alongside. - Supports **arbitrary camera viewpoints and shot sizes**, including partial body visibility — unlike prior pipelines that assume a fixed egocentric rig. - Validated on a real robot for **grasping, cutting, wiping, and pouring** learned directly from internet video. - Because it is an engine rather than a fixed corpus, data quality improves automatically as its component models are swapped for better ones. Licensing is **non-commercial research only** (FAIR Noncommercial Research License v1 for data; the code carries mixed MIT / CC-BY-NC-ND / AGPL components).

Open · custom · custom