Understand activity
Temporal labels, narrations, and a task-matched evaluation split.
Research topic · Ego data
Ego data for embodied learning.
The wearer's view keeps intent, hands, objects, motion, and language close to action. It supplies human-scale supervision for embodied models, but not robot dynamics or executable controls.
76 reviewed records4 current 2026 signals6 representative devices6 2026 transfer studies
Start with your task
Choose the supervision you need, then check coverage, access, and rights. These are editorial starting points, not a ranking or a guarantee of training suitability.
Model hand–object interaction
Hand and object geometry, calibration, and interaction annotations.
Learn across viewpoints
Paired ego–exo observations and temporal alignment.
Transfer to a robot
Human demonstrations plus a method-specific embodiment bridge and robot-side evaluation.
Compare starting datasets
Use and signal summaries checked against linked sources on . Access and license below come from the index and require separate confirmation.
Ego4D
Dataset recordActivity understanding, episodic memory, and forecasting.
Video · Narrations · Hand–object annotations · Audio / gaze / 3D scans in subsets
Signal source ↗Extra modalities and annotations vary by subset; check the benchmark split.
Access: Gated · License: Ego4D License Agreement
Ego-Exo4D
Dataset recordCross-view alignment and skilled activity understanding.
Synchronized ego–exo video · Audio · Gaze · 3D
Signal source ↗Paired views support perception research; they are not robot control labels.
Access: Gated · License: Ego-Exo4D License Agreement
EgoDex
Dataset recordHuman hand trajectories still need a robot-specific mapping and validation.
Access: Open · License: CC-BY-NC-ND-4.0
HOI4D
Dataset recordGeometry-aware hand–object interaction.
RGB-D · Hand pose · Object pose · Segmentation
Signal source ↗Check object categories and annotation coverage against your target task.
Access: Unknown · License: MIT
See the supervision
Frames show how wearer view, synchronized viewpoints, and hand pose expose different supervision.
ObserveLong-form activity in the wearer’s view
AlignFirst- and third-person skill in sync
ActHand pose for dexterous manipulationLinked dataset records remain the authority for exact rigs, calibration, sensor channels, access, and rights.
Human-reviewed index
Source datasets
One canonical record per human-worn first-person or synchronized ego–exo source corpus. Search and filter the source-reviewed records; access status is not a usage license.
What has been reviewed?
Corpus inclusion ledger: . We check that a record identifies a human-worn first-person or synchronized ego–exo source corpus and exclude aliases and downstream repackagings.
Inclusion review does not verify every field. Access and license are reported separately; unknown means not verified, not unavailable. Open access does not grant unrestricted use.
Data-focus filters include only records with source-checked selection profiles. Other records remain searchable in All data. A missing classification is not evidence that a signal is absent. Extra modalities may exist only in subsets.
Each record links to its source for current terms and field details. Signal-source reviews have their own date; they do not refresh the corpus inclusion date.
Data focus uses source-checked profiles for 4 of 76 records; it does not search descriptions. Select All data to include unclassified records.
8 shown · 76 reviewed
- ADTAccess not verified
Publisher not recorded · 2023
Aria Digital Twin: A New Benchmark Dataset for Egocentric 3D Machine Perception
- Catalog-reported signals
- Not verified
- Reported license
- Not verified — check source terms
Inclusion ledger: 2026-08-31 · Signals not independently checked
- AEAAccess not verified
Publisher not recorded · 2024
Aria Everyday Activities Dataset
- Catalog-reported signals
- Not verified
- Reported license
- Not verified — check source terms
Inclusion ledger: 2026-08-31 · Signals not independently checked
- Assembly101Access not verified
assembly-101 · 2022
Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activities
- Catalog-reported signals
- Not verified
- Reported license
- Not verified — check source terms
Inclusion ledger: 2026-08-31 · Signals not independently checked
- CaptainCook4DAccess not verified
CaptainCook4D · 2023
CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities
- Catalog-reported signals
- Not verified
- Reported license
- Not verified — check source terms
Inclusion ledger: 2026-08-31 · Signals not independently checked
- Charades-EgoAccess not verified
gsig · 2018
Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos
- Catalog-reported signals
- Not verified
- Reported license
- Not verified — check source terms
Inclusion ledger: 2026-08-31 · Signals not independently checked
- ChildLensAccess not verified
Publisher not recorded · 2026
ChildLens: An egocentric video dataset for activity analysis in children
- Catalog-reported signals
- Not verified
- Reported license
- Not verified — check source terms
Inclusion ledger: 2026-08-31 · Signals not independently checked
- CMU-MMACAccess not verified
Publisher not recorded · 2008
Guide to the Carnegie Mellon University Multimodal Activity (CMU-MMAC) Database
- Catalog-reported signals
- Not verified
- Reported license
- Not verified — check source terms
Inclusion ledger: 2026-08-31 · Signals not independently checked
- CoMindOpen
ETH Zurich and collaborators · 2026
A dual-view egocentric interaction dataset with two synchronized head-mounted cameras, paired exocentric views, scene and object scans, actions, hand poses, gaze, and social cues.
- Catalog-reported signals
- Video / RGB · Depth / 3D · Pose / calibration · Objects / hands · Audio
- Reported license
- CC-BY-4.0
Inclusion ledger: 2026-08-31 · Signals not independently checked
Why Ego matters in 2026
The question is no longer only whether first-person video helps. New work tests how human Ego data scales, what interaction structure must be recovered, and how much robot-side alignment still remains.
Human data now shows a scaling signal
EgoScale reports 20,854 hours of action-labelled human video and a log-linear relationship between pre-training scale and validation loss, with that loss tracking downstream real-robot performance.
EvidenceEgoScale · 2026Collection can move beyond the lab
AoE combines a neck-mounted smartphone holder, a cross-platform capture app, and cloud–edge processing to make distributed, long-duration first-person collection a practical research pipeline.
EvidenceAoE · 2026The transferable unit is structured interaction
HumanEgo lifts hand–object interaction into an embodiment-neutral representation, while EgoAVFlow models shared 3D motion for manipulation and active vision. Both move beyond copying pixels or wrist tracks alone.
The embodiment gap is now an explicit stage
EgoEngine generates robot-view observations and executable trajectories; Ego2Robot retargets actions and synthesizes robot embodiments at scale. The bridge is engineered and evaluated—not assumed.
Evidence baseFoundational Ego papers7 papers
01 · CVPR · 2022
Ego4D: Around the World in 3,000 Hours of Egocentric VideoEstablished large-scale first-person benchmarks for episodic memory, hand–object interaction, social understanding, and activity forecasting.
02 · NeurIPS · 2022
Egocentric Video-Language PretrainingBuilt EgoClip and EgoNCE to learn transferable video–language representations from narrated first-person activity.
03 · CVPR · 2022
HOI4D: A 4D Egocentric Dataset for Category-Level Human-Object InteractionAdded RGB-D sequences and geometry-aware annotations for hands, objects, poses, motion, and interaction state.
04 · CVPR · 2023
Learning Video Representations From Large Language ModelsUsed learned narrators to densify and synchronize language supervision for long-form video pre-training.
05 · ICCV · 2023
EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the BackboneMoved video–language fusion into the backbone so one pre-trained representation can support contrastive and multimodal tasks.
06 · CVPR · 2024
Ego-Exo4D: Understanding Skilled Human Activity from First- and Third-Person PerspectivesPaired ego and exo video with audio, gaze, 3D, IMU, and expert language for skill understanding and cross-view learning.
07 · CVPR · 2024
EgoExoLearn: Bridging Asynchronous Ego- and Exo-centric Procedural ActivitiesStudied how a first-person performer can follow and align with an asynchronous third-person demonstration.
From human view to robot learning
Ego data supplies observation and human demonstrations. Transfer methods must still bridge viewpoint, body geometry, action space, and robot dynamics before closed-loop evaluation.
01
Capture signals
Human-worn RGB, depth, gaze, IMU, audio, or body sensing.
02
Learn state and intent
Hands, objects, language, temporal context, and next actions.
03
Bridge embodiment
Align views, retarget motion, render robots, or distill policies.
04
Validate on a robot
Train or adapt a policy, then measure closed-loop behavior.
Official method video · silent
From human demonstrations to robot training data
Ego2Robot visualizes action retargeting, robot-arm synthesis, curation, and robot-side evaluation.
Transfer evidence
2026 Ego-to-Robot studies
Current 2026 work spans scaling, structured interaction, observation synthesis, active vision, and policy co-training. Most entries are recent preprints, so reported results are evidence to inspect—not settled consensus.
Compare transfer requirements
Author-reported evidence, not a cross-paper leaderboard. Tasks and evaluation settings differ.
- Input
- EgoDex; EgoVerse; ViTRA; ANT
- Method
- Robot-data synthesis
- Robot demonstrations
- Joint pre-training with robot data
- Validation
- VLA pre-training across morphology and scene perturbations, followed by real-robot deployment.
Limit: The reported synthesized hours include morphology-specific renderings of the source behavior; they are not the same as newly recorded task hours.
- Input
- Egocentric human RGB demonstrations
- Method
- Robot-data synthesis
- Robot demonstrations
- No target-task real-robot demonstrations reported
- Validation
- Policies are evaluated in simulation and on real robots, including zero-shot visuomotor dexterous manipulation.
Limit: The system recovers and synthesizes robot-compatible supervision; raw human RGB video is not used as an executable action stream.
- Input
- Egocentric human walking videos; Robot navigation demonstrations
- Method
- Policy co-training
- Robot demonstrations
- Human-derived data plus robot demonstrations
- Validation
- Evaluated on a fruit-search navigation task with a mobile robot.
Limit: The action conversion is designed for ground navigation and does not directly solve dexterous manipulation or arbitrary embodiments.
- Input
- Purpose-recorded egocentric human demonstrations
- Method
- Structured policy learning
- Robot demonstrations
- No robot demonstrations reported
- Validation
- Reports zero-shot transfer across real robot hardware using short human demonstration collections.
Limit: The method depends on recovering a reliable structured hand–object representation from the human video.
- Input
- Egocentric human manipulation videos
- Method
- Active-view policy
- Robot demonstrations
- No robot demonstrations reported
- Validation
- Real-world experiments evaluate manipulation under viewpoint changes and active camera motion.
Limit: The shared flow representation still requires accurate 3D motion estimation and a controllable active-vision setup.
- Input
- 20,854 hours of action-labelled egocentric human video; Aligned human–robot demonstrations
- Method
- Human-data scaling
- Robot demonstrations
- Small aligned human–robot set plus downstream robot data
- Validation
- Evaluated on real-robot manipulation with a 22-DoF robotic hand across multiple tasks.
Limit: The result supports human-data pre-training at scale; it does not remove the need for robot-side alignment and closed-loop evaluation.
Robot-data synthesis · arXiv · 2026Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data
Retargets hand motion, removes the human arm, renders target robot embodiments, and curates the resulting trajectories at multiple quality levels.
Open paperRobot-data synthesis · arXiv · 2026EgoEngine: From Egocentric Human Videos to Robot Learning
Transforms egocentric RGB demonstrations into robot-view observation video and executable action trajectories under embodiment and feasibility constraints.
Open paperStructured policy learning · arXiv · 2026HumanEgo: Learning Robot Manipulation from Egocentric Human Videos
Lifts hand–object interactions into an entity-level representation and trains a flow-matching policy with auxiliary interaction objectives.
Open paperActive-view policy · arXiv · 2026EgoAVFlow: Learning Unified 3D Flow for Robot Manipulation and Active Vision
Uses a shared 3D flow representation to predict manipulation actions, future scene motion, and camera trajectories from egocentric demonstrations.
Open paperHuman-data scaling · arXiv · 2026EgoScale: Scaling Egocentric Human Data for Robot Learning
Pre-trains on 20,854 hours of action-labelled human video, then aligns the representation with a smaller paired human–robot mid-training stage.
Open paperWhat to capture
Choose the rig from the supervision the model needs—not from the camera name. Richer spatial and intent signals improve alignment, while adding calibration, consent, and processing cost.
Browse all indexed equipmentWearable RGB
Head- or chest-mounted cameras; single or multi-camera rigs
Long-form video, hands, objects, speech, and activity context
Scales well, but video alone does not provide metric hand pose, depth, or robot actions.
Spatial headset
Project Aria, Apple Vision Pro, HoloLens, and related SLAM headsets
Calibrated video, head pose, gaze, IMU, hand pose, and spatial maps
Richer geometry improves retargeting, while proprietary formats and device-specific calibration raise processing cost.
Intent and body sensing
Eye tracking, IMU, EMG, pressure, touch, or synchronized body sensors
Attention, motion, muscle activity, contact, and pre-action intent cues
Adds valuable supervision, but synchronization, consent, calibration, and missing channels become first-order constraints.
Capture equipment index
6 representative systems
Start with the sensor stack, calibration path, wearer comfort, and export format. These are representative systems—not a ranking. Verify current specifications at the linked source.
Project Aria Gen 2
Spatial research glasses
Sensor stack
RGB, tracking cameras, eye tracking, audio, and IMU
DAS Ego
Panoramic head-mounted rig
Sensor stack
Six wide-angle RGB cameras and IMU
Orbbec EGO RGB-D
Head-mounted RGB-D rig
Sensor stack
Stereo depth, RGB, and IMU
Tobii Pro Glasses 3
Eye-tracking glasses
Sensor stack
Scene video, binocular gaze, and audio
LANYVE-OMNI 4P
Lightweight multi-camera headset
Sensor stack
Four synchronized RGB cameras and high-rate IMU
AoE smartphone rig
Neck-mounted phone kit
Sensor stack
Smartphone RGB plus app and cloud–edge pipeline
Maintain the topic
Suggest a dataset or paper, request data, or report a source error.