1. What Is Actually Slowing Down Progress in Physical AI?
The common assumption is that robotics is held back by weak algorithms or limited computing power. In reality, the bigger obstacle is data. Training a robot to perform even a simple task, like pouring a cup of coffee or plugging in a cable, requires far more physical world information than most companies currently have access to. This is different from how large language models were built. Text for LLMs was already sitting on the internet, ready to be scraped at very little cost. Physical interaction data does not exist in the same way. According to Encord's own analysis of robotics data, there is no public corpus of robot demonstrations comparable to Common Crawl or LAION, so every dataset of meaningful size has to be collected from scratch, recorded, labeled, and structured before a model can learn anything useful from it.
2. Why Robotics Companies Are Struggling to Find Training Data
According to Vineeth Velmurugan, who leads robot learning at data infrastructure company Encord and previously worked at OpenAI's robotics division and warehouse automation firm Berkshire Grey, the scale of the problem is significant. As reported by TechCrunch, he estimates that closing the current data gap for robotics would require a dataset roughly five times larger than the entire video library of YouTube.
That number puts the challenge into perspective. It is not a matter of collecting a bit more footage here and there. It requires building an entirely new data supply chain, one that does not yet exist at industrial scale.
Self driving car companies faced a similar issue years ago and solved it by collecting real world driving data directly. That approach worked for cars, but it is difficult to replicate for robotics because the range of physical tasks a robot might need to perform is far wider than driving a vehicle from point A to point B. Training robots purely from existing video content scales more easily, but video alone lacks the depth and precision that comes from real physical interaction.
3. How Companies Are Manufacturing Data From Scratch
Since usable training data is not readily available, companies working in this space have started building it themselves. There are currently two primary methods being used across the industry.
3.1 Egocentric Video Capture
The first method involves recording video from the perspective of a human worker performing everyday tasks. Cameras worn on the body capture natural hand movements and interactions with objects, giving models a first person view of how tasks are actually completed. This footage is gathered from real work environments, including factories, to reflect authentic physical conditions.
3.2 Teleoperated Robots and Leader Follower Rigs
The second method uses what is known as a leader follower rig. This setup involves two robotic arms, one controlled directly by a human operator and a second arm that mirrors those movements in real time. This allows precise physical actions to be recorded and later used to train robotic systems.
These rigs are often used to capture tasks that seem simple to humans but are surprisingly difficult for machines, such as pouring liquid or stacking small objects like poker chips. A related report from The Spokesman-Review described this work firsthand, noting that pilots can spend hours repeating the same task, such as pouring coffee, so robots can learn to correct their own mistakes over time.
Facilities built for this purpose are often stocked with common household items, including books, fake plants, storage containers, and cables, since many robots are ultimately intended to operate inside homes rather than only in industrial settings. In more specialized environments, operators also test tasks like plugging and unplugging network cables in server racks, a task that highlights just how far robotic dexterity still needs to come. Human fingers offer a level of precision and flexibility that most mechanical grippers cannot yet match.
4. What New Data Types Are Emerging in Physical AI?
Beyond video and teleoperation, companies are now experimenting with newer forms of data capture that could offer more detailed insight into human movement and cognition.
4.1 Muscle Signal Sensors
One emerging method uses sensors placed on the forearm to detect electrical signals generated by muscle activity. The goal is to build a more accurate three dimensional understanding of hand movement, something that standard video often fails to capture with enough precision.
4.2 Brain Wave Headsets and Mental State Signals
A more experimental approach involves brain wave headsets that measure mental states such as error recognition, intent, and surprise while a person performs a task. This method is currently being tested in an early stage collaboration with Zander Labs, a German neuroscience company that specializes in noninvasive brain-computer interface technology.
Per the TechCrunch report, a Zander neuroscientist supervising the work explained that the amount of brain activity recorded during a task can offer clues about when a model needs to apply more computing effort. The idea is that brain activity could indicate the exact moments when a task becomes mentally demanding, not just physically complex. This work is still in a trial phase, with an initial dataset planned to test performance gains before any decision is made to scale it further.
- Trial goal: build an initial brain wave-tagged dataset
- Testing method: run the dataset through customer robotics models
- Decision point: evaluate real performance gains before scaling
5. Why Annotated Data Matters More Than Raw Footage
Collecting raw video or sensor data is only part of the process. For this data to actually improve a model, it needs detailed annotation, meaning specific descriptions of what is happening in each clip, such as identifying when a hand tightens a bolt or lifts an object.
Estimates suggest that this kind of detailed annotation can be worth around one hundred times more than raw, unlabeled footage when it comes to training models for specific tasks. The cost of producing annotated data is higher, roughly twenty times more than raw footage, but the value it adds far outweighs that additional cost. This is why annotation has become just as important as the data collection process itself.
6. How Physical AI Data Economics Differ From Language Model Training
It is worth understanding why physical AI cannot simply follow the same playbook that worked for language models. Text based AI systems benefited from a massive, pre-existing pool of internet content that could be gathered at minimal expense. Physical interaction data offers no such shortcut.
This gap is also drawing serious investor attention. Encord recently raised additional funding to expand this work, with the company noting that it expects more than 400 million intelligent robots to come online over the next four years, pushing the physical AI industry toward tens of billions of dollars in annual value. Every clip, every sensor reading, and every annotated action has to be intentionally created, and that changes the underlying economics of building physical AI systems. Progress depends less on discovering new algorithms and more on building the infrastructure needed to generate high quality, real world data at scale.
7. What This Means for the Future of Robotics
The broader takeaway is that robotics may not be limited by intelligence or model design as much as it is limited by access to the right kind of data. Companies working across multiple robotics clients are in a unique position to notice patterns and gaps in training data before individual companies would recognize them on their own.
As experimentation continues with new data types, including muscle sensors and brain wave signals, the industry is signaling a shift. The next major advancement in physical AI may not come from a smarter model architecture, but from a more complete and detailed understanding of how humans physically interact with the world around them.
8. Frequently Asked Questions
8.1 What is physical AI training data?
Physical AI training data refers to information collected from real world human movement and interaction, including video, sensor readings, and annotated actions, used to teach robots how to perform physical tasks.
8.2 Why is physical AI data harder to collect than language model data?
Unlike text, which was widely available online, physical interaction data does not exist in large quantities and must be actively created through methods like video capture, teleoperation, and sensor recording.
8.3 How does brain wave data help train robots?
Brain wave data is being explored as a way to detect mental states like intent, error, and surprise during a task, which could help models understand when a physical action requires more focus or complexity, based on early trials between Encord and Zander Labs.
9. Final Thoughts
Physical AI is entering a phase where data creation, not just model development, will determine how quickly humanoid robots and automated systems improve. The methods being tested today, from egocentric video to brain wave signals, represent early steps toward solving a problem that has no shortcut. As this space matures, the companies willing to invest in building high quality, real world data are likely to shape how fast physical AI moves from experimental to practical.