From Playing a Xylophone to Training Robots
This demo explores a simple idea: using the 3D orientation of wearable IMUs to control a human model in real time.
Two Shimmer3R devices are used to track the upper arm and wrist. Their orientation is mapped onto a 3D arm model, allowing the user’s movement to be reproduced virtually. In this example, the virtual arm is used to play a xylophone.
For the demo video, the Shimmer devices were simply tucked into an arm sleeve. This was convenient for a quick demonstration, but it is not ideal for motion tracking because the sensors can move relative to the body. Better and more consistent results should be possible using a proper harness or more rigid sensor attachment.
There is nothing particularly novel about using IMUs for motion tracking. The purpose of this demo is instead to show how quickly applications like this can be built using Shimmer3R, together with the SDKs, example code and development tools we provide.
Where could this go next?
One interesting direction is embodied AI and robot learning.
A person could wear egocentric camera glasses while carrying out everyday tasks, with multiple IMUs capturing how their arms and body move. This would create synchronized egocentric video and human motion data that could potentially support areas such as imitation learning and learning from demonstration.
Much robot-training data is still collected through simulation, teleoperation or controlled data-collection environments. Large-scale collection of multimodal human demonstration data in real homes and workplaces could help capture a much wider variety of objects, layouts, behaviours and regional differences in how everyday tasks are performed.
For example, the same task may be carried out differently depending on the home, tools available, cultural context or individual habits. Capturing this diversity may be important for developing embodied AI systems that need to operate outside highly controlled environments.
The xylophone demo is only a small example, but the underlying idea is broader: wearable sensing can provide a relatively simple bridge between human movement, digital environments and future embodied AI systems.
