Controlling a physical robot arm through natural movement in mixed reality.
This project is a complete mixed-reality teleoperation system that connects a digital twin in Unity to a physical six-degree-of-freedom robot arm. A user wearing a Meta Quest 3 can manipulate the virtual arm naturally and watch those movements transfer to the real robot in near real time. The result brings together mechanical design, additive manufacturing, embedded control, inverse kinematics, XR development, and network communication in one system.
The arm was designed in SOLIDWORKS and manufactured on an Ender 3 Pro 3D printer. Six servos provide the physical degrees of freedom, while a Raspberry Pi Zero 2 W runs the arm's control software. The Pi receives a continuous stream of joint angles from the digital twin, converts those angles into servo-ready commands, and updates the physical arm.
A PCA9685 16-channel, 12-bit PWM driver controls the servos. Because the Raspberry Pi cannot safely provide the current required by six motors, the servo driver uses a dedicated external power supply. Separating control from motor power made the system more reliable and protected the Pi from voltage drops and current spikes.
The virtual twin followed a multi-stage asset pipeline. The original SOLIDWORKS assembly was exported as an STL, rigged with bones in Blender, exported again as an FBX, and imported into Unity. Unity's XR tools provide the Quest 3 mixed-reality environment, controller and hand tracking, and passthrough view. The user can therefore see the physical room while directly manipulating the virtual robot arm.
The Quest application combines the rigged robot, mixed-reality integration, tracking input, and a virtual work surface into a focused teleoperation scene. This made it possible to test the digital twin from a first-person perspective and compare virtual motion against the physical mechanism shown in the demo video.
Inverse kinematics was the most difficult engineering challenge. A Chain IK Constraint uses the Blender armature to coordinate the upper arm, forearm, and wrist, while a Multi-Aim Constraint integrates shoulder rotation with the rest of the chain. Getting the basic rig moving was only the first step; making it behave like the real mechanism required carefully constraining each joint's allowable axes.
Two C# scripts manage the shoulder and upper-arm constraints. Using Unity's localEulerAngles, each script continuously locks the axes that should not move: X and Z for the shoulder, and Y and Z for the upper arm. The forearm and wrist could then be solved by the IK chain without additional constraint scripts. This approach turned a general 3D rig into a digital twin that respects the motion limits of the printed arm.
Unity represents the required joint range from -90° to 90°, while the servos accept values from 0° to 180°. Instead of coupling hardware-specific conversion logic to the Unity project, the Raspberry Pi performs the mapping by adding 90° to each incoming angle. Joint-specific calibration was also required: the forearm's mechanical home position begins at 30°, so its command path needed a separate offset and extensive testing.
Unity transmits the live joint angles to the Pi over UDP using the Pi's local IP address. UDP was selected for its low overhead and suitability for a continuous stream where the newest pose matters more than retransmitting an outdated one. The Pi receives each packet, applies the angle mapping and calibration, and sends the final commands through the PWM driver to the six servos.