Watch a Microsoft engineer use Kinect and a Sphero to simulate telekinesis

Telekinesis, defined as the psychic ability allowing one to influence a physical system without physical interaction, lives largely in the world of science fiction. Yet as we continue to see time and time again, technology is quickly helping drop the second word of that phrase and bring what we see on the big screen to the real world.

The latest example comes from Microsoft engineer Mike Taulty, who works on the company’s Developer and Platform Group. Watch him use hand gestures to move a ball around a room without ever touching it (the good bit starts at 1:45):

The ball in question is a Sphero. Designed by Orbotix, the robotic toy is controlled by a smartphone or tablet via Bluetooth. The latest version we covered is the Sphero Ollie (previously known as Sphero 2B).

You’re probably a lot more familiar with Microsoft’s Kinect, the motion-sensing input device for Xbox and Windows, though it’s worth noting that the Kinect SDK 2.0 was released recently. The new software development kit (SDK) essentially allows anyone to use the second-generation Kinect (which launched with the Xbox One) with their Windows PC.

Taulty used it to build an app for Kinect that runs on his laptop and interfaces with the Sphero. Instead of using touch as input, he can use the position of his hands with respect to his body.

This system relies on Kinect’s skeletal tracking, which by the way is significantly more accurate in the second-generation sensor. Here’s a recap of the improvements Microsoft made to the feature over the first generation:

The enhanced fidelity of the depth camera, combined with improvements in the software, have led to a number skeletal tracking developments. In addition to now tracking as many as six complete skeletons (compared to two with the original sensor), and tracking 25 joints per person (as compared to 20 with the original sensor), the tracked positions are more anatomically correct and stable—and the range of tracking is broader. This enables and simplifies many scenarios, including more stable avateering, more accurate body position evaluation, crisper interactions, and more bystander involvement in interactive scenarios.

It’s pretty amazing that all Taulty needed to do was write some JavaScript code, and suddenly he could control the rotation of the Sphero with just his left hand and its direction of movement with his right. Microsoft envisions digital displays in shopping centers, museums, advertising displays, gaming arcades, and public spaces in general that let passersby “manipulate products or objects themselves.”

For detailed instructions on how to set this up yourself, check out Taulty’s in-depth guide: Playing with Kinect for Windows V2 and Sphero from a Windows 8.1 Store App.