5 ways Project Tango, Google’s 3D mapping initiative, can change how we interact with tech

Sanjay Patel is CEO and co-founder of Personify, an immersive video technology company.

As you’ve most likely seen, Google recently announced Project Tango, a bold new vision for the future of the smartphone. Mainstream advancements in this space started with Microsoft Kinect, and continued with Intel’s Perceptual Computing initiative and Apple’s acquisition of PrimeSense.

With Project Tango, we are one massive step closer to a world where all PCs, tablets and smartphones have 3D sensors. 3D imaging systems are able to capture depth information of a scene, in addition to the color information captured by the conventional 2D sensor already in our devices.

When 3D imaging is everywhere, what will be the impact?

First, let’s take a deeper look at Project Tango itself. Tango is a mobile device prototype that can track your motion and map your surroundings with its built in sensors and processing capabilities. To do this, it uses a rear-facing 4MP camera, a 320×180 depth sensor, a 180-degree field-of-view fisheye camera and a 120-degree field-of-view front facing camera.

[We’ll be exploring a plethora of ways to grow your mobile business at VentureBeat’s Mobile Summit next week. Sign up for an invite now!]

At the heart of this breakthrough technology is the Myriad 1 vision processor designed by a company called Movidius that can deliver one teraflop of computing power for motion tracking and 3D mapping without draining the battery. Those are some serious sensing and processing capabilities!

With Project Tango, Google has, yet again, taken the bold path of getting the developer community involved to build applications and new user experiences for the device.

So what can we expect to see from those developers who have early access to Project Tango prototypes? Immersive experiences and augmented reality applications that see, process, recognize and synthesize your surroundings are all part of the possibilities with Tango.

Imagine being able to use a smartphone to guide someone who is visually impaired through unfamiliar territory, or being able to capture dimensions within your home that can you can utilize while shopping for new furniture or scanning in an object, reconstructing it digitally and printing it on a 3D printer. Of course, the possibilities for gaming applications are endless and virtually untapped at this point.

This is just the beginning. The key to Project Tango and the groundbreaking technologies being driven by Microsoft, Intel, Apple and others is 3D imaging. Below I’ve outlined five of the most impactful innovations that will soon become a reality as 3D imaging reaches the mainstream.

Gestures

Love ‘em or hate ‘em, we’re all becoming more comfortable interacting with our devices with a pinch or flick, and soon we’ll be comfortable interacting with a wave or gaze.

This will be particularly important in environments where it’s not easy or convenient to touch a screen or mouse, such as in an automobile or in your living room when your PC is across the room. While there are innovative companies working on gesture input with conventional 2D cameras, the experience becomes much broader and more robust with the addition of the depth information captured by a 3D imaging system.

Computational imaging

Thanks to broad-appeal apps like Instagram, the world has become very familiar what can be accomplished by simple image filters applied to a conventional 2D photo. 3D imaging opens an enormous range of possibilities for what we can do to a photo or a video stream computationally.

Depth data gives us a 3D view of the scene in front of the camera. Features include real-time background subtraction, stunning bokeh effects, refocusing and relighting of photos after they are taken — even changing the viewing perspective or the angle at which the picture was taken. The new HTC One smartphone offers a glimpse of what is possible. It has a world-facing dual-camera which captures enough 3D information to enable some very cool computational photo effects.

3D scanning

3D printing is booming. 3D printers are an ideal output device for the 3D data captured by 3D imaging systems. Imagine being able to wave your smartphone in front of an object and capture that object as a high-resolution 3D surface mesh, and then print it on your 3D printer.

Immersive experiences

The acquisition of Oculus Rift by Facebook has re-awakened the virtual and augmented reality space big time. Devices such as Project Tango that map the world around you in 3D are great platforms for immersive experiences where digital content is added to the world around you. The 3D information captured by Tango enables a seamless fusion between the digital world and physical world, which enriches the overall experience.

Robust facial recognition

Fingerprint sensors are becoming more common for protecting valuable data (and digital wallets) on our smartphones. The next advancement on this arc is for devices to automatically recognize user’s faces visually, in a highly robust manner. 3D imaging enables this to happen by capturing a set of 3D points along the user’s face, which can make the authentication process more accurate. To contrast, techniques that use conventional 2D cameras can be easily foiled with a picture.

At my company, Personify, we’ve been investing in the future of digital imaging for four years. Innovative companies like Google are paving the way for our vision to become a reality and by tapping the developer community, they’re certain to raise the bar of applications that will be available from the very beginning of Project Tango-derived smartphones.