Omnipresence and the coming age of ‘remote reality’

The coming age of remote reality will give us a kind of omnipresence in which we can virtually teleport anywhere in the world at any time. Unlike virtual reality (VR), where we can immerse ourselves in worlds that are entirely synthetic, remote reality gives us real power in our real world.

The key difference between the omnipresence of tomorrow and the telepresence of the past is the sense of immersion provided by the audio and visual detail that allows us to feel directly connected to the remote world. Immersive displays and audio systems will render this detail at full life size, and full surround audio to create a strong sense of reality, making it easier to “suspend disbelief” and allow ourselves to feel the experience. When we see people at full life-size, we have a stronger social response to them than when they appear miniaturized or with poor resolution. Immersive detail lets us feel a human-to-human bond and to see and act as if we were right there.

Encoding and transporting streams from multiple ultra-high resolution cameras and microphones in real-time is referred to as full-field communication and opens a huge number of new business opportunities: telemedicine will shift from just talking heads to full diagnostic detail, remote technicians will operate and repair complex machinery, shoppers will examine remote real estate, gems, rare manufacturing materials, electronics and other visually complex products.

People will soon be able to fly drones to tour exotic locations, pilot robots to meet with people, and shop at the most exclusive boutiques and exotic bazaars in the world, see the first-hand action of world-class athletes on the field or hear the full, visceral thrill of the music on stage right there with the band. All delivered live and in real time. Let’s look at an example in the domain of field repair:

Mike Smith has been a field technician for more than 20 years, repairing equipment for one of the world’s largest shipping companies. These days, he works as much from his office as in the field, using a full field communication system to assist and train new technicians who are located at the complex machines that need his expertise. Mike uses a full wall-sized display with retina-quality resolution that lets him see the full space of the remote engine rooms he services. Mike tunes into the panoramic camera that the junior technician wears atop her safety helmet, allowing Mike to obtain a full 360 degree panoramic view of the site. Not only can Mike see what the technician sees, the system shows exactly where the junior technician’s gaze is directed on the equipment, so Mike knows where his colleague’s attention is aimed. The junior technician’s equipment also contains a 360 degree directional microphone array allowing Mike to hear the precise location of troubled sounds from the equipment they are diagnosing.

Mike can draw in midair as if he were drawing right on the equipment, making diagrams of what actions the junior technician should take. The junior technician’s AR goggles show a video of Mike’s hands and diagrams, which maintain precise alignment with the equipment even when the junior technician moves her head. Mike can call up schematics and reference material and show his colleague exactly what needs to happen projected into the junior technician’s AR display. While they discuss the situation and clues with each other, intelligent listening software identifies topics and automatically retrieves pertinent information (part numbers, prior trouble-ticket reports, related problems and more) often before they even think of looking it up themselves. The result: Mike and his remote colleagues repair problems in less than one tenth the time and cost it used to take, keeping the fleet afloat and global trade channels open.

Network is the key

That is just one example of the power that omnipresence brings to real-world problems; imagine similar capabilities applied to medicine, sports, tourism, shopping, and more. However, digital realities require enormous amounts of data to match human sensitivities that are capable of processing over 500 gigabits per second of sound and light. Although advanced compression techniques can achieve 300 to 1 reduction and other optimizations can further reduce the throughput requirement to single-digit gigabits per second, networks of the future will need to carry orders of magnitude more data than they can transport efficiently today.

Another requirement for full immersion is to match the immediacy of actions in the real world. Miller showed in early human-computer interaction research that systems must respond at less than 1/10th of a second or else people can sense the delay (latency). Additionally, some of the effects of motion sickness that a few people experience in virtual reality systems are caused by the delay between that person’s motion and the system’s response — shorter latencies are needed to reduce such vertigo.

All of this makes clear that while there are still many breakthroughs needed on the camera, VR rendering, and display technologies that are core to any immersive experience, another important component is the network. Omnipresence requires transporting the remote reality between cameras, servers, and end users. In addition, cloud-based computing infrastructures can provide the necessary power to generate realistically rendered augmented information and to run computationally intensive computer-vision algorithms for object recognition, tracking, and prediction.

We’re nearly there. At the same time that camera and display resolutions are advancing, the networks of tomorrow, such as 5G wireless, promise to provide greater than 100 Mbps wireless pipes with a target delay of no more than 1ms. Video compression technologies are improving not just in terms of compression ratio but also in how fast they perform, approaching the requisite latency of less than 100 ms. The technologies are nearly there, but are we ready? Given the nearly god-like power of omnipresence along with virtual, augmented and remote reality, how many realities can a person actually live in?

Just one, of course, and it’s getting a whole lot bigger.

Bo Begole is VP of Huawei’s Media Labs, based in Santa Clara, Calif.