Mixed Reality differs from Augmented Reality (AR) in that, in addition to mixing real and virtual images, it offers the possibility of interacting with virtual elements. For example, in a video game, a character may perceive a greeting from our physical hand. Or the virtual sculpture will react to the movement of our hands when we flip or rotate it. The business magazine B-MAG has prepared an overview of five examples of mixed reality (MR) that will progress in the near future.
What the future holds for us
HoloLens glasses have made great strides in enterprise applications, but they are still relatively expensive and bulky, so they are not yet everyday devices. However, new technologies are likely to enable mixed reality (MR) applications. Only time will tell what remains hype and what reaches mass distribution.
Giant screens with Mixed Reality
Imagine if your glasses could show you a giant movie screen while riding the bus. Thanks to direct projection onto the retina (which we will discuss at the end of the article), transparent glasses can offer the user a cinematic experience close to a conventional movie theater. There aren’t any yet, but eyewear manufacturer GLOW provides the ability to select a movie with a hand gesture and project it directly onto the lens of the device.
Next-generation video games
Made a lot of noise – Pokemon Go. This video game offers augmented reality features, but its developers wanted to go further. Thus, in the last demo, you can see the possibility of user interaction with virtual creatures using hands. It seems that Apple is also interested in these applications and plans to develop mixed reality technologies.
Education and MR
One area destined to benefit from mixed reality is education. A recent example is HoloMusic XP, a tool that helps students learn to play the piano using mixed reality and gamification. According to a recent study, VR, AR, and MR technologies are expected to improve information retention, motivation, and concentration of users during the learning period.
Repair of equipment in extreme conditions
In an environment like outer space, it’s impossible to send an entire team of engineers to service complex devices. So in 2020, NASA turned to mixed reality for one of its astronauts to upgrade the Cold Atom Laboratory, a quantum laboratory aboard the International Space Station. On our planet, it is already used to control desalination plants, water treatment plants, and wind farms.
Geospatial Modeling
Project Anywhere XR is a technology initiative to render complex maps in 3D. This includes everything from orographic features to urban environments. Apart from mixed reality glasses, an important aspect is that all content comes from the cloud in real-time.
What is required for an optimal mixed reality experience?
In addition to mobility, which is as basic a requirement as two wheels for a bicycle, mixed reality has to solve several important problems. Failure in any of these can break the dive and lead to unwanted effects such as dizziness or confusion.
- Minimum delay
The slightest inconsistency between controller instructions and the displayed image can spoil the impression. This is known as a delay. For example, the new Apple glasses are rumored to be paired with the phone using Wi-Fi 6E technology, which ensures minimal latency.
- Advanced Environment Recognition
In order to integrate elements into a scene, it is necessary to have full recognition or mapping of the environment. For this, in addition to cameras, it is ideal to have depth sensors such as LiDAR, which receive a laser-scanned and three-dimensional image of the room. Finally, AI algorithms are required to process all this data.
- Ability to use the management interface
Unlike augmented reality, the mixed reality examples we’ve seen require some sort of technology to interact with virtual objects. Cameras and depth sensors can identify the user’s hands, but mixed reality glasses will likely need additional controllers. An alternative would be ringing with motion sensors, such as those patented by Apple.
- Ability to integrate virtual objects
This is one example of the challenges that mixed reality faces. If you want to wear glasses with this technology for a long time, all elements must be integrated into the scene. Ultimately, this means that their proportions and focus correspond to nearby physical objects. They also need to take into account eye movement.
Latest Breakthroughs in Mixed and Augmented Reality Technologies
Of the four problems we have mentioned, the last one is probably the most difficult. Fortunately, technological innovations are already being developed that could provide a solution. As already noted, the problem is not so much that the virtual object may appear more out of focus as the distance increases.
After all, if the room is plotted on a three-dimensional map, it is easy to assign characteristics to the depicted object. No, the problem is that the virtual object behaves logically according to retinal focus and eye movement.
This is where one of the most exciting technological breakthroughs of recent years comes into play: retinal projection. This technology, which is still in its infancy, projects images directly onto the retina rather than onto screens. This allows for seamless integration of images into the environment, which will benefit both mixed reality glasses and augmented reality glasses.
In addition to the patent recently obtained by Apple, some prototypes offer functionality in this sense. One of them will be Light Drive glasses, presented by Bosch at the CES 2020 technology show. This model, not yet on the market, projects images in the form of laser beams onto the retina.
Another technology that is likely to be used in mixed reality glasses will be developed by the Swiss company CREAL. As seen in this video, their device projects a hologram known as a light field onto the lenses.
In this case, this involves projecting six thousand images per second to each eye at different angles and depths, emulating the behavior of light. Thus, the eye can change the focus of an object depending on which part of the scene it is looking at.
Regardless of the prevailing system, it is likely that glasses of this type will depend on the processing power of the paired device to remain lightweight and manageable. One example of mixed reality in this sense is the Nreal glasses, which are associated with an Android phone.
With this approach, longer battery life and 5G connectivity can be offered, which will be critical for the mass adoption of these devices. However, this will be only an intermediate stage. Ultimately, mixed reality glasses are expected to give way to smart contact lenses like the prototype unveiled at CES 2021.