Transforming How Users Experience Reality with AR & MR
So far on our blog, we’ve spent a lot of time on what’s broadly called ‘Practical AR.’ We’ve looked at use cases in which AR solves everyday problems like what furniture to buy or what artwork would look good in your living room, and we’ve highlighted how ARKit and ARCore are helping developers make dynamic AR experiences on the phones everyone already has. Augmented Reality Apps.
In our upcoming Consumer Applications of Augmented Reality whitepaper, we talk a lot more about how practical Augmented Reality is poised to revolutionize the consumer experience over the next few years. We also spend time looking at a somewhat surprising DigiCapital forecast which predicts that the Augmented Reality Apps market will actually be four times bigger than the virtual reality (VR) market by 2021. The reasons for this seem pretty simple - virtual reality, for all the interest it has generated, doesn’t have many practical uses for consumers beyond gaming. Its headsets are clunky and require a ton of computing power, and thus far the barriers to entry for the everyday consumer have kept VR a niché market.
Where we haven’t looked yet is the fuzzy world of mixed reality (MR) - a world so fuzzy, in fact, that no one has quite provided a definitive answer as to what actually makes a mixed reality experience. Confusing definitions abound on the internet, but the easiest way to think of mixed reality is that it’s more immersive than simple Augmented Reality Apps but still isn’t totally computer generated: It’s the merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interact in real-time.
What kind of mixed reality is available now?
We’re just now beginning to see consumer-facing mixed reality hardware make it to market, and nearly all of these devices are built on the Windows Mixed Reality (WMR) platform. WMR became a standard feature of Windows 10 in Q4 of last year, and Microsoft tapped several of the biggest names in hardware - Samsung, Dell, Asus, Lenovo, and HP - to build headsets that run on WMR. With this line of devices, the idea was for them to be cheaper and less-graphically intensive than high-end VR headsets from HTC, Occulus, and others. The ability to run WMR headsets on lower-resolution graphics settings means that they can pair with something as inexpensive as an ultrabook, adding to their potential reach with consumers.
Not surprisingly, the majority of early use cases of Augmented Reality Apps on the WMR app store are games, and most of these standalone apps would qualify more as full virtual reality. Responses to the headsets have been mixed, with many reviewers pointing out that WMR isn’t all that different from traditional VR. Still, with the headsets’ reduced price points and a centralized platform to build on, it’s clear that major enterprise is still betting on this type of mixed reality finding its market over time.
Meta and Microsoft: The Next Generation of MR
Beyond what’s currently available on the consumer market is where things begin to get truly innovative. Microsoft has been acquiring startups to contribute to its own mixed reality headset for years, and the enterprise and developer versions of its HoloLens are finally available.
Hololens looks more like an Augmented Reality app on steroids, at least based on its video demonstrations and Microsoft’s hypothetical use cases for the product. Its success (or failure) is likely to hinge on the degree of investment the company can get from third-party developers, who also have options like the hardware from Meta. Like Hololens, the appeal of the Meta 2 is in its ability to render extremely lifelike holograms that a user can fully interact with via intuitive hand controls, and the company has already scored impressive partnerships with Dell and Dassault Systèmes’ (makers of the 3D CAD SOLIDWORKS apps). Meta 2 is a wired device, so it seems like it's being positioned as something more like a monitor replacement than a device to walk around with. Like Microsoft, Meta is heavily courting developers at the moment, and 2018 looks to be a make-or-break year for both systems as they attempt to rise above the scrum in an increasingly crowded mixed-reality headset market.
Magic Leap (Almost) Says, “Hello World!”
Magic Leap operated in stealth mode for years before announcing its mammoth $542 million series B funding round in October of 2014, and the technology that film producer and former Legendary Pictures Thomas Tull told FastCompany, was “So badass you can’t believe it,” and the company has since upped its funding total to nearly $2 billion dollars without releasing even a demonstration of what its product might look like.
That all changed last year, with the announcement that the Magic Leap One Creators Edition would launch in the early part of this year. After glowing-yet-reserved (largely due to non-disclosure restrictions) write-ups in Wired, Fortune, and the MIT Technology Review, the world finally got a look at the Magic Leap One headset and an accompanying profile of the company’s CEO Rony Abovitz in Rolling Stone.
While the development world continues to wait on the any-day-now release of Magic Leap’s SDK and ‘Creator Portal,’ the company has announced a number of exciting collaborations that aim to demonstrate just how groundbreaking Magic Leap’s technology really is. Pitchfork was invited to the company’s Florida headquarters to demo Tónandi, a mixed-reality app that Magic Leap has been developing with the band Sigur Rós for the past four years. Though the report was again scant on actual hardware details, Marc Hogan’s description of experiencing the app certainly sounds unlike anything that’s yet been seen anywhere in the AR/VR space:
Pitchfork/ Magic Leap
The real-world demo space I’m in is decorated to resemble a living room, and the tónandi adjust to account for a table with oversized hardcover book on top of it. At one point, waving my hands near the table allows me to summon up the roar of lead singer Jónsi Birgisson’s signature bowed guitar; in another corner, banging on virtual white puffy wisps releases different tones of what is unmistakably his voice. Whatever I’m seeing, it isn’t just something pasted over my surroundings, but something that acknowledges those surroundings, and therefore seems more real.
Each user of the app can have a different experience. The three members of Sigur Rós check out this latest iteration of the app the same day I do, and we compare mental notes afterward. When I ask drummer-keyboardist Orri Dýrason about drum sounds I swear I heard in the app, he protests, telling me he didn’t hear any. But later on, the Magic Leap team confirms to me that there are some drums near the end—you just have to find them.
The company also recently announced a landmark deal with the NBA. Announcing the plan alongside league commissioner Adam Silver, Abovitz hinted that the partnership will bring viewers an entirely new way to experience live sports: In addition to providing an immersive viewing experience, Abovitz told the media that Magic Leap “senses you...gets an emotional state, biomarkers, and information about the world around you,” and then adjusts the end-user experience based on these inputs.
Just how Magic Leap will do this remains as mysterious as the company itself, but the announcement did give us Shaq wearing the headset, so it was generally a win all around:
Though details on where Magic Leap really wants to take things with mixed reality are still fuzzy, the overarching theme of everything we’ve seen come out of its headquarters in the last few months makes it seem as if Abovitz and his team are aiming for nothing short of transforming how users experience reality - perhaps even transforming the nature of reality itself.