How AR can transform Art & Music


“To many, technology seems unapproachable. But through art, technology becomes more human and even an artistic medium in its own right.”

-Heather Day, abstract expressionist and Facebook Artist-in-Residence

Augmented Reality Apps

Over the past few weeks, we’ve highlighted a number of ways that AR can help make our worlds feel more personal, unique, and special: From cosmetics and home decor to sifting through millions of options to finding wall art that speaks to us, augmented reality has become an increasingly valuable tool in personalizing and beautifying our spaces.

But augmented reality is so much more than selfie filters and precisely-scaled furniture placement: To artists looking to explore new creative frontiers, AR represents an entirely novel expressive medium, one capable of conveying an immersive artistic vision quite unlike anything on the scene even a year or two ago.

Many early Augmented Reality experiences centered around museums and artifacts, with graphical overlays used to enhance a visitor’s understanding of a piece and overall experience. With the development of several new tools for creating AR experiences, a new wave of artists has taken to creating pieces explicitly designed for augmented reality’s unique properties and the experience it can offer an observer.

Each of these three artists seems a natural fit for augmented reality: They create big, expansive pieces that seek to transport the viewer to a world that is somehow set apart from the everyday. These three projects represent great examples of how technology can not only augment existing artwork with new information, but can offer creators an entirely new mode of artistic expression.

Heather Day [museum experience]

Platform: Facebook Camera

Heather Day, an abstract expressionist living in San Francisco, had already done commissioned pieces for the likes of Dropbox, Airbnb, and Snap when she was asked by Facebook to create the first major piece of art built on Facebook’s recently-enhanced camera platform. Working alongside Facebook’s Applied Machine Learning group, Day worked with traditional media at the company’s Menlo Park, CA headquarters as the Facebook team worked to bring her pieces to life in AR: 

Any time she poured paint, made a brush stroke, drew a pattern, or made any other type of mark, the AML team captured those movements on camera and added them to a digital library.

The AML team worked with Heather to determine which images should be given to the animators, and what movements they should make in the living, breathing AR installation. Within two weeks, they built technology that would recognize the specific location for the art and analyze the scene geometry that made Heather's virtual installation stick to real surfaces.

The result is an arrestingly beautiful mix of brush strokes and paint waterfalls, viewable only through the Facebook camera platform. Speaking to CNET, Day highlighted the platform’s ability to surprise and delight, which can be plainly seen in Facebook’s demo video featuring employees seeing the installation for the first time:

 "You can just be walking and say, 'Oh my god, I'm standing in a painting right now...It's new territory."

Nancy Baker Cahill [virtual trespassing]

Platform: iOS

LA-based visual artist Nancy Baker Cahill is known for large, almost explosive canvas pieces. She began experimenting with creating virtual reality art last year with Google’s Tilt Brush, and she told Forbes about the creative freedom she gained from working with this brand of multimedia: “Being able to draw in 360 degrees around my body and in space—that serves as an amplification and expansion.”

Screen Shot 2018-02-23 at 8.54.48 AM.png
She recently partnered with Drive Studios to port her work into AR, and the result is the gorgeous standalone iOS app 4thWall, which brings four of the artist’s works to life in augmented reality. Speaking again to Forbes, Baker Cahill highlighted a key component of AR: It only requires a smartphone, and that makes her augmented reality work far more accessible than her VR pieces: “In VR, I went and tried a number of experiences that were not related to art—more like examples of gaming experiences. I had that thing happen where your heart starts to pound and you realize, “Oh my God, this is actually the perfect next step.” VR is the most immersive medium we have right now. On the other hand, you need to have hardware to experience it. So then translating it into AR means being able to share it and democratize the experience of it.

Brian Eno [opening spaces]

Platform: Microsoft HoloLens

Musician Brian Eno is no stranger to exploring how technology can enhance and transform the live music experience, having previously helped to create an app for Coldplay’s song Hypnotised. Now, working with his frequent collaborator Peter Chilvers, Eno has launched Bloom: Open Space, a ‘generative musical experience’ that lets users create original pieces by interacting with holograms. You can read more about the full experience over at Wired, and if you’re fortunate enough to be able to make it to The Transformatorhuis (Trafo House) at Westergasfabriek in Amsterdam, the installation is currently running through February 25. For the rest of us, the video demo will have to suffice for now, but it’s likely that the project is being used to test the waters for many more future immersive projects on Microsoft’s much-hyped HoloLens platform


Read More:

Leave a Comment