How 3D-printing, robotics and mixed reality created this HoloLens art piece

Transformation Mask is an experiential sculpture piece that brings art and technology together as it morphs from bird mask to human-merged-with-machine. It’s the result of a collaboration between Canadian artist Shawn Hunt and Microsoft Vancouver’s in-house maker space, The Garage, and leverages electronics and mechanical engineering in a physical transformation, before beginning an immersive holographic experience with the HoloLens.

But while the project is certainly interesting enough from a technical perspective – the way it blends 3D-printed and robotics elements with HoloLens mixed reality holograms via Bluetooth is sure to get anybody with a penchant for geeky tinkering very excited – what makes it truly ground-breaking is the way it uses those technologies to explore aesthetics, identity, and artistic expression in new ways.

Shawn Hunt is a successful Vancouver-based artist who – like many people in British Columbia – is of mixed descent, combining Scottish, French and indigenous Heiltsuk heritage. Rather than seeing that conflict of identities as a problem, however, Hunt has always leveraged it as a part of his art:

“I have never felt like I really belonged to any one particular movement, culture, category, or clique. As an artist this has given me an incredible amount of freedom. I don’t feel that my work is conceptual, traditional, artefact or craft. It is neither ancient nor modern. Instead, I feel as though my work has elements of all of these categories. This is a freedom that allows me to distort, subvert, hijack and remix these categories in order to offer new points of view. I want to challenge the viewers’ preconceptions. I like the idea of art being like a catalyst, or a flash point. I think art is most powerful when it poses questions, not when it gives the viewer the answers. My goal is to make the viewer think.”

So it made perfect sense for him to take this subversion and remixing to the next step by incorporating cutting-edge technologies in his work. That led him to accept Microsoft’s invitation to essentially come and play with all the shiny toys in their impressive Vancouver HQ. Hunt and his team of artists then collaborated with a host of designers and engineers to explore how Robotics, 3D-Printing and Mixed Reality could help tell traditional stories in a new and powerful way.

The raven in Heiltsuk mythology is seen as the “ultimate trickster” and that fluid identity seemed like a perfect idea to explore using those tools. The interactive installation they came up with features a bird mask in a traditional indigenous design which gradually morphs into a cyborg through a mixture of physical robotics mechanisms and virtual Mixed Reality elements. As the experience unfolds, the mask appropriates the traditional aspects of metamorphosis with the transformation from bird mask to human, yet in this adaptation the human mask has been altered, upgraded, and merged with the machine. Incorporating aspects of technology, sound and space, each part of the work reflects Hunt’s interest in how we understand and identify with the term indigenous.

“This work presents a new trajectory for engagement and exploration of First Nations practice; one that points towards technology and innovation as aspects that expand traditional practices and opens new avenues for interpretation,” says Hunt.

The process started with the production of a paper prototype of the mask, and the team also employed a variety of traditional methods, from initial hand drawn sketches to a full size foam core mock up. These were then replicated into a digital prototype mounted on an aluminium skeleton for hinge measurements. True to traditional form, there are no straight lines on the raven (a concept diametrically opposed to polygonal hard surface 3D modelling) and complex compound curves were lofted to generate the primary beak and head surfaces while procedurally generated hex patterns filled the voids for the raven’s eyes and nostrils.

At every step, the structures were digitally recreated and animated to avoid complications in the hardware build phase. Overall, the mask required nearly 300 hours of 3D-printing time. The model was digitally sliced into over 20 sections to meet printer specifications, with components assembled on open beam aluminium rails with interlocking elements between the printed pieces. The finished mask was made up of over 20 individually 3D-Printed components and measured over a meter.

The mask’s behaviour was achieved through an array of electronics, sensors, processors and mechanical elements. An ultrasonic range finder detects the presence of an attendee and automatically triggers the experience. Three linear actuators attached to an aluminium skeleton drive the primary phases of motion. Several microcontrollers work in unison to control individual addressable LEDs and behaviour states for the raven’s eyes projected through Windows phones. The mask transforms with the help of three linear servos driven by two microcontrollers, and communicates with itself and the outside world through electromagnetic and optical signals.

Modified LED strips line the inside of the mask triggering the hardware and software cues throughout the various stages of the experience. The HoloLens impressive computing power orchestrates all these elements, synchronizing the software and hardware over a Bluetooth connection.The holographic experience, where the animated volumetric drawings appear in concert with particle simulations and spatial sounds, was sequenced and rendered in real time using Unity3D. To design it, Shawn started by familiarizing himself with a range of technologies. Drawings that started as volumetric digital sculptures in Tilt Brush were later refined using 2D digital inking to perfect the traditional lines and forms Hunt intended to express. The 3D drawings were further enhanced with purpose-built skeletal systems to create organic animation. Sound was also a key part of the experience, with custom sound design being implemented first, inspiring the timing and intensities for visuals rather than adding it at the end. The result is a powerfully synchronized audio-visual-mechanical experience.

“Art comes in many forms, can easily transcend cultures and utilize technology to become something new. Life is all about these infinite possibilities and combinations,” says Andy Klein, HoloLens Mixed Reality Designer.

And that’s precisely what makes the Transformation Mask such an interesting use case for HoloLens and Mixed Reality Technology: Rather than separating artistic and technical functions, these were very closely blended throughout the conceptual, design and execution phases of the project. Just as the traditional indigenous masks blurred the line between human and animal, reality and myth, technology brings together human and machine, the physical and the virtual. It is clear that if we are to create compelling, engaging and inspiring content that also resonates culturally with audiences, that’s an approach that content makers would do well to learn from and adopt.