Spatial sound will be a critical component of Mixed Reality

Back in late October of 2014, I had the pleasure of meeting with Wyclef Jean and his long-time manager at her New Jersey home to discuss Augmented Reality and how important it would soon be to the music and entertainment industry.  When I arrived, Clef and his manager were busy discussing an array of important business decisions for the week as I sat patiently waiting, watching CNN on a nearby TV in the living room area.

Finally, when called in, his manager excitedly asked me to show Clef several of the AR examples I had demoed for her a few weeks earlier. Quickly, I went through some sample activations, including a bottle of Heinz Ketchup that came to life with a 3D recipe booklet.  At that very moment, I witnessed Clef sit back in his chair as a light bulb seemed to instantly turn on in his head.

He then proceeded to cut me off and ask, “Do you know what binaural sound is? I want to be the binaural sound of this technology. I don’t just want my music to play through your phone, I want to bring supersonic-stereo capabilities to this.”

Wyclef immediately got it. Not only could he see the promise of immersive technologies, he could hear it.

The 💜 of EU tech

The latest rumblings from the EU tech scene, a story from our wise ol' founder Boris, and some questionable AI art. It's free, every week, in your inbox. Sign up now!

“Remember when you had regular TV and then you had TV with ‘Dolby’ sound?” he asked. “That’s what I want. I want to make that ketchup sound so crisp and realistic, it sounds just like it’s squirting out of the bottle onto your shirt.”

I will admit, at the time I was a bit dazed and confused, but in the best of ways. My head was shaking no but my mouth was saying “yes!”

Fast forward to today, and it’s pretty clear there has been a lot of focus on the really cool visual stuff: the holograms, animations and many mind-blowing visual elements that Mixed Reality and optic displays will bring into our daily lives.

Equally important – but perhaps not as front and center in today’s discussion – is how these holograms and immersive experiences will sound, and the technology that will convince our brains we are really experiencing these digital transformations. With this in mind, let’s jump into what Clef dubbed “supersonic stereo” – otherwise known as 3D spatial sound.

What is Spatial Sound?

Spatial sound is the natural progression from ‘Stereo’ and ‘Surround Sound,’ to the next level of being able to hear and interpret noises, like we do in the real world.

Dolby surround sound spiked in the 1990’s when flat screen TV’s and home cinema set-ups became all the rage, and people tried to recreate intimate cinema environments within their own homes. These home speaker systems have not changed much over the past couple of decades, until recent advancements in technology.

Since 2011, 3D speaker arrays have been installed into cinemas across the globe with platforms such as Auro-3D, Dolby Atmos and DTS:X providing the tools with which sound engineers can create and mix amazing 3D soundscapes, soundtracks and scores.

A 3D speaker array combines two surround sound systems, one placed reasonably higher than the other, to bring ‘height’ into effect and therefore allow the audio to replicate the real world, with layers of sound that we continually interpret.

Essentially, the same theory that works for a fully spatial, atmospheric 3D experience over speakers can be applied into a headphone environment to make the listener feel like they are engaged in a real scenario, with a true feeling of space.

Now that axes are available as referenced below, sound sources and layers of sounds can be positioned, manipulated and augmented to recreate real world ambiance and atmosphere.

With the development of movement tracking in mobile devices and HMD’s (head mounted displays), the ability to add the 3rd dimension of height into audio experiences is changing the way audio is being created, mixed and delivered over headphones.

How we hear 3D in the real world

Positional sound and directional tracking of individual sound sources, along with multiple layers of spatial ambiance, all combine to convince our ears that we are in a perceivable ‘space’, with subtle timings of sound travelling between objects to tell us how far away they are.

Sound, unlike light, can travel around corners and provide us with information about actions and events we would otherwise have no reference to, essentially providing us with a sonic warning system. So when these objects are out of our line of sight, the way our brains determine what’s happening all around us is derived purely from sound.

For example, if we want to grab someone’s attention in the real world we call their name, shout, beep a horn or raise an alarm. More simply, if we’re having a conversation with friends or colleagues, we are able to speak our turn and our voice dictates where we are sitting and will draw attention from those around us. This is referred to as sound localization.

With 3D audio we can now subtly grab a user’s attention towards specific areas within an experience by creating ‘cues’ and ‘prompts’ which subconsciously points the user into following the laid out journey in the direction intended. This 3D audio-led narrative is already becoming the best friend of storytellers in VR.

Because binaural headphones and some HMD’s (head mounted displays) have a 360-degree field of hearing (FOH), as users move around, sound follows, helping to place you within the context of the experience.

Why is Spatial Sound Critical to Mixed Reality?

Often an afterthought by many developers, spatial sound is essential for providing a believable immersive experience. To bring 3D holograms to life in our physical environments, they must have a realistic sound aspect anchored to them to convince our brain that what we are experiencing is real. When users detect a mismatch between their senses, the optical illusion starts to break down and creates disbelief.

Take for example the above dinosaur hologram, the sound of its footsteps, roar and all of the environmental noise must be placed accurately and in time with the visual elements, almost as if they are a part of the hologram to make this an exceptionally believable scenario.

The movement of sound throughout a ‘spatial’ scene will need to be as close to real as possible so the user can barely tell the difference between virtual and real. The skill of great design and expert mixing will be a valuable commodity in this new 3D medium.

Inevitable automation of these processes and new AI mixing tools will help to make this style of audio creation more accessible to the masses, further down the road.

Key benefits of Spatial Sound in Mixed Reality:

According to Dave Black, Co-Founder of 3D audio firm Mixed Immersion, below are some key benefits spatial sound will bring to Mixed Reality in the near future.

• The introduction of spatial audio into a Mixed Reality environment will provide the user with life-like perspective, depth and atmosphere to direct and immerse them within their virtual experience.
• Mixed Reality will be a massive beneficiary of such advancements in spatial audio, due to its ability to convince us that environments we experience are bigger or smaller than they actually are, by building up layers of atmosphere and ambiance, which we are already constantly processing through our subconscious. The timing between these layers is what gives us spatial ‘cues’ and information.
• The reflections of sound, otherwise known as ‘reverberation’, are what cause echoes and are clearly recognizable to us in many cases, like when a church responds to the organ or how a footstep squeaks on a basketball court. All of these effects can be replicated within spatial audio environments and the immense impact that they have in Mixed Reality will result in almost lifelike levels of presence and realism.
• Spatial, 3D audio is already deepening our sense of presence in Virtual Reality and we believe that the addition of this type of atmospheric, responsive sound in a Mixed Reality environment will be the change we need to experience truly reflective, holographic experiences.
• When planning innovative and immersive experiences using AR, VR and Mixed Reality, it’s always important to remember how critical it is to develop audio and visuals at a 50/50 ratio.

The shift from stereo to 3D audio will be monumental for music, film, entertainment and media consumption, in a much bigger way than the previous super-sessions of mono to stereo and stereo to surround sound.

Spatial audio is a way of mixing and delivering layers into this new 3D world of sound; and the best experiences will still need the craft and skill of amazing sound designers, engineers, musicians and producers to create truly immersive, virtual audio worlds.