Blog homepage

Why VR will put Game Audio in the Spotlight

Game Audio / Spatial Audio / VR Experiences

sense of presence in VR

 

The sense of Presence

The word that is on everyone’s lips when describing VR (Virtual Reality) is  “immersion”.  A cool word indeed. To achieve this ultimate goal however, developers need to simulate the perception of being physically present in a virtual environment, and we call this the sense of presence“Presence is a psychological state or subjective perception in which even though part or all of an individual's current experience is generated by and/or filtered through human-made technology, part or all of the individual's perception fails to accurately acknowledge the role of the technology in the experience.”  (The International Society of Presence Research - ISPR)

Creating this level of true immersion is a real challenge. While graphics already made tremendous improvements in the past few years, audio still has a long way to go in regards to reaching its true potential of playing a major role in this new and still evolving field of VR.

 

Spatial immersion and multi-sensory uses in VR gaming

In console and PC games, immersion can be separated into three categories: narrative immersion (players being invested in a story), tactical immersion (players performing tactile operations that involve skills), and strategic immersion (more cerebral, associated with mental challenges).  In VR however, a fourth category is added: spatial immersion, which convinces the player the artificial world surrounding them is real.  Among the former three categories, spatial immersion is also of great importance when it comes to triggering the sense of presence and delivering a believable and immersive environment.

To position ourselves physiologically, our visual system is used to help guide our bodies within our environment, our ears provide crucial information about our surroundings, and smell provides complimentary surrounding information.

Audio goes beyond vision in the sense that it is not limited to our field of view, but rather provides a 3D-audio feedback (including 360-degree, height and depth) information about our surrounding environment.

VR immersion via audio

In order to create realistic spatial immersion, ideally, we would want to be able to use all of our five senses and have full sensory processing (process of using our bodies effectively within an environment). The problem at the current stage in VR development is that we are unable to utilize all five senses. Sight and hearing are used as the primary senses, touch is not used very effectively, and taste and smell are not integrated at all at this point. With that in mind, audio is perhaps the most powerful medium we can develop in order to “hack” the user’s brain and create a complete 3D-audio experience that offers spatial immersion, as it gives the representation of a realistic space by providing virtual sources (which can’t be necessarily seen in the scene or action) rendered through a simulated acoustic environment.

 

3D-audio experience to create spatial immersion

The first step to enhancing spatial immersion through audio realism is audio positioning. A pattern emerging from recent studies (Skalski & Whitbred, 2010) showed that better sound positioning in games (surround VS stereo) had a much higher impact on the player’s spatial immersion and enjoyment than visual quality (HD VS standard).

The brain uses subtle differences in loudness, tone and timing between our two ears to localize sound sources in three-dimensional positions (horizontal, vertical, and distance). In VR, viewers are literally in the middle of a scene, so the process of determining the location of sound sources in a simulated acoustic environment is the most important audio improvement towards creating natural listening experiences.

 binaural

Early in media history, we discovered that the human ears pick up audio in three dimensions and the first binaural recording goes back to 1881.

“Binaural recording is a method of recording sound that uses two microphones, arranged with the intent to create a 3-D stereo sound sensation for the listener of actually being in the room with the performers or instruments. This effect is often created using a technique known as "dummy head recording", wherein a mannequin head is outfitted with a microphone in each ear.” (Wikipedia, “Binaural Recording”)

In games, 3D-audio should be created with real-time processing since we can’t anticipate where sound objects will be located corresponding to the player's location and head movements.  So, about a century after its invention, binaural recording is now being revisited with binaural processing, and engineers and investors are working hard to develop the best binaural algorithms suitable for VR.

Indeed, because of its nature in regards to the required use of stereo headphones rather than speakers to replicate 3D positioning (necessary in VR to isolate the player from the outside world and have audio playback that matches head/ears movements relative to the field of vision), binaural has been logically chosen by the professionals as the new standard for 3D-audio processing instead of the usual surround techniques (limited to horizontal and distance audio positioning).

 

Conclusion

Many audio challenges, considerations, and needs are being exposed as we work with VR.  By exploring these further, we will be able to push the boundaries of the VR and gaming industry altogether.  And, although the gaming industry has been improving visual quality in the last generation of consoles, with the VR development on a rapid rise, the mindset within the VR industry is quick focusing on audio as the VR community is realizing that better audio realism will increase the player’s enjoyment and engagement levels, elevate the state of immersion, and ultimately, the sense of presence.

3D-audio positioning is just one of the areas where audio improvements are being pursued as a direct result of VR developments.  Audio working tools are also in continuous development with new technologies catering to VR needs and its uniqueness when it comes to the mixing techniques required, in order to enhance VR experiences.

Audio is the next medium for the industry to explore and focus on, and VR is certainly helping put audio the spotlight!

 

 
 
 

Alex Riviere

Audio Director

CCP Games

Alex Riviere

Audio Director

CCP Games

Audio Director with 10+ years of professional experience in audio across multiple industries. Pioneer Audio Director / Audio Lead for VR Games.

 @CCP_alexriviere

Comments

Leave a Reply

Your email address will not be published.

More articles

Peggle Blast! Peg Hits and the Music System

As a primer for their GDC 2015 talk, Peggle Blast! Big Concepts, Small Project, PopCap Team Audio...

30.5.2016 - By RJ Mattingly

Overwatch - Game Audio Using Wwise (part 2/2)

Continued... During Audiokinetic's Wwise Tour, Project Audio Director Scott Lawlor, and Senior...

27.9.2016 - By Audiokinetic

Your Spatial Audio Testing Playground: The Wwise Audio Lab

The Spark With the rise of Virtual Reality, developers started asking for advice on the most...

4.10.2017 - By Simon Ashby

The Wwise Adventure Game 

The Genesis This project started six years ago back in 2012 as an answer to our users requesting a...

14.3.2018 - By Simon Ashby

Recreating the Sounds of a Legendary Universe for VR - Star Trek: Dark Remnant

Dave & Buster’s launched their 3rd experience “Star Trek: Dark Remnant”, at more than 120...

22.10.2019 - By Hexany Audio

Wwise Wworld Wwide Online Expo Recap

The year 2020 arrived in synchronicity with Audiokinetic's 20th anniversary. Today, Wwise is the...

7.5.2020 - By Audiokinetic

More articles

Peggle Blast! Peg Hits and the Music System

As a primer for their GDC 2015 talk, Peggle Blast! Big Concepts, Small Project, PopCap Team Audio...

Overwatch - Game Audio Using Wwise (part 2/2)

Continued... During Audiokinetic's Wwise Tour, Project Audio Director Scott Lawlor, and Senior...

Your Spatial Audio Testing Playground: The Wwise Audio Lab

The Spark With the rise of Virtual Reality, developers started asking for advice on the most...