Introduction To Audio In VR: Opportunities and Challenges

空间音频 / VR 体验

Among the many new technologies on the market, virtual reality is one of the most in demand, not only by consumers but also by the enterprise and the government sectors.

By 2020, the VR industry is expected to reach $120 billion. In the report by Digi Capital, it states that the arrival of high-end VR equipped with powerful CPU and GPU chipsets and premium features will be the initial consumer VR market drivers. Companies, like Samsung with their Gear VR, are expected to make big changes in the industry. The same report mentioned that the Korean tech giant is making great improvements in addressing user demands, such as display and wireless audio output. It is evident that Samsung is focusing on VR with the massive upgrades on the Galaxy S8. As featured by O2, the handsets come with VR-focused features, such as the ‘infinity display,’ a 64-bit octa core processor (4 GB RAM), and Bluetooth 5.0. These technologies promise to deliver a reliable and robust audio connection (four times the range, two times the speed, and eight times the broadcast message capacity) for a more immersive VR experience.

banner-1571999_960_720 (1).jpg

When it comes to virtual reality, it’s common for many users to discuss VR apps based solely on the image content presented to them and less on its aural score. Just how important is 3D audio in virtual reality? The truth is that for VR to be truly immersive, it needs convincing sound to match. Badly implemented audio in VR can be off-putting and can impact user acceptance.

What 3D audio offers is a little map in your brain, Engadget author Mona Lalwani explained, even when you are not viewing objects it will allow you to know where things are even if they are not in your field of view.

“The premise of VR is to create an alternate reality, but without the right audio cues to match the visuals, the brain doesn't buy into the illusion. For the trickery to succeed, the immersive graphics need equally immersive 3D audio that replicates the natural listening experience,” Lalwani stated.

However, there are fundamental problems that need to be addressed, one of which is externalising sounds. In television shows and films, you will notice that the sound of the narrator is different from the people you’re actually seeing on the screen due to the way in which it was recorded (it’s recorded closer than ‘on set’ dialogue with a condenser microphone). The voice of the narrator is the audio inside the viewer’s head, which is what developers need to avoid to achieve a seamless virtual environment for users.

Developers need to ensure that the perspectives are convincing. How you give players information about distance is vital for users to be able to localise something accurately in the virtual reality space. There’s a need to recreate the acoustic behaviour in VR for it to be convincing with a delicate balance between the volume of the sound waves (directly to the ears, bounce around the room, the length of reverberation, the ratio between direct and indirect sound, and perceived loudness).

Another way to recreate a natural listening experience is by making a binaural recording that creates a clear distinction between left and right sound. It is an important element to  successful 3D audio as it helps the brain of the user to pinpoint the exact source of the sound. However, it may not apply to all directions as the sounds coming from the back and the front are more ambiguous. A response called ‘Head-Related Transfer Function’ (HRTF) is created when the sound from the front interacts with the outer ears, neck, shoulders, and head. This audio gets coloured with modification that assists the brain in solving confusion. Overall, binaural recording embodies the core of a personalised immersive audio.

Developers also need to consider the limitation of a human brain. Audio’s power is to give users information or to influence their emotional state, but there’s a limit to the amount of auditory information that they can process. Film editor Walter Murch said it needs to apply the ‘Law of Two and a Half,’ wherein a human can easily isolate two sets of sounds, but the third sound takes out the ability of the brain to distinguish individual elements. The sound needs to be light and shaded when presented to users. However, more than two positional cues at a time can still be effective if there’s a need to briefly disorient the player on purpose.

Although this article may hold a possible solution, the truth is that the audio for VR is still a work in progress. But, the combination of 3D audio and head-tracking seems to compliment and make virtual reality complete.

"Audio, from an evolutionary perspective, is the thing that makes you turn your head quickly when you hear a twig snap behind you," said Joel Susal, director of Dolby’s AR and VR business. "It's very common that people put on the headset and don't even realise they can look around. You need techniques to nudge people to look where you want them to look, and sound is the thing that has nudged us as humans as we've evolved."

TechJVB

Blogger

Freelance

TechJVB

Blogger

Freelance

TechJVB is certified audiophile and gaming expert with expertise in AR, VR, AI and the likes. She has attended several tech conferences in Europe and Asia. She has been invited to be a guest speaker at different schools in Manchester to inspire young minds to enter STEM fields and explore the bigger potential and future of technology.

评论

Andrew Menino

June 25, 2017 at 01:38 pm

Great article! Looking forward to investigating the "Law of Two and a Half"! Thanks a lot

留下回复

您的电子邮件地址将不会被公布。

更多文章

反馈的乐趣

Wwise 2017.1中引入了3D 总线和总线辅助发送,这让模块化地使用声音引擎成为了可能,也让Wwise做到了之前从未想过的事情。我准备带你们跳出使用Wwise创作游戏音频的惯例...

23.11.2017 - 作者:内森 哈里斯(NATHAN HARRIS)

《Blade Runner: Revelations》

《Blade Runner: Revelations》是 Seismic Games 和 Alcon Media Group 最近发布的一款以《Blade Runner》系列经典影片为蓝本的 VR...

25.9.2018 - 作者:_HEXANY AUDIO_

玩家有话说:电子游戏音乐必须是自适应的

电子游戏音乐本身就与线性媒体有所不同。另外,线性和非线性音乐的作曲要求也很不一样。在霍华德·肖 (Howard Shore) 为《魔戒》系列电影创作配乐时,他事先知道哪些角色会在银幕上出现 20 秒或...

1.7.2019 - 作者:Valerio Velardo 博士

《使命召唤手游》的空间音频探索

当我们提起“空间音频”的时候,我们在聊什么?

15.6.2020 - 作者:腾讯天美工作室群

Impacter 与 Unreal | 利用游戏物理控制 Impacter 插件

简介 Impacter 是最近针对 Wwise 开发的一款撞击建模插件原型(详可参阅这篇博文)。在本文中,我将介绍如何使用 Impacter 来实现声音并整合到采用 Unreal Engine...

14.9.2021 - 作者:肖恩·索拉汉(SEAN SORAGHAN)

为何要选用 Wwise 来打造 3D 互动音乐体验

Dirty Laundry by Blake Ruby 移动 VR 应用:音频设计详解。

26.10.2021 - 作者:朱利安•梅西纳 (Julian Messina)

更多文章

反馈的乐趣

Wwise 2017.1中引入了3D 总线和总线辅助发送,这让模块化地使用声音引擎成为了可能,也让Wwise做到了之前从未想过的事情。我准备带你们跳出使用Wwise创作游戏音频的惯例...

《Blade Runner: Revelations》

《Blade Runner: Revelations》是 Seismic Games 和 Alcon Media Group 最近发布的一款以《Blade Runner》系列经典影片为蓝本的 VR...

玩家有话说:电子游戏音乐必须是自适应的

电子游戏音乐本身就与线性媒体有所不同。另外,线性和非线性音乐的作曲要求也很不一样。在霍华德·肖 (Howard Shore) 为《魔戒》系列电影创作配乐时,他事先知道哪些角色会在银幕上出现 20 秒或...