Lumote: Using Wwise to Drive In-Game Visual FX

게임 오디오 / 사운드 디자인

Hello to everyone out there in Wwise user land! My name is Paul Ruskay and I am the Audio Director at Studio X Labs in Vancouver, Canada. In this blog, I am going to explain how, in our upcoming Indie 3D Puzzle Platformer title: Lumote (PC, Switch, PS4, XBox One), I used the Wwise Audio Metering plugin to drive in-game VFX, such as particles, lighting, camera shaking and mesh warping. Due to Lumote being an indie title, I was given a great amount of time and freedom to experiment with the Metering plugin and to learn how to use it in the context of the game’s world and narrative. Since I was designing sound elements to specifically drive visual effects, I found many opportunities to create unique visual and auditory moments with this setup.

lumote2

Lumote is a colourful and beautiful 3D-Puzzle-Platformer slated for release on February 20, 2020 on Steam . It will be available on Switch, XBox One and PS4 later in 2020.

Overview

The process of creating game audio typically has the sounds being created after an animator or VFX artist has completed their work. Our task, as sound designers, is to create audio that will match to these finalized animations and visual effects. For Lumote’s development, I used the Wwise Metering plugin to “harvest” the sound design’s waveforms, in order to generate a RTPC value string that was fed back into the game engine, to drive various visual effects and animation systems. With a team of four devs and no animators, we needed to adapt our approach and we soon realized that a lot of visual interest could be generated in a procedural manner, by having the audio drive many of the game’s VFX systems.  This approach to audio-driven VFX has led to a game that feels very organic and fluid. I have an extensive portfolio of game audio production and this approach of driving visual effects from waveforms was new to me and a lot of fun.

The topics for this Blog are as follows:

  1. Description of Lumote
  2. Development Team
  3. Lumote’s Audio Design
  4. Metering Plugin Setup
  5. Examples from Lumote of audio driven VFX

About Lumote

Lumote is a game about two opposing powers. Blue power is controlled by Lumote and Red power is controlled by the Mastermote, the antagonist in our game. The player navigates Lumote through this 3D-Puzzle Platformer, in which the player has only two abilities: the ability to possess creatures and the ability to jump. Possession allows the Lumote to take control of the world's inhabitants, (the “Motes”) and to use their unique abilities to solve increasingly difficult puzzles. Depending on which energy controls a Mote, their behaviour will either help (with Blue Energy) or hinder (with Red Energy) Lumote’s goal of reaching the bottom of the world map and take control away from the Mastermote.

lumote3

Lumote is a 3D-Puzzle Platformer that will take the player on a 6-hour journey to the bottom of the world map. It was developed in a custom game engine (rEngine), coded for this project by members of the Lumote Dev Team (Kyle Rocha and Aaron Arnt)

The world of Lumote is broken into three biomes, each comprising three towers that need to be converted from Red Energy to Blue Energy before Lumote can proceed to the next series of puzzles. Each tower represents a Mote type and a specific game mechanic, that you must master by completing the tower. There are five Mote types in the game (Small Dumbats, Large Dumbats, Hover Movers, Jellyfish and Divertors). Each Mote has its own special ability that Lumote must master in order to reach the end of the game.

 

Lumote4

Early concept art for Motes. Some great character design that works beautifully in the context of the puzzle design. 

 

Development Team:

Lumote5

From left to right:
Michelle Rocha , Puzzle Design & Code             
Kyle Rocha, Code & Art   
Aaron Arnt , Code & Rendering          
Paul Ruskay, Music, Sound Design & Wwise Integration                


We all worked remotely, so our development environment was Perforce, Slack, rEngine and Wwise. We kept meetings to a minimum and used Slack as the primary means of communication throughout the project. Everyone worked within their respective silo, while easily being able to Slack someone when needed. Looking back, it's a great way to make a game, which provided me with the time and creative freedom to experiment with audio-driven VFX.
Luminawesome Games is an indie studio, located in Vancouver, Canada, and is made up of veterans from the video game industry. Every team member has worked on and shipped AAA titles, so there was a lot of development experience applied to Lumote. In October 2014, Kyle and Michelle were one of the winning teams in an Unreal Game Jam, with their mini game, Bump. They loved the concept and decided to develop it into a full game. Aaron joined in 2015 and it was decided then to move away from the Unreal Engine. Over the course of 2015, Kyle and Aaron coded a custom graphics engine, called rEngine, specifically for Lumote. I joined the project in March 2016 and was charge of music composition, sound design and Wwise integration.

Lumote’s Audio Design

The main goal of the sound design for this game was to create a world that would feel alive and expressive, whenever a player interacted with it. Lumote needed to sound and look organic, electronic and otherworldly. We were a small team with no animators, and so we needed to optimize our resources. With that in mind, we quickly realized that many of the Mote’s animations could be generated in a procedural manner, from their respective sound design. By using the Metering plugin, we were able to create an RTPC value string from the game object’s decibel meter, in its associated audio folder in Wwise. Kyle programmed the VFX in rEngine, which would take the RTPC output from Wwise and send it to lighting, particle emissions, camera shaking and mesh warping systems. So Lumote’s visual effects became literal translations of the waveforms being triggered from the game objects Wwise folder. That meant I would need to think about audio in a visual way. Below is a breakdown of what I came up with:

Lumote6

A constant pulsing sound applied to lighting would generate a strobe effect. The same sound applied to particles would create a constant stream. It was interesting to have to think about how the audio wave would translate visually as an animation, especially within the context of character responses.

 

 

The most time consuming task was developing the five varieties of Motes, as all five needed to communicate both their character State and their reactions to Lumote. The other challenging part of Lumote’s audio design was having to create two different sound sets for each energy type, per Mote (Red and Blue). At any given time in the game, the environment and the Motes could carry either Red Energy (Mastermote possessed) or Blue Energy (Lumote possessed) and as a result, each energy type needed to have a its own special audio treatment. This resulted in having two sound design versions for every game interaction, based on energy type. I had to effectively sound design the game twice to make all this work. Also, the Mote's sound design needed to blend harmonically with the music, as many of the Mote's voices were generated from samples of musical instruments (flutes, bells, whistles, etc). The musical style that best suited the environment was ambient, as I needed to create music that felt “underwater” and “otherworldly”. The ambient music would also not distract the player, who would need to concentrate to try and solve the spatial 3D puzzles. There are also no spoken words in Lumote’s character reactions, only vocalizations. This meant that no language localization recordings were needed, but it also meant the game’s narrative would be non-verbal and would have to be told through audio design and music alone. 

My inspiration would come from games like Inside, Limbo and Ori and The Blind Forest, that all have great “non-verbal” narratives that sustain interest, build tension and create convincing gameplay moments that communicate the emotional state of the main character.

Even though Lumote’s core gameplay loop of solving puzzles has a very simple sequence of audio Events, I discovered that there was an amazing amount of subtlety and variation that could be created within the game, for example, as Lumote possessed and changed a Mote’s energy State.  

Mote’s Core Audio Event Loop:
Idle – Hello – Goodbye (Blue or Red)
Idle – Hello – Possession Start – Energy Change - Move Start – Move End – Possession End – Idle Start
 
Lumote Core Audio Event Loop:
Walk
Jump
Double Jump
Land (RTPC driven based on falling distance)
Possession Start
Possession End
Mote Hello Reaction

 

Lumote has different reactions for each Mote type, based on the Energy State. It is apprehensive when greeting Red Motes and pleasantly cordial when meeting a Blue Mote.

 

Metering Plugin Setup and Wwise Setup

My goal was to create a Wwise project that would allow me to have maximum control over all game audio assets, with a minimal amount of programming support. To accomplish this, Kyle and I “paper-designed” the audio Event layer, based on game object groupings that shared common Events. This way we could have 80% of the audio Events being passed to Wwise before I even started the audio production. Our Switch categories and Common Events were:

Lumote7

With rEngine passing audio Events to Wwise, I set out to start building my folder structures and populating them with audio assets. By getting 80% of the Events passed from rEngine at the start of the sound development process, I was able to work independently and could create and test all of the audio elements with very little programming support. With a Dev team composed of only four people, it was obvious everyone needed to be as independent as possible.

 

lumote8

Kyle created a series of OutputMeter presets that I could assign to game objects to drive their lighting, animation and particles. The OutputMeter_Voice was used 90% of the time, because that was the preset that linked the RTPC from Wwise to the Mote's visual systems. 

AkGameObject, AkSoundEngine.PostEvent:

This is how we passed the RTPC from Wwise to the game’s engine:

1. We set the Mote Type Switch for each character in the game.

2. We then set PostEvent with a common Event like Common_OnMoveBegin and Common_OnMoveEnd.

3. We then used AK::SoundEngine::Query::GetRTPCValue() to read back the metering parameters from Wwise, based on the audio folder’s decibel meter. 

4. We post-processed those metering parameters in the game to modify the amplitude and frequency of a Sine wave, which is what drives the actual lighting, mesh animations, camera shaking and emissive material parameters.

 

What I found working with Metering Plugin:

1. We used the Metering plugin to reinforce Motes States and created creature voices for a range of States (idle, surprised, stop, start, power change).

2. The Metering plugin worked great as long as there was no “hangover” signal from the game object’s previous audio Event. I needed to spend time working in the Event layer to make sure a game object’s sound design had completely stopped before the next SFX was triggered. Any audio overhang from a previous audio Event would takeover the Metering plugin and not pass the new audio RTPC to the game engine. I also found that using Pause Events on a folder would also takeover the Metering plugin. It was best to stop playing a folder completely before triggering the next audio Event, so the Metering plugin remained available for the new audio signal.

3. Adding verb FX on Mote subfolders allowed for both audio and visual blending between different Mote States. It helped blend the separate sound folders into a cohesive whole.

4. Linking Impacts to the Pitch RTPC (for Impacts) yielded really interesting visual results. The lower pitched versions of the SFX created slower and broader visual effects.

lumote090

Conclusions

1. Lumote's sound design turned into an evolutionary process as I began to play around with the Metering plugin. When we hooked camera shaking to the audio of the Mastermote, it really brought the Tower Sequences to life.

2. Be open to crazy ideas and always pursue the most enjoyable and memorable experience for the player. Also, ask yourself: Is there too much sound in the game? I there enough “negative space” in the audio to allow the player to be comfortable in the game environment? 

3. "Paper design" the Event layer and Switch categories early.

5. Research the goals of creatives in other fields (coding, art, and design), and ask yourself how you can help the people working in those fields achieve their goals with audio?

 

 

Paul Ruskay

Audio Director

Studio X Labs

Paul Ruskay

Audio Director

Studio X Labs

Paul Ruskay has been an Audio Director at Studio X Labs in Vancouver Canada since founding this postproduction studio in 1999. He has produced audio for over 40 game titles during the past 26 years. He is best known for his music on Homeworld and Strike Suit Zero.

www.studioxlabs.com

댓글

Elise Baldwin

February 05, 2020 at 07:17 pm

Wonderful article and multi-disciplinary creative use of Wwise data. Super cool!

댓글 달기

이메일 주소는 공개되지 않습니다.

다른 글

이미지 기반 파라미터를 이용한 오픈 월드 앰비언트 디자인

Blend Container는 강력한 시퀀싱 및 믹싱 도구가 될 수 있습니다. 단순히 그것의 기능을 배우는 것이 게임 사운드 디자이너의 생각에 온갖 종류의 새로운 아이디어를...

13.3.2020 - 작성자: 톰 토디아 (TOM TODIA)

UI 설계 관점에서 UI 오디오 접근하기 - 제 2부

이 글에서는 UI 디자이너*의 관점에서 UI 오디오에 접근하는 방식을 보여드리려고 합니다. 이를 통해 심미적으로나 기능적으로 화합적인 UI를 제작하는 데에 도움이 되었으면 합니다....

20.10.2020 - 작성자: 조셉 마척(JOSEPH MARCHUK)

가상 음향을 통해 소리 풍경 가청화하기

이 시리즈에서는 과거, 현재, 그리고 새로운 잔향 기술을 집중적으로 살펴보고 몰입적 공간적 관점에서 이 기술을 검토해봅니다. 이전 글에서는 가상 현실에서 몰입적인 잔향을 제작하는...

16.3.2021 - 작성자: 브누아 알라리 (BENOIT ALARY)

누구나 사용할 수 있는 WAAPI - 제 1부: 개요

안녕하세요. 저는 왕양 (汪洋) 이라고 합니다 (혹은 ‘씨 예’, 溪夜라고도 불립니다). 저는 작년 하반기에 WAAPI에 대해 알게 되었습니다 (Wwise 저작 API). 저같이...

30.3.2021 - 작성자: 토마스 왕 (THOMAS WANG, 汪洋)

게임 사운드 보관 | 제 2부: '컨커 최악의 날'과 미스터리한 MP3

오늘 소개할 이야기는 뜻밖의 결과, 다시 말해 우연한 발견에 관한 이야기입니다. 연구 프로젝트가 의도한 대로 끝나지 않았기 때문에 복잡하게 느껴지실 수도 있지만 끝까지...

29.9.2021 - 작성자: 파니 러비야르 (Fanny REBILLARD)

게임 음악은 단순히 그냥 음악이 아니다: 제 2부

게임 음악이란 무엇일까요? 상호작용 음악이란 무엇일까요? 이 질문에 답하기란 생각만큼 그리 간단하지 않습니다. 올리비에 더리비에르(Olivier Derivière)는 이 글을 통해...

27.10.2021 - 작성자: 올리비에 더리비에르 (OLIVIER DERIVIÈRE)

다른 글

이미지 기반 파라미터를 이용한 오픈 월드 앰비언트 디자인

Blend Container는 강력한 시퀀싱 및 믹싱 도구가 될 수 있습니다. 단순히 그것의 기능을 배우는 것이 게임 사운드 디자이너의 생각에 온갖 종류의 새로운 아이디어를...

UI 설계 관점에서 UI 오디오 접근하기 - 제 2부

이 글에서는 UI 디자이너*의 관점에서 UI 오디오에 접근하는 방식을 보여드리려고 합니다. 이를 통해 심미적으로나 기능적으로 화합적인 UI를 제작하는 데에 도움이 되었으면 합니다....

가상 음향을 통해 소리 풍경 가청화하기

이 시리즈에서는 과거, 현재, 그리고 새로운 잔향 기술을 집중적으로 살펴보고 몰입적 공간적 관점에서 이 기술을 검토해봅니다. 이전 글에서는 가상 현실에서 몰입적인 잔향을 제작하는...