Blog homepage

SpectralMultiEffect by zerocrossing

Interactive Audio / Sound Design

SpectralMultiEffect is a plug-in for Wwise. It was created with the intent of enhancing interactivity in games, and of providing a rich source of experimentation and audio transformation to sound designers. Two major features are provided: A variety of effects to modify the audio signal, and a system to enable access to information from the audio data itself for direct use in the game.

SpectralMultiEffect offers many ways to transform the signal through processing applied to its frequency spectrum. Up to three effects can be set up in series, and all available parameters can be controlled at runtime through the Wwise RTPC feature. This includes the ability to switch effects on or off, or to swap them at runtime. 

I have noticed sound designers are interested in finding ways of enhancing interactions between sound content and the game. For this plug-in project, I wanted to provide the ability to use data obtained from the audio signal to directly influence or control aspects of the game's behavior. During playback, time or frequency domain data obtained from the signal can be made available to the application for any desired use. This can be useful in establishing a close connection between the game and the audio content.

A set of callbacks is provided, which developers can use to obtain data from the audio in multiple formats in real-time. As plug-ins can be attached to channels or sounds directly, it is also possible to set up multiple streams of data, through separate instances of the plug-in, each with its own callbacks.

Whereas audio plug-ins for games are often created to solve a specific and well-known technical problem, SpectralMultiEffect can be thought of as a creative tool. I am hoping it will help encourage an exploratory approach to working with audio which can lead to finding interesting sounds, creating an engaging and interactive experience for your game.

The following features are available:

1. Eleven different effects applied in the frequency domain, including pitch changes, multi-band delay, and spectrum freezing, among others.

2. Obtaining information from time or frequency domain signal data, to be sent to the application, enabling the possibility of synchronization between audio and other game elements.

3. Audio visualization in the Wwise Authoring Tool while monitoring the application.

4. Presets to demonstrate possible sound transformations. These can be used as a starting point for your own.

Using SpectralMultiEffect

Audio Effects

Three slots are available for effects to be applied in series. They can be selected and switched on or off at any time during the plug-in's operation. Additionally, it is possible to perform pitch-shifting within a range of -/+ 3 octaves without affecting sound duration.

effects
Effect selection area in the Wwise Authoring Tool

While the plug-in is active, incoming audio data is converted to the frequency domain. Processing is done at this level, before converting back to the time domain for playback. 

Once you have selected effects for use, you can control their parameters using the RTPC system. There are many available, but you only need to control those involved in the effects that you have enabled.

Obtaining audio signal data through plug-in callbacks.

User-defined callbacks are available to prepare and obtain data from the audio signal.

Options include peaks or RMS in the time domain, and frequency bins or bands in the frequency domain. Data can be sent to the user either before or after applying effects processing.

To obtain data from the audio signal for your application:

1. Create an instance of the effect in the Wwise project, assigning a unique name tag that will identify it to the application at runtime.

Tag1
Adding a tag to identify a plug-in instance

callbacks
Callback declarations in ZcSpectralMultiEffectFXCallback.h

2. Implement the Initialization, Termination, and Execute callbacks. These are declared in the file: ZcSpectralMultiEffectFXCallback.h. (This file will be copied to your Wwise include directory during the plug-in's installation, so you just need to add an #include directive to use it in your source code.)

The initialization callback is used to configure the preferred data format. This is done by filling in the passed in ZcSpectralMultiEffectDataFormat structure. 

During playback, the Execute callback will provide the application with data from the signal in the format that was requested. The identifying tag set by the user will also be received, to facilitate managing multiple plug-in instances of the plug-in.

The termination callback notifies that a particular plug-in instance's operation is ending.  Any clean-up work that you may need to do to match actions taken during initialization can be done here.

Authoring Tool Monitoring

When monitoring your application in the Wwise Authoring Tool, you can visualize signal data in the plug-in's Effect Editor window. This can be done with either time or frequency domain data, as can be seen in the screenshots below.

peak
Time Domain - Peaks
rms
Time Domain - RMS
bins
Frequency Domain - Bins
bands
Frequency Domain - Bands

Presets

A few presets are included, which can be selected in the Authoring Tool, from the drop-down menu on the bottom right of the Effect Editor window. They show examples of settings that may be interesting to you. You can generate factory presets based on them, or create your own, and use them as you would for any other plug-in.

Supported Platforms

SpectralMultiEffect is initially available on mobile (Android / iOS) and desktop (Mac / Windows) platforms. I am considering expanding support, including game consoles. If you would like to see SpectralMultiEffect available on your project's platform please contact me through the website, and let me know. Based on interest from users I will consider platforms to be added soon.

Subscribe

Audio Samples

Here are three sets of samples demonstrating audio processing with SpectralMultiEffect.

Spoken Voice File

This file begins with an unprocessed voice, listen as it gradually transforms using different combinations of effects in SpectralMultiEffect.

Piano Sound Files

The first audio sample is the original unprocessed. Each following sample is processed in different ways with  SpectralMultiEffect.

Gong Sound Files

The first audio sample is the original unprocessed. Each following sample is processed in different ways with  SpectralMultiEffect.

 

Javier Arciniegas

CEO zerocrossing

zerocrossing

Javier Arciniegas

CEO zerocrossing

zerocrossing

Javier Arciniegas has worked on audio programming in the game industry in Canada and Japan for sixteen years. He has two Masters' degrees, in music composition and electronic music and recording media from Mills College in the United States. He is based in Tokyo, Japan. zerocrossing Co., Ltd. offers audio programming services, including sound engine and plug-in development, integration, customization, and other related work.

Comments

Leave a Reply

Your email address will not be published.

More articles

A Day Without Sound

As a lifelong music lover and someone who has worked with sound for nearly twenty years, it occurred...

12.12.2017 - By Dr. Karen Collins

Building a Career in Sound Today: Film, Games and Shades of Gray

On one side there’s sound for films- linear, on the other there’s sound for video games-...

9.7.2019 - By David Grinbaum

Music for Games Should be More than Just Music: Part 2

What is video game music? What is interactive music? The answers to these questions are not as...

20.11.2020 - By Olivier Derivière

Interactive Music: A 'Choose Your Own Adventure' Style Ballad

After a successful crowdfunding campaign in 2018, inXile Entertainment began full production of the...

28.5.2021 - By Alexander Brandon

Tell Me Why | Audio Diary Part 1: Ambience and Voiceover

"Tell Me Why" is a single-player narrative adventure game developed by DONTNOD, published by Xbox...

10.6.2021 - By Louis Martin

Tell Me Why | Audio Diary Part 4: Mixing and Mastering

"Tell me Why" was released on Xbox and PC and it fully supports 5.1 surround sound. Working on a...

1.7.2021 - By Mathieu Fiorentini

More articles

A Day Without Sound

As a lifelong music lover and someone who has worked with sound for nearly twenty years, it occurred...

Building a Career in Sound Today: Film, Games and Shades of Gray

On one side there’s sound for films- linear, on the other there’s sound for video games-...

Music for Games Should be More than Just Music: Part 2

What is video game music? What is interactive music? The answers to these questions are not as...