Blog homepage

PixelSound | An Audio Toy

Interactive Audio / Interactive Music

Whenever I start a new creative project, I always get a new toy - some cool plugin, or musical instrument, or spiffy microphone - something to just play around with for no other reason than to have fun. Making time for play encourages “childlike discovery” which is when good ideas happen.

"PixelSound" is an audio toy that makes a sound based on the color of the pixel your mouse is pointing at. It's a Unity/Wwise project (partly inspired by the excellent program PhotoSounder) that loads up a jpeg, then sets RTPCs to red, green, blue, and grayscale values including x/y screen position.

It's also a way to use Wwise as a sound design tool. Need to create some related variations for a random container? Set up a mix of assorted stems; let it run while recording the output; cut up the results and pick the ones you like. It's a common enough technique ...

Using the color values of pixels to set Wwise event parameters is just a way to generate pseudo-random variations. You can set up patterns to trigger specific effects, or load up a photo of your dog just to see what happens. This kind of "seemingly but not really completely by chance" algorithm can be considered "stochastic", a composition method famously utilized by Xenakis to produce interesting tonalities.

To play, fire up the Unity project, and click on the screen. Move the mouse around to hear how the event reacts to the colors and screen position. Click again to stop.

Download the project here.

You can drop your own .jpg files into the Assets/Pictures folder, then drag them into the "Picture" field of the GetPixelParameters script attached to the "pictureSprite" game object. This formats and resizes the picture to the screen, starts/stops the selected Wwise event, and sets the RTPCs.

The Wwise project contains some demonstration and test events that modify volume, pitch, trigger rates, EQ, and other parameters. They produce a variety of noises you can play with -- but of course, the whole point is to have fun messing around with your OWN sounds and pictures. There's also a Wwise Recorder instance on the Master Audio Bus, in case you want to record the output for later editing.

PixelSound

I can possibly see this technique being useful in a game:

  • Modify the ambience based on the color of a procedurally generated sky
  • Change filter settings based on the greyscale value of a scene
  • Randomize UI/selection sounds based on screen position

Granted, it's a whimsical technique that could maybe support creative audio solutions. But mostly, it's just supposed to be fun to play around with, and experiment with the powerful features available to sound designers when using Wwise as a synthesizer.

What does blue sound like to you today?

Peter "pdx" Drescher

Peter "pdx" Drescher

Peter "pdx" Drescher produces audio content, implementation, and code for games and mobile devices. He is a talented sound designer, accomplished musician/composer, published author, experienced programmer, and skilled audio engineer. His ProTools project studio "Twittering Machine" is located in the SF Bay area.

www.twittering.com/

Comments

Leave a Reply

Your email address will not be published.

More articles

A Brief History of Music in Video Games

Music is a key part of all modern video games. You only have to look at Persona 4’s oddball mix of...

25.9.2018 - By TechJVB

Automotive Sound Design: bridging the gap between sound design & automotive engineering, using Wwise

My involvement in automotive sound design began back during my college years while I was working in...

12.2.2019 - By Brett Sceli

Botanical Dimensions

In October of last year, my team and I helped put together an amazing experience for Intel in...

14.5.2019 - By Braeger Moore

The Differences Between Working in Game Audio and Film: Part 1

Have you been working in the film industry and are wondering what it would be like to work in game...

18.6.2019 - By Jesper Ankarfeldt

Music Design & “Project Hiraeth”: Why I Spent a Year Making a Score for a Game that Doesn’t Exist

Project Hiraeth is the world’s first video game score license to be fully adaptive, professionally...

9.7.2019 - By Seth Wright

The Making of Krotos

Igniter Live Synth, the Krotos plug-in for real-world or sci-fi vehicle and engine sound effects,...

3.9.2019 - By Orfeas Boteas

More articles

A Brief History of Music in Video Games

Music is a key part of all modern video games. You only have to look at Persona 4’s oddball mix of...

Automotive Sound Design: bridging the gap between sound design & automotive engineering, using Wwise

My involvement in automotive sound design began back during my college years while I was working in...

Botanical Dimensions

In October of last year, my team and I helped put together an amazing experience for Intel in...