PixelSound | An Audio Toy

상호작용 오디오 / 상호작용 음악

Whenever I start a new creative project, I always get a new toy - some cool plugin, or musical instrument, or spiffy microphone - something to just play around with for no other reason than to have fun. Making time for play encourages “childlike discovery” which is when good ideas happen.


"PixelSound"

"PixelSound" is an audio toy that makes a sound based on the color of the pixel your mouse is pointing at. It's a Unity/Wwise project (partly inspired by the excellent program PhotoSounder) that loads up a jpeg, then sets RTPCs to red, green, blue, and grayscale values including x/y screen position.

It's also a way to use Wwise as a sound design tool. Need to create some related variations for a random container? Set up a mix of assorted stems; let it run while recording the output; cut up the results and pick the ones you like. It's a common enough technique ...

Using the color values of pixels to set Wwise event parameters is just a way to generate pseudo-random variations. You can set up patterns to trigger specific effects, or load up a photo of your dog just to see what happens. This kind of "seemingly but not really completely by chance" algorithm can be considered "stochastic", a composition method famously utilized by Xenakis to produce interesting tonalities.

To play, fire up the Unity project, and click on the screen. Move the mouse around to hear how the event reacts to the colors and screen position. Click again to stop.

Download the project here.

You can drop your own .jpg files into the Assets/Pictures folder, then drag them into the "Picture" field of the GetPixelParameters script attached to the "pictureSprite" game object. This formats and resizes the picture to the screen, starts/stops the selected Wwise event, and sets the RTPCs.

The Wwise project contains some demonstration and test events that modify volume, pitch, trigger rates, EQ, and other parameters. They produce a variety of noises you can play with -- but of course, the whole point is to have fun messing around with your OWN sounds and pictures. There's also a Wwise Recorder instance on the Master Audio Bus, in case you want to record the output for later editing.

PixelSound

**Project update - "VPixelSound"**

VPixelSound is basically the same idea as PixelSound, except it uses video instead of jpegs. You drop in a .mov and select a position (center screen by default). RTPCs are set to red/green/blue/greyscale values as before, but I added some State processing as well, whereby color values set Wwise states on a ten step scale (i.e. blue_00, blue_01 ... blue_10).

Using this version, Wwise events are controlled by moving picture, rather than by moving mouse. The series of colors that generate the audio is also repeatable, in case you wanted to craft an event for a specific sequence of images. Multiple cursor positions and color averaging options are available as well.

Most videos will provide interesting input for sonic experimentation. I used QuickTime Screen Recording to create the included samples, but any capture method will do. Note: no soundtrack needed, and any video format Unity supports should work.

Remember, this project is an audio toy; the point is to try out your own videos and Wwise events to see what noises you can produce. Is it useful as a Serious Sound Design Tool? <shrug> All I know is it sure is fun to play with! (and isn't that an important aspect of any creative work?)

Video demo:


The "VPixelSound" project can be downloaded here.

I can possibly see this technique being useful in a game:

  • Modify the ambience based on the color of a procedurally generated sky
  • Change filter settings based on the greyscale value of a scene
  • Randomize UI/selection sounds based on screen position

Granted, it's a whimsical technique that could maybe support creative audio solutions. But mostly, it's just supposed to be fun to play around with, and experiment with the powerful features available to sound designers when using Wwise as a synthesizer.

What does blue sound like to you today?

Peter "pdx" Drescher

Peter "pdx" Drescher

Peter "pdx" Drescher produces audio content, implementation, and code for games and mobile devices. He is a talented sound designer, accomplished musician/composer, published author, experienced programmer, and skilled audio engineer. His ProTools project studio "Twittering Machine" is located in the SF Bay area.

www.twittering.com/

댓글

댓글 달기

이메일 주소는 공개되지 않습니다.

다른 글

라우드니스를 처리하는 최상의 방법, 제 1강: 라우드니스 측정 (1부)

방송과 같은 푸시 서비스, 멀티 플랫폼 상호작용 엔터테인먼트, 인터넷 스트리밍과 같은 다양한 미디어 형식에서 라우드니스는 단지 신호 전달 표준이 아니라 소비자 선호에 직접적인...

12.11.2019 - 작성자: 지에 양 (Jie Yang, 디지몽크)

새로운 Impacter 플러그인 알아보기

개요 Impacter(임팩터)는 기존의 SoundSeed Impact 플러그인을 영감으로 하는 새로운 음원 플러그인입니다. 이 플러그인은 '타격음' 사운드 파일을 저작 도구로...

20.5.2021 - 작성자: 라이언 돈 (RYAN DONE)

Wwise 2021.1에서 시도해볼 10 가지

Wwise 런처에서 Wwise 2021.1을 다운로드할 수 있게 되었다는 소식입니다. 오브젝트 기반 파이프라인, 방사 이미터(radial emitter), WAQL을 포함한 다양한...

9.6.2021 - 작성자: 매스 마라티 소노로 (MADS MARETTY SØNDERUP)

게임 음악은 단순히 그냥 음악이 아니다: 제 1부

게임 음악이란 무엇일까요? 상호작용 음악이란 무엇일까요? 이 질문에 답하기란 생각만큼 그리 간단하지 않습니다. 올리비에 더리비에르(Olivier Derivière)는 이 글을 통해...

20.10.2021 - 작성자: 올리비에 더리비에르 (OLIVIER DERIVIÈRE)

NFL 킥오프 2020: 텅 빈 경기장에 관중 사운드 시스템 도입

실제 팀과 경기장별 오디오 파일을 사용하는 동적 시스템 이 글은 Sports Video Group News(스포츠 비디오 그룹 뉴스)에 게시된 원본 글을 가져온 것입니다. 이번...

1.12.2021 - 작성자: 댄 대일리 (Dan Daley)

‘잇 테이크 투(It Takes Two)’ 사운드 비하인드 스토리 | Hazelight 오디오 팀과의 Q&A

Hazelight Studios(헤이즈라이트 스튜디오)에서 제작한 잇 테이크 투(It Takes Two)는 분할 스크린 액션 어드벤처 플랫폼 협동 게임입니다. 이 게임은 엄청나게...

5.4.2022 - 작성자: Hazelight (헤이즐라이트)

다른 글

라우드니스를 처리하는 최상의 방법, 제 1강: 라우드니스 측정 (1부)

방송과 같은 푸시 서비스, 멀티 플랫폼 상호작용 엔터테인먼트, 인터넷 스트리밍과 같은 다양한 미디어 형식에서 라우드니스는 단지 신호 전달 표준이 아니라 소비자 선호에 직접적인...

새로운 Impacter 플러그인 알아보기

개요 Impacter(임팩터)는 기존의 SoundSeed Impact 플러그인을 영감으로 하는 새로운 음원 플러그인입니다. 이 플러그인은 '타격음' 사운드 파일을 저작 도구로...

Wwise 2021.1에서 시도해볼 10 가지

Wwise 런처에서 Wwise 2021.1을 다운로드할 수 있게 되었다는 소식입니다. 오브젝트 기반 파이프라인, 방사 이미터(radial emitter), WAQL을 포함한 다양한...