The concept behind Project Echo

"Project Echo" is an immersive audiovisual experience created by sound artist and founder of POSTRED Beso Kacharava. It is an exciting project that aims to take visitors on a journey through the strange corridors of one's subconscious. Your Echo guides you through this experience. Echo is an amorphous and enigmatic companion and may remind you of otherworldly fictional encounters, such as Twin Peaks, Annihilation, or Arrival.

The concept for Project Echo has been around for a long time with Beso Kacharava. His idea was to offer visitors a unique opportunity to encounter and connect with their inner selves. The idea became significant after POSTRED partnered up with UNESCO's designation of Tbilisi as the World Book Capital 2021, Beso explained. “This opportunity was a catalyst that helped me transform the idea into a tangible project. I aimed to employ various art mediums to breathe life into this concept, focusing on creating an intimate space where individuals could connect with themselves.”

Project Echo utilizes a unique structure where visitors encounter a series of generative ‘rounds,’ each featuring a combination of sights and soundscapes—the ‘leads.’ We engineered each lead to resonate with the visitor, potentially evoking emotions, memories, or sensations.

Between each round, the journey loops back to the Echo's chamber - where the Echo, dwelling in the core, poses a question, a ‘riddle’ to the visitor. You select an answer on the tablet, which determines the course of the journey, leading the visitor to the next round. The environment restructures itself into a new scene.

In an unexpected twist at round Eight, the journey abruptly halts, and Echo reappears, this time metamorphosed into a darker entity, a reflection of the abyss within. This shadowy Echo is like a nod to Jung's shadow archetype, Lovecraftian darkness, or the suspense-filled works of Stephen King. Answering Echo's probing question takes the visitor off to round Nine.

Upon reaching this final stage, the visitor undergoes a symbolic rebirth, culminating in a return to the beginning. Having come full circle, the journey resets, and the next visitor is awaiting. This cyclical design encourages multiple visits and guarantees a unique and new experience with each run-through.

Project Echo - Trailer for UNESCO’s World Book Capital 2021 in Tbilisi (Vimeo)

Impact

Since its launch, Project Echo has received an overwhelming response from visitors, with many describing the experience as transformative and emotionally intense. The sheer demand for the experience was evident as slots filled rapidly, leading to a fully booked calendar throughout the exhibition's scheduled timeframe. Such was the flow of visitors that the exhibition had to be extended by an additional month to accommodate the influx of eager attendees.

The duration of each visitor's stay within the installation varied significantly. While some spent a brief ten minutes, others were so engrossed that they lingered for up to forty minutes. To our surprise and quiet delight, a few attendees were so moved by Project Echo that they left with tears in their eyes.

Technical Execution

We wanted it to allow visitors to immerse themselves in the installation's meaning and purpose.  But before we could start working, we needed to find a perfect setting for which we would later design the sounds and the visuals. So we installed huge mirrors in one of the rooms at Artarea, Tbilisi, Georgia, which would serve as Echo's chamber, where each visitor would sit on a very comfortable sofa surrounded by the circular 7-speaker audio system with a subwoofer. The mirrors were the starting point for designing the visuals, and we wanted them to enhance the visitors' feeling of depth and dimensionality. Hence, the mirrors had to fit and connect flawlessly. This was a very challenging task for the team who installed the mirrors.

The technical prototype of Project Echo, made by the sound designer George Murghulia with MaxMSP, was helpful in the early development of the technical layout, but we eventually shifted to Unity 2021+Wwise. This combination offered more opportunities for creating more complex audiovisual systems.

The visitor’s experience starts with the waiting mode. The sounds and visuals pulsate in sync, helping to prepare the visitor for what is about to come. Selecting the language on the tablet starts the experience. It leads the visitor to “Echo’s Chamber,” where Echo asks the visitor a question, which, upon answering, leads the visitor into the round corresponding to the question.

Each round had its own set of possible sounds in Wwise and visuals in Unity, which interact dynamically and generatively. The Hero's Journey inspired the progression of the experience between the rounds and the selection of feelings each round had to portray.

After going through each round and experiencing the rebirth, the program returns to the waiting mode, signaling the completion of the experience. However, visitors can experience it again without leaving the room.

"One of our challenges was to create smooth transitions between the different scenes without disrupting the visitors' immersion and highlighting the importance of their choice, which should not have felt right or wrong," said George Murgulia. Later, we will discuss the sounds controlling the visuals, which helped make impactful transitions between rounds.

The audio approach

We created all audio systems in Wwise, utilizing immersive custom multi-channel layout support, adapted to the particular environment and circular speaker system.

Each sound had to transport visitors into a specific location and resonate with them. The challenge was designing and preparing a selection of sounds and layers that could work together compellingly while benefiting from a generative nature. “The sounds had to be specific enough to remind the audience of unique sounds they could encounter during their lifetime while also being ambiguous enough to fit people of diverse backgrounds,” said Beso Kacharava. The team settled on settings that included familiar environments like the seaside, a stadium, a plane, and a subway, while also having more designed and rare environments like a mine, a futuristic city, etc.

The objective was to create a dynamic and expressive atmosphere, so Beso and his team approached it like we design sounds for video game environments. We started with a sound structure that included "bed tracks" and hundreds of layers, reflecting specific sounds such as cars passing by, birds twittering, calls, shouts, etc.

Another challenge was to determine the appropriate periodicity of the sounds. The team experimented with different ideas, eventually settling on a controlled generative approach in Wwise. "Listening to real rainforest sounds inspires me in a meditative way," said project's author Beso Kacharava. "You will start to hear patterns between birds communicating, waves of winds and raindrops hitting surfaces; by understanding that concept, we can go beyond just assigning randomization values for intervals between sounds, the magic starts when you find the right blend of ‘controlled randomness’ and right sounds. This way, the environment resonates with the listeners." With this approach, we had to consider how elements of the environments would work outside the DAW before implementing the sounds. Because the pattern relationships can be easily lost during randomization.

Let's discuss a simple example of dogs contributing to a village’s signature soundscape. Usually, one dog starts to bark, and all other dogs follow. You have two choices to recreate that in Wwise; either you render the groups of dogs barking sequences into files and then export them to Wwise, or you need to break down the logic of the dog’s social interaction and reconstruct it. Both approaches work; however, while being more challenging, the second one can feel more authentic.

For that, we need Random Containers containing dog bark random sub-containers. Although the samples in the sub-containers should contain short samples, they can contain more than one bark to feel right and can be small sequences on their own, basically the smallest sets of barks that make sense.

After preparing the Random Containers, we build different sequences for the dog barks. Remember, there is always one mischievous dog that starts the chaos. For example, in this set, the 4 dogs are separated with a trigger delay gap (randomized up to 3 seconds), which feels like real dogs barking in a village.

Through testing and guided by what feels good, we create sequences that will be triggered randomly as an element of the village soundscape.

The same can be done with birds.

An important consideration is that an individual source(a dog, a bird, or a creak) does not disappear after producing the sound, so the soundscape should feel right. 

Approach to music and designing a library

The music for "Project Echo" was composed by Zviad Mgebry. He aimed to produce music centered on timbres and emotions rather than being restricted to a particular genre. "The concept of freedom inspired me; I intended to create a musical form that, while being adaptive to visitors' decisions, would still feel musically natural," said Zviad. 

For this project, we needed to create a musical language and timbres that blended seamlessly with sounds and visuals. To that end, Zviad worked with POSTRED's internal audio tools team to develop an Echo Toolkit virtual musical instrument.

We designed two types of instruments for the audio tool: fixed and procedural. 

• Fixed instruments were programmed to follow specific rules and could be performed expressively. 
• Procedural instruments, conversely, generated sounds based on algorithms and variable parameters. 

To create the Echo Toolkit, we utilized the sampler Kontakt for the prototype but decided to stick with it until the end because of its strong scripting and modulation capabilities. "I wanted to transform traditional instruments into something new to create more xylophone-like instruments. Imagine instead of performing a single Taiko drum like a percussive instrument, you play a melody with it all over the keyboard," said Zviad.

My initial idea was to utilize algorithms that would adapt seamlessly to visitors' choices and produce a dynamic soundtrack. But you know how experimental art projects go when more important projects are under the ticking clock. Ultimately, we decided to use a safer approach that would be as effective but also work "out of the box" - with less risk. We deeply sampled the output of the Echo Toolkit under different conditions and emulated its functionality in Wwise.

Initially, Zviad positioned the sounds in each of the seven speakers but later changed the approach by animating the sounds and instruments through the sound field. The acoustics of Echo's space were already excellent, so we did not need to use any extra reverbs. 

Approach to dynamic visuals

Research and experiments for shaping Echo’s visuals

Luka Lebanidze tells the whole experience from his perspective: As the technical lead on Project Echo, I wanted to achieve a unique yet very natural experience in the execution. From the start, I knew I had to find an approach that would melt visual and audio together in an integral way.

Everyone in the audio world is used to seeing moving images that procedurally react to audio: curves, particles, meters, and waves. As sound artists, we see them everywhere: in music apps, plugins, hardware, and even video games. While these visuals are prevalent, they are still captivating and beautiful. This organic nature of waveforms has always captivated me; during the last few years, I had discovered the oscilloscope music genre for myself - music where sounds create images. Inspired by the exciting and crazy things the artists in that niche do with sounds, lasers, and oscilloscopes, I decided to try it as a geeky hobby, and I am glad I did.

Through playing, I observed something that has fascinated me since then. When you watch through an oscilloscope, combine two signals, and tune them in different combinations, you see their fluctuating interaction visually. 

Sometimes, it is chaos, but sometimes, when you find "sweet spots," the image vibrates and dances interestingly. What you hear is what you see. 

In the following video, you can see that signal #1, which draws a cube, is fixed to a frequency of 55 Hz, which corresponds to note A; when we start to slowly tune signal #2 up, which looks like a pyramid, the square image starts to break and reappear again in different "twisted" versions. 

Using octaves and different musical intervals brings exciting visual results. So, for note A, the sweet spots would be 55*n frequencies. When cycling through different frequencies, images form exciting shapes in places where the audio sounds feel good (‘harmonious’, if we can use this word in this context). Again, what you see is what you hear. When you tune the signals to the frequencies that might not have any relation and don't sound good, the visual becomes a real mess - noise chaos.

Tuning signal #2 to 440 Hz, the A note creates a more stable figure because it is in octaves to signal #1 of 55 Hz, which becomes the fundamental frequency. The real magic happens when you slightly detune signal #2 to 440.37 Hz. The image starts to pulsate; an exciting texture emerges from the lines and rotates, which was not present before. The figures form into complex but beautiful Mandala-like patterns. 

The kind of images you can see through oscilloscopes always look to me in the way ‘they meant’ to be in nature.  I think there is something fundamental about the image of waves interacting with each other. Vibration is one of nature's fundamental phenomena and may even apply to everything.

People with experience working with hardware oscilloscopes know a pattern called Lissajous Figures. Lissajous figures are produced by the intersection of two sinusoidal curves, the axes at right angles. I wanted to explore in depth what Lissajous Patterns can do and find an approach that would work in a game engine for Echo.

(The results of some of the geeky experiments I mentioned are available as background animations on POSTRED's game audio page.)

The best tool I found for quickly experimenting with Lissajous Figures is the online Desmos Graphic Calculator.

By experimenting with parameters and trigonometric equations in the Desmos Graphic Calculator, I was able to build a unique logic for particle movement in Unity. I translated my Desmos equations to Unity VFX graph particle systems; we could generate satisfying and dynamic visuals by feeding in real-time parameters from Wwise. By adding multiplier variables (m, n, i, j, etc.) to equations, you can affect the symmetry and the twist of the figure; these variables are assigned to game parameters received from Echo's sounds to morph the particle figures.

This is an example of very complicated knots that you can achieve. I call them "elven figures." If you take a closer look at the images, which are chaotic and orderly, you can feel the same behavior I described in oscilloscope visuals. 

On a side note, Desmos is handy for explaining simple audio and synthesis concepts to our academy sound and music students. It helps to demonstrate graphically how different sounds interact and what happens during physical processes like filtering, doppler, additive, and subtractive synthesis.

This is an example of particles animated through a similar equation created in Desmos and later translated into particle movement in Unity’s VFX graph.

Idea execution in Wwise and Unity

The next step was binding the patterns to audio waveforms and frequencies. We used the Wwise Meter plug-in to convert the audio levels into Game Parameters. These were connected to the equations designed using Desmos and integrated using the VFX graph in Unity.

This image is a straightforward example of a waiting mode sound: a pulsating low tone played before the experience starts. The particles start to twist when the audio amplitude rises. The amplitude gets converted into a Game Parameter, which feeds into the particle equation as one of the variables. Also, sound designer George Murgulia attached the Game Parameter value output to Arduino, controlling the LED intensity that guided visitors to the sofa in the darkness.

For more complex interactions with visuals, where the amplitudes of different frequencies in the audio spectrum affect the visuals, you either need a custom solution to get the audio spectrum data out of Wwise or create several auxiliaries using manually separated frequency bands. We should not worry about what happens to the audio here because the signals are only used to drive the shaders and particles in Unity.

Making Echo’s figure speak

The image of "Echo's" entity was inspired by one of Carl Jung's Archetypes: the shadow, which refers to the unconscious aspect of our psyche that contains repressed or hidden aspects of our personality. 

I wanted to use a very intuitive shape with a concept fundamental to nature so the sphere idea resonated with me because a sphere is an object with all its edges equally spaced from the center. The sphere is an ideal object that cannot exist in real nature, so if we think of the concept of Project Echo, the Echo holds the pure answer, and we can never know the real essence of the answer; so the right thing to do seemed to make the sphere's surface fluctuate and to hide the sphere's shape in noise. Similar to the chorus/mandala effect we found in our experiments with an oscilloscope and Desmos graphing calculator. So, I fed the noise into the sphere's geometry and assigned the fluctuation multipliers to Echo's voice's amplitude and frequency spectrum. We recorded POSTRED’s managing partner Tina Babakishvili as Echo's voice. When Tina talked, Echo's Geometry responded and morphed.

A Funny Incident:

Despite our initial excitement, we encountered issues we hadn't anticipated, which we discovered on the second day that Echo was open.

As it turned out, since the equations running the visuals used the game engine’s game time to calculate motion, the program started fine but slowly and continuously picked up speed, which was not noticeable during sessions. We missed that since the program ran up to a maximum of 1 hour during our development tests. It was an important detail we overlooked because, after 3-9 hours of running, the procedural animation of the visuals got so fast that it led to a completely different and unexpected experience for our visitors. Which I would describe as more intense than originally intended. 

Of course, we couldn't leave things in disarray for long. After some head-scratching, we found the issue that had caused the installation to go haywire. We added a time parameter reset for each cycle of the experience, so the application running time no longer affected the experience. So, if you arrived in the evenings of the first two days, you may have experienced a “limited edition” of Project Echo!

Final thoughts

“When you sit in a dark room on a very comfortable sofa surrounded by seven speakers and see beautiful visuals in mirrors, which create a feeling of endless space in total darkness, you start to think about your existence. The feeling is unique: testing the experience every day before the project's launch was a great memory,” said Saba Baidoshvili, the developer who worked with me on this project.

We would be happy to do Echo II if there is a chance and time. It was a very interesting way to see what can be done with the same tools that we use every day but with a totally different twist.