As sound designers, we are usually focused on how sound can affect a player’s immersion or enjoyment of a game - certainly important. However, for this blog post I wanted to try and focus on a slightly less common topic, how sound can affect a player’s experience in relation to their 'performance'. How can we improve–or worsen–a player’s ability to succeed with sound?
I’m going to separate this post into three major categories:
- Why and how does sound affect our performance
- Specific studies done with games to test player performance
- My conclusions, additional facts, and practical applications
Do keep in mind that while a lot of this post is based on actual science, a lot is also based on my own hypothesis and conclusions from all the bits and pieces of research I’ve dug through. I’ll try my best to clearly identify facts from my own observations or opinions. Reading research papers is not the most exciting thing in the world, so I’ll mostly only be focusing on results rather than the methods of the studies, unless the methods are particularly relevant to the discussion.
Why Sound?
Why would we even consider sound as a tool for affecting performance?
Auditory reaction time is significantly faster than Visual reaction time
In several studies done with athletes between 2007-2010 to test simple reaction time, the subjects scored 40-50 ms faster with auditory as apposed to visual reaction time. (1) If you’ve ever tried playing a twitch FPS or MOBA with 100 ms ping as apposed to a50 ms ping, you’d understand just how significant this small amount of time is in many gaming situations. This is an obvious argument for a clear and well-mixed game where audio cues can always be heard along with their visual.
Audio produces a stronger physiological response than Visuals alone
This is mostly common knowledge at this point. We all know that music and sound along with a visual is significantly more arousing or stressful than a visual by itself. This is significant to performance however, not just immersion and enjoyment. According to the Yerkes-Dodson Law, a small amount of stress or anxiety helps us complete complex tasks more effectively, but once we reach a certain level of stress we begin to perform worse. (2) Of course, what is high stress or arousal will vary significantly from person to person and the measurements of stress are very subjective. The core concept of balancing stress to maintain optimal performance is very important in games and is reinforced in several of the studies I’ll mention later in this post.
Audio can alert the player to things outside of their field of view (FOV)
A very obvious function for audio is that it can alert the player to things they cannot see. This benefit is most prevalent in games where the FOV is very limited—first person games for example—and least beneficial in games with a wide field of view such as isometric games.
Relevant Studies With Specific Games
Legend of Zelda: Twilight Princess
I’d like to begin with a study done in 2010 and 2012 using the action/adventure game Legend of Zelda: Twilight Princess, as it lays out a good foundation for a lot of the major concepts for the rest of this section. The study tested how effectively players were able to complete sections of the game within three different conditions:
- with the in-game audio on
- with the in game-audio off
- and with the in-game audio off and unrelated music playing in the background
The expectation was that players would perform best with the in-game audio on, but the actual result ended up being a little more complicated. The average player actually scored highest when playing the game with unrelated music in the background! Don’t mute your games yet though, as that only applied to players who were unfamiliar with the franchise or had less experience with games. The players who scored higher in all conditions as well as those who had experience with the franchise scored highest with the in-game audio on. Also, even players who scored lower with in-game audio would see their scores improve as they played the game more with the in-game audio condition. So what does this mean?
Basically, this all comes down to two things, cognitive load and arousal. The players who were less experienced had to focus more on learning the controls, figuring out timing, breaking down unfamiliar visuals, and so on. The in-game audio ended up acting as a distraction for those players who were already experiencing a high cognitive load from the visual. The more experienced players however did not have to think as much about the basics, like controls and what cues mean, so they could effectively take advantage of the additional information given to them by the in-game audio without experiencing an excess cognitive load.
Now the remaining question is why was unrelated music able to improve a player’s performance? This basically ties back to the idea of a slight bit of arousal helping improve performance, but since it’s unrelated to the actual gameplay, it ends up not having as strong of a cognitive load. So basically, it helps players focus because of a slight increase in arousal but does not distract them.
Osmos, Flat Out Total Carnage, Amnesia: The Dark Descent
This study doesn’t actually measure player performance but instead measures physiological changes caused by in-game audio using a bioharness—a device worn to record heart rate, respiratory rate, and skin temperature. As mentioned before however, physiological changes will almost always have some effect on performance, whether negative or positive depends on the severity of the change. The other interesting thing about this study is that it uses three different games in very contrasting genres; Osmos is a fairly slow pace puzzle game with relaxing music, Flat Out Total Carnage is a demolition racing game, and Amnesia: The Dark Descent is a first-person horror exploration game. The study took these three games and had participants play them with the in-game audio on and off, measuring differences in heart rate and respiratory rate between the two states.
There were some variations in the results between the three games but heart rate was consistently higher in the audio-on group for all three games. The least significant difference was found in Osmos, in which the audio-off group had a min of 61 bpm and a max of 78 bpm, while the audio-on group had a min of 68 bpm and a max of 84 bpm. This is to be expected since it is a puzzle game with mostly relaxing music and sounds.
Flat Out Total Carnage was the game with the most significant increase in heart rate. The min max for the audio-off group was 64 bpm and 77 bpm, while for the audio-on group it was 57 bpm and 91 bpm. The audio-on group would reach near max bpm fairly quickly and sustain that level for most of the play session. This result makes a lot of sense when you consider that because it is a demolition racing game it has very high-energy music and a lot of big, loud impact sounds throughout.
Amnesia: The Dark Descent was in my opinion tested in a somewhat ineffective way as only the opening level was used, in which there are no jump scares or any real action and only exploration and puzzle solving. This means that basically only ambient sounds and music were really tested, leading to potentially less significant results than if more action intensive moments were included in the test. This game did however still end up with the most varied changes of heart rate and the highest minimum heart rate in the audio group. The min max for the audio-off group was 52 bpm and 77 bpm and for the audio-on group it was 74 bpm and 90 bpm. This variety is most likely caused by randomized, horror ambient sounds that slightly startled players.
Basically this study proves that audio can increase arousal in the player and that the type of audio and type of game has an effect on how dramatic the increase is. Like previously mentioned, an increase in arousal can improve or decrease performance. While there is no actual rule as to how much arousal is ideal, and the level varies a lot from person to person, we can combine these ideas of arousal and cognitive load to affect how well the player can actually perform.
Conclusions, Opinions, Practical Applications: Designing Sound with Player Performance in Mind
A Clean Mix, Remembering the Visual
Okay, so this one is probably very obvious but still worth thinking about in the context of player performance. The more sounds there are going on at once, the higher the player’s cognitive load, therefore leading to reduced performance or delayed reaction to important sounds if the cognitive load is too great. We also need to consider things like visuals and gameplay because they add to this load as well. So, for example a very difficult section of a game, or something with a lot of rapid visuals, may benefit from a reduced number of sounds and a quieter mix overall. The goal in this case would be to keep the player focused on the sounds that matter and the visuals that matter as well.
Attention-Grabbing Sounds, Transients and Frequencies
Human beings have different reaction times for different types of sounds. The speed of human reaction time to pure-tone auditory stimuli is in agreement with equal-loudness curves, with sounds we perceive as louder also producing quicker reaction times.(5) Humans react more quickly to sounds with rapid changes in volume than slow changes, even if both sounds reach the same peak volume.(6) Keeping this in mind, it’s best to design sounds that provide important information to the player with rapid change in volume (high-transient), and have a significant amount of information in the 2 -5 kHz range–where human ears are the most sensitive.
Multiplayer or Single-Player
How we approach designing a sound for a multi-player centric game as apposed to a single-player centric game may be completely different–even if the animations and visuals are identical. What information becomes important is different in the two, also how much information we want to give away is important to consider as well. I’ll give an example of this with some of my own work with the indie game Blockspell. There is a high damage ability in the game called “inferno” which is basically like a large magical flame-thrower. Because this is a multiplayer game, I decided to give a strong attack to the sound even though the animation just has flames gradually fade in. I decided to do this to be able to warn nearby enemy players that someone has begun using inferno, so that they can attempt to quickly escape before they get caught in the flames. If this were a single-player game I would have just done something to better fit the dynamics of the animation.
Music, Ambiences, The “Cocktail Party Effect”
Sometimes music can be very gameplay supportive, like in the 3D Legend of Zelda games where 'battle' music begins when there are enemies nearby, or in stealth games like Metal Gear Solid where the music changes to reflect how alert the enemies are to your presence. Ambiences can also provide similar information by having different ambiences for a safe or dangerous area. In most cases though, they don’t provide any information that is relevant to gameplay, and are more for setting the mood and helping with immersion.
If performance is of a concern however, how can we compose music or design ambiences to improve–or hinder–player performance? We can apply a lot of the same principles as those we apply to creating attention-grabbing sounds. If we want the music or ambiences to not be distracting we should avoid them being too loud or dense, avoid too many percussive or high transient elements; avoid excessive arousal by keeping tempo or sound density low. A more specific example with ambience would be designing random ambient bird chirps to slowly rise in volume rather than pop out right away; this helps the birds feel non-diegetic and keep them from grabbing the attention of the player if they are unimportant to the gameplay.
One other very important thing to keep in mind is the Cocktail Party Effect. This effect is basically the way that humans can pick out individual sounds, even when many sounds are happening at once.(7) This is important to consider because we have been trained to focus on sounds we deem important even when there are a lot of other things going on. So when composing music or designing ambiences, it’s incredibly important to avoid including sounds that are similar to any gameplay relevant sounds which a player is listening for, unless you are deliberately trying to confuse or distract them. An example would be a game that has music with interesting percussive elements like door slams or anvil hits; and the problem would be that this game is a fantasy action game where these sounds already exist in the game world, leading to many confusing situations where the player has trouble differentiating between the music and actual in-game events.
Of course, the opposite can be done if you are trying to grab attention or increase arousal in the player for the sake of enjoyment. It’s very common in racing games to have very high tempo and exciting music for example, this leads to more aggressive and exciting play styles but also means that players tend to make more mistakes with the music on.(8) It doesn’t really matter that the player is making mistakes though because this heightened arousal would make the player feel more excited about the game and help with immersion.
Make The Game Harder Or Just More Exciting
A really good example of games with audio that makes them harder, is horror games. Most horror games are not nearly as scary or stressful with the sound off. Music and ambiences in these types of games will often have elements that could be confused for in-game events, like violin screeches, distant howls, and creaking. In most horror games, stressing out the player is a good thing, so think about how to maximize player stress using all the things we previously mentioned in an inverse manner.
Does any of this really matter?
Kinda? Is player performance the most important thing to consider when designing a game’s audio experience?
- No. Not at all. This is just one small thing to consider along with things like immersion, storytelling, and what’s probably the most important–fun!
Does the type of game determine the importance of audio’s influence over player performance?
- Yes, yes, and yes. Player performance does not matter very much in a turn based JRPG compared to something like a competitive FPS game.
Why did I even bother writing this whole blog post about audio’s effect on player performance?
- Understanding how sound can affect player performance is just another tool we can use in the name of making games a more fun and compelling experience.
1 Neuroscience & Medicine, 2010, 1, 30-32
doi:10.4236/nm.2010.11004 Published Online September 2010 (http://www.SciRP.org/journal/nm)
2 Yerkes RM, Dodson JD (1908). "The relation of strength of stimulus to rapidity of habit-formation". Journal of Comparative Neurology and Psychology. 18: 459–482.
3 Tan, S. L., Baxa, J. P., & Spackman, M. P. (2010). Effects of built-in audio versus unrelated background music on performance in an adventure role-playing game. International Journal of Gaming and Computer-Mediated Simulations, 2 (3), 1-23. & Tan, S. L., Baxa, J., & Spackman, M. P. (2012). Effects of built-in audio versus unrelated background music on performance in an adventure role-playing game. In Ferdig, R.E. & de Freitas, S. (Eds.) Interdisciplinary advancements in gaming, simulations and virtual environments: Emerging trends, pp. 142-164. Hershey, PA: IGI Global.
4 How Does In-Game Audio Affect Players? by Raymond Usher https://www.gamasutra.com/view/feature/168731/how_does_ingame_audio_affect_.php
5 The Relationship Between Loudness and Reaction Time in Normal Hearing Listeners by Lynne Marshall & John F. Brandt Pages 244-249 (1979)
6 Evaluating Warning Sound Urgency With Reaction Times by Clara Suied and Patrick Susini, Stephen McAdams (2008)
7 The benefit of binaural hearing in a cocktail party: Effect of location and type of interferer The Journal of the Acoustical Society of America 115, 833
Monica L. Hawley and Ruth Y. Litovsky (2004)
8 Effects of built-in audio versus unrelated background music on performance in an adventure role-playing game. International Journal of Gaming and Computer-Mediated Simulations, 2 (3), 1-23; Tan, S. L., Baxa, J., & Spackman, M. P. (2012)
댓글
George Levy
June 05, 2018 at 08:03 pm
Thanks for the excellent piece. I never realized there was a direct relation between having audio playing and one's heart rate. Definitely makes sense and I can totally see how having your heart pounding out of your chest can make a game feel far more immersive. I'm gonna wear a Fitbit the next time I'm playing a first-person shooter game to see how my heart handles it.
George Levy
June 05, 2018 at 08:03 pm
Thanks for the excellent piece. I never realized there was a direct relation between having audio playing and one's heart rate. Definitely makes sense and I can totally see how having your heart pounding out of your chest can make a game feel far more immersive. I'm gonna wear a Fitbit the next time I'm playing a first-person shooter game to see how my heart handles it.
Bruno Hanstenreiter
January 31, 2020 at 02:24 pm
Great article!