How does having runtime DSP in games affect the user’s experience?
Runtime DSP is a huge leap in game audio, and I’ve been incredibly happy to see it occur over the past few years. It is another example of the pro audio industry paying attention to games. Runtime DSP helps solve two of the big creative challenges we have in game audio: repetition and interaction with the environment.
Repetition is an issue in games because we are actually pretty good at recognizing when we hear the exact sound twice. Prior to the advent of recording, that literally never occurred in human experience. No vocal utterance or gunshot or sound of garbage cans crashing is ever exactly the same. But in games, when we play back WAV files, we present the same sound; this breaks the illusion we’re trying so hard to create. By using DSP in games, we can alter—subtly or not so subtly—how sounds sound each time they’re played. So rather than hearing the same dinosaur roar over and over, we can take an animal sound and pitch shift it, flange it, whatever, using run-time DSP effects. And each time we need the dinosaur to roar, we can pick different DSP parameters with different automations, giving us a unique sound each time it's heard. That’s only possible with in-game, runtime DSP effects. DSP gives tremendously more flexibility in our sound effects generation.
DSP also lets us tailor sounds to specific environments of contexts. For example, suppose I have character dialog, but that character may or may not be using their radio to communicate. By storing the unprocessed dialog and running the “radioize” DSP at runtime, that gives us far more flexibility and even allows for cases where a line starts radioized, but mid-sentence the player might turn off their radio because we’ve gotten close enough to them to hear them “acoustically.” And of course, now we can emulate damage to the radio, also by using some DSP techniques that trash the sound or otherwise degrade it.
And of course, this doesn’t even touch some of the Virtual Reality issues we’ve already discussed.
I’d love to hear your thoughts on how exactly making professional-grade DSP effects available to game developers pushed the industry forward.
Back in 2000 at Microsoft, we tried very hard to get run-time DSP effects into games for the reasons outlined above, and designed the Xbox audio system accordingly. We even put in a dedicated 56000 DSP chip specifically to support these effects—we knew that pro audio DSP companies were very familiar with that chip and had experience programming them. And there were a couple attempts to use custom DSP effects, but in the end it was a bit too cumbersome to use and the industry wasn’t quite ready for it.
The emergence of game audio engines/platforms such as Wwise and FMOD, combined with the increase in CPU power and memory finally enabled people to be able to create DSP effects and put them directly into game engines. One other thing we’re starting to see is DSP effects specifically designed for the game industry—for example there are effects available designed to create sound variations or to specifically create creature sounds. That to me is very exciting!
In your article “6 Surprising Facts About Composing Music for Video Games” you write, “Because you don’t know when various story elements will actually happen (it depends on the player), when composing for video games you have to write your music to be flexible, and know how to quickly change from ‘wandering around’ music to ‘battling for your life,’ while not sounding obvious or abrupt.” How can composers handle this context-switching naturally?
There have literally been whole books written about this! Part of the fun of working in games is that we’re still figuring this all out…What does it mean to take music—an inherently linear medium—and integrate it into video games—an inherently non-linear medium? Some of the techniques involve breaking music into small(ish) segments from a couple measure to several full musical phrases. As the game plays, special software in the game called the “audio engine” take those smaller segments and line them up to play. Sort of like Mozart’s “Musical dice” game, but with a bit more finesse.
Another technique is vertical layering. Imagine you’re doing a horror game and you’d like to be able to adjust how creepy the music is. You might record some aleatoric high string parts. When the music plays normally, though, you keep these muted. Only as the player walks down the increasingly scary hallway might you start to slowly fade in those string tracks.
Those two techniques, “horizontal resequencing” and “vertical resequencing” are at the heart a lot of game music. All the major game audio engines out there (program like Wwise, FMOD Studio, Elias, CRI, Fabric) support those basic ideas along with various enhancements. That said, there are a lot of other things we try, and people are always coming up with new ideas and implementations, either extensions of horizontal and vertical resequencing or altogether new techniques.
For those hoping to work in the video game sound design industry, what advice would you have? Anything stand out in your career as instrumental to opening up opportunities?
If you’re specifically talking about sound design, then you should get well versed in the various game audio engines, and even game engines themselves. In a study a very large percentage of “game sound designer wanted” job postings (yes, many companies hire full-time employees for this), require knowledge of game audio tools such as Wwise and FMOD, and many also specifically list “scripting,” which is a type of simple computer programming. Properly placing sounds and music into games is actually quite technical, and companies are more and more expecting that their sound design team have the skillset to do it. Fortunately for sound designers (and composers), the most popular game audio tools are completely free to download and learn. It’s only if a specific game wants to use the technology does the company charge a fee, and then it is the game developer who pays the license.
In my own career, I was fortunate enough to have gotten specific skills (music composition and computer programming) just at a time when both of those were needed to make a career in game audio. So I would definitely say, based on both creative and technical needs of the current day, you can do your career a favor by making sure you have a good grasp of the tools, technologies and how they are used to address the creative and artistic challenges we have in games.
One word of caution, though. In games, we sometimes get so caught up in the tech, that it's easy to forget that, at the end of the day, it’s all about what comes out of the speakers. The very best tech won’t make up for mediocre sound design or poorly mixed music. A successful game sound designer will both push the technology and keep the creative/aesthetic bar high.
As a follow-up question, I thought your piece on how video game composers can get more gigs by venturing into sound design was excellent. To turn the question around, how can sound designers get more composing gigs?
I think that’s a bit harder route, mainly because most successful composers have spent years or decades mastering an instrument and/or practicing composing. So if you were, let’s say a tremendous Foley artist, but had never learned how to read music or play an instrument, you’d find it quite difficult to suddenly start composing.
That said, a lot of people who primarily sound designers have also worked in music, either as a composer outright or as a player. So first thing would be to get your composing/production chops up to snuff. Listen to game music that’s out there for the kinds of jobs you’re looking for. Interested in working for a company like Zynga on professionally produced mobile games? Listen to their music. Practice your production skills.
And then don’t be shy! I’ve been doing game music for 30 years and I still get nervous the first time I submit a piece to the game designer. You are probably a better composer than you think you are.
Also find your forte. I know one very in demand composer who is knowing for being able to great “cute” music, just the sort that keeps his dance card filled in mobile/casual games. And I know that “big orchestra” is not something where I can compete with the likes of Gordy Haab, Neal Acree or Lennie Moore. It’s a little humbling, but recognizing your own strengths and weaknesses can go a long way to sustaining a career in game music/sound.
Have you used iZotope’s Iris for game sound design?
I have Iris in my standard ambiences template. I’m a very visually oriented person, so I find the ability to shape the harmonic content of sound incredibly useful. I used it in a bed for a game I’m currently working on called “Mutant Football League.” One of the team’s stadium is in space, and I’ve gotten some great sci-fi/alien ambiences for it using Iris.
What other iZotope products have you used? RX?
I’m not sure that I know any game audio team that hasn’t used RX. I’ve received VO files from game developers that on first listen sounded utterly unusable, but contained some really good performances. RX, in the right hands, can work miracles on those.