Musical Acoustics &

Psychoacoustics
Donald Byrd
5 Nov. 2007

Copyright 2006-07, Donald Byrd

Review: Representations of Music

Three basic forms (representations) of music
Audio: most important for most people (general
public)
MIDI files: often best/essential for some musicians,
especially for pop, rock, film/TV
Notation: often best/ essential for musicians (even
amateurs) & music scholars
Essential difference: how much explicit structure

Music holdings of Library of Congress: over

10M items
Includes over 6M pieces of sheet music and 100Ks of
scores of operas, symphonies, etc.: all notation!

Differences are profound

rev. 8 Sep. 2006

Review: Representations of Music &

Audio
Digital Audio

Audio (e.g., CD, MP3):

like speech

Time-stamped Events
Time-stamped
Events
(e.g., MIDI file): like
unformatted text

Musiclike
Notation
Music Notation:
text with complex
formatting

1 Sep. 2006

Review: Rudiments of Musical

Acoustics
Need some musical acoustics for almost
anything in digital audio
Acoustics: branch of physics that
studies sound (of any kind)
Concepts like frequency & amplitude

Psychoacoustics: study of how sound is

perceived; mostly psychology
Concepts like pitch, loudness, timbre

1 Sep. 2006

Materials for Studying Audio

What are interesting sounds really like?
Sine waves, etc. are boring (cf. addsynenv)
Sounds of acoustic instruments are rich
Vary in every way: with pitch, loudness, time

Musical instrument samples

Audacity audio editor
For Windows, Mac OS 9 and X, Linux
Download from
http://audacity.sourceforge.net/

Programs in (e.g.) R
10 Sept. 2006

Review: Parameters of Musical Sound

Four basic parameters of a definite-pitched
musical note
1. pitch: how high or low the sound is: perceptual

analog of frequency
2. duration: how long the note lasts
3. loudness: perceptual analog of amplitude
4. timbre or tone quality

Above is decreasing order of importance for

most Western music
and decreasing order of explicitness in CMN
(Conventional Music Notation)!
10 Sept. 2006

Scholars (and others) Beware! (1)

Plausible (at the time!) assumptions
Men have more teeth than women (ancient)
Diseases cant be transmitted by invisible
organisms (19th century)
Stomach ulcers cant be caused by
organisms (20th century)

What you expect & what you see/hear

A recent discovery about kitchen sponges
1923 New York Times headline: Dinosaurs
Cavort in Film for Doyle

rev. 31 Oct. 2007

Scholars (and others) Beware! (2)

What you expect & what you hear

Don & the Kurzweil 250 flute sound

Hammond organ vs. a real pipe organ
Don, a famous musician, & K250 handclaps
Huron on what he knew & learned
R. Moog at Kurzweil & piano touch

rev. 20 Sep. 2006

Musical Acoustics (1)

Acoustics involves physics
Musical (opposed to architectural, etc.) acoustics

Frequency (=> pitch)

Amplitude (=> loudness)
Spectrum, envelope, & other characteristics => timbre
Partials vs. harmonics

Psychoacoustics involves psychology/perception

Perceptual coding (for lossy compression: MP3, etc.)

Timbre
Old idea (thru ca. 1960s?): timbre produced by static
relationships of partials, plus envelope
but attack often more distinctive than steady state!
Rich (interesting) sounds are complex; nothing is static
Time domain (waveform) vs. frequency domain (spectrum,
spectrogram) views

rev. 11 Sept. 06

Materials for Studying Acoustics

Most useful
Piano (preferably a grand)
String instruments (preferably large & bowed)
Drinking glasses filled to different levels
Simple percussion: sleighbell, rainegg, etc.
Recording equipment
Audio editor, spectral-analysis program

Also helpful
Other musical instruments
Tuning fork
Strobe light
Echoic & anechoic chambers

rev. 31 Oct. 07

10

Modes of Vibration &

Simple Vibrating Systems
Waves can be transverse or longitudinal
Which is water? Which is sound?

Simple vibrating systems & waveforms

Tuning fork
Not many other examples!

Whats the waveform?

4 April 07

11

Complex Vibrations & Resonance

Modes of vibration

Air column closed at both ends vibrates at f, 2f, 3f

where f = frequency determined by length
Closed at one end, open at other: f, 3f, 5f
where f = half the frequency of previous case

Resonance
When a system that can vibrate at a certain frequency is
acted on by periodic disturbance at same frequency,
vibrations of large amplitude can be produced. Backus
Ex: swing in playground
Ex: string & body of musical instrument
Resonance: tendency of a system to oscillate at maximum
amplitude at a certain frequency.

4 April 07

12

How String Instruments

Make Sounds
Vibrating strings
Waveform is almost a sawtooth => lots of
harmonics!
Bowing
Rosin to increase friction
Special effects: sul ponticello, col legno, etc.

Plucking

4 April 07

13

How Woodwinds Make

Sounds
Vibrating air columns
Without reeds: flute
Acts like open column

With reeds: oboe, clarinet, bassoon

Clarinet acts like half-open column
Why?
Same length => an octave lower

Overblowing
What about brass instruments?

4 April 07

14

How Percussion
Instruments Make Sounds
Guess

4 April 07

15

Acoustic Phenomena
Sympathetic vibration
The harmonic series
Beats
Chorus effect

Harmonics (on bowed strings) show:

Modes of vibration
Nodes
Relationship to sul ponticello?

Architectural acoustics
Resonance (as in I Am Sitting in a Room)
Standing waves

31 Oct. 07

16

Psychocoustic Phenomena
A matter of psychology (and, to some extent,
music theory)
Basis for auditory illusions
Endless glissando
Mystery melody

Practical application: perceptual coding used in

lossy compression (MP3, WMA, etc.)
Masking
Threshhold of audibility

4 April 07

17

Creating Interesting Sounds (1)

Periodic, nearly-, & non-periodic signals
Approaches to creating sounds

Additive synthesis: like painting

Subtractive synthesis: like sculpture
Sampling: like collage
Others: modulation, physical modeling

Most possible with analog or digital

hardware, but analog is limited

11 Sept. 2006

18

Creating Interesting Sounds (2)

Cf. Electronic Music Tutorial (Ishkur)
Additive Synthesis
Fourier's Theorem
Any periodic signal => sum of harmonically-related
sine waves

Envelopes: continuous & piecewise linear

Important special case: ADSR
Attack, Decay, Sustain, Release

Phase, interference, & beats

Phase by itself rarely important, but relationships are
Diagrams & demo in CECM Acoustics Primer, Sec. 8
Interference between channels or very close partials

rev. 13 Sept. 2006

19

Additive Synthesis & Envelopes

addsynenv does additive synthesis of up to six partials
Each has arbitrary partial no., starting phase, "ADSR"
envelope
Partial no. can be non-integer => not harmonic
ADSR = Attack/Decay/Sustain/Release (3 breakpoints)
but addsynenv allows much more complex envelopes
Plays one note with waveform specified by partial nos. &
their envelopes (maybe also phases)
Simultaneously displays spectrogram or sonogram
but not waveform
Phase in real world normally has little effect, but can be
critical in recording & digital worlds (e.g., cancellation)

Additive synthesis cant create aperiodic (non-definite

pitch) sounds
...or many realistic attacks

rev. 13 Sept. 06

20

Creating Interesting Sounds (3)

Early CCRMA (Stanford) studies of acoustic
instrument sounds
Envelope for each partial with a few segments
Similar to addsynenv

Subtractive Synthesis
Early (analog: Moog, etc.) synthesizer model
Signal source: sine/square/sawtooth (for
additive)
or noise & filters (for subtractive)
LFO to add vibrato, tremolo, glissando, etc.
ADSR envelope for entire sound
11 Sept. 2006

21

Periodicity & Definite Pitch

Periodic waveform: clearcut
definite pitch
With sharp corners: much
high-freq. energy

square wave; loud trumpet

Without sharp corners:

little high-freq. energy

sine wave; low, soft flute

Almost-periodic waveform:
definite pitch

piano

Somewhat periodic
waveform: complex or
multiple pitches

bells

Aperiodic waveform: noisy,

no definite pitch

cymbal, bass drum

31 Oct. 07

22

Real-World Musical Sounds (1)

The Attack/Sustain/Release (ASR) model for notes
Attack, Sustain, Release modified from recordings
Attack includes Decay from ADSR model

Used in the Kurzweil 250 (1984), etc.

Original version had only 2 MB for all samples

Piano had diff. samples for 2 loudness levels
and diff. sound for every 4-6 semitones
1-2 sec. per sample for A+S+R

How good did the K250 really sound?

COUNTDOWN, by Christopher Yavelow
An opera for the nuclear age
the orchestral accompaniment is in reality a Kurzweil250 digital sampler, synchronized to the baton of the
conductor
http://www.yavelow.com/docs/countdown.html

12 Feb. 07

23

Real-World Musical Sounds (2)

Nowadays, can afford unlimited sustain
but also need diff. sounds for many (8?) diff.
loudness levels (multisampling)
All Together Now, Electronic Musician, Jan. 2007

and diff. sound for every semitone or two

W/ unlimited sustain, takes gigabytes just for
piano!

19 Feb. 07

24

Timbre Space & Envelopes (1)

Timbre space
Includes all possible timbres
Not well understood; may be 3-dimensional

Original FourierRandomTimbre covers only a

tiny part
Some ways to enlarge it:
Overall envelopes
Allow (inharmonic) partials
Envelopes for partials

Formants show timbre involves context: not

really a property of a note!
Timbre as "the psychoacoustician's
multidimensional wastebasket category"
29 Oct. 07

25

Timbre Space & Envelopes (2)

Envelope
Cf. Wikipedia article on timbre
Amplitude structure of a sound, so called because sound
just "fits" inside its envelope

Piecewise-linear envelope
Envelope that can be drawn with a relatively small number
of straight lines and no curves
Important in music synthesis because can approximate
many complex real-world sounds well with very little data
Special case: "ADSR" (Attack - Decay - Sustain - Release)
envelopes; common on synthesizers, etc.
Samplers may combine A & D

29 Oct. 07

26

Uncompressed Audio Files are Big

1 byte = 8 bits (nearly always)
How much data on a CD?
CD audio is 44,100 samples/channel/sec. * 2
bytes/sample * 2 channels = 176,400
bytes/sec., or 10.5 MByte/min.
CD can store up to 74 min. (or 80) of music
10.5 MByte/min. * 74 min. = 777 MBytes
Actually more: also index, error correction
data, etc.

22 Sep. 2006

27

Compressed Audio: Lossless & Lossy

Dont confuse data compression and dynamicrange compression (a.k.a. audio level
compression, limiting)
Codec = COmpressor/DECompressor
Lossless compression
Standard methods (LZW: .zip, etc.) dont do much for
audio
Audio specific methods

MLP used for DVD-Audio

Apple & Microsoft Lossless

Lossy compression
Depends on psychoacoustics (perceptual coding)

16 Feb. 06

28

Specs for Some Common Audio Formats

13 Feb. 06

29

Psychoacoustics & Perceptual Coding

Pohlmann, Ken (2005). Principles of Digital
Audio, 5th ed., Chapter 10: Perceptual Coding
Rationale: much better data compression
Physiology of ear and critical bands
Not fixed frequency: any sound creates one or more
critical bands

Masking
Depends on relative loudness & frequency
Noise is much better than pitched sounds

Threshhold of hearing
Depends greatly on frequency

22 Sep. 2006

30

Compressed Audio: Lossy Compression

General method
1. Divide signal into sub-bands by frequency
2. Take advantage of:

Masking (shadows), via amplitude within

critical bands
Threshhold of audibility (varies w/ frequency)
Redundancy among channels
MPEG-1 layers I thru III (MP-1, 2, 3), AAC get better &
better compression via more & more complex
techniques
There is probably no limit to the complexity of
psychoacoustics. --Pohlmann, 5th ed.
However, there probably is an asymptotic limit to
compression!

Implemented in hardware or software codecs

22 Feb. 06

31

Another Analog vs. Digital Battle

Merton, Orren (2006, February). The Sum of All
Tracks. Electronic Musician 22,2, pp. 57-63
About summing (mixing) with analog vs. digital
hardware
Tested with excellent hardware, panel of experts
Methodology not really scientific
But is this fair? Does it really matter? How can
you do better with subjective phenomena?
Electronic Musician isnt a scholarly journal
Critical: blind study => results probably meaningful
Cf. http://createdigitalmusic.com/2006/09/06/analogsumming-pm8-for-people-who-dont-trust-softwaremixing/

rev. 25 Sep. 2006

32

Scientific Evaluation of Subjective

Phenomena: Lossy Compression (1)
Pohlmann, Ken (2005). Principles of Digital
Audio, 5th ed., pp. 403-413
Discusses perceptual coding performance
evaluation
Want to evaluate as objectively as possible!
One idea: measure artifacts
Difference between original & compressed =>
distortion
Psychoacoustic model can estimate NMR (Noise-toMasking Ratio)

25 Sep. 2006

33

Scientific Evaluation of Subjective

Phenomena: Lossy Compression (2)
Pohlmann: best way to evaluate is to
exhaustively listen
Tests with excellent hardware, panel of experts
but serious scientific methodology
Double-blind, ITU-R methods, CCIR 5-pt scale,
>=50 subjects, etc.
Similar methods used in validation of MP3,
AAC, etc.
and ff123 et als evaluation of Ogg Vorbis
Our listening test: at http://www.informatics.
indiana.edu/donbyrd/N560Site-Fall06/Worst/
25 Sep. 2006

34

Auditory Illusions 1
Wikipedia article auditory illusion
Deutsch's Musical Illusions
E.g., Mysterious melody, Chromatic illusion
http://psy.ucsd
.edu/~ddeutsch/psychology/deutsch_research1
.html

Visual and Auditory Illusions (from UBC)

Often analogous illusions
E.g., Shepard's Tones, Tritone Paradox
http://www.cs.ubc.
ca/nest/imager/contributions/flinn/Illusions/Illu
sions
.html
25 Sep. 2006
35

Auditory Illusions 2: Note, Chord, or

Scale?

Issue is partials each with its own envelope

Clue: relative frequencies of partials
Clue: shapes of envelopes
Try to simplify the sound
Do you experience fusion?

rev. 25 Jan. 2007

36

Auditory Illusions: Shepards Tones

Also called (less accurately) endless glissando
Demo on Web
http://www.cs.ubc.ca/nest/imager/contributions/flinn/Illusions/Illusi
ons.html

Wikipedia version has a more interesting timbre

Again, its partials & envelopes

Clue: relative freqs. of partials are important,
unusual
Clue: envelopes of partials are important
Used by J.C. Risset in computer-synthesized
music
in one case, with rhythm equivalent

rev. 5 Nov. 2007

37

Visual Analogue of Shepards Tones

rev. 25 Jan. 2007

38

Auditory Illusions: Mysterious Melody

Hard to recognize, unless you already
know what it is!
Clue: probably easier for Americans

rev. 5 Nov. 2007

39