Lectures 1 Rabiner Speech Processing

ECE 259B
Fundamentals of Speech
Recognition Lecture 1
Introduction/Overview of
Automatic Speech
Recognition
12/28/2009
Recognition-Overview of ASR
Why Digital Processing of

Speech?
digital processing of speech signals (DPSS)
enjoys an extensive theoretical and
experimental base developed over the past 75
years
much research has been done since 1965 on
the use of digital signal processing in speech
communication problems
highly advanced implementation technology
(VLSI) exists that is well matched to the
computational demands of DPSS
there are abundant applications that are in
widespread use commercially
12/28/2009
The Speech Stack

Speech Applications coding, synthesis,
recognition, understanding, verification,
language translation, speed-up/slow-down
Speech Algorithms speech-silence,

voiced-unvoiced, pitch, formants
Speech Representations temporal,
spectral, homomorphic, LPC
Fundamentals acoustics, linguistics,
pragmatics, speech perception
12/28/2009
Speech Recognition-2001
(Stanley Kubrick View in 1968)
12/28/2009
Apple Navigator -- 1988
12/28/2009
The Speech Advantage

Reduce costs
reduce labor expenses while still providing
customers an easy-to-use and natural way to
access information and services
New revenue opportunities
24x7 high-quality customer care automation
access to information without a keyboard or
touch-tones
Customer retention
provide personal services for customer
preferences
improve customer experience
12/28/2009
The Speech Circle

Voice reply to customer
Customer voice request
What number did you
want to call?
Text-to-Speech
Synthesis
TTS
ASR
Automatic Speech
Recognition
Data
Whats next?
Words spoken
Determine correct number
I dialed a wrong number
Dialog
Management
(Actions) and
Spoken
Language
Generation
12/28/2009 (Words)
DM &
SLG
SLU
Spoken Language
Understanding
Meaning
Billing credit
Automatic Speech Recognition

Goal: Accurately and efficiently convert a
speech signal into a text message
independent of the device, speaker or the
environment.
Applications: Automation of complex
operator-based tasks, e.g., customer care,
dictation, form filling applications,
provisioning of new services, customer
help lines, e-commerce, etc.
12/28/2009
Pattern Matching Problems

speech
A-to-D
Converter
speech
Feature
Analysis
Pattern
Matching
symbols
recognition
speaker recognition
speaker verification
word spotting
automatic indexing of speech recordings
12/28/2009
Basic ASR Formulation (Bayes Method)

Speakers
Intention
Speech
Production
Mechanisms
s(n)
Acoustic
Processor
Speaker Model
Linguistic
Decoder
^
W
Speech Recognizer
W = arg max P (W | X )
W
P ( X | W ) P (W )
= arg max
P( X )
W
= arg max PA ( X | W ) PL (W )
W
Step 3
12/28/2009
Step 1
Step 2
10
Steps in Speech Recognition

Step 1- Acoustic Modeling:
Modeling assign probabilities
to acoustic realizations of a sequence of
words. Compute PA(X|W) using statistical
models (Hidden Markov Models) of acoustic
signals and words
Step 2- Language Modeling:
Modeling assign probabilities
to sequences of words in the language. Train
PL(W) from generic text or from transcriptions
of task-specific dialogues.
Step 3- Hypothesis Search:
Search find the word
sequence with the maximum a posteriori
probability. Search through all possible word
sequences to determine arg max over W.
12/28/2009
11
Step 1-The Acoustic Model

i we build acoustic models by learning statistics of the acoustic
features, X , from a training set where we compute the variability
of the acoustic features during the production of the sounds
represented by the models
i it is impractical to create a separate acoustic model, PA ( X | W ), for
every possible word in the language--it requires too much
training data for words in every possible context
i instead we build acoustic-phonetic models for the ~50 phonemes
in the English language and construct the model for a word by
concantenating (stringing together sequentially) the models for
the constituent phones in the word
i similarly we build sentences (sequences of words) by concatenating
word models
12/28/2009
12
Step 2-The Language Model

the language model describes the probability of a sequence of
words that form a valid sentence in the language
a simple statistical method works well based on a Markovian
assumption, namely that the probability of a word in a sentence
is conditioned on only the previous N-words, namely an N-gram
language model
PL (W ) = PL (w1,w 2 ,...,w k )
k
= PL (w n | w n 1,w n 2 ,...,w n N )
n =1
i where PL (w n | w n 1,w n 2 ,...,w n N ) is estimated by simply

counting up the relative frequencies f of N -tuples in a
large corpus of text
12/28/2009
13
Step 3-The Search Problem

the search problem is one of searching the space of
all valid sound sequences, conditioned on the word
grammar, the language syntax, and the task
constraints, to find the word sequence with the
maximum likelihood
the size of the search space can be astronomically
large and take inordinate amounts of computing
power to solve by heuristic methods
the use of methods from the field of Finite State
Automata Theory provide Finite State Networks
(FSN) that reduce the computational burden by
orders of magnitude, thereby enabling exact solutions
in computationally feasible times, for large speech
recognition problems
12/28/2009
14
Basic ASR Formulation

The basic equation of Bayes rule-based speech
recognition is
W = arg max P ( W | X)
s(n), W
P(W) P( X | W)
= arg max
P ( X)
W
= arg max P ( W ) P ( X | W )
Speech
Analysis
Xn
Decoder
where X=X1,X2,,XN is the acoustic observation (feature

vector) sequence.
= w w ...w
W
1 2
M
is the corresponding word sequence, P(X|W) is the
acoustic model and P(W) is the language model
12/28/2009
15
^W
TTS
DM
Speech
Recognition
Process
ASR
SLU
Acoustic
Acoustic
Model
Model
(HMM)
(HMM)
Input
Speech
Feature
Feature
Analysis
Analysis
(Spectral
(Spectral
Analysis)
Analysis)
Pattern
Pattern
Classification
Classification
(Decoding,
(Decoding,
Search)
Search)
Language
Language
Model
Model
(N-gram)
(N-gram)
12/28/2009
Utterance
Utterance
Verification
Verification
(Confidence
(Confidence
Scores)
Scores)
Hello World
(0.9) (0.8)
Word
Word
Lexicon
Lexicon
16
Speech Recognition Processes

Choose task => sounds, word vocabulary,
task syntax (grammar), task semantics
text training data set => word lexicon, word
grammar (language model), task grammar
speech training data set => acoustic models
Evaluate performance
speech testing data set
Training algorithm => build models from

training set of text and speech
Testing algorithm => evaluate
performance from testing set of speech
12/28/2009
17
Feature Extraction
Goal: Extract robust features (information) from
the speech that are relevant for ASR.
Feature
Feature
Extraction
Extraction
Method: Spectral analysis through either a
bank-of-filters or through LPC followed

by non-linearity and normalization (cepstrum).
Acoustic
Acoustic
Model
Model
Pattern
Pattern
Classification
Classification
Language
Language
Model
Model
Utterance
Utterance
Verification
Verification
Word
Word
Lexicon
Lexicon
Result: Signal compression where for each window of speech
samples where 30 or so cepstral features are extracted (64,000 b/s ->

5,200 b/s).
Challenges: Robustness to environment (office, airport, car),
devices (speakerphones, cellphones), speakers (acents, dialect, style,

speaking defects), noise and echo. Feature set for recognition
cepstral features or those from a high dimensionality space.
12/28/2009
18
Robustness
Robustness
Robustness
Problem:
Rejection
Rejection
Unlimited
Unlimited
Vocabulary
Vocabulary
A mismatch in the speech signal between the

training phase and testing phase can result in performance degradation.
Methods:
Traditional techniques for improving system robustness

are based on signal enhancement, feature normalization or/and
model adaptation.
Perception Approach:
Extract fundamental acoustic information in narrow bands

of speech. Robust integration of features across time and frequency.
12/28/2009
19
Methods for Robust Speech

Recognition
Training
Signal
Enhancement
Testing
12/28/2009
Signal
Features
Normalization
Features
Model
Adaptation
Model
20
Acoustic Model
Goal: Map acoustic features into distinct
Acoustic
Acoustic
Model
Model
Feature
Feature
Extraction
Extraction
phonetic labels (e.g., /s/, /aa/).
Pattern
Pattern
Classification
Classification
Language
Language
Model
Model
Utterance
Utterance
Verification
Verification
Word
Word
Lexicon
Lexicon
Hidden Markov Model (HMM): Statistical method for
characterizing the spectral properties of speech by a parametric random

process. A collection of HMMs is associated with a phone. HMMs are
also assigned for modeling extraneous events.
Advantages: Powerful statistical method for dealing with a wide
range of data and reliably recognizing speech.
Challenges: Understanding the role of classification models (ML
Training) versus discriminative models (MMI training). What comes after

the HMMare there data driven models that work better for some or all
vocabularies.
12/28/2009
21
HMM for speech

Phone model : z (/Z/)
z1
z1
i
1
z1
z2
z1
z3
Word model: is (/IH/ /Z/)i
ih1
12/28/2009
ih2
ih3
z1
z1
z1
z2
z1
z3
22
Isolated Word HMM

a11
1
a22
a12
b1(Ot)
a33
a23
a44
a34
a55=1
a45
a13
a24
a35
b2(Ot)
b3(Ot)
b4(Ot)
b5(Ot)
Left-right HMM highly constrained state sequences

12/28/2009
23
Word Lexicon
Acoustic
Acoustic
Model
Model
Goal:
Map legal phone sequences into words

according to phonotactic rules. For example,
David
Feature
Feature
Extraction
Extraction
/d/ /ey/ /v/ /ih/ /d/
Pattern
Pattern
Classification
Classification
Language
Language
Model
Model
Utterance
Utterance
Verification
Verification
Word
Word
Lexicon
Lexicon
Multiple Pronunciation:
Several words may have multiple pronunciations. For example

Data
Data
/d/ /ae/ /t/ /ax/

/d/ /ey/ /t/ /ax/
Challenges:
How do you generate a word lexicon automatically; how do you add

new variant dialects and word pronunciations.
12/28/2009
24
Language Model
Goal:
Mapping words into

phrases and sentences based on
task syntax.
Acoustic
Acoustic
Model
Model
Feature
Feature
Extraction
Extraction
Handcrafted:
Deterministic grammars that are

knowledge-based. For example,
Yes on my credit (card) please
Pattern
Pattern
Classification
Classification
Language
Language
Model
Model
Utterance
Utterance
Verification
Verification
Word
Word
Lexicon
Lexicon
Statistical:
Compute estimate of word probabilities (N-gram model). For example

Yes on my credit card please
0.4
Challenges:
0.6
How do you build a language model rapidly for a new task.

12/28/2009
25
Pattern Classification
Goal:
Combine information (probabilities) from the acoustic

model, language model and word lexicon to generate
an optimal word sequence (highest
Feature
Feature
Extraction
probability).
Extraction
Method:
Decoder searches through all possible recognition

choices using a Viterbi decoding algorithm.
Acoustic
Acoustic
Model
Model
Pattern
Pattern
Classification
Classification
Language
Language
Model
Model
Utterance
Utterance
Verification
Verification
Word
Word
Lexicon
Lexicon
Challenges:
How do we build efficient structures (FSMs) for decoding and searching

large vocabulary, complex language models tasks;
features x HMM units x phones x words x sentences can lead to
search networks with 10 22 states
FSM methods can compile the network to 10 8 states14 orders
of magnitude more efficient
What is the theoretical limit of efficiency that can be achieved
12/28/2009
26
Unlimited Vocabulary ASR

The basic problem in ASR is to find the sequence
of words that explain the input signal. This implies
the following mapping:
Features
HMM states
HMM units
Phones
Words
Robustness
Robustness
Rejection
Rejection
Unlimited
Unlimited
Vocabulary
Vocabulary
Sentences
For the WSJ 64,000 vocabulary, this results in a network

of 1022 bytes!
State-of-the-art methods including fast match, multi-pass
decoding and A* stack provide tremendous speed-up at
a cost of increased complexity and less portability.
Advances in weighted finite state transducers have
enabled us to represent this network in a unified
8
mathematical framework with only 10 bytes!
12/28/2009
27
Weighted Finite State

Transducers (WFST)
Unified Mathematical framework to ASR
Efficiency in time and space
Word:Phrase
WFST
WFST
Phone:Word
WFST
WFST
HMM:Phone
WFST
WFST
State:HMM
WFST
WFST
12/28/2009
Combination
Combination Optimization
Optimization
Search
Network
28
Weighted Finite State

Transducer
Word Pronunciation Transducer
ey:/.4
dx:/.8
ax:data/1
d: /1
ae:/.6
12/28/2009
Data
t:/.2
29
Algorithmic Speed
-up for
Speed-up
Speech Recognition
AT&T (Algorithmic)
Moore's Law (hardware)
Relative Speed
30
AT&T
25
20
Community
15
10
5
0
1994
1995
1996
1997
1998
1999
2000
2001
2002
Year
12/28/2009
North
NorthAmerican
AmericanBusiness
Business
vocabulary:
40,000
words
vocabulary:
40,000
words
Fundamentals
of Speech
branching
Recognition-Overview
of85
ASR
branchingfactor:
factor:
85
30
Utterance Verification
Acoustic
Acoustic
Model
Model
Goal:
Identify possible recognition errors

Feature
Feature
Extraction
and out-of-vocabulary events. Potentially
Extraction
improves the performance of ASR, SLU and DM.
Pattern
Pattern
Classification
Classification
Language
Language
Model
Model
Method:
Utterance
Utterance
Verification
Verification
Word
Word
Lexicon
Lexicon
A confidence score based on a hypothesis test is associated with

each recognized word. For example:
Label:
Recognized:
Confidence:
credit please
credit fees
(0.9) (0.3)
Challenges:
Rejection of extraneous acoustic events (noise, background speech,

door slams) without rejection of valid user input speech.
12/28/2009
31
Robustness
Robustness
Rejection
Rejection
Rejection
Problem:
Extraneous acoustic events, noise,
background speech and out-of-domain speech
deteriorate system performance.
Unlimited
Unlimited
Vocabulary
Vocabulary
Measure of Confidence:
Associating word strings with a verification cost that
provide an effective measure of confidence
(Utterance Verification).
Effect:
Improvement in the performance of the recognizer,
understanding system and dialogue manager.
12/28/2009
32
TTS
DM
Input
Speech
ASR
SLU
Feature
Feature
Extraction
Extraction
State
-of-the-Art
State-of-the-Art
Performance?
Acoustic
Acoustic
Model
Model
Pattern
Pattern
Classification
Classification
(Decoding,
(Decoding,
Search)
Search)
Language
Language
Model
Model
12/28/2009
Utterance
Utterance
Verification
Verification
Recognized
Sentence
Word
Word
Lexicon
Lexicon
33
Word Error Rates

CORPUS
TYPE
Connected Digit
Strings--TI Database
Connected Digit
Strings--Mall
Recordings
Connected Digits
Strings--HMIHY
RM (Resource
Management)
ATIS(Airline Travel
Information System)
NAB (North American
Business)
Broadcast News
Spontaneous
Switchboard
Call Home
12/28/2009
Spontaneous
Conversational
VOCABULARY
WORD
SIZE
ERROR RATE
11 (zero-nine,
0.3%
oh)
11 (zero-nine,
2.0%
oh)
5.0%
Read Speech
11 (zero-nine,
oh)
1000
Spontaneous
2500
2.5%
Read Text
64,000
6.6%
News Show
210,000
13-17%
Conversational
Telephone
Conversational
Telephone
45,000
25-29%
28,000
40%
Factor of
17
increase
in digit
error rate
2.0%
34
12/28/2009
35
North American Business
12/28/2009
vocabulary:
40,000
branching
vocabulary:
40,000words
words
Fundamentals
of
Speech branching
factor:
factor:85
85of ASR
36
Broadcast News
12/28/2009
37
Dictation Machine
12/28/2009
38
Algorithmic Accuracy for

Speech Recognition
Word Accuracy
80
70
60
50
40
30
1996
1997
1998
1999
2000
2001
2002
Year
12/28/2009
Switchboard/Call
Switchboard/CallHome
Home
Vocabulary:
40,000
words
Vocabulary:
40,000
words
Fundamentals
of Speech
Perplexity:
of ASR
Perplexity:85
85
39
Vocabulary Size
Growth in Effective
Recognition Vocabulary Size
10000000
1000000
100000
10000
1000
100
10
1
1960
1970
1980
1990
2000
2010
Year
12/28/2009
40
Human Speech Recognition vs ASR
12/28/2009
41
Human Speech Recognition vs ASR
M ACHINE ERROR (%)
100
10
x100
1
x10
x1
0.1
0.001
0.01
0.1
Machines
Outperform
Humans
1
10
HUMAN ERROR (%)
12/28/2009
Digits
RM-LM
NAB-mic
WSJ
RM-null
NAB-omni
SWBD
WSJ-22dB
42
Voice
-Enabling Services:
Voice-Enabling
Technology Components
What number did you
want to call?
Text-to-Speech
Synthesis
ASR
TTS
Automatic Speech
Recognition
Whats next?
Words spoken
Dialog
Management
and Spoken
(Actions)
Language
Generation
12/28/2009 (Words)
DM &
SLG
SLU
Spoken Language
Understanding
Meaning
Billing credit
43
Spoken Language Understanding (SLU)
Goal:: Interpret the meaning of key words and phrases in the recognized
speech string, and map them to actions that the speech understanding
system should take
accurate understanding can often be achieved without correctly recognizing
every word
SLU makes it possible to offer services where the customer can speak naturally
without learning a specific set of terms
Methodology:: Exploit task grammar (syntax) and task semantics to

restrict the range of meaning associated with the recognized word string;
exploit salient words and phrases to map high information word
sequences to appropriate meaning
Performance Evaluation:: Accuracy of speech understanding system on
various tasks and in various operating environments
Applications:: Automation of complex operator-based tasks, e.g.,
customer care, catalog ordering, form filling systems, provisioning of new
services, customer help lines, etc.
Challenges: What goes beyond simple classifications systems but below
full Natural Language voice dialogue systems
12/28/2009
44
Voice
-Enabling Services:
Voice-Enabling
Technology Components
What number did you
want to call?
Text-to-Speech
Synthesis
ASR
TTS
Automatic Speech
Recognition
Whats next?
Words spoken
Dialog
Management
(Actions) and
Spoken
Language
Generation
12/28/2009 (Words)
DM &
SLG
SLU
Spoken Language
Understanding
Meaning
Billing credit
45
Dialog Management (DM)
Goal:: Combine the meaning of the current input with the interaction history to
decide what the next step in the interaction should be
DM makes viable complex services that require multiple exchanges between the system and
the customer
dialog systems can handle user-initiated topic switching (within the domain of the
application)
Methodology:: Exploit models of dialog to determine the most appropriate
spoken text string to guide the dialog forward towards a clear and well understood
goal or system interaction
Performance Evaluation:: Speed and accuracy of attaining a well defined task
Applications:: Customer care (HMIHY), travel planning, conference registration,
goal, e.g., booking an airline reservation, renting a car, purchasing a stock, obtaining
help with a service
scheduling, voice access to unified messaging
Challenges: Is there a science of dialogueshow do you keep it efficient (turns,
time, progress towards a goal); how do you attain goals (get answers). Is there an
art of dialogues. How does the User Interface play into the art/science of
dialoguessometimes it is better/easier/faster/more efficient to point, use a mouse,
type than speak multimodal interactions with machines
12/28/2009
46
Customer Care IVR and HMIHY

Customer Care IVR
sm
HMIHY
10 seconds
Sparkle Tone
Thank you for calling AT&T
Sparkle Tone
AT&T, How may I help you?
30 seconds
Network Menu
Account Verification Routine
13 seconds
LEC Misdirect Announcement
26 seconds
Account Verification Routine
58 seconds
Main Menu
38 seconds
LD Sub-Menu
20 seconds
8 seconds
Reverse Directory Routine
Total Time to Get to Reverse

Total Time to Get to Reverse
12/28/2009
Directory Lookup: 28 seconds!!!
Directory Lookup: 2:55 minutes!!!
47
HMIHY sm How Does It Work

z
z
z
Prompt is AT&T. How may I help you?

User responds with totally unconstrained fluent speech
System recognizes the words and determines the meaning
of users speech, then routes the call
Dialog technology enables task completion
HMIHY
Account
Balance
12/28/2009
Calling
Plans
Local
Unrecognized
Number
...
48
HMIHY Example Dialogs

sm
Irate Customer
Rate Plan
Account Balance
Local Service
Unrecognized Number
Threshold Billing
Billing Credit
12/28/2009
Customer Satisfaction
decreased repeat
calls (37%)
decreased OUTPIC
rate (18%)
decreased CCA (Call
Control Agent) time
per call (10%)
decreased customer
complaints (78%)
49
Customer Care Scenario
12/28/2009
50
TTS Closest to the Customers Ear

What number did you
want to call?
Text-to-Speech
Synthesis
TTS
ASR
Automatic Speech
Recognition
Whats next?
Words spoken
Dialog
Management
(Actions) and
Spoken
Language
Generation
12/28/2009 (Words)
DM &
SLG
SLU
Spoken Language
Understanding
Meaning
Billing credit
51
Speech Synthesis
text
12/28/2009
Linguistic
Rules
DSP
Computer
D-to-A
Converter
speech
52
Speech Synthesis
Synthesis of Speech for effective humanmachine communications
reading email messages over a telephone
telematics feedback in automobiles
talking agents for completion of transactions
call center help desks and customer care
handheld devices such as foreign language
phrasebooks, dictionaries, crossword puzzle
helpers
announcement machines that provide things
like stock quotes, airlines schedules, updates
of arrivals and departures of flights
12/28/2009
53
Giving Machines High Quality Voices and

Faces
U.S. English Female:
U.S. English Male:
Spanish Female::
Natural
Speech
12/28/2009
54
Speech Synthesis Examples

Soliloquy from Hamlet:
Gettysburg Address:
Third Grade Story:

12/28/2009
55
Speech Recognition Demos
12/28/2009
56
Au Clair de la Lune
12/28/2009
57
Information Kiosk
12/28/2009
58
Multimodal Language Processing

Unified Multimodal Experience Access to
information through voice interface, gesture or both.
Multimodal Finite State Combination of speech,

gesture and meaning using finite state technology
MATCH: Multimodal
Access To City Help
12/28/2009
59
Are thereRecognition-Overview
any cheap Italian
places in this neighborhood?
of ASR
MIPad Demo--Microsoft
12/28/2009
60
Voice-Enabled Services
Desktop applications -- dictation, command and control of
desktop, control of document properties (fonts, styles, bullets, )
Agent technology simple tasks like stock quotes, traffic
reports, weather; access to communications, e.g., voice dialing,
voice access to directories (800 services); access to messaging
(text and voice messages); access to calendars and appointments
Voice Portals convert any web page to a voice-enabled site
where any question that can be answered on-line can be
answered via a voice query; protocols like VXML, SALT, SMIL,
SOAP and others are key
E-Contact services Call Centers, Customer Care (HMIHY) and
Help Desks where calls are triaged and answered appropriately
using natural language voice dialogues
Telematics command and control of automotive features
(comfort systems, radio, windows, sunroof)
Small devices control of cellphones, PDAs from voice
commands
12/28/2009
61
Milestones in Speech and

Multimodal Technology Research
Small
Vocabulary,
Acoustic
Phoneticsbased
Isolated
Words
Filter-bank
analysis;
Timenormalization;
Dynamic
programming
1962
12/28/2009
Large
Vocabulary;
Syntax,
Semantics,
Medium
Large
Vocabulary,
Vocabulary,
Template-based Statistical-based
Isolated Words;
Connected Digits;
Continuous
Speech
Pattern
recognition; LPC
analysis;
Clustering
algorithms; Level
building;
1967
1972
Connected
Words;
Continuous
Speech
Hidden Markov
models;
Stochastic
Language
modeling;
Continuous
Speech; Speech
Understanding
Spoken dialog;
Multiple
modalities
Stochastic language
understanding;
Finite-state
machines;
Statistical learning;
1977
1982
1987
Year
Very Large
Vocabulary;
Semantics,
Multimodal
Dialog, TTS
1992
Concatenative
synthesis; Machine
learning; Mixedinitiative dialog;
1997
2002
62
Future of Speech Recognition

Technologies
Very Large
Vocabulary,
Limited Tasks,
Controlled
Environment
Dialog
Systems
2002
12/28/2009
Very Large
Vocabulary,
Limited Tasks,
Arbitrary
Environment
Unlimited
Vocabulary,
Unlimited Tasks,
Many
Languages
Robust
Systems
Multilingual
Systems;
Multimodal
Speech Enabled
Devices
2005
2008
Year
2011
63
Issues in Speech Recognition
12/28/2009
64
Issues in Speech Recognition
Speaker trained
Speaker independent
Amount of training material
Quiet office
Home
Noisy surroundings (factory floor,
cellular environments, speakerphones)
High quality microphone, close

talking/noise cancelling microphone
Telephone (carbon button/electret)
Switched telephone network
IP network (VoIP)
Cellular network
12/28/2009
Feedback to users
Instructions
Requests for repeats
Rejections
Tolerance for recognition errors
Syntax constrained (language model)

Viable semantics
Human factors
Highly motivated, cooperative

Casual
Recognition task
Small (2-50 words)

Medium (50-250 word)
Large (250-2,000,000 words)
Speaker characteristics
Transducer and transmission system
Vocabulary size and complexity

(perplexity)
Speaking environment
Isolated words/phrases
Connected word sequences
Continuous speech (essentially
unconstrained)
Recognition mode
Input speech format
Fail soft systems

Human intervention on errors/confusion
Correction mechanisms built in
System complexity
Computation/hardware
Real-time response capability
65
Overview of Speech
Recognition Processes
12/28/2009
66
OverviewofSpeech
RecognitionProcesses
Template
VQHMM
FS
Speech
Transcriptions
logEn Zn
Temporal,
Spectral,
Cepstral,LPC
Features
d(X,Y)DTW
Word/
Sound
Models,
Templates
Dictionary
Syntax
Recognized
Input
TemplatesModels
12/28/2009
67
Statistical Pattern Recognition

The basic speech recognition task may be defined as follows:
a sequence of measurements (speech analysis frames)
on the (endpoint detected) speech signal of an utterance
defines a pattern for that utterance
this pattern is to be classified as belonging to one of
several possible categories (classes) (for word/phrase
recognition) or to a sequence of possible categories (for
continuous speech recognition)
the rules for this classification are formulated on the
basis of a labeled set of training patterns or models
The type of measurement (temporal, spectral, cepstral, LPC
features) and the classification rules (pattern alignment and
distance, model alignment and probability) are the main
factors that distinguish one method of speech recognition from
another
12/28/2009
68
Issues in Pattern Recognition
Training phase is required; the more training data, the better the
patterns (templates, models)
Patterns are sensitive to the speaking environment, transmission
environment, transducer (microphone), etc. (This problem is known
as the speech robustness problem).
No speech specific knowledge is required or exploited, except in the
feature extraction stage
Computational load is (more or less) linearly proportional to the
number of patterns being recognized (at least for simple recognition
problems, e.g., isolated word tasks)
Pattern recognition techniques are applicable to a range of speech
units, including phrases, words, and sub-word units (phonemes,
syllables, dyads, etc.)
Extensions possible to large vocabulary, fluent speech recognition
using word lexicons (dictionaries) and language models (grammars
or syntax)
Extensions possible to natural language understanding systems
12/28/2009
69

1. Fundamentals (Lectures 2-6)
speech production (acoustic-phonetics, linguistics)
speech perception (auditory (ear) models, neural
models)
pattern recognition (statistical, template-based)
neural networks (classification methods)
2. Speech/Endpoint Detection (Lecture 7)

algorithms
speech features
12/28/2009
70

3. Speech Analysis/Feature Extraction (Lectures 7-9)
temporal parameters (log energy, zero crossings,

autocorrelation)
spectral parameters (STFT, OLA, FBS, spectrograms)
cepstral parameters (cepstrum, -cepstrum, 2cepstrum)
LPC parameters (reflection coefs, area coefs, LSP)
4. Distance/Distortion Measures (Lecture 10)
12/28/2009
temporal (quadratic, weighted)

spectral (log spectral distance)
cepstral (cepstral distance)
LPC (Itakura distance)
71

5. Time Alignment Algorithms (Lectures 11-12)
linear alignments
dynamic time warping (DTW, dynamic programming)
HMM alignments (Viterbi alignment)
6. Model Building/Training (Lectures 13-14)
template methods
clustering methods
HMM methodsViterbi, Forward-Backward
vector quantization (VQ) methods
12/28/2009
72

7. Connected word modeling (Lecture 15)
dynamic programming
level building
one pass method
8. Testing/Evaluation Methods (Lecture 16)
12/28/2009
word/sound error rates

dictionary of words
task syntax
task semantics
task perplexity
73

9. Large Vocabulary Recognition (Lectures 17-18)
phoneme models
context dependent models
discrete, mixed, continuous density models
N-gram language models
Natural language understanding
insertions, deletions, substitutions
other factors
12/28/2009
74
Putting It All Together
12/28/2009
75
Speech Recognition Course

Topics
12/28/2009
76
What We Will Be Learning
speech production modelacoustics, articulatory concepts, speech production models

speech perception modelear models, auditory signal processing, equivalent acoustic
processing models
signal processing approaches to speech recognitionacoustic-phonetic methods,
pattern recognition methods, statistical methods, neural network methods
fundamentals of pattern recognition
signal processing methodsbank-of-filters model, short-time Fourier transforms, LPC
methods, cepstral methods, perceptual linear prediction, mel cepstrum, vector
quantization
pattern recognition issuesspeech detection, distortion measures, time alignment and
normalization, dynamic time warping
speech system design issuessource coding, template training, discriminative methods
robustness issuesspectral subtraction, cepstral mean subtraction, model adaptation
Hidden Markov Model (HMM) fundamentalsdesign issues
connected word modelsdynamic programming, level building, one pass method
grammar networksfinite state machine (FSM) basics
large vocabulary speech recognitiontraining, language models, perplexity, acoustic
models for context dependent sub-word units
task-oriented designsnatural language understanding, mixed initiative systems, dialog
management
text-to-speech synthesisbased on unit selection methods
12/28/2009
77

Lectures 1 Rabiner Speech Processing

Transféré par

Informations du document

Titre original

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Lectures 1 Rabiner Speech Processing

Transféré par

Droits d'auteur :

Formats disponibles

ECE 259B

Why Digital Processing of

The Speech Stack

Speech Algorithms speech-silence,

Apple Navigator -- 1988

The Speech Advantage

The Speech Circle

Determine correct number

I dialed a wrong number

Automatic Speech Recognition

Pattern Matching Problems

Basic ASR Formulation (Bayes Method)

Steps in Speech Recognition

Step 1-The Acoustic Model

Step 2-The Language Model

i where PL (w n | w n 1,w n 2 ,...,w n N ) is estimated by simply

Step 3-The Search Problem

Basic ASR Formulation

where X=X1,X2,,XN is the acoustic observation (feature

Speech Recognition Processes

Training algorithm => build models from

Method: Spectral analysis through either a

bank-of-filters or through LPC followed

Result: Signal compression where for each window of speech

samples where 30 or so cepstral features are extracted (64,000 b/s ->

Challenges: Robustness to environment (office, airport, car),

devices (speakerphones, cellphones), speakers (acents, dialect, style,

A mismatch in the speech signal between the

Traditional techniques for improving system robustness

Extract fundamental acoustic information in narrow bands

Methods for Robust Speech

phonetic labels (e.g., /s/, /aa/).

Hidden Markov Model (HMM): Statistical method for

characterizing the spectral properties of speech by a parametric random

Advantages: Powerful statistical method for dealing with a wide

range of data and reliably recognizing speech.

Challenges: Understanding the role of classification models (ML

Training) versus discriminative models (MMI training). What comes after

HMM for speech

Word model: is (/IH/ /Z/)i

Isolated Word HMM

Left-right HMM highly constrained state sequences

Map legal phone sequences into words

/d/ /ey/ /v/ /ih/ /d/

Several words may have multiple pronunciations. For example

/d/ /ae/ /t/ /ax/

How do you generate a word lexicon automatically; how do you add

Mapping words into

Deterministic grammars that are

Compute estimate of word probabilities (N-gram model). For example

How do you build a language model rapidly for a new task.

Combine information (probabilities) from the acoustic

Decoder searches through all possible recognition

How do we build efficient structures (FSMs) for decoding and searching

Unlimited Vocabulary ASR

For the WSJ 64,000 vocabulary, this results in a network

Weighted Finite State

Weighted Finite State

Moore's Law (hardware)

Identify possible recognition errors

A confidence score based on a hypothesis test is associated with

Rejection of extraneous acoustic events (noise, background speech,

Word Error Rates