Vous êtes sur la page 1sur 10

0

BASICDigital Signal Processing g g g


()

MathematicalBasics()

SelfIntroduction Self Introduction


Ph.D.,ElectronicEngineering, TohokuUniversity,Sendai,Japan(2004). M.Sc.,SystemsEngineering, M Sc Systems Engineering PakistanInstituteofEngineering&AppliedSciences (PIEAS),Islamabad,Pakistan(1999) (PIEAS) Islamabad Pakistan (1999)

FirstSemesterFY2011

ADAPTIVESIGNALPROCESSING
Muhammad Tahir Akhtar
AssistantProfessor,TheCenterforFrontierScienceandEngineering(CFSE), TheUniversityofElectroCommunications,Chofu,Tokyo,JAPAN The Uni ersit of Electro Comm nications Chof Tok o JAPAN WestCampus,BuildingW1,Room216 (akhtar@ice.uec.ac.jp) 2011 06 02

ResearchInterests:
Adaptivesignalprocessinganditsapplicationinactive noisecontrols(ANC)systems Independentcomponentanalysis(ICA)andblindsource separation(BSS) Biomedicalsignalprocessingandanalysis

2011 06 02

MuhammadTahirAkhtar

2011 06 02

MuhammadTahirAkhtar

AdaptiveSignalProcessing

Contents
Introductiontoadaptivefiltering p g Wienerfiltering&MethodofSteepestDescent LeastMeanSquare(LMS)algorithmandits L tM S (LMS) l ith d it variants,e.g.NormalizedLMSalgorithm RecursiveLeastSquares(RLS)algorithm KalmanFiltering anditsrelationshipwithRLS Kalman Filtering and its relationship with RLS algorithm(iftimepermits) Reference: f AdaptiveFilterTheory,4thEd.,bySimonHaykin
2011 06 02 MuhammadTahirAkhtar 2011 06 02 MuhammadTahirAkhtar

BASICDigital Signal Processing

ADAPTIVESIGNALPROCESSING 1. Introduction
1.1.AfewPreliminaries

IntroductiontoAdaptiveFiltering

IntroductiontoAdaptiveFiltering

AFewPreliminaries(1/6) A Few Preliminaries (1/6)


Filter:
Afilterisadeviceorsystemthatprocesses(orreshapes)the inputsignalaccordingtosomespecificrules togeneratean input signal according to some specific rules to generate an outputsignal.

AFewPreliminaries(2/6) A Few Preliminaries (2/6)


LinearFilter:
Afilterissaidtobelinear,ifthefiltered,smoothedor predictedvalueattheoutputofdeviceisalinear predicted value at the output of device is a linear combinationoftheobservationappliedtothefilterinput, otherwise,thefilterisnon linear. otherwise, the filter is nonlinear.

StatisticalInformation: l f Remarks:
Afilterthatreshapesthefrequencycomponentsofthe p , g, p input,e.g.,Lowpass SignalSmoothing Prediction,etc, Prediction etc
2011 06 02 MuhammadTahirAkhtar

Instatisticalapproachtothelinearfilteringproblem,we assumetheavailabilityofcertainstatisticalparameters (mean,correlationfunction,etc.)oftheusefulsignaland unwantedadditivenoise.

2011 06 02

MuhammadTahirAkhtar

IntroductiontoAdaptiveFiltering

IntroductiontoAdaptiveFiltering

AFewPreliminaries(3/6) A Few Preliminaries (3/6)


Stationary: y
Asignalissaidtobestationary,ifstatistical properties/parametersofthesignalremainunchanged (for properties/parameters of the signal remain unchanged (for certaintime).Forstationaryinputs,theresultingsolutionis commonlyknownastheWienerFilter,whichissaidtobe commonly known as the Wiener Filter, which is said to be optimuminthemeansquareerrorsense.

AFewPreliminaries(4/6) A Few Preliminaries (4/6)


ErrorPerformanceSurface:
Aplot ofthemeansquarevalueoftheerrorsignalversus theadjustableparametersofalinearfilterisreferredtoas the adjustable parameters of a linear filter is referred to as errorperformancesurface.

OptimalSolution:
Theminimumpointoftheerrorperformancesurfaceisthe WeinerSolution orOptimalSolution

ErrorSignal:
TheErrorSignal isdefinedasthedifferencebetween some desiredresponseandtheactualfilteroutput.

2011 06 02

MuhammadTahirAkhtar

2011 06 02

MuhammadTahirAkhtar

IntroductiontoAdaptiveFiltering

IntroductiontoAdaptiveFiltering

AFewPreliminaries(5/6) A Few Preliminaries (5/6)


AdaptiveFilter: p
Itisaselfdesigningfilter,whichusesrecursivealgorithm, p p g andperformsintheenvironmentwherecompletesignal statisticsarenotknown. Thealgorithmstartsfromsomeinitialconditions Inastationaryenvironment,afterrecursiveiterationsof y , thealgorithm,itconvergestotheWienerSolution. y g Inanonstationaryenvironment,thealgorithmthen offersatrackingcapability. Strictlyspeaking,theadaptivefiltersarenonlinear dueto theirdatadependency

AFewPreliminaries(6/6) A Few Preliminaries (6/6)


LinearAdaptiveFilter: p
Anadaptivefilteris saidtobelinear,iftheestimateofthe quantityofinterestiscomputedrecursively/adaptively(at quantity of interest is computed recursively/adaptively (at theoutputofthefilter)asalinearcombinationof the availablesetofobservationsappliedtothefilterinput. available set of observations applied to the filter input. Otherwise,itissaidtobenonlinear. ,

FilterStructures: Filter Structures:


Transversalfilterstructure(FIRfilter) IIRfilterstructure fl Latticefilterstructure Nonlinearfilterstructure(e.g.,Volterra filter)
2011 06 02 MuhammadTahirAkhtar

2011 06 02

MuhammadTahirAkhtar

10

IntroductiontoAdaptiveFiltering

11

AdaptiveFilter(1/3) Adaptive Filter (1/3)


BASICDigital Signal Processing

ClosedLoopAdaptation: Closed Loop Adaptation:


Measuretheoutcome(comparetothedesiredoutput)to optimizedtheperformance. ti i d th f

ADAPTIVESIGNALPROCESSING ADAPTIVE SIGNAL PROCESSING 1. Introduction


1.1.AfewPreliminaries 1 1 A few Preliminaries 1.2.AdaptiveFilter&AdaptiveAlgorithms

2011 06 02

MuhammadTahirAkhtar

2011 06 02

MuhammadTahirAkhtar

IntroductiontoAdaptiveFiltering

12

IntroductiontoAdaptiveFiltering

13

AdaptiveFilter(2/3) Adaptive Filter (2/3)

AdaptiveFilter(3/3) Adaptive Filter (3/3)

2011 06 02

MuhammadTahirAkhtar

2011 06 02

MuhammadTahirAkhtar

IntroductiontoAdaptiveFiltering

14

IntroductiontoAdaptiveFiltering

15

AdaptiveAlgorithms(1/4) Adaptive Algorithms (1/4)


Objective: j
Tosetthefilterparameters,insuchawaythatitsoutputtriesto minimizeameaningfulobjectivefunctioninvolvingthedesired g j g signal. Thearemanyrecursivealgorithms fortheadaptationof linearadaptivefiltering. Thechoiceofonealgorithmovertheotherisdetermined byoneormoreofthefollowingfactors.

AdaptiveAlgorithms(2/4) Adaptive Algorithms (2/4)


Missadjustment: j
Itisaquantitativemeasureofamountbywhichthefinal valueofthemeansquareerror,averagedoveranensemble q g ofadaptivefilters,deviatesfromtheminimummeansquare errorthatisproducedby theWeinerfilter.

Tracking:
Whenthealgorithmoperatesinanonstationary environment,thealgorithmisrequiredtotrackstatistical , g q variationintheenvironment.

RateofConvergence:
Numberofiterationsrequiredforthealgorithmtoconverge.

Robustness:
Smalldisturbancecanonlyresultinasmallestimationerror. Small disturbance can only result in a small estimation error
2011 06 02 MuhammadTahirAkhtar

2011 06 02

MuhammadTahirAkhtar

IntroductiontoAdaptiveFiltering

16

IntroductiontoAdaptiveFiltering

17

AdaptiveAlgorithms(3/4) Adaptive Algorithms (3/4)


ComputationalRequirement: p q
Computationalcomplexity: Thenumberofoperations(i.e.,*, /,+,,etc),requiredtomakeonecompleteiterationofthe ) q p algorithm. MemorySize: Whichisrequiredtostorethedataandthe program. Theinvestment requiredtoprogramthealgorithmoverthe computer.

AdaptiveAlgorithms(4/4) Adaptive Algorithms (4/4)


NumericalProperties: p
NumericalStability: Thisisinherentcharacteristicsofan adaptivefilteringalgorithm. NumericalAccuracy: Determinedbynumberofbitsusedin numericalrepresentationofdata samples,andfilter coefficients.

Structure:
Howthealgorithmcanbeimplementedinhardware? g p Forexample,simple,pipelined,complex,modular/non modular,etc.
2011 06 02 MuhammadTahirAkhtar 2011 06 02 MuhammadTahirAkhtar

IntroductiontoAdaptiveFiltering

18

IntroductiontoAdaptiveFiltering

19

AdaptationApproaches(1/4) Adaptation Approaches (1/4)


Stoc ast c G ad e t pp oac : StochasticGradientApproach:
Thecostfunction orperformancemeasure orperformance index is meansquareerror between filter output and desired ismean square errorbetweenfilteroutputanddesired response. Thefilterconverges(orattemptstoconverge)totheWeiner The filter converges (or attempts to converge) to the Weiner Solution. Theiterationprocedureforadaptationisbasedonthe The iteration procedure for adaptation is based on the methodofSteepestDescent.Thismethodisbasedon GradientVector,thevalueofwhichdependsontwo Gradient Vector the value of which depends on two parameters: C Correlationmatrixoftapinputvector l ti t i ft i t t crosscorrelationvector betweenthedesiredresponse andtapinputvector d
2011 06 02 MuhammadTahirAkhtar

AdaptationApproaches(2/4) Adaptation Approaches (2/4)


StochasticGradientApproach: Stochastic Gradient Approach:
Inmanypracticalsituations thesecorrelationsarenot available,andthegradientvectoriscomputedfrom instantaneousvalues ofcorrelationmatrixandcross correlationvectorcomputedfrompresentdata. Thus,recursiveequationforstochasticgradientbased Thus recursive equation for stochastic gradient based algorithmis:
updated value Old value of of tap - weight = tap - weight vector vector Tap - input (Error Signal) + (step - size ) vector

2011 06 02

MuhammadTahirAkhtar

IntroductiontoAdaptiveFiltering

20

IntroductiontoAdaptiveFiltering

21

AdaptationApproaches(3/4) Adaptation Approaches (3/4)


StochasticGradientApproach: Stochastic Gradient Approach:
Stochasticgradientapproach +Transversalfilterstructure LeastMeanSquare(LMS)Algorithm ( ) l h Stochasticgradientapproach +Latticefilterstructure GradientAdaptiveLattice(GAL)Algorithm Gradient Adaptive Lattice (GAL) Algorithm Thisstochasticgradientapproachbasedadaptivefiltersare This stochastic gradient approach based adaptive filters are widelyusedforsimplicityandrobustperformance. Thedisadvantage: slowconvergencespeed Th di d t l d

AdaptationApproaches(4/4) Adaptation Approaches (4/4)


LeastSquare(LS)Estimation: LeastSquare (LS) Estimation:
Costfunctionissumofweightederrorsquare IterativeprocedureisbasedonMethodofLeastSquares d b d h d f Twoapproaches BlockEstimations RecursiveEstimations ecu s e st at o s RecursiveLeastSquares(RLS)algorithm KalmanFiltering:
updated value Old value of of the state = the state
2011 06 02

Kalman Innovation + Gain Vector


MuhammadTahirAkhtar

2011 06 02

MuhammadTahirAkhtar

22
BASICDigital Signal Processing

IntroductiontoAdaptiveFiltering

23

FourTypesofApplications(1/5) Four Types of Applications (1/5)

ADAPTIVESIGNALPROCESSING 1. Introduction
1.1.AfewPreliminaries 1.2.AdaptiveFilter&AdaptiveAlgorithms 1 2 Ad ti Filt & Ad ti Al ith 1.3.Applications

A.SystemModeling(Identification): y g( )
ToIdentifythemodeloftheunknownplantbasedon y p theknowledgeoftheplantinputx andoutputd TheAdaptivefilteradjustsitsparameterstomatchor identifyunknownsystems

Applications: SystemIdentification Layeredearthmodeling

2011 06 02

MuhammadTahirAkhtar

2011 06 02

MuhammadTahirAkhtar

IntroductiontoAdaptiveFiltering

24

IntroductiontoAdaptiveFiltering

25

FourTypesofApplications(2/5) Four Types of Applications (2/5)

FourTypesofApplications(3/5) Four Types of Applications (3/5)


B.InverseModeling:
Theadaptiveprocessortriestoundotheeffectsofa plant. Itattemptstorecovertheoriginalsignal.Alsoknownas deconvolution Difficulty: Noiseinthemeasurements.(Randomnessinplant) h ( d l ) Applications: A li ti Equalizationofcommunicationchannel BlindDeconvolution Blind Deconvolution

2011 06 02

MuhammadTahirAkhtar

2011 06 02

MuhammadTahirAkhtar

IntroductiontoAdaptiveFiltering

26

IntroductiontoAdaptiveFiltering

27

FourTypesofApplications(4/5) Four Types of Applications (4/5)

FourTypesofApplications(5/5) Four Types of Applications (5/5)


C.SignalPrediction:
Predictsthecurrentsignalx(=desiredsignal)basedonthe pastx(delayedx)

Applications: Applications Predictivecoding(speechcoding) Autoregressivespectralanalysis Autoregressive spectral analysis


2011 06 02 MuhammadTahirAkhtar 2011 06 02 MuhammadTahirAkhtar

IntroductiontoAdaptiveFiltering

28

IntroductiontoAdaptiveFiltering

29

FourTypesofApplications(5/5) Four Types of Applications (5/5)


C.InterferenceCancellation:Tocancelunknown
interferencewhichismixedwiththedesiredsignal.

ApplicationExample EchoCancellation(1/3) Application Example Echo Cancellation (1/3)


A.TelephoneLineEchoCancellation:
Echocausedbyimpedancemismatchatthehybridconnection (connectsa2wirecircuitwitha4wirecircuit).Causesvoicesignalto bouncebacktothetalker.Thisreflection,alongwithpathdelays, bounce back to the talker This reflection along with path delays causestalkeratbothendstohearanecho. ThisisespeciallybadforVoIPbecausethepathdelayisusuallyalot longerthatinPOTS(PlainOldTelephoneService). Ifthedelaybetweenthespeechandechoisshort,theechoisnot noticeable,butperceivedasaformofspectraldistortionor noticeable but perceived as a form of spectral distortion or reverberations.If,ontheotherhand,thedelayexceedsafewtensof milliseconds,theechoisdistinctlynoticeable.

Sincev (n)andv (n)are comingfrom same noisesource,henceare Since v1(n) and v2(n) are coming from same noise source hence are correlated witheachother p g , y( ) ( ), ( ) ( ), Whenadaptivefilterconverges,y(n) v1(n), andhencee(n) s(n), the desiredsourcesignal. ThisschemecanbeusedforAdaptiveNoiseCancellation,whenan appropriatereferencenoisesignalisavailable.
2011 06 02 MuhammadTahirAkhtar 2011 06 02

MuhammadTahirAkhtar

IntroductiontoAdaptiveFiltering

30

IntroductiontoAdaptiveFiltering

31

ApplicationExample EchoCancellation(2/3) Application Example Echo Cancellation (2/3)

ApplicationExample EchoCancellation(3/3) Application Example Echo Cancellation (3/3)


B.AcousticEchoCancellation:
Acousticechoarisesinhandsfreecommunicationsystems.

y(n)=Farendspeechsignal(referenceinputtoadaptivefilter) r(n)=echooffarend(reference)signal x(n)=nearendspeechsignal s(n)=nearendinputsignal=r(n)+x(n)=(desiredresponseforAF) whenadaptivefilterconverges,e(n) x(n) h d i fil ( ) ( )


2011 06 02 MuhammadTahirAkhtar 2011 06 02 MuhammadTahirAkhtar

32
BASICDigital Signal Processing

WienerFilters

33

ReviewofStochasticProcesses(1/6) Review of Stochastic Processes (1/6)


Stoc ast c/ a do Stochastic/RandomProcess: ocess:
Itdescribestimeevolutionofsomestatisticalphenomenon accordingtoprobabilisticlaws. according to probabilistic laws

ADAPTIVESIGNALPROCESSING 1. 1 Introduction 2. WienerFilters


2.1.ReviewofStochasticProcess

TimeSeries: i i
Oneparticularrealization ofadiscretetimestochasticprocess iscalledadiscretetimeseriesortimeseries: [ ( ), ( ), [u(n),u(n1),,u(nM)] , ( )]
Presentvalue Pastvalues

2011 06 02

MuhammadTahirAkhtar

2011 06 02

MuhammadTahirAkhtar

WienerFilters

34

WienerFilters

35

ReviewofStochasticProcesses(2/6) Review of Stochastic Processes (2/6)


StrictlyStationaryProcess:
Forastrictlystationarystochasticprocess,itsstatistical p parametersareinvariant toashiftoftime. Meanvaluefunction: (n) = E{ u(n) } (n)=E{u(n)} Autocorrelationfunction: r(n,nk)=E {u(n)u*(nk)} r(k);r(0)=E{|u(n)|2 } Covariancefunction: c(n,nk)=E[{u(n) }{u(nk) (nk)} c(k); c(0) c(0) =2

ReviewofStochasticProcesses(3/6) Review of Stochastic Processes (3/6)


WideSenseStationaryProcess:
Astochasticprocessiscalledwidesensestationary,ifits statisticalparametersarefound tobeinvariant toashiftof time.

MeanErgodicTheorem: Mean Ergodic Theorem:


AstochasticprocessissaidtobemeanErgodic,iff,
2 lim E ( n ) = 0 N

where ( n ) =

1 N

N 1 n =0

u(n )
(n = E ( ( n ) )
MuhammadTahirAkhtar

istimeaveragedmean,and is time averaged mean and


2011 06 02 MuhammadTahirAkhtar 2011 06 02

WienerFilters

36

WienerFilters

37
u(n) u ( n 1) u( n ) = M u ( n M + 1)

ReviewofStochasticProcesses(4/6) Review of Stochastic Processes (4/6)


Signal~UnderlyingProcess:
Statistics: Scienceofinterpretingnumericaldata,suchas acquiredsignal Probability: Usedtounderstandprocess thatgeneratethe P b bili U d d d h h signals Example: C id fli i E l Considerflippingacoin1000times,togeneratea i 1000 ti t t binarylevelrandomsignal,head 1,andTail 0. Theprocess creatingthesignalu(n)={1,0,1,0,0,0,1,1,}has The process creating the signal u(n) {1 0 1 0 0 0 1 1 } has anexactprobabilityof0.5,however,thesignalacquired by thisprocessmaynothaveexactmeanof0.5. this process may not have exact mean of 0.5. Theprobabilitiesofunderlyingprocessareconstant,but q g y g , statisticsoftheacquiredsignalmaychange,eachtimethe experimentisperformed.ThisiscalledStatisticalVariation, fluctuation,ornoise.
2011 06 02 MuhammadTahirAkhtar

ReviewofStochasticProcesses(5/6) Review of Stochastic Processes (5/6)


CorrelationMatrix:
Thetapinputvectorisdefinedas:

Thecorrelationmatrixoftapinputisdefinedas:
u(n ) u ( n 1) H R = E u( n ) u ( n ) = E u ( n ) u ( n 1) L u ( n M + 1) M u ( n M + 1) r (1) L r ( M 1) r ( 0) r ( 0) r ( M 2) L r ( 1) = M O M r ( M + 1) ( M + 2) L r ( 0)

2011 06 02

MuhammadTahirAkhtar

WienerFilters

38

39

ReviewofStochasticProcesses(6/6) Review of Stochastic Processes (6/6)


PropertiesofCorrelationMatrix:
1. 2. 3. 4. 4 Hermition Toeplitz p Almostalways+ve definite,andalwayssemi+ve definite. Canbecomputedrecursively,forexample Can be computed recursively for example

FurtherReading:
Chapter0and1ofTextBook Chapter 0 and 1 of Text Book

NextLecture:
ContinueourdiscussiononWienerFilters MayhaveaQuiz aboutTodaysLecture

where

r ( 0) RM +1 = r

rH RM

r H = [ r (1) r ( 2) L, r ( M )] ), ),

QUESTIONS?
MuhammadTahirAkhtar 2011 06 02 MuhammadTahirAkhtar

2011 06 02