Chapter1

Discrete Sources and Entropy

sourcedoesnot emita character attimeindex/,wesaythatthesourceproducesanull characterat time r. A null character,therefore,is a kind of "virtual" symbol denoting the absenceof a "real" symbol at /. This convention allows many discrete-timeasynchronoussources to be approximatedassynchronous sources. The symbols emitted by a source in a physical communication systemmust be representedsomehow.In digital communication systems, this is typically done using binary representation. For example,a sourcewith M:4 might have its symbolsrepresentedby a pair of binary digits.In this case, aomight be representedas00,a, as01,and so on. It is common to refer to symbolsrepresentedin this fashion assourcedata. Information theory makes an important distinction between data and information. T}aetwo conceptsare not the sameand, generally,data is not equivalentto information. To seethis, consideran information sourcethat has an alphabetwith only one symbol.The sourcecan emit only this symbol and nothing else.The representationof this symbol is "data" but, clearly, this data is completely uninformative. There is no "surprise" whatsoevertied to this data. Since information carries the connotation of uncertainty in what will come next, the information contentof this sourceis zero. The information content of a sourceis an important attribute and can be measured. In his original paper (which founded the science of information theory), Shannongavea precisemathematicaldefinition of the average amount of information per sourcesymbol. This measureis called the entropy of the source and is conveyed defined as H(A) : p*log2Q,lp). t.2.5

ltu

Ltr' dt cfor

frc
br
gt l:l n
fl:

pcn

Ca &tr

st
lrrT (n'

m:9

Equation 1.2.5tells us that the information content of a sourceis determinedby the individual symbol probabilities. Note that for the case of our uninformative M:7 source, the probabilityof its symbolis unity and fI: 0. In the definition given above for entropy,we take the logarithm to the base2 of the reciprocalof the probabilities.Itis alsopossible to define entropyusingany other base logarithm. For entropy defined as in Equation 7.2.5,the unit of measure of entropy is called a "bit." If we were to replace the logarithm to the base2 in Equation 1.2.5 by the natural logarithm(log to the basee), the entropyis saidto be measured in "natural units" or "nats." The most commonpracticeis to measureentropy in bits.

rh (d erri pd

cwilfi 1ffi tllff iltiilttilmt]iliIttllitl

What is the entropy of a 4-arysourcehaving symbol probabilities P, : {0.5,0.3,0.15,0.05}? Solution: H(A): -f .15log2(100/15) :1..6477bits. .5Iog,2(2) * .3tog2(1013) +.051o92(20) Sir dt
EE

(Recall that logr(x) :In(x)lln(2)

e$ ftc m

,|.2 Section

Discrete Information Sources and Entropy 7

The first term on the right-hand side of Equation L.2.7 issimply H(A).The sec_ ond term is cailed a conditio;ar entropy.It is the uo""roiniii"noopy) 'rurr \vu of B given A andiswrittenH(BlA).Thus,wemay*rit" . H(C) : H(A,B) : H(A) + H(B]A). 1.2.8 (Note the similarityof Equation 1.2.g with theJogarithm . of p,,',,: ' OO,p).Itis a simple t matter to prove that H(A, B): H(B) + "r 77(Ap) ai well. Now, if B is statisticallyindependentLf a,

-, H(BtA)

In,4oo,tosr(rlp1,): )o,\
'l

nltos(t1p):H(B).

In this case, the total entropy is simply the sum of the entropiesof the two sources. on the other hand, if B is depende"t on A, ibe shown inat n1q,! <H(.8). (The proof of this gives rise to a standard "uoin information theorem theory that says,,side information never increases entropy.':)Therefore,

H(A,B)=H(A) + H(B), with equality if and onlyif A and.Barestatistically independent.

r.2.,

Many computer backplanes and memory systems gqnlor a parity bit as asimplemeansof error detection. Let A be an informationtoui."'*i,i alphabet equally probable and let B: 1= l0,r,z,3l.Let eachsymbola be {0, 1} be a parity lenerator with

if u' , : { o (1 if
WhatareH(A),H(B),and H(A, B)?

a:oora:3 a:'l.ora:2

Solution: Fromthe d".fiiili9" of_entropy, H (A) 4 . ! . logr(4): 2. Likewise, the two sym_ bo"ls in B eachhaveprobability _gowever, 0.5,t. ,{'alll. trr"e Lnoitio"al probabilitie spr(bla) : 1, pr(110) Pr(010) : 6, : s, pr(111) Pr(011) : 1, : g, pr(tl2) : 1, Pr(012) : 1, pr(113) Pr(013) : 6.

Therefore.

H(BtA): j o,j o, litogze/ np).

'rlog(.r) : 0,thisexpressionevaluarestof(81,4) *4.0.2s.(1 .logr(1)-0.togr(0)): :t":lt$ 0 whichsimplysays thatB is completely determinedby.4. Therefore, H ( A , B ): H ( A ) + H ( B I A ): 2 * 0 : 2 . Source B contributes no informadon to - the -' compound signair. (Source B is saidto be "redundant.")

Chapter1

Discrete Sources and Entropy

1.2.3 Entropyof SymbolBlocks and the ChainRule Another useful applicationfor entropy is in finding the information content of a block of symbols. Suppose s,oTce a froJyl a seqye.n:e of n symbols (so, ? s1,... , s,_,) with eachs, drawn from alphabet 1. The entropy of this block is denoted as H(A* A.,, ... , A,_.t) where the arguments -4, indicate that trr" ,y-uoi at inoex 'oepe"nds r was drawn iiom alphabetA' This Fl ii a joint its vatue on the joint probablity of thesymbols "ttiffiin"e in
as

Srtrc
teanr to

tu

emrop of fu

theblocl.rf we,.s'r'r1,*r*;;il5i;;, :,r,... , .rn_1) asa compound symbol, similarry to ther preceanfJis"ussron, -----o$rvveusrv'' we rv'lil ws can use use oui our prevlous result to express the entropy
H ( A 0 ,A 1 , .. . ,A n _ t ) : H ( A o ) + H ( A 1 ,A 2 , .. . ,A n _ t l Ao).

.6I,, {",

joint entropv conditioned il:tJffi',',11Jfiil#n,t1fl:'",X113,::1""',' on Ao. rt "

argument inductivelv andplugging theresutts back intorhefirstequa,Tfi?Jt"lt#is
H(A0,A1,...,An_,) : H(Ao) + H(AlA0) + H(A21A0,A) +...H(A,-11A,,...,A,_2). This result is known y-tlelhain rule for entopy. Since'F(BrA) = H(B),one immldiate consequence of Equation 1.2.10 is . .,An_lA;) : H (Ay A2,. H (AlAs) + H (A2,.. ., An_l A0, A1).

Houwer- Itrrrq deuoe rrtnsd tbe same sfta 6qnel 3ry1 rn slr qlntiok -d uncertainr-T ia t

urlrcl
Rigoroul_r$or Solurli* fr

1.2.10

H (A o ;'.,A n- r ) = 2u( e) , ,=0

n_I

where pi-,rn, _.i*r

7.2.1\

Pu*-'-'.t.: l-dcl

with equality if and only if all of the symbolsin the sequence are statisticallyindependent' (A sourcewith this p.op"rtt;:;]i; a memoryress source;it doesn,t,,remember" from one time to ttt^"tt"i*ti,;yd;i, it haspreviousryemitted.) whle developedr.2.r0 and 1.2.11 we have assumtd;;;".e source.4 emitreda, of the symbors, theseequationsare also true ir rrr" ,yii.i."are emitted by differentsources. when alr of the symbotrur" the samei"r"r'nutio" source, probabilities of the sourcedo and if the ",nitiJ;;n, not_cha"!" ou"r ai.e, then att oi trr" .Flterms summation under the in I.2.1I are equalto the eritropy of source A, and.we have H(Ao;", An-1)< n. H(A), L2.I2

-r_

unal.Cq

For an inforrnati possible?

l|tff1Tt*:ffi1Jr:y.'jr'.T

svmbols are statisticallv independent. Equality inr.2.r2

Solution: The d must lie in the ra each term in the ample 1.2.2. *-e sa withp^:1 ,{- e

:fi:'Jfrfi ;t-dffi;"

Suppose a memoryress sourcewith,4 : {0,1}having equal symbolprobabilities emits a

sequence aseventh svmlor istransmitted which isthe }",i::llt"::1:11!+oi'1'.^'yr*; excrusive-or orihe symbors emitteo by "".,l#jr',?:',:ilfi1flij|i:L'|:1..,",1e

Therefore.

Section 1.2
5

Discrete Information Sources and Entropy 9

Solution: Letb:

> es, where ) O denotes a summationmodulo 2 and s,e,4. The t=0

entropyof the sequence is H(Ao,Ar, '.. ,As, b). By the chain rule for entropiegthis.is equiva_ lent to

H (Ao, A1,..., As,b): H (Ai + H (ArtAi + H (A2t A0, A1) + ...H (btAo;.., As).
if the svmbolsare statisticallv independent. (staristicalindepenI:y::::yJ*l:)^-r,!,) oencemeansthat Aitells.usnothing aboutA,it i + j). Sincethe first six symbolsare drawn irom tne samesource,l, the first six_terms on the right-hand side of the foregoing expressionare all e.qual, and since Po:Pr:0'5, theseterms sumlo 6. In the last term onihe right-handside,the six symbols Ao;",As completely determine b so knowledge of these six symbols leaves no uncertainty in b and H(blAo,..., Ar) : 0.Therefor e,H(Ao,Al, ..., Ar, b1: 6.

Rigorously showthat H(blAo,... ,Ar):

0 in exampl e 1..2.4.

Solution: From the definition of entropy,we can write

L

H ( b t A o , . . . , A ) : ) Pu,o,,.e, log2(1' otoo, ..,a,), , fP

b=0

where p6140;..,dr.is. th: conditional probabilityof b givenAo;,.,As.Now, for any givenAoi, ,,As, -p61a,;..,;::0;; b is either 0 or 1,(depending on h,... 0,4), so we have either Pbt.t;",Ar: L, dependingon the value of D in the summation. Therefore, H(blAo,".,A): -0.log2(0) * 1.log2(1):0 * 0:0.

For an information source having alphabet '4 with l,4lsymbols,what is the range of entropies possible? solatton: The definition of entropy is given by Equation 1.2.5. Sincethe symbol probabilities must lie in the range 0 = r. for eveiy symbor ,iogr(1,1p^)> 0 for every symuot.Therefore, 4-= each term in the summation in Equation i.Z.S i, non_negative. Therefore, H(A) -_0. In Ex_ ample L.2.2,wesaw that the entropy is maximizedwhen ev'ery symbol in a is equally.probable with p ^ : I I lAl.Consequently,

H(1.7'
Therefore.

: log,(rAl). irilog2(l'41)

0- H(A) <log2(t,41).

I.2.t3

10

ChapterI

Discrete Sources and Entropy

1.3 SOURCE CODING

'1.3.1 Mapping Functions and Efficiency Example 1'2'6 showed that arbitrary information sourcescan have a considerable range of possibleentropies. The entropy of a sourceis the averageinformation carried per symbol' Sinceeachsymbol will either be transmitt"J-1intrr" caseof a communication system) or stored(in the caseof a storage system), and sinceeachuseof the channel (or eachunit of storage)has somearsoCiat"d .ort, it is clearlydesirable to obtain the most information possibleper symbol (on the uu"*g"). If we have an inefficient source'i'e', H(A) < toglia)' our systemcan be made moie cost efTective through the useof a sourceencoder. A sourceencodercan be looked at as a data processing elementwhich takesan input sequence of so, s1,''' symbols s, e ,4 from the informationsourceand produces an output sequence s6,si,"' usingsymbolssi drawn t o^ i ria, afuhabetr.'rrr"s" ,y*_ bols are calledcodewordl ]\e objective of the encoderis to processthe input in such a way that the averageinformation transmitted (or stored) per channel ur" approaches H(l) (If ,4 is.a compoundsymbol,,u"h u, in-"ru*pt" "1o."ry r.2.3,H(A)would actuallybe a joint entropy.) In its simplest form, the encodercan be viewedasa mapping of the sourcealphabet A to a codealphabetB. Mathematically, this is repres"ni"a u, C:A--+B, since the encodedsequence must eventuallybe decoded, the function c must be invertible'This meanstlere must existanothei function c-1 suchthat if C(a) ,->b then c-'(b) e a. This.is possibre o"ry ii zloy b i, unique,i.e., for every be.B, there is ? exactly one a e A such that C(a) rt 6, uo! for every'ae A,'thereis exactly oneb e B

[d0*htur h;n'ffirtm,r

[fufrriililS
l--qm" @@

wd.lttud fu 0ffi%, C
ffituruund$d

sqp

lfue r ffiru

,n*O;LmdffiC

ddlpffi dcnmudfr$ "il crhe{deM

:::lrtlffr'(b)

kt-{ hc a "fl-m -4 6 5lrmsmnnm hte f:n,q,nI riHc

thatevery i e Ahasa non-zero probability l.(we areassuming of

The sourceencoderactsasa data-processing unit. In this section, we will discuss someof the detailshow data processing effectsinlormation.In particular,we must be concerned with the.relationships betweeninformation carried by code words b and information carried by sourcesymbolsa. Beforediving into itrese details,however,it is to illustrate the function of a source by means of the following J"""fJlt" "rr.6d",

Let A be a 4-arysource with symbolprobabilities asgivenin Examplel.2.l.Letc be an encoder which mapsthe symbolsin,4 into stringsof binary digits,asfollows: po:0.5 pr:0.3 pt:0.L5 pr:0.05 C(a) ,> 0 C ( a r )e > l e C ( a r )r > l l g C(ar) r--> ll1

Since the s HlAx A r::fi ber of bits per n

The efficiencl i mitted bir_sare r

Section i.3

Source Coding

11

Let L- be the number of binary digits in code word b^. rf the code words are transmitted one binary digit at a time, the averagenumber of transmifte; binary digits p", loo" word is given by

L : 9^o*t*: .s(1) + .3(2) : 1..70. + .1s(3) + .05(3)

From Example1.2.r,H(A):1,'6477 bits.The efficiency of rhis encode r is H (A)lL : 0.96924.rt the source encoder were not used,we would need two binary digits to represent each source symbol. The efficiency of the uncoded source would be n1A11z":0.g2385.put another way, almost 18% of the transmitted binary digits for the uncoded source would be redundant, whereasonly about 3o/o of the transmitied b-inary digits tio- a ur",Juiaurr,.

More sophisticateddata processingby the encoder can ^ also be carried out. Supposethat symbols emitted by sourc! A *"r" first groupeJ i.rto ordered pairs (a,,a) and the code wordsprodu."a uy the encoder *"r" bur"d on thesepairs.The set of all possiblepairs (au a) is called the Cartesian product of set A with itself and is denotedby AX A. The encoding process then becomes a functionof two variables and canbe denotedby a mappin g CA X A_+ B or by a function C1a)a,5,, U.

L:t A be a 4-atymemoryless sourcg probabilities yith the symbol givenin Example 1.3.1. Since 1 is a memoryless source, the probability or anygiven pair of syiborsis giventy pr(o,,-if: Pr(a,)'Letthe encoder mappairsbt symuots intothe codewordsshown in the follovling i:{i!,

\aaal
As; Ag a6 a1 ao, a2 4g743 ay aj ay a1 ay 42 ay a3

Pr(a,,a)

b.

(oro) az, ao
421 41

Pr(ai,a)

.25
.rl

.075 .025 .15 .09 .045 .015

00 100 1100 11100 101 010 0110 111100

az, az a2;a3
431 dg 431 A1

ay az
431 43

.075 .045 .022s .0075 .025 .015 .0075 .0025

1101 0111 111110 1111110 11101 tr1.101, 11111110 71.1111,1,r

Since thesymbols fro-T1 areindependent, weknowfromEquatio nL.2.r2that theentropy f"r J;h;t;bol, theaverage !1(A\9:zH(A):3.2954.Sinc" 4ry!Pr(b^1:pr(ai,aj) ' "num_ berof bitsper transmitted code wordi;

T: 2P't' m:0

- 3.3275.

The efficiency of the encoderis therefore H(A X A)/L : .99035. Less than1% of the transmitted bits are redundant.

!f

Chapter1

Discrete Sources and Entropy

If this result seemsa bit unsettling,,it should.It appearswe have gotten ,,something for nothing" out of the encoder. However, t"t'r'irrirrt about what the result means'H(B)> 1{'4) saysthere is more.uncertaintyin B than there is in,4. where did this uncertainty come irom? t d: discussion;t;;; ie saia we courd alrow the encoderto pick from a selection of b symbols with aat random.If the encoderis 'otossing a coin," the extra uncertainty in "r*"i","4 a comesfrom this random process andnot from A. There is indeed more information in r u* ,t ,F(B) _ u67 i" useless in terms of getting more information " "*cess out.i;. Th;;rcoder is like the town gossip:Along with the factsiome other unretatedor specurative tidbits. 1.3.3 A Brief Digression on Encryption All of this is not to say that such an encoder is useless. The difficulty and expenseof building a decoder,isdirectry r"tut"o,.to th"^";;;;'ot'rt" signarbeing decoded. Suppose you wanted to makeit.noi" aini"ult for asriJopv rrriro party to decodea private message betweenyourserf and a_trienJ. o;;;;;;Irdo so would be to consrruct an encodersuchas that in the example ;il;".;;;;e;;

drc d eflifr Futfirrer

"Arry tuf,Pcl h &lllg e nr

$ldea$t hrimr

information,, H(B)- H(A),is useress your friend but courdbe terrib-ry to frustrating ," ,*rJr.pp"r. so,of varueto you). 1urra,

areoftenc atle d en* yption encoders.The ir functi,o;, t,.o*-"'":lt*"[t":T H] J tT#; taintyto thetransmittid m"rsage i.e., raise , trre^entrofy, *itrroutadding to the edge conveyed knowrby the tn"rrug"lfh"'"*".,

srtfr:ctto-r-ml

et"'j'nfJir'" or ,'l."i"i and rhe output ffifllftiif:li,li;1Ji',iiX|u"-.' "n,,opv ""tput

a(r): {10100101 1000001 100111011} andtheinitialcontents of theregisters arellt =0) : 5 ( = 101). fiT:'#rr*:Jl,.oo"t is a finite statemachine. Bv directexamination, the outputsequence

Let the information source havealphabet A:1g,1].with po=pr:0.5.ThenH(e1=1. y", j' encoder c havealphabet u"Jr"t ,lj irr" haie uinary {:J0, !, b : (brbrbir. The encoder representation "t"-"nir'ot., ii shown rri rr^g*" 1.3.1 berow wh;;;.,addirion,, brocks are

In the stea{*r elementsof .8 sri

The enco&rof Frr

of 8from t rry

elemens of A ril pseudorandon m sequenceb fouod fr

{E}
This exaryf is too simpleNot

: [o t ol . l-ol b(I +1) Lr3 ;Jt.''Li-1""'

followed b-v either "crackin-e- this 69 reverse-engin6pain example can be er ble. We shall sa1-m

1.3.4 Summary of Secti

bo
bl

Figure 1.3.1: Encoder for Example 1.3.3

L,et'ssummarizec forming the infun to improve the trar dancy in the souro

Section1.3 whereall additions are modulo2.Let ns: pr(D :0),nt: probability staleasthe vector pa :

Source Coding

15

pr(b-: 1), etc.,and define the

rr... ntlr. Lno time t, Pugivesthe probability of the encoder being in each of the eight possiblestates. 1' Ty By drawing a state diagram for the encoder of Figure 1311,we .un .hor" that the probability stateas a function of time is given by

Ps(t + t1 :

.50000.500 .50000.500 0.500.5000 0.500.5000 0 0.50 0 0 0.5 0 0.50 0 0 0.5 0 0 0.50 0.50 0 0 0.50 0.50

PoU),

subjectto the constraint

7

t-v

)o'

= t'

p"(t * 1): puQ).Bydirect In the steady_state, substitution, we seethat rr,:'l.f8,for all elementsof B satisfiesthis condition.Tirerefore,

/ H(B): -)r1tog2@):3
j:o

> H(A).

The encoderof Figure 1''3.1 is a pseudorandom number generatorwhich generates all elements of I from 1 through7 if input a;.O,-with equiprobable a10 andr: i,ih"'"n.oder generates all elements of B with eqpal probability. The increase in entropy is due to the memory of. the pseudorandom number

sequence isround o"-,f;:*'#_:i;ff#,11ffiff:[t'J

given above, rhe encoder's output

4 , 2 ,5 0 L , 2 i D ) : { 0 ,0 , ! , 2 ,4 , 2 , 4 , 3 , 6 , 66 , , 7 , 50 , ,1 ,3 ,6 ,6 , 6 I . This examplewould not make a very good encryption code because the encoder is too simple' Note, for instance, that 0 is at-ways roiloieJ by either 0 or 1., 2 is always followed by either 4 or 5,and so o".tl .*p"ri in would haveno difficulty "cracking" this code by compiling a list of what "".tyption coae woras follow each other and reverse-engineering the encoder(try it!). However,the basictheme outlined in this examplecan be expandedon to create an encryption systemwhich is rather formidable' we shat saymore on the subjectof encryptio" i" cituft., s. '1.3.4 Summaryof Section.1.3 Let's summarizewhat we've learned in this section. Source coding consistsof transforming the information source'salphabetto a code word alphabet.one usefor this is to improve the transmissionefficiencyby data this application,redundancy in the source message "ornpr"rriorr.In is removed so that the averagenumbei of bits actually

Section 1.4

Huffman Coding 17

words in, a dictionary,your computer would probably deliver the secondpunctuation above'You would have to add iome rules governing sentencestructure to delineate betweenthe two punctuationsgiven above.Suppose yiu did this.consioer the sentence TIME FLIES LIKE AN ARROW (noun) (verb) (predicate). If your program could parseTIMEFLIESLIKEANARROW tencestructurerule given above,what would it do with using the example sen_

FRUIT FLIES LIKE A BANANA (wouldn'r youif you were a fruit fly?) ft9 Fnglish languageis not generally self-punctuating . because it containssufficient ambiguitiesthat somemessages could be correctly pu.i"o irr-t*o different ways IF I WANTED To 'ICK oNE vs. IF I WANT ED To 'ICK oNE and' in other cases, the rules connectingpossiblewords to valid sentences are complicated.It is for this reasonthat we ur" ,fu""r, commas, and other punctuation marks in written language. A prefix code is a -code having punctuation built in to the structure (rather than added, in usingspecial punctuatioi symbols)This is accomflisheouy designing the codesuchthat no codeword is a prefix of someoiher (longer) coie woro. Suchcodesare saidto satisfytheprefix conditioi.They are alsosaid tobe instantaneously decodable.

For the codein Example1,.3.Z,decodethe sequence 10000011 11 1111011 101 assuming the codewords are transmittedbit_seriallyfrom left

to rieht.

s.olutilry Scanthe sequence from left to right, and place a commaafter every string of bits that is a valid codeword. observe that neither r noi ro is a valid ,oo" *orJ, Lut 100is a valid code word. Parsingthe sequence in this way,we get

100, 00, 0111, 111110, 11101,

which decodes as
AO A1 A0 AO A2 AI A2 A2 As Ao.

Notice that this parsingis unique.For example, there are no codewordssuchas 1000or 01111. Any code which satisfiesthe piefix condition is structured so that whenevera string of n bits is is no possibitity of some otr,", 1r,ong".)code word which l:::::i:1tl::j".:.^d," T::d,"jhgre b"gt*f the sameway as the first recognizable code *;;J;;, t";J;;;

1.4.2 Construction of Huffman Codes Huffman codes are constructed usine a tree-building algorithm. The algorithm is defined in a manner which quaranteesthe code will satisfy the prefix coniition. The

18

Chapter 1

Discrete Sources and Entropy

first step in the procedureis to list the sourcesymbolsin a column in descending order of probability.This is illustrated in Example r.4.2.The sourcesymbols form the leaves of the tree.In Examplel.4.2,symbolaowith probabilitypo: 0.j is at the top of the col_ umn,and symbolarwith pr: 0.05is at the bottom. Step2 of the algorithm is to combinesymbols, starting with the two lowestprobability syrnbols, to^form a new compoundsymbol.In Examlb r.4.2,symbols a.r'and. a, are combinedto form compoundsymbol ararwithprobabilityp pr* pr:\lZO.ft ,r:l combining of two symbolsforms a branchln the tiee with stemsgoing " from the compound symbol to the two symbolswhich formed it. The top stem is labeled with a ,,0,, and the bottom stem is labeled with a "1-".(We could equally well label the stemsas "1"'and "0", respectively, instead.)After this step,the tree in example1.4.2 consists of the three symbols_ao, a11a2, a3withtheir respective probabilities. Step two is then repeated,using the two lowest probability symbols from the new set of symbols'This processcontinuesuntil all of the original sym-bols have been combined into a singlecompound symbol,asa1a2a3;having irobability 1. The final tree is shown inExampleI.4.2. Code words.are by reading the labels of the tree stemsfrom right to -assigned left back to the original symbor. For inshnJe,in Examprer.4.2,thesymbola, is asfilned codeword 111.Code-wordassignments for the other symbolsare done in th! same*way. Note that the final result is the sameasthe codeword Lssignments in Example 1.3.1.

fhhbr ffiirh l3Jerl m&-ces. (uiftu mUUn

sh
th:mn twrl

lompl qrmbolA estprobC tiesm. sible_Th.t candidate smallerm Ilm at tbe be$ tovbnitr ing to fu traebact 1.4.3 Hardware

Constructa Huffman code for the 4-ary sourcealphabetof Example 1.3.2. Solution* Seethe tree constructionin the following Figure: Probability Symbol

Code Word 0

0.50

0.30

r1

10

0.15

d2

110

0.05

o3

111

Hardwmc using del alphabeq and desigl technical I using a rc entry givi ampb dr code nord included I The souro ROMbp sequential word- Sem Your hemd Dcq cientlY'ler discussed thedeod bia are rel ister is de lookuptafr notthe c|n
teins the d

24

Chapter 1

Discrete Sources and Entropy !t

rf (n, a) is alreadyin the dictionaryin step 2;the en-coder is processing a stringof sym_ bols that has occurr.ed. at.least once pieviously. setting the next value of rz to this addressconstructsa rinked rist that alrowsthe stiing or ri-uor, to be traced. rf (n, a) is not already in the dictionary in stJp 2,ihe encoderis encounteringa new string that has oo1.b"": previouslyprocessed.lt hansmitscode symbolzl,which lets the receiver know the diciionary uioi"r, of the lastsourcesymbol in the previous string'whenever the encodertransmitsa code symbol,it also createsa new dictionary entry (n' c)' In this entty,n is a pointer to the last sourcesymbotin ttr" previousstring of sourcesymbolsand a is the root symborwhich begin, , ;; ,;;;;. code word n is then reset to the address, of (0,a).Th9 "0" pointer m tlms po-i-.rt, to the null character which indicatesthe beginningof the new string. th" "rr,ry Jictionary-building and code symbol transmission pro."r, is illustrateJin the"n"oaer,s folrowin! &ampte.

A binary informationsourceemitsthe sequence of symbols110001011001011 10001111 etc. construct the encodingdictionary and determine thesequenceor t.anrmitt"o code symbols. solution: Initialize the dictionary as shown above.Sincethe sourceis binary,the initial dictionary will contain only the null entry and the entries (0,0i;; (q1ilil;.nary addresses 0, 1, and 2, respectively. The initial valuei for n andm aren: 0 and m:' 3.T\eencoder,soperationis then describedin the followins table.

l-5-3 The De@

sourcesymbol 1 7 0 0 0 1, 0 1 1, 0 0 1, 0 I 1, 1. 0 0 0

presentn

ptesenlm
J J

transmlt

next n

dictionary entry

The deco trarrsrmitr exa@E does not decoderi

0 2 2 1 1 2 2 3 1 5 6 1 2
J

2 4 7 5 6 2
J

4 5 o 6 7 7 8 8 9 9 9 10 11 11. 72 12
1J

2 2
I

2 2 1 1 5 2 4 2
J

2,L 2,0 1,0 5,1 4,1 3,0 6,0 1 ,1

L Rer
gm

RN 3. So0
nel

6 1

1 5 6 1, 2
3

2
+

If the dr struct a I root srml dictionar ated rith sent bv tl

13 13 I4 1,4

1, 5
f)

4,0

ffi
o,r
Decode tl this exaq will help y

2
3

11

Section,l.5 The encoder,s dictionary to this

DictionaryCodesand Lempel_ZlV Codino

25

point is asfollows:

dictionary address

dictionary entry

0 1, 2 3
4 5
o

0,null 0,0 0,1

)1

2,0 1,0 4,1 3,0 6,0 1,1

J'I

7 8 9 10 1i 12
IJ

I4

4,0 6,1. no entryyet

1.5.3 The Decodingprocess The decoderat.thereceivermust alsoconstructa dictionaryidentical to the one at the transmitterand it must decode tt" t"""iu.o-il"*r;ffi;t';.Notice from examplehow the encoderconstructs the previous code symborr"ro,,i.rngs and that does not transmit -?ny the encoder *rrar as ii tras sourc""ryrrruorr. T operation of the decoder "oo. is governed by the'followinfobservations:

t l:t"rifflo,ot

unt codeword means that a new dictionary entry musrbe

3ff#lfi:T:::|;:h5ift tos""i'o*tr'i'i"aon"';;;;i;,;;;;;;;;;;il",
Decodethe received code-words transmitted in Exampler.5.r. (Note:As you

2' Pointer n for this new dictionary entry is the sameas the received code wordn; 3' Sourcesymbol a for this entry is not yet known, sinceit is the rootsymbol next string(whichhasnot y"f of the b""n transmittedby the encoder). If the addrer. oi-tl1. nexJdictionary entry is m,wesee that the decoder struct a partial entry (n, ?) since can onry con_ it -urt await the n"*t ."."iu"d code word to rina tne root symbol a for this entry' It can,however,fill in the missing symbol a in its dictionaryentry at address previous *-t.'i-",unalso decoderrr"-rour"" symbolstring associ_

with ,i'J-oi.,i;;;.}';ffiij,,on lxil;:,fi,',f u. itoccur! J.1i"13"',fri11.:T,Ha:Fi"'1 that

fonow along

30

Chapter 1

Discrete Sources and Entropy

small set of compound sourcesymbols, within some small range about nH(A),makes up the great majority of all blocks emitted by the source.Tlis subsetof tompound symbolsis called the typicalsel.As a consequence, a sourceencoderthat assigns code words to elementsof the typical set using nH(A) bits per code word will achievedata compressionwith very high efficiency.We will make this idea more formal when we discuss Shannon'ssource-coding theorem in Chapter 9. Now, generally,nH(A) is not an integer. Huffrnan codes must ,oroundup,, the number of code bits used to representelemLntsof the typical set to obtain an integer number of bits per code word. Arithmetic codesdo not o-ott ir. 1.6.2 The ArithmeticCodingMethod How is it possibleto have a code word which does not contain an integer number of bits? supposeeacha, e A is encodedinto a real number B,lyingin the iriterval 0 = 8i17. Clearly this idea might require more bits to represent since alarger number of bits B,, could be necessary to representa real numbet. Now'supposethit we encode the sequence rorr"' by addingtogetherscared versionsofthe B,according to b: Bo*A$r+ Lzl}z+..., where eachB,is the code number correspondingto s, and where the A, are monotonically decreasing scalefactors also lying in the interval bet*een 0 and i. If we pick the A, in sucha way that it is possibleto later decompose b back into the original ,"qu"rr." of B,s,this code can be decoded.Now, imagin" we ,eprerent each iri binary fixedB, point format. For now, we'll also assumewe can use extremely long iixed-point numbers to do this.As we add the successive terms of b, the bits representingthe different scaledB, are added together.Thus,it is the sum of theserepresentation which representsthe entire codedblock.In a very real way,we canregardthe bit positions usedto representb as being. "shared" among severaldifferent B,.Th" net effect of this sharing is that_e_ach B, effectivelyusesa non-integernumber of'bits (on the average). We now turn to the details of how to carry this out. Aiithmetic coal"! begins by assigningeach element of.A a subinterval of the real numbers between 0 and 1. The length of the assigned interval is set equal to the symbolprobability.

'\an

TL
af,

When dl anyDrl

Any m bloctdl Ihr Assumel stepc;;

i:l Li= Hi= RE

I.JN

kt us ilftr

il
For the sa

Sotuir the rcsuhr

Let Abe the informationsource givenin Example1.2.1. Assign each a,e ,4,a fractionof the real numberinterval0 = Bt<1,. Solution: po:0.5 + assign aoto the interval1o: [0,0.5), :0.3 pr + assign a, to the interval1, : [0.5,0.g), p,t: 0.I5 + assign azto theinterval1, : [0.g,0.95), pr:0.05 + assign a3to theinterval1.: [0.95, 1). Anybwifi

whichcml

Section 1.6

Arithmetic eoding 31

The arithmetic encodingprocessconsists of constructinga code interval (rather than a code number b).whiclliniquery aeGues a brock or ruZ."r*iu" sourcesymbols. when this interval,lo, is complet"d, i;*iiih;ve upper and lower boundswithin which any b will satisfythe toding condition discussedabove.i.e..

4:fL,A,L=b<H. 7.7.6 Any convenient b within this range is a suitabre code word representingthe entire block of symbols. The calcuration of these intervars is based on the following simple argorithm.
ffi:1" each a, e t hasbeenassigned aninterval{ : [sa,si. c;ry out the following ./:0 Li:o
Hj:1

REPEAT a: H,- L, readnext a, Lyr: Lj+ A. S(, Hyr: Lj+ A. Sh 1: jI1. UNTIL all a,havebeenencoded. Let us illustrate this procedureby means of an example.

For the source of exampleL.6.1, encode the sequence

aL aOaOa3 a2.

solutiorc Using the intervals found in Example 1.6.1,and applying the algorithm, we ' obtain the resultsshownin the following table:

Lj 0a1 7ao 2a0 3a3 4a2

Hj

Lnt

Hi*,

0 0.5 0.5 0.5 0.57725

11 0.8 0.65 0.575 0.575

0.3 0.15 0.07s 0.00375

0.5 u.5 0.5 0.57I25 0.57425

0.8 0.65 0.575 0.575 0.5748125

Any b within the final interval will suffice for a code word. one convenientchoice might be b :0.57470703125. which can be representedusing an 11-bit fixed-point binary number format.

,l.6 Section
Df !o
l-

Arithmetic-Coding 33

is assigned its own interval L When the decoder detectsthat it has decoded the STOP symbol,it exits the decodingloop.

Df

n [It
E

For the source and encoderof example,J..6.2, d.ecode b :0.57470703t25. solution: The step-by-step resultsof the algorithm are given in the fo'owing table:

s h
D

next Il

next L

next A

ai

t
D
D

0 0.5 0.5 0.5 0.57725

1 0.8 u.o) 0.575 0.575

1, 0.3 0.15 0.075 0.00375

4
Io t3
r2

0.5 0.5 0.575 0.5 u.)/) 0.57725 0.57481.25 0.57425

U.OJ

0.8

0.3 0.15 0.075 0.00375 0.0005625

a1 a0 ao a3 a2

In tracingthrough the stepsof exampre 1.6.3:{9} must pay attentionto the precision with which you calculatethe quantity (bry1t aiitiJ other variables. Roundoff error in this calculationcan lead to an erroneousanswer. 1.6.4 Other lssues in Arithmetic Coding In the practical application of arithmetic codes,there are severalother issuesthat the code implementatiol usually needs to address. on. ,.t oi issuesarisesfrom the fact that computersand hardwarl encoders alwayshave finite frecision in their arithmetic. we must therefore pay attention to the irru"r-or;;.;#; overflow and underflow. Let us suppos: H I. are represented as n-bit binary numbers. Let us further -that .and suppose we representthe subintervalcdcurations . ,s,usingf uit, p.".i_ s";;l A ! sion'Finally,letus suppose the machine's numericur "i It has ptJ.iJon is limited top bits. been shownthat avoidanceof overflow or underflo* requires "rro., f=n-2 and f*n=p. This limits the sizeof the sourcealphabet,4 to

Vl= zr,
since,Sr, ^S, are limited to /bits of precision.For example, suppose p : 16 bits and, n: 9 bits.Then/< 7 and our aiphabetAis therefore ri-it"'Ji" iis symbors. Another issuethat usuallyis important in practiceis the need for being able to transmit and decodethe information ion the fly.;'rn the Jg;rithms presentedearlier,