INFORMATION THEORY

INTRODUCTION

Baltasar Beferull Lozano Baltasar.Beferull@uv.es, Escuela T ecnica Superior de Ingenier a Universitat de Val` encia Pedro M. Crespo pcrespo@ceit.es Tecnun Universidad de Navarra Escuela de Ingenieros

'

Single-user Information Theory

Shannons information theory deals with limits on data compression (source coding) and reliable data transmission (channel coding) rates How much can data be compressed ? How fast can data be reliably transmitted over a noisy channel ? Two basic single-user theorems (Shannon 1948): Source coding Theorem: The entropy of the source X , H (X ), is the minimum rate at which data can be compressed losslessly Channel coding Theorem: The capacity of the channel p(y |x), C = maxX I (X ; Y ), is the maximum rate at which data can be reliably transmitted. Since his seminal 1948 papers, Shannons basic theorems have been extended (by him and others) in several directions: Dierent and more rigorous proofs Rate-Distortion theory Source coding theorem for more complex sources (stationary, universal, . . . ) Capacity of more complex channels (continuous-time, memory, state, . . . )

&

'

Until recently, Information Theory was considered (by most) an esoteric theory with no apparent relation to the real world With advances in technology (algorithms, hardware, software), today the picture is very dierent: Single-user Data compression techniques close to optimal Single-user Modulation and Error-correction techniques close to optimal

&

'

Multi-user or Network Information Theory

Extends Shannons basic theorems to communication networks with several sources communicating over a network with several senders and receivers.
X X

NETWORK
X X X X

Several new ingredients: cooperation, side information, superposition of information, interference, etc.... First paper on Network Information Theory was (again) by Shannon, Two-way communication channels, Proc. 4th Berkeley Symp. on Math. Stat. Prob., pp. 611-644, 1961 He didnt nd the optimal rates Problem still open
&

'

Lots of research activity in the 70s and early 80s with many neat results, but very little activity until mid 90s Many problems still open Absolutely no interest from communication community (just a useless theory with oversimplied models that ignore this and that) Almost no-one saw the intuition that is built in the various Theorems Wireless communication, Internet and advances in technology are reviving this area: Lots of fundamental and applied research going on Some work on old open problems but some new problems as well Many sessions in conferences and PhD theses . . . Impact on real systems and ultimately on standards Whats going on ? Several ideas originated from Network Information Theory, such as: Water-lling Superposition coding Successive cancelation Slepian-Wolf lossless Binning
&

'

Wyner-Ziv lossy Binning Side-Information Writing on dirty paper ... are beginning to have real impact on practical source and channel coding in networks.

&

'

What is this course all about ?

Goal: provide main results for single-user and multi-user information theory and optimal code design Theoretical limits for dierent important problems Intuition and key properties to build optimal codes (road to optimal codes!) Emphasis: basic ideas, main proof techniques (and tricks) and basic code structure We will focus on: Discrete memoryless channels and sources Gaussian channel and source

&

'

Basic channel coding models that will be discussed (because they contain most of the key ideas in Multi-user Information Theory): Review of single-user memoryless channel

W
Multiple-access channel

p(y |x)

W1 W2
Broadcast channel

X1 X2

p(y |x1 , x2 )

1, W 2) (W

(W1 , W2 )

p(y1 , y2 |x)

Y1 Y2

1 W 2 W

&

'

Because lack of time, we will not cover: Channels with state information

p(s) S W
Interference channel

p(y |x, s)

W1 W2
Relay channel

X1 X2

p(y1 , y2 |x1 , x2 )

Y1 Y2

1 W 2 W

f (.) Y1 W
&

X1 Y W

p(y, y1 |x, x1 )

'

Basic channel coding problems that will be discussed

Single-user channel coding (Class Notes: II)
M

E
ENCODER

xi (M )

CHANNEL p(y |x)

Yi

D
DECODER

M
Estimate

Multiple Access Channel (Class Note III)

M1

E1
Encoder 1

n X1

M2

E2
Encoder 2

n X2

p(y |x1 , x2 )

Yn

D
Decoder

1 , M 2 ) (M

Channel

Broadcast Channel (Class Notes: IV)

Y1n (M1 , M2 )
encoder

decoder 1

D1
decoder 2

1 M

X n p(y1 , y2 |x) Y2n

D2

2 M

&

'

10

Basic source coding problems that will be discussed

Single-user source coding (Class Notes: II)

U

E
Encoder

D
Decoder

Distributed lossless source coding (Class Notes: V)

Encoder 1

E1 E2
Encoder 2

R1

D
V
R2
Joint Decoder

, V ) (U

Distributed lossy coding with side information (Class Notes: VI)

Encoder 1 Decoder

E1 E2
Encoder 2

R1

R2

&

'

11

We know optimal rates and code design for some of these problems (or sub-classes of these problems): Multiple-access Distributed lossless source-coding Distributed lossy source coding with side information Degraded broadcast channel Degraded relay channel Channels with state information for some particular cases Strong interference channel Gaussian multiple description coding But many basic problems remain open, and a general theory is yet to be developed.

This course will NOT cover: Complete code design, encoding & decoding algorithms and performance analysis Other complementary Master courses: Digital Communication Theory Advanced Wireless Communications and Mobility Advanced Coding Techniques Advanced Signal Processing for Communications
&

'

12

Course requirements and Grading

Participation in class (5% of grade) Homeworks (30% of grade) Although in-group discussion is ok, try to solve all the problems individually This will be your best possible preparation for the nal exam Problems will not be plug-and-chug, will require time and thinking (but you will hopefully learn a lot!!) Goal: make you learn well the concepts and interpret solutions correctly Final exam (65% of grade) Set of problems and questions of similar level to the homework assignments. The tentative day for this exam will be 17/12/2012 (Friday)

&

'

13

Other details: course prerequisites and web sites

Course Prerequisites: Calculus Algebra Basic Probability Basic Random Processes Basic Programming Web sites for course material: http://aulavirtual.uv.es Lecture slides Homework assignments Complementary material (list of papers, etc...)

&

'

14

Bibliography
Basic Bibliography: A. El Gamal, Y.-H. Kim. Network Information Theory, Cambridge University Press, 2012. T.M. Cover, J.A.Thomas. Elements of Information Theory, Second Edition, Wiley 2006. Set of slides Complementary Bibliography: R. Yeung. A rst course in Information Theory, Kluwer 2002. R.G. Gallager, Information Theory and Reliable Communications. Wiley, 1968. J.M. Wozencraft, I.M. Jacobs. Principles of Communication Engineering. Wiley, 1965. D. Mackay, Information Theory, Inference, and Learning Algorithms, Cambridge University Press, 2003. G. Kramer, Topics in Multi-User Information Theory, Foundations and Trends in Communications and Information Theory, now Publishers Inc., 2008. G. Keshet, Y. Steinberg, N. Merhav, Channel Coding in the Presence of Side
&

'

15

Information, Foundations and Trends in Communications and Information Theory, now Publishers Inc., 2008. C. Schlegel, A. Grant, Coordinated Multiuser Communications, Springer, 2006. R. Yeung, S.-Y. R. Li, N. Cai, Z. Zhang, Network Coding Theory, Foundations and Trends in Communications and Information Theory, now Publishers Inc., 2006. I. Csiszar, J. Koerner, Information Theory: Coding Theorems for Discrete Memoryless Systems, Academic Press, 1981. R. M. Gray, Entropy and Information Theory, Springer-Verlag, 1990 (Revised version available on-line from R. M. Grays web page at Stanford) T. Berger. Rate-Distortion Theory: A Mathematical Basis for Data Compression, Prentice Hall, 1971 C. E. Shannon, The Mathematical theory of Communication, U. of Illinois Press, 1963. Several journal papers.

&