Information Theory

and Coding
By
Prof. Budhaditya Bhattacharyya

Prof. Budhaditya Bhattacharyya, SENSE, VIT, University

Why Information and Coding

Theory?

In general for any mode of communication, reduction of error is a

must.
Reducing it to ZERO is purely hypothetical.
In most of the scenario:
Probability of Error (Pe) = exp (-kEb)
If Eb is increased then Pe will decrease.
But is it at all possible???
The power of the signal (Pi) = Eb x Rb
Where Rb = Bit rate and Eb = Energy per bit
Two possibilities are :
increase Eb, keep Rb to decrease in order to maintain same Pi.
increase Rb, keep Eb to decrease to maintain the same Pi.
So Pe 0 if Rb 0

Prof. Budhaditya Bhattacharyya, SENSE, VIT, University

Why Information and Coding

Theory?
Shannon Theorem: (A brief overview :)
In 1948 Shannon Published his paper titled A
Mathematical Theory of Communication
In brief Shannon suggested that
As long as the rate of transmission is within a particular
limit or channel capacity, then it is possible to have an Error
Free Transmission.
Hence for Pe 0 it is not mandatory to
have Rb 0
If Rb< C then also Pe 0 is obtainable.

http://historycomputer.com/ModernComputer/thinkers/Shannon.html

Prof. Budhaditya Bhattacharyya, SENSE, VIT, University

Why Information and Coding

Theory?

Shannon Theorem: (A brief overview :)

The disturbances which occur on a communication

channel do not limit the accuracy of the transmission but
limits the rate of transmission of information .

Hence the information should be quantified.

Information Theory is a mathematical science that is

characterised by a quantitative approach to the notion of
information.

Prof. Budhaditya Bhattacharyya, SENSE, VIT, University

Information and its types

There are three types of information:
Syntactic: Related to the number of symbols used to build msg.
Semantic : Related to the meaning of the message.
Pragmatic :Related to the usage and effect of the message.
Check this four sentences:
1.

Saurav Ganguly scored a brilliant century at lords.

2.

At Lords, a brilliant centuary was scored by Saurav Ganguly.

3.

There is a traffic congestion on highway leading from Vellore to

Chennai.

4.

There is a huge rush of traffic on NH47 in India.

Prof. Budhaditya Bhattacharyya, SENSE, VIT, University

The basic of Source and Channel

Coding
Source

Channel

Destination

Source
Encoder

Source
Decoder

Channel
Encoder

Channel
Decoder
Channel
The theoretical communication between
source and destination
The Practical communication between
source and destination

Prof. Budhaditya Bhattacharyya, SENSE, VIT, University

The basic of Source and Channel

Coding
Source

Channel

Destination

Source
Encoder

Source
Decoder

Channel
Encoder

Channel
Decoder
Channel

The importance of Shannons Theory was to find out the

optimum source encoded and channel encoder for a given
channel

A specific Example:

Consider three cities A, B and C. Now suppose a communication

link needs to be established between city B and A.

The status of weather should be communicated between the two

cities.

Let city B AND C is having the following statistics:

City B weather Statistics

Probability

State
Information

Binary
levels

SUNNY

00

RAINY

01

CLOUDY

10

FOGGY

11

City C weather Statistics

Probability

State
Information

Binary
levels

1/8

SUNNY

1110

1/8

RAINY

110

CLOUDY

10

1/2

SMOGGY

City C weather Statistics

City B weather Statistics

State
Information

Binary
levels

1/8

SUNNY

1110

01

1/8

RAINY

110

CLOUDY

10

CLOUDY

10

FOGGY

11

1/2

SMOGGY

Probability

State
Information

Binary Probability
levels

SUNNY

00

RAINY

The basic reason behind the change of coding is COST. This can be
achieved by efficient transmission of the same information using lower
or lesser number of bits/ message/second (Lavg)

For the second scenario (city C)

Lavg = SUNNY +RAINY + CLOUDY +SMOGGY

= Bits required for transmission of status x probability of each state

= 4 x (1/8) + 3 x (1/8) +2 x (1/4) +1 x (1/2) = 1.875 binits/msg

City C weather Statistics

City B weather Statistics

State
Information

Binary
levels

1/8

SUNNY

1110

01

1/8

RAINY

110

CLOUDY

10

CLOUDY

10

FOGGY

11

1/2

SMOGGY

Probability

State
Information

Binary Probability
levels

SUNNY

00

RAINY

The link between city C to A convey less information hence lesser

number of binits.

The probability of occurrence of any event is highly important for

designing best possible code