Chapter 7 Information Theory and Coding

Chapter 7: Information Theory &
Coding 1
Chapter 7: Information Theory and Coding
Introduction to Information Theory
Measure of Information:
Common-Sense Measure of Information:
The amount of information received from a message is inversely
proportional to the certainty (probability) of its occurrence.
The sun will rise from the east tomorrow:
100% probability Zero information content
Singapore will have an earthquake next week:
Very low probability Very high information content

Coding 2
Measure of Information:
Digital Communication Measure of Information:
The source is generating symbols (e.g. characters from a keyboard).
More convenient to consider the information content on a per symbol
basis rather than an entire message.
The information content of a symbol s
i
is measured as:
where P
i
is the probability of occurrence of symbol s
i
;
i.e., the higher the probability that a symbol occurs, the smaller the
information content of the symbol, measured in terms of number of
bits.
bits
P
1
log ) I(s
i
2 i
|
|
.
|
\
|
=
Coding 3
Average Information Content of Symbols (Entropy)
Very often the symbol-stream generated from a digital
communication source have probability of occurrence that
fluctuates wildly.
For example, you press certain keys more often than other keys
(e.g. a, e, i, o, u occurs more often than z, q, x, w).
Hence the symbol-stream coming out of a source have fluctuating
information content.
It is more convenient to consider the average information content
per symbol in a long message rather than the instantaneous
information content.
In practical symbol-stream, the probability of occurrence of a
symbol P(s
i
) depends not only on how often the symbol is
generated. It also depends on the occurrence of other symbols. For
example, if you type q, it is almost certain that u will follow.
For simplicity, this chapter assumes the generation of s
i
are
statistically independent; i.e. occurrence of s
i
does not depend
on the occurrence of previous symbols.
Coding 4
Derivation of Entropy Equation
Assume a source has M symbols: s
1
, s
2
, s
3
, . , s
M
.
Let probability of occurrence of these symbols be P
1
, P
2
, P
3
, , P
M
,
respectively.
On the average, s
i
will occur NP
i
times, where i = 1 or 2 or M.

N symbols
Assume we sent a long message consisting of N symbols:

(e.g. s
1
s
3
s
5
s
2
s
1
. s
2
)

Since information content of s
i
is bits
P
1
log ) I(s
i
2 i
|
|
.
|
\
|
=
information content of NP
i
occurrence of s
i
in the N symbols
will be:
bits
P
1
log NP
i
2 i
|
|
.
|
\
|
Coding 5
Symbol Information content of
symbol
Average information content
contributed by symbol in the
message
S
1

S
2

s
M
bits
P
1
log ) I(s
2
2 2
|
|
.
|
\
|
=
bits
P
1
log ) I(s
M
2 M
|
|
.
|
\
|
=
bits
P
1
log ) I(s
1
2 1
|
|
.
|
\
|
=
bits
P
1
log NP
1
2 1
|
|
.
|
\
|
bits
P
1
log NP
2
2 2
|
|
.
|
\
|
bits
P
1
log NP
M
2 M
|
|
.
|
\
|
Total information content of the message (N symbols):
bits
P
1
log NP
P
1
log NP ...
P
1
log NP
P
1
log NP I
M
1 i
i
2 i
M
2 M
2
2 2
1
2 1 total
=
|
|
.
|
\
|
=
|
|
.
|
\
|
+ +
|
|
.
|
\
|
+
|
|
.
|
\
|
=
Coding 6
Total information content of the message (N symbols):
bits
P
1
log NP
P
1
log NP ...
P
1
log NP
P
1
log NP I
M
1 i
i
2 i
M
2 M
2
2 2
1
2 1 total
=
|
|
.
|
\
|
=
|
|
.
|
\
|
+ +
|
|
.
|
\
|
+
|
|
.
|
\
|
=
Average information content per symbol in the N-symbols message is:

P
1
log P
N
I
H
M
1 i
i
2 i
total
=
|
|
.
|
\
|
= =
bits/symbol
where H is called the source entropy.
Coding 7
Example 1.1
Find the entropy of a source that emits one of three symbols A, B and C in
a statistically independent sequence with probabilities , and ,
respectively.
Solution:
P(A) = ; P(B) = ; P(C) =
Information content of A, B and C are:
bits 2 (4) log
P(C)
1
log ) I(C
bits 2 (4) log
P(B)
1
log ) I(B
bit 1 (2) log
P(A)
1
log ) I(A
2 2
2 2
2 2
= =
|
|
.
|
\
|
=
= =
|
|
.
|
\
|
=
= =
|
|
.
|
\
|
=
Entropy H =
|
|
.
|
\
|
+
|
|
.
|
\
|
+
|
|
.
|
\
|
P(C)
1
log P(C)
P(B)
1
log P(B)
P(A)
1
log P(A)
2 2 2
= x 1 + x 2 + x 2 = 1.5 bits/symbol
Coding 8
Example 1.2
A discrete source emits one of five symbols once every millisecond. The
symbol probabilities are P
1
= 1/2, P
2
= 1/4, P
3
= 1/8, P
4
= 1/16,
P
5
= 1/16. Find the source entropy and information rate.
Solution:
Symbol rate r
s
= 1/(1 ms) = 1 K symbols/s .
Source entropy H =
=
|
|
.
|
\
|
5
1 i
i
2 i
P
1
log P
( ) ( ) ( ) ( )
875 . 1
16 log
16
1
x 2 8 log
8
1
4 log
4
1
2 log
2
1

1/16
1
log
16
1
x 2
1/8
1
log
8
1
1/4
1
log
4
1
1/2
1
log
2
1
2 2 2 2
2 2 2 2
=
+ + + =
|
.
|
\
|
+
|
.
|
\
|
+
|
.
|
\
|
+
|
.
|
\
|
=
bits/symbol
Information rate = H r
s
= 1875 b/s .
Coding 9
Source Coding
The block diagram below shows a Digital Communication System.
We shall concentrate on the Source Encoder and Source Decoder block.
Coding 10
Assume the Discrete Information Source is transmitting a message
m in the form of a symbol stream.
The symbol stream consists of the interleaving of 3 symbols: s
1
, s
2
, s
3

each having probability of occurrence P(s
i
) = 1/3 ; i.e., m = s
1
s
3
s
2
s
3
s
1
s
2
.
The function of the Source Encoder is to code each symbol with a
binary codeword:
e.g. : symbol codeword
s
1
00
s
2
01
s
3
10
Hence the output of the Source Encoder is m = 0010011000 .
The Source Decoder is the inverse function of the Source Encoder.
The Source Decoder output is hence m = s
1
s
3
s
2
s
3
s
1
s
2
, assuming the
channel noise does not cause any bit error in the transmission.
If the three symbols do not have equal probability, the Source Encoder
will generate codewords which differs in length (i.e. variable length).
High Probability symbol Short Codeword; Low Probability symbol
Long Codeword. In general, variable-length codewords will reduce the
bits transmitted at the source encoder output; i.e. there is data
compression.
Coding 11
Example 2.1
A source is generating symbols from a source alphabets of three possible
symbols, s
1
, s
2
and s
3
. These symbols have equal probability of
occurrence.
a) Give the minimum-length codeword for each symbol.
b) What is the code efficiency ?
Solution:
Since all the symbols are equiprobable, they have equal-length codeword.
This is the case of fixed-length coding.
For fixed-length coding, the minimum codeword length can be obtained as follow:
Let X be the number of symbols in the source alphabets; ( i.e., X = 3 symbols )
where
m
denotes the largest integer not exceeding m .

+
=
integer an not is log if 1 log
integer an is log if log
2 2
2 2
X X
X X
n
log
2
X = log
2
3 = 1.58
Not integer
Use 2
nd
eqn
a)
Coding 12
where
m
denotes the largest integer not exceeding m .

+
=
integer an not is log if 1 log
integer an is log if log
2 2
2 2
X X
X X
n
Since log
2
X = log
2
3 = 1.58, use the lower equation to compute the minimum
length of the codeword:

bits 2 1 1 1 58 . 1 1 3 log 1 log
2 2
= + = + = + = + = X n
Each codeword is 2-bit long. A possible coding is:
Symbol Codeword
s
1
00
s
2
01
s
3
10
Note: Each codeword is unique (Uniquely Decodable Property)
Coding 13
b)
Code Efficiency q is defined as
where H is the source entropy, is the average codeword length.
n
H
=
n
58 . 1 3 log
3 / 1
1
log
3
1
3 / 1
1
log
3
1
3 / 1
1
log
3
1
P
1
log P H
2
2 2 2
3
1 i
i
2 i
= =
|
.
|
\
|
+
|
.
|
\
|
+
|
.
|
\
|
=
|
|
.
|
\
|
=

=
2 n =
bits/symbol
bits
Code efficiency
X 100%
% 79 100% x
2
58 . 1
100% x
n
H
= = =
Coding 14
Example 2.2 (Repeat of Example 2.1, but with unequal symbol probability)
A source is generating symbols from a source alphabets of three possible
symbols, s
1
, s
2
and s
3
. These symbols have probability of occurrence 5/8.
2/8 and 1/8, respectively.
a) Give the minimum-length codeword for each symbol.
b) What is the code efficiency ?
c) What is the compression ratio ?
Solution:
Since there are differing symbol probabilities, the source encoding strategy is:
(i) the higher the probability of a symbol, the shorter is the codeword.
(ii) Assign the minimum number of bits per codeword.
Hence a possible code set is:
Symbol Codeword Probability
s
1
0 5/8
s
2
10 2/8
s
3
11 1/8
Note: Each codeword is unique (Uniquely Decodable Property)
Each codeword is not a prefix of other codewords (Prefix-Free Property)
a)
Coding 15
b)
Code Efficiency
n
H
=
n
299 . 1
8 / 1
1
log
8
1
8 / 2
1
log
8
2
8 / 5
1
log
8
5
P
1
log P H
2 2 2
3
1 i
i
2 i
=
|
.
|
\
|
+
|
.
|
\
|
+
|
.
|
\
|
=
|
|
.
|
\
|
=
=
n
bits/symbol
X 100%
The average codeword length is computed based on the equation:
=
=
M
1 i
i i
) P(s n n where M is number of symbols in source alphabets
(i.e., M = 3), n
i
is number of bits for codeword of s
i
.
Symbol Codeword P(s
i
) n
i

s
1
0 5/8 1
s
2
10 2/8 2
s
3
11 1/8 2
Coding 16
% 94 100% x
375 . 1
299 . 1
100% x
n
H
= = =
Code efficiency
=
=
M
1 i
i i
) P(s n n
Symbol Codeword P(s
i
) n
i

s
1
0 5/8 1
s
2
10 2/8 2
s
3
11 1/8 2
375 . 1
8
1
x 2
8
2
x 2
8
5
x 1 ) P(s n n
3
1 i
i i
=
+ + = =

=
bits
c) Compression Ratio =
45 . 1
375 . 1
2
n
n
(variable)
(fixed)
= =
where is from Example 2.1 . 2 n
(fixed)
=
Coding 17
There is no problem decoding prefix-free code set as
shown below :-
Example 2.2 is an example of Prefix-Free Coding.
Symbol Codeword
s
1
0
s
2
10
s
3
11
Prefix-Free Coding
Assume input to source encoder is a symbol stream: s
1
s
3
s
2
s
1
s
1
s
3
s
2
s
1
0 1 1 1 0 0
Source Encoder
s
1
s
3
s
2
s
1
0 1 1 1 0 0
Source Decoder
Coding 18
Assume input to source encoder is a symbol stream: s
1
s
3
s
2
s
1
s
1
s
3
s
2
s
1
0 1 0 0 1 0
Source Encoder
Symbol Codeword
s
1
0
s
2
01
s
3
10
Non-Prefix-Free Coding
(codeword s
1
uses a prefix of codeword s
2
)

There is problem decoding non-prefix-free code set :-
s
1
s
3
s
2
s
1
0 1 0 0 1 0
Source Decoder
or
0 1 0 0 1 0
s
2
s
1
s
1
s
3
?
?
}
Ambiguous
Result
Coding 19
Huffman Coding
A prefix-free variable-length code.
Can achieve the shortest average code length
for a given input alphabet.
Coding 20
Example 2.3
A source has five possible symbols, s
1
, s
2
, s
3
, s
4
, s
5

with probabilities 0.1, 0.2, 0.1, 0.4, 0.2,
respectively.
a) Obtain the codewords for the symbols using
Huffman coding. Assume binary 0 is to be sent as
often as possible.
b) Calculate the code efficiency and compression
ratio.

Coding 21
Solution:
a)
P(s
1
) = 0.1; P(s
2
) = 0.2; P(s
3
) = 0.1; P(s
4
) = 0.4; P(s
5
) = 0.2;
Huffman encoding algorithm:
1. List symbols in order of decreasing probability.
2. Since binary zero is to be sent more often: assign 0, 1, to the two lowest
probabilities, respectively.
3. Add the two lowest probabilities and re-list in order of decreasing probability.
4. Repeat step 2 and 3 until the list contains only two probabilities in decreasing
order. Assign 0 and 1 to the final two probabilities.
5. Obtain the codeword of each symbol by working backward (against the arrow
direction) and noting the sequence of 0 and 1 along the way.
s
4
s
2
s
5
s
1
s
3
0.4
0.2
0.2
0.1
0.1
0.4
0.2
0.2
0.2
0.4
0.4
0.2
0.6
0.4
s
4
s
2
s
5
s
1
s
3
00
10
11
010
011
symbol
codeword
Coding 22
b)
Code Efficiency
n
H
=
n
n
bits/symbol
X 100%
The average codeword length is computed based on the equation:
=
=
M
1 i
i i
) P(s n n where M is number of symbols in source alphabets
(i.e., M = 5), n
i
is number of bits for codeword of s
i
.
122 . 2
1 . 0
1
log 0.1 x 2
2 . 0
1
log 0.2 x 2
4 . 0
1
log 4 . 0
P
1
log P H
2 2 2
5
1 i
i
2 i
=
|
.
|
\
|
+
|
.
|
\
|
+
|
.
|
\
|
=
|
|
.
|
\
|
=
=
s
4
s
2
s
5
s
1
s
3
00
10
11
010
011
symbol codeword P(s
i
) n
i
0.4 2
0.2 2
0.2 2
0.1 3
0.1 3
Coding 23
% 96 100% x
2 . 2
122 . 2
100% x
n
H
= = =
Code efficiency
=
=
M
1 i
i i
) P(s n n
bits
s
4
s
2
s
5
s
1
s
3
00
10
11
010
011
symbol codeword P(s
i
) n
i
0.4 2
0.2 2
0.2 2
0.1 3
0.1 3
2 . 2
0.1 x 3 0.1 x 3 0.2 x 2 0.2 x 2 0.4 x 2 ) P(s n n
5
1 i
i i
=
+ + + + = =

=
Coding 24
Compression Ratio =
36 . 1
2 . 2
3
n
n
(variable)
(fixed)
= =
where is the average codeword length for Huffman coding. 2 . 2 n
(variable)
=
(variable)
(fixed)
n
n
Compression Ratio =

+
=
integer an not is X log if 1 X log
integer an is X log if X log
n
2 2
2 2
fixed
(where X is the source alphabet; i.e. X = 5 symbols

m
denotes the largest integer not exceeding m ).
Since log
2
X= log
2
5 = 2.32 (not an integer), use the lower equation to compute
the minimum length of the codeword:

bits 3 1 2 1 32 . 2 1 5 log 1 X log n
2 2 (fixed)
= + = + = + = + =
Coding 25
Some Observation on Huffman Coding
If binary 1 is to be sent more often, the each node will have
assignment:
If there is no preference on which binary digit to be sent more often
then we could randomly assign 0, 1 or 1, 0 to the upper and lower
branch of the node. In the earlier example, there are 4 nodes.
Therefore there could be 2
4
= 16 possible Huffman code solutions.
However each solution will produce the same code efficiency and
compression ratio.
In the earlier example, we consistently placed the summed equal
probability at the top. If we consistently place the summed equal
probability at the bottom, the length of codeword will have the largest
variance, i.e. the smallest code length is 1, the largest code length is 4.
Huffman code, like any other variable-length code can sometime give
rise to error propagation; i.e. a 1-bit error will cause the rest of the
symbols to be decoded wrongly at the source decoder.
Coding 26
Channel Coding
The block diagram below shows a Digital Communication System. We
shall concentrate on the Channel Encoder and Channel Decoder blocks.
Two approaches to Channel Coding: (i) Block Coding (ii) Convolutional
Coding. We shall concentrate on Block Coding, in particular a class
of code called Systematic Linear Block Code.
Coding 27
Discrete
Information
Source
Source
Encoder
Channel
Encoder
m
u
First three stages of a Digital Communication System:
The output of the source encoder is a bit stream m.
The channel encoder partitions the bit stream into message blocks,
where each block is k bits long. Then by some encoding rule, it adds
p parity bits to each message block, and outputs data blocks, where
each data block is n = k + p bits long.
For example if k = 4 and p = 3 (n = 7) :-
Channel
Encoder
1110 0111
0101110 0010111

message blocks

message blocks

parity blocks
Coding 28
In the above example, the channel encoder outputs data block of
n = 7 bits which consists of k = 4 bits message block. In general a
block code with codeword n-bit long consisting of k-bit message is
called an (n, k) block code.
The purpose of adding p bits to each message block is to allow the
channel decoder to detect and/or correct any error bit in the date block
after passing through the noisy channel.
If the noise power is not too high, the channel decoder will correct any
error bit and present a bit-stream consisting of k-bit message blocks to
the source decoder:
Channel
Decoder
Source
Decoder
0101110 0010111
1110 0111
In practice, there are usually error bits in some data blocks; but channel
coding in general will reduce the BER(bit error rate) of the matched
filter demodulator.
Coding 29
Characteristics of Linear Block Code
If the message block has k = 3 bits, there will be 8 possible
message blocks, as shown below:
0 0 0
0 0 1
0 1 0
0 1 1
1 0 0
1 0 1
1 1 0
1 1 1
Let us assume the channel encoder uses some encoding rule to
append p = 3 bits to each of the 8 message block, as shown below:
0 0 0 0 0 0
1 1 0 0 0 1
1 0 1 0 1 0
0 1 1 0 1 1
0 1 1 1 0 0
1 0 1 1 0 1
1 1 0 1 1 0
0 0 0 1 1 1
8 possible
message
blocks
Codewords
generated by
channel encoder
Coding 30
The addition used in Linear Block Code is modulo-2 addition.
0 + 0 = 0; 0 + 1 = 1; 1 + 0 = 1; 1 + 1 = 0
0 1 1 ; 1 0 1 ; 1 1 0 ; 0 0 0 = = = =
For convenience, we will use + instead of but remember
that it is modulo-2 addition. Thus
0 0 0 0 0 0
1 1 0 0 0 1
1 0 1 0 1 0
0 1 1 0 1 1
0 1 1 1 0 0
1 0 1 1 0 1
1 1 0 1 1 0
0 0 0 1 1 1
The output of the channel encoder as shown below is a linear block
code where the sum of any two codewords is another codeword in the
code set.
Codewords
generated by
channel
encoder:
p
i
are parity bits
m
i
are message
bits
p
1
p
2
p
3
m
1
m
2
m
3
Coding 31
Hamming Weight Defined as the number of 1 in a codeword.
Example: u = 1 1 0 1 1 0
Hamming weight = 4
Hamming Distance between two codewords u
1
and u
2
is the number
of positions where the digits are different.
Example:
u
1
= 0 1 1 1 0 0
u
2
= 1 0 1 1 0 1

3 positions where digits are different, hence
Hamming distance d(u
1
, u
2
) = 3
Coding 32
Note: Hamming Distance between two codewords u
1
and u
2
is the
Hamming Weight of u
1
+ u
2
(remember + is modulo-2 addition)

Example:
u
1
= 0 1 1 1 0 0
u
2
= 1 0 1 1 0 1
u
1
+ u
2
= 1 1 0 0 0 1
Hamming Weight = 3
(Hamming distance d(u
1
, u
2
) = 3 from previous example)

0 0 0 0 0 0
1 1 0 0 0 1
1 0 1 0 1 0
0 1 1 0 1 1
0 1 1 1 0 0
1 0 1 1 0 1
1 1 0 1 1 0
0 0 0 1 1 1
To find the minimum distance for the code below, you need to find
the smallest distance between any pair of codewords.
Compute distance for:
row 1 to 2, 1 to 3, , 1 to 8
row 2 to 3, 2 to 4, , 2 to 8

row 7 to 8 .
Then find the value of the
minimum distance.
Coding 33
Fortunately for Linear Block Code, minimum distance of the code is
equal to the minimum weight of the non-zero codewords:
0 0 0 0 0 0
1 1 0 0 0 1
1 0 1 0 1 0
0 1 1 0 1 1
0 1 1 1 0 0
1 0 1 1 0 1
1 1 0 1 1 0
0 0 0 1 1 1
Codeword Weight
0
3
3
4
3
4
4
3
Minimum weight
of non-zero
codeword = 3.
Hence minimum
distance of code
= 3
Coding 34
Error Detection and Correction Capabilities of
Linear Block Code
Let the minimum distance be d
min
.
Can detect up to e = d
min
- 1 bit errors per codeword.

Can correct up to t = bit errors per codeword, where
(

2
1 d
min

m
denotes the largest integer not exceeding m.
Example 3.1
A code has minimum distance d
min
= 4 . What is its error detection and
correction capabilities ?
Solution:
Can detect up to e = d
min
1 = 4 1 = 3 error bits per codeword.

Can correct up to t = error bit per codeword.

1 5 . 1
2
3
2
1 d
min
= =
(
=
(

Coding 35
Encoding and Decoding Procedure for Linear Block Code
Channel
Encoder
0100 1100
1010100 0111100

message blocks

message blocks

parity blocks
We have shown earlier on an example of the Channel Encoder
generating n-bit codewords (n = 7) from message blocks each k-bit
long (k = 4).
How are the parity bits generated ?
The parity bits (together with the message bits) are generated by a
Generator Matrix G.
Coding 36
For example:
1010100 0111100
Channel
Encoder
u = mG 0100 1100

message blocks

message blocks

parity blocks
m u
First message block is represented by a vector m = [1 1 0 0]
The generator matrix for this example is given by:
(
(
(
(
1 0 0 0 1 1 1
0 1 0 0 1 1 0
0 0 1 0 1 0 1
0 0 0 1 0 1 1
G =

Parity submatrix P
Must have unique row
or column pattern
Identity matrix I. Dimension k x k
where k = length of message vector
Coding 37
To obtain the vector u at the output of the channel encoder:
u = mG =
(
(
(
(
1 0 0 0 1 1 1
0 1 0 0 1 1 0
0 0 1 0 1 0 1
0 0 0 1 0 1 1
[1 1 0 0]
= [0 1 1 1 1 0 0]
where codeword
4 3 2 1 3 2 1
m m m m p p p u =
p
1
= 1 + 1 + 0 + 0 = 0
p
2
= 1 + 0 + 0 + 0 = 1
p
3
= 0 + 1 + 0 + 0 = 1
and ) 1100 ( ) (
4 3 2 1
= m m m m
p
1
p
2
p
3
m
1
m
2
m
3
m
4
Coding 38
To obtain the second vector u for message vector m = [0 1 0 0]:
u = mG =
(
(
(
(
1 0 0 0 1 1 1
0 1 0 0 1 1 0
0 0 1 0 1 0 1
0 0 0 1 0 1 1
[0 1 0 0]
= [1 0 1 0 1 0 0]
where codeword
4 3 2 1 3 2 1
m m m m p p p u =
p
1
= 0 + 1 + 0 + 0 = 1
p
2
= 0 + 0 + 0 + 0 = 0
p
3
= 0 + 1 + 0 + 0 = 1
and ) 0100 ( ) (
4 3 2 1
= m m m m
p
1
p
2
p
3
m
1
m
2
m
3
m
4
Coding 39
Example 3.2
The generator matrix for a (6,3) block code is given below. Find all the
code vectors of this code. Obtain the equation of each parity bit as a
function of the message bits.
Solution:
(
(
(
1 0 0 0 1 1
0 1 0 1 0 1
0 0 1 1 1 0
G =
For a (6,3) block code, n = 6 bits, k = 3 bits; (i.e., message block has 3 bits).
There are 2
k
= 2
3
= 8 possible code vectors for the message block:-
0 0 0
0 0 1
0 1 0
0 1 1
1 0 0
1 0 1
1 1 0
1 1 1
m
1
m
2
m
3
Coding 40
The question requires us to compute the 8 codevectors u for the 8
message vectors m.
Channel
Encoder
u = mG
m u
For example, to compute u for m = [1 1 1] :-
u = mG = [1 1 1]
(
(
(
1 0 0 0 1 1
0 1 0 1 0 1
0 0 1 1 1 0
where codeword
3 2 1 3 2 1
m m m p p p u =
p
1
= 0 + 1 + 1 = 0
p
2
= 1 + 0 + 1 = 0
p
3
= 1 + 1 + 0 = 0
and ) 111 ( ) (
3 2 1
= m m m
= [0 0 0 1 1 1]
p
1
p
2
p
3
m
1
m
2
m
3
Coding 41
The 8 u codevectors for the 8 m vectors are given below:
0 0 0 0 0 0
1 1 0 0 0 1
1 0 1 0 1 0
0 1 1 0 1 1
0 1 1 1 0 0
1 0 1 1 0 1
1 1 0 1 1 0
0 0 0 1 1 1
0 0 0
0 0 1
0 1 0
0 1 1
1 0 0
1 0 1
1 1 0
1 1 1
m
1
m
2
m
3
p
1
p
2
p
3
m
1
m
2
m
3

u vectors

m
vectors
Coding 42
To obtain the equation of each parity bit as a function of the
message bits, refer the the generator matrix below :-
(
(
(
1 0 0 0 1 1
0 1 0 1 0 1
0 0 1 1 1 0
G =
m
1
m
2
m
3
m
1
m
2
m
3
p
1
p
2
p
3

P I
3
p
1
= m
2
+ m
3
p
2
= m
1
+ m
3
p
3
= m
1
+ m
2
Coding 43
Decoding Procedure for Linear Block Code
In Example 3.2, the channel encoder outputs data block of
n = 6 bits which consists of k = 3 bits message block and
p = 3 parity bits.
The purpose of adding p bits to each message block is to allow the
channel decoder to detect and/or correct any error bit in the date block
after passing through the noisy channel.
If the noise power is not too high, the channel decoder will correct any
error bit and present a bit-stream consisting of k-bit message blocks to
the source decoder:
Channel
Decoder
Source
Decoder
011100 000111
100 111
How does the channel decoder detect and/or correct error bits ?
Coding 44
Supposing the channel encoder is transmitting a (6, 3) block code using the
generator matrix as in Example 3.2.
If there is no channel noise, the 6-bit codeword being received by the
channel decoder is one of the eight codevectors shown below.
0 0 0 0 0 0
1 1 0 0 0 1
1 0 1 0 1 0
0 1 1 0 1 1
0 1 1 1 0 0
1 0 1 1 0 1
1 1 0 1 1 0
0 0 0 1 1 1
p
1
p
2
p
3
m
1
m
2
m
3

u vectors
Let us assume that the vector u = 1 1 0 1 1 0 is sent by the channel
encoder (highlighted in green) but because of channel noise the
channel decoder received it as r = 1 1 0 1 0 0 . (Blue bit 1 becomes
red bit 0 due to channel noise).
Coding 45
Channel encoder sent: u = 1 1 0 1 1 0
Channel decoder received: r = 1 1 0 1 0 0
How does the channel decoder detect bit error ?
How does the channel decoder know where are the bits that have
errors so that that they can be corrected ?
The channel decoder system is shown below:
syndrome
calculator
error-pattern
generator
error-bit
correction
r
s
e
u
s = r H
T
u = r e
Channel Decoder
Remove
parity
bits
m
The syndrome calculator refers to a matrix H
T
which will be explained in a
moment.
Coding 46
Let us define a parity check matrix
H = [ I
n-k
P
T
] = [ I
3
P
T
]

(
(
(
1 0 0 0 1 1
0 1 0 1 0 1
0 0 1 1 1 0
G =

P I
3
From example 3.2, the generator matrix G is:
(
(
(
0 1 1 1 0 0
1 0 1 0 1 0
1 1 0 0 0 1
H =

I
3
P
T
(
(
(
(
(
(
(
(
0 1 1
1 0 1
1 1 0
1 0 0
0 1 0
0 0 1
H
T
=

I
3
P
Coding 47
syndrome
calculator
error-pattern
generator
error-bit
correction
r
s
e
u
s = r H
T
u = r e
Channel Decoder
Remove
parity
bits
m
The syndrome calculator computes an error syndrome vector s using
the equation s = r H
T
.
If s has all zero elements, then it is assumed that
the received vector r has no error bit.
If s has non-zero elements, it is assumed that the received
vector r has one or more error bits.
Based on the pattern of the error syndrome, it is possible to know
which bits are in error in the received vector r.
Coding 48
syndrome
calculator
error-pattern
generator
error-bit
correction
r
s
e
u
s = r H
T
u = r e
Channel Decoder
Remove
parity
bits
Assume the received vector is r = 1 1 0 1 0 0 = r
1
r
2
r
3
r
4
r
5
r
6

The syndrome calculator computes an error syndrome vector s using
the equation s = r H
T
.
(
(
(
(
(
(
(
(
0 1 1
1 0 1
1 1 0
1 0 0
0 1 0
0 0 1
s = [1 1 0 1 0 0]
= [s
1
s
2
s
3
] = [1 0 1]
s
1
= 1 + 0 + 0 + 0 + 0 + 0 = 1
s
2
= 0 + 1 + 0 + 1 + 0 + 0 = 0
s
3
= 0 + 0 + 0 + 1 + 0 + 0 = 1
r
1
r
2
r
3
r
4
r
5
r
6
Error
syndrome
pattern
shows r
5
is
error bit
Detect error
because non-zero
pattern
m

Coding 49
The error-pattern generator is a look-up table with the error syndrome
vector as the pointer.
error syndrome vector error-pattern
r
1
r
2
r
3
r
4
r
5
r
6
1 0 0 1 0 0 0 0 0
0 1 0 0 1 0 0 0 0
0 0 1 0 0 1 0 0 0
0 1 1 0 0 0 1 0 0
1 0 1 0 0 0 0 1 0
1 1 0 0 0 0 0 0 1

In the previous slide we show that the error syndrome vector has the
value s = [1 0 1] indicating that bit r
5
is an error bit.
The error syndrome vector value [1 0 1] points to the error pattern
value e = [0 0 0 0 1 0] highlighted in red above.
The error bit r
5
is corrected by the modulo-2 addition of r and e :
r 1 1 0 1 0 0
u = r + e e 0 0 0 0 1 0 +
u 1 1 0 1 1 0
where u is the corrected codeword which is similar to the
codeword generated at the output of the channel encoder.
Coding 50
The function of the last block of the channel decoder is to remove the
parity bits of the codeword u, leaving the message block m.
1 1 0 1 1 0 1 1 0
p
1
p
2
p
3
m
1
m
2
m
3
m
1
m
2
m
3
Coding 51
Example 3.3
A (7, 4) code has codeword u = =
4 3 2 1 3 2 1
m m m m p p p
7 6 5 4 3 2 1
u u u u u u u
The parity bits are related to the message bits by the following rule:
4 3 2 3
4 3 1 2
4 2 1 1
m m m p
m m m p
m m m p
+ + =
+ + =
+ + =
a) Obtain the generator matrix.
b) Obtain the parity check matrix.
c) Find all the codewords.
d) What is the minimum distance of the code ? Hence compute the error
detection and correction capability of this code.
e) For the following code vector r at the input of the channel decoder,
determine whether it is a valid codeword. If it is not, show how the
channel decoder correct any error bit.
i. r = 1 0 0 1 0 1 0
ii. r = 1 0 0 1 1 0 1
Coding 52
Solution:
a)
4 3 2 3
4 3 1 2
4 2 1 1
m m m p
m m m p
m m m p
+ + =
+ + =
+ + =
(
(
(
(
1 0 0 0 1 1 1
0 1 0 0 1 1 0
0 0 1 0 1 0 1
0 0 0 1 0 1 1
G =
m
1
m
2
m
3
m
4
p
1
p
2
p
3
m
1
m
2
m
3
m
4

P

I
4
Generator matrix
b) Parity check matrix H = [ I
n-k
P
T
] = [ I
3
P
T
]
H =

I
3

P
T
(
(
(
1 1 1 0 1 0 0
1 1 0 1 0 1 0
1 0 1 1 0 0 1
Refer to Question:
(n, k) code where
n = 7, k=4
Coding 53
c) The codeword can be obtained from the matrix equation: u = m G
where m can have one of 16 values:
m = [0 0 0 0], [0 0 0 1], [0 0 1 0], . , [1 1 1 1]
For example, for m = [1 0 1 0] the corresponding code vector u is :-
u = m G = [1 0 1 0]
(
(
(
(
1 0 0 0 1 1 1
0 1 0 0 1 1 0
0 0 1 0 1 0 1
0 0 0 1 0 1 1
= [1 0 1 1 0 1 0]
p
1
p
2
p
3
m
1
m
2
m
3
m
4
p
1
= 1 + 0 + 0 + 0 = 1
p
2
= 1 + 0 + 1 + 0 = 0
p
3
= 0 + 0 + 1 + 0 = 1
where
) (
4 3 2 1
m m m m
No need to compute for in the u vector as the identity
matrix portion of the G matrix will automatically reproduce [1 0 1 0].
Coding 54
For m = [0 0 0 0], [0 0 0 1], [0 0 1 0], . , [1 1 1 1], the corresponding
u codevectors are:
0 0 0 0
0 0 0 1
0 0 1 0
0 0 1 1
0 1 0 0
0 1 0 1
0 1 1 0
0 1 1 1
1 0 0 0
1 0 0 1
1 0 1 0
1 0 1 1
1 1 0 0
1 1 0 1
1 1 1 0
1 1 1 1
0 0 0 0 0 0 0
1 1 1 0 0 0 1
0 1 1 0 0 1 0
1 0 0 0 0 1 1
1 0 1 0 1 0 0
0 1 0 0 1 0 1
1 1 0 0 1 1 0
0 0 1 0 1 1 1
1 1 0 1 0 0 0
0 0 1 1 0 0 1
1 0 1 1 0 1 0
0 1 0 1 0 1 1
0 1 1 1 1 0 0
1 0 0 1 1 0 1
0 0 0 1 1 1 0
1 1 1 1 1 1 1
m = m
1
m
2
m
3
m
4
u = p
1
p
2
p
3
m
1
m
2
m
3
m
4 weight
0
4
3
3
3
3
4
4
3
3
4
4
4
4
3
7
Table 3.3
Coding 55
e) i) The first operation of the channel decoder is to compute the error
syndrome vector s, to ascertain the validity of the received codeword:
s = r H
T
From b), the parity check matrix H was obtained as:
(
(
(
1 1 1 0 1 0 0
1 1 0 1 0 1 0
1 0 1 1 0 0 1
H =
(
(
(
(
(
(
(
(
(
1 1 1
1 1 0
1 0 1
0 1 1
1 0 0
0 1 0
0 0 1
H
T
=
d) Minimum distance of the code is the minimum weight of the code
shown in Table 3.3 in the earlier slide;
i.e., minimum distance d
min
= 3
Can detect d
min
-1 = 2 error bits in codeword.
Can correct

1 5 . 1
2
3
2
1 d
min
= =
(
=
(

error bit in codeword.
Coding 56
s = r H
T
= [1 0 0 1 0 1 0]
(
(
(
(
(
(
(
(
(
1 1 1
1 1 0
1 0 1
0 1 1
1 0 0
0 1 0
0 0 1
= [0 0 1]
From the question r = [1 0 0 1 0 1 0]
r
1
r
2
r
3
r
4
r
5
r
6
r
7
s
1
= 1 + 0 + 0 + 1 + 0 + 0 + 0 = 0
s
2
= 0 + 0 + 0 + 1 + 0 + 1 + 0 = 0
s
3
= 0 + 0 + 0 + 0 + 0 + 1 + 0 = 1
s
1
s
2
s
3
where
Non-zero error
syndrome
indicating
invalid
codeword; i.e.,
there is error
in codeword
Error syndrome
value points to
error in bit r
3
r 1 0 0 1 0 1 0
e 0 0 1 0 0 0 0
u 1 0 1 1 0 1 0
Received codeword
Error pattern generation
Corrected codeword
+
Error Correction:
u = r + e
(modulo-2 addition)
r
1
r
2
r
3
r
4
r
5
r
6
r
7
Coding 57
s = r H
T
= [1 0 0 1 1 0 1]
(
(
(
(
(
(
(
(
(
1 1 1
1 1 0
1 0 1
0 1 1
1 0 0
0 1 0
0 0 1
= [0 0 0]
From the question r = [1 0 0 1 1 0 1]
r
1
r
2
r
3
r
4
r
5
r
6
r
7
s
1
= 1 + 0 + 0 + 1 + 1 + 0 + 1 = 0
s
2
= 0 + 0 + 0 + 1 + 0 + 0 + 1 = 0
s
3
= 0 + 0 + 0 + 0 + 1 + 0 + 1 = 0
s
1
s
2
s
3
where
Zero error
syndrome
indicating valid
codeword; i.e.,
there is no
error in
codeword
e) ii)

Chapter 7 Information Theory and Coding

Transféré par

Informations du document

Description originale:

Copyright

Formats disponibles

Partager ce document

Partager ou intégrer le document

Options de partage

Avez-vous trouvé ce document utile ?

Ce contenu est-il inapproprié ?

Droits d'auteur :

Formats disponibles

Chapter 7 Information Theory and Coding

Transféré par

Droits d'auteur :

Formats disponibles

Chapter 7: Information Theory &

Chapter 7: Information Theory &

Vous aimerez peut-être aussi