TC-502 INFORMATION THEORY

SOURCE CODING
 SOURCE CODING
An important problem in digital communications and
computer storage is the efficient transmission and storage of
information.
Transmission and storage of digital data require the
information to be represented in a digital or binary form.
Thus there is a need to perform source coding, the process of
encoding the information source message to a binary code
word, and lossless and faithful decoding from the binary code
word to the original information source message.
The goal of source coding for digital systems is two-fold:
1. Uniquely map any arbitrary source message to a binary
code and back again
2. Efficiently map a source message to the most compact
binary code and back again
 SOURCE CODING
Source coding is shown in figure below
The source encoder efficiently maps the text to a
binary representation and the source decoder
performs the reverse operation.

It should be noted that the channel is assumed noiseless. In the

presence of noise, channel codes are also needed.
SOURCE CODING
 SOURCE CODING
A source code is identified as:
A non-singular code if all the code words are distinct.
A block code of length if the code words are all of
fixed length .
All practical codes must be uniquely decodable.
The following observations can be made:
A block code of length which is non-singular is
uniquely decodable
It is a non-trivial exercise to prove whether a code is
uniquely decodable in general
A code is proven to be NOT uniquely decodable if there
is at least one instance of a non-unique decoding
 Instantaneous Codes
Although usable codes have to be at least uniquely
decodable, there is a subclass of uniquely decodable
codes, called instantaneous codes, that exhibit
extremely useful properties for the design and analysis
of such codes and the practical implementation of the
encoding and decoding processes.
 Instantaneous Codes
A code that is uniquely decodable but not
instantaneous requires the past and current code
symbols to be buffered at the receiver in order to
uniquely decode the source message. An instantaneous
code allows the receiver to decode the current code
word to the correct source message immediately upon
receiving the last code symbol of that code word (e.g.,
the decoding is performed on the fly).
Figure below graphically depicts the universe of all codes and how
the different classes of codes we have described are subsets of one
another.
That is, the class of
all instantaneous
codes is a subset
of the class of all
uniquely
decodable codes
which in turn is a
sub-class of all
non-singular
codes.
 Construction of Instantaneous Codes
The prefix condition not only makes it easy to
determine whether a given code is instantaneous or
not but it can also be used to systematically design
an instantaneous code with specified lengths for the
individual code words.
The problem can be stated as follows. For a source
with symbols it is required to design the
individual code words with specified code word
lengths of 1 , 2 ,, such that the code is
instantaneous (i.e., it satisfies the prefix condition).
 Construction of Instantaneous Codes
To design the code the code word lengths are sorted
in order of increasing length. The code words are
derived in sequence such that at each step the
current code word does not contain any of the other
code words as a prefix. A systematic way to do this is
to enumerate or count through the code alphabet.
 Decoding Instantaneous Codes
Since an instantaneous code has the property that the
source symbol or message can be immediately decoded
upon reception of the last code character in the current
code word the decoding process can be fully described by
a decoding tree or state machine which can be easily
implemented in logic.
The receiver simply jumps to
the next node of the tree in
response to the current code
character and when the leaf
node of the tree is reached the
receiver produces the
corresponding source symbol,
, and immediately returns to
the root of the tree. Thus
there is no need for the
receiver to buffer any of the
received code characters in
order to uniquely decode the
sequence.
Properties of Instantaneous Codes
Sensitivity to Bit Errors