Introduction

to
Video Compression

Fast Forward Your Development www.dsp-ip.com

 Compression
p
Compression Representing information by less
bit than the original information
Lossless Compression Original information
and compressed information are identical.
identical
example LZ, TAR and other compression
techniques.
techniques
Lossy Compression Compressed info is not the
same as uncompressed
ncompressed infoinfo. E
Example:
ample MP3
MP3,
JPEG etc
Lossy
L compression
i is
i often
ft MODEL BasedB d
Compression

Fast Forward Your Development www.dsp-ip.com

Compression
p terms
Encoder Module which compress the
information
Decoder Module which decompress the
information
CODEC (en)CODer / DEcoder
Channel the medium which the information is
passed through for example ADSL line or disk

Channel Decoder
Encoder

Disk

Fast Forward Your Development www.dsp-ip.com

Sampling
p g
Digitization of analog signal
Voice - Sampling in time only
only. The number of
samples per second is the Frequency F. in
telephony voice sampling frequency is 8KHz to
telephony,
transfer 4KHz of sound.
Image
I - Sampling
S li in
i 2D space - Fx,
F F Fy. On
O thi
this
screen the images are sampled 1024x768
Video
Vid - Sampling
S li iin 2D
D space andd ti
time Fx,
F F Fy,
Fn. On TV Fn is 25 or 30 fps (frames per second)

Fast Forward Your Development www.dsp-ip.com

Quantization
Lowering the number of bits needed to describe
an object.
object
For example I scored 66 in the test an object with
100 steps need 7 description bits.
bits
How is she looking ? 10 ! Needs only 5 bits for
d
description.
i ti
Could we lower the number of bits more? What
i the
is th minimum?
i i ?
Quantization is not always linear - ADPCM

Fast Forward Your Development www.dsp-ip.com

MODEL BASED COMPRESSION

Voice
i
Audio
Video

Fast Forward Your Development www.dsp-ip.com

Model Base Transform - example
Transforming the information from a
representation of similar importance
to all variables to a representation of
varying importance
The
Th poor
sheriff
h iff problem
bl
The sheriff needs to telegram the
d
dangers criminal
i i l profile
fil tto
headquarters.
A telegram costs a cent for each
bit.
Sending it bit by bit would finish
the poor sheriff salary.

Fast Forward Your Development www.dsp-ip.com

Solution
The sheriff and headquarters
agree on a base
b that
h wouldld
describe images according to a
base known to both sides.
The transformation base is:
Mouth
Nose
Eyes
Ears
The values are the sizes of each
item quantized to millimeters
Mad Max will be transmitted as:
8 14 12 17
8,14,12,17

Fast Forward Your Development www.dsp-ip.com

Model Based Compression (Multimedia)

Pre- Model Based Quantize / Entropy

Processing Transform Prioritize Reorder Codingg

Fast Forward Your Development www.dsp-ip.com

Human Visual System
The human eye has two basic light receptors:
Rods Light Intensity receptors
Cons Colored light receptors

Fast Forward Your Development www.dsp-ip.com

The Human Eye
Rods Concentration >> Cons Concentration
Green Discrimination << Red,
Red Blue
Discrimination
Low
L Frequency
F > High
Hi h Frequency
F

Fast Forward Your Development www.dsp-ip.com

Model Based transformations
RGB (3 equally quantized colors) ->
>
YUV (Light Intensity + two color channels)
Pixel
Pi l based
b dd domain
i ->> Frequency
F d
domain
i

Fast Forward Your Development www.dsp-ip.com

Speech coding
In speech coding, the vocal tract is used as a
model:

Fast Forward Your Development www.dsp-ip.com

Audio / Music Coding
In general Audio Coding, the ear is used as a
model:
Frequencies -> Frequency bands
Masking and Temporal Masking are used

Fast Forward Your Development www.dsp-ip.com

IMAGE COMPRESSION

Fast Forward Your Development www.dsp-ip.com

What is a digital image?

Audio PCM
One 1-D array of
sample
BMP Image
Three 22-D
D arrays of
numbers
representing Red,
Green and Blue
values

Fast Forward Your Development www.dsp-ip.com

Image Compression? Why?

Image size = 720*580

720 580
3 Image Layers RGB =720*580*3
8 Bits per pixel 720
720*580*3*8
580 3 8
= 10022400 bits
Lots
L t off bit
bits ffor one L
Lena

Fast Forward Your Development www.dsp-ip.com

Color matching
How can we compare
colors so that the
content creators and
consumers know
kno whathat
they are seeing?
Many
M diff
differentt ways
including CIE
chromacity
h it di
diagram

Fast Forward Your Development www.dsp-ip.com

Video Color Transforms
Largely derived from older analog methods of coding
color for TV.
TV Luminance is separated from color
information.
YIQ is used to transmit TV signals in North America and
Japan.This coding also makes its way into VHS video
tape coding in these countries since video tape
technologies also use YIQ.
In Europe, video tape uses the PAL or SECAM codings,
which
hi h are based
b d on TV thath uses a matrix i transform
f
called YUV.
Finally,
Finally digital video mostly uses a matrix transform
called YCbCr that is closely related to YUV

Fast Forward Your Development www.dsp-ip.com

YUV Separation

Fast Forward Your Development www.dsp-ip.com

YUV Color Model

YUV codes a luminance signal (for gamma-corrected

signals) equal to Y , the luma".
Chrominance refers to the difference between a color
and a reference white at the same luminance. (U and V)

The transform is:

Fast Forward Your Development www.dsp-ip.com

RGB->YUV Color Transform
G
G
B B

Y
U

R
R

Fast Forward Your Development www.dsp-ip.com

YIQ Color Model

YIQ is used in NTSC color TV broadcasting.

broadcasting
Again, gray pixels generate zero (I;Q)
chrominance signal.
g
I and Q are a rotated version of U and V .

Th ttransform
The f is:
i

Fast Forward Your Development www.dsp-ip.com

YCbCr Color Model

1. The Rec. 601 standard for digital video uses

another color space YCbCr which closely
related to the YUV transform.
2. The YCbCr transform is used in JPEG image
compression and MPEG video compression.

For 8-bit coding:

Fast Forward Your Development www.dsp-ip.com

Sampling
p g
Digitization of analog signal
Voice - Sampling in time only
only. The number of
samples per second is the Frequency F. in
telephony voice sampling frequency is 8KHz to
telephony,
transfer 4KHz of sound.
Image
I - Sampling
S li in
i 2D space - Fx,
F F Fy. On
O thi
this
screen the images are sampled 1024x768
Video
Vid - Sampling
S li iin 2D
D space andd ti
time Fx,
F F Fy,
Fn. On TV Fn is 25 or 30 fps (frames per second)

Fast Forward Your Development www.dsp-ip.com

Quantization
Lowering the number of bits needed to describe
an object.
object
For example I scored 66 in the test an object with
100 steps need 7 description bits.
bits
How is she looking ? 10 ! Needs only 5 bits for
d
description.
i ti
Could we lower the number of bits more? What
i the
is th minimum?
i i ?
How to describe an object with one bit?

Fast Forward Your Development www.dsp-ip.com

Color based decimation
Our eyes have better resolution and scaling for
luminance then for color.
color
Compress color by using 4:2:0 method

Fast Forward Your Development www.dsp-ip.com

Counting the bits
How much can we save by color compression?
3*Image size in RGB 24 bit color representation.
representation
1 + 2*1/4 Image size in 4:2:0 YUV representation.
Compression ratio is 2 !!
Actual saving is bigger due to different Y and UV
quantization.
quantization

Fast Forward Your Development www.dsp-ip.com

Linear Transform
If the signal
g is formatted as a
vector, a linear transform can
be formulated as a matrix-
vector product that transform
the signal into a different
domain.
Examples:
K-L Transform
Discrete Fourier Transform
Discrete cosine transform
Discrete wavelet transform
Fast Forward Your Development
Energygy compaction
p p
property:
p y
The transformed signal vector
has few, large coefficients and
many nearly zero small
coefficients. These few large
coefficients can be encoded
efficiently with few bits while
retaining the majority of
energy of the original signal.
www.dsp-ip.com
Block-based Image Coding

Block-based
Block based image Advantages:
coding scheme: Parallel processing can
partitions the entire be applied
pp to p
process
image into 8 by 8 or individual blocks in
parallel.
16 by 16 (or other
Redundant information
size) blocks. in close proximity (like
The coding algorithm cache)
is applied to
individual blocks
independently.

Fast Forward Your Development www.dsp-ip.com

Transform - DCT
The DCT transform the data from pixel intensity
to frequency intensity.
intensity
Low frequency are important high frequency less

1 7 7 (2m 1)u (2n 1)v

(Youll 4 get Flaunch
(u , v) cos even if you cos dont remember m n the
0;
u 0 v 0 16 16
f (m, n) 7 7
IDCT 1f
formula l above)
b (2 m) 1)u (2n 1)v
8
F (u , v) cos cos 0 m, n 7; m n 0.
u 0 v 0 16 16

Fast Forward Your Development www.dsp-ip.com

DCT Coefficients Quantization

Fast Forward Your Development www.dsp-ip.com

AC Coefficients
AC coefficients are first
weighted
i h d withi h a quantization
i i 1 2 6 7 15 16 28 29

matrix: 3 5 8 14 17 27 30 43

C(i,j)/q(i,j)
C(i j)/q(i j) = Cq(i,j)
(i j) 4 9 13 18 26 31 42 44

Then quantized. 10 12 19 25 32 41 45 54

Then they are scanned in a 11 20 24 33 40 46 53 55

zig-zag order into a 1D 21 23 34 39 47 52 56 61

sequence to be subject to AC 22 35 38 48 51 57 60 62
Huffman
ff encoding.
di
36 37 49 50 58 59 63 64

Question: Given a 8 by 8
array how to convert it into a
array,
Zig-Zag scan order
vector according to the zig-
zag
g scan order? What is the
algorithm?
Fast Forward Your Development www.dsp-ip.com
DCT Basis Functions

Fast Forward Your Development www.dsp-ip.com

DCT compression Example
Original Image

Fast Forward Your Development www.dsp-ip.com

DCT 1 coefficient

Fast Forward Your Development www.dsp-ip.com

DCT 6 coefficients

Fast Forward Your Development www.dsp-ip.com

DCT 20 coefficient

Fast Forward Your Development www.dsp-ip.com

 JPEG Image Coding Algorithms

Quantization
Q DC
8x8 Matrix DC DPCM Huffman
block
DCT Q
Zig Zag AC
AC Scan Huffman
Code books

JPEG Encoding Process

Fast Forward Your Development www.dsp-ip.com

 Generalization of JPEG Coding

Transform Entropy
Color Frequency
Color, Quantize Reorder Coding

JPEG Encoding Process

Fast Forward Your Development www.dsp-ip.com

 Video Coding Basics

Fast Forward Your Development www.dsp-ip.com

Video Coding
Video coding is often implemented as encoding
a sequence off iimages.Motion
i compensationi
is used to exploit temporal redundancy between
successive
i frames.
f
Examples: MPEG-I, MPEG-II, MPEG-IV,
H.263, H.263+, H264
Existing video coding standards are based on
JPEG image compression as well as motion
compensation.

Fast Forward Your Development www.dsp-ip.com

 Video Coding Standardization
Scope
Only restrictions on the Bitstream, Syntax, and
Decoder are standardized:
Permits the optimization of encoding
Permits complexity
p y reduction
Provides no guarantees on quality

Fast Forward Your Development www.dsp-ip.com

Video Encoding

Buffer control
Current
frame x(t) r Bit stream
+ DCT Q VLC Buffer

Q-1 This is a simplified block
diagram where the
encoding of intra coded
IDCT frames is not shown.

Xp(t): predicted ^r(t): reconstructed residue

frame
+
^(t) reconstructed
x(t): t t d
Motion ^ current frame
x(t) x(t-1) Frame
Estimation &
Compensation Buffer
Motion vectors
Fast Forward Your Development www.dsp-ip.com
Video Encoding
Color Frequency
Transform Buffer control
Transform

+ Q Reorder Entropy

Q-1 This is a simplified block
diagram where the
encoding of intra coded
Tf-1 frames is not shown.

Xp(t): predicted ^r(t): reconstructed residue

frame
+
^(t) reconstructed
x(t): t t d
Motion ^ current frame
x(t) x(t-1) Frame
Estimation &
Compensation Buffer
Motion vectors
Fast Forward Your Development www.dsp-ip.com
Forward Motion Estimation

1 2 3 4 1 2 4
3
5 6 7 8 5 7 8
6
9 10 11 12 9 11 12
10
0
13 15 16
13 14 15 16 14

Current frame constructed From

different parts of reference frame Reference frame

Fast Forward Your Development www.dsp-ip.com

 Video sequence : Tennis frame 0, 1

previous frame current frame

50 50

100 100

150 150

200 200

50 100 150 200 250 300 350 50 100 150 200 250 300 350

Fast Forward Your Development www.dsp-ip.com

Frame Difference

Frame Difference :frame 0 and 1

Fast Forward Your Development www.dsp-ip.com

What is motion estimation?
Motion Vector Field of frame 1
50

-50

-100

-150

-200

-250
0 50 100 150 200 250 300 350 400

Fast Forward Your Development www.dsp-ip.com

What is motion compensation ?
Motion compensated frame

50

100

150

200

50 100 150 200 250 300 350

Fast Forward Your Development www.dsp-ip.com

Motion Compensated
p Frame Difference

Motion Compensated Frame Difference :frame 0 and 1

Frame Difference :frame 0 and 1

Fast Forward Your Development www.dsp-ip.com

Frame Types
Three types
yp of frames:
Intra (I): the frame is coded as if it is an image
Predicted (P): predicted from an I or P frame
Bi-directional (B): forward and backward predicted
from a pair of I or P frames.
A typical
i l fframe arrangement iis:
I1 B1 B2 P1 B3 B4 P2 B5 B6 I2
P1, P2 are both
b h fforward-predicted
d di d ffrom I1. B1, B2 are
interpolated from I1 and P1, B3, B4 are interpolated
from P1, P2, and B5, B6 are interpolated from P2, I2.
New Coding standards added other frame types: SP,
SI, D

Fast Forward Your Development www.dsp-ip.com

Macro-blocks and Blocks

Y(16x16) Cr (8x8)
RGB

Cb (8x8)

16x16x3

Fast Forward Your Development www.dsp-ip.com