Audio/Video compression

More about video compression and MPEG

Alain Bouffioux December, 20, 2006

Agenda
About video formats

Video compression
Audio/Video synchronisation The MPEG model

The MPEG model and its situation in a communication/storage context

2 AV Compression / Alain Bouffioux December, 20, 2006

Agenda
About video formats

Video compression
Audio/Video synchronisation The MPEG model

The MPEG model and its situation in a communication/storage context

3 AV Compression / Alain Bouffioux December, 20, 2006

Some video formats (1)

Max. component video signal bandwidth: 6 MHz. CCIR601 (CCIR is now ITU-R): Video sampling frequency: 13.5 MHz for 525 & 625 line standards (Shannon requirement) Synchronous with line (& image) sampling frequency Fsampling= 864*Fh for 625 line system (50Hz countries) Fsampling= 858*Fh for 525 line system (60Hz countries) Why synchronous? Points at the same place

RGB format

4 AV Compression / Alain Bouffioux December, 20, 2006

Some video formats (2)

YCbCr format Cb = B-Y, Cr = R-Y Eye is more sensitive to luminance than to chrominance (lower resolution needed for chrominance)
Red Green Blue Y Matrix LPF Cb Cr Modu SubC lator +

Composite Video

5 AV Compression / Alain Bouffioux December, 20, 2006

Some video formats (3)

The 4:2:2 format
Y sampling @ 13.5 MHz C sampling @ 6.75 MHz 8 bits per pixel 720 active points per line 576 lines active lines per image (2 fields) (625 lines) and 480 active lines (525 lines) Pixels are not square (e.g. for 480 lines, only 640 active points are needed - VGA format) Image size 720*576 or 720*480

The 4:2:0 format

Vertical chrominance resolution reduced by a factor 2 (average on two successive lines)

6 AV Compression / Alain Bouffioux December, 20, 2006

Some video formats (4)

SIF format (Source Intermediate Format) Half the vertical & horizontal resolution of 4:2:0 For 50Hz countries:
Luminance: 360*288 Chrominance: 180*120

CIF format (Common Intermediate Format)

Intermediate format used in videoconferencing (communication between US & Europe) resolution: 360*288 Sampling frequency: 30 Hz

QCIF (Quarter CIF)

Half the vertical & horizontal resolution of CIF.

7 AV Compression / Alain Bouffioux December, 20, 2006

Agenda
About video formats

Video compression
Audio/Video synchronisation The MPEG model

The MPEG model and its situation in a communication/storage context

8 AV Compression / Alain Bouffioux December, 20, 2006

Video compression in MPEG-1&2 (1/6)

Principles
removal of intra-picture redundancy : Image is decomposed in 8*8 pixels sub-images. Each sub-image contains redundant information DCT transformation (in frequency domain) de-correlates the input signal.( most energy in low spatial frequencies) removal of inter-picture redundancy : coding of difference with an interpolated picture (moving vectors) high frequent spatial frequencies quantized with lower resolution than low ones(remove irrelevancy) Zig-zag scan and VLC (remove redundancy)

9 AV Compression / Alain Bouffioux December, 20, 2006

Video compression in MPEG-1&2 (2/6)

Result 4:2:2 CCIR 601 resolution : 166 Mbps (=25images/sec *576lines* 720pixels* 2(lum & chrom) *8bits) 3-4 Mbps (mean) in MPEG2 4:2:0 SIF resolution : 30 Mbps (=25 images/sec *288 lines *352pixels* 1.5(lum & chrom) *8bits) 1.2 Mbps (CBR) in video CD (MPEG1)

10 AV Compression / Alain Bouffioux December, 20, 2006

Video compression in MPEG-1&2 (3/6)

Spatial redundancy reduction (DCT example)
1 39 1 44 1 50 1 59 1 59 1 61 1 62 1 62 1 44 1 51 1 55 1 61 1 60 1 61 1 62 1 62 1 49 1 53 1 60 1 62 1 61 1 61 1 61 1 61 15 3 15 6 16 3 16 0 16 2 16 1 16 3 16 1 15 5 15 9 15 8 16 0 16 2 16 0 16 2 16 3 15 5 15 6 15 6 15 9 15 5 15 7 15 7 15 8 1 55 1 56 1 56 1 59 1 55 1 57 1 57 1 58 1 55 1 56 1 56 1 59 1 55 1 57 1 57 1 58 1 26 0 - 1 -1 2 -2 3 -1 7 - 6 -1 1 - 9 - 2 -7 -2 0 -1 -1 1 2 0 2 -1 0 0 -3 2 -4 -5 -3 2 1 2 0 -1 -2 2 -3 0 1 0 -1 0 2 -2 0 -1 0 -1 1 2 1 -3 0 -1 0 1 1 1 -1 1 -1 0 0 1 -1 -1 0

DC T

Q uan tsa ton i i

1 58 -1 -1 0 0 0 0 0 0 -1 0 0 0 0 0 0 -1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

z g -zag scan i 158 0 -1 -1 -1 -1 EO B

11 AV Compression / Alain Bouffioux December, 20, 2006

Video compression in MPEG-1&2 (4/6)

Temporal redundancy reduction
I c r a se o f n e co p r ss on m e i r t ae I : I tr - oded p c t r n a c i u e P :P r d c t d p c t r e i e i u e B :B i- ir c tona ly n t r o a t d p c t r d e i l i e p l e i u e B i- ir c tona lp r d c ton d e i e i i

O r e ro f d p r sen t ton e a i

P r d c ton e i i

O r e ro f d tr n s ss on a m i i

12 AV Compression / Alain Bouffioux December, 20, 2006

Video compression in MPEG-1&2 (5/6)

Model of a possible encoder
R egu a t r l o Fom a t r con ve rs on i and p c u re it reo rde r M o ton i e s m a ton ti i 1 DC T / DC T Q V LC M u lt ip ex l B u fe r f 1Q / Da t a

I pu t n p c t re s i u

M o ton i ve c o rs t an d m od e s

P c t re iu s o re and t p red c ton i i

13 AV Compression / Alain Bouffioux December, 20, 2006

Video compression in MPEG-1&2 (6/6)

MPEG1 en MPEG2 video features
MPEG1
sequential picture resolution : SIF format 288(240)*356*24,25 or 30 Hz

MPEG2
sequential or interlaced various levels : low level (SIF: 288*356), main level (CCIR601: 576 * 720), high 1440 level (HDTV: 1152*1440), high level (EQTV: 1152*1920) various profiles (toolboxes) : simple profile (No B picture), main profile (=MPEG1+interlaced), SNR scalable profile (allows graceful degradation (noise improvement at same resolution), spatial scalable profile (hierarchical coding : improvement at higher resolution), high profile.

14 AV Compression / Alain Bouffioux December, 20, 2006

MPEG-4 AVC or H264

Defined about 10 years after MPEG-2 Similar structure used as for MPEG-2, but with some improvements And two additional functional blocks:
Intra-frame prediction: exploit the redundancy between neighboring blocks within a frame. A block is predicted using previously decoded neighboring blocks. Deblocking filter: filter applied at the block edges (except picture boundaries) in order to improved perceptual quality

Cost/benefits
Higher complexity Compression efficiency: factor 2

More than compression Network adaptation layer defining packets adapted for each transport network Uses
HD-DVD (MPEG2 & H264) & HD-camcorder Video services over IP networks Mobile television

15 AV Compression / Alain Bouffioux December, 20, 2006

Agenda
About video formats

Video compression
Audio/Video synchronisation The MPEG model

The MPEG model and its situation in a communication/storage context

16 AV Compression / Alain Bouffioux December, 20, 2006

Synchronisation
Synchronisation in the multimedia context
refers to the mechanism that ensures a temporal consistent presentation of the audio-visual

information to the user

17 AV Compression / Alain Bouffioux December, 20, 2006

Intramedia synchronisation
C ap u r m e t e ti T = C on s an t t P r sen a ton m e e t i ti

A ud o i s g na l i

E n code r

N e w o rk t

D e code r

A ud o i s g na l i

T 1 T 2 = T 1

T between capture & presentation = Constant Same clock frequency & Data on time Need for corresponding tools

18 AV Compression / Alain Bouffioux December, 20, 2006

Intermedia synchronisation
C ap u r m e t e ti T _a ud o = C on s a n t i t P r sen a ton m e e t i ti

A ud o i s gn a l i E n cod e r V d eo i s gn a l i C ap u r m e t e ti T _ v deo = C on s t n tT _A ud o i a = i N e w o rk t D e code r

A ud o i s gn a l i V d eo i s gn a l i P r sen a ton m e e t i ti

T_Audio = T_Video Sampled at the same time Presented at the same time) Possible tools : common time base and presentation control (media synchronisation with the common time base) Ex.: Lip_sync (requirement: |delay_difference| < 80msec)

19 AV Compression / Alain Bouffioux December, 20, 2006

Recovery of clock in CBR

CBR = Constant Bit Rate if the clock to recover is synchronous with transport clock Recovery of clock but not of common time base Remark : possibility to slave stream from DSM (Digital Storage Media) to local clock
Tm e n o m a ton ca rred i I f r i i b y ea ch sa p e m l

me ti C BR s tr a em P r ce ss ng o i F illng e ve l i l 50 % P ha se e rr r o F ile r t VCO R e co ve r d e c o ck l

20 AV Compression / Alain Bouffioux December, 20, 2006

Recovery of clock and time base in VBR

VBR = Variable Bit Rate Need for insertion of time stamps (OUTPUT TIME)
Output time stamp says for example : It is now 16h25 Receiver adjusts its own horloge to the received time stamp

Recovery of clock & of common time base

C o ck l C oun e r t D a a s tr a t em D a a s tr a t em Tm e s a p s i tm C oun e r sa p e t m l ( Tm e s a p s ) = i tm Tm e s a p s i tm V BR S tr a em Tm e s a p i tm e x tr c ton a i C hanne l R e co ve ry o f c o ck & m e l ti R e co ve r d c o ck e l Tm e n o m a ton ca rred i i f r i i on y b y m e s a p s l ti tm

F irs t m e s a p ti tm O he rs t

Tm e coun e r i t E rr r o

F ile r t

VCO

21 AV Compression / Alain Bouffioux December, 20, 2006

Synchronisation with common time base

Insertion of time stamp (=INPUT TIME)
Input time stamp says : Input has been sampled at 16h29. Receiver presents the sample at (its input time stamp + maximum encoding and decoding delay). Alternative: transmission of presentation time stamp (input time+delay)
S a p e " m e c o ck " m l T i l A sse b e fr m e m l a P r ce ss ng o i

M ed a np u t i i Tm e i sa p tm

Tm e c o c k i l C h a nn e l

Tm e c o c k i l R e co ve r d ) ( e M ed a i ou pu t t

C o pa rson o f m e c o ck m i ti l w ih sa p ed m e c o c k t m l ti l

B u fe rn g f i

P r ce ss ng o i

22 AV Compression / Alain Bouffioux December, 20, 2006

Getting data on time

On time Not too late, not too early No buffer over- or underflow Flow control : not applicable in broadcasting Common time base and Definition of a standard target decoder that describes the data consumption pattern of the receiver. Remark: Direct MPEG (Microsoft) does not use time information for clock recovery but relies on flow control

23 AV Compression / Alain Bouffioux December, 20, 2006

Streams
Idea of continuity (pipelining)

Carry time information for clock recovery

No flow control (allows broadcasting) The emitter must have a precise knowledge of the receiver data consumption pattern (explicit in MPEG STD) Just-in-time Shorter delay and smaller buffer size than with flow control Two aspects in synchronisation : Clock recovery & timing control (model & buffering)

24 AV Compression / Alain Bouffioux December, 20, 2006

Requirement on for stream transport

Data information BER (Bit Error Rate) requirement No repetition of frame possible FEC (Forward Error Correction) Time information No jitter

25 AV Compression / Alain Bouffioux December, 20, 2006

Agenda
About video formats

Video compression
Audio/Video synchronisation The MPEG model

The MPEG model and its situation in a communication/storage context

26 AV Compression / Alain Bouffioux December, 20, 2006

What is MPEG ? (1/2)

Moving Picture Expert Group International standard (ISO/IEC) Interoperability & economy of scale Various standard developed/under development Not only audio/video compression MPEG-1/2/4 - Compression of audio and video and multiplexing in a single stream Definition of the interface not of the codecs room for improvement

MPEG-1 : until 1.5 Mbps, for DSM Progressive picture, stereo (Dolby surround)

27 AV Compression / Alain Bouffioux December, 20, 2006

What is MPEG ? (2/2)

MPEG-2 : Various bit rates (CBR & VBR) Program stream for DSM, transport stream for network Interlaced picture, 5.1 audio channels Definition of various video levels (e.g. CCIR601 resolution: 4-9 Mbps, HDTV:15-25 Mbps) and profiles MPEG-3 : Cancelled, integrated in MPEG-2 (Initially : for HDTV) MPEG-4 : standard for audio, video and graphics in interactive 2D and 3D multimedia communication. (Initially : low bit rate for real-time personal communication) MPEG-7 : Multimedia contents description interface MPEG-21 : Focus on multimedia distribution and on DRM aspects.

28 AV Compression / Alain Bouffioux December, 20, 2006

The MPEG model (1/2)

A ud o i s gna l i

A ud o i en c ode r M u lt p e xe r i l T ran sm s s on i i cha nne l D g ita l s to rage m ed um i i or N e w o rk t D em u lt ip e xe r l

A ud o i de c ode r

A ud o i s gna l i

V deo i s gna l i

V deo i en c ode r

V deo i de c ode r

V deo i s gna l i

C ap tu red s gna s i l

P re s en ted s gna s i l

29 AV Compression / Alain Bouffioux December, 20, 2006

The MPEG model (2/2)

Compression of audio & video and multiplexing in a single stream

Guarantees intramedia and intermedia synchronisation.

MPEG defines an interface
bitstream syntax timing of the bitstream STD specifying timing requirement (ideal model)

Consequences:
Decoder should compensate deviations from STD Network should correct jitter introduced by the channel (RTD-LJ)

MPEG stream must be adapted to transmission channel formatting, error correction, channel coding (b.v.video-CD)

30 AV Compression / Alain Bouffioux December, 20, 2006

Components of the MPEG standard

The MPEG standard is composed of 3 main parts :
Audio : Specifies the compression of audio signals Video : Specifies the compression of video signals System : specifies how the compressed audio and video signals are combined in the multiplexed stream (program stream or transport stream).

Each part specifies :

The bitstream syntax The timing requirement and the related information (bit rate, buffer needs)

31 AV Compression / Alain Bouffioux December, 20, 2006

Synchronisation Mechanism (1/2)

S TC

A ud o i npu t i

A sse b e m l aud o fr m e s , i a S a p e ST m l C f rP TS o

A ud o i en code r

S a p e ST m l C f rPC R o ( CR ) S M u ltp e xe r i l and sys tm e en code r

V de o i npu t i

A sse b e m l p cu r s, i t e S a p e ST m l C f rP TS o

V de o i en code r

T r n s ss on a m i i channe l

A ud o i ou p u t t

C o pa r son m i P TS and S TC an d p r sen t ton e a i

A ud o i de code r

S ys tm de code r e an d de u ltp e xe r m i l C o pa r son m i P TS and S TC an d p r sen t ton e a i

V de o i ou p u t t

V de o i de code r

E x tr c ton a i o fPC R ( CR ) S

S TC

32 AV Compression / Alain Bouffioux December, 20, 2006

Synchronisation Mechanism (2/2)

PCR for TS & SCR for PS (but same concept)

Clock & time base recovery: Time-stamping at OUTPUT (PCR included in TS multiplex, SCR in pack header)
Audio & video clock locked to STC easy recovery (see next slide) Synchronisation of audio & video to common time base (Time stamping at Input) STD is defined (because of the absence of flow control) streams are such that STD buffers never over- or underflow In TS, many program in a single stream but unique clock per program.

Time information No Jitter requirement for transport

33 AV Compression / Alain Bouffioux December, 20, 2006

Clock recovery in receiver

V d e o c o ck i l d v de r i i PC R E r ro r L ow P a s s F i lt r e ( I t g ra t r ) ne o VC O A ud o c o ck i l d v de r i i

V deo i c o ck l

S TC S TC C o u n t r) ( e P TS L o a d f irs tPC R D e co d e d a ud o i C om p a r is o n P TS a n d S TC a nd P re s e n a to n t i S TC

A ud o i c o ck l

A ud o i ou t u t p

34 AV Compression / Alain Bouffioux December, 20, 2006

MPEG-2 program & transport streams

Program streams:
Relatively error free environment program stream packet may have variable and great length Single time base

Transport streams:
environment where errors are likely many programs (independent time base) Transport stream packet : fixed, 188 bytes Contains tables

35 AV Compression / Alain Bouffioux December, 20, 2006

MPEG in a communication context (1)

Typical communication system
I f m a ton nor i sou rce D g it l i a npu t i S ou rce Fom a t r en code C hanne l b its C hanne l M u lt ien code p ex l O t er h sou rce s M odu at l e M u ltp e i l a cce ss

E n c ryp t

B its trea m

S yn ch ro n sa ton i i

D g it l i a w a ve f m or

C hanne l

Fom a t r I f m a ton nor i s nk i

S ou rce de code D g it l i a ou t u t p

D e c ryp t

C hanne l de code

D e u lt im p ex l

D e odu - M u ltp e m i l at l e a cce ss O t er h de s tna ton i i

36 AV Compression / Alain Bouffioux December, 20, 2006

MPEG in a communication context (2)

MPEG : Source coding only (bit rate reduction) + multiplexing The MPEG stream must be adapted to the channel in what concern its physical characteristics and in order to get the required QoS (Quality of Service) & Security
Encryption Channel coding (forward error correction, interleaving, modulation codes) multiplexing & formatting modulation (frequency allocation) multiple access method

Some channels : CD/DVD - satellite - cable - ATM - 1394

37 AV Compression / Alain Bouffioux December, 20, 2006

MPEG-2 in a communication context (3)

A simple view of MPEG-2 in the communication context
ES E e en a ry ( lm t S tr a ) em TS M u lt ip e x ng l i V d eo i E n code r A ud o i en cod e r TS ( pr n o g rm s ) a TS ( r n spo r tS tr a ) Ta em or PS P r g r m S tr a ) ( o a em A da p a ton t i o he t t ch ann e l

A ud o , i v d eo i so u rce s

C ab e l

A da p a ton t i o he t t ch ann e l PS M u lt ip e x ng l i PS ( pr 1 o g rm ) a A da p a ton t i o he t t ch ann e l

S a e llie t t

D sc i

M PEG 2 co p r s s o n a ye r m e i l

M PEG 2 s y s e a ye r tm l

DVB ,DVD ...

38 AV Compression / Alain Bouffioux December, 20, 2006

39 AV Compression / Alain Bouffioux December, 20, 2006