LOSS CONCEALMENTS FOR REAL-TIME

MULTIMEDIA OVER THE INTERNET

Benjamin W. Wah
Department of Electrical and Computer Engineering
and the Coordinated Science Laboratory
University of Illinois at Urbana-Champaign
Urbana, IL 61801, USA
January 14, 2005

Loss Concealments for Real-Time Multimedia over the Internet

Outline

Motivations
Real-time multimedia
Voice-over-IP
Image downloads from Web servers
Real-time audio over IP networks
Real-time video transmissions
Characteristics of IP networks
Loss rate
Burstiness
End-to-end delay
Packet rate
Loss concealments without a channel-loss model
Trade-os among delay, bandwidth, and (perceptual) quality
Modied codec design (standard compliant versus quality)
Reconstruction algorithm
Benjamin W. Wah

Loss Concealments for Real-Time Multimedia over the Internet

Outline

VoIP Speech Quality

Interactive real-time communications:
End-to-end delay due to codec, network, and jitter buer
ITU G.114: one way delay, < 150 ms acceptable, < 400 ms noticeable
Mobile loop one-way delay about 100 ms; Mobile-VoIP-Mobile: about 300ms
Acoustic echo: due to PSTN wiring or PC setup
Noticeable for delays more than 30ms
Loss: some degradations on voice samples tolerable
Low bandwidth/congestion: last-mile problem
Long-burst or frequent short-burst intolerable
Codec
Codec in tandem: code conversions at hosts or gateway, causing degraded
quality and increased delay
Using PC as phone: Speaker and microphone not optimal for phone conversation
Standard low bit-rate speech codecs: Error propagation
Benjamin W. Wah

Loss Concealments for Real-Time Multimedia over the Internet

Outline

Delay-Quality Trade-os in Image Downloads

Lena at 0.125bpp 2 out of 8 packets lost in UDP, 470 ms end-to-end delay

TCP
JPEG2000 SDC
30.97 dB
4.01 sec.


UDP
JPEG2000 SDC
20.51 dB
0.71 sec.

UDP
Frequency-based MDC
25.21 dB
0.71 sec.


Some quality degradations can be tolerated in real-time downloads

Benjamin W. Wah

Loss Concealments for Real-Time Multimedia over the Internet

Outline

Network Environments: Packet Network

IPv4: best-eort, no real-time support
Packet size: less than MTU to avoid fragmentation
Packet rate: 20 - 30 packets per second
IPv6: best-eort, may support real-time trac
Wireless: future IP-based


Loss unavoidable in packet networks

Benjamin W. Wah

Loss Concealments for Real-Time Multimedia over the Internet

Outline

Network Environments: Transport-Layer Protocol

TCP
- Reliable but not suitable for real-time transmissions
- Connection oriented, more secure
- Allowed through rewalls
UDP
- Lossy and unreliable
- No congestion control mechanism to slow the ow
- Not permitted through rewalls
TCP in real-time mode
- Provide connection-oriented transmission without congestion avoidance
- Suitable for real-time multimedia for rewall compatibility


Loss of real-time multimedia not handled at the transport layer

Benjamin W. Wah

Loss Concealments for Real-Time Multimedia over the Internet

Outline

Network Environments: Application-Layer Protocol

H.323: umbrella standard for interoperability

RTP: no loss recovery scheme for real-time multimedia



Packet losses in real-time multimedia left for end-point applications

Benjamin W. Wah

Loss Concealments for Real-Time Multimedia over the Internet

IP Trac Loss Characteristics

IP Trac Loss Characteristics

Example connections
Connection
Loss rate
UIUC-Berkeley
low-medium
UIUC-W. China
medium-high
UIUC-Central Europe
high

0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0

1
0.9
UIUC-Berkeley
UIUC-W. China
UIUC-C. Europe

Distribution

Loss Rate

Loss behavior

UIUC-Berkeley
UIUC-W. China
UIUC-C. Europe

0.7
0.6
0.5
0.4

10
15
Time of day, hour

20

Loss rate can go up to 50%

Benjamin W. Wah

0.8

4
5
Burst length

Most losses have short burst lengths

7

Loss Concealments for Real-Time Multimedia over the Internet

IP Trac Loss Characteristics

Reducing Unrecoverable Loss by Interleaving

Bursty losses are dicult to handle
Interleaving: disperse bursty losses to isolated losses
P (f ail | i): prob. of losses that cannot be recovered under interleaving factor i

0.2

UIUC-Berkeley (i=1)
UIUC-Berkeley (i=2)
UIUC-Berkeley (i=3)
UIUC-Berkeley (i=4)

0.15

P(fail|i)

P(fail|i)

0.25

0.1
0.05
0
0

10
15
Time of day (hour)

20

0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0

UIUC-W. China (i=1)

UIUC-W. China (i=2)
UIUC-W. China (i=3)
UIUC-W. China (i=4)

10
15
Time of day (hour)

P(fail|i)

0.3

20

0.5
0.45
0.4
0.35
0.3
0.25
0.2
0.15
0.1
0.05
0

UIUC-C. Europe (i=1)

UIUC-C. Europe (i=2)
UIUC-C. Europe (i=3)
UIUC-C. Europe (i=4)

10
15
Time of day (hour)

20

Small interleaving factor 2 4 is enough

Multiple-description coding is promising
Benjamin W. Wah

Loss Concealments for Real-Time Multimedia over the Internet

Problem Addressed

Loss Concealment Problem

Design, analyze and evaluate robust end-to-end loss-concealment schemes

Allow reliable and real-time multimedia transmissions
Unreliable IP networks, like the Internet and wireless wide area networks

Benjamin W. Wah

Loss Concealments for Real-Time Multimedia over the Internet

Loss concealments for low bit-rate coded speech

Coded UDP
Packet

Description 1
Coded UDP
Packet

Lost

Frame with
even samples

Reconstructed
frame with
odd samples

Played Speech
Sequence

Frame with
odd samples

Depacketization
and Decoding

Frame with
even samples

Coding and
Packetization

Samplebased
Interleaving

Original speech
sequence

Description 0

Deinterleaving

Coder-Independent Sample-Based MDC of Low Bit-Rate Speech

Drawbacks:
Aliasing: caused by down sampling
Coding-frame time span lengthened

Benjamin W. Wah

10

Synthetic Tests

Tests on FS-CELP 1016

Original voice stream
SDC (no loss)
Sample-based MDC (no loss)
2-way LSP-based MDC
Both descriptions recd
Only 1 description recd

2-way LSP-based MDC (PW)

Both descriptions recd

Internet
Tests

Only 1 description recd

SDC (UIUC-Slovak, 46% loss)

Adaptive MDC
Adaptive MDC (PW)

Synthetic Tests (With Concealments)

Original voice stream
Loss %
0%
15%
30%
45%

FS-CELP G.723ACELP

G.732G.729
MP/MLQ

Loss Concealments for Real-Time Multimedia over the Internet

Loss concealments for low bit-rate coded speech

Coder-Dependent Parameter-Based MDC for Low Bit-Rate Speech

Played Speech
Sequence

Reconstructed
frame with
param. set S2

Decoded frame

Coded UDP
packet with Lost
param. set S2

Modified decoding

Description 1

Frame with
param. set S1

Deinterleaving

Coded UDP
packet with
param. set S1

Depacketization

Parameterbased
Interleaving

Coded frame

Modified coding

Original speech
sequence

Description 0

MDC design by correlation analysis

Interleave correlated parameters to multiple descriptions and reconstruct lost
parameters from description(s) received
Replicate uncorrelated parameters to multiple descriptions after reducing the
amount of parameters generated
Analyze speech sequence to determine critical parameters to replicate
Benjamin W. Wah

11

Loss Concealments for Real-Time Multimedia over the Internet

Loss concealments for JPEG2000 images

Coder-Dependent MDC for JPEG2000 Images

subdescription
00

subdescription
01

subdescription
02

......
Description 0

Original
Image

......
Description 1
subdescription
10

subdescription
11

subdescription
12

......

Packet sequence
subdescription
00

subdescription
10

subdescription
01

intradescription redudancy
interdescription redudancy

subdescription
11

UDP packet

subdescription
02

subdescription
12

......

interleaving set

Generate subdescriptions that minimize intra-description redundancies, without

sacricing inter-description redundancies
Probability of losing both subdescriptions in an interleaving set is less than 5%
Presence of intra-description redundancies reduces coding eciency
Benjamin W. Wah

12

Loss Concealments for Real-Time Multimedia over the Internet

Loss concealments for JPEG2000 images

Frequency-Based MDC System for JPEG2000 Images

Packet1

hdr

0
original
image

Packet3

Packet2

hdr

hdr

1
2
3
subdescriptions

Packet2
hdr

decoded
image

Packet4
1

Lost

description 1
description 2
Benjamin W. Wah

Sender side
Receiver side

Packet3
1

Packet4

description
Packet1

lost data

reconstructed
subbands
13

Loss Concealments for Real-Time Multimedia over the Internet

Loss concealments for JPEG2000 images

An Illustration on Smoke
Sending 512 512 smoke compressed at 0.25 bpp by JPEG2000
Between UIUC and Thailand1 (www.iced.moe.go.th)
Five out of the sixteen packets were lost in UDP

SDC and TCP

SDC and UDP

Frequency-Based MDC+UDP

PSNR: 30.96 dB,

PSNR: 22.03 dB,

PSNR: 28.72 dB,

Delay: 13.03 s.

Delay: 0.46 sec.

Delay: 0.46 sec.

Benjamin W. Wah

14

Loss Concealments for Real-Time Multimedia over the Internet

Future Work

Future Work
Low bit-rate coded speech
Bandwidth-quality trade-os
Exploiting voice properties in generating protection and concealments
MP3 coded audio
Exploiting redundancies in stereo audio for reconstruction
JPEG2000 coded images
Real-time TCP protocol with retransmissions to improve image quality

Benjamin W. Wah

15