Vous êtes sur la page 1sur 70

 ❄ ✲ ✲ ✲ ✲

 ✲ ✲ ❄ ✛ ✛ ✛ ✛

{ a 1 ,

, P ( a K ) a k

1 k K P ( a K )

K =1 P ( a k ) = 1

, a K }

P ( a 1 ) ,

k

a 1 a 2 a 3 a 4

 a 1 → 00 a 2 → 01 a 3 → 10 a 4 → 11 a 1 → 0 a 2 → 10 a 3 → 110 a 4 → 111

P ( a 1 ) = 1 2 , P ( a 2 ) = 1 4 , P ( a 3 ) = P ( a 4 ) = 1

8 ,

a k P ( a k ) I ( a k ) = log 2 P ( a k ) I ( a k ) = 0 P ( a k ) = 1

K

k =1

P ( a k ) log 2 P ( a k ) .

2 L L 2 L L H τ s H/τ s

, a K }

, b J }

P ( b j | a k ) a k b j { 0 , 1 }

{ b 1 ,

{ a 1 ,

1

✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏

1

1

{ a 1 ,

, s K ( t)

τ c s 1 ( t ) ,

, a K }

τ c

, b J } τ c

{ b 1 ,

, b J }

τ c

{a 1 ,

, a K } { b 1 ,

C R R < C R > C R C

X X Pr X X

[0 , 1] U X

Pr X ( X ) = 1

U ⊂ X V ⊂

X U ∩ V = Pr X ( U ∪ V ) = Pr X ( U ) + Pr X ( V )

x X { x } p X ( x ) = Pr X [ { x } ] U X X

Pr X [ U ] = p X ( x )

x ∈U

p X : X → R

p X ( x ) 0 , x ∈ X

p X ( x ) = 1

x ∈X

Pr X X = ( X , p X )

X = ( X , p X ) V

Pr X [ V = v ] = Pr X [ { x ∈ X | V ( x ) = v } ] ,

V = v V ( x ) = v V : X → R

E X [ V ] = p X ( x ) V ( x ) .

x ∈X

( X , p X ) V X Pr X [ V ] = 0 V = ( V , p V ) x ∈ V p V ( x ) p X ( x ) p V ( x ) = p X ( x ) / Pr X [ V ] U X U V X U ∩ V V

Pr X [ U | V ] = Pr V [ U ∩ V ] = Pr X [ U V ]

Pr X [ V ]

.

Pr X [ U | V ] X

U V X Pr X [ U , V ] = Pr X [ U ∩ V ]

U V

Pr X [ U | V ] = Pr X [ U ]

Pr X [ V | U ] = Pr X [ V ]

Pr X [ U , V ] = Pr X [ U ]Pr X [ V ]

U V u v

Pr X [ U = u, V = v ] = Pr X [ U

= u ]Pr X [ U = u ]

XY X × Y p XY X Y p X p Y p XY

p X ( x ) = Pr XY [ X = x ] = p XY ( x, y ) ,

y ∈Y

p Y ( y ) = Pr XY [ Y = y ] = p XY ( x, y ) .

x

∈X

( X , p X ) ( Y , p Y ) X Y p X ( x ) > 0 Y = y X = x

Pr XY [ Y = y | X = x ] = p XY ( x, y )

p X ( x ) .

Y = y X = x X ×{ y } { x }×Y X ×

Y { ( x, y ) }

X = x Y = y

Pr XY [ X = x | Y = y ] = p XY ( x, y )

p Y ( y ) .

X = x Y = y

p XY ( x, y ) = p X ( x ) p Y ( y ) .

X × Y X Y

X Y X Y XY X × Y Pr[ U ] X Y XY U X Y X × Y Pr Pr X Pr Y Pr XY

p X p Y p XY ( x, y ) ∈ X × Y

p ( x ) =

p ( y ) = p Y ( y )

p ( x, y ) = p XY ( x, y ) p ( x | y ) = Pr XY [ X = x | Y = y ]

= x ]

p ( y | x ) = Pr XY [ Y = y | X

p X ( x )

X X

(

n

n > 0

(

) X n 1 X n

x 1 ,

x 1 ,

, x n ,

,

x n ) ∈ X n

p ( x 1 , x 2 ,

, x n ) = Pr[ X 1 = x 1 , X 2 = x 2 ,

, X n = x n ]

X n

n > 0

( x 1 ,

, x n ) ∈ X n

p

( x 1 ,

, x n ) =

n

p

( x i )

i =1

n > 0

Pr[ X 1 = x 1 , X 2 = x 2 ,

, X n = x n ] = Pr[ X 2 = x 1 , X 3 = x 2 ,

, X n+1 = x n ] .

n > 0

Pr[ X n+1 = x n+1 | X 1 = x 1 , X 2 = x 2 ,

, X n = x n ] = Pr[ X n +1 = x n+1 | X n = x n ]

Pr[ X n+1 = x n+1 | X n = x n ] n p ( x n+1 | x n )

p ( x 1 , x 2 ,

, x n ) = p ( x 1 ) p ( x 2 | x 1 )

p ( x n | x n 1 )

( p ( x )) x X Π = ( p ( x 2 | x 1 )) ( x 1 ,x 2 ) X 2 ( λ ( x )) x X )

Pr[ X n +1 = x n+1 | X 1 = x 1 , X 2 = x 2 ,

, X n = x n ]

= Pr[ X n+1 = x n +1 | X n +1 = x n +1 ,

, X n = x n ]

p ( x 1 , x 2 ,

, x n ) = p ( x 1 ,

, x ) p ( x +1 | x 1 ,

, x )

p ( x n | x n 1 ,

, x n )

( Y i ) i> 0 Y = X Y i = ( X i , X i +1 ,

, X i + )

XY X = x Y = y p ( x | y ) p ( x ) x y y x x p ( x ) p ( x | y ) p ( x | y ) p ( x ) x p ( x | y ) p ( x ) x x x x x x y

x p ( x | y ) = 1 x = y x I ( x ) = f ( p ( x )) f ()

x I ( x ) = f ( p ( x )) f () x p ( x ) x y f ( p ( x, y )) = f ( p ( x ) p ( y )) = f ( p ( x )) + f ( p ( y )) I ( x ) = λ log p ( x ) λ < 0 p ( x ) λ

X { 0 , 1 } p (0) = p (1) = 1 / 2 X = 0 X = 1

I (0) = λ log p (0) = λ log 2 = 1 ,

λ = 1/ log 2 I ( x )

I ( x ) = log 2 p ( x ) .

n n

, a 15 }

k 0 k < 16 P ( a k ) = 1 /16 a k I ( a k ) = log 2 (1 / 16) = 4

k

a k I ( a k ) = log 2 p ( a k )

{ a 0 ,

XY y ∈ Y x ∈ X y p ( x ) x p ( x | y )

x ∈ X y ∈ Y

I ( x ; y ) = log 2

p ( x | y )

p ( x ) .

p ( x, y ) = p ( x | y ) p ( y ) = p ( y | x ) p ( x )

I ( x ; y ) = I ( y ; x ) = log 2

p ( x, y )

p ( x ) p ( y ) .

I ( x ; y ) I ( x ; y ) > 0 I ( x ; y ) < 0 I ( x ; y ) = 0

a 1 a 2 a 1 a 2 b 1 b 2

I (x, y ) (x, y )

x

a

a

1

2

P ( y | x )

✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏ ✏

1

1

y

b

b

1

2

p ( b 1 | a 1 ) = p ( b 2 | a 2 ) = 1 p ( b 1 | a 2 ) = p ( b 2 | a 1 ) = p ( a 1 ) = p ( a 2 ) = 1 / 2

p ( a 1 , b 1 ) = p ( a 2 , b 2 ) = 1

2

p ( a 1 , b 2 ) = p ( a 2 , b 1 ) =

2 ,

p ( b 1 ) = p ( b 2 ) = 1 / 2 ( a k , b j )

I ( a 1 ; b 1 ) = I ( a 2 ; b 2 ) = log 2 2(1 ) I ( a 1 ; b 2 ) = I ( a 2 ; b 1 ) = log 2 2 .

< 1/2 I ( a 1 ; b 1 ) I ( a 1 ; b 2 ) b 1 a 1 b 2 a 1 = 1 /2

I ( x ; y )

x y p ( x | y ) = 1 y = x

I ( x ; y ) = log 2

p ( x | y )

p

( x )

= log 2

1

p ( x ) .

x

x ∈ X

I ( x ) = log 2 p ( x ) .

p ( x ) = 1 I ( x ) = 0

XY x

y

x ∈ X y ∈ Y

I ( x | y ) = log 2 p ( x | y ) .

I ( x ; y ) = I ( x ) I ( x | y ) ,

x y x x y

XY

X Y XY

I ( X ; Y ) =

x ∈X y ∈Y

p ( x, y ) I ( x ; y ) =

x ∈X y ∈Y

p ( x, y ) log 2

p ( x, y )

p ( x ) p ( y ) .

X

X

H ( X ) = p ( x ) I ( x ) = p ( x ) log 2 p ( x ) .

x

∈X

x

∈X

X Y XY

H ( X | Y ) =

p ( x, y ) log 2 p ( x | y ) .

x ∈X y ∈Y

I ( X ; Y ) = H ( X ) H ( X | Y ) .

X K H ( X )

H ( X ) log 2 K

X

log 2

log 2 z < ( z 1) log 2 e z > 0 , z = 1

log 2 z = ( z 1) log 2 e

z = 1

H ( X ) log 2 K 0

H ( X ) log 2 K

=

=

p ( x ) log 2

x

p ( x ) log 2

x

1

p ( x )

x

1

Kp ( x ) .

p ( x ) log 2 K

z = 1 /( Kp ( x ))

 H ( X ) − log 2 K ≤ ≤

log 2 e p ( x )

x

Kp ( x ) 1

1

log 2 e

x

1

K

x

p ( x ) = 0 ,

1 /( Kp ( x )) = 1 x

p ( x ) = 1 /K x

2 L L log 2 K

X

XY

I ( X ; Y ) X Y

I ( X ; Y ) 0 ,

X Y

I ( X ; Y ) 0

I ( X ; Y ) = p ( x, y ) log 2

x,y

p ( x ) p ( y )

p ( x, y )

log 2 e

x,y

p ( x, y ) p ( x ) p ( y ) p ( x, y )

1

log 2 e p ( x ) p ( y ) p ( x, y )

x,y

x,y

log 2 e p ( x ) p ( y ) 1 = 0 ,

x

y

p ( x ) p ( y ) /p ( x, y ) = 1 ( x, y ) x y p ( x, y ) = p ( x ) p ( y ) I ( X ; Y ) = 0 X Y

y Y X X Y

XY

H ( X | Y ) H ( X ) .

· · · −− ·

X X l l X X = l 1 X l X { 0 , 1 } m ∈ X | m|

X { 0 , 1 }

ϕ X { 0 , 1 } x X

( x 1 ,

X

{ 0 , 1 }

, x ) ϕ ( x 1 ) ··· ϕ( x )

X X = ( X , p X ) X p X

X = ( X , p X ) H ϕ X ϕ | ϕ|

ϕ

| ϕ | = | ϕ( x ) | p X ( x ) ,

x ∈X

E ( ϕ) = H

| ϕ| .

X L > 0 X L X L p X Φ X

L Φ = lim

L

→∞

1

L

( x 1 ,

,x

L ) ∈X L

p X ( x 1 ,

, x L ) | Φ( x 1 ,

, x L ) | ,

X Ψ

E (Φ) = H ( X )

L (Φ) .

X K X n

log 2 K n < 1 + log 2 K.

n < log 2 K

, x K } x k k 1

n n 2 n K

n < log 2 K n 2 n < K X

X = { x 1 ,

n X X H ( X ) /n H ( X ) log 2 K E 1 E = 1

 H ( X ) = log 2 K n = log 2 K X = { 0 , 1 , , 9 }

H ( X ) / 4 = (log 2 10) / 4 0 , 83

, 99} X

X 2 H ( X 2 ) = log 2 100 = 2 H ( X ) 2 7 = 128 X 2 H ( X 2 ) / 7 = (2 log 2 10) / 7 0 , 95 X 3 0 , 996

X 2 = { 00, 01,

ϕ L X L

| ϕ L |

1

L

< L + log 2 K.

H ( X ) / log 2 K H ( X ) = log 2 K

n n

{a 1 , a 2 , a 3 } a 1 1 a 2 10 a 3 100

✭✭

❤ ❤❤

❤ ❤❤

◗ ◗ ◗ ◗✏ ✏ ✏ ✭ ✭ ✭ ✭ ✭✭

❤ ❤❤

❤ ❤❤

✭✭

{ a 1 , a 2 , a 3 , a 4 }

 a 1 m 1 = 00 m 1 = 00 m 1 = 0 a 2 m 2 = 01 m 2 = 01 m 2 = 10 a 3 m 3 = 10 m 3 = 001 m 3 = 110 a 4 m 4 = 11 m 4 = 011 m 4 = 111

✟ ✟ ✟ ✟ ✘ ✘ ✘ ✘ ✘ ✘ ✘

❳ ❳

❍ ❍ ❍ ❍ ❍ ❍ ❍ ❳ ❳ ❳ ❳

✘ ✘

</