Lecture Slides for

INTRODUCTION TO

Machine Learning
2nd Edition
ETHEM ALPAYDIN
The MIT Press, 2010
alpaydin@boun.edu.tr
http://www.cmpe.boun.edu.tr/~ethem/i2ml2e

CHAPTER 3:

Bayesian Decision
Theory

Classification
Example: Credit

scoring
Differentiating
between low-risk
and high-risk
customers from
their income and
savings

Lecture Notes for E Alpaydn 2010 Introduction to Machine Learning 2e The MIT
Press (V1.0)

Probability and Inference

Result of tossing a coin is {Heads,Tails}
Random var X {1,0}

Bernoulli: P {X=1} = poX (1 po)(1 X)

Sample: X = {xt }Nt =1

Estimation: po = # {Heads}/#{Tosses} = t
xt / N
Prediction of next toss:
Heads if po > , Tails otherwise
Lecture Notes for E Alpaydn 2010 Introduction to Machine Learning 2e The MIT
Press (V1.0)

Classification
Credit scoring: Inputs are income and

savings.
Output is low-risk vs high-risk
Input: x = [x1,x2]T ,Output: C ( {0,1} )
Prediction: C 1 if P (C 1|x ,x ) 0.5
1
2
choose

C 0 otherwise

or
C 1 if P (C 1|x1 ,x2 ) P (C 0|x1 ,x2 )
choose
C 0 otherwise
Lecture Notes for E Alpaydn 2010 Introduction to Machine Learning 2e The MIT
5
Press (V1.0)

Bayes Rule

prior

posterior

likelihood

P C p x | C
P C | x
p x
evidence

P C 0 P C 1 1

p x p x | C 1 P C 1 p x | C 0 P C 0
p C 0| x P C 1| x 1

Lecture Notes for E Alpaydn 2010 Introduction to Machine Learning 2e The MIT
Press (V1.0)

Bayes Rule: K>2 Classes

p x | Ci P Ci
P Ci | x
p x
p x | Ci P Ci
K
p x| Ck P Ck
k1

P Ci 0 and P Ci 1
i 1

choose Ci if P Ci | x max k P Ck | x
Lecture Notes for E Alpaydn 2010 Introduction to Machine Learning 2e The MIT
Press (V1.0)

Losses and Risks

- Menerima low-risk meningkat-kan
keuntungan
- Menolak high-risk menurunkan
kerugian
Kerugian :
- Menerima high-risk costumer
Kehilangan keuntungan
- Menolak low-risk costumer

Lecture Notes for E Alpaydn 2010 Introduction

to Machine Learning 2e The MIT Press (V1.0)

Losses and Risks

Actions: i ( memilih kelas Ci jika diberikan

input)
Loss : ik ( kesalahan memilih Ci jika kls
sebenarnya Ck ) K
R i | x ikP Ck | x

Resiko melakukan
k1 aksi i (Duda and Hart,
choose i if R i | x min kR k | x
1973)

Lecture Notes for E Alpaydn 2010 Introduction to Machine Learning 2e The MIT
9
Press (V1.0)

Losses and Risks: 0/1 Loss

0 if i k
ik
1 if i k
K

R i | x ikP Ck | x
k1

P Ck | x
k i

1 P Ci | x
For minimum risk, choose the most probable class
Lecture Notes for E Alpaydn 2010 Introduction to Machine Learning 2e The MIT
Press (V1.0)

10

Losses and Risks: Reject

0 if i k

ik if i K 1 , 0 1
1 otherwise

loss karena memilih K 1

Resiko reject
K

R K 1|x P Ck |x
k 1

Resiko memilih kelas Ci

R i |x P Ck |x 1 P Ci |x
k i

Lecture Notes for E Alpaydn 2010 Introduction to Machine Learning 2e The MIT
11
Press (V1.0)

The optimal decision rule is to

choose Ci if R( i | x) R( k | x), k i and
R( i | x) R( K 1 | x)
reject

if R ( K 1 | x) R( i | x), i 1, 2,..., K 1

atau
choose Ci if P Ci | x P Ck | x k i and P Ci | x 1
reject
otherwise

Lecture Notes for E Alpaydn 2010 Introduction to Machine Learning 2e The MIT Press
(V1.0)

12

Discriminant Functions
choose Ci if gi x max kgk x

gi x , i 1, ,K

R i | x

gi x P Ci | x
p x | C P C
i
i

K decision regions R1,...,RK

R i x | gi x maxkgk x
Lecture Notes for E Alpaydn 2010 Introduction to Machine Learning 2e The MIT
13
Press (V1.0)

K=2 Classes
Dichotomizer (K=2) vs Polychotomizer

(K>2)
g(x) = g1(x) g C
2(x)
1 if g x 0
choose

C2 otherwise

Lecture Notes for E Alpaydn 2010 Introduction to Machine Learning 2e The MIT
14
Press (V1.0)

Utility Theory
Melakukan keputusan yang rasional ketika

tidak yakin
dengan keadaan yg sebenarnya (state)
Prob of state k given exidence x: P (Sk|x)
Fungsi utility Uik ( seberapa baik

melakukan aksi i ketika state adalah Sk )

Expected utility:
EU i | x UikP Sk | x
k

Choose i if EU i | x max EU j | x
j

Lecture Notes for E Alpaydn 2010 Introduction to Machine Learning 2e The MIT
15
Press (V1.0)

Association Rules
Association rule: X Y
People who buy/click/visit/enjoy X are also

likely to buy/click/visit/enjoy Y.
A rule implies association, not necessarily
causation.

Lecture Notes for E Alpaydn 2010 Introduction to Machine Learning 2e The MIT
16
Press (V1.0)

Association measures
Support (X Y):
# customers who bought X and Y
P X,Y
# customers
Confidence (X Y):
P X,Y
P Y | X
P(X)
# customers who bought X and Y

# customers who bought X

Lift (X Y):

P X,Y P(Y| X)

P(for
X)EPAlpaydn
(Y) 2010P(Introduction
Y)
Lecture Notes
to Machine Learning 2e The MIT
Press (V1.0)

17

Apriori algorithm (Agrawal et al.,

1996)
For (X,Y,Z), a 3-item set, to be frequent (have

enough support), (X,Y), (X,Z), and (Y,Z) should

be frequent.
If (X,Y) is not frequent, none of its supersets
can be frequent.
Once we find the frequent k-item sets, we
convert them to rules: X, Y Z, ...
and X Y, Z, ...

Lecture Notes for E Alpaydn 2010 Introduction to Machine Learning 2e The MIT
Press (V1.0)

18

P C1 | x
Log odds:
log

P C2 | x

Lecture Notes for E Alpaydn 2010 Introduction to Machine Learning 2e The MIT
Press (V1.0)

19

 Jika A dan B dua kejadian , dengan P ( A) 0,

peluang bersyarat B diberikan A, didefinisikan sebagai:
P( A B)
P B A
P ( A)
Teorema Bayes :

A1 , A2 ,..., Ak
Jika kejadian-kejadian
adalah partisi dari ruang sampel S maka
untuk kejadian B sembarang dari S
sedemikian sehingga
berlaku:
P
( B Ai ).P ( Ai )
P ( Ai P(B)>0
B)
P ( Ai B )
k
P( B)
P( B Ai ).P( Ai )
i 1

20