Statistics 512 Notes 15: Properties of Maximum

Likelihood Estimates Continued

Computation of maximum likelihood estimates
Example 2: Logistic distribution. Let
1
, ,
n
X X K
be iid with
density
2
exp{ ( )}
( ; )
(1 exp{ ( )})
x
f x
x

+
,
, x < < < <
.
The log of the likelihood simplifies to:
1 1
( ) log ( ; ) 2 log(1 exp{ ( )})
n n
i i
i i
l f X n nX X

+

Using this, the first derivative is
1
exp{ ( )}
'( ) 2
1 exp{ ( )}
n
i
i
i
X
l n
X

Setting this equal to 0 and rearranging terms results in teh

equation:
1
exp{ ( )}
1 exp{ ( )} 2
n
i
i
i
X n
X

+
. (*)
Although this does not simplify, we can show the equation
(*) has a unique solution. The derivative of the left hand
side of (i) simplifies to

( )
2
1 1
exp{ ( )} exp{ ( )}
0
1 exp{ ( )}
1 exp{ ( )}
n n
i i
i i
i
i
X X
X
X





>
+
+

Thus, the left hand side of (*) is a strictly increasing
function of . Finally, the left hand side of (*) approaches
0 as and approaches n as . Thus, the
equation (*) has a unique solution. Also the second
derivative of
( ) l
is strictly negative for all ; so the
solution is a maximum.
How do we find the maximum likelihood estimate that is
the solution to (*)?
Newtons method is a numerical method for approximating
solutions to equations. The method produces a sequence of
values
(0) (1)
, , K that, under ideal conditions, converges
to the MLE

MLE
.
To motivate the method, we expand the derivative of the
log likelihood around
( ) j
:
( ) ( ) ( )

0 '( ) '( ) ( ) ''( )
j j j
MLE MLE
l l l +
Solving for

MLE
gives
( )
( )
( )
'( )

''( )
j
j
MLE
j
l
l

This suggests the following iterative scheme:
( )
( 1) ( )
( )
'( )
''( )
j
j j
j
l
l

+

.
The following is an R function that uses Newtons method
to approximate the maximum likelihood estimate for a
logistic distribution:
mlelogisticfunc=function(xvec,toler=.001){
startvalue=median(xvec);
n=length(xvec);
thetahatcurr=startvalue;
# Compute first deriviative of log likelihood
firstderivll=n-2*sum(exp(-xvec+thetahatcurr)/(1+exp(-
xvec+thetahatcurr)));
# Continue Newtons method until the first derivative
# of the likelihood is within toler of 0
while(abs(firstderivll)>toler){
# Compute second derivative of log likelihood
secondderivll=-2*sum(exp(-xvec+thetahatcurr)/(1+exp(-
xvec+thetahatcurr))^2);
# Newtons method update of estimate of theta
thetahatnew=thetahatcurr-firstderivll/secondderivll;
thetahatcurr=thetahatnew;
# Compute first derivative of log likelihood
firstderivll=n-2*sum(exp(-xvec+thetahatcurr)/(1+exp(-
xvec+thetahatcurr)));
}
list(thetahat=thetahatcurr);
}
Issues with Newtons method:
1. The method does not work if
( )
''( ) 0
j
l .
2. The method does not always converge.
3. The method may converge to a local but not global
maximum.
4. The starting value is important since different starting
values can converge to different local maxima. If the
log likelihood is not concave, try different starting
values.
Another useful method for computing maximum
likelihood estimates is the EM algorithm (Chapter 6.6).
Good book on computation in statistics: Numerical
Methods of Statistics, John Monahan
Confidence interval
Theorem 6.2.2: Assume
1
, ,
n
X X K
are iid with pdf
0
( ; ) f x
for
0

such that the regularity conditions (R0)-(R5) are
satisfied. Then
( )
0
0
1

0,
( )
D
MLE
n N
I

_


,
Approximate
(1 )
confidence interval for :
/ 2
1

( )
MLE
MLE
z
nI

t
For the logistic distribution model,
( )
2
2 2
1
exp{ ( )}
( ) log ( ; ) 2
1 exp{ ( )}
n
i
i
i
X
I E f X E
X

1
1
1
1

+ 1 ]
]

Problem: It is hard to compute

( ) I
.
Solutions:
(1) Use the observed Fisher information rather than the
expected information.
Observed Fisher information =
2

2 1
log ( ; )
MLE
n
i
i
f X

Another approximate
(1 )
confidence interval for
besides that given by Theorem 6.2.2 is:
/ 2
2

2
1
1

log ( ; )
MLE
MLE
n
i
i
z
f X

(2) Parametric bootstrap: In the nonparametric bootstrap,

we approximated the true distribution F by the empirical
distribution

n
F . In the parametric bootstrap, we
approximate the true distribution F by the density

( ; )
MLE
f x . To obtain a bootstrap percentile confidence
interval:
1. Draw an iid sample
* *
1
, ,
n
X X K
from

( ; )
MLE
f x .
2. Compute ,
* * *
1

( , , )
b MLE n
X X K
3. Repeat steps 1 and 2 for b=1,...,M.
4. The bootstrap percentile confidence interval endpoints
for are the / 2 and 1 / 2 quantiles of
* *
1

( , , )
M
K .
The function rlogis(n,location=theta,scale=1) generates n
iid logistic random variables in R.
# Function that forms percentile bootstrap confidence
# interval for parameter theta of logistic distribution
percentcibootlogisticfunc=function(X,m,alpha){
# X is a vector containing the original sample
# m is the desired number of bootstrap replications
thetahat=mlelogisticfunc(X)$thetahat;
thetastar=rep(0,m); # stores bootstrap estimates of theta
n=length(X);
# Carry out m bootstrap resamples and estimate theta for
# each resample
for(i in 1:m){
Xstar=rlogis(n,location=thetahat,scale=1);
thetastar[i]=mlelogisticfunc(Xstar)$thetahat;
}
thetastarordered=sort(thetastar); # order the thetastars
cutoff=floor((alpha/2)*(m+1));
lower=thetastarordered[cutoff]; # lower CI endpoint
upper=thetastarordered[m+1-cutoff]; # upper CI endpoint
list(thetahat=thetahat,lower=lower,upper=upper);
}
Optimality (Efficiency) of the MLE:
We will now show that for large samples, the MLE does as
well as any other possible estimator in some sense.
We start by presenting a lower bound on the variance of an
unbiased estimator for an estimator in finite samples.
Theorem 6.2.1 (Rao-Cramer Lower Bound): Let
1
, ,
n
X X K
be iid with pdf
( ; ) f x
for . Assume that the
regularity conditions (R0)-(R4) hold. Let
1
( , , )
n
Y u X X K
be a statistic with mean
1
( ) [ ( , , )] ( )
n
E Y E u X X k

K
. Then
2
[ '( )]
( )
( )
k
Var Y
nI

.
Proof: We will follow the proof in the book.