STATS 200 (Stanford University, Summer 2015)

Lecture 11:

Frequentist Confidence Sets

Earlier in the course, we discussed methods to use data produce a single estimate of an
unknown parameter . An alternative approach is to use data to report a set of values of
that we believe contains the true value of . Specifically, if we have data X, we can specify a
rule (i.e., a function) that maps each possible data value x to some subset of the parameter
space C(x) . Then the set C(X) is a random set that serves as a sort of set-valued
estimator of . Frequentists and Bayesians use different terminology for such objects:
If C(X) is based on certain frequentist principles (to be defined in Section 11.1 below),
it is typically called a confidence set.
If C(X) is based on certain Bayesian principles (to be defined later in the course), it
is typically called a credible set.
This lecture deals with the notion of frequentist confidence sets. Bayesian credible sets will
be addressed later in the course.

11.1

Confidence Sets and Confidence Levels

For confidence sets to be a useful tool of statistical inference, we must quantify the notion
of a set of values of that we believe contains the true value of .
Definition of Confidence Level
A confidence level of a confidence set C(X) for a parameter is a number 0 such
that P [ C(X)] for all .
Confidence levels are typically expressed as a percentage, and we typically refer to a
confidence set for with confidence level using this percentage (e.g., a 95% confidence
set for when = 0.95).
Note that the random part of the event C(X) is C(X), not . For this reason, it is
good to think and speak in terms of whether or not the confidence set C(X) contains
the parameter (rather than whether or not the parameter falls in the confidence set).
Higher confidence levels correspond to a greater probability that the confidence set
contains the unknown parameter. However, the higher the confidence level is, the
larger (and hence less informative) the confidence set tends to be. As an extreme case,
note that C(X) = would be a 100% confidence set, but of course this confidence set
would be totally useless.
If P [ C(X)] = for all , then the confidence level and confidence set are sometimes
called exact. However, other authors use the word exact to refer to a confidence set that
is calculated without resorting to asymptotic approximations. To avoid confusion, we will
avoid using the word in either sense.

Lecture 11: Frequentist Confidence Sets

The Confidence Level

Notice that if is a confidence level of a particular confidence set, then any number smaller
than is also a confidence level of that set. For example, a 99% confidence interval is also
a 95% confidence interval. Recall that a similar issue arises in hypothesis testing, where we
use the term size to refer to the smallest number that is a level of the test. It would be
convenient if we had a similar term (e.g., confidence size) to refer to the largest number
that is a confidence level of a confidence set, but unfortunately there is no such standard
terminology. Thus, for simplicity, we will assume that whenever we refer to the confidence
level of a confidence set, we mean the largest number that is a confidence level of that set.
For example, if we refer to a 95% confidence interval, we will assume that 0.95 is the largest
number that is a confidence level of that interval.
Constructing Confidence Sets
In simple problems, it is sometimes clear how to construct confidence sets with a desired
confidence level.
Example 11.1.1: Let X1 , . . . , Xn iid N (, 2 ), where R and 2 > 0 are both unknown.
Let 0 < < 1, and suppose we wish to construct a confidence set for with a confidence
level of 1 . As usual, define
X=

1 n
Xi ,
n i=1

S2 =

1 n
2
(Xi X ) .
n 1 i=1

(11.1.1)

Then by Theorem 3.3.2 of Lecture 3,

X

tn1
S 2 /n
for all R and all 2 > 0. Now let n, /2 denote the number such that P (T > n, /2 ) = /2
for a tn1 random variable T . Then for all R and all 2 > 0,

X
1 = P,2 n, /2 <
< n, /2

S 2 /n

S
S2
= P,2 X n, /2
< < X + n, /2
n
n

Then the set

S2
S2

0 R X n, /2
< 0 < X + n, /2

n
n

is a confidence set for with confidence level 1 .

(11.1.2)

It is often the case that, as in Example 11.1.1, a confidence set takes the form of an interval.
Hence, it is common to refer to such a set as a confidence interval.

Lecture 11: Frequentist Confidence Sets

Note: For confidence intervals involving data with a continuous distribution, it usually
makes no difference whether the endpoints are included in the interval. In such cases,
the interval is sometimes written in the form (center) (half-width) if such a form
is convenient. For example, the interval in (11.1.2) might be written as

S2
X n, /2
.
n
Such notation should of course be avoided in settings where the inclusion or exclusion
of the endpoints is actually relevant.

11.2

Confidence Sets as Inverted Hypothesis Tests

As with other topics in the course, we would like to develop a general method by which we
can construct a confidence set with a desired confidence level for a wide variety of problems.
The following theorem provides such a method.
Theorem 11.2.1. For every 0 , let R0 be the rejection region of a hypothesis test of
H0 = 0 versus H1 0 with level . Then C(X) = {0 X R0 } is a confidence
set for with confidence level 1 .
Proof. For every , P [ C(X)] = P (X R ) = 1 P (X R ) 1 .
Thus, Theorem 11.2.1 essentially states that confidence sets are inverted hypothesis tests.
A confidence set with confidence level 1 is simply all of the values of the parameter that
would not be rejected as the null hypothesis value in a test with level .
Multiple Unknown Parameters
Theorem 11.2.1 still applies even if there are other unknown parameters besides . In such
cases, the null hypothesis H0 = 0 is composite, with the other parameters unconstrained.
(Of course, the presence of the additional unknown parameters may complicate the process
of finding a test with level .)
Example 11.2.2: Let X1 , . . . , Xn iid N (, 2 ), where R and 2 > 0 are both unknown.
Let 0 < < 1, and suppose we wish to construct a confidence set for with a confidence
level of 1 . To construct such a set, we can begin by finding a test of H0 = 0 versus
H1 0 with level . We saw in Example 9.0.2 of Lecture 9 that the likelihood ratio test
of these hypotheses is to reject H0 if and only if
X 0

n, /2 .
S 2 /n

(11.2.1)

Then a confidence set for with confidence level 1 is the set of all 0 R such that the
inequality in (11.2.1) does not hold, i.e., such that
X 0

< n, /2 .
S 2 /n

(11.2.2)

Lecture 11: Frequentist Confidence Sets

The inequality in (11.2.2) holds if and only if

S2
S2
X n, /2
< 0 < X + n, /2
.
n
n
Then the set

S2
S2

< 0 < X + n, /2
0 R X n, /2

n
n

is a confidence interval for with confidence level 1 . This confidence interval is the same
as the confidence interval derived in Example 11.1.1.

11.3

Asymptotic Confidence Intervals

If the sample size is large, asymptotic results can be used to approximate sizes of hypothesis
tests. Confidence sets based on such asymptotically motivated tests may be called asymptotic
confidence intervals. The stated confidence level of such sets is only approximate, with the
quality of the approximation improving as the sample size increases.
Wald Confidence Intervals
The simplest asymptotic confidence intervals are those based on Wald tests. Recall that one
version of the Wald test of H0 = 0 versus H1 0 rejects H0 if and only if

In (n ) n 0 z/2 ,
(11.3.1)
while the other version rejects H0 if and only if

Jn n 0 z/2 ,

(11.3.2)

where z/2 is the number such that P (Z z/2 ) = /2 for a N (0, 1) random variable Z. Then
the Wald test in (11.3.1) would fail to reject H0 = 0 if and only if 0 is in the set

z
z

/2
/2
< 0 < n +
,
0 n

(
)
(
)

n n
n n

(11.3.3)

while the Wald test in (11.3.2) would fail to reject H0 = 0 if and only if 0 is in the set

z/2
z/2

0 n
< 0 < n + .

Jn
Jn

(11.3.4)

The sets in (11.3.3) and (11.3.4) are called Wald confidence intervals, noting that these sets
are indeed guaranteed to be intervals.

Lecture 11: Frequentist Confidence Sets

Note: The construction of Wald confidence intervals now makes it clear why we did
not consider the test that rejects H0 if and only if

In (0 ) n 0 z/2
when we defined Wald tests. Since 0 appears twice in the test statistic above, we
cannot state a general formula for how to invert this test to obtain a confidence set.

We can now see that commonly employed nave confidence intervals of the form
(estimator) (normal quantile)(standard error of estimator)
are essentially just Wald confidence intervals.
Example 11.3.1: Let X1 , . . . , Xn iid Exp(), where > 0 is unknown, and let 0 < < 1.
In Example 10.1.1 of Lecture 10, we found that both versions of the Wald test of H0 = 0
versus H1 0 with approximate level are to reject H0 if and only if

n
n 0 z/2 .
(11.3.5)

2n
Then the Wald confidence interval for with approximate confidence level 1 is the set
of all 0 > 0 such that the inequality in (11.3.5) does not hold. Thus, the set

2n

< 0 < n + z/2

0 > 0 n z/2

n
n

n (1 + z/2 )}
n (1 z/2 ) < 0 <
= {0 > 0
n
n
z/2
z/2
1
1
= {0 > 0
(1 ) < 0 <
(1 + )}
n
n
Xn
Xn
is the Wald confidence interval for with approximate confidence level 1 .

It can be seen from their construction that Wald confidence intervals are always centered at
the MLE (unless the interval runs into the boundary of the parameter space).
Note: Some authors define the Wald confidence interval as simply the set of all 0
for which the Wald test statistic is less than the critical value, regardless of whether
such 0 values are in the parameter space. In other words, some authors allow Wald
confidence intervals to spill over outside the parameter space. For example, they would
take the Wald confidence interval in Example 11.3.1 to be
{0 R

z/2
z/2
1
1
(1 ) < 0 <
(1 + )},
n
n
Xn
Xn

even if the lower endpoint of this interval is negative. However, we will stick to our
definition, which requires Wald intervals (and confidence sets in general) to be subsets
of the parameter space.

Lecture 11: Frequentist Confidence Sets

Since Wald tests are not invariant to reparametrization, Wald confidence intervals are not
invariant to reparametrization either. Thus, Wald confidence intervals for and = g() may
not match in the logical way. For example, some value can be in the Wald confidence
interval for while the corresponding value g( ) is not in the Wald confidence interval
for = g(), or vice versa.
Example 11.3.2: If we redo Example 11.3.1 with = 1/ as the unknown parameter, then
we find that
z/2
z/2
{0 > 0 X n (1 ) < 0 < X n (1 + )}
n
n
is the Wald confidence interval for with approximate confidence level 1 . Clearly this
interval does not match the interval for , in the sense that neither intervals endpoints
are the multiplicative inverses of the other intervals endpoints.

Score Confidence Sets

An alternative approach is to base asymptotic confidence sets on score tests. Recall that a
score test of H0 = 0 rejects H0 if and only if

1
In (0 )

`Xn (0 ) z/2 .

Then the score test would fail to reject H0 = 0 if and only if 0 is in the set

`Xn (0 ) < z/2 .

0

In (0 )

The set in (11.3.6) is called a score confidence set.

(11.3.6)

Example 11.3.3: Let X1 , . . . , Xn iid Exp(), where > 0 is unknown, and let 0 < < 1.
In Example 10.2.1 of Lecture 10, we found that the score test of H0 = 0 versus H1 0
with level is to reject H0 if and only if

20
1
n( X n ) z/2 .
(11.3.7)
n
0
Then the score confidence set for with approximate confidence level 1 is the set of all
0 > 0 such that the inequality in (11.3.7) does not hold. Thus, the set

20
1

0 > 0
n( X n ) < z/2 = {0 > 0 n 1 0 X n < z/2 }

n
0

z/2
z/2
1
1
(1 ) < 0 <
(1 + )}
= {0 > 0
n
n
Xn
Xn
is the score confidence set for with approximate confidence level 1 . Thus, the score
confidence set is the same as the Wald confidence interval found in Example 11.3.1. Of
course, this makes sense, since we already saw in Example 10.1.1 and Example 10.2.1 that
the Wald and score tests were the same for this particular setup.

Lecture 11: Frequentist Confidence Sets

Solving for 0 to obtain a closed-form solution for a score confidence set is not always possible.
Often we must resort to approximations or numerically computed solutions. Even when a
closed-form solution can be obtained, its form may not be particularly simple.
Example 11.3.4: Let X1 , . . . , Xn iid Poisson(), where > 0 is unknown. Let 0 < < 1.
The score function is
`Xn () =

n
Xn
n
1 n
[ Xi log n log(Xi !)] = Xi n = n(
1),
i=1
i=1

i=1

while the Fisher information is

In () = E [`Xn ()] =

n
1
n
E
(
Xi ) = .

i=1

Then the score test of H0 = 0 versus H1 0 with approximate level is to reject H0

if and only if

0
Xn
n(
1) z/2 .
(11.3.8)
n
0
Then the score confidence set for with approximate confidence level 1 is the set of all
0 > 0 such that the inequality in (11.3.8) does not hold. Thus, the set

0
Xn

n(
1) < z/2
0 > 0

n
0

Xn
n 0 < z/2 }
= {0 > 0
0
0 2
2
z }
= {0 > 0 ( X n ) 20 X n + 20 <
n /2

2
2

z/2
z/2

Xn +
= 0 > 0 X n +

( Xn )

2n
2n

2
2

z/2
z/2

(
)
< 0 < X n +
+
Xn +
Xn

2n
2n

2
2
2
2

z/2
z/2

X n z/2
n z/2
z/2
+ 2 < 0 < X n +
+ z/2
+ 2
= 0 > 0 X n +

2n
n
4n
2n
n
4n

is the score confidence set for with approximate confidence level 1 . Note that the Wald
confidence interval for with approximate confidence level 1 is

Xn
Xn

0 > 0 X n z/2
< 0 < X n + z/2
.

n
n

Thus, the score confidence interval is slightly wider than the Wald confidence interval, and
its center is shifted slightly toward larger parameter values.

Lecture 11: Frequentist Confidence Sets

Since score tests are invariant to reparametrization, score confidence sets are invariant to
reparametrization as well. Thus, Wald confidence intervals for and = g() will match
in the logical way. More precisely, we mean that a value is in the score confidence set
for , if and only if the corresponding value g( ) is in the score confidence set for = g().
Example 11.3.5: If we redo example 11.3.3 with = 1/ as the unknown parameter, then
we find that
1
1

z/2
z/2

0 > 0 X n (1 + ) < 0 < X n (1 )

n
n

is the score confidence interval for with approximate confidence level 1 . This interval
matches the interval for in the sense that each intervals endpoints are the multiplicative
inverses of the other intervals endpoints.

Likelihood Ratio Confidence Sets

Confidence sets can also be constructed by inverting a likelihood ratio test. The test can be
constructed using the exact distribution of the likelihood ratio statistic (as in Example 11.2.2)
or using an asymptotic approximation based on Theorem 10.3.1 of Lecture 10. However, the
form of the likelihood ratio statistic is often more complicated than that of the Wald or score
test statistics, so such likelihood ratio confidence sets often must be computed numerically.
Example 11.3.6: Let X1 , . . . , Xn iid Exp(), where > 0 is unknown, and let 0 < < 1.
In Example 10.3.2 of Lecture 10, we found that the likelihood ratio test of H0 = 0 versus
H1 0 with approximate level is to reject H0 if and only if
n

2 log{[0 X n exp(1 0 X n )] } w ,

(11.3.9)

2
where w = z/2
is the number such that P (W w ) = for a 21 random variable W . Then
the likelihood ratio confidence set for with approximate confidence level 1 is the set of
all 0 > 0 such that the inequality in (11.3.9) does not hold. Thus, the set
n

2
}
{0 > 0 2 log{[0 X n exp(1 0 X n )] } < z/2

2
z/2

= 0 > 0 0 exp(0 X n ) <

exp 1

2n

Xn

is the likelihood ratio confidence set for with approximate confidence level 1 . We
cannot simplify this set any further, but we could find the set numerically for any particular
observed data X n = xn .

Philosophical Issues
When a confidence set is constructing by inverting a hypothesis test, it can also be affected
by the various philosophical issues with hypothesis testing that were discussed in Section 8.5
of Lecture 8. As before, these issues are merely things that should be kept in mind, not
necessarily an argument against the use of confidence sets.