Tnlno

Eollor.t

AppliedLinear
Regression
Models

'F

John Neter
Universitl, of Georgia

MichaelH. Kutner
The Cleveland Clinic Foundation

ChristopherJ. Nachtsheim
Unit ersity of Minneso ta

WilliamWasserman
Syracuse Llniversitt'

IBWIN
Chicago .
L c .n d o n .

Bo g atd .
Ma d ri d .

concerned about Our Environment

al
,nW|t
yet
In recognitionof the {act that our company is a large end-userof ftagile
D- -l|
resources,we at IRWINcan assureyou that everyetfort is made to
replenis-hable
Lt
m e e to r e xc eedE nvi ronmental P rotecti onA gency(E P A )recommendati onsandrequ i re.
mentsfor a'greener"workplace.
policies' both companywroe
To preservethese natural assets,a number ol environmental
recycled paper in
and departmenlspeciiic, have been rmplemented.From the use of 50o/o
o u r te xtb ookstothepri nti ngol promoti ona|materi a|sw i threcyc|edstockandsoyi nkstoour
and replacingenviron'
office paper recycllngprogram,we are committedto reducingwaste
mentallyunsafeproductswith safer alternatives'

@ The McGraw-Hill Companies,Inc., 1983.1989,and 1996

No part of this publicationmay be reproduced,storedin a letrieval system,oI transAll rights reserved.
or otherwise'
mitted, in any form or by any means,electronic,mechanical,photocopying,recording'
without the prior written permissionof the publisher'
TrademarkAcknowledgments:
IBM and IBM pc are registefedtrademarksoflnternational BusinessMachinescorporation.
Disclaimer
or implied, regardingthe enclosedcomRichard D. Irwin, Inc. makesno waranties, eitherexpressed
particularpurpose.The exclusionof imfor
any
its
fitness
or
merchantability
package,
its
puter software
not apply to you' This warranty promay
plied *urrantiei is not permitted by somestates.The exclusion
you
may havewhich may vary from
that
rights
other
be
may
There
rights.
legal
you
with
specific
uid.,
stateto state.
Irwin Book Team
Publisher:Tom Casson
Seniorsponsoringeditor: Richard T. Hercher,Jr'
editor:Gail Korosa
Seniordevelopmental
Seniormarketingmanager:Brian Kibby
Projecteditor: LYnneBasler
Productionsupervisor:Laurie Kersch
Designer:MatthewBaldwin
Coverdesigner:Stuart Paterson
AssistantManager,Graphics:CharleneR. Perez
Compositor/ArtStudio:GraphicComposition,Inc'
Typeface:10/12TimesRoman
Printer:R. R. Donnelley& SonsCompany

Library of Congress Cataloging-in-Publication Data

Applied linear regressionmodels / John Neter ' ' ' [et al']'

3rd ed'

p .c m
lncludes index.
ISBN 0-256-08601-X
1. Regressionanalysis.I. Neter, John
QA2 7 8 .2.A 651996
5 1 9 .5 ' 36-dc20
Prfitted in the United States of America
3 4 5 6 7 890D O210987

95-37447

D,
Cnlpren

to
MatrixApproach
Linear
Simple
AnalYsis
Regression
The matrix approach
Matrix algebrais widely usedfor mathematicaland statisticalanalysis'
extensive systemf
permits
it
since
is practicllly a necessityin multiple regressionanalysis,
upon efficiently'
operated
and
compactly
ofequationi ancllarge anays ofdata to be denoted
(A
fuller treatmenl
algebra.
matrix
to
In this chapter,we tirstiake up a brief introduction
5'1') Then wc
Reference
as
such
texts
of matrix algebra may be found in specialized
chapten'
previous
in
discussed
model
apply matrix-methodsto the simple linear regression
application
the
regression,
linear
simple
for
ettnougn matrix algebrais not really required
regression'which
of matrix methoclsto this casewill provide a useful transition to multiple
will be taken uP in Parts II and III.
parts of thit
Readerslamiliar with matrix algebra may wish to scan the introductory
regressioR
in
methods
matrix
of
use
the
with
chapter and l.cus upon the later parti dealing

analysis.

5.1 Matrices
Detinition of Matrix
columns. An exampled
A matrix is a rectangulararray of elementsarrangedin rows and
i:
a matrix is:
Column Column
1)

I l- 16,000 nf
Row
Row2 | 33,000
Row3 L 21,000

176

47
|
35 J

Secriotr5.1 Matrices

177

Other examplesof matricesare:

ol

[r

j 1 2 l6 ' l

l+

l_srol

l: rs e 8 l

Thesetwo matriceshave dimensionsof 2 x 2 and 2 x 4, respectively.Norc thut in giving

the dimensionof a matrix, we always specifythe numberof rows first urrtlthcrrlhc number
of columns.
As in ordinary algebra,we may use symbolsto identify the elenrcnlsol :r rrurlrix:

" /:l
a,
a2,
|

i :l l
i :2

tacl ,

em:.
rtl),
l eiri

atz
azz

j :3
on1
ozz l

Note that the first subscriptidentifiesthe row numberand the secondthc r.rlurrrrrnurnber.
We shall usethe generalnotationaii for the elementin the i th row and thc rlr t.olrrrln.In
i
o u r a b o v ee x a mp l ei, : 1,2 and7 - 1,2,3.
A matrixmay be denotedby a symbolsuchasA, X, or Z.The symbolis irr lrolllircc to
identify that it refersto a matrix. Thus, we might definefor the abovemarrrr:

lers.
.tior,
hich
rhi:
siol

j :2

att otz

A:l

'''
Lazt az2 a:z )

Referenceto the matrix A then implies referenceto the 2 x 3 arrayjust givcrr.

Another notationfor the matrix A just given is:
A:

i :1,2:

[ai j l

j :1,2,3

This notationavoidsthe needfor writing out all elementsof the matrixby stllirr::,rrlv thc
generalelement.It canonly be used,of course,whentheelementsof a matrixrrr.r,
svrrrlrols.
To summarize,
a matrix with r rows and c columnswill be represented
citlrer.irrlirll:
le o

(5.1)

A_

att

art

cl ,;

a-tt

ar

at;

a; r

o;t

a;;

A,t

A,t

Q.;

an(i

umr
000)
's b1'
I the

or in abbreviatedform;
A :[a;1)

i :1,...,r;

or simply by a boldfacesymbol, suchas A.

j -1,..

.C

178

Chapter5

Matrix Approachto SimpleLinear RegressionAnalysis

Comments
L Do not think of a matrix as a number.It is a set of elementsarrangedin an array.Only
when the matrix has dimensionI x I is there a singlenumberin a matrix, in which caseone
can think of it interchangeablyas either a matrix or a number.
2. The following is not a matrix:

ls
l
l"l
ll0
I s

I
ls l
16_]

sincethe numbersarenot arrangedin columnsand rows.

SquareMatrix
A matrix is said to be squareif the number of rows equals the number of columns. Two
examplesare:

l+ e7ll
L3

l-1
on arz ort
I
I
ol azz or,
I
I
Lo:, az2 ax J

Vector
A matrix containingonly one column is called acolumnvector or simply avector.Two
examplesarc;

^ :fi
L'ol l .:[r
l" r i ]
Lcsl

The vector A is a 3 x I matrix, and the vector C is a 5 x I matrix.

A nratrix containingonly one row is called a row vector.Two examplesare:

B' : [15 25 50]

F' : [/i

fr]

We use the primc syrnbol for row vectors for reasonsto be seenshortly. Note that the row
v e c to rB /i sa I x 3 nratri xandtherow vectorF/i sa1x 2matri x.
A singlesubscriptsufficesto identify the elementsof a vector.

Transpose
The transposeof a matrix A is another matrix, denoted by A', that is obtained by interchangingcorrespondingcolumnsand rows of the matrix A.
For example,if:

,i,:li'i]

Sulirn 5.1 Matices

l7g

then the transposeA/ is:

7 3l

A' :l' -

2x3

l0

L5

4)

Note that the first column of A is

the-firstrow_of Ar, and similarly
the sec.nclc.lunrn of A
is the-secoJrdiow
of A'..corr.espo"ai"gly,it"irst
row of A has becomerhe rirsrcorumn
of A/' and to onior. that theiime"r;;;
;f ;, indicatedunder the symbor
A. bcconres
reversedfor the dimensionof A,.
As another example, consider:

,t, : f ,J] ,g,:ro 7 lol

Thus, the transposeof a column vector
is a row vector, and vice versa.
This is the reason

iaeniriv
arowvector,
since
ir maybethought
oras

;:1.ffi::3:?:'Jft'*i":,:lf:to
In general,we have:
T

I o'
'X":l^'
lu' l

ot,f
;l:
or,

J|

'z
Row
Column
index index

(5.3)

A':
cXr

l-rort
I

o rt

l ::l

tl

Lor,

ffiLf;"i:ment

or,

i:1,".'r; j:r,...,c

1'il
"\

tlit)

i:1,...,c:

,v\
Row
index

i=1,...,r

Column
index

in thei th rowandthe7thcolumnin A

is foundin the7throw andirh

Equality of Matrices
Two matrices A and B are said to
be equal if they have the same
dimension and if all

:[Tii,","*T:;i;f;::;n5:i';:"*rserv,irtwomatricesareequar,rheircorresponding

th e n A:B i mp l i e s :

:fi]
,r,:li,l
,x,
at:4

a 2 =7

a3:3

Similarly, if:

fo,, onl
orrl
.!.:l o ,
Lo,

ot,_J

n:f

3x2

f n 21
t+ tl
L t:

9l

180

Chapter5

Matix Approachto SimpleLinearRegression

Analysis
th e n A :Bi mP l i es:
att:17

at2:2

azt:

14 azz: 5

ay:

13 azz:9

RegressionExamples
ln regressionanalysis,one basicmatrix is the vectorY, consistingof then observationson
lhe responsevariable:

Ir, I

Y :1:1 'l

(s.4 )

nxl

lY^)
Note that thc transposeY' is the row vector:

Yl

: [Ir

Yz

Yn]

lXn

Another basicmatrix in regressionanalysisis the X matrix, which is definedas follows

for simple linear regressionanalysis:

(5.6.1

I":l:
nx2

[l ::1
:l
l:

L t x' )

The nratrixX consistsof a column of ls and a column containingthe n observationson the

predictorvariatrlcX. Note that the transposeof X is:

(57,

x, =l;, ;, . ":]

5.2 MatrixAdditionand Subtraction

Adding or subtractinglwo matricesrequiresthat they have the samedimension.The sum,
or difference,of two matricesis anothermatrix whoseelementseachconsistof the sum.or
Jiff'erence,
of the correspondingelementsof the two matrices.Suppose:

:1 11
A :l 1l
B
3
x
2
6J
4

3x2

[:

l:

Section5.2 M ut ri.rAt lJ i t i r trtund Subtraction

181

then:

4 + 2 1 lz 6l
5+3 | :14 8 |
L3+3 6+4) L6 r0l
It+r

A + B : lz + 2
3x2
Similarly:

- t2 4 - 2 1 [ o 2 1
: 1I2t _
5 _ 3 1 :lo 2l
L 3 - 3 6 - 4 1 L0 2)

A_B
3x2
ns on

In general,if:

: [ai i l

r x.c

B :l btj l

l :1,

;-

rX c

then:

(s.8)
1I"u

: lail * bill and

^,;j

: [a;i - b;i)

Formula (5.8) generalizesin an obviousway to additionand subtractionof more than two

matrices.Note also that A + B : B * A, as in ordinary algebra.
rllows

flegressionExample
The regressionmodel:
Y i :E { Y i I* ei

f;*'Y'i""n

on thc

i :1,...,n

comPactrY
in matrix notation. First, let us define the vector of

/5 q l

E{Y}
nX l

the mean

I ur r , t l
I twrlI

l,,r,l

and the vector of the error teffns:

e sunl
,um,()l

Y in (5.4), we can write the regression

r82

Chapter5

Matrix Approachto SimpleLinear RegressionAnalysis

because:

tll:-faall
.-f::l:|;ulrll
*'.1

L"lL'''.i
L'll',,,]

the
Thus, the observations vector Y equals the sum of two vectors, a vector containing
expectedvalues and another containing the error terms'

5.3 MatrixMultiPlication
Multiplication of a Matrix by a Scalar
a number. In multiplication of n '.
an ordinary number or a symbol reple!941!1ng
L-scalqt_i_s
example.I
matrix by a sialar, every element of the matrix is multiplied by the scalar.For
supposethe matrix A is given bY:

o:13
i]
Then 4A, where 4 is the scalar,equals:

oo:ol?i] : [,: 212)sl

Similarly,l.A equals:

7rl

^ o : ^ []3] : [ 3 ]3 r l
whereI denotesa scalar.
the '
If every element of a matrix has a common factor, this factor can be taken outside
example:
For
a
scalar.
matrix and treated as

I s 2 7 1: ' "ls
f:

Lt r r s . l

el

e. l

S i rn i l a rl y:

T5

21

li il r ls
s l: t L3
li
L,i. ).1

'11

;l

we have:
ln general,if A = loil and.1.is a scalar,

(5.rr)

trA:Al-lLaij

Section5.3 Matrix Multiplication

1g3

tllultiplication of a Matrix by a Matrix

Multiplication of a matrix by a matrix may appearsomewhat
complicatedat flrst, but a little
practice will make it a routine operation.
Considerthe two matrices:

i] ,y,:u
,1,:11
f]
The product AB will be a 2 x 2 matrix whose elements
are obtained by finding the cross
products of rows of A with columns of B and summing
the crossp.oauar. For instance,to
find the element in the first row and the first column
of the product AB, we work with the
first row of A and the first column of B, as follows:
Al

*"* Ifzll
6l
l-f-4-l
no'* zf t ) Lb"j8 J
^
Col. I Col. 2

AB
nowr

[33

L]

Col. I

We take the crossproducts and sum:

2 (4 )+ 5 (5 ): 3 3
The number 33 is the element in the first row and first column
of the matrix AB.
To find the element in the first row and secondcolumn
of AB, we work with the first
row of A and the secondcolumn of B:

*o*'l+ll
[D-3-l l-+
Row2[4 t )

ls

i6-ll

LEll

Col. 1 Col.2

AB
Rowr [33 s2l

LJ

C ol . 1 C ol .2

Thesumof thecrossproductsis:
2(6)+5(8):52
Continuingthisprocess,we find theproductAB to be:

i][l:]: i;i#l
f:,:l'^
Let us consider anotherexample:

3i] ,*,:f;]
,x,:l;
ri:f;33l[;]:[r]

Chapter5

Matrix Approachto SimpleLinear RegressionAnalysis

When obtainingthe productAB, we saythat A is postmultipliedby B or B is premultiplied by A. The reasonfor this preciseterminology is that multiplication rules for ordinary
algebrado not apply to matrix algebra.In ordinary algebra,xy : yx.In matrix algebra,
AB + BA usually.In fact, even though the product AB may be defined,the product BA
may not be definedat all.
In general,the productAB is only definedwhen the numberof columnsin A equalsthe
numberof rows in I| so that therewill be correspondingtermsin the crossproducts.Thus,
in our previoustwo cxamples,we had:
Equal

E qual

A,/\B
I x:

\,v

7x2

:AB
2x2

A/\B
2x3

3xl

AB
2xl

\,v

I )i nrcrrsi on
ol protl uct

Dimension
of product

row sinAandt he
N o te th a rrl retl i rrrcnsi orro1' theproductA B i sgi venbythenumberof
ol coltrrrrns
in ll. Note alsothat in the secondcasethe productBA would not be
rrurrrbcl
of columnsin B is not equalto the numberof rows in A:
tlclirrctlsirrcr'tlrc rrrrrrrbcr
Unequal

B z/\
3xl

A
2x3

llcrc is rrrrollrel
cxirrnpleof matrixmultiplication:

;ttt - ['rrr
L,1: I

ar2 ottl
A ZZ

A ZZ)

f utt btrl
Ib^ bzzI

Lbt, bt, )
,r,,b,, * arrb^ * arrb, arrbr,* arrbr,+ orrbrrl
- |
* arrb22
* ozzbn)
f ,,,,b,, l azzbzt * arrb, arrb,,2

Irrg c rrc r' :rl .iAl h:rsdi mensi onrxcandB hasdi mensi oncxs,theproduct ABisa
r. x .r whoseelementin the i th row and lth columnis:
matrixol tlirrrcrrsion
c

\L

aitbtrj

so lh ill:

(5 .11 )

A tl

: [i o,obo,l
i'l
Lk=r

'l'hus. in thc lirlcgoinu example,the elementin the first row and secondcolumn of the
grlorluct
Ali is:
3

,L.ortbm:

atbt2 * arrbr, * arrby

as indeedwe found by taking the crossproductsof the elementsin the first row of A and
secondcolumn of B and summing.

Section 5.4

Special Types of Matrices

185

Additional Examples
I q 2 1 1a , 1

t.

| + o ,+ 2 a . 1

+8,;)
Is sjL,;J:Lr,i
fzl
:;:s1
3
s
l
12
l1 l: t ' + 3 2 + s 2 1
L)l

2.

Here, the product is a 1 x I matrix, which is equivalentto a scalar'.l'lrLrs.lhc rnatrix

producthereequalsthe number38.

r,,i,l
tiiiltnr:t1,i

-).

RegressionExamples. A product frequently neededis Y'Y, where Y is tlrt' vectrlr of

observationson the responsevariableas definedin (5.4):

It, I

t5 .l 3 r

y'y:[X r
rxr

lv, l
y,,1
| .' l:lyi
|r ..l: I
L t,,-l

yz
-

+r l+ ...*1,]l

I) ],1

Note thatY'Y is a I x I matrix,or a scalar.We thushavea compactwly ()l l r irirrr :r srrrrr

of squaredterms:Y/Y : rV?.
We alsowill needX'X, which is a 2 x 2 matrix.whereX is definedin tr.r' r

Ir x,l

(5.14t

x,-l:ll [,, r\,

[r I "' rllr
X'X:l
ll
r\,
i*z Lx' x2
:
",,11, x n )I Ltx,
Lr

andX/Y, which is a2 x l matrix:

(s.rs)

I v ,I

I "'tl
2xt
",":['
LX'

X2

Xr)

| ':' l : [L ,\,),1
' , ,I

| I
LY,,J

5.4 SpecialTypesof Matrices

Certain specialtypes of matricesarise regularly in reurcssionanalysis.We considerthe
mostimoortantof these.

186

Chapter5

Matrix Approachto SimpleLinear RegressionAnalysis

Symmetric Matrix
If A : A', A is said to be symmetric.Thus, A below is symmetric:

[4r 4 6 1
:
1
2 5l
A

3x3

A,
3x3

L6 5 3l

:[li\]

A symmetric matrix necessarilyis square.Symmetric matricesarise typically in regression

analysiswhen we premultiply a matrix, say,X, by its transpose,X'. The resultingmatrix,
X/X, is symmetric,as can readily be seenfrom (5.14).

Diagonal Matrix
elementsare all zeros,suchas:
matrix is a square
matrix whoseoff-diagonal
A diasonal
v
..- :
t- - ._ _ ,

,l
,i,:[i+l] .*.:
,
[l l]
We will often not show all zeros for a diagonal matrix, presentingit in the tbrm:
-'l
At^l
'| l |

a1
0l+x+

3x3

ln:

l0

azl
-J

Two importanttypesof diagonalmatricesarethe identity matrix and the scalarmatrix.

Identity Matrix. The identity matrix or unit matrix is denotedby I. It is a diagonalmatrix
whoseelemenlson the main diagonalareall ls. Premultiplyingorpostmultiplying anyr x r
matrix A by thc r x r identity matrix I leavesA unchanged.For example:

atz onf
o oltt [a,,
[t
I
'
lA:10I
0llo,
a22 or..l:
L0 0

ll

[43'

a3z an J

f o , d t z o r:I
ou azz or,
I
I
Loy

a32 an )

we have:
Si rn i l a rl y.

AI:

f o,, atz a,,l I t o o- l f o,, ar z o,:I

azz onl
lor r azz atr ll0 I 0l:lot'
Lo'

a32 a33l [0

ll

Lo:'

a3z ai .3J

Note that the identity matrix I thereforecorrespondsto the number I in ordinary algebra,
x.
s i n c ew e h a v etherethat 1 . x : x .l :
In general,we have for any r x r matrix A:

(5.16)

AI:IA:A

Section5.4 SpecialTypes
of Matices

187

Thus, the identity matrix can be insertedor droppedfrom a matrix expressionwheneverit

is convenientto do so.
Scalar Matrix. A scalarmatrix is a diagonalmatrix whosemain-diagonalelementsare
the same.Two examplesof scalarmatricesare:

| \ 0 0l

l?
9l l; ^ 0r
Lo 2)
[o o i_]
A scalarmatrix can be expressedas ,r"I,where), is the scalar.For irrslrrrrcc:

lz ol_,Ir ol : zr
,):'lo iJ

fo

[;, o o l
[r o o l
r
o
l:^
lo
I o l:1 1
lo
L0 0 i._j

L0 0 r _l

Mu l ti p l y i n g a n rx rma tri xA bytherxrscal armatri x),Ii sequi val crrr r' rrrrrl ri ;rl yi n, u
{ by the scalar),.

Vectcir and Matrix with All Elements lJnity

A columnvectorwith all elements
1 will bedenotedby 1:

['l

(5.17)

tLIJll

rX l

and a squarematrix with all elementsI will be denotedby J:

Ir

(5.18)

ll

f :1,
'xr I r

r_l

For instance,we have:

lr l
r :ltl

3x r

L r_ j

fr I r l
I rl
I :lr

3x.i

[r

t]

Note that for an n x I vector I we obtain:

l'l
lxl

: [l

|-'l

"L;J
:

[r1l:

11

| 8tt

('lruptcr 5

Matrix Approach to Simple Linear Regression Analysis

and:

ltl

=l, lu

[r

rl

Li

il:

rr: li

L'J

J
nX n

Zero Vector
A zero vector is a vector containing only zeros. The zero column vector will be denoted

bvo:

'*,:f:]

(5.1e)

For example,we have:

0
3x1

: [3]
L0_l

5.5 LinearDependenceand Rankof Matrix

Linear Dependence
Considerthe followins marrix:

:l

[r 2 s
d , : lz 2 t o
L3 4 rs rl

Let us think now of the columnsof this matrix as vectors.Thus, we view A as being made
up of four column vectors. It happenshere that the columns are interrelated in a special
manner.Note that the third column vector is a multiple of the first column vector:

[Lr,sl; j :L,3Il j ]
We say that the columnsof A tre linearly dependent.They containredundantinformation,
so to speak,sinceone column can be obtainedas a linear combinationof the others.
we definethe set of c column vectorscI,..., c" in an r x c matrix to be linearly
dependentif one vectorcan be expressedas a linear combinationof the others.If no vector
in the set can be so expressed,we definethe set of vectorsto be linearly independent.A
rnoregeneral,though equivalent,definitionis:
(5.20) When c scalarsi,1, . . . , i.", not all zero,can be found suchthat:

i.1cr+ L2c2+...+ lccc - 0

Section
5.6 Inverseof a Matrix

189

where 0 denotes the zero column vector, the c column vectors are linearly dependent.lf the only set of scalarsfor which the equality holds is
lr : 0, ...,1. : 0, the setof c columnvectorsis linearlyindependent.
T o i l l u s tra tefo ro u re xampl e,l t = 5, l z = 0. l : : -1,1+ :0l eads to:

'[i].'[1]
'Ir].'[i]:[l]
Hence,the column vectorsarelinearly dependent.Note that someof the,l.; equalzerohere.
For linear dependence,
it is only requiredthat not all .1,,be zero.

Rank of Matrix
The rank of a matrix is definedto be the maximum numberof linearly independentcolumns
in the matrix. We know that the rank of A in our earlierexamplecannotbe 4, sincethe four
columnsare linearly dependent.we can, however,find threecolumns (1, 2, and 4) which
arelinearlyindependent.
Thereareno scalars.].r.1,, 14 suchthat),1cl + ),rcr+ ).oco : 0
other than Lt : Lz - L4 : 0. Thus, the rank of A in our exampleis 3.
The rank of a matrix is uniqueand can equivalentlybe definedas the maximum number
of linearly independentrows. It follows that the rank of an r x c matrix cannot exceed
min(r, c), the minimum of the two valuesr and c.
When a matrix is the productof two matrices,its rank cannotexceedthe smallerof the
two ranksfor the matricesbeing multiplied. Thus, if C : AB, the rank of C cannotexceed
min(rank A, rank B).

5.6 Inverseof a Matrix

In ordinary algebra,the inverseof a numberis its reciprocal.Thus, the inverseof 6 is l . A
o
numbermultipliedby its inversealwaysequalsl:
I

.l
x.-

__.h_l

6
1

6
=-f,.X

:x

._{:

In ma tri xa l g e b rath
, e i nverseof a matri xA i s i urothclrrrrtri x.tl crrotcd
bv A -1, such
that:

(s.21)

r\A ' -.

whereI is the identitynrutrix.'l'hus.rrrrrrirr.

thc idcrrtitynratrixI playsthe samerole as the
numberI in ordinaryalgebra.An invelscof a matrix is definedonly for squarematrices.
Even so, many squarematricesdo not have an inverse.If a squarematrix does have an
inverse,the inverseis unique.

190

Chapter5

Matrix Approachto SimpleLinear RegressionAnalysis

Examples
l. The inverseof the matrix:

: f' t 41il
,!., Ll
IS:

: i-.l
A-,
L
2x2
srllcc:

A _1A :
o I:

A A -I _
l . ' l ' l rci rrv c rs ol
r' ' thematri x:

4l
I r - . 24112
1
1
3
rl
1.3

.41I

: [;?]:'

l: il11-11:
[;?]:'
o ol

[r
A :10 4 0l

3x3

A-r
JAJ

since:

A',A

L0 0 2)

l i ;;l

g oli, o
[+
=
:,
ro i o ro .3 'l:i ; ? 3 -l
o 2J [o o rJ
Lo o i_lLo

Nolc thlt tltt' ittvcrsc o1'a diagonalmatrix is a diagonalmatrix consistingsimply of the

rcciprocirls
ol tlrc clcntentson the diagonal.

Finding the lnverse

lJp to this Poirrt.rhc inverseof a matrix A has been given, and we have only checkedto
nrakcsurc it is the invcrseby seeingwhether or not A-1A _: I. But how doesone find the
inversc,anclwlrcn tklcsi1exist?
An inversetll a squarer x r matrix existsif the rank of the matrix is i.. Such a matrix
is said to be ntttrsittl4ulur
<>rof .full rank. An r x r matrix with rank lessthan r is saidto be
:;ingulctror not ol'.litll ntnk, anddoesnot havean inverse.The inverseof an r x r matrix of
lull rankalsohasrankr'
Finding the inverseof a matrix can often requirea largeamountof computing.We shall
take the approachin this book that the inverseof a2 x 2 matrix and a 3 x 3 matrix can be
calculatedby hand.For any largermatrix,oneordinarilyusesa computeror a programmable

Section 5.6

Inverse of a Matrix

191

calculatorto find the inverse,unlessthe matrix is of a specialform suchas a diagonalmatrix.

It can be shown that the inverses for 2 x 2 and 3 x 3 matrices are as follows:
1 . If:

* , : lx3 l
then:

(s.22)

4-r
2x2

o''1, l

: lo
Lc

f d

-bf

LD

Dl

:I D D I
a I
l-c

-l

dJ

where:
(5 .2 2 a )

D :ad-bc

D is called the determinant of the matrix A. If A were singular,its determinantwould equal

zero and no inverseof A would exist.

2. rf:

B :l i 2;l

3x3

Ls

k)

then:

r5 ? 1 r

B- 1
3x 3

f a b , f - ' : lDle
ld n f |
Ls h k)

B cl
E rl

LG H

K)

where:

7:(ek-fh)lz
p:-(dk-fil/Z
(5.23a)
6:(dh-edlZ

B:-(bk-ch)lz
6:(ak-cC)lZ
g:-(ah-b7)lZ

C:(bf-ce)lZ
F:-(af-cd)lZ
K-(ae-bd)/Z

and;

(s.23b)

Z : a(ek- fh) - b(dk -fil

Z is called the determinantof the matrix B.

Let us use (5.2D to find the inverseof:

A:ll

f't

r,1

LJ

TI

, I

We have:
a:2

b:4

c:3

d:1

* c(dh - eg)

192

Chapter5

Matrix Approachto SimpleLinear RegressionAnalysis

D : ad - bc :2(1) - 4(3): -19

Hence:

I
-4
ftl
:f
=
T
A -,= l
I

2l

l_ 3

L -1 0

-l0 l

-.r

.41

. 3 -. 2 )

as was given in an earlierexample.

When an inverse A-l has been obtained by hand calculationsor from a computer

:"tri;:;ffi
::
ilH,lf;1'ffi
:"iil,Trilr;l,ffr"h
il#tti;tfs!:iff
RegressionExample
Ihe principal inversematrix encounteredin regressionanalysisis the inverseof the matrix
X ' X i n (-5l.4 l :

f n tX,l
:1"",
txi-l
I'T
Using rule (5.22),we have:

a:n

b:LXi

,:LXt

d:LX?

so that:

D:,,fxi - (I",) (f r,): "lt rf -

rr v.i2l
\ut'i)
l:nl,1x,

nl

4.

Henc c :

(-5.24
)

> x?
- EXi I
|
,>d-W
,
|
I
(X ' X ; -' : ; ^>A:-ZP
I
2x2

-X X i

t,>,,x-W ;t1;|:VF )
Si n c eX X ; :

:> X ?

ttX anclX (X ; - I)'

- n* 2,w ecan si mpl i fy(5.24):

r'
-t
I
rTP
t(X,
t(x=r
_,
_
I
I
--.
( X' X) - ':
fLr

(5.24a)

2x2

l -X 1l

L ;A;;7

;6;vP )

-*)2

Section 5.6

Inverse of a Matrix

Usesof lnverseMatrix
In ordinary algebra,we solve an equation of the type:
5Y :20
by multiplying both sidesof the equation by the inverse of 5, namely:

lrsrr: lrrur
f)

and we obtain the solution:

: !p0,1:4

,
-5

In matrix algebra,if we have an equation:

AY:

we corespondinglypremultiply both sidesby A-l , assumingA has an inverse:

A -IA Y :

A -rC

SinceA-lAY : IY : Y, we obtain the solution:

Y:

A -l C

To illustratethis use, supposewe have two simultaneousequations:

2y, I 4yr:29
31tfY ' :l Q
which can be written as follows in matrix notation:

lz +l
fir I : Irol
ll[ r : l
L3

L lo l

The solutionof theseequationsthen is:

[ v , ' l - [ z 4 . 1 -[r2 o l

L ,,l :L : rl L ' o l
Earlier we found the required inverse, so we obtain:

[r,l - l.t
Hence,yt :2

4.1
[20'l: [r.l

and yr - 4 satisfythesetwo equations'

193

194

Chapter5

Matrix Approachto SimpleLinear RegressionAnalysis

5.7 Some BasicTheoremsfor Matrices

We list here,without proof, somebasictheoremsfor matriceswhich we will utilize in later
work.
( 5 ?5)

A+B:B*A

(s.26)

(A + B )f C: A + (B + C)

(s.27)

(AB)C: A(BC)

(s.28)

c (A + B ): CA + Cg

(s.2e)

). (A + B ): ), A + i" B
(A,),: A

(s.30)
(s.31)

(A + B )/: A ' + B '

(s.32)

(A B )/ : B ' A /

( 5 3?\

(A B C), : C, B , A ,

r 5 34)

(A B )-' - B -rA -r

( 5 ?5\

(A B C )-' - C -tB -tA -r

( 5 ?6\

(s.37)

- O

(A -t;-t
(A ' )-t:

(A -r)'

5.8 RandomVectorsand Matrices

A random vector or a random matrix contains elementsthat are random variables. Thus,
the observationsvector Y in (5.4) is a random vector since the Y, elements are random
rtariables.

Expectationof Random Vector or Matrix

Supposewe have r : 3 observationsin the observationsvectorY:

Y
3xl

:[l]

The expectedvalue of Y is a vector,denotedby E{Y}, that is definedas follows:

I t t t ' lI

E { Y } : lE lY z ll

3x1

Luttrt-l

Section 5.8

RandomVectors and Matrices

195

Thus, the expectedvalue of a random vector is a vector whose elementsare the expected
values of the random variables that are the elementsof the random vector. Similarly, the
expectationof a random matrix is a matrix whose elementsare the expectedvalues of the
correspondingrandom variablesin the original matrix. We encountereda vector of expected
valuesearlierin (5.9).
In general,for a random vector Y the expectationis:

(s.38)

E{Y} : lEIYiJl

i :I,...,n

nX I

and for a random matrix Y with dimension n x p, the expectationis:

(s.39)

E{Y} : lE{Yii}l

i :1,...,ni

j :I,...,P

nx p

RegressionExample. Supposethe numberof casesin a regressionapplicationis n : 3

The threeerror terms t, 2,t3 eachhave expectationzero.For the error termsvector:

I
3xl

=
["]

we have:

E{e} :
3x1

since:

I t { ' ' }I

0
3x1

I ol

Lf[;]l:L3l
,lariance-Covariance Matrix of Random Vector
Consider again the random vector Y consisting of three observationsYr, Yz, L,. The
variancesof the three random variables.o:{Yi}, and the covariancesbetweenany two
of the randomvariables,o {Yi, Y1lt,are assembledin the yariance-covariance
matrix of Y ,
denotedby 62{Y}, in the following form:

(s.40)

62{ Y} :

y:}l
o{Y1,Y2l
f " t { rr}
"ty,,
o { Y r, Y , } o 2 { y rl o t Y r,\ }
I
|
lo{ Y r, Y , } o { \ , Y 2 } o ' { Y ) J

Note that the variancesare on the main diagonal,and the covarianceo {Y;, Yi} is found
in the i th row and 7th column of the matrix. Thus, o {I2, I, } is found in the secondrow, first
column, ando {Y,, 12} is found in the flrst row, secondcolumn. Remember,of course,that
o { Y r,Yri ' - o { Yr,I2 } . S i n c eo { Y ;,Y ;} : o{Y i , Y ;} for al l i t' j .o2{ Y } i s a symmerri c
matrix.

196

Chapter5

Matrix Approachto SimpleLinear RegressionAnalysis

It follows readily that:

(5.41)

o2{y}: E{ty - E{y}lty - E{y}l'}

For our illustration,we have:

or{Y} :

,''{[l-ritil

E{Yt}

Y2-E[Y2]

-U,rr"]
"

Multiplying the two matricesand then taking expectations,we obtain:

Lclcationin Product

ExpectedValue

(Yt _ E{YJ)z
(yt- E{Yt})(Y2-E{Y2})
(Yt - ElYl ))(r: - E{yz]j)
(Y2- EIy\DVt - EIYJ)
etc.

Row l, column 1
llow l, column2
l{ow l, column3
ll.ow 2, column I
etc.

o2lYrl
o lY1, Y2lr
o {Yr,Yr}
o {Yr, Yr]r
etc.

This. ol'course, lcadsto the variance-covariance

matrix in (5.40).Rememberthe definitions
ol'variarrccand covariance
in (A.15) and (A.21),respectively,
whentakingexpectations.
To gerrctlrliz.c.
the variance-covariance
matrix for an n x I randomvectory is:

( 5 4) l

(' 2{ Y }

olYr,Yr)
oz IYr|

| o"lY' t', .r]

Yrl

:lI'

o {Y t, Y r\
o {Y2,Ynl'

nX,l

Lo{ Y"'Yt\

o lYn, Y2)

Note againtharo21Y] is a symmetricmatrix.

Regressionllxarnple. Let us return to the examplebasedon n :3 cases.Supposethat
thethreeerrorlerttlshaveconstantvariance,o2le,] : oz,andareuncorrelated
sothato {ei,
e; ) : 0 l-ori I .i. "l'he variance-covariancematrix for the random vector e of the previous
examplcis tlrcrclirrcasfollows:

62{}
3x3

fo ' o o l
o2 0 |
Io o "r ]

:10

Notc:thrrtrtll r,:trianccsareo2 and all covariancesare zero. Note also that this variancectlvarianccttr:tlrix is a scalarmatrix, with the common varianceo2 the scalar.Hence,we
c:rncxprcssthc variunce-covariance
matrix in the following simplefashion:
o2 {El 3x3

o ol

fo,

"' lo 0 lJ
L0

L0

0 o' )

[t
o2l =

o2l
3x3

I o l:l

ol

o o2 o I

Section 5.8

Randont Vettors and Matrices

197

Some Basic Theorems

Frequently,we shallencountera randomvectorW which is obtainedhy prcrrrultiplyingthe
randomvectorY by a constantmatrix A (a matrix whoseelementsarc Iixctl):

(5.43)

W: AY

for thiscaseare:
Somebasictheorems
E{A} : A

(s.44)
{\

E{W} :E{AY} :AE{Y}

l l '\ l

62{w} : o2{AY}: Ao2{Y}A'

(5.46)

matrix of Y.
whereo2{Y} is the variance-covariance

Example. As a simple illustrationof the use of thesetheorems,consider:

A
2x2

2x1

Y
2xl

We then have by (5.45):

E {w ) :
2x 1

Ir

Ir

-1
I

I Iu ,r,]l: IE{ rr}r{

- r,} l
EtY)
I L r tr ,tl L r ty,]+

andby (5.46):

o21W1:
2x2

o { Y,.r ,} l[| tl
I r - ' lI o ? 1 Y,1
r
t
I
J ["1 r r r. ,1 " r 1 vr)yl- t r I
o2{y) - o2{yrl
f "'{v,} + 02lyzJ- 2olyr,yrl
o21Yrl- o'lyr\
o2lyrl + o21rry*2o{Y,, ,,, ]
L

Thus:
o21w,l- o'lYt - Y2J: o2{yrl+ o21vry- 2olYr,Yr)
o21wrl- o'lY, * Y) : o2{Yrl+ 02lY2}* 2olYr,Y2l
o lW1,W2)- olyt * yz,y, * y2|: o'lyrj - o2lyrj

198

Chapter5

Matrix Approachto SimpleLinear RegressionAnalysis

5 e simp'e'';::J"::il"::::l

again,ha,
::J.T:, Remember

*::;:::ff:T:

we will not presentany new results,but shall only statein matrix terms the results obtained
earlier.We beein with the normal error regressionmodel (2.1):

(s.41)

Yi:

flol BrX,+ e,

i:1.....n

This implies:

Y t : f ro* B , X ,* e ,
Y z : f l0 * B t X rt e ,

(5.47a)

Y n:frol B tX r+ e,
We definedearlierthe observationsvectorY in (5.4),the X matrix in (5.6),andthe I vector
in (5.10). Let us repeatthese definitions and also define the p vector of the regression
coefficients:

[r x ,I

[r,I
(s.48)

[' , I

: l' ' I
,i,
,g
,:[fl]
.L :ll"'l
Itl
L ',1
Lr, _l
[ t x, , ]

Y:
n Xl

Now we can write (5.47a)in matrix terms compactlyas follows:

(s.4e)

=x

nX l

since:

p+

nX 2 2\ I

nX l

I,
f'll
tll [ r x1'l[!:1.1:,

lL..l
:ll
r,l Lt xn)

L',_J

IBo*F,x,1 [''l

Ino*B,x,+e,)

l a o + 0 ,"
l *' | ' : l : |

,' * P ' x ,+ ' =

|

u,'*,*,,)
Lr,*'u,r,)
L"l Lr,*
Note that Xp is the vector of the expectedvalues of the I; observationssince E{Y;}
Bo + B rx ,;h ence:

(s.s0)

E{Y} :
nX l

f.
e

Xp
nX l

Section 5.9

Simple Linear Regression Moclel in Matrix Terms

lgg

whereE{Y} is definedin (5.9).

The column of ls in the X matrix may be viewed as consistingof the dummy variable
Xo : I in the alternativeregressionmodel (1.5):
Yi : FsXo -t BtX; * e;

where Xn : I

Thus, the X matrix may be consideredto contain a column vector of the dummy variable
Xo and anothercolumn vector consistingof the predictorvariableobservationsX;.
with respectto the error terms, regressionmodel (2.1) assumesthat E{;} : 0,
o21e;\ : o2, and that the e; are independentnormal random variables.The condition
E { r;} : 0 i n ma tri xte rm si s :

(s.s
r)

E{} :0
nX l

nX l

since(5.51)states:

Iat.,tl I o-l
oI
lt
{.
,
}
l_l
l: ll: l

fur',rjLr l

The condition that the error terms have constant variance o2 and that all covariances
6l i , j l fo ri
I j a re z e ro(s i n cethee, arei ndependent)i sexpressedi nmatri xterms
through the variance-covariancematrix of the error terms:

/5 52 r

00
f",
o2o
l0

^ :
o'{e}
I
nx n'

L0

ol
ol
.l

:l

::
00

"t)

Sincethis is a scalarmatrix, we know from the earlierexamplethat it can be expressedin

the following simple fashion:

(5.52a)

62{E} trxn

o2r
nxn

Thus,thenormalerrorregressionmodel(2.1,)i n matri xtermsi s:

(s.s3)

Y:XF+e

where:
e is a vector of independent normal random variables with E{e} :
oz{e} : 621

0 and

Chapter 5

Matrix Approach to Simple Linear Regression Analysis

5.10 LeastSquaresEstimationof RegressionParameters

Normal Equations
The normalequations(1.9):

(5 .5 4 )

nbo+brlXi:

f Y,

boLxi * brL x! : 2 x,Y,

in nratrixternrsare:

(.5..s-5
)

X'X

2x.2 zxl

X/Y
2xl

wlrcrc b is thc vcctor of the leastsquaresregressioncoefficients:

'|'

b : [?'l

(5 .5 5 a )

2x1

Lbr )

To seelhis. rccirllthatwe obrainedx/X in (5.14)and x'Y in (5.15).Equation(5.55)thus

I

["i,;;i][i:1:t;';]
-l
I n b r + b t r x i l_ [ t t ,
I a o : x ,+ b t , x ?) L r " , r , l
Thescirrcprccisclythe normalequationsin (5.54).

E stimated Reg ressi on Coeff i c i ents

Ttrohtuilrlhc eslirnatedregressioncoefficientsfrom the normal equations(5.55)by matrix
rncthotls.
rvc Plerrrultiplyboth sidesby the inverseof X/X (we assumethis exists):

(x'x)-rx'xb- (X'x;-t*'t
W eth e nl i nd, si ncc(X ' X ;-t;1' " : f andIb : b:

tS 5 6 r

5
2x1

(X/X)-tX,y
2x2

2xl

The estimatorsbo and b, tn b arethe sameas thosegiven earlierin ( 1.l0a) and ( l. l0b). We
shall demonstratethis by an example.

Sectbn 5. I0

Least Squares Estimation of Regression parameters

201

Example. We shall use matrix methodsto obtain the estimatedregressioncoefficientsfor

the TolucaCompanyexample.The dataon the I and X variableswere given in Table l.l.
Using thesedata, we def,ne the Y obserrrationsvector and the X matrix as follows:

I r 8ol

[:eel

(a)
" : l'?' |

15 57\

(b)*: ll': l

1,,,
J

We now require the following matrix products:

(5.58)

l- r I . . .
X ',X : | ^.
180 3 0

Li^j

80l
I 1r ,0
t' l |
| _ t 2s 1.7s0'l
Toll'
: I - Ll.7so t42.3ooJ
70)
Lr

(5.59)

l -r

[:eoI

r l | 1 2 rI

I 6 1z.soz
I
7o.l
:
|
l: L 7 , 1 8I 0

X'Y:l^.
180 30

1323J
Using (5.22),we find the inverseof X'X:

(5.60)

- 33il:;,,
: [_33]1li
(x,x)-,
]

In subsequentmatrix calculationsutilizing this inverse matrix and other matrix results. we

shall actually utilize more digits for the matrix elementsthan are shown.
Finally, we employ (5.56)to obtain:

7 .8 0 7 .l
t5.6lr b
- ? o-: l ""^'
r X 'Xr - ,X' y:
I lt' 1 7 2- .0000s0s
9 9 ]:1r:I t el7.
- : |lb,l
ts0J
l-.oo:s:s
.1l

_l e z . nI

| 3.s102l

or bo: 62.37 andbr :3.5102. Theseresultsagreewith the onesin Chapterl. Any

differenceswould havebeendue to roundins effects.
Comments
I . To derivethe normalequationsby the methodof leastsquares,we minimizethe quantity:

Q:DIYi _ $o+ Fl))2

In matrix notation:

(s.62)

Q: (Y - Xp),g _ Xp)

Expanding, we obtain:

e :Y'Y - F'x'Y- Y'xF+ p'x'xp

2ll2

Clnpter5

MatrixApproachto SimpleLinearRegression
Analysis
since(Xp)/ : p'X'by (5.32)-Note now that Y'Xp is I x 1, henceis equalro its transpose,
which accordingto (5.33)is p'X'Y. Thus,we find:
(5 .6 3 )

Q:Y'Y -2F'x'Y+ p'x'xp

To find the value of p that minimizes Q, we differentiatewith respectto po and Br.Let'.

a ' ^.

(5.64)

f| ao1|
I adoI

ap(s'=| og I
la h )

Then it follows that:

(s.6s)

e
a F (Q : -2 x ' Y + 2 x ' x p

Equatingto the zero vector,dividing by 2, and substitutingb for p givesthe matrix form of the
leastsquaresnormalequationsin (5.55).
2. A comparisonof the normalequationsand X'X showsthat wheneverthe columnsof
X'X are linearly dependent,the normal equationswill be linearly dependentalso. No unique
solutionscan thenbe obtainedfor bo and b,. Fortunately,
in mostregressionapplications,
the
teadingto uniquesolutionsfor bn and0,.
columnsof X'X arelinearlyindependent,

5.11 FittedValuesand Residuals

Fitted Values
Let the vector of the fitted values Y, be denotedby Y:

-^lv, l

(s.66)

n\1

:l l' lt, l
l ^' l

L Y' l
In matrix notation,we then have:

(s.67)

Y:
nX l

xb
nx2 2x1

f a^+u,x,l
I ui+a',x"
I

Ltl[l:]

fLot?'l:
t-l
J
Ir,* r,",-]

Section 5.II

FittedValues and Residuals

203

Example. For the TolucaCompanyexample,we obtainthe vector of fitted valuesusins

the matricesin (5.57b)and (5.61):

(5 .6 8 )

I r 80l

[:+z.rs
I

Li toj'

[r228_]

v:xb:I l, I f,3jl.,l:| 16e47

|
'

The fitted valuesare the same,of course,as in Table 1.2.

Hat Matrix. We can expressthe matrix result for t in (S.OZ)as follows by usins the
expressionfor b in (5.56):

t : X(X'X)-'X'y
or, equivalently:

(s.6e)

Y:HY
nX l

nX n nX l

where:

(5.69a)

: X(X'x)-rx'

nX tr

We see from (5.69) that the fitted values Y, canbe expressedas linear combinationsof
the responsevariableobservationsf,, with the coefficientsbeing elementsof the matrix
H. The H matrix involves only the observationson the predictorvariableX, as is evident
from (5.69a).
The squaren xn matrix H is calledthe/zatmatrix.ltplays animportantrolein diagnostics
for regressionanalysis,as we shall seein Chapter9 when we considerwhetherresression
resultsareunduly influencedby one or a few observations.The matrix H is symmetricand
hasthe specialproperty(calledidempotency):

(s.70)

HH:

ln general,a matrix M is saidto be idempolenlif MM : M.

Residuals
Let the vector of the residualse, :

Y .-Y .he
'1

denotedbv e:

[''I

(s.1r)
nX l

l",l

1",
l

2(l.l

('lnt1t11'v5

Matrix Approach to Simple Linear Regression Analysis

In matrix notation, we then have:

(s.72)

e=Y
nX l

nX l

Y:

nX 1

nX l

-Xb
nX l

Example. For the Toluca Company example, we obtain the vector of the residuals by
usingthe resultsin (5.57a)and (5.68):

[:+z.es
I I s r . oz l

[:eeI

(s.73)

'',i'll-or;or
I
":|'1'
l|
lzzz
) 1r,r.rrl|
'0"-l

The residualsare the sameas in Table 1.2.

Variance-CovarianceMatrix of Residuals. The residualse,, like the fitted values i,
can be expressedas^linearcombinationsof the responsevariable observationsI;, using the
resultin (5.69)for Y:
e= Y -?:Y -H Y :fl -mY
We thus have the important result:

(s.74)

e:(IH)Y
nX I

nxn

nxn nX l

where H is the hat matrix defined in (5.69a). The matrix I - H, like the matrix H, is
symmetric and idempotent.
The variance-covariancematrix of the vector of residualse involves the matrix I _H:

(s.7s)

o2{ e} = o2(I-H )
nXn

and is estimatedby:

(5 .7 6 )

:MsE(I-H)
s21e1
nX n

Note
The variance-covariancematrix of e in (5.75) can be derivedby meansof (5.46). Since
e : (I - H)Y, we obtain:
or1e1: (I - H )62{ Y }(I - H )'
Now oz{Y} : o2{s} : o2lfor the normalerrormodelaccordingto (5.52a).Also, (I - H)/
I - H becauseof the symmetryof the matrix. Hence:
a':

o21e1: o2(I-H )I(I-H )

: o2(t - H )(I - H )

i,
Y.:i--

Section5.12 AnahsisofVarianceResults

205

lnv iewof t hef ac t t hat t hem at r i x l - H i s i d e m p o t e n t , w e k n o w t h a t ( I - H ) ( I - H ) : I - H

andwe obtainformula(5.15).

5.12 Analysisof VarianceResults

Sums of Squares
To seehow the sumsof squaresareexpressedin matrix notation, we begin u'ith the total sum
of squaresSSZO,definedin (2.43).It will be convenientto use an algebraicallyequivalent
expression:

.lYi\2
SSIO: LtYi - Yl' : L,rz -

(s.11)

We know from (5.13)that:

Y,Y:L,Y?
The subtractionterrn (rY)2 ln in matrix form usesJ, the matrix of ls definedin t5.18),as
follows:

(r v')r

(s.7s)

: (1) v'rv
\n/

For instance,if n :2, we have:

(i),, ,,rIl l][l;]:

(Yt+y2)(Yr+Y,)

Hence,it follows that:

(s.19\

sszo:Y/Y- (1)v'r

by Y'Y in matrix terms,so 55p : L el : t(ri - f; tr can

Just as X ff is represented
be reoresentedas follows:

(s.80)

SSE: e'e : (Y - Xb)'(Y - Xb)

whichcanbe shownto equal:

SSE: Y'Y - b'X'Y

(5.80a)
Finally,it canbe shownthat:

(s.81)

ssR: b'x'y- (1)v'n

Chapter 5

Matrix Approach to Simple Linear Regression Analysis

Example. Let us find SSE for the Toluca Company example by matrix methods, using (5.80a).Using (5.57a),we obtain:

Y/Y: [399 121

[:se
I
lr 2r l

32311 | : 2,74s,173
l:l
L323)

andusing(5.61)and (5.59),we find:

b,yly:t6237 3.s102t
:2.6s0,348
Ir,]:?33]
sSE: y'y - tr'xiy : 2,745,1',73
: 54,g25
- 2,690,34g
which is the sameresull as that obtainedin Chapterl. Any differencewould havebeendue
to rounding effects.
Note
To illustratethederivationof thesumsof squares
expressions
in matrixnotation,considerSSE:

ssE: e/e: (y _ xb)/(y_ xb) : y,y _ 2b,x,y+ b,x,xb

In substitutingfur therightmostb we obtainby (5.56):

ssE: y'y - 2btx,y+ b,x,x(x/x)-,x,y

: y'y - 2b'x'y + b/Ix'y
tn droppingI andsubtracting,
we obtaintheresultin (5.80a).

Sums of Squaresas Quadratic Forms

TheANOVA sumsof squarescanbe showntobequadraticforms.Anexampleof aquadratic
form of rhe observationsY, when n :2 is:

(s.82)

5 Y ?+ 6 Y t Y z + 4 Y :

Note that this expressionis a second-degree

polynomial containingterms involving the
squaresof the observationsand the crossproduct.We can express(5.82)in matrix termsas
follows
(5 .8 2 a )

Is
lY_-t1
' Y rll:
L.-

where A is a symmetric matrix.

llli:l: "'o"

Section 5.12

Analysis ofVariance Results

In general,a quadratic form is defined as:

(5.83)

Y 'A Y :iio,,y,y.
rx l

where

' t' t

E ti !1

aii : aii

A is a symmetric n x n maftix and is called the matrix of the quadratic

form.
The ANovA sums of squares.ssro, ,s,tE, and ssR are ali quadratic forms,
as can be
seenby reexpressingb/X/. From (5.67),we know, using (5.32),ihat:

b'X' : (Xb)': t'

We now use the result in (5.69) to obtain:

b'X': (HY)'
SinceH is a symmetric
matrixsothatH, : H, we finallyobtain,using(5.32):
(5.84)

b,X, _ Y,H

This result enablesus to expressthe ANovA sumsof squaresas follows:

(5 .8 5 a )

sszo:
Y'ir- (;) r]'

(s.8sb)

S S E : Y ,(I _ H )Y

(5.85c)

ssR:

"' ["

- ( ;) ' ]

"

Eachof thesesumsof squarescan now be seento be of the form y/Ay, wherethe three
A
matricesare:

(5 .8 6 a )

(s.86b)
(5.86c)

'-

( 1) r
\n/

I-H
\n/
"- 11)r

Since each of these A matrices is symmetric, sszo, ssE, and ssR are quadratic forms,
with the matricesof the quadraticforms given in (5.86). Quadraticforms play an important
role in statisticsbecauseall sumsof squaresin the analysisof variancefor linear statistical
modelscan be expressedas quadraticfoms.

208

Chapter5

MatrixApproachto SimpleLinear RegressionAna[ysis

5.13 lnferencesin RegressionAnalysis

point estimator
As we saw in earlier chapters,all interval estimatesareof the following form:
point
estimator'
the
of
deviations
standard
plus and minus a certain number of estimated
of the
deviation
standard
estimated
the
and
Similarly, all tests require the point estimator
Matrix
squares'
of
sums
various
tests,
variance
point esiimator or, in the caseof analysisof
estimated standard
algebra is of principal help in inference making when obtaining the
of the sums
equivalents
matrix
the
given
deviations and sums of squares.We have already
for
expressions
matrix
the
on
chiefly
here
focus
of squaresfor the analysisof variance.We
interest'
of
the estimatedvariancesof point estimators

RegressionCoefficients
Ihe variance-covariancematrix of b:
0 2 {b }
2x2

o ' lu o l o { b g , b t J l
: I
b e } o 2 lb ] )
Io{b1.

o2{b} : ozlx'X)-l
2x2

(s.88)
or, using (5.24a):

foz
t-

62{ b}
2x2

(5 .8 8 a )

-t

_ ln
I
L

o2x2
*, ;1x,
-Vy

-X oz

,(xi - x)'
oZ

-X o2
-.---.:;

----.-----=E(Xi - X)'

t(xi - x)'

When MSE is substitutedfor o2 in (5.88a),we obtainthe estimatedvariance-covariance

matrixo1'b.tienotedbY s2{b}:

(s.8e)

f usr _r MSE(X:)
- -xusE^
t( xi-x)'
s:1 b:M
1 sE( x' x) - ' I:l '
? ) ,1 ::*"
MSE
rvr

-Xusn

tt",;f

,6=W

(2.22b) and of, b', in (2.3b) and the

In (5.88a),you will recognizethe variancesof bo in
in (5'89) are familiar
covariancetr ao ana a, iln 1+.s1.Likewise, the esiimated variances
from earlier chaPters.
exampleby matrix
Example. We wish to find s2{bo}and s2{b,} for the TolucaCompany
methods.Using the resultsin Figure 2'2 andin (5'60)' we obtain:

(5.90)

,o, |: MSE(x'x)-r' : .2'56+

s21b1

.287415-.003535 I

I
f-.oo:srs .oooo5o5l

-8.428 I
: I oas.:+
)2o4o)
-a.+za
[

Section 5.13

Thus, s2{b0}:
Chapter 2.

Inferences in Regression Analysis

209

685.34and s2{b,} : .12040.Theseare the sameas the resultsobtainedin

Note
To derivethe variance-covariance
matrix of b, recall that:

b : (x / x )-ix , Y : A Y
whereA is a constantmatrix:
(X /X )-rX /

A:
Hence,by (5.46)we have:

o2{b} : Ao2{Y}A'
Now o2{Y} : d2L Fufther,it follows from (5.32)andthe fact that (X'X)-l is symmerricthat:

A': X(X'X)-r
We find therefore:

62{b} : 1x'xy-tx' o2lx(x'x)- I

: o2(x'X)-rx'x1x'x;-r
: o2(x'x)-1I
: o2(x'x)-1

hleanResponse
To estimatethe meanresponseat Xp,let us definethe vector:

(s.91)

Xh:lo
zxl

l-r I
L"hJ

or

X'h:il

Xnl

lx2

The fitted value in matrix notation then is:

(s.e2)

?lr:x'nb

since:

x;b : Ir

l-i" I

: [lr] : 11,
xnll lo | : [bo* b1X1)
L"I J

Note that X[b is a I x I matrix; hence,we can write the final result as a scalar
The varianceof ?1r,givenearlierin (2.29b),in matrix notationis:

(s.e3)

o2{?nl - o'x'h(x'x) - I xft

The varianceof Y7,in (5.93) canbe expressed

asa functionof o2 1b1.ttrevariance-covariance
matrix of the estimatedregressioncoefficients,by making use of the resultin (5.88):

(5.93a)

o2It'nl : x'ncz{b}xn

210

Chapter5

Matrix Approachto SimpleLinear RegressionAnalysis

The estimatedvarianceof Il, given earlierin (2.30),in matrix notationis:

t21?o1: MSE(x/h(x'x)-txr,)

(s.e4)

Example. We wish to find s21ir) for the TolucaCompanyexamplewhen X1 : 65. We

define:

x; : tl 6sl
and use the result in (5.90) to obtain:

s21?^1:
xi,s21n1x,
...lass:q
lf r l
=rl o
' ll - s.+ zs- 8.428
.r 2 o 4 o ) lo:r!ls' :z
This is the sameresult as that obtained in Chaoter 2.
Note
Theresultin (5.93a)canbe deriveddirectlyby using(5.46),sincein : X,r,b:

o21?1,|:xioz{b}Xr,
Hence:

= 1t
o21i'01
of:

(5.e5)

,ulJ*',ii',,
"!;,'Ji']
[;,]

o'{i'n\: oz{bo}*2X1,olbo,b)+ x|ozlbJ

Using the results from (5.88a), we obtain:

) ,:. .
-r'ht

o2

o 2 *2

>(Xi - X)'
-

'

2X t \-X)o2

X?.o2

>1x, - x't2 ,(xi - x)2

which reducesto the familiar expression:

(5.95a)

o r lr' ) : " r l! * t x n - x ) ' z ]

ln t(xt - x)'/1

Thus,we seeexplicitlythat the varianceexpression

in (5.95a)containscontributions
from
o21Ul,o21b.,1,
ando {bo,b1}, whichit musraccording
(A.30b)since?1,: bo+brX1,
to theorem
is a linearcombinationof bo and,br.

Prediction of New Observation

The estimatedvariances2{pred},givenearlierin (2.38),in matrixnotationis:

(s.96)

L__

s2{pred}: MS E (I + X ;(X ' X )-rX ft )

Problems

2ll

Cited Reference
Graybill, F. A. Matices with Applications in statistics. 2nd ed.
Belmont, calif.:
Wadsworth,1982.

Problems
For the matricesbelow,obrain(1) A + B, (2) A _ B, (3) AC, (4) AB,, (5)
B,A.

[r +l
6

A :12

Ir :l

n :lr

+l

lz sJ

L3 8 J

T"
c:ll or 0t.l
l
L)

Statethe dimensionof eachresultingmatrix.

5.2. For the matricesbelow,obtain(1) A + C, (2) A - C, (3) B/A, (4)
AC,, (5) C,A.

":fi]

lz rl

[: sl

o :l; ; I

.: I I 9
|
l) r I

14 8_l

12 4l

Statethe dimensionof eachresultingmatrix.

5.3. Showhow the followingexpressions
arewrittenin termsof matrices:(1) yi _ ii : ei,
(2 )t X,e i : 0 . A s s u m ei : 1, ..., 4.
t/ s'+. tr'lavordeterioration.Theresultsshownbelow
wereobtainedin a small-scale
experimenr
to studythe relationbetweenoF of storagetemperature(X) andnumberof weeks
before
flavordeterioration
of a food productbeginsto occur(y).
i :1 2 3
Xi :8 4
Yi:
7.8

9.0

0-4-8
10.2
I 1.0

l r .1

Assumethat first-orderregression
model(2.1)is applicable.Usingmatrixmethods.find
(1) Y'Y, (2) X'X, (3) X/Y.
5'5' Consumerfinance.The databelow show,for a consumerlinancecompany
operating
in six cities,the numberof competingloan companiesoperatingin the city (X)
andthe
numberperthousand
of thecompany'sroansmadein thatcity thatarecurrentryderinquent
(Y ):

r23456
Xi :4 1 2334
yi:
t6
5

10 15
13 22
Assumethat flrst-orderregression
model(2.1)is applicable.using matrixmethods,find
(l ) Y ' Y . (2 ) X' ,X,(3 ) X' ,Y.

('lurltrcr 5

Mutrix Approach to Simple Linear Regression Analysis

\i 5.6. Refer to Airfreight breakageproblem 1.21.Using matrix merhods,find (l) y/y, (2)

x'x, (3)x/Y.

Referto Plastic hardnessProblem1.22.Using marrixmethods,find (1) y,y, (2) x/x,

(3) X/Y.
Let B be definedas follows:

" :L[r i;0 35 J 1

a. Are the column vectorsof B linearly dependent?
b. What is the rank of B?
c. What must be the determinantof B?
5.9. Let A be definedas follows:

-1

r - l8l
A:10 3 ll
1055J
[n

a. Are the column vectorsof A linearly dependent?

b. Restatedefinition (5.20) in terms of row vectors.Are the row vectorsof A linearly
dependent?
c. What is the rank of A?
d. Calculatethe determinantof A.
5.10. Find the inverseof eachof the followins matrices:

^:[3i]

B:

l+6 53 2110
|

Lr0 r6J

Checkin eachcasethat the resultingmatrix is indeedthe inverse.

Find the inverseof the followins matrix:

[s r 3l

A:14 0 5l
Lr e 6)

Checkthat the resultingmatrix is indeedthe inverse.

5 .1 2 .Referto Flavor deteriorationProblem5.4.Find (XrX;-t.
5 .13 . Referto ConsumerfinanceProblem5.5.Find (X,y;-t.
5 .r4 . Considerthesimultaneous
equalions:
4 y , i 7yr:

2yr * 3yr:12
a. Write theseequationsin matrixnotation.

)5

Problems

213

b. Using matrix methods,find the solutionsfor y, and yr.

5.15. Considerthe simultaneous
equations:
5yti _Zyt:8
23y14- 7lz : 28
a. Write theseequationsin matrix notation.
b. Usingmatrixmethods,find the solutionsfor y, andyr.
5.16. Considertheestimated
linearregression
functionin theform of (1.15).Write expressions
in this form for the fitted valuesi in matrix termsfor I : 1, . . ., 5.
5.17. Considerthefollowingfunctionsof therandomvariablesyr, yr, and,y.:
W t:Y r+ Y 2+ Y 3
W z: Y t - Y z
W z:Y r-Y z-Y z
a. Statethe abovein matrix notation.
b. Find the expectationof the randomvectorW.
c. Find the variance-covariance
matrix of W.
5.18' considerthe followingfunctionsof therandomvariablesyr, yz, y3, and r_r:
1

W r : ; ( Y t +Y 2 +Y 3 +Y 4 l
Wr:

ll
i(Y,

-r Yr) ,(y3

+ y4l

a. State the above in matrix notation.

b. Find the expectation of the random vector W.
c. Find the variance-covariance matrix of W.
5.19. Find the matrix A of the quadratic form:

3Y ?+ 1 o y y 2 + 1 7 y 1
5.20. FindthematrixA of thequadratic
form:
lYl -8YrY2+8Y:
5.21.Forthematrix:
[s -l
)f
A:1"
1_l
L2
y, andyr.
findthequadratic
formof theobservations
5.22.Forthematrix:

1041
0 3 ol
4 0 eJ
find the quadratic form of the observations; Yr, Yr, andY,

t
*_>---*

214

Chapter5

Matix Approachto SimpleLinear RegressionAnlalysis

'l S.ZZ.Referto Flavor deteriorationProblems5.4 and5.12.

a. Using matrixmethods,obtainthe following: (1) vectorof estimatedregression
coefficients,(2) vector of residuals,(3) s.tR, (4) ssE, (5) estimatedvariance-covariance
matrixof b, (6) pointestimateof E lYo] whenX1 = -6, (7) estimatedvarianceof io
when X7,: -6.
b. What simplificationsarosefrom the spacingof the X levels in the experiment?
Find the hat matrix H.
Finds2{ei.
5.24. Referto ConsumerfinanceProblems5.5 and5.13.
using matrixmethods,obtainthe following:(1) vectorof estimatedregression
coefficients,(2) vector of residuals,(3) ssR, (4) ssE, (5) estimatedvariance-covariance
matrix of b, (6) point estimateof E {Y1,}whenX1 : 4, (j) s2{pred}whenX o : 4.
b. From your estimatedvariance-covariance
matrix in part (a5), obtain the following:
(1) s{ bn,b,}; (2) s21br};
(3) s{b,}.
a,

c . Find the hat matrixH.

d. Finds2{e}.

1 5.25. Referto Airfreight breakageProblems1.21and5.6.

a. Using matrixmerhods,
obtainthefollowing:(l) (X/D-r, (2) b, (3) e, (4) H, (5) ,SSE,
(6) s2{b},(7) i, whenXn = 2,18;s21io1whenX1,- 2.
b. Fromparl(a6),obtainthefollowing:(l) 12{br};(2) s{Do,b,i; (3) s{bo}.
c. Find thc ntarrixofthe quadraticform for SSR.
5.26. Refer to PlastichardnessProblems1.22and5.7.
a. Llsingnrarrixnrerhods,
obtainthefollowing:(1) (X,X)-1, (2) b, (3) i, <+lH, (5)SSE,
(6 ) s?{b} ,(7) s2{pred}
w henX 7,- 30.
b. Irrorn pilrr (a6). obrain the following: (l) s2{bo};Q) s{bo, D, }; (3) s{b1}.
Obtain tlrc nr:rtrix ol'the quadratic form for SSE.

Exercises
5.27. Referto regression-through-the-origin
model(4.10).Setup the expectation
vectorfor .
A s s u methati :1,...,4.
5.28. Considermodel (4.10) for regressionthroughthe origin and the estimarorb, given
in (4.14).Obtain(4.14)by utilizing(5.56)with X suitablydefined.
5.29. Considerthe leastsquaresestimatorb givenin (5.56).Using matrixmethods,showthat
b is an unbiasedestimator.
5.30. Showthat i, in (5.92) canbeexpressed
in matrixtermsasb/X;,.
5.31. Obtain an expressionfor the variance-covariance
matrix of the fined valuesf,, I :
1. .... n. in termsof thehatmatrix.

b-