ARMA Processes

Brian Borchers April 3, 2001

Some notation

In the following, we will make use of forward and backward shifts in time. The B operator is dened by Bzn = zn1 while the F operator is F zn = zn+1 . Note that B and F are not numbers but rather operators that act on a time series zn . We can extend this notation to include powers of B and F B k zn = znk and F k zn = zn+k . Also, we can build polynomials from the B and F operators. For example, (1 B + 0.5B 2 )zn = zn zn1 + 0.5zn2 . Another important example is the backward dierence operator zn = (1 B)zn = zn zn1 .

White Noise

We will frequently make use of a white noise. The white noise process has 2 An normally distributed with mean 0, variance A , and autocovariance k = 0, k = 1, 2, . . . and autocorrelation k = 0, k = 0, 1, . . .. White noise can easily be generated in MATLAB using the randn command. Using our formula for the spectrum of a stationary process from its autoco2 variance, its easy to show that the white noise process should have I(f ) = 2A , 0 f 1/2. In the limit as n goes to innity, the spectrum is constant for all frequencies. However, for any actual realization of the white noise process, the sample spectrum will contain considerable noise.

The ARMA process

An autoregressive moving average (ARMA) process is obtained by applying a recursive lter to white noise. In terms of the elements of the zn and an sequences, zn = 1 zn1 + 2 zn2 + . . . + p znp + an 1 an1 . . . q anq . The terms 1 zn1 through p znp are the autoregressive portion of the lter. The terms an through q anq are a moving average of the white noise input process. Notice that this has the form of the recursive IIR lter that we previously considered, except that the rst coecients have been normalized to 1. In terms of our operator notation, (B)zn = (B)an where (B) = 1 1 B . . . p B p and (B) = 1 1 B . . . q B q . Note the unusual notational convention of minus signs in front of each coecient. Once weve written the lter in our operator notation, its possible to obtain the transfer function. Let (B) . (B) = (B) Then zn = (B)an . We can expand (B) as (B) = 1 + 1 B + 2 B 2 + . . . where the coecients k can be obtained by Taylor series expansion. Note that the constant coecient is always 1 and that this time (only) weve used positive signs in front of the coecients. An alternative is to let (B) = 1 (B) = In this case, an = (B)zn . Clearly, (B) = 1 (B). We can expand (B) in a Taylors series as (B) = 1 1 B 2 B 2 . . . 2 (B) . (B)

Example 1 Consider the ARMA process Zn 0.5Zn1 = An 0.3An1 + 0.2An2 . Here (B) = 1 0.5B and (B) = 1 0.3B + 0.2B 2 . Using Maple to compute the Taylors series, we obtain (B) = (B) = 1 + 0.2B + 0.3B 2 + 0.15B 3 + . . . (B)

Thus 1 = 0.2, 2 = 0.3, and 3 = 0.15. Similarly, (B) = (B) = 1 0.2B 0.26B 2 0.038B 3 . . . . (B)

Thus 1 = 0.2, 2 = 0.26, and 3 = 0.038.

Stationarity and Invertibility

Unfortunately, it is easy to write down an ARMA process which is not stationary. For example, let Zn = 2Zn1 . Then V ar(Zn ) = 4V ar(Zn1 ), and the process is clearly not stationary. It can be shown that if

|j | <
j=1

then the ARMA process is stationary. This happens if the series (B) converges for every B with |B| 1. Since (B) is a rational function, it can also be shown that the series converges for every B with |B| 1 if the complex zeros of (B) lie outside the unit circle. If we have a stationary process, then since Zn = (B)An , and the expected values of An are all 0, the expected value of Zn is also 0. A related issue is that of invertibility. Recall that we can write zn in terms of an and previous values of znk . That is, (B)zn = an or zn = an + 1 zn1 + 2 zn2 + ... This inverted form of the process provides a very useful way of generating a random sequence according to our ARMA process. However, this innite sum must be truncated in practice. If the j coecients do not decay to zero, then it isnt possible to approximate this innite sum by truncating it. We say that the process is invertible if

|j | <
j=1

Since (B) is a rational function, the series is invertible if the complex zeros of (B) lie outside of the unit circle. Example 2 Recall the ARMA process of example 1. In this case, since (B) = 1 0.5B, the only zero of (B) is at B = 2, which is outside of the unit circle, so the process is stationary. The zeros of (B) are at B = 0.75 2.1i, so the process is also invertible.

Finding the autocovariance and autocorrelation of an ARMA process

In order to nd the autocovariance of an ARMA process, we start with the model in the recursive lter form. Zn = 1 Zn1 + 2 Zn2 + . . . + p Znp + An 1 An1 . . . q Anq . Next, we multiply both sides by znk and take expected values. Since E[Zn ] = E[znk ] = E[A] = 0, Cov(Zn , Znk ) = E[Zn Znk ]. Thus Cov(Zn , Znk ) = 1 Cov(Zn1 , Znk ) + . . . + p Cov(Znp , Znk ) +Cov(Znk , An ) 1 Cov(Znk , An1 ) . . . q Cov(Znk , Anq )

So, k = 1 k1 + . . . + p kp + ZA (k) 1 ZA (k 1) . . . q ZA (k q) where ZA (j) = Cov(Znk , Anj ) Since Znk is independent of the white noise at times after n k, these covari ances are 0. Also, since Znk = j=0 j Ankj , the remaining covariances are 2 given by Cov(Znk , Ankj ) = j A . Thus ZA (j) = 0 2 j A j>0 j0

So, we can express the autocovariance at lag k as

2 k = 1 k1 + . . . + p kp + A (k 0 k+1 1 . . . q qk )

When k q + 1, this simplies to k = 1 k1 + . . . + p kp . Another important case is k = 0. The variance 0 is given by

2 0 = 1 1 + . . . + p p + A (1 1 1 . . . q q )

These recurrence relations can be solved to obtain the autocovariance and autocorrelation. 4

The power spectrum can be obtained by substituting B = e2piif in the transfer function 2if 2 ) 2 (e . I(f ) = 2a (e2if ) These computations can all be performed for arbitrary ARMA(p,q) processes. However, in practice, the most important processes have p and q quite small, and general solutions for these particular ARMA(p,q) processes have been developed. As an example, consider the second order autoregressive process Zn = 1 Zn1 + 2 Zn2 + An It can be show that this process is stationary if 1 + 2 < 1, 2 1 < 1, and 1 < 2 < 1. Because (B) = 1 this process is always invertible. To compute the autocovariance, we multiply the above formula by Znk and take expected values. Cov(Zn , Znk ) = 1 Cov(Zn1 , Znk ) + 2 Cov(Zn2 , Znk ) + Cov(An , Znk ) When k = 0, we get 0 = 1 1 + 2 2 + Cov(An , Zn ) But Zn = 1 Zn1 + 2 Zn2 + An , and An is independent of Zn1 and Zn2 , so 0 = 1 1 + 2 2 + Cov(An , An ) or
2 0 = 1 1 + 2 2 + A

When k > 0, Cov(An , Znk ) = 0, and we get k = 1 k1 + 2 k2 In terms of the autocorrelation function, we have 0 = 1 and 1 = 1 0 + 2 1 . Solving this equation for 1 , we get 1 = For k > 2, we get k = 1 k1 + 2 k2 k > 2 Example 3 Consider an AR(2) process with 1 = 0.5 and 2 = 0.3. We generated a random sequence according to this process. Figure 1a shows 5 1 1 2

10 zn 0 200 400 n 600 800 1000

10 0 1 n 0.5 0 0 1 rn 0.5 0 0

10 n

15

20

10 n

15

20

Figure 1: An AR(2) process with 1 = 0.5, 2 = 0.3.

the rst 1000 points of this random process. Figure 1b shows the theoretical autocorrelation. Figure 1c shows the autocorrelation as estimated from the 20,000 point sequence. Next, consider the ARMA(1,1) process Zn 1 Zn1 = An 1 An1 Here (B) = 1 1 B and (B) = 1 1 B. We need to make sure that the roots of (B) and (B) are outside of the unit circle. This process is stationary if 1 < 1 < 1 and invertible when 1 < 1 < 1. We can also compute (B) = 1 + (1 1 )B + . . .. The recurrence relations for the autocovariance give 2 0 = 1 1 + A (1 1 1 )
2 1 = 1 = 0 1 A

k = 1 k1 k 2 These equations can be solved for the autocovariance. We can then convert the solution to an autocorrelation function. The result is 1 = (1 1 1 )(1 1 ) 2 1 + 1 21 1

k = 1 k1 k >= 2 Box, Jenkins, and Reinsel contains specic solutions for the autocorrelations of a variety of ARMA(p,q) processes with small values of p and q. 6