Stationary ARMA processes - Economics · De–nitions and Review MA processes AR processes ARMA Processes White noise processes I De–nition A white noise process is a sequence fε

De�nitions and Review MA processes AR processes ARMA Processes

Stationary ARMA processes

Helle Bunzel

ISU

January 30, 2009

Helle Bunzel ISU



Introduction I

In this part of the lectures, we will start with some de�nitions, newand review.

After this we will focus on ARMA processes.

They are important because they are used VERY frequently to modelerror processes of time series.We will look at:

What they are and their properties.A bit about how to estimate them.

Helle Bunzel ISU



White noise processes I

De�nition

A white noise process is a sequence fεtg∞t=�∞ , where

E (εt ) = 0

E�ε2t�= σ2

E (εi εj ) = 08i 6= j

De�nitionAn independent white noise process, is a white noise process where εi andεj are independent 8i 6= j

Helle Bunzel ISU



White noise processes II

De�nitionA Gaussian white noise process, is a white noise process where

εt � N�0, σ2

�Note that Hamilton de�nes the Gaussian process a little bit di¤erently.He starts with an independent white noise process. Does that matter?

Helle Bunzel ISU



Expectations I

This part is mostly review.

We typically consider a sequence of random variables fYtg∞t=�∞ ,

from where we get a sample fytg∞t=�∞

In fact we usually only get a sample fytgTt=1 from fYtgTt=1 .

From one sequence of random variables fYtg∞t=�∞ we can imagine

getting many sample sequences:�y1t∞t=�∞ ,

�y2t∞t=�∞ , ...,

�y It∞t=�∞ .

The mean of Yt is

E (Yt ) �Z ∞

�∞yt fYt (yt ) dyt

Helle Bunzel ISU



Expectations IIWe could estimate this mean by taking the average of the t 0thobservation from each of the I sequences above: 1I ∑I

i=1 yit

Note that

limI!∞

1I

I

∑i=1

y it = E (Yt )

Also note that this part of Hamilton is a little unclear.

Helle Bunzel ISU



Expectations IIITypical time series and their means:

Yt = µ+ εt , E (Yt ) = µ

Yt = βt + εt , E (Yt ) = βt

Note that the mean can depend on t!

In general

E (Yt ) = µt

Similarly

V (Yt ) �Z ∞

�∞(yt � µt )

2 fYt (yt ) dyt

Helle Bunzel ISU



Autocovariance IDe�nition

The j 0th autocovariance of Yt is

γtj � Cov (Yt ,Yt�j ) = E�(Yt � µt )

�Yt�j � µt�j

��Recall that this de�nition implies that

γtj =Z ∞

�∞

Z ∞

�∞(yt � µt )

�yt�j � µt�j

�fYt ,Yt�j (yt , yt�j ) dytdyt�j

Helle Bunzel ISU



Autocovariance II

This is also equivalent to the way Hamilton de�nes autocovariance:

γtj =Z ∞

�∞

Z ∞

�∞(yt � µt )

�yt�j � µt�j

�fYt ,Yt�1,...,Yt�j (yt , yt�1, ..., yt�j ) dytdyt�1...dyt�j

because

fYt ,Yt�j (yt , yt�j ) =Z ∞

�∞

Z ∞

�∞fYt ,Yt�1,...,Yt�j (yt , yt�1, ..., yt�j ) dyt�1...dyt�j+1

Note as a special case that

γt0 = Cov (Yt ,Yt�0) = V (Yt )

Helle Bunzel ISU



Autocorrelation I

Recall that the correlation between two stochastic variables is

ρ =COV (X ,Y )pV (X )V (Y )

Similarly the autocorrelation function is de�ned as:

De�nition

The j 0th autocorrelation of Yt is

ρts =γtsp

V (yt )V (yt�s )

Note that by de�nition ρt0 = 1

Helle Bunzel ISU



Stationarity IDe�nitionA process is said to be covariance-stationary or weakly stationary if

E (Yt ) = µ 8t,E [(Yt � µ) (Yt�j � µ)] = γj 8t, j

An important consequence of covariance-stationarity: Cov (Yt ,Yt�j )depends only on j , the time-distance between the two variables.

Helle Bunzel ISU



Stationarity IIDe�nitionA process is said to be strictly stationary if the joint distribution of(Yt ,Yt+j1 ,Yt+j2 , ....,Yt+jn ) depends only on j1, j2, ..., jn and not on t.

Note that a strictly stationary process is covariance-stationary. Why?

De�nitionA process is said to be Gaussian if the joint density fYt ,Yt+j1 ,Yt+j2 ,....,Yt+jn isGaussian for any choice of j1, j2, ..., jn.

Note that a covariance-stationary Gaussian process is strictlystationary. Why?

Helle Bunzel ISU



Stationarity III

Why is stationarity important?

For most series we have only one observation at each point in time.Therefore, if µt is not the same every period, we have only oneobservation to estimate it with.If all observations share the same parameter, we can potentially usethem all to estimate the parameter.How many observations do we have available to estimate γs ?

Helle Bunzel ISU



Ergodicity I

De�nitionA covariance stationary process is said to be ergodic for the mean if

plimT!∞1T

T

∑t=1yt = E (Yt )

It turns out that a su¢ cient condition for a covariance stationaryprocess to be ergodic for the mean is that

∞

∑j=0

��γj �� < ∞ (1)

Helle Bunzel ISU



Ergodicity II

De�nitionA covariance stationary process is said to be ergodic for the secondmoments if

plimT!∞1

T � jT

∑t=1(yt � µ) (yt�j � µ) = γj 8j

Su¢ cient conditions for this to hold are given in Hamilton, Chapter 7.

It turns out that if Yt is a Gaussian process, (1) is also su¢ cient forergodicity of the second moment.

Helle Bunzel ISU



Stationarity and Ergodicity, Example I

Consider the following process:

Yt = µ̃+ εt

where both µ̃ and εt are random variables which are independent ofeach other and

µ̃ � N�0,λ2

�.

Furthermore εt is a Gaussian white noise process with mean 0 andvariance σ2.

Helle Bunzel ISU



Stationarity and Ergodicity, Example II

Note that

E (Yt ) = E (µ̃) + E (εt ) = 0

and

γtj = E [YtYt�j ] = E [(µ̃+ εt ) (µ̃+ εt�j )]

= E�µ̃2 + µ̃εt + µ̃εt�j + εt εt�j

�= E µ̃2 + E (εt εt�j ) =

�λ2 if j 6= 0

λ2 + σ2 if j = 0

This means that the series is covariance stationary.

Helle Bunzel ISU



Stationarity and Ergodicity, Example III

Note however that

∞

∑j=0

��γj �� = σ2 +∞

∑j=0

λ2 = ∞

and

1T

T

∑t=1Yt =

1T

T

∑t=1(µ̃+ εt ) = µ̃+

1T

T

∑t=1

εt ! µ 6= E (Yt ) = 0

Helle Bunzel ISU



First order MA processes

In this part of the course we�ll assume that everything is stable.

MA stands for "Moving Average".

Let fεtg be a white noise process and µ and θ be constants.

The process Yt de�ned as

Yt = µ+ εt + θεt�1

is a moving average process of order 1.

Notation: MA(1)

Note that we can also write it as

Yt = µ+ (1+ θL) εt

Helle Bunzel ISU



Properties of MA(1) processes I

When looking at the properties of a timeseries process, �nding themean, variance, autocovariance and autocorrelation functions as wellas determining conditions for stationarity are the primary steps.

First the mean:

E (Yt ) = E (µ) + E ((1+ θL) εt ) = µ

Variance:

V (Yt ) = Eh(Yt � µ)2

i= E

h(µ+ εt + θεt�1 � µ)2

i= E

h(εt + θεt�1)

2i=�1+ θ2

�σ2

Helle Bunzel ISU



Properties of MA(1) processes II

Autocovariances. Start with the �rst:

γ1 = E [(Yt � µ) (Yt�1 � µ)]

= E [(εt + θεt�1) (εt�1 + θεt�2)]

= θE�ε2t�1

�= θσ2

The second:

γ2 = E [(Yt � µ) (Yt�2 � µ)]

= E [(εt + θεt�1) (εt�2 + θεt�3)] = 0

All higher autocovariances are 0 as well.

Helle Bunzel ISU



Properties of MA(1) processes IIISummary:

E (Yt ) = µ

V (Yt ) =�1+ θ2

�σ2

γj =

�θσ2 if j = 10 otherwise

Clearly an MA(1) process is covariance stationary regardless of thevalues of the parameters!

The autocorrelation:

ρ1 =γ1p

V (Yt )V (Yt�1)=

θσ2�1+ θ2

�σ2=

θ�1+ θ2

�Note the mean parameter(s) has no in�uence on the autocovarianceand autocorrelation functions.

Helle Bunzel ISU



Properties of MA(1) processes IVAlso note that di¤erent processes can share the same autocorrelation.Consider the MA(1) process

Yt = µ+ εt + ψεt�1

where ψ = 1θ .

Then

ρ1 =ψ�

1+ ψ2� = 1

θ�1+

� 1θ

�2�=

θ2 1θ

θ2�1+

� 1θ

�2� = θ�θ2 + 1

�

Helle Bunzel ISU



Properties of MA(1) processes V

Pictures of the autocorrelation functions:

Helle Bunzel ISU



Higher order MA processesAn MA(q) process is given by

Yt = µ+ εt + θ1εt�1 + θ2εt�2 + ...+ θqεt�q ,

Yt = µ+q

∑j=0

θj εt�j , θ0 = 1,

Yt = µ+q

∑j=0

θjLj εt , θ0 = 1

where fεtg is white noise and (µ, θ1, ..., θq) are any real numbers.Again look at the properties of the process:

Mean:

E (Yt ) = E (µ) + E

"q

∑j=0

θjLj εt

#= µ

Helle Bunzel ISU



Higher order MA processes

Variance:

V (Yt ) = γ0 = Eh(Yt � µ)2

i= E

h(εt + θ1εt�1 + θ2εt�2 + ...+ θqεt�q)

2i

= E

q

∑i=0

θi εt�iq

∑j=0

θj εt�j

!= E

q

∑j=0

θ2j ε2t�j

!, θ0 = 1

= σ2

1+

q

∑j=j

θ2j

!

Helle Bunzel ISU




Autocovariance:

γj = E [(Yt � µ) (Yt�j � µ)]

= E [(εt + θ1εt�1 + θ2εt�2 + ...+ θqεt�q)

� (εt�j + θ1εt�j�1 + θ2εt�j�2 + ...+ θqεt�j�q)]

First note that if j > q, γj = 0Hence assume that j � q.

Helle Bunzel ISU




Then:

γj= E [(εt + θ1εt�1 + θ2εt�2 + ...+ θqεt�q)

� (εt�j + θ1εt�j�1 + θ2εt�j�2 + ...+ θqεt�j�q)]

= E�θj ε

2t�j + θj+1θ1ε

2t�j�1 + θj+2θ2ε

2t�j�2 + ...+ θqθq�j ε

2t�q�

= (θj + θj+1θ1 + θj+2θ2 + ...+ θqθq�j ) σ2

= σ2q

∑i=j

θi θi�j , θ0 = 1

There are q � j + 1 terms in the sum.

Helle Bunzel ISU




In conclusion, we can write the autocovariance function as

γj =

�σ2 ∑q

i=j θi θi�j , θ0 = 1 if j � q0 if j > q

Again the process is covariance stationary as well as ergodic for themean.

Helle Bunzel ISU




A picture of the autocorrelation function:

Helle Bunzel ISU



MA(oo) processes I

Consider the following MA(q) process:

Yt = µ+q

∑j=0

ψj εt�j

Consider the process we get if q ! ∞ :

Yt = µ+∞

∑j=0

ψj εt�j

This process is covariance stationary the coe¢ cients are squaresummable

∞

∑j=0

ψ2j < ∞

Helle Bunzel ISU



MA(oo) processes II

Sometimes absolute summability is assumed instead:

∞

∑j=0

��ψj �� < ∞

The results are a bit harder to prove (we won�t), but they are asexpected:

E (Yt ) = µ

V (Yt ) = limq!∞

�ψ20 + ψ21 + ψ22 + ...+ ψ2q

�σ2 = σ2

∞

∑i=0

ψ2i

γj = limq!∞

σ2q

∑i=j

ψiψi�j = σ2∞

∑i=j

ψiψi�j

Helle Bunzel ISU



MA(oo) processes III

It turns out that an MA (∞) process with absolute summablecoe¢ cients also have absolutely summable autocovariances. Hencethe process is ergodic for the mean.

Finally, if the fεtg process is Gaussian, the process is ergodic for allmoments.

Helle Bunzel ISU



AR processes

An AR(p) process is de�ned as

yt = c + φ1yt�1 + φ2yt�2 + ...+ φpyt�p + εt

where εt again is white noise.

This is exactly a q�th order di¤erence equation.

Let us consider the properties of an AR(1) process.

Helle Bunzel ISU



AR(1) processes I

We are considering

yt = c + φyt�1 + εt ,(1� φL) yt = c + εt

Recall that as long as jφj < 1, we can re-write this as

yt =c

1� φ+

∞

∑j=0

φj εt�j

Assume that jφj < 1 in what follows.We have now written yt as a MA (∞) process with ψj = φj .

Helle Bunzel ISU



AR(1) processes II

When jφj < 1 we also know that

∞

∑j=0

��ψj �� = ∞

∑j=0jφjj = 1

1� jφj < ∞

such that the AR(1) process is ergodic for the mean.

The mean of the AR(1) process is:

E (yt ) = E

"c

1� φ+

∞

∑j=0

φj εt�j

#=

c1� φ

Helle Bunzel ISU



AR(1) processes III

The variance:

V (yt ) = E

"�yt �

c1� φ

�2#= E

24 ∞

∑j=0

φj εt�j

!235= E

"∞

∑j=0

φj εt�j∞

∑i=0

φi εt�i

#=

∞

∑j=0

φ2jσ2

= σ21

1� φ2

Helle Bunzel ISU



AR(1) processes IVAnd the j 0th autocovariance is:

γj = E��yt �

c1� φ

��yt�j �

c1� φ

��= E

"∞

∑i=0

φi εt�i∞

∑l=0

φl εt�j�l

#

= E

"∞

∑i=j

φi εt�i∞

∑l=0

φl εt�j�l

#

= E

"∞

∑m=0

φm+j εt�(m+j)∞

∑l=0

φl εt�j�l

#

= E

"∞

∑m=0

φm+j εt�j�m∞

∑l=0

φl εt�j�l

#

Helle Bunzel ISU



AR(1) processes V

γj = E

"∞

∑m=0

φm+j εt�j�m∞

∑l=0

φl εt�j�l

#

= E

"∞

∑l=0

φl+jφl ε2t�j�l

#= σ2φj

∞

∑l=0

φ2l

= σ2φj

1� φ2

Finally the autocorrelation is given by

ρj =σ2

φj

1�φ2

σ2 11�φ2

= φj

Helle Bunzel ISU



AR(1) processes VI

Note that this is exactly the dynamic multiplier of the di¤erenceequation.

The stable AR(1) process is clearly covariance-stationary.

Graphs of AR(1) processes:

Helle Bunzel ISU



AR(1) processes VII

Helle Bunzel ISU



AR(1) processes

High serial correlation:

Helle Bunzel ISU



AR(1) processes, alternative method I

An alternative way of �nding the moments of the process:

Assume that we are working with a stable and stationary process.

We can the use the di¤erence equation itself to determine themoments:

yt = c + φyt�1 + εt =)E (yt ) = E (c + φyt�1 + εt ) =)E (yt ) = c + φE (yt�1)

Because we assumed stationarity, we know that E (yt ) = E (yt�1) , so

E (yt ) = c + φE (yt�1) =)E (yt ) = c + φE (yt ),E (yt ) =

c1� φ

Helle Bunzel ISU



AR(1) processes, alternative method II

To �nd the second moment, use the face that µ = c1�φ to re-write

the basic equation:

yt = c + φyt�1 + εt ,yt = µ (1� φ) + φyt�1 + εt ,

(yt � µ) = φ (yt�1 � µ) + εt

Now square both sides

(yt � µ)2 = φ2 (yt�1 � µ)2 + ε2t + 2φ (yt�1 � µ) εt

Helle Bunzel ISU



AR(1) processes, alternative method III

And take expectations:

E (yt � µ)2 = φ2E (yt�1 � µ)2 + E�ε2t�+ 2φE [(yt�1 � µ) εt ]

γ0 = φ2γ0 + σ2 ,

γ0 =σ2

1� φ2

Finally, to �nd the autocovariances, we start from:

(yt � µ) = φ (yt�1 � µ) + εt

and multiply by (yt�j � µ) :

(yt � µ) (yt�j � µ) = φ (yt�1 � µ) (yt�j � µ) + (yt�j � µ) εt

Helle Bunzel ISU



AR(1) processes, alternative method IV

Taking expectations:

E [(yt � µ) (yt�j � µ)] = φE [(yt�1 � µ) (yt�j � µ)]

+E [(yt�j � µ) εt ]

γj = φγj�1 )

γj = φjγ0 =φj

1� φ2σ2

Helle Bunzel ISU



Stationarity of Time Series ProcessesAR(1) Example

Consider

yt = a0 + a1yt�1 + εt

Assume that the process started at 0 with y0. Then

yt = a0t�1∑i=0ai1 + a

t1y0 +

t�1∑i=0ai1εt�1

Find the means of yt and yt�s :

E (yt ) = a0t�1∑i=0ai1 + a

t1y0

E (yt+s ) = a0t+s�1∑i=0

ai1 + at+s1 y0

These means are clearly not identical, so the series is NOT stationary.Helle Bunzel ISU



AR(p) processes I

A p0th order autoregressive process satis�es

yt = c + φ1yt�1 + φ2yt�2 + ...+ φpyt�p + εt

If the roots of

1� φ1z � φ2z1 � ...� φpz

p = 0

lie outside the unit circle we can write this process as acovariance-stationary MA (∞) process.

Quick aside: The wording "outside the unit circle".

Helle Bunzel ISU



AR(p) processes II

We write

yt = µ+ ψ (L) εt

where

ψ (L) =�1� φ1L� φ2L

1 � ...� φpLp��1

and

∞

∑j=0

��ψj �� < ∞

Helle Bunzel ISU



AR(p) processes IIISince we know we are dealing with a stationary process, we can �ndthe moments from the equation itself, so

yt = c + φ1yt�1 + φ2yt�2 + ...+ φpyt�p + εt )µ = c + φ1µ+ φ2µ+ ...+ φpµ ,

µ =c

1� φ1 � φ2 � ...� φpAnd the autocovariances we get from:

yt = µ�1� φ1 � φ2 � ...� φp

�+ φ1yt�1 + ...+ φpyt�p + εt

(yt � µ) = φ1 (yt�1 � µ) + ...+ φp (yt�p � µ) + εt

(yt � µ) (yt�j � µ) = φ1 (yt�1 � µ) (yt�j � µ) + ...

+φp (yt�p � µ) (yt�j � µ) + (yt�j � µ) εt

Helle Bunzel ISU



AR(p) processes IV

If j = 0 :

γ0 = φ1γ�1 + ...+ φpγ�p + σ2

Because of stationarity

γ0 = φ1γ1 + ...+ φpγp + σ2

For j > 0 :

γj = φ1γj�1 + φ2γj�2 + ...+ φpγj�p

This system can be solved in a similar manner to the p0th orderdi¤erence equations.

Helle Bunzel ISU



ARMA Processes IWe can combine AR processes with MA processes to getARMA (p, q) processes:

yt = a0 +p

∑i=1aiyt�i +

q

∑i=0bi εt�i

This too can be written as an MA (∞) process if the consitions forstability of the AR part of the process are met.:

yt �p

∑i=1aiyt�i = a0 +

q

∑i=0bi εt�i ,

1�p

∑i=1aiLi

!yt = a0 +

q

∑i=0bi εt�i ,

yt =a0 +∑q

i=0 bi εt�i∏pi=1 (L� αi )

Helle Bunzel ISU



ARMA Processes II

yt =a0 +∑q


Here αi , i = 1, ..., p are the roots of

1�p

∑i=1aiLi = 0

Note that

µ = E (yt ) =a0

∏pi=1 (L� αi )

Helle Bunzel ISU



ARMA Processes III

We can continue to re-write the process as

yt =a0 +∑q


,

yt � µ =∑qi=0 bi εt�i

∏pi=1 (L� αi )

,

yt � µ =∏qi=1 (L� βi )

∏pi=1 (L� αi )

εt

Where βi , i = 1, ..., q are the roots of

1+q

∑i=1biLi = 0

Helle Bunzel ISU



ARMA Processes IV

We got

yt � µ =∏qi=1 (L� βi )

∏pi=1 (L� αi )

εt

This formulation makes it clear that we can over-parametrize theprocess. If βi = αj for some i , j , we could write the same process asan ARMA(p � 1, q � 1), which has two fewer parameters.

Helle Bunzel ISU



ARMA(1,1) Example I

ARMA(1,1) Example:

yt = a1yt�1 + εt + β1εt�1

First we need to �nd the autocovariances.

Use Yule-Walker equations:E (ytyt ) = a1E (ytyt�1) + E (yt εt ) + β1E (yt εt�1)

E (yt�1yt ) = a1E (yt�1yt�1) + E (yt�1εt ) + β1E (yt�1εt�1)

E (yt�2yt ) = a1E (yt�2yt�1) + E (yt�2εt ) + β1E (yt�2εt�1)

::

E (yt�syt ) = a1E (yt�syt�1) + E (yt�s εt ) + β1E (yt�s εt�1)

Helle Bunzel ISU



ARMA(1,1) Example IIFrom the �rst equation we get

γ0 = a1γ1 + σ2 + β1E ((a1yt�1 + εt + β1εt�1) εt�1),γ0 = a1γ1 + σ2 + β1 (a1 + β1) σ2

Second equation:

γ1 = a1E (yt�1yt�1) + E (yt�1εt ) + β1E (yt�1εt�1),γ1 = a1γ0 + β1E ((a1yt�2 + εt�1 + β1εt�2) εt�1),γ1 = a1γ0 + β1σ

2

Plug into expression for γ0 :γ0 = a1

�a1γ0 + β1σ

2�+ σ2 + β1 (a1 + β1) σ2 ,γ0�1� a21

�= a1β1σ

2 + σ2 + β1 (a1 + β1) σ2

γ0 =1+ β21 + 2a1β1(1� a21)

σ2

Helle Bunzel ISU



ARMA(1,1) Example III

Then �nd γ1 :

γ1 = a1γ0 + β1σ2 = a1

1+ β21 + 2a1β1(1� a21)

σ2 + β1σ2

=a1 + a1β21 + 2a

21β1 + β1 � β1a

21

(1� a21)σ2

=a1 + β1 + a1β21 + a

21β1

(1� a21)σ2 =

(1+ a1β1) (a1 + β1)

(1� a21)σ2

Now look at the third equation:

E (yt�2yt ) = a1E (yt�2yt�1) + E (yt�2εt ) + β1E (yt�2εt�1),γ2 = a1γ1

Helle Bunzel ISU



ARMA(1,1) Example IV

Fourth:

E (yt�3yt ) = a1E (yt�3yt�1) + E (yt�3εt ) + β1E (yt�3εt�1),γ3 = a1γ2

And s�th:

E (yt�syt ) = a1E (yt�syt�1) + E (yt�s εt ) + β1E (yt�s εt�1),γs = a1γs�1

Note that only the �rst few are hard.

Now for the autocorrelations. First note:qV (yt )V (yt�s ) =

qV (yt )V (yt ) = V (yt ) = γ0

Helle Bunzel ISU



ARMA(1,1) Example VThen calculate the �rst autocorrelation:

ρ1 =γ1p

V (yt )V (yt�1)=

γ1γ0=

(1+a1β1)(a1+β1)

(1�a21)σ2

1+β21+2a1β1(1�a21)

σ2

=(1+ a1β1) (a1 + β1)

1+ β21 + 2a1β1The second:

ρ2 =γ2p

V (yt )V (yt�2)=

γ2γ0=a1γ1γ0

= a1ρ1

The third:

ρ3 =γ3p

V (yt )V (yt�3)=

γ3γ0=a1γ2γ0

= a1ρ2 = a21ρ1

Helle Bunzel ISU



ARMA(1,1) Example VI

Similarly for all s � 2,

ρs = a1ρs�1 = as�11 ρ1

Helle Bunzel ISU


Documents

Stationary ARMA processes - Economics · De–nitions and Review MA processes AR processes ARMA Processes White noise processes I De–nition A white noise process is a sequence fε