Introduction to Time Series Analysis. Lecture 8

Introduction to Time Series Analysis. Lecture 8.

1. Review: Linear prediction, projection in Hilbert space.

2. Forecasting and backcasting.

3. Prediction operator.

4. Partial autocorrelation function.

1

Linear prediction

GivenX1, X2, . . . , Xn, the best linear predictor

Xnn+m = α0 +

n∑

i=1

αiXi

of Xn+m satisfies theprediction equations

E(Xn+m − Xn

n+m

)= 0

E[(

Xn+m − Xnn+m

)Xi

]= 0 for i = 1, . . . , n.

This is a special case of theprojection theorem.

2

Projection theorem

If H is a Hilbert space,

M is a closed subspace ofH,

andy ∈ H,

then there is a pointPy ∈ M

(theprojection of y on M)

satisfying

1. ‖Py − y‖ ≤ ‖w − y‖

2. 〈y − Py, w〉 = 0

for w ∈ M.

y

y−Py

Py

M

3

Projection theorem for linear forecasting

Given1, X1, X2, . . . , Xn ∈{

r.v.sX : EX2 < ∞}

,

chooseα0, α1, . . . , αn ∈ R

so thatZ = α0 +∑n

i=1 αiXi minimizes E(Xn+m − Z)2.

Here,〈X, Y 〉 = E(XY ),

M = {Z = α0 +∑n

i=1 αiXi : αi ∈ R} = s̄p{1, X1, . . . , Xn}, and

y = Xn+m.

4

Projection theorem: Linear prediction

Let Xnn+m denote the best linear predictor:

‖Xnn+m − Xn+m‖2 ≤ ‖Z − Xn+m‖2 for all Z ∈ M.

The projection theorem implies the orthogonality

〈Xnn+m − Xn+m, Z〉 = 0 for all Z ∈ M

⇔ 〈Xnn+m − Xn+m, Z〉 = 0 for all Z ∈ {1, X1, . . . , Xn}

⇔E(Xn

n+m − Xn+m

)= 0

E[(

Xnn+m − Xn+m

)Xi

]= 0

5

Linear prediction

That is, theprediction errors (Xnn+m − Xn+m) are orthogonal to the

prediction variables (1, X1, . . . , Xn).

Orthogonality of prediction error and1 implies we can subtractµ from all

variables (Xnn+m andXi). Thus, for forecasting, we can assumeµ = 0.

6

One-step-ahead linear prediction

Write Xnn+1 = φn1Xn + φn2Xn−1+ · · · + φnnX1

Prediction equations: E((Xn

n+1 − Xn+1)Xi

)= 0, for i = 1, . . . , n

⇔n∑

j=1

φnjE(Xn+1−jXi) = E(Xn+1Xi)

⇔n∑

j=1

φnjγ(i − j) = γ(i)

⇔ Γnφn = γn,

7

One-step-ahead linear prediction

Prediction equations: Γnφn = γn.

Γn =

γ(0) γ(1) · · · γ(n − 1)

γ(1) γ(0) γ(n − 2)...

. .....

γ(n − 1) γ(n − 2) · · · γ(0)

,

φn = (φn1, φn2, . . . , φnn)′, γn = (γ(1), γ(2), . . . , γ(n))′.

8

Mean squared error of one-step-ahead linear prediction

P nn+1 = E

(Xn+1 − Xn

n+1

)2

= E((

Xn+1 − Xnn+1

) (Xn+1 − Xn

n+1

))

= E(Xn+1

(Xn+1 − Xn

n+1

))

= γ(0) − E(φ′

nXXn+1)

= γ(0) − γ′

nΓ−1n γn,

whereX = (Xn, Xn−1, . . . , X1)′.

9

Mean squared error of one-step-ahead linear prediction

Variance is reduced:

P nn+1 = E

(Xn+1 − Xn

n+1

)2

= γ(0) − γ′

nΓ−1n γn

= Var(Xn+1) − Cov(Xn+1, X)Cov(X, X)−1Cov(X, Xn+1)

= E(Xn+1 − 0)2− Cov(Xn+1, X)Cov(X, X)−1Cov(X, Xn+1),

whereX = (Xn, Xn−1, . . . , X1)′.

10






11

Backcasting: Predicting m steps in the past

GivenX1, . . . , Xn, we wish to predictX1−m for m > 0.

That is, we chooseZ ∈ M = s̄p{X1, . . . , Xn} to minimize‖Z −X1−m‖2.

The prediction equations are

〈Xn1−m − X1−m, Z〉 = 0 for all Z ∈ M

⇔ E((

Xn1−m − X1−m

)Xi

)= 0 for i = 1, . . . , n.

12

One-step backcasting

Write the least squares prediction ofX0 givenX1, . . . , Xn as

Xn0 = φn1X1 + φn2X2 + · · · + φnnXn = φ′

nX,

where the predictor vector is reversed: nowX = (X1, . . . , Xn)′.The prediction equations are

E((Xn0 − X0) Xi) = 0 for i = 1, . . . , n

⇔ E

n∑

j=1

φnjXj − X0

Xi

= 0

⇔

n∑

j=1

φnjγ(j − i) = γ(i)

⇔ Γnφn = γn.

13

One-step backcasting

The prediction equations are

Γnφn = γn,

which is exactly the same as for forecasting, but with the indices of the

predictor vector reversed:X = (X1, . . . , Xn)′ versusX = (Xn, . . . , X1)′.

14

Example: Forecasting AR(1)

AR(1) model: Xt = φ1Xt−1 + Wt

linear prediction ofX2: X12 = φ11X1

Prediction equation: γ(0)φ11 = γ(1)

= Cov(X0, X1)

= φ1γ(0)

⇔ φ11 = φ1.

15

Example: Backcasting AR(1)


linear prediction ofX0: X10 = φ11X1

Prediction equation: γ(0)φ11 = γ(1)

= Cov(X0, X1)

= φ1γ(0)

⇔ φ11 = φ1.

16






17

The prediction operator

For random variablesY, Z1, . . . , Zn, define the

best linear prediction of Y given Z = (Z1, . . . , Zn)′

as the operatorP (·|Z) applied toY :

P (Y |Z) = µY + φ′(Z − µZ)

with Γφ = γ,

where γ = Cov(Y, Z)

Γ = Cov(Z, Z).

18

Properties of the prediction operator

1. E(Y − P (Y |Z)) = 0, E((Y − P (Y |Z))Z) = 0.

2. E((Y − P (Y |Z))2) = Var(Y ) − φ′γ.

3. P (α1Y1 + α2Y2 + α0|Z) = α0 + α1P (Y1|Z) + α2P (Y2|Z).

4. P (Zi|Z) = Zi.

5. P (Y |Z) = EY if γ = 0.

19

Example: predicting m steps ahead

Write Xnn+m = φ

(m)n1 Xn + φ

(m)n2 Xn−1 + · · · + φ(m)

nn X1

Γnφ(m)n = γ(m)

n ,

with Γn = Cov(X, X),

γ(m)n = Cov(Xn+m, X)

= (γ(m), γ(m + 1), . . . , γ(m + n − 1))′.

Also, E((Xn+m − Xnn+m)2) = γ(0) − φ(m)′γ(m)

n .

20






21

Partial autocovariance function


γ(1) = Cov(X0, X1) = φ1γ(0)

γ(2) = Cov(X0, X2)

= Cov(X0, φ1X1 + W2)

= Cov(X0, φ21X0 + φ1W1 + W2)

= φ21γ(0).

Clearly,X0 andX2 are correlated throughX1.

In the PACF, we remove this dependence by considering the covariance of

theprediction errorsof X12 andX1

0 .

22

Partial autocovariance function

For AR(1) model: X12 = φ1X1,

X10 = φ1X1,

so Cov(X12 − X2, X

10 − X0) = Cov(φ1X1 − X2, φ1X1 − X0)

= Cov(W2, φ1X1 − X0)

= 0.

23

Partial autocorrelation function

The Partial AutoCorrelation Function (PACF) of a stationary

time series{Xt} is

φ11 = Corr(X1, X0) = ρ(1)

φhh = Corr(Xh − Xh−1h , X0 − Xh−1

0 ) for h = 2, 3, . . .

This removes the linear effects ofX1, . . . , Xh−1:

. . . , X−1, X0, X1, X2, . . . , Xh−1︸︷︷︸

partial out

, Xh, Xh+1, . . .

24

Partial autocorrelation function

The PACFφhh is also the last coefficient in the best linear prediction of

Xh+1 givenX1, . . . , Xh:

Γhφh = γh Xhh+1 = φ′

hX

φh = (φh1, φh2, . . . , φhh).

25

Example: Forecasting an AR(p)

For Xt =

p∑

i=1

φiXt−i + Wt,

Xnn+1 = P (Xn+1|X1, . . . , Xn)

= P

(p∑

i=1

φiXn+1−i + Wn+1|X1, . . . , Xn

)

=

p∑

i=1

φiP (Xn+1−i|X1, . . . , Xn)

=

p∑

i=1

φiXn+1−i for n ≥ p.

26

Example: PACF of an AR(p)

For Xt =

p∑

i=1

φiXt−i + Wt,

Xnn+1 =

p∑

i=1

φiXn+1−i.

Thus,φhh =

φh if 1 ≤ h ≤ p

0 otherwise.

27

Example: PACF of an invertible MA(q)

For Xt =

q∑

i=1

θiWt−i + Wt, Xt = −∞∑

i=1

πiXt−i + Wt,

Xnn+1 = P (Xn+1|X1, . . . , Xn)

= P

(

−

∞∑

i=1

πiXn+1−i + Wn+1|X1, . . . , Xn

)

= −

∞∑

i=1

πiP (Xn+1−i|X1, . . . , Xn)

= −

n∑

i=1

πiXn+1−i −

∞∑

i=n+1

πiP (Xn+1−i|X1, . . . , Xn) .

In general,φhh 6= 0.

28

ACF of the MA(1) process

−10 −8 −6 −4 −2 0 2 4 6 8 100

0.2

0.4

0.6

0.8

1

θ/(1+θ2)

MA(1): Xt = Z

t + θ Z

t−1

29

ACF of the AR(1) process

−10 −8 −6 −4 −2 0 2 4 6 8 100

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

φ|h|

AR(1): Xt = φ X

t−1 + Z

t

30

PACF of the MA(1) process

0 1 2 3 4 5 6 7 8 9 10

−0.2

0

0.2

0.4

0.6

0.8

1

MA(1): Xt = Z

t + θ Z

t−1

31

PACF of the AR(1) process

0 1 2 3 4 5 6 7 8 9 10

0

0.2

0.4

0.6

0.8

1

AR(1): Xt = φ X

t−1 + Z

t

32

PACF and ACF

Model: ACF: PACF:

AR(p) decays zero forh > p

MA(q) zero forh > q decays

ARMA(p,q) decays decays

33

Sample PACF

For a realizationx1, . . . , xn of a time series,

thesample PACF is defined by

φ̂00 = 1

φ̂hh = last component of̂φh,

whereφ̂h = Γ̂−1h γ̂h.

34






35

Documents

Introduction to Time Series Analysis. Lecture 8