ARMA(\(p, q\)) Processes
-
\(\{X_t\}\) is an ARMA(\(p, q\)) process if \(\{X_t\}\) is stationary solution to
\[
\phi(B)X_t = \theta(B)Z_t
\]
for every \(t\), where \(\{Z_t\} \sim WN(0, \sigma^2)\), and \(\phi(z) = 1 - \phi_1 z - \dotsm - \phi_p z^p\)
& \(\theta(z) = 1 + \theta_1 z + \dotsm + \theta_q z^q\) have no common roots.
-
\(\{X_t\}\) is an ARMA(\(p, q\)) process with mean \(\mu\) if \(\{ X_t - \mu \}\) is an ARMA(\(p, q\)) process.
Causality
-
\(\{X_t\}\) is causal if we can write
\[ X_t = \sum_{j=0}^\infty \psi_j Z_{t-j} = \psi(B)Z_t, \]
where \(\psi(z)\) \( = \sum_{j=0}^\infty \psi_j z^j = \psi_0 + \psi_1 z + \psi_2 z^2 + \dotsm \), with
\(\sum_{j=0}^\infty \mid \psi_j \mid < \infty \).
-
\(\psi_j\)'s are called the \(\psi\)-coefficients.
Determining \(\psi\)-coefficients
-
To determine \(\psi_j\)'s,
-
We start with the identity
\(
(1 - \phi_1 z - \dotsm - \phi_p z^p)(\psi_0 + \psi_1 z + \dotsm)
= 1 + \theta_1 z + \dotsm + \theta_q z^q
\)
-
Equating coefficients of \(z^j, j = 0, 1, \dotsm\), we find that
\(z^0: \psi_0 = 1\)
\(z^1: \psi_1 - \phi_1 \psi_0 = \theta_1 \Rightarrow \psi_1 = \theta_1 + \phi_1 \psi_0 = \theta_1 + \phi_1\)
-
In general,
\[
\psi_j = \theta_j + \sum_{i=1}^p \phi_i \psi_{j-i}
\]
for \(j = 0, 1, 2, \dotsm\), with convention \(\theta_0 = 1, \theta_j = 0\) for \(j > q\), and \(\psi_j = 0\) for \(j < 0\).
-
We can solve for \(\psi_0, \psi_1, \psi_2, \dotsm\) recursively.
Examples
-
AR(2): \(X_t - 0.7 X_{t-1} + 0.1 X_{t-2} = Z_t\)
(Example 3.1.2 from textbook)
The autoregressive polynomial for this process has the factorization
\( \phi(z) = (1 - 0.5z)(1 - 0.2z) \), and is therefore zero at \(z=2\) and \(z=5\).
Since these zeros lie outside the unit circle, we conclude that \(\{X_t\}\) is a causal AR(2)
process with coefficients \( \{ \psi_j \} \) given by
-
\(\psi_0 = 1\),
-
\(\psi_1 = 0.7 \),
-
\(\psi_2 = 0.7^2 - 0.1\),
-
\(\psi_j = 0.7 \psi_{j-1} - 0.1 \psi_{j-2}, j = 2, 3, \dots\).
While it is simple matter to calculate \(\psi_j\) numerivally for any \(j\), it is
possible also to give an explicit solution of these difference equations using the
theory of linear difference equations.
-
ARMA(1, 1) process: \(X_t - 0.5 X_{t-1} = Z_t + 0.4 Z_{t-1}\)
(see photo - 15:41 Oct 3)
\(\phi(z) = 1 - 0.5 z \) & \(\theta(z) = 1 + 0.4 z \)
\(
(\psi_0 + \psi_1 z + \psi_2 z^2 + \dotsm)(1-0.5z) = 1 + 0.4z
\)
\( z^0: \psi_0 = 1 \)
\( z^1: \psi_1 - 0.5\psi_0 = 0.4 \Rightarrow \psi_1 = 0.9 \)
\( z^2: \psi_2 - 0.5\psi_1 = 0 \Rightarrow \psi_2 = 0.5(0.9) = 0.45 \)
\( \vdots \)
\( z^j: \psi_j - 0.5 \psi_{j-1} = 0 \Rightarrow \psi_j = 0.5 \psi_{j-1} \text{ for } j \ge 2 \)
-
R Command: ARMAtoMA
Invertibility
-
\(\{X_t\}\) is invertible if we can write
\[
Z_t = \sum_{j=0}^\infty \pi_j X_{t-j} = \pi(B)X_t,
\]
where \(\pi(z) = \sum_{j=0}^\infty \pi_j z^j = \pi_0 + \pi_1 z + \pi_2 z^2 + \dotsm\),
with \(\sum_{j=0}^\infty \mid \pi_j \mid < \infty \).
-
\(\pi_j\)'s are called the \(\pi\)-coefficients.
Determining \(\pi\)-coefficients
-
To determine \(\pi_j\)'s,
-
We start with the identity
\(
(1 + \theta_1 z + \dotsm + \theta_q z^q)(\pi_0 + \pi_1 z + \dotsm)
= 1 - \phi_1 z - \dotsm - \phi_p z^p
\)
-
Equating coefficients of \(z^j, j = 0, 1, \dotsm\), we find that
\(z^0: \pi_0 = 1\)
\(z^1: \pi_1 + \theta_1 \pi_0 = - \phi_1 \Rightarrow \pi_1 = - \phi_1 - \theta_1 \pi_0 = - \phi_1 - \theta_1\)
\(\vdots\)
-
In general,
\[
\pi_j = - \phi_j - \sum_{i=1}^q \theta_i \pi_{j-i}
\]
for \(j = 0, 1, 2, \dotsm\), with convention \(\phi_0 = -1, \phi_j = 0\) for \(j > p\), and \(\pi_j = 0\) for \(j < 0\).
-
We can solve for \(\pi_0, \pi_1, \pi_2, \dotsm\) recursively.
Examples
- MA(1):
\( X_t = Z_t + \theta Z_{t-1} \)
-
ARMA(1, 1) process:
\(
X_t - 0.5 X_{t-1} = Z_t + 0.4 Z_{t-1}
\)
ACVF and ACF of ARMA(\(p, q\)) Processes
-
MA(\(q\)) processes:
\[
X_t = Z_t + \theta_1 Z_{t-1} + \dotsm + \theta_q Z_{t-q}
\]
-
AR(\(p\)) processes:
\[
X_t - \phi_1 X_{t-1} - \dotsm - \phi_p X_{t-p} = Z_t
\]
-
ARMA(\(p, q\)) processes:
\[
\begin{align*}
& X_t - \phi_1 X_{t-1} - \dotsm - \phi_p X_{t-p} \\
& = Z_t + \theta_1 Z_{t-1} + \dotsm + \theta_q Z_{t-q}
\end{align*}
\]
-
MA(\(q\)) is a special case of linear processes
-
ACVF is given by
\(
\gamma(h) =
\begin{cases}
\sigma^2 \sum_{j=0}^{q - \mid h \mid} \theta_j \theta_{j + \mid h \mid} & \text{ if } \mid h \mid \leq q \\
0 & \text{ if } \mid h \mid > q
\end{cases}
\)
-
ACF is given by \(\rho(h) = \frac{\gamma(h)}{\gamma(0)}\)
Examples
-
MA(2): \(X_t = Z_t + 0.3 Z_{t-1} - 0.1 Z_{t-2}\)
-
R command: ARMAacf
-
R example: ACF of some MA(\(q\)) processes
Remarks
-
For an MA(\(q\)) process, ACF is 0 for all lags \(\mid h \mid > q\).
-
Converse is also true: if the ACF of a stationary process \(\{X_t\}\) is 0
beyond some lag \(q\), then \(\{X_t\}\) is an MA(\(q\)) process.
-
This suggests that if \(\hat{\rho}(h)\) is small for \(\mid h \mid > q\), then
an MA of order \(q\) (or less) may be appropriate.
Example
Consecutive daily overshorts from an underground gasoline tank at a filling station
in Colorado.
oshorts.txt
ACVF of AR(\(p\)) Processes - Method 1
-
To calcualte ACVF of a causal AR(\(p\)) process, we can express the process as
a linear process, i.e.,
\(
X_t = \sum_{j=0}^\infty \psi_j Z_{t-j}, \{Z_t\} \sim WN(0, \sigma^2)
\)
-
Then, the ACVF is readily calculated by
\(
\gamma(h) = \sigma^2 \sum_{j=0}^\infty \psi_j \psi_{j + \mid h \mid}
\).
ACVF of AR(\(p\)) Processes - Method 2
(convenient for finding closed-form solution)
-
Obtain a homogeneous difference equation
\( \gamma(h) - \phi_1 \gamma(h-1) - \dotsm - \phi_p \gamma(h - p) = 0 \), \(h \ge p\)
-
Let \(\xi_1, \dotsm, \xi_k\) denote the distinct roots of \(\phi(z)\), each with multiplicity \(r_1, \dotsm, r_k\).
\( r_1 + \dotsm + r_k = p\)
-
Then, \(\gamma(h) = \xi_1^{-h}P_1(h) + \dotsm + \xi_k^{-h}P_k(h)\) for \(h \ge 0\),
where \(P_i(h)\) is a polynomial in \(h\) of degree \(r_i - 1\).
-
So, \(\gamma(h)\) decays exponentially as \(h \rightarrow \infty\).
(short-memory processes)
-
\(P_i(h), i = 1, \dotsm, k\), are determined uniquely from initial condictions:
-
\(\gamma(0) - \phi_1 \gamma(1) - \dotsm - \phi_p \gamma(p) = \sigma^2\)
-
\(\gamma(h) - \phi_1 \gamma(h-1) - \dotsm - \phi_p \gamma(h-p) = 0, 1 \le h < p\)
-
Examples:
-
AR(2): \(X_t - 0.7X_{t-1} + 0.1 X_{t-2} = Z_t\),
\(\{Z_t\} \sim WN(0, \sigma^2)\)
\( \phi(z) = 1 - 0.7 z + 0.1 z^2 = (1-0.2z)(1-0.5z) \)
So, roots are \(\xi_1 = 5\) and \(\xi_2 = 2\)
The general solution is \(\gamma(h) = \xi_1^{-h}P_1(h) + \xi_2^{-h} P_2(h)\) for \(h \ge 0\)
\( \gamma(h) = 5^{-h} \beta_{10} + 2^{-h} \beta_{20} \), where \(\beta_{10} \) and \(\beta_{20}\) are determined by
\(
\begin{cases}
\gamma(0) - 0.7 \gamma(1) + 0.1 \gamma(2) = \sigma^2 \\
\gamma(1) - 0.7 \gamma(0) + 0.1 \gamma(1) = 0
\end{cases}
\)
note that \(\gamma(0) = \beta_{10} + \beta_{20}\),
\(\gamma(1) = \frac{\beta_{10}}{5} + \frac{\beta_{20}}{2}\),
and \(\gamma(2) = \frac{\beta_{10}}{25} + \frac{\beta_{20}}{4}\)
We then have
\(
\begin{cases}
0.864 \beta_{10} + 0.675 \beta_{20} = \sigma^2 \\
-0.48 \beta{10} - 0.15 \beta_{20} = 0
\end{cases}
\)
\(\Rightarrow \beta_{10} = - 0.7716 \sigma^2 \) & \(\beta_{20} = 2.4691 \sigma^2 \)
Therefore, \(\gamma(h) = -0.7716 \sigma^2 5^{-h} + 2.4691 \sigma^2 2^{-h}\) for \(h \ge 0\).
-
R Example: ACF of some AR(2) processes
(see photo - 15:45 Oct 8)
ACVF of AR(\(p\)) Processes - Method 3
(convenient for finding numerical solution)
-
We obtain Yule-Walker equations:
-
\(\gamma(0) - \phi_1 \gamma(1) - \dotsm - \phi_p \gamma(p) = \sigma^2\)
-
\(\gamma(h) - \phi_1 \gamma(h-1) - \dotsm - \phi_p \gamma(h-p) = 0, h \ge 1\)
-
(1) Solve the first \(p + 1\) equations for \(\gamma(0), \dots, \gamma(p)\)
-
(2) Then, use the subsequent equations to solve for \(\gamma(p + 1), \dots\), successively.
-
Example:
-
AR(2): \(X_t - 0.7 X_{t-1} + 0.1 X_{t-2} = Z_t\),
\(\{Z_t\} \sim WN(0, \sigma^2)\)
(1) Find \(\gamma(0)\), \(\gamma(1)\), and \(\gamma(2)\) using
\(
\begin{cases}
\gamma(0) - 0.7 \gamma(1) + 0.1 \gamma(2) = \sigma^2 \\
\gamma(1) - 0.7 \gamma(0) + 0.1 \gamma(1) = 0 \\
\gamma(2) - 0.7 \gamma(1) + 0.1 \gamma(0) = 0
\end{cases}
\)
(2) Find \(\gamma(h), h \ge 3 \) using \(\gamma(h) = 0.7 \gamma(h-1) - 0.1 \gamma(h-2)\)
ACVF of ARMA(\(p, q\)) Processes - Method 1
-
To calculate ACVF of a causal ARMA(\(p, q\)) process, we can express the process as
a linear process, i.e.,
\(
X_t = \sum_{j=0}^\infty \psi_j Z_{t-j}, \{Z_t\} \sim WN(0, \sigma^2)
\)
-
Then, the ACVF is readily calcualted by
\(
\gamma(h) = \sigma^2 \sum_{j=0}^\infty \psi_j \psi_{j + \mid h \mid}
\).
ACVF of ARMA(\(p, q\)) Processes - Method 2
-
Obtain a homogeneous difference equation
\( \gamma(h) - \phi_1 \gamma(h-1) - \dotsm - \phi_p \gamma(h-p) = 0, h \ge m \)
where \(m = \max(p, q+1)\).
-
Let \(\xi_1, \dotsm, \xi_k\) denote the distinct roots of \(\phi(z)\), each with multiplicity
\(r_1, \dotsm, r_k\).
-
\(r_1 + \dotsm + r_k = p\)
-
Then, \(\gamma(h) = \xi_1^{-h}P_1(h) + \dotsm + \xi_k^{-h}P_k(h)\) for \(h \ge m-p\),
where \(P_i(h)\) is a polynomial in \(h\) of degree \(r_i - 1\).
-
So, \(\gamma(h)\) decays exponentially as \(h \rightarrow \infty\).
-
\(P_i(h), i = 1, \dotsm, k\), and \(\gamma(j), 0 \leq j < m-p\), are
determined uniquely from initial conditions:
\[
\begin{align*}
& \gamma(h) - \phi_1 \gamma(h-1) - \dotsm - \phi_p \gamma(h-p) \\
& = \sigma^2 \sum_{j=0}^{q-h} \theta_{h+j} \psi_j, 0 \leq h < m
\end{align*}
\]
-
Example:
ARMA(1, 1): \(X_t - 0.5X_{t-1} = Z_t + 0.4 Z_{t-1}\), \(\{Z_t\} \sim WN(0, \sigma^2)\)
(easy to check it's causal and invertible)
\(p = 1, q = 1, \text{ and } m = 2\)
(1) \( \phi(z) = 1 - 0.5z \Rightarrow \text{ root } \xi = 2 \) with \(r = 1\)
(2) General solution: \(\gamma(h) = 2^{-h} \cdot \beta_{10}\) for \(h \ge m-p =1\)
where \(\gamma(0)\) and \(\beta_{10}\) are determined by
\(\gamma(0) - 0.5 \gamma(1) = 1.36 \sigma^2\)
\(\gamma(1) - 0.5 \gamma(0) = 0.4 \sigma^2\)
or
\(\gamma(0) - 0.25 \beta_{10} = 1.36 \sigma^2\)
\(0.5 \beta_{10} - 0.5 \gamma(0) = 0.4 \sigma^2\)
We have \(\beta_{10} = 2.88 \sigma^2\) and \(\gamma(0) = 2.08 \sigma^2\)
Thus, \(\gamma(0) = 2.08 \sigma^2\) and \(\gamma(h) = 2.88 \sigma^2 \cdot 2^{-h}\) for \(h \ge1\).
Note:
rhs \(= \sigma^2 \sum_{j=0}^1 \theta_j \psi_j = \sigma^2(\psi_0 + \theta_1 \psi_1)\) (when \(h = 0\)) (\(\theta_1 = 0.4\), \(\psi_1 = 0.9\), \(\psi_0 = 1\))
rhs \(= \sigma^2 \theta_1\) (when \(h=1)\)
ACVF of ARMA(\(p, q\)) Processes - Method 3
-
Numerical determination of ACVF from equations
-
\( \gamma(h) - \phi_1 \gamma(h-1) - \dotsm - \phi_p \gamma(h-p) \)
\( = \sigma^2 \sum_{j=0}^{q-h} \theta_{h+j} \psi_j, \text{ for } 0 \leq h < m \)
-
\( \gamma(h) - \phi_1 \gamma(h-1) - \dotsm - \phi_p \gamma(h-p) = 0, \text{ for } h \ge m = \max(p, q+1)\)
can be carried out by
-
finding \(\gamma(0), \dotsm, \gamma(p)\) from the first \(p + 1\) equations
-
finding \(\gamma(p+1), \dotsm \), successively from subsequent equations.
-
Example:
ARMA(1, 1):
\( X_t - \phi X_{t-1} = Z_t + \theta Z_{t-1}, \mid \phi \mid < 1 \) and
\(\{Z_t\} \sim WN(0, \sigma^2) \)
(general solution in textbook)
Here consider
ARMA(1, 1): \(X_t - 0.5X_{t-1} = Z_t + 0.4 Z_{t-1}\), \(\{Z_t\} \sim WN(0, \sigma^2)\)
(1) Find \(\gamma(0)\) and \(\gamma(1)\) using
\(\gamma(0) - 0.5 \gamma(1) = 1.36 \sigma^2\)
\(\gamma(1) - 0.5 \gamma(0) = 0.4 \sigma^2\)
\(\Rightarrow \gamma(0) = \dotsm\) and \(\gamma(1) = \dotsm\).
(2) Find \(\gamma(h), h \ge 2\) using
\(\gamma(h) - 0.5 \gamma(h-1) = 0 \Rightarrow \gamma(h) = 0.5 \gamma(h-1)\)
We then have \(\gamma(h) = 0.5^{h-1} \gamma(1)\)
Characteristics of ACVF/ACF of MA, AR, & ARMA
-
MA(\(q\)): Cuts off at lag \(q\);
i.e., \(= 0\) for \(h > q\).
-
AR(\(p\)): Decreases geometrically as \(h\) increases.
-
ARMA(\(p ,q\)): Decreases geometrically as \(h\) increases.
Partial Autocorrelation Function (PACF) of ARMA(\(p, q\)) Processes
-
MA(\(q\)) model can be identified from its ACF: non-zero up to lag \(q\)
and zero afterwards.
-
We need a similar tool for identifying AR(\(p\)) model;
this role can be filled by PACF.
PACF
-
The PACF of a stationary time series
\(\{X_t\}\) at time lag \(h\) denoted as \(\alpha(h)\),
is defined as
\[
\begin{align*}
\alpha(0) & = 1 \\
\alpha(h) & = \phi_{hh}, h \ge 1,
\end{align*}
\]
where \(\phi_{hh}\) is the last component of \(\boldsymbol{\phi}_h = \boldsymbol{\Gamma}_h^{-1} \boldsymbol{\gamma}_h\),
with \(\boldsymbol{\phi}_h = (\phi_{h1}, \dotsm, \phi_{hh})'\),
\(\boldsymbol{\Gamma}_h = [\gamma(i - j)]_{i,j = 1}^h\), and
\( \boldsymbol{\gamma}_h = (\gamma(1), \dotsm, \gamma(h))' \).
-
From Chapter 2, the one-step BLP
\( P_h X_{h+1} = \mu + \phi_{h1}(X_h - \mu) + \dotsm + \phi_{hh}(X_1 - \mu) \)
where \(\boldsymbol{\phi}_h\) is determined by \(\boldsymbol{\Gamma}_h \boldsymbol{\phi}_h = \boldsymbol{\gamma}_h\).
-
So, \(\alpha(h)\) is equal to the coefficient associated with \(X_1\) in the expression of \(P_h X_{h+1}\).
-
Consequently, \(\alpha(h)\) can be computed recursively using Durbin-Levinson algorithm.
Examples
-
AR(\(p\)) process:
\(X_t = \phi_1 X_{t-1} + \dotsm + \phi_p X_{t-p} + Z_t \),
\(\{Z_t\} \sim WN(0, \sigma^2)\)
From Chapter 2,
\( P_n X_{n+1} = \phi_1 X_n + \dotsm + \phi_p X_{n+1-p} \) for \(n \ge p\)
-
For \(h = p\),
\(P_h X_{n+1} = \phi_1 X_h + \dotsm + \phi_p X_{h+1-p} = \phi_1 X_p + \dotsm + \phi_p X_1\)
Then, \(\alpha(p) = \phi_p\)
-
For \(h > p\),
\(P_h X_{n+1} = \phi_1 X_h + \dotsm + \phi_p X_{h+1-p} + \dotsm + 0 \cdot X_1\)
That is \(\alpha(h) = 0\).
-
For \(h < p\),
\(\alpha(h) \) can be found from definition.
-
MA(1) process:
\(X_t = Z_t + \theta Z_{t-1}, \mid \theta \mid < 1\) &
\(\{Z_t\} \sim WN(0, \sigma^2)\)
(Problem 3.12, see photo - 15:39 Oct 10)
PACF for MA(1) decays exponentially. So do PACF's for MA(\(q\)) and ARMA(\(p, q\)).
R Examples
-
R command: ARMAacf with option 'pacf = TRUE'
-
PACF of some AR(\(p\)) processes
-
PACF of some MA(\(q\)) processes
Characteristics of ACF & PACF of MA, AR, & ARMA
-
ACF:
-
MA(\(q\)): Cuts off at lag \(q\); i.e., \(= 0\) for \(h > q\).
-
AR(\(p\)): Decreases geometrically as \(h\) increases.
-
ARMA(\(p, q\)): Decreases geometrically as \(h\) increases.
-
PACF:
-
MA(\(q\)): Decreases geometrically as \(h\) increases.
-
AR(\(p\)): Cuts off at lag \(p\); i.e., \(= 0\) for \(h > p\).
-
ARMA(\(p, q\)): Decreases geometrically as \(h\) increases.
Sample PACF
-
Given observations \(x_1, \dotsm, x_n\)
-
Suppose \(x_i \ne x_j\) for some \( i \ne j \).
-
The sample PACF at time lag \(h\),
denoted as \(\hat{\alpha}(h)\), is defined as
\[
\begin{align*}
\hat{\alpha}(0) & = 1 \\
\hat{\alpha}(h) & = \hat{\phi}_{hh}, 1 \leq h < n,
\end{align*}
\]
where \(\hat{\phi}_{hh}\) is the last component of \(\hat{\boldsymbol{\phi}}_h = \hat{\boldsymbol{\Gamma}}_h^{-1} \hat{\boldsymbol{\gamma}}_h \),
with \(\hat{\boldsymbol{\phi}}_h = (\hat{\phi}_{h1}, \dotsm, \hat{\phi}_{hh})'\),
\( \hat{\boldsymbol{\Gamma}}_h = [\hat{\gamma}(i - j)]_{i, j = 1}^h \), and
\( \boldsymbol{\gamma}_h = (\hat{\gamma}(1), \dotsm, \hat{\gamma}(h) )' \).
Remarks
-
\(\hat{\alpha}(h)\) is expected to reflect the properties of \(\alpha(h)\).
-
\(\hat{\alpha}(h)\) can be computed recursively using Durbin-Levinson algorithm.
Sample PACF of AR(\(p\)) Processes
-
If \(X_1, \dotsm, X_n\) is a realization of an AR(\(p\)) process \(\{X_t\}\)
with \(\{Z_t\} \sim \text{ iid } (0, \sigma^2)\),
-
then, for large \(n\), \(\hat{\alpha}(\cdot)\)'s at lag greater than \(p\) are
approximately independent \(\mathcal{N}(0, 1/n)\) random variables.
-
So, a simple method to identify AR order is:
-
If \(\mid \hat{\alpha}(k) \mid \leq 1.96 / \sqrt{n}\) for \(k > p\),
then, an AR of order \(p\) (or less) may be appropriate.
R Example
-
R command: acf with option 'type = partial'
-
The sunspot numbers for the years 1770-1869;
sunspot.txt
Forecasting ARMA(\(p, q\)) Processes
-
Consider a causal ARMA(\(p, q\)) process
\( \phi(B) X_t = \theta(B) Z_t, \{Z_t\} \sim WN(0, \sigma^2) \).
-
We assume \(p \ge 1\) and \(q \ge 1\) without loss of generality.
One-Step Prediction of ARMA(\(p, q\))
-
Let \(m = \max(p, q)\)
-
Define
\(
W_t =
\begin{cases}
\sigma^{-1} X_t & t = 1, \dotsm, m \\
\sigma^{-1} \phi(B) X_t & t > m
\end{cases}
\)
-
We apply the innovations algorithm to \(\{W_t\}\), the transformed process.
-
By doing so, a drastic simplification may be achieved in the calculations.
-
Applying the innovations algorithm to \(\{W_t\}\) yields
\(
\hat{W}_{n+1} =
\begin{cases}
\sum_{j=1}^n \theta_{nj}(W_{n+1-j} - \hat{W}_{n+1-j}) & 1 \leq n < m \\
\sum_{j=1}^q \theta_{nj}(W_{n+1-j} - \hat{W}_{n+1-j}) & n \ge m
\end{cases}
\)
where \(\theta_{nj}\) and \(r_n\)\( = \mathbb{E}(W_{n+1} - \hat{W}_{n+1})^2 \)
are found recursively, where \(\hat{W}_1 = 0\) and \(\hat{W}_k = P_{k-1}W_k\) for \(k > 1\).
-
Finally, we find \(\hat{X}_{n+1} = P_n X_{n+1} \) from \(\hat{W}_{n+1}\) as follows:
-
\( \hat{X}_{n+1} = \sum_{j=1}^n \theta_{nj} (X_{n+1-j} - \hat{X}_{n+1-j}), 1 \leq n < m \)
-
\( \hat{X}_{n+1} = \phi_1 X_n + \dotsm + \phi_p X_{n+1-p} + \sum_{j=1}^q \theta_{nj} (X_{n+1-j} - \hat{X}_{n+1-j}), n \ge m \)
-
And \(\mathbb{E}(X_{n+1} - \hat{X}_{n+1})^2 = \sigma^2 r_n \)
-
It can be shown that \(X_t - \hat{X}_t = \sigma(W_t - \hat{W}_t) \) for all \(t \ge 1\).
-
The one-step predictors \(\hat{X}_2, \hat{X}_3, \dotsm\) are determined recursively.
Remarks
-
\(\theta_{nj}\) vanishes when \(n \ge m\) and \(j > q\).
-
\(\theta_{nj}\) and \(r_n\) depend only on \(\phi_1, \dotsm, \phi_p\) and
\(\theta_1, \dotsm, \theta_q\) but not on \(\sigma^2\).
-
If \(\{X_t\}\) is invertible, then \(\theta_{nj} \rightarrow \theta_j, j = 1, \dotsm, q\),
and \(r_n \rightarrow 1\) as \(n \rightarrow \infty\).
Examples
-
Prediction of AR(\(p\)) process: for \(n \ge p\),
\( P_n X_{n+1} = \phi_1 X_n + \dotsm + \phi_p X_{n+1-p} \)
-
Prediction of MA(\(q\)) process: for \( n \ge 1\),
\( \hat{X}_{n+1} = \sum_{j=1}^{\min(n,q)} \theta_{nj} (X_{n+1-j} - \hat{X}_{n+1-j}) \)
where \(\theta_{nj}\)'s are found by applying innovations algorithm to \(\{W_t\}\).
-
In this case, \(W_t = \sigma^{-1}X_t\) for all \(t\).
-
Prediction of ARMA(1, 1) process,
\( X_t - \phi X_{t-1} = Z_t + \theta Z_{t-1} \)
where \(\{Z_t\} \sim WN(0, \sigma^2)\) and \(\mid \phi \mid < 1\):
\( \hat{X}_{n+1} = \phi X_n + \theta_{n1}(X_n - \hat{X}_n) \)
for \(n \ge 1\), where \(\theta_{n1}\) is found by applying innovations algorithm to \(\{W_t\}\).
-
In this case,
\(r_0 = \frac{1 + 2 \theta \phi + \theta^2}{1 - \phi^2}\);
\(\theta_{n1} = \frac{\theta}{r_{n-1}}\);
\( r_n = 1 + \theta^2 - \frac{\theta^2}{r_{n-1}}\).
