Hanwen Ju | Notes on Prediction Error

Back to Blogs 2026.07.01

Notes on Prediction Error

These are my notes after reading Chapter 7 of The Elements of Statistical Learning. The main question is simple: after fitting a model, how should we think about the error it will make on data that was not used to train it?

1. Several Errors That Sound Similar

The most immediate quantity is the training error:

\[ \overline{err} = \frac{1}{N} \sum_{i=1}^{N} L(y_i, \hat f(x_i)). \]

This is easy to compute, but it is usually too optimistic. The model has already seen these responses, so a flexible model can reduce this number by fitting not only the signal but also part of the noise.

For a fixed training set \(\mathcal{T}\), the conditional test error is the expected loss on a new independent observation:

\[ Err_{\mathcal{T}} = \mathbb{E}_{X,Y} \left[ L(Y, \hat f(X)) \mid \mathcal{T} \right]. \]

If we also average over possible training sets, we get the expected test error:

\[ Err = \mathbb{E}_{\mathcal{T}} \left[ Err_{\mathcal{T}} \right] = \mathbb{E}_{X,Y,\mathcal{T}} \left[ L(Y,\hat f(X)) \right]. \]

ESL also discusses in-sample error. Here the input points \(x_i\) are fixed at the observed training inputs, but the responses are imagined as newly sampled:

\[ Err_{in} = \frac{1}{N} \sum_{i=1}^{N} \mathbb{E}_{Y_i^0} \left[ L(Y_i^0, \hat f(x_i)) \mid \mathcal{T} \right]. \]

This separates two issues: evaluating on the same input locations, but not reusing the same noisy labels. It is useful for understanding why training error is biased downward.

2. Optimism

Optimism is the gap between in-sample error and training error:

\[ op = Err_{in} - \overline{err}. \]

A correction to my original notes: optimism is usually positive in expectation, not negative. The training error is called optimistic because it is smaller than the error we expect on newly sampled responses.

Under squared-error loss, fixing the predictors and averaging over the responses gives the expected optimism:

\[ \omega = \mathbb{E}_{\mathbf{y}}(op) = \frac{2}{N} \sum_{i=1}^{N} \operatorname{Cov}(\hat y_i, y_i). \]

This formula is a compact way to say that overfitting is related to how strongly the fitted values move with the observed noisy responses. If \(\hat y_i\) changes a lot when \(y_i\) changes, then the model is using the training labels very aggressively.

Suppose the data follow:

\[ y_i = f(x_i) + \varepsilon_i, \quad \varepsilon_i \overset{i.i.d.}{\sim} \mathcal{N}(0,\sigma^2). \]

The goal is to learn \(f\), not the realized noise \(\varepsilon_i\). When a model is too flexible, it may behave like:

\[ \hat f(x_i) \approx y_i = f(x_i) + \varepsilon_i. \]

This makes the training error look small, but the fitted function has absorbed random noise that will not repeat in future samples.

3. Effective Degrees of Freedom

For many linear fitting methods, the fitted values can be written as:

\[ \hat{\mathbf{y}} = S\mathbf{y}, \]

where \(S\) is a smoother matrix. In this case:

\[ \sum_{i=1}^{N} \operatorname{Cov}(\hat y_i, y_i) = \sigma^2 \operatorname{tr}(S). \]

The quantity \(\operatorname{tr}(S)\) is called the effective degrees of freedom. Therefore:

\[ \mathbb{E}_{\mathbf{y}}(Err_{in}) = \mathbb{E}_{\mathbf{y}}(\overline{err}) + \frac{2\sigma^2}{N} \operatorname{tr}(S). \]

For ordinary least squares with \(d\) parameters, \(\operatorname{tr}(S)=d\), giving the familiar correction:

\[ \omega = \frac{2d\sigma^2}{N}. \]

For ridge regression,

\[ \hat{\mathbf{y}} = X(X^TX+\lambda I)^{-1}X^T\mathbf{y}. \]

Its effective degrees of freedom are:

\[ df(\lambda) = \operatorname{tr} \left[ X(X^TX+\lambda I)^{-1}X^T \right] = \sum_{j} \frac{\eta_j}{\eta_j+\lambda}, \]

where \(\eta_j\) are the eigenvalues of \(X^TX\). As \(\lambda\) increases, \(df(\lambda)\) decreases, so the model becomes less sensitive to individual noisy responses.

4. Bias-Variance Decomposition

For regression with squared error, assume:

\[ Y = f(X) + \varepsilon, \quad \mathbb{E}[\varepsilon]=0, \quad \operatorname{Var}(\varepsilon)=\sigma^2. \]

At a fixed point \(x_0\), averaging over possible training sets and response noise:

\[ Err(x_0) = \mathbb{E} \left[ (Y-\hat f(x_0))^2 \mid X=x_0 \right]. \]

This decomposes into:

\[ Err(x_0) = \underbrace{\sigma^2}_{\text{irreducible error}} + \underbrace{ \left( \mathbb{E}[\hat f(x_0)] - f(x_0) \right)^2 }_{\text{bias}^2} + \underbrace{ \mathbb{E} \left[ \left( \hat f(x_0)-\mathbb{E}[\hat f(x_0)] \right)^2 \right] }_{\text{variance}}. \]

A very simple model tends to have high bias and low variance. A very flexible model tends to have low bias and high variance. The useful model is rarely the one with the smallest training error; it is the one that manages this tradeoff best for new data.

5. How to Estimate Prediction Error

In practice, we usually do not know \(Err\). We estimate it. Common approaches include:

Validation set: simple and honest, but can waste data when the sample size is small.
Cross-validation: repeatedly holds out parts of the data and averages the error.
Bootstrap: resamples the data to estimate uncertainty and prediction error.
Analytic corrections such as \(C_p\), AIC, and related optimism-based criteria.

The conceptual link is that all of these methods try to correct the same problem: training error is not the same as future prediction error.

6. A Small Bayesian Note

A Bayesian model can express prediction uncertainty through the posterior predictive distribution:

\[ p(y_0 \mid x_0, \mathcal{T}) = \int p(y_0 \mid x_0, \theta) p(\theta \mid \mathcal{T}) d\theta. \]

This does not remove prediction error, but it changes how uncertainty is represented. Instead of only producing one fitted value, the model produces a distribution over possible future responses. From that distribution, we can evaluate expected loss under a chosen loss function.

What I Want to Remember

Prediction error is not just one number. Training error, in-sample error, conditional test error, and expected test error answer different questions. The most important warning is that a model can look good because it has learned the particular noise pattern in the training set. Optimism, effective degrees of freedom, and bias-variance decomposition are three different ways to describe this same danger.

Reference

Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Second Edition. Springer. Chapter 7.