Chengyuan Zhang

Chengyuan Zhang

Ph.D. candidate in the Department of Civil Engineering, McGill University.

Proof: unbiasedness of ordinary least squares (OLS)

less than 1 minute read

Published:

Consider the linear regression model: \(\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon},\) where:

  • \(\mathbf{y}\) is an \(n \times 1\) vector of observations.
  • \(\mathbf{X}\) is an \(n \times p\) design matrix (full column rank).
  • \(\boldsymbol{\beta}\) is a \(p \times 1\) vector of unknown parameters.
  • \(\boldsymbol{\varepsilon}\) is an \(n \times 1\) vector of errors with
  • \(\mathbb{E}[\boldsymbol{\varepsilon}|\mathbf{X}] = \mathbf{0}\).

Ordinary Least Squares (OLS) Estimator:

\[\hat{\boldsymbol{\beta}}_{\text{OLS}} = (\mathbf{X}^\top \mathbf{X})^{-1}\mathbf{X}^\top \mathbf{y}.\]

Proof of Unbiasedness:

  1. Substitute \(\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}\) into the OLS estimator:

    \[\hat{\boldsymbol{\beta}}_{\text{OLS}} = (\mathbf{X}^\top \mathbf{X})^{-1}\mathbf{X}^\top(\mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon}).\]

  2. Distribute \(\mathbf{X}^\top\):

    \[\hat{\boldsymbol{\beta}}_{\text{OLS}} = (\mathbf{X}^\top \mathbf{X})^{-1}(\mathbf{X}^\top\mathbf{X}\boldsymbol{\beta} + \mathbf{X}^\top\boldsymbol{\varepsilon}).\]

  3. Simplify \(\mathbf{X}^\top\mathbf{X}\boldsymbol{\beta}\):

    \[\hat{\boldsymbol{\beta}}_{\text{OLS}} = \boldsymbol{\beta} + (\mathbf{X}^\top \mathbf{X})^{-1}\mathbf{X}^\top \boldsymbol{\varepsilon}.\]

  4. Take the conditional expectation given \(\mathbf{X}\):

    \[\mathbb{E}[\hat{\boldsymbol{\beta}}_{\text{OLS}}\|\mathbf{X}] = \mathbb{E}[\boldsymbol{\beta} + (\mathbf{X}^\top \mathbf{X})^{-1}\mathbf{X}^\top \boldsymbol{\varepsilon}\|\mathbf{X}].\]

  5. Since \(\boldsymbol{\beta}\) is constant and \(\mathbb{E}[\boldsymbol{\varepsilon}|\mathbf{X}] = \mathbf{0}\):

    \[\mathbb{E}[\hat{\boldsymbol{\beta}}_{\text{OLS}}\|\mathbf{X}] = \boldsymbol{\beta} + (\mathbf{X}^\top \mathbf{X})^{-1}\mathbf{X}^\top \mathbb{E}[\boldsymbol{\varepsilon}|\mathbf{X}] = \boldsymbol{\beta}.\]

Thus:

\[\mathbb{E}[\hat{\boldsymbol{\beta}}_{\text{OLS}}|\mathbf{X}] = \boldsymbol{\beta} \implies \mathbb{E}[\hat{\boldsymbol{\beta}}_{\text{OLS}}] = \boldsymbol{\beta}.\]

Conclusion: The OLS estimator is unbiased.

You May Also Enjoy

A Detailed Introduction to Gaussian Velocity Fields (GVF) Based on Gaussian Processes

8 minute read

Published:

In autonomous driving, modeling and understanding the interactions between the ego vehicle and its surrounding vehicles is crucial for safe and efficient navigation. One important challenge is dealing with varying traffic densities and dynamic environments, where the number of surrounding vehicles can fluctuate. Traditional models, which rely on a fixed number of surrounding vehicles, may struggle in such conditions.

Gaussian Processes (GP) for Time Series Forecasting

5 minute read

Published:

Time-series forecasting is a critical application of Gaussian Processes (GPs), as they offer a flexible and probabilistic framework for predicting future values in sequential data. GPs not only provide point predictions but also quantify uncertainty, making them particularly useful in scenarios where confidence in predictions is important.

Connections among autoregressive (AR) processes, Cochrane-Orcutt correction, Ornstein-Uhlenbeck (OU) processes, and Gaussian Processes (GP)

8 minute read

Published:

In this post, we’ll explore four important concepts in time series modeling and stochastic processes: Autoregressive processes, Cochrane-Orcutt correction, Ornstein-Uhlenbeck (OU) processes, and Gaussian processes (GPs). After explaining each concept, we will also examine their connections and differences. In the end, we will provide some literature of the applications in driving behavior (car-following) modeling.

Modeling Autocorrelation: FFT vs Gaussian Processes

4 minute read

Published:

Autocorrelation is a key property of time series data, describing the dependency of a variable on its past values. Both the Fourier Transform (FT) and Gaussian Processes (GP) can model autocorrelation, but they operate in fundamentally different domains: FFT in the frequency domain and GP in the time domain. Despite their differences, the two methods are mathematically connected through the spectral representation theorem. This blog explores the core concepts, their mathematical underpinnings, and practical differences.