Chengyuan Zhang

Chengyuan Zhang

Ph.D. candidate in the Department of Civil Engineering, McGill University

Posts by Tags

Cochrane-Orcutt correction

Connections among autoregressive (AR) processes, Cochrane-Orcutt correction, Ornstein-Uhlenbeck (OU) processes, and Gaussian Processes (GP)

7 minute read

Published:

In this post, we’ll explore four important concepts in time series modeling and stochastic processes: Autoregressive processes, Cochrane-Orcutt correction, Ornstein-Uhlenbeck (OU) processes, and Gaussian processes (GPs). After explaining each concept, we will also examine their connections and differences. In the end, we will provide some literature of the applications in driving behavior (car-following) modeling.

Fourier transform

Modeling Autocorrelation: FFT vs Gaussian Processes

4 minute read

Published:

Autocorrelation is a key property of time series data, describing the dependency of a variable on its past values. Both the Fourier Transform (FT) and Gaussian Processes (GP) can model autocorrelation, but they operate in fundamentally different domains: FFT in the frequency domain and GP in the time domain. Despite their differences, the two methods are mathematically connected through the spectral representation theorem. This blog explores the core concepts, their mathematical underpinnings, and practical differences.

Frobenuis product

Matrix derivative of Frobenius norm involving Hadamard product

less than 1 minute read

Published:

Problem: Solve $\frac{\partial\left|\boldsymbol{A}\circ (\boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})\right|_ {F}^{2}}{\partial\boldsymbol{W}}$ and $\frac{\partial\left|\boldsymbol{A}\circ ( \boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})\right|_{F}^{2}}{\partial\boldsymbol{X}}$, where $\circ$ denotes the Hadamard product, and all variables are matrices.

GLS

From Ordinary Least Squares (OLS) to Generalized Least Squares (GLS)

4 minute read

Published:

Ordinary Least Squares (OLS) is one of the most widely used methods for linear regression. It provides unbiased estimates of the model parameters under the assumption that the error terms are independent and identically distributed (i.i.d.) with constant variance. However, real-world data often violate these assumptions. When the errors exhibit heteroskedasticity (non-constant variance) or correlation, OLS estimates remain UNBIASED (see this post) but lose their efficiency, leading to incorrect standard errors and confidence intervals.

Gaussian processes

A Detailed Introduction to Gaussian Velocity Fields (GVF) Based on Gaussian Processes

7 minute read

Published:

In autonomous driving, modeling and understanding the interactions between the ego vehicle and its surrounding vehicles is crucial for safe and efficient navigation. One important challenge is dealing with varying traffic densities and dynamic environments, where the number of surrounding vehicles can fluctuate. Traditional models, which rely on a fixed number of surrounding vehicles, may struggle in such conditions.

Gaussian Processes (GP) for Time Series Forecasting

5 minute read

Published:

Time-series forecasting is a critical application of Gaussian Processes (GPs), as they offer a flexible and probabilistic framework for predicting future values in sequential data. GPs not only provide point predictions but also quantify uncertainty, making them particularly useful in scenarios where confidence in predictions is important.

Connections among autoregressive (AR) processes, Cochrane-Orcutt correction, Ornstein-Uhlenbeck (OU) processes, and Gaussian Processes (GP)

7 minute read

Published:

In this post, we’ll explore four important concepts in time series modeling and stochastic processes: Autoregressive processes, Cochrane-Orcutt correction, Ornstein-Uhlenbeck (OU) processes, and Gaussian processes (GPs). After explaining each concept, we will also examine their connections and differences. In the end, we will provide some literature of the applications in driving behavior (car-following) modeling.

Modeling Autocorrelation: FFT vs Gaussian Processes

4 minute read

Published:

Autocorrelation is a key property of time series data, describing the dependency of a variable on its past values. Both the Fourier Transform (FT) and Gaussian Processes (GP) can model autocorrelation, but they operate in fundamentally different domains: FFT in the frequency domain and GP in the time domain. Despite their differences, the two methods are mathematically connected through the spectral representation theorem. This blog explores the core concepts, their mathematical underpinnings, and practical differences.

HMM

Hadamard product

Matrix derivative of Frobenius norm involving Hadamard product

less than 1 minute read

Published:

Problem: Solve $\frac{\partial\left|\boldsymbol{A}\circ (\boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})\right|_ {F}^{2}}{\partial\boldsymbol{W}}$ and $\frac{\partial\left|\boldsymbol{A}\circ ( \boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})\right|_{F}^{2}}{\partial\boldsymbol{X}}$, where $\circ$ denotes the Hadamard product, and all variables are matrices.

OLS

Proof: unbiasedness of ordinary least squares (OLS)

less than 1 minute read

Published:

Consider the linear regression model: \(\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon},\) where:

  • \(\mathbf{y}\) is an \(n \times 1\) vector of observations.
  • \(\mathbf{X}\) is an \(n \times p\) design matrix (full column rank).
  • \(\boldsymbol{\beta}\) is a \(p \times 1\) vector of unknown parameters.
  • \(\boldsymbol{\varepsilon}\) is an \(n \times 1\) vector of errors with
  • \(\mathbb{E}[\boldsymbol{\varepsilon}|\mathbf{X}] = \mathbf{0}\).

From Ordinary Least Squares (OLS) to Generalized Least Squares (GLS)

4 minute read

Published:

Ordinary Least Squares (OLS) is one of the most widely used methods for linear regression. It provides unbiased estimates of the model parameters under the assumption that the error terms are independent and identically distributed (i.i.d.) with constant variance. However, real-world data often violate these assumptions. When the errors exhibit heteroskedasticity (non-constant variance) or correlation, OLS estimates remain UNBIASED (see this post) but lose their efficiency, leading to incorrect standard errors and confidence intervals.

Ornstein-Uhlenbeck processes

Connections among autoregressive (AR) processes, Cochrane-Orcutt correction, Ornstein-Uhlenbeck (OU) processes, and Gaussian Processes (GP)

7 minute read

Published:

In this post, we’ll explore four important concepts in time series modeling and stochastic processes: Autoregressive processes, Cochrane-Orcutt correction, Ornstein-Uhlenbeck (OU) processes, and Gaussian processes (GPs). After explaining each concept, we will also examine their connections and differences. In the end, we will provide some literature of the applications in driving behavior (car-following) modeling.

PGM

Fundamental Probabilistic Graphical Models

less than 1 minute read

Published:

Tail-to-tail, head-to-tail, and head-to-head structures are three common types of graphical models. In this post, we will introduce these structures and discuss their implications for probabilistic modeling and inference.

autocorrelation

Connections among autoregressive (AR) processes, Cochrane-Orcutt correction, Ornstein-Uhlenbeck (OU) processes, and Gaussian Processes (GP)

7 minute read

Published:

In this post, we’ll explore four important concepts in time series modeling and stochastic processes: Autoregressive processes, Cochrane-Orcutt correction, Ornstein-Uhlenbeck (OU) processes, and Gaussian processes (GPs). After explaining each concept, we will also examine their connections and differences. In the end, we will provide some literature of the applications in driving behavior (car-following) modeling.

Modeling Autocorrelation: FFT vs Gaussian Processes

4 minute read

Published:

Autocorrelation is a key property of time series data, describing the dependency of a variable on its past values. Both the Fourier Transform (FT) and Gaussian Processes (GP) can model autocorrelation, but they operate in fundamentally different domains: FFT in the frequency domain and GP in the time domain. Despite their differences, the two methods are mathematically connected through the spectral representation theorem. This blog explores the core concepts, their mathematical underpinnings, and practical differences.

From Ordinary Least Squares (OLS) to Generalized Least Squares (GLS)

4 minute read

Published:

Ordinary Least Squares (OLS) is one of the most widely used methods for linear regression. It provides unbiased estimates of the model parameters under the assumption that the error terms are independent and identically distributed (i.i.d.) with constant variance. However, real-world data often violate these assumptions. When the errors exhibit heteroskedasticity (non-constant variance) or correlation, OLS estimates remain UNBIASED (see this post) but lose their efficiency, leading to incorrect standard errors and confidence intervals.

autoregressive processes

Introduction to Autoregressive (AR) Processes

6 minute read

Published:

Autoregressive (AR) processes are a class of time series models used to describe a variable that is correlated with its past values. AR models are widely applied in various fields such as economics, engineering, finance, and traffic modeling, among others. In this post, we will introduce the concept of AR processes, their mathematical formulation, and the different types of AR models used in time series forecasting.

Connections among autoregressive (AR) processes, Cochrane-Orcutt correction, Ornstein-Uhlenbeck (OU) processes, and Gaussian Processes (GP)

7 minute read

Published:

In this post, we’ll explore four important concepts in time series modeling and stochastic processes: Autoregressive processes, Cochrane-Orcutt correction, Ornstein-Uhlenbeck (OU) processes, and Gaussian processes (GPs). After explaining each concept, we will also examine their connections and differences. In the end, we will provide some literature of the applications in driving behavior (car-following) modeling.

driving behavior

A Detailed Introduction to Gaussian Velocity Fields (GVF) Based on Gaussian Processes

7 minute read

Published:

In autonomous driving, modeling and understanding the interactions between the ego vehicle and its surrounding vehicles is crucial for safe and efficient navigation. One important challenge is dealing with varying traffic densities and dynamic environments, where the number of surrounding vehicles can fluctuate. Traditional models, which rely on a fixed number of surrounding vehicles, may struggle in such conditions.

forecasting

Modeling Autocorrelation: FFT vs Gaussian Processes

4 minute read

Published:

Autocorrelation is a key property of time series data, describing the dependency of a variable on its past values. Both the Fourier Transform (FT) and Gaussian Processes (GP) can model autocorrelation, but they operate in fundamentally different domains: FFT in the frequency domain and GP in the time domain. Despite their differences, the two methods are mathematically connected through the spectral representation theorem. This blog explores the core concepts, their mathematical underpinnings, and practical differences.

heteroskedasticity

From Ordinary Least Squares (OLS) to Generalized Least Squares (GLS)

4 minute read

Published:

Ordinary Least Squares (OLS) is one of the most widely used methods for linear regression. It provides unbiased estimates of the model parameters under the assumption that the error terms are independent and identically distributed (i.i.d.) with constant variance. However, real-world data often violate these assumptions. When the errors exhibit heteroskedasticity (non-constant variance) or correlation, OLS estimates remain UNBIASED (see this post) but lose their efficiency, leading to incorrect standard errors and confidence intervals.

hierarchical model

Random Effects and Hierarchical Models in Driving Behaviors Modeling

4 minute read

Published:

In many driving behavior studies, we model how a following vehicle responds to the movement of a lead vehicle. For example, the Intelligent Driver Model (IDM) uses a set of parameters \(\boldsymbol{\theta} = (v_0, T, a_ {\text{max}}, b, s_0)\) to describe a driver’s response in terms of desired speed, time headway, maximum acceleration, comfortable deceleration, and minimal spacing. A critical challenge, however, is that not all drivers behave the same way. Some maintain larger headways, others brake more aggressively, and still others prefer smoother accelerations.

Heterogeneity and Hierarchical Models: Understanding Pooled, Unpooled, and Hierarchical Approaches

4 minute read

Published:

Hierarchical models are powerful tools in statistical modeling and machine learning, enabling us to represent data with complex dependency structures. These models are particularly useful in contexts where data is naturally grouped or exhibits multi-level variability. A critical aspect of hierarchical models lies in their hyperparameters, which control the relationships between different levels of the model.

hyperparameters

Heterogeneity and Hierarchical Models: Understanding Pooled, Unpooled, and Hierarchical Approaches

4 minute read

Published:

Hierarchical models are powerful tools in statistical modeling and machine learning, enabling us to represent data with complex dependency structures. These models are particularly useful in contexts where data is naturally grouped or exhibits multi-level variability. A critical aspect of hierarchical models lies in their hyperparameters, which control the relationships between different levels of the model.

logsumexp

matrix derivative

Matrix derivative of Frobenius norm involving Hadamard product

less than 1 minute read

Published:

Problem: Solve $\frac{\partial\left|\boldsymbol{A}\circ (\boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})\right|_ {F}^{2}}{\partial\boldsymbol{W}}$ and $\frac{\partial\left|\boldsymbol{A}\circ ( \boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})\right|_{F}^{2}}{\partial\boldsymbol{X}}$, where $\circ$ denotes the Hadamard product, and all variables are matrices.

proof

Proof: unbiasedness of ordinary least squares (OLS)

less than 1 minute read

Published:

Consider the linear regression model: \(\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon},\) where:

  • \(\mathbf{y}\) is an \(n \times 1\) vector of observations.
  • \(\mathbf{X}\) is an \(n \times p\) design matrix (full column rank).
  • \(\boldsymbol{\beta}\) is a \(p \times 1\) vector of unknown parameters.
  • \(\boldsymbol{\varepsilon}\) is an \(n \times 1\) vector of errors with
  • \(\mathbb{E}[\boldsymbol{\varepsilon}|\mathbf{X}] = \mathbf{0}\).

random effects

Random Effects and Hierarchical Models in Driving Behaviors Modeling

4 minute read

Published:

In many driving behavior studies, we model how a following vehicle responds to the movement of a lead vehicle. For example, the Intelligent Driver Model (IDM) uses a set of parameters \(\boldsymbol{\theta} = (v_0, T, a_ {\text{max}}, b, s_0)\) to describe a driver’s response in terms of desired speed, time headway, maximum acceleration, comfortable deceleration, and minimal spacing. A critical challenge, however, is that not all drivers behave the same way. Some maintain larger headways, others brake more aggressively, and still others prefer smoother accelerations.

regression

Proof: unbiasedness of ordinary least squares (OLS)

less than 1 minute read

Published:

Consider the linear regression model: \(\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon},\) where:

  • \(\mathbf{y}\) is an \(n \times 1\) vector of observations.
  • \(\mathbf{X}\) is an \(n \times p\) design matrix (full column rank).
  • \(\boldsymbol{\beta}\) is a \(p \times 1\) vector of unknown parameters.
  • \(\boldsymbol{\varepsilon}\) is an \(n \times 1\) vector of errors with
  • \(\mathbb{E}[\boldsymbol{\varepsilon}|\mathbf{X}] = \mathbf{0}\).

From Ordinary Least Squares (OLS) to Generalized Least Squares (GLS)

4 minute read

Published:

Ordinary Least Squares (OLS) is one of the most widely used methods for linear regression. It provides unbiased estimates of the model parameters under the assumption that the error terms are independent and identically distributed (i.i.d.) with constant variance. However, real-world data often violate these assumptions. When the errors exhibit heteroskedasticity (non-constant variance) or correlation, OLS estimates remain UNBIASED (see this post) but lose their efficiency, leading to incorrect standard errors and confidence intervals.

time series

Introduction to Autoregressive (AR) Processes

6 minute read

Published:

Autoregressive (AR) processes are a class of time series models used to describe a variable that is correlated with its past values. AR models are widely applied in various fields such as economics, engineering, finance, and traffic modeling, among others. In this post, we will introduce the concept of AR processes, their mathematical formulation, and the different types of AR models used in time series forecasting.

Gaussian Processes (GP) for Time Series Forecasting

5 minute read

Published:

Time-series forecasting is a critical application of Gaussian Processes (GPs), as they offer a flexible and probabilistic framework for predicting future values in sequential data. GPs not only provide point predictions but also quantify uncertainty, making them particularly useful in scenarios where confidence in predictions is important.

Modeling Autocorrelation: FFT vs Gaussian Processes

4 minute read

Published:

Autocorrelation is a key property of time series data, describing the dependency of a variable on its past values. Both the Fourier Transform (FT) and Gaussian Processes (GP) can model autocorrelation, but they operate in fundamentally different domains: FFT in the frequency domain and GP in the time domain. Despite their differences, the two methods are mathematically connected through the spectral representation theorem. This blog explores the core concepts, their mathematical underpinnings, and practical differences.

tricks

Random Effects and Hierarchical Models in Driving Behaviors Modeling

4 minute read

Published:

In many driving behavior studies, we model how a following vehicle responds to the movement of a lead vehicle. For example, the Intelligent Driver Model (IDM) uses a set of parameters \(\boldsymbol{\theta} = (v_0, T, a_ {\text{max}}, b, s_0)\) to describe a driver’s response in terms of desired speed, time headway, maximum acceleration, comfortable deceleration, and minimal spacing. A critical challenge, however, is that not all drivers behave the same way. Some maintain larger headways, others brake more aggressively, and still others prefer smoother accelerations.

Heterogeneity and Hierarchical Models: Understanding Pooled, Unpooled, and Hierarchical Approaches

4 minute read

Published:

Hierarchical models are powerful tools in statistical modeling and machine learning, enabling us to represent data with complex dependency structures. These models are particularly useful in contexts where data is naturally grouped or exhibits multi-level variability. A critical aspect of hierarchical models lies in their hyperparameters, which control the relationships between different levels of the model.

Matrix derivative of Frobenius norm involving Hadamard product

less than 1 minute read

Published:

Problem: Solve $\frac{\partial\left|\boldsymbol{A}\circ (\boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})\right|_ {F}^{2}}{\partial\boldsymbol{W}}$ and $\frac{\partial\left|\boldsymbol{A}\circ ( \boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})\right|_{F}^{2}}{\partial\boldsymbol{X}}$, where $\circ$ denotes the Hadamard product, and all variables are matrices.