Chengyuan Zhang

Chengyuan Zhang

Ph.D. candidate in the Department of Civil Engineering, McGill University.

Matrix Derivative of Frobenius norm involving Hadamard Product

less than 1 minute read

Published:

Problem: Solve $\frac{\partial\left|\boldsymbol{A}\circ (\boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})\right|_ {F}^{2}}{\partial\boldsymbol{W}}$ and $\frac{\partial\left|\boldsymbol{A}\circ ( \boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})\right|_{F}^{2}}{\partial\boldsymbol{X}}$, where $\circ$ denotes the Hadamard product, and all variables are matrices.

Solution

We define $\boldsymbol{Z}=\boldsymbol{A}\circ (\boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})$, then we have

\[\partial\|\boldsymbol{A}\circ (\boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})\|_F^2\\ = \partial \boldsymbol{Z}:\boldsymbol{Z}\\ =2\boldsymbol{Z}:d\boldsymbol{Z}\\ =2\boldsymbol{Z}:d(\boldsymbol{A}\circ (\boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X}))\\ = 2\boldsymbol{Z}:\boldsymbol{A}\circ d(-\boldsymbol{W}^\top\boldsymbol{X})\\ = -2(\boldsymbol{A}\circ\boldsymbol{Z}):(d\boldsymbol{W}^\top\cdot\boldsymbol{X}+\boldsymbol{W}^\top\cdot d\boldsymbol{X})\\ = -2(\boldsymbol{A}\circ\boldsymbol{Z}):(d\boldsymbol{W}^\top\cdot\boldsymbol{X})-2(\boldsymbol{A}\circ\boldsymbol{Z}):( \boldsymbol{W}^\top\cdot d\boldsymbol{X})\\ = -2(\boldsymbol{A}\circ\boldsymbol{Z})\boldsymbol{X}^\top:d\boldsymbol{W}^\top-2\boldsymbol{W}( \boldsymbol{A}\circ\boldsymbol{Z}):d\boldsymbol{X}\\ = -2\boldsymbol{X}(\boldsymbol{A}\circ\boldsymbol{Z})^\top:d\boldsymbol{W}-2\boldsymbol{W}( \boldsymbol{A}\circ\boldsymbol{Z}):d\boldsymbol{X}.\\\]

Therefore, we have \(\frac{\partial\|\boldsymbol{A}\circ (\boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})\|_F^2}{\partial \boldsymbol{W}}=-2\boldsymbol{X}(\boldsymbol{A}\circ\boldsymbol{Z})^\top=-2\boldsymbol{X}( \boldsymbol{A}\circ\boldsymbol{A}\circ (\boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X}))^\top,\) and \(\frac{\partial\|\boldsymbol{A}\circ (\boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})\|_F^2}{\partial \boldsymbol{X}}=-2\boldsymbol{W}(\boldsymbol{A}\circ\boldsymbol{Z})=-2\boldsymbol{W}( \boldsymbol{A}\circ\boldsymbol{A}\circ (\boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})).\)

Notes

  1. The Frobenuis product of two matrices $\boldsymbol{A}$ and $\boldsymbol{B}$ is defined as $\boldsymbol{A}: \boldsymbol{B}=\text{tr}(\boldsymbol{A}^\top\boldsymbol{B})$.
  2. $\boldsymbol{A}:\boldsymbol{B}\circ\boldsymbol{C}=\boldsymbol{A}\circ\boldsymbol{B}:\boldsymbol{C}$.

You May Also Enjoy

Connections among autoregressive (AR) processes, Cochrane-Orcutt correction, Ornstein-Uhlenbeck (OU) processes, and Gaussian Processes (GP)

8 minute read

Published:

In this post, we’ll explore four important concepts in time series modeling and stochastic processes: Autoregressive processes, Cochrane-Orcutt correction, Ornstein-Uhlenbeck (OU) processes, and Gaussian processes (GPs). After explaining each concept, we will also examine their connections and differences. In the end, we will provide some literature of the applications in driving behavior (car-following) modeling.

Modeling Autocorrelation: FFT vs Gaussian Processes

4 minute read

Published:

Autocorrelation is a key property of time series data, describing the dependency of a variable on its past values. Both the Fourier Transform (FT) and Gaussian Processes (GP) can model autocorrelation, but they operate in fundamentally different domains: FFT in the frequency domain and GP in the time domain. Despite their differences, the two methods are mathematically connected through the spectral representation theorem. This blog explores the core concepts, their mathematical underpinnings, and practical differences.

Proof: unbiasedness of ordinary least squares (OLS)

less than 1 minute read

Published:

Consider the linear regression model: \(\mathbf{y} = \mathbf{X}\boldsymbol{\beta} + \boldsymbol{\varepsilon},\) where:

  • \(\mathbf{y}\) is an \(n \times 1\) vector of observations.
  • \(\mathbf{X}\) is an \(n \times p\) design matrix (full column rank).
  • \(\boldsymbol{\beta}\) is a \(p \times 1\) vector of unknown parameters.
  • \(\boldsymbol{\varepsilon}\) is an \(n \times 1\) vector of errors with
  • \(\mathbb{E}[\boldsymbol{\varepsilon}|\mathbf{X}] = \mathbf{0}\).

From Ordinary Least Squares (OLS) to Generalized Least Squares (GLS)

5 minute read

Published:

Ordinary Least Squares (OLS) is one of the most widely used methods for linear regression. It provides unbiased estimates of the model parameters under the assumption that the error terms are independent and identically distributed (i.i.d.) with constant variance. However, real-world data often violate these assumptions. When the errors exhibit heteroskedasticity (non-constant variance) or correlation, OLS estimates remain UNBIASED (see this post) but lose their efficiency, leading to incorrect standard errors and confidence intervals.