Chengyuan Zhang

Chengyuan Zhang

Postdoctoral Researcher, McGill University

Matrix derivative of Frobenius norm involving Hadamard product

less than 1 minute read

Published:

Problem: Solve $\frac{\partial\left|\boldsymbol{A}\circ (\boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})\right|_ {F}^{2}}{\partial\boldsymbol{W}}$ and $\frac{\partial\left|\boldsymbol{A}\circ ( \boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})\right|_{F}^{2}}{\partial\boldsymbol{X}}$, where $\circ$ denotes the Hadamard product, and all variables are matrices.

Solution

We define $\boldsymbol{Z}=\boldsymbol{A}\circ (\boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})$, then we have

\[\begin{aligned} \partial\|\boldsymbol{A}\circ (\boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})\|_F^2 &= \partial \boldsymbol{Z}:\boldsymbol{Z} \\ &= 2\boldsymbol{Z}:d\boldsymbol{Z} \\ &= 2\boldsymbol{Z}:d(\boldsymbol{A}\circ (\boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})) \\ &= 2\boldsymbol{Z}:\boldsymbol{A}\circ d(-\boldsymbol{W}^\top\boldsymbol{X}) \\ &= -2(\boldsymbol{A}\circ\boldsymbol{Z}):(d\boldsymbol{W}^\top\cdot\boldsymbol{X} +\boldsymbol{W}^\top\cdot d\boldsymbol{X}) \\ &= -2(\boldsymbol{A}\circ\boldsymbol{Z}):(d\boldsymbol{W}^\top\cdot\boldsymbol{X}) \\ &\quad -2(\boldsymbol{A}\circ\boldsymbol{Z}):(\boldsymbol{W}^\top\cdot d\boldsymbol{X}) \\ &= -2(\boldsymbol{A}\circ\boldsymbol{Z})\boldsymbol{X}^\top:d\boldsymbol{W}^\top -2\boldsymbol{W}(\boldsymbol{A}\circ\boldsymbol{Z}):d\boldsymbol{X} \\ &= -2\boldsymbol{X}(\boldsymbol{A}\circ\boldsymbol{Z})^\top:d\boldsymbol{W} -2\boldsymbol{W}(\boldsymbol{A}\circ\boldsymbol{Z}):d\boldsymbol{X}. \end{aligned}\]

Therefore, we have

\[\begin{aligned} \frac{\partial\|\boldsymbol{A}\circ (\boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})\|_F^2}{\partial\boldsymbol{W}} &= -2\boldsymbol{X}(\boldsymbol{A}\circ\boldsymbol{Z})^\top \\ &= -2\boldsymbol{X}(\boldsymbol{A}\circ\boldsymbol{A}\circ (\boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X}))^\top, \end{aligned}\]

and

\[\begin{aligned} \frac{\partial\|\boldsymbol{A}\circ (\boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})\|_F^2}{\partial\boldsymbol{X}} &= -2\boldsymbol{W}(\boldsymbol{A}\circ\boldsymbol{Z}) \\ &= -2\boldsymbol{W}(\boldsymbol{A}\circ\boldsymbol{A}\circ (\boldsymbol{Y}-\boldsymbol{W}^\top\boldsymbol{X})). \end{aligned}\]

Notes

  1. The Frobenuis product of two matrices $\boldsymbol{A}$ and $\boldsymbol{B}$ is defined as $\boldsymbol{A}: \boldsymbol{B}=\text{tr}(\boldsymbol{A}^\top\boldsymbol{B})$.
  2. $\boldsymbol{A}:\boldsymbol{B}\circ\boldsymbol{C}=\boldsymbol{A}\circ\boldsymbol{B}:\boldsymbol{C}$.

You May Also Enjoy

Introduction to Autoregressive (AR) Processes

5 minute read

Published:

Autoregressive (AR) processes are a class of time series models used to describe a variable that is correlated with its past values. AR models are widely applied in various fields such as economics, engineering, finance, and traffic modeling, among others. In this post, we will introduce the concept of AR processes, their mathematical formulation, and the different types of AR models used in time series forecasting.

Fundamental Probabilistic Graphical Models: Tail-to-Tail, Head-to-Tail, and Head-to-Head

7 minute read

Published:

Probabilistic graphical models (PGMs) provide a compact, visual language for reasoning about joint distributions over many random variables. In directed acyclic graphs (DAGs), three elementary three-node structures — tail-to-tail, head-to-tail, and head-to-head — serve as the building blocks that determine when two variables are (conditionally) independent. Understanding these three patterns is the key to reading any larger Bayesian network.

A Detailed Introduction to Gaussian Velocity Fields (GVF) Based on Gaussian Processes

8 minute read

Published:

In autonomous driving, modeling and understanding the interactions between the ego vehicle and its surrounding vehicles is crucial for safe and efficient navigation. One important challenge is dealing with varying traffic densities and dynamic environments, where the number of surrounding vehicles can fluctuate. Traditional models, which rely on a fixed number of surrounding vehicles, may struggle in such conditions.