Matrix Calculus

Matrix calculus tutorial based on Jan R. Magnus' "A gentle introduction to matrix calculus" - a comprehensive guide to matrix differential calculus with applications in statistics, econometrics, and optimization.

Core Concepts

Matrix calculus rests on two pillars and requires six tools:

Two Pillars

1. Correct definition of matrix derivative (Section 3)
2. The concept of differential (Section 4)

Six Tools (Section 2)

Tool	Description	Key Property
Trace	Sum of diagonal elements: $\operatorname{tr} A = \sum_i a_{ii}$	$\operatorname{tr} A'B = \operatorname{tr} BA'$ (cyclic permutations)
Linear/Quadratic Forms	$Ax$ and $x'Ax$	$x'Ax = 0 ; \forall x \iff A$ is skew-symmetric
Kronecker Product	$A \otimes B$	$(A \otimes B)(C \otimes D) = AC \otimes BD$
Vec Operator	Stacks columns: $\operatorname{vec} A$	$\operatorname{vec} ABC = (C' \otimes A)\operatorname{vec} B$
Commutation Matrix	$K_{mn}$ transforms $\operatorname{vec} A \to \operatorname{vec} A'$	$K_{mn}' = K_{mn}^{-1} = K_{nm}$
Duplication Matrix	$D_n$ transforms $\operatorname{vech} A \to \operatorname{vec} A$ for symmetric $A$	Essential for symmetric matrix derivatives

The Correct Definition of Matrix Derivative

The only correct definition for the derivative of a matrix function $F(X)$ is:

$$DF(X) = \frac{\partial \operatorname{vec} F(X)}{\partial (\operatorname{vec} X)'}$$

Each row contains partial derivatives of one element of $F$ with respect to all elements of $X$. Each column contains partial derivatives of all elements of $F$ with respect to one element of $X$.

Key consequences:

• Derivative of scalar function $a'x$ is $a'$ (row vector, not column)
• Derivative of $\operatorname{tr} X$ is $(\operatorname{vec} I)'$ (row vector)

First Differential and Identification Theorem

Differential Rules

For matrices $X, Y$ and scalar $\alpha$:

• $dA = 0$ (constant)
• $d(\alpha X) = \alpha , dX$
• $d(X') = (dX)'$
• $d(X + Y) = dX + dY$
• $d(XY) = (dX)Y + X(dY)$
• $d\operatorname{tr} X = \operatorname{tr} dX$ (square $X$)
• $d|X| = |X| \operatorname{tr} X^{-1} dX$ (nonsingular $X$)
• $d\log|X| = \operatorname{tr} X^{-1} dX$
• $dX^{-1} = -X^{-1}(dX)X^{-1}$ (nonsingular $X$)

First Identification Theorem

$$d f(x) = A(x) , dx \iff Df(x) = A(x)$$

For matrices:
$$d \operatorname{vec} F(X) = A(X) , d\operatorname{vec} X \iff DF(X) = A(X)$$

Workflow: Compute differential $\to$ Identify derivative from coefficient of $dX$

Common Derivatives

Function	Derivative
$a'x$	$a'$
$x'Ax$	$x'(A + A')$ (or $2x'A$ if $A$ symmetric)
$\operatorname{tr} X'AX$	$(\operatorname{vec} C)'$ where $C = (A + A')X$
$\log	X'X
$\operatorname{tr} X^k$	$k(\operatorname{vec} X'^{k-1})'$
$AX^{-1}B$	$-(X^{-1}B)' \otimes (AX^{-1})$

Optimization

Unconstrained Optimization

1. Compute differential $df(x) = a(x)' dx$
2. Set $a(x) = 0$ and solve

Constrained Optimization (Lagrange)

For constraint $g(x) = 0$, define Lagrangian:
$$\mathcal{L}(x) = f(x) - \lambda g(x)$$

For matrix constraint $G(X) = 0$:
$$\mathcal{L}(X) = f(X) - \operatorname{tr} L' G(X)$$

First-order conditions: $\partial f/\partial x' = \lambda \partial g/\partial x'$ and $g(x) = 0$

Second Differential and Hessian

Second Identification Theorem

$$d^2 f(x) = (dx)' B(x) dx \iff Hf(x) = \frac{B(x) + B(x)'}{2}$$

Important: Must symmetrize $B(x)$ - the matrix in the quadratic form is not necessarily symmetric.

Hessian Matrix

The Hessian contains second-order partial derivatives:
$$Hf(x) = \frac{\partial^2 f(x)}{\partial x \partial x'} = \frac{\partial}{\partial x'}\left(\frac{\partial f(x)}{\partial x'}\right)'$$

Chain Rule for Second Differentials

If $z = f(y)$ and $y = g(x)$:
$$d^2 z = (dA)B(x)dx + A(y)(dB)dx$$

where $dz = A(y)dy$ and $dy = B(x)dx$.

Note: Cauchy invariance does NOT hold for second differentials.

Matrix Forms for Hessian

Proposition 15 (Hessian identification):

• $d^2 f(X) = \operatorname{tr} A(dX)'B , dX \iff Hf(X) = \frac{1}{2}(A' \otimes B + A \otimes B')$
• $d^2 f(X) = \operatorname{tr} A(dX)B , dX \iff Hf(X) = \frac{1}{2}K_{qn}(A' \otimes B + B' \otimes A)$

Practical Tips (Tacit Knowledge)

1. Think of matrices as units, not individual elements
2. Prove theorems for $n=2$ and $n=3$ first - if it works for $n=3$, it probably works in general
3. Test with diagonal matrices when a matrix is symmetric
4. To prove $A = B$, prove $C = A - B = 0$ (often easier)
5. Use $\operatorname{tr} C'C = 0$ to prove $C = 0$
6. Always use the correct definition of matrix derivative
7. For Hessian, start with first differential, not first derivative
8. Symmetrize $B(x)$ in $d^2 f(x) = (dx)'B(x)dx$

Applications

The reference material includes worked examples:

1. Least Squares (Section 6): Constrained and unconstrained
2. Maximum Likelihood (Section 13): Multivariate normal
3. ML with Parameters in Design Matrix (Section 14)
4. Eckart-Young Theorem (Section 15): Matrix approximation

Reference

For complete theory, proofs, and exercises, see references/magnus_matrix_calculus.md - the full text of "A gentle introduction to matrix calculus" by Jan R. Magnus.

Resources

references/

• magnus_matrix_calculus.md - Complete tutorial text with all theorems, proofs, and exercises