Linear Algebra 2: Special Matrices and vectors

This post is TLDR part 2 of chapter 1 on Linear Algebra from Deep Learning Book by Goodfellow.

Diagonal Matrix

• $\mathbf{D}$ is diagonal matrix iff $D_{i,j} = 0, \forall i \ne j$
• Diagonal matrix can be written as $\mathbf{D} = diag(\mathbf{v})$ where $\mathbf{v} = D_{i,j} \forall i=j$
• Diagonal matrix are useful because of following properties:
  • Fast multiplication: \(diag(\mathbf{v})*\mathbf{x} = \mathbf{v} \odot \mathbf{x}\)
  • Fast inversion: \(diag(\mathbf{v})^{-1} = diag([\dfrac{1}{v_1}, \dfrac{1}{v_2},..., \dfrac{1}{v_n}])\)

Symmetric matrix

• A matrix $\mathbf{A}$ is said to be symmetric if: \(\mathbf{A} = \mathbf{A}^T\)
• $A_{i,j} = A_{j,i}$
• A matrix must be square to be symmetric
• Symmetric matrices arise when entries are generated by function of arguments that do not depend on order: e.g. distance measurement: $A_{i,j}$ measures distance between points $i$ and $j$, then : $A_{i,j} = A_{j,i}$

Unit vector

• A vector is callled unit vector if its L² norm is 1. \(\vert\vert\mathbf{x}\vert\vert_2 = 1\)

Orthogonal vectors

• Two vectors $\mathbf{x}$ and $\mathbf{y}$ are said to be orthogonal, if: \(\mathbf{x}^T \mathbf{y} = 0\)
• If both vectors are non-zero norm, the angle between the vectors should be $90°$ for vectors to be orthogonal
• In $\mathbb{R}^n$, there are at most $n$ mutually orthogonal vectors.
• If orthogonal vectors $\mathbf{x}$ and $\mathbf{y}$, both have a unit norm (i.e. both are unit vectors), then they are said to be orthonormal vectors.

Orthogonal Matrix

• A square matrix
• All rows are mutually orthonormal vectors
• If \(\mathbf{A} = \begin{bmatrix}\mathbf{- x_1^T -}\\\mathbf{- x_2^T -}\\...\\\mathbf{- x_n^T -} \end{bmatrix}\) is an orthogonal matrix then: vectors: $\mathbf{x_i} \forall i = 1,2,…,n$ are mutually orthogonal.
  • $\mathbf{x_i}^T\mathbf{x_j} = 1, if \ i=j$
  • $\mathbf{x_i}^T\mathbf{x_j} = 0, if \ i\ne j$
• From above point, it can be shown that, if $\mathbf{A}$ is orthonormal: $\mathbf{A}\mathbf{A}^T = \mathbf{A}^T\mathbf{A} = \mathbf{I}$
• SO for orthonormal matrix: \(\mathbf{A}^T = \mathbf{A}^{-1}\)
• Advantage: Inverse is transpose, so is very easy to compute.

September 10th, 2020 by Bipin Lekhak