1 Introduction

In this lecture, we look at how to use eigenspaces defined with principal components computed with PCA:

• Difference-From-Feature-Space (DFFS)

• Distance-In-Feature-Space (DIFS)

which are both defined in Probabilistic visual learning for object representation B. Moghaddam and A. Pentland (PAMI 1997).

2 Representation with Principal Components

Having applied PCA to a dataset $\lbrace\mathbf{x}_n\rbrace_{n=1,\cdots,N}$ with $\forall n, \mathbf{x}_n \in \mathbb{R}^D$, we note $\overline{\mathbf{x}}\in \mathbb{R}^D$ the mean vector and $\mathrm{U}=\lbrack\mathbf{u}_1, \mathbf{u}_2, \cdots, \mathbf{u}_D\rbrack$ the $D$ eigenvectors (principal components) of the covariance matrix $\mathrm{S}$ are sorted such that:
$$\lambda_{1}\geq \lambda_{2} \geq \cdots \geq \lambda_{D} \geq 0$$ with $\lambda_n$ the eigenvalue associated with eigenvector $\mathbf{u}_n$, $\forall n$.

Because $\mathbf{u}_i^T\mathbf{u}_j=0, \forall i\neq j$ (orthogonality), $\mathbf{u}_n^T\mathbf{u}_n=1,\forall n$ (unit norm), the $D$ eigenvectors form a orthonormal basis of the space $\mathbb{R}^D$.

Any vector $\mathbf{x}\in \mathbb{R}^D$ can be reconstructed in that basis of eigenvectors: $$\mathbf{x}= \overline{\mathbf{x}} +\sum_{i=1}^D \alpha_i \ \mathbf{u_{i}}$$ or with notation $\tilde{\mathbf{x}}=\mathbf{x}-\overline{\mathbf{x}}$ $$\tilde{\mathbf{x}}= \sum_{i=1}^D \alpha_i \ \mathbf{u_{i}}$$ with the coordinates $\alpha_i=(\mathbf{x}-\overline{\mathbf{x}})^T \mathbf{u}_i$, $\forall i=1,\cdots, D$.

The vector of coordinates $\pmb{\alpha}=[\alpha_1, \cdots, \alpha_D]^T$ represents $\mathbf{x}$ in a space centered on the mean $\overline{\mathbf{x}}$ and orthonormal basis defined by the eigenvectors $\lbrace\mathbf{u}_i\rbrace_{i=1,\cdots,D}$:

$$\mathbf{x}= \left\lbrack\begin{array}{c} x_1\\ x_2\\ \vdots\\ x_D\\ \end{array}\right\rbrack=\left\lbrack\begin{array}{c} \overline{x}_1\\ \overline{x}_2\\ \vdots\\ \overline{x}_D\\ \end{array}\right\rbrack + \sum_{i=1}^D \alpha_i \ \mathbf{u}_i$$ or $$\mathbf{x} =\underbrace{ \left\lbrack\begin{array}{c} \overline{x}_1\\ \overline{x}_2\\ \vdots\\ \overline{x}_D\\ \end{array}\right\rbrack}_{\overline{\mathbf{x}}} + \alpha_1 \underbrace{ \left\lbrack\begin{array}{c} u_{1,1}\\ u_{1,2}\\ \vdots\\ u_{1,D}\\ \end{array}\right\rbrack}_{\mathbf{u}_1} + \alpha_2 \underbrace{\left\lbrack\begin{array}{c} u_{2,1}\\ u_{2,2}\\ \vdots\\ u_{2,D}\\ \end{array}\right\rbrack}_{\mathbf{u}_2} +\ ...\ + \alpha_D \underbrace{\left\lbrack\begin{array}{c} u_{D,1}\\ u_{D,2}\\ \vdots\\ u_{D,D}\\ \end{array}\right\rbrack}_{\mathbf{u}_D}$$

This representation with $\pmb{\alpha}$ is a more useful representation than the original representation of $\mathbf{x}$ in the space centered on the origin $\pmb{0}_{D}$ and with the standard basis:
$$\mathbf{x}= \left\lbrack\begin{array}{c} x_1\\ x_2\\ \vdots\\ x_D\\ \end{array}\right\rbrack= \left\lbrack\begin{array}{c} 0\\ 0\\ \vdots\\ 0\\ \end{array}\right\rbrack +\sum_{i=1}^D x_i \ \mathbf{e}_i = \underbrace{ \left\lbrack\begin{array}{c} 0\\ 0\\ \vdots\\ 0\\ \end{array}\right\rbrack}_{\pmb{0}_D} +x_1 \underbrace{ \left\lbrack\begin{array}{c} 1\\ 0\\ \vdots\\ 0\\ \end{array}\right\rbrack}_{\mathbf{e}_1} + x_2 \underbrace{\left\lbrack\begin{array}{c} 0\\ 1\\ \vdots\\ 0\\ \end{array}\right\rbrack}_{\mathbf{e}_2} +\ ...\ + x_D \underbrace{\left\lbrack\begin{array}{c} 0\\ 0\\ \vdots\\ 1\\ \end{array}\right\rbrack}_{\mathbf{e}_D}$$

3 PCA Whitening

PCA Whitening corresponds to normalising (dividing) coordinates $\alpha$’s with their respective standard deviation (square root of their corresponding eigenvalue):

$$\alpha_i^W = \frac{\alpha_i}{\sqrt{\lambda_i}}$$

See Whitening transformation on Wikipedia for additional information.

Whitening (or batch normalisation) is a standard operation computed in Neural Networks (e.g. see Decorrelated Batch Normalization (CVPR 2018)).

4 DFFS and DIFS

A subspace $F$ is created using the first $d$ principal components associated with the $d$ highest eigenvalues with: $$0\leq d \leq D$$ The remaining subspace noted $F^{\perp}$ is described by the $(D-d)$ remaining eigenvectors.

$$\mathbb{R}^D= F \oplus F^{\perp}$$ and $$F \cap F^{\perp} =\lbrace 0 \rbrace$$ Any vector $\tilde{\mathbf{x}}\in\mathbb{R}^D$ centered on the mean $\overline{\mathbf{x}}$ can be decomposed into a vector in $F$ and another vector in $F^{\perp}$: $$\tilde{\mathbf{x}}= \underbrace{\sum_{i=1}^d \alpha_i \ \mathbf{u_{i}}}_{\tilde{\mathbf{x}}_F\in F} +\underbrace{\sum_{i=d+1}^D \alpha_i \ \mathbf{u_{i}}}_{\tilde{\mathbf{x}}_{F^{\perp}}\in F^{\perp}}$$

Question: Show $\|\tilde{\mathbf{x}}\|^2=\|\tilde{\mathbf{x}}_F\|^2+\|\tilde{\mathbf{x}}_{F^{\perp}}\|^2$

We define two distances as a measure of similarity between any vector $\mathbf{x}\in \mathbb{R}^D$ and a PCA-learned eigenspace $F$.

5 Relation to Multivariate Normal Distribution

The Multivariate Normal Distribution is a Probability Density Function defined that can be used for describing the distribution of a random vector $\mathbf{x}$ in $\mathbb{R}^D$.

Only 2 parameters are needed to compute the Multivariate Normal:

• $\pmb{\mu}$ its mean

• $\Sigma$ its covariance matrix

Having a set of observations (dataset $\lbrace \mathbf{x}_n\rbrace_{n=1,\cdots,N}$) for random vector $\mathbf{x}$, then:

• $\pmb{\mu}$ can be estimated by its sample mean $\overline{\mathbf{x}}$,

• $\Sigma$ can be estimated by the covariance matrix $\mathrm{S}$ computed with the dataset.

$$p(\mathbf{x})=\frac{1}{(\sqrt{2\pi})^D} \det(\mathrm{S})^{1/2} \exp\left(-\frac{1}{2} (\mathbf{x}-\overline{\mathbf{x}})^T\mathrm{S}^{-1} (\mathrm{x}-\overline{\mathrm{x}}) \right)$$ Since $\mathrm{S}=\mathrm{U}^T\Lambda\mathrm{U}$ with $\mathrm{U}=[\mathbf{u}_1,\cdots,\mathbf{u}_D]$ (eigenvectors) and $\Lambda$ the diagonal matrix with the eigenvalues: $$\Lambda= \left\lbrack \begin{array}{cccc} \lambda_1&0&0&0\\ 0&\lambda_2&0&0\\ \vdots&&\ddots&\\ 0&0&&\lambda_D\\ \end{array} \right\rbrack$$ We have (using properties of linear algebra with $\mathrm{U}$ and orthonormal matrix): $$\mathrm{S}^{-1}=\mathrm{U}^T\Lambda^{-1}\mathrm{U}$$ so we have $$\begin{array}{lcl} (\mathbf{x}-\overline{\mathbf{x}})^T\mathrm{S}^{-1} (\mathrm{x}-\overline{\mathrm{x}}) &=&(\mathbf{x}-\overline{\mathbf{x}})^T\mathrm{U}^T\Lambda^{-1}\mathrm{U} (\mathrm{x}-\overline{\mathrm{x}})\\ &=& \lbrack\mathrm{U}(\mathbf{x}-\overline{\mathbf{x}})\rbrack^T\Lambda^{-1}\lbrack\mathrm{U} (\mathrm{x}-\overline{\mathrm{x}})\rbrack\\ &=&\pmb{\alpha}^T\Lambda^{-1}\pmb{\alpha}\\ &=&\sum_{i=1}^D \frac{\alpha_i^2}{\lambda_i}\\ &\simeq& \underbrace{\sum_{i=1}^d \frac{\alpha_i^2}{\lambda_i}}_{DIFS} +\frac{1}{\rho}\ \underbrace{\sum_{i=d+1}^D \alpha_i^2}_{DFFS}\\ \end{array}$$ The term $(\mathbf{x}-\overline{\mathbf{x}})^T\mathrm{S}^{-1} (\mathrm{x}-\overline{\mathrm{x}})$ is the Mahalanobis distance in the space $\mathbb{R}^D$ and it can be approximated as a weighted sum of the DFFS and DIFS. The parameter $\rho$ can be approximated as the average of the eigenvalues that were left out: $$\rho=\frac{1}{D-d} \sum_{i=d+1}^D \lambda_i$$

See wikipedia example of a Multivariate Normal Distribution in $\mathbb{R}^2$