2.1 Data

Consider that we have a set of vectors \(\lbrace \mathbf{x}_n \rbrace _{n=1,\cdots ,N}\) in \(\mathbb{R}^D\). This means that any vector \(\mathbf{x}_n\) has \(D\) coordinates as follows: \[ \mathbf{x}_n = \left\lbrack \begin{array}{c} x_n[1]\\ x_n[2]\\ \vdots\\ x_n[D]\\ \end{array} \right\rbrack \]

2.2 Mean

We can define the mean \(\overline{\mathbf{x}}\) such that \[ \overline{\mathbf{x}}=\frac{1}{N}\sum_{n=1}^{N} \mathbf{x}_n \] Spatially, the mean can be understood as the center of gravity of the clouds of points \(\lbrace \mathbf{x}_n \rbrace _{n=1\cdots N}\).

\(\overline{\mathbf{x}}\) is a also vector of dimension \(D\): \[ \overline{\mathbf{x}} = \left\lbrack \begin{array}{c} \frac{1}{N} \sum_{n=1}^N x_n[1]\\ \frac{1}{N} \sum_{n=1}^N x_n[2]\\ \vdots\\ \frac{1}{N} \sum_{n=1}^N x_n[D]\\ \end{array} \right\rbrack \]

2.3 Covariance matrix

The covariance matrix \(\mathrm{S}\) is defined as: \[ \mathrm{S}=\frac{1}{N} \sum_{n=1}^{N} (\mathbf{x}_n-\overline{\mathbf{x}}) (\mathbf{x}_n-\overline{\mathbf{x}})^{T} \]

with defining \(\mathbf{\tilde{x}}_n=\mathbf{x}_n-\mathbf{\overline{x}}\), \(\forall n\) then \[ \mathrm{S}=\frac{1}{N} \sum_{n=1}^{N} \mathbf{\tilde{x}}_n \ \mathbf{\tilde{x}}_n^T \] or \[ \mathrm{S}=\frac{1}{N} \sum_{n=1}^N \left \lbrack \begin{array}{cccc} (\tilde{x}_{n}[1])^2 & (\tilde{x}_{n}[1]\times \tilde{x}_{n}[2]) &\cdots &(\tilde{x}_{n}[1] \times \tilde{x}_{n}[D])\\ &&&\\ (\tilde{x}_{n}[2]\times \tilde{x}_{n}[1])& (\tilde{x}_{n}[2])^2 &&\\ &&&\\ &&\ddots&\\ &&&\\ &&& (\tilde{x}_{n}[D])^2 \\ \end{array} \right\rbrack \]

Defining the \(D\times N\) matrix \(\mathrm{\tilde{X}}=[\tilde{x}_{1},\cdots,\tilde{x}_{N}]\) then the covariance matrix is \[ \mathrm{S}=\frac{1}{N} \mathrm{\tilde{X}}\mathrm{\tilde{X}}^T \] The covariance matrix \(\mathrm{S}\) is of size \(D\times D\).

3.1 Finding the first principal component

The solution is such that the derivatives of \(\mathcal{E}\) are zeros, so we need to find \((\mathbf{u}_1,\lambda_1)\) such that:

\[ \frac{\partial\mathcal{E}}{\partial \mathbf{u}_1}=0 \quad (\star) \] and \[ \frac{\partial\mathcal{E}}{\partial \lambda_1}=0 \quad (\star \star) \]

3.2 How to find the direction with second most maximum variance?

Once we have the direction \(\mathbf{u}_1\) with its associated eigenvalue \(\lambda_1\) (variance of the projections of centered data on \(\mathbf{u}_1\)), the direction \(\mathbf{u}_2\) with second maximum variance can be found such that: \[ \mathbf{u}_2^T\mathrm{S}\mathbf{u}_2 \] is maximised with the constraints \(\|\mathbf{u}_2\|=1\) and \(\mathbf{u}_2^T\mathbf{u}_1=0\) (\(\mathbf{u}_2\) is orthogonal to \(\mathbf{u}_1\)).

4.1 Computing PCA

From a set of vectors \(\lbrace \mathbf{x}_n \rbrace_{n=1,\cdots,N}\)

1. Compute the mean vector \(\overline{\mathbf{x}}\in \mathbb{R}^D\)
2. Center each vector observation \(\mathbf{\tilde{x}}_n=\mathbf{x}_n-\overline{\mathbf{x}}\)
3. Compute the covariance matrix \[ \mathrm{S}=\frac{1}{N} \sum_{n=1}^{N} \mathbf{\tilde{x}}_n \mathbf{\tilde{x}}_n^T \]
4. Compute the eigenvectors of \(\mathrm{S}\) and sort them from the one associated with the highest eigenvalue , to the one associated with the lowest eigenvalue.

\[ \text{eigenvectors} \quad \equiv \quad \text{Principal components} \]

Note the \(D\) principal components \(\mathrm{U}=\lbrack\mathbf{u}_1, \mathbf{u}_2, \cdots, \mathbf{u}_D\rbrack\) of the covariance matrix \(\mathrm{S}\) are sorted such that:
\[ \lambda_{1}\geq \lambda_{2} \geq \cdots \geq \lambda_{D} \geq 0 \] with \(\lambda_n\) the eigenvalue associated with eigenvector \(\mathbf{u}_n\), \(\forall n\).

4.2 Using PCA

Each vector in the set can be written as a linear combination of the mean and the eigenvectors: \[ \mathbf{x} = \overline{\mathbf{x}} +\sum_{j=1}^D \alpha_j\ \mathbf{u}_j \] Principal components associated with highest \(d\) eigenvalues provide a good reconstruction: \[ \mathbf{x} = \overline{\mathbf{x}} +\underbrace{\sum_{j=1}^d \alpha_j\ \mathbf{u}_j}_{\text{reconstruction}} +\underbrace{\sum_{j=d+1}^D \alpha_j\ \mathbf{u}_j}_{\text{reconstruction error} } \]