Basic Math for Principal Component Analysis

To begin, Principle Component Analysis (PCA) is a statistical tool for exploratory analysis that is famous for reducing the dimensionality of the data. It is a technique that involves vector space transformation to represent data in another form. It is useful for feature extraction and feature elimination, removing redundant features. The outcome of a PCA operation is a set of orthogonal variables, called the principal components. To learn PCA from a technical angle, I recommend the work by J Shlens (ref.[1]).

1. Intro to Matrix Operations
Before going into the math, there are a few technical terms in matrix algebra we have to be familiar with. First, eigenvector and eigenvalue. An eigenvector v is of a linear transformation (or simply a square matrix A) is a non-zero vector that changes only by a scalar factor (i.e. by the eigenvalue, λ) when that transformation is applied to it. We can say, [matrix].[eigenvector] = [eigenvalue].[eigenvector]; or we can write: Av = λv.  If the eigenvalue is negative, the direction is reversed. Refer to the figure on the right for a geometric description.

An eigenvector, with a real nonzero eigenvalue, points in a direction that is stretched by the transformation and the eigenvalue is the factor by which it is stretched.

Another term is about an orthogonal matrix. Let a square matrix A is invertible (it can be inversed) or non-singular.

A square matrix is orthogonal with real entries if, and only if, its transpose (A^T) is equal to its inverse (A^-1), that is AA^T yields an identity matrix I.  The square matrix is also normal if it satisfies A^TA = AA^T.

Importantly, if the matrix is orthogonal then it is necessarily invertible, unitary, and normal. The determinant of an orthogonal matrix is either +1 or −1. Next, we learn about another special matrix called a diagonal matrix, i.e. a square matrix with non-zero diagonal elements and all-zero remaining (non-diagonal elements). When a square matrix A is diagonalizable, there exists an invertible matrix B (a matrix that can have an inverse) such that B^-1AB produces a diagonal matrix.

Matrix S is symmetric only if it has a characteristic of S^T = S. Such matrix is also orthogonally (orthonormally) diagonalisable.

An example of a square matrix related to PCA is called the covariance matrix. More specifically, in this matrix we have the variances on the diagonal entries of this matrix, and the covariances on the off-diagonal entries. Take a look below. Here, the red box denotes the variance of variable height, whereas the green box shows the covariance between the score and age.

Suppose we have a huge dataset which has too much information. Can we re-express the original dataset optimally in a new form? Now the word 'optimality' means we do not want correlated or redundant features. After a series of matrix operations, the new or transformed dataset will have a new coordinate system. The whole idea of PCA is to force as much of the variation as possible into fewer dimensions, you can throw away the rest without losing much information. Let's relate PCA with the previous ideas on matrices:

In PCA, each eigenvector is a unit vector pointing in the direction of a new coordinate axis. The axis with the highest eigenvalue is said to be the axis that explains the largest variation, meaning an eigenvalue represents variance. The directions with the largest variances are the most “important” or principal. 

2. Technical Definition

Suppose we have a dataset of m × n matrix X, where the n columns are the number of observations and the m rows are the variables. We wish to linearly transform this matrix X into another matrix Y, also of dimension m × n, so that for some m × m transformation matrix P, we will obtain Y = PX.

So, PCA attempts to express correlated m variables into uncorrelated new variables through some sort of transformation. In matrix algebra, all possible correlations of any variable-pairs can be captured using the covariance matrix C_X. How do we decouple the variables? It turns out, we can get this by maximizing the variances or diagonal elements, and minimizing the co-variances or off-diagonal entries of the covariance matrix. So our goal is to obtain a new data Y such that the covariance matrix of the transformed data C_Y is strictly diagonal. To put it in another way, if we can find the transformation matrix P in such a way that C_Y is diagonal, then our objective is met.

The solution is based on an important property of matrix algebra on C_X called the eigen-decomposition. The effort to make C_Y a diagonal matrix is essentially to find P = E^T. How to get this new matrix? It turns out that the orthonormal matrix E can be found by diagonalizing the covariance matrix of the original dataset, C_X = EDE^T , where E is the matrix of eigenvectors which applies the transformation on X; and matrix D has diagonal elements containing the eigenvalues in descending order. In short, the principal components are formed by:
- The “new directions” of our data as indicated by the eigenvectors;
- The “magnitude”, or importance of each direction, as indicated by the eigenvalues.

Often, the normalized eigenvectors by the variance are also called the coefficients or loadings of the principal components. Remember, these coefficients define how the new principal component is formed as a linear combination of the original variables. Each column in the eigenvector matrix contains coefficients for each component in the descending order of the variance.

Dimensionality reduction works as follows. Suppose our original data is an m × n matrix X with m dimension and n data points.

If we want to transform the points to k dimensions (where, k < m), then select the first k eigenvectors of the matrix C_X sorted decreasingly according to the eigenvalues, and form a matrix with them, then use them as k × m matrix P. The resulting matrix is Y with size (k × m)(m × n) = k × n.

3. Closing remarks

PCA does not simply do a rotation. Normalization is always involved prior to PCA, which can be done by mean-centering the data and divide them with the standard deviation. This is important if our data contain variables of different scales, e.g. kilogram and pounds.

PCA and ICA are two slightly different things. While PCA expresses the data into a set of orthogonal variables, ICA simply represents data into an independent set of variables (not necessarily orthogonal). In ICA, the order of importance is irrelevant. Prior to ICA, data whitening is performed and PCA is one method to whiten the data. The figure on the right is common on the internet. PCA is predominantly used as a dimensionality reduction technique, for example, in computer vision, facial recognition, and image compression. It is also used for finding patterns in a high dimensional dataset in the field of finance, data mining, biomechanics, bioinformatics, etc.

References:
(1) "A tutorial on Principal Components Analysis" by J. Shlens (2014). Very nice technical tutorial!
(2) https://towardsdatascience.com/a-step-by-step-explanation-of-principal-component-analysis-b836fb9c97e2
(3) https://stats.stackexchange.com/questions/612/is-pca-followed-by-a-rotation-such-as-varimax-still-pca