PCA — keep the directions with the most variance
Principal component analysis rotates the columns into a new set of axes. The first axis is the direction of greatest variance, the second is the greatest remaining variance at a right angle to the first, and so on. Projecting onto the first few axes is the dimensionality reduction. The loadings say which original columns make up each axis.
The axes are ordered by variance, not by usefulness for a later classifier. A rare class can sit along a low-variance direction that PCA throws away. If the projection is a prelude to prediction, check the downstream score against the number of components instead of stopping at 90% variance by habit.
Scale the columns unless they already share a unit. PCA on unscaled data is a ranking of column ranges. Also center the data; the algorithm assumes a mean of zero, and the implementation subtracts it.
Settings that change this fit
- n components
- How many axes to keep. The scree plot of explained-variance ratio is the picture that justifies the number.
- svd solver
- Full is exact on a small CSV. Randomized is an approximation that only pays off when both rows and columns are large.