Truncated SVD — reduce a matrix you should not center
Truncated SVD keeps the largest singular components of the matrix as it stands, without subtracting the mean. That is the right decomposition for a sparse matrix of counts or TF-IDF weights: centering would fill in the zeros and destroy the sparsity. On a dense, centered table it is close to PCA. On a sparse table PCA is the wrong tool and this is the right one.
Latent semantic analysis is this decomposition applied to a term-by-document matrix. The components are topics only in the loose sense that words which co-occur end up with similar weights. They are not a clustered vocabulary, and the sign of a component is arbitrary.
The singular values drop off. Components past the elbow add reconstruction of noise, which for text means rare co-occurrences. Pick the cutoff from that curve, then check a few reconstructed rows to see whether the surviving components still say something.
Settings that change this fit
- n components
- How many singular vectors are stored. This caps both the reduction and the cost.
- algorithm
- Randomized is the practical choice past a few hundred columns. ARPACK is exact enough to prefer on a small dense CSV.