Random forest classifier — many trees, one vote
A random forest fits a crowd of trees. Each tree sees a bootstrap sample of the rows and, at every split, a random subset of the columns. The class is the majority vote. The randomization is the point: trees that all split on the same obvious column do not become wiser when you average them, and column subsampling stops that.
Out-of-bag rows, the ones left out of a tree's bootstrap sample, are a free test set for that tree. The out-of-bag error is an honest estimate as long as the rows are independent. It is not honest when several rows are repeated measurements of the same object; the forest will have seen the twin.
Feature importance from impurity drop favors columns with many possible cut points. A continuous column beats a binary column of equal usefulness. Permuting a column and watching the vote change is the check that does not have that bias.
Settings that change this fit
- n estimators
- How many trees vote. Past a few hundred the vote stabilizes and extra trees only cost time.
- max features
- Columns considered at each split. Sqrt is the usual classification setting. Using every column makes the trees copies of each other.