Random forest regressor — averaged trees for a numeric target
Each tree predicts a leaf mean, and the forest predicts the average of those means. Averaging is what cuts the variance. The bias of a shallow tree remains: if every tree is too shallow to capture a curve, the average is still too shallow. Depth has to be enough for one tree to be flexible; the forest's job is to make that flexibility less noisy.
Out-of-bag predictions are the average over trees that did not train on that row. They underestimate error when rows are grouped (several lines per customer) because a sibling row still entered the tree. Split by customer, not by line, or the out-of-bag number is flattery.
The forest cannot predict outside the range of training targets. Every leaf mean lies between the smallest and largest y it saw. A forecasting problem that must exceed the historical maximum needs a different model.
Settings that change this fit
- n estimators
- Trees in the average. Increase until the out-of-bag score stops moving.
- max features
- Columns tried per split. For regression, trying about a third of the columns is the usual starting point, not the square root used for classification.