Gradient boosting classifier — each tree fixes the last
Gradient boosting starts with a constant prediction and adds a shallow tree that steps the predictions toward the labels, then another tree on the remaining error, and so on. The learning rate shrinks each step. A low rate with many trees usually beats a high rate with few trees, because each individual tree stays too weak to memorize a row.
Early stopping watches a validation set and halts when that set stops improving. Without it, extra trees keep fitting noise. The staged predictions are useful on their own: plot validation loss against the number of trees and the curve tells you where the ensemble peaked.
Boosting is sensitive to label noise in a way a random forest is not. One mislabeled row keeps attracting later trees. If the labels are messy, cap the depth at 2 or 3 and keep the learning rate small.
Settings that change this fit
- learning rate
- Shrinkage on each tree. 0.1 is a common start. 0.01 needs more trees and usually generalizes better.
- max depth
- Depth of each weak tree, not of the ensemble. Depth 1 is a stump. Depth 6 is already a strong learner and easier to overfit.