Gradient boosting regressor — trees aimed at the residual
Regression boosting fits the next tree to the gradient of the loss, which for squared error is the residual. Each tree is a small correction added to the running prediction, multiplied by the learning rate. The sum can follow a curve that no single shallow tree could draw, and it can also follow noise with the same enthusiasm.
Watch the validation loss by stage. It typically falls and then rises. The rising part is later trees fitting residuals that are noise. Early stopping at the bottom of that curve is the regularization that matters most; the subsample fraction, which fits each tree on a fraction of the rows, is the one that matters next.
Squared-error boosting is dominated by large residuals. If a few targets are orders of magnitude off, they become the residual the next tree chases. An absolute-error loss, or a cap on those targets, changes which rows the ensemble spends its capacity on.
Settings that change this fit
- learning rate
- Multiplier on each correction. Smaller values need more trees and stop the early trees from using up the whole residual.
- n estimators
- Number of corrections. Set it high and let early stopping choose, or you are guessing where the validation curve turned.