Decision tree regressor — predict with a piecewise constant
A regression tree splits the columns with thresholds, and each leaf predicts the mean of the training targets that landed there. The prediction is a step function. It will never extrapolate: a row beyond the training range of a column receives the mean of the nearest leaf, not a continuing slope.
Splits minimize squared error, so a leaf's mean is the right constant for that loss and a bad constant for absolute error. A few enormous targets will own the choice of threshold. If the target has heavy tails, those rows should be winsorized or the loss should change before the tree is grown.
Depth works the same way as in classification. A deep tree puts a different mean on every handful of rows and the training error goes to zero while the steps no longer repeat on new rows.
Settings that change this fit
- max depth
- How many thresholds a prediction can depend on. Shallow trees are coarse and stable. Deep trees are lookup tables.
- min samples leaf
- The smallest sample behind a predicted mean. A leaf of one row predicts that row's target and nothing else.