XGBoost regressor — boosted trees plus leaf regularization
XGBoost adds the same sequence of residual corrections as gradient boosting and penalizes the size of the leaf weights. The penalty lets you grow a slightly deeper tree without every leaf becoming a copy of one training target. Row and column subsampling per tree are available in the same fit, so the ensemble disagrees with itself on purpose.
For a continuous target the default squared error is usually right, and the first plot to make is predicted against actual, not a feature-importance bar. Importance will highlight a column the trees split on early; it will not show whether the predictions are biased high in one region.
Missing entries take a learned direction at each split. That is helpful when the missingness is random and harmful when missingness itself carries the signal and you would rather encode it as its own column.
Settings that change this fit
- lambda
- L2 penalty on leaf weights. Raise it when a deep tree matches the training residuals exactly.
- subsample
- Fraction of rows used per tree. 0.8 is a common damper. 1.0 uses every row every time and overfits sooner.