XGBoost classifier — boosted trees with a penalty
XGBoost runs the boosting loop and adds an explicit penalty on leaf weights, plus column and row subsampling. The penalty is why a deep XGBoost tree overfits less than an equally deep ordinary boosted tree. Missing values are sent down a learned default direction, so you do not have to impute before training.
The controls that move accuracy on a typical CSV are the learning rate, the number of rounds, max depth, and min child weight. Min child weight refuses to create a leaf supported by too little hessian, which on a rare class stops the model from building a leaf for three noisy rows. Subsample and colsample add the same kind of disagreement a forest gets from bootstrapping.
Scale pos weight matters only for a rare positive class. It rebalances the loss; it does not change the tree structure by itself. Set it from the class ratio, then check precision and recall separately. Accuracy will hide a model that never predicts the rare class.
Settings that change this fit
- eta
- Learning rate per round. Lower eta wants more rounds. The pair should be tuned together, not one at a time.
- min child weight
- Minimum hessian in a leaf. Raise it when the training metric is perfect and the validation metric is not.