KNN regressor — average the targets of nearby rows
KNN regression predicts the average target of the k nearest training rows. Distance weighting replaces that average with a weighted one so a next-door row counts more than a row at the edge of the neighborhood. There is no global slope. Two regions of the table can have completely different relationships between the columns and the target.
The prediction is always inside the range of the neighbors' targets, so the model cannot extrapolate a trend past the last observed row. At the edge of the data the neighbors are all on one side and the average is biased inward. That bias is the main reason a KNN score looks worse than a linear model on a problem that really is a straight line.
Scale the columns. Then choose k by validation error, not by the training error, which falls all the way to zero at k=1 because each row is its own neighbor.
Settings that change this fit
- k
- Neighborhood size. k=1 copies the nearest target. Large k averages in rows from other regimes.
- weights
- Uniform is a plain average. Distance gives nearby rows more of the average and usually helps at moderate k.