MLP regressor — a small net for a numeric target
An MLP regressor is the same stack of linear maps and nonlinearities as the classifier, with a single linear output and a squared-error loss. One hidden layer can represent a smooth curve that linear regression cannot. It can also represent a wiggly curve that fits 200 training targets and misses the next 50.
Standardize the columns and consider standardizing the target. The output unit is linear, but the hidden units saturate when their inputs are large, and a target in the tens of thousands makes the loss huge relative to a default learning rate. Early stopping on a validation split is the control that usually matters more than adding a second hidden layer.
Refit a few seeds. Non-convex training means the reported RMSE is one run's RMSE. If the seeds disagree by more than you can tolerate, the network is too wide for the table.
Settings that change this fit
- hidden layer sizes
- Capacity. A narrow layer underfits. A wide layer needs early stopping or weight decay or it will interpolate the training targets.
- learning rate
- Step size. If the loss jumps around, it is too large. If it is still falling slowly at the iteration cap, it is too small or the cap is.