Bias, Variance and Overfitting

A model’s error on new data comes from two sources, and reducing one often increases the other.

Underfitting

Fitting a straight line to clearly curved data leaves a systematic error that no amount of extra data can remove.

Overfitting

A very flexible model, such as a high-degree polynomial, can pass through every training point, including the random noise. It then performs poorly on new data because it has learned the noise instead of the pattern.

Example: A simple model has a training error of 2 and a test error of 3. After adding many extra features, the training error falls to 0.5, but the test error rises to 6. The widening gap between the two errors signals overfitting.

Controlling It

• Train-test split: keep some data aside and measure the error only on that unseen portion.
• Cross-validation: repeat the split several times so the result does not depend on one lucky split.
• Regularization: add a penalty on large coefficients, as in ridge and lasso regression, to keep the model simpler.
• More data or fewer features: both reduce the room for the model to chase noise.