Regularization & Generalization

Regularization & Generalization#

Regularization in ML#

Regularization is a technique to prevent overfitting by adding a penalty to the model’s complexity.

  • In ML, models (like Linear Regression, Neural Networks) may learn too much noise from training data → they fit training data perfectly but fail on unseen data.

  • Regularization reduces this by penalizing large weights.

Types of Regularization#

  • L1 (Lasso) → penalty = sum of absolute values of weights → forces some weights to zero (feature selection).

  • L2 (Ridge) → penalty = sum of squares of weights → shrinks coefficients but keeps all features.

  • Elastic Net → mix of L1 and L2.

👉 General equation (for regression):

\[ Loss = \text{MSE} + \lambda R(\beta) \]

Where:

  • \(\lambda\) = regularization strength

  • \(R(\beta)\) = penalty term (L1/L2/ElasticNet)

✅ Effect: keeps weights small → simpler model → less variance.

Generalization in ML#

Generalization is a model’s ability to perform well on unseen data, not just the training set.

Why is it important?#

  • A model that memorizes training data (overfits) has low generalization.

  • A model that is too simple (underfits) also fails to generalize.

Relationship with Regularization#

  • Regularization improves generalization by preventing the model from becoming too complex.

  • By shrinking weights, the model focuses on important patterns, not noise.

Visual Intuition#

  • Underfitting (high bias): model too simple → poor training & test accuracy.

  • Overfitting (high variance): model too complex → high training accuracy, poor test accuracy.

  • Good Generalization: balance of bias & variance → good test accuracy.

Regularization helps move from overfitting → toward better generalization.

✅ In short:

  • Regularization = technique to control complexity by penalizing large weights.

  • Generalization = model’s ability to work well on unseen data.

  • Regularization → improves generalization.

Contents