Overfitting vs Underfitting in Machine Learning

Quick Overview:

• Overfitting vs underfitting describes two common problems in machine learning where a model either learns the training data too closely or fails to learn important patterns.

• Overfitting occurs when a machine learning model memorizes the training data, including noise and minor details, instead of learning general patterns.

• Underfitting occurs when a machine learning model is too simple to learn the key patterns in data, resulting in poor performance on both training and test data.

• In this blog you will explore overfitting vs underfitting in machine learning, their causes and signs, the bias-variance tradeoff, model complexity, and ways to improve model performance.

Overfitting vs Underfitting in ML: Key Differences

Before going in detail of each concept individually, here is a side-by-side comparison of how overfitting vs underfitting differ across the factors that matter most when diagnosing a model. 

Let’s understand the topic more clearly using a overfitting vs underfitting graph below:

Now let’s explore each overfitting vs underfitting​ in machine learning in detail.

Also Read: Linear Regression in Machine Learning: Types, Algorithm, and Implementation

What Is Overfitting in Machine Learning?

Overfitting occurs when  a machine learning model learns the training data too slowly. It will not just capture the genuine underlying pattern but also the noise, random variations or other details which are not required for the data.

The result is a model that performs extremely well on the data it was trained on but fails to generalize to new, unseen data, since it has essentially memorised the training set rather than learned from it.

Also Read: Supervised vs Unsupervised Learning: Differences, Examples, and Applications

Signs of an Underfitted Model

Some common signs of an underfitted model includes:

• Low training accuracy

• Low validation or test accuracy

• High training error

• High validation error

• Poor predictions on both training and unseen data

For example, if a ML model achieves only 65% accuracy on training data and 63% on test data, it may be underfitting because it has not learned the underlying patterns effectively.

Causes of Underfitting

Underfitting can occur because of:

• An overly simple model

• Insufficient or irrelevant features

• Excessive regularisation

• Inadequate training

• Poor feature engineering

• Incorrect model selection

Relationship Between Model Complexity and Overfitting

As model complexity increases, a model becomes capable of learning increasingly detailed patterns from the training data. Beyond an appropriate level of complexity, it may begin fitting noise rather than useful patterns.

This can cause training performance to keep improving while validation or test performance starts declining.

Also Read: Logistic Regression in Machine Learning: Algorithm, Types, Examples, and Applications

Training Accuracy vs Test Accuracy

A useful way to identify overfitting while discussing overfitting vs underfitting in machine learning is to compare training and test performance.

Example:

• Training accuracy: 98%

• Validation accuracy: 76%

• Test accuracy: 74%

The large difference suggests that the model performs well on data it has already seen but struggles with unseen observations. 

Real-World Impact of Overfitting

An overfitted model can be genuinely dangerous in production, since it often looks excellent during development and testing, only to underperform once it encounters real-world data that differs even slightly from the training set. This can lead to costly business decisions built on predictions that seem reliable in testing but do not hold up in practice, from mispriced financial risk models to medical diagnostic tools that fail on new patient populations. 

Understanding underfitting in detail is also important while discussing the difference  between overfitting vs underfitting machine learning​. Let’s explore what is underfitting.

Also Read: Decision Tree Algorithm in Machine Learning: How It Works, Types, and Examples

What Is Underfitting in Machine Learning?

Underfitting occurs when a model is too simple to capture the important patterns in the training data. Because the model does not learn enough from the available information, it performs poorly on both training and unseen data.

In overfitting vs underfitting machine learning, underfitting represents the opposite problem: the model has insufficient complexity or information to represent the underlying relationship in the data.

Signs of an Underfitted Model

Some common signs of underfitting of a ML model includes:

• Low training accuracy

• Low validation or test accuracy

• High training error

• High validation error

• Poor predictions on both training and unseen data

For example, if a model achieves only 65% accuracy on training data and 63% on test data, it may be underfitting because it has not learned the underlying patterns effectively.

Also Read: Random Forest Algorithm in Machine Learning: Working, Types, Pseudocode

Causes of Underfitting

Underfitting can occur because of:

• An overly simple model

• Insufficient or irrelevant features

• Excessive regularisation

• Inadequate training

• Poor feature engineering

• Incorrect model selection

Relationship Between Model Complexity and Underfitting

A model with very low complexity may not have enough flexibility to represent relationships within the dataset. Increasing model complexity appropriately can allow the model to capture more meaningful patterns.

However, simply making a model more complex is not always the right solution. Feature quality, data preparation, and model selection also affect performance.

Also Read: Gradient Boosting Algorithm in Machine Learning: How It Works, and Practical Examples

Training and Validation Performance

An underfitted model generally performs poorly on both training and validation datasets.

For example:

• Training accuracy: 68%

• Validation accuracy: 65%

• Test accuracy: 64%

Since the model cannot perform well even on its training data, it may not have learned enough useful information.

Real-World Impact of Underfitting

An underfitted model tends to be less dangerous than an overfitted one, since its poor performance is usually obvious early rather than hidden until deployment. Still, an underfit model provides little real business value, since it fails to capture the patterns needed to make useful predictions, which can mean missed opportunities, poor decision support, or simply wasted investment in a model that never becomes reliable enough to use. 

Overfitting and underfitting are two sides of the bias-variance tradeoff. Understanding overfitting vs underfitting bias variance helps explain why these two problems occur in machine learning. So now let’s explore what is a bias-variance tradeoff.

Also Read: Support Vector Machine (SVM) in Machine Learning: Algorithm, Types, Kernels, and Examples

Bias-Variance Tradeoff in Machine Learning

The bias-variance tradeoff helps explain the relationship between model complexity, overfitting, and underfitting. A model needs to balance bias and variance to generalize effectively to unseen data.

What Is Bias?

Bias refers to error caused by overly simplistic assumptions in a model. A model with high bias may fail to capture important relationships in the data. This is commonly associated with underfitting.

What Is Variance?

Variance refers to how much a model's predictions change when it is trained on different datasets. A model with high variance can become overly sensitive to the training data and may learn noise or random variations. This is commonly associated with overfitting.

Also Read: Machine Learning Algorithms: Working, Types and Examples

High Bias and Underfitting

High bias means the model is too simple or makes overly restrictive assumptions about the data.

For example, using a simple linear model to represent a highly complex non-linear relationship may result in underfitting.

High Variance and Overfitting

High variance means the model is highly sensitive to the training data.

A highly complex decision tree, for example, may learn very specific patterns from the training dataset and perform poorly on new observations.

Bias vs Variance

Conclusion

Now you know about overfitting vs underfitting in machine learning and how both affect model performance. Overfitting learns the training data too closely, while underfitting fails to learn important patterns.

By balancing model complexity, bias, and variance, you can build models that perform better on unseen data. Techniques such as regularisation, feature selection, and choosing the right model complexity can help achieve this balance.