Cross Validation in Machine Learning: Types, Steps, and Examples

Quick Overview:

• Cross validation in machine learning evaluates model performance by training and validating it across multiple data splits instead of relying on one train-test split.

• Common types include K-Fold, Stratified K-Fold, LOOCV, Leave-P-Out, Repeated K-Fold, Time Series, and Group K-Fold cross-validation.

• Different techniques are used for different datasets, such as Stratified K-Fold for classification and Time Series or Group K-Fold for structured data.

• In this blog you will explore cross validation in machine learning in detail along with its types, methods, advantages and disadvantages and implementation using Python.

Build the skills to drive technology and AI-led business transformation with the Executive Programme in Technology & AI Leadership. Gain practical insights to make informed technology decisions and lead AI initiatives. 

What Is Cross Validation in Machine Learning?

Cross validation in machine learning is a machine learning model evaluation technique which is used to check how well a model is performing on unseen data. Instead of relying on a single train-test split, the cross validation divides the dataset into two or more parts and then trains and evaluates the model so many times.

The cross validation techniques in machine learning​ gives a more reliable estimate of the model performance and helps to identify whether a model is overfitting or working well.

Now let’s understand how cross validation in machine learning works.

How Does Cross Validation in Machine Learning Work?

The common process of working of cross validation techniques in machine learning​ is dividing the available data into two or more subsets which are called as folds. The model is then trained on some of the folds and being tested on the remaining folds. The process repeats until every fold is used for testing.

Let’s take a cross validation in machine learning example for 5-fold cross validation:

In the above cross validation in machine learning diagram: 

1. The dataset is divided into five equal or nearly equal parts.

2. The model is trained on four folds.

3. The remaining fold is used for validation.

4. The process is repeated five times, with a different fold used for validation each time.

5. The scores from all five runs are averaged to get the final performance estimate.

Also Read: Machine Learning Algorithms: Working, Types and Examples

Cross-validation vs Simple Train-Test Split

A train-test split divides the dataset into two parts. One part is used to train the model, while the other is used for testing. Cross validation techniques in machine learning​ divides the data into multiple folds and performs several training and validation cycles.

The key difference between both are:

For smaller datasets, cross validation can provide a more dependable estimate of model performance.

Also Read: Linear Regression in Machine Learning: Types, Algorithm, and Implementation

Now that you understand what is cross validation in machine learning​, let’s understand what different types of cross validation in machine learning are there for different datasets and use cases.

Types of Cross Validation in Machine Learning

There are several types of cross validation in machine learning including K-Fold, Stratified K-Fold, LOOCV, Leave-P-Out and more. 

Let’s explore each of them in detail to find the appropriate method depending on the dataset structure and the type of problem you want to solve.

1. K Fold Cross Validation in Machine Learning

K fold cross validation in machine learning divides the dataset into K equal or nearly equal folds. The model uses K-1 folds for training and one fold for validation. This process continues until each fold has been used for validation once.

For example, with 5-fold cross validation, the model runs five times and produces five validation scores. The average of these scores represents the model's cross-validation performance.

2. Stratified K-Fold Cross-Validation

Stratified K-fold cross validation is mainly used for classification problems. It keeps approximately the same proportion of each class in every fold. This is particularly useful when the classes are imbalanced.

For example, if a dataset contains 70% positive and 30% negative samples, stratified sampling attempts to maintain a similar ratio across the folds.

Also Read: Decision Tree Algorithm in Machine Learning: How It Works, Types, and Examples

3. Leave-One-Out Cross-Validation (LOOCV)

In Leave-One-Out Cross-Validation, one data point is used for validation while all remaining data points are used for training. The process is repeated for every data point in the dataset.

For a dataset containing 100 observations, the model is trained and validated 100 times. This makes LOOCV computationally expensive for large datasets.

4. Leave-P-Out Cross-Validation

Leave-P-Out Cross-Validation leaves P observations out for validation and uses the remaining observations for training. The number of possible combinations can become very large as the dataset grows, making this method expensive for larger datasets.

For example, if P is 2, two observations are used for validation in each iteration.

Also Read: Supervised vs Unsupervised Learning: Differences, Examples, and Applications

5. Repeated K-Fold Cross-Validation

Repeated K-fold cross validation runs K-fold cross validation multiple times using different splits of the dataset. It can provide a more stable performance estimate but requires additional computation.

For example, 5-fold cross validation can be repeated three times. This produces 15 validation results that can be averaged.

6. Time Series Cross-Validation

Time series cross validation is designed for datasets where observations follow a time order, such as stock prices, sales, or weather data.

Unlike standard cross validation, it does not randomly mix past and future observations. Training data generally comes from earlier periods, while validation data comes from a later period.

This helps prevent information from the future from influencing model training.

Also Read: K Means Clustering in Machine Learning: Algorithm, Steps, Examples, and Applications

7. Group K Fold Cross Validation in Machine Learning

Group K fold cross validation in machine learning is useful when observations belong to groups and data from the same group should not appear in both training and validation sets. This helps reduce data leakage and provides a more realistic evaluation.

For example, if a dataset contains multiple records from the same customer, all records belonging to one customer can be kept within the same fold.

Now you also understand what different types of cross validation in machine learning are available for different use cases. Let’s see how to implement cross validation in machine learning python.

Implementation of Cross Validation in Machine Learning using Python

Cross validation in machine learning python can be implemented using scikit-learn, which compares a single train-test split against several of the cross-validation types covered above.

import numpy as np

import pandas as pd

from sklearn.model_selection import (

    KFold, StratifiedKFold, LeaveOneOut, LeavePOut, RepeatedKFold,

    TimeSeriesSplit, GroupKFold, cross_val_score, train_test_split

)

from sklearn.linear_model import LogisticRegression

from sklearn.preprocessing import StandardScaler

from sklearn.metrics import accuracy_score

# Build a sample loan-approval dataset

np.random.seed(42)

n = 200

income = np.round(np.random.uniform(20000, 120000, n))

credit_score = np.round(np.random.uniform(300, 850, n))

existing_loans = np.random.randint(0, 4, n)

score = (income / 120000) 0.5 + (credit_score / 850) 0.5 - existing_loans * 0.1

approved = ((score + np.random.normal(0, 0.1, n)) > 0.45).astype(int)

df = pd.DataFrame({

    "Income": income, "CreditScore": credit_score,

    "ExistingLoans": existing_loans, "Approved": approved

})

X = df[["Income", "CreditScore", "ExistingLoans"]].values

y = df["Approved"].values

scaler = StandardScaler()

X_scaled = scaler.fit_transform(X)

model = LogisticRegression()

Single train-test split, for comparison:

X_train, X_test, y_train, y_test = train_test_split(

    X_scaled, y, test_size=0.2, random_state=42, stratify=y

)

model.fit(X_train, y_train)

print("Single split accuracy:", accuracy_score(y_test, model.predict(X_test)))

Output: 

Single split accuracy: 0.875 

K Fold and Stratified K Fold cross validation in machine learning:

kf = KFold(n_splits=5, shuffle=True, random_state=42)

kf_scores = cross_val_score(model, X_scaled, y, cv=kf)

print("K-Fold scores:", kf_scores, "mean:", kf_scores.mean())

skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

skf_scores = cross_val_score(model, X_scaled, y, cv=skf)

print("Stratified K-Fold mean:", skf_scores.mean())

Output:

K-Fold scores: [0.9   0.85  0.975 0.875 0.825] mean: 0.885

Stratified K-Fold mean: 0.89

Repeated K-Fold cross-validation:

rkf = RepeatedKFold(n_splits=5, n_repeats=3, random_state=42)

rkf_scores = cross_val_score(model, X_scaled, y, cv=rkf)

print("Repeated K-Fold mean:", rkf_scores.mean(), "std:", rkf_scores.std())

Output: 

Repeated K-Fold mean: 0.89 std: 0.03958114029012641

Leave-One-Out and Leave-P-Out cross-validation (kept to a small subset, since both scale poorly):

small_idx = np.random.choice(n, 40, replace=False)

X_small, y_small = X_scaled[small_idx], y[small_idx]

loo = LeaveOneOut()

loo_scores = cross_val_score(model, X_small, y_small, cv=loo)

print("LOOCV mean accuracy:", loo_scores.mean())

tiny_idx = np.random.choice(n, 12, replace=False)

X_tiny, y_tiny = X_scaled[tiny_idx], y[tiny_idx]

lpo = LeavePOut(p=2)

lpo_scores = cross_val_score(model, X_tiny, y_tiny, cv=lpo)

print("Leave-P-Out mean accuracy:", lpo_scores.mean(), "total rounds:", lpo.get_n_splits(X_tiny))

Output:

LOOCV mean accuracy: 0.85

Leave-P-Out mean accuracy: 0.8636363636363636 total rounds: 66

Time series and Group K-Fold cross-validation:

tscv = TimeSeriesSplit(n_splits=5)

ts_scores = cross_val_score(model, X_scaled, y, cv=tscv)

print("Time series split mean:", ts_scores.mean())

groups = np.random.randint(0, 20, n)  # simulate 20 customer groups

gkf = GroupKFold(n_splits=5)

gkf_scores = cross_val_score(model, X_scaled, y, cv=gkf, groups=groups)

print("Group K-Fold mean:", gkf_scores.mean())

Output:

Time series split mean: 0.8606060606060606

Group K-Fold mean: 0.8955311355311355

Now that you have seen how different cross validation techniques in machine learning​ work, try implementing them yourself to get hands-on experience .

Also Read: Random Forest Algorithm in Machine Learning: Working, Types, Pseudocode

Exercise: Implement Cross Validation in Machine Learning Using Python

Dataset

 Use the Breast Cancer Wisconsin dataset available in scikit-learn. Your goal is to predict whether a tumour is malignant or benign.

Tasks

1. Load the dataset using load_breast_cancer().

2. Separate the features (X) and target (y).

3. Split the data into training and testing sets using train_test_split().

4. Train a LogisticRegression model and calculate the test accuracy.

5. Apply 5-Fold Cross-Validation using KFold.

6. Apply Stratified K-Fold Cross-Validation using StratifiedKFold.

7. Apply Repeated K-Fold Cross-Validation with 5 folds and 3 repeats.

8. Apply Leave-One-Out Cross-Validation on a smaller subset of the dataset.

9. Compare the mean accuracy obtained from each method.

10. Identify which cross-validation method gives the most consistent results based on the standard deviation.

Also Read: Gradient Boosting Algorithm in Machine Learning: How It Works, and Practical Examples

Starter Code

import numpy as np

from sklearn.datasets import load_breast_cancer

from sklearn.model_selection import (

    train_test_split,

    KFold,

    StratifiedKFold,

    RepeatedKFold,

    LeaveOneOut,

    cross_val_score

)

from sklearn.linear_model import LogisticRegression

from sklearn.preprocessing import StandardScaler

# Load the dataset

data = load_breast_cancer()

X = data.data

y = data.target

# Scale the features

scaler = StandardScaler()

X_scaled = scaler.fit_transform(X)

# Create the model

model = LogisticRegression(max_iter=1000)

# Write your code below

Questions to Answer

After running your code, answer these questions:

• What accuracy did you get from the single train-test split?

• What was the mean accuracy for K-Fold Cross-Validation?

• How did Stratified K-Fold compare with regular K-Fold?

• What was the mean accuracy and standard deviation for Repeated K-Fold?

• Why might Leave-One-Out Cross-Validation take longer to run?

• Which method produced the most stable results?

• Why is Stratified K-Fold useful for classification problems?

Also Read: Overfitting vs Underfitting in Machine Learning

Challenge

Modify the code to use Support Vector Machine (SVM) instead of Logistic Regression.

Compare the SVM and Logistic Regression results using Stratified K Fold Cross Validation in machine learning and report the mean accuracy for both models.

Now let’s explore some of the pros and cons of Cross validation in Machine Learning.

Advantages and Limitations of Cross Validation in Machine Learning

Cross validation provides a stronger evaluation approach than relying on a single data split. However, it also has some limitations.

Advantages of Cross Validation

• Provides a more reliable estimate of model performance.

• Uses different portions of the dataset for validation.

• Helps identify potential overfitting.

• Makes better use of smaller datasets.

• Can help compare different machine learning models.

• Supports model and hyperparameter selection.

Limitations of Cross Validation

• Requires more computation because the model is trained multiple times.

• Can take considerable time with large datasets or complex models.

• The wrong cross-validation method can lead to misleading results.

• Data leakage can still occur if preprocessing is performed incorrectly before cross validation.

• Time-series and grouped data require specialised validation strategies.

Also Read: Bias Variance Tradeoff in Machine Learning: Concept, Examples, and Techniques

Conclusion

So, cross validation in machine learning in simple words, is an important technique for evaluating how well a model can perform on unseen data. By training and validating a model across multiple data splits, it provides a more dependable performance estimate than a single train-test split.

Different cross validation in machine learning types, such as K-fold, Stratified K-fold, LOOCV, Time Series, and Group K-fold, are used for different types of datasets. Choosing the right method can help you evaluate models more effectively and avoid misleading results.