Choose the machine learning model.
Identify the hyperparameters that need tuning.
Define possible values or ranges for each hyperparameter.
Select a tuning method such as Grid Search or Random Search.
Train the model using different combinations.
Evaluate each combination using a suitable metric.
Select the best-performing combination.
Train the final model using the selected hyperparameters.
Evaluate the final model on unseen test data.
For example, imagine you are training a Random Forest model and want to tune the number of trees.
You could test:
50 trees
100 trees
200 trees
300 trees
The model can be evaluated for each value, and the value that performs best during validation can be selected.
If you are confused with Grid or Random search, do not worry, you will get to know about these hyperparameter tuning methods in machine learning in upcoming sections. Now let’s see what is the difference between Hyperparameters vs Parameters.
Hyperparameters vs Parameters in Machine Learning
Hyperparameters and parameters are different but both can influence model performance. These are some key differences between both:
Feature | Hyperparameters | Parameters |
When selected | Before or during the training setup | Learned during training |
Selected by | Data scientist or tuning process | Machine learning algorithm |
Examples | Learning rate, tree depth, number of trees | Weights and biases |
Purpose | Control the learning process | Learn patterns from data |
For example, in a neural network, the learning rate is a hyperparameter. The weights and biases are parameters that the model learns during training.
Now let’s explore what kind of hyperparameter tuning parameters are used by different machine learning algorithms.
Common Hyperparameters in Machine Learning
Different machine learning algorithms use different hyperparameters for hyperparameter tuning. The best ones depend on the model being trained.
1. Decision Trees
Decision trees use hyperparameters that control their structure and complexity.
Common hyperparameters include:
Maximum depth: It controls how deep the tree can grow. A very deep tree can capture complex patterns but may overfit the training data.
Minimum samples split: It defines the minimum number of samples required to split an internal node.
Minimum samples leaf: It defines the minimum number of samples that must be present in a leaf node.
2. Random Forest
Random Forest combines multiple decision trees to make predictions.
Common hyperparameters for hyperparameter tuning include:
Number of trees: It controls how many decision trees are used in the ensemble. Increasing this value can improve stability, but it also increases computation.
Maximum depth: It controls the depth of each tree.
Maximum features: It determines how many features can be considered when looking for a split.
3. K-Nearest Neighbors
K-Nearest Neighbors, or KNN, makes predictions based on nearby data points.
Important hyperparameters include:
Number of neighbours
Distance metric
Weighting method
The number of neighbours, represented by K, determines how many nearby observations influence the prediction. A small K can make the model more sensitive to individual observations, while a larger K can produce smoother predictions.
4. Support Vector Machine
Support Vector Machines use hyperparameters that influence the decision boundary.
Common hyperparameters include:
C: It controls the trade-off between having a simple decision boundary and correctly classifying training examples.
Kernel: It determines how the data is transformed to find a suitable decision boundary.
Gamma: It controls the influence of individual training examples for certain kernels, such as the RBF kernel.
5. Logistic Regression
Logistic regression also has several hyperparameters for hyperparameter tuning in machine learning that can be tuned.
Common examples include:
Regularization strength: It controls how strongly the model is penalised for large coefficients.
Solver: It determines the optimisation method used to train the model.
Penalty: It specifies the type of regularization.
6. Neural Networks
Neural networks have several hyperparameters that can significantly affect training.
Common examples include:
Learning rate: It controls how quickly the model updates its weights.
Batch size: It determines how many training examples are processed before an update.
Number of layers and neurons: It controls the network's architecture and capacity.
Number of epochs: It determines how many times the model goes through the training dataset.
Now that you have a good idea of what is hyperparameter tuning in machine learning and parameters used by different algorithms, let’s explore different types of hyperparameter tuning methods in machine learning.
Also Read: Linear Regression in Machine Learning: Types, Algorithm, and Implementation
Types of Hyperparameter Tuning Methods in Machine Learning
There are several methods for hyperparameter tuning in machine learning. Some are simple and require manual decisions, while others automate the search process. Below are some of the most used methods:
1. Manual Search
Manual search means a person selects a small number of hyperparameter combinations by hand, usually based on intuition or prior experience, trains and evaluates the model on each one, and picks the best performer. It requires no special tooling but does not scale well and depends heavily on the practitioner's judgment.
Code:
from sklearn.model_selection import cross_val_score
from sklearn.ensemble import RandomForestClassifier
manual_configs = [
{"n_estimators": 100, "max_depth": 5, "max_features": "sqrt"},
{"n_estimators": 200, "max_depth": 8, "max_features": "sqrt"},
{"n_estimators": 300, "max_depth": 10, "max_features": "log2"},
]
best_score, best_config = -1, None
for cfg in manual_configs:
model = RandomForestClassifier(random_state=42, **cfg)
score = cross_val_score(model, X_train, y_train, cv=5).mean()
print(cfg, "CV accuracy:", round(score, 4))
if score > best_score:
best_score, best_config = score, cfg
print("Best manual config:", best_config)
Testing the three configurations gave cross-validation accuracies of 0.7844, 0.7656, and 0.7719. The smallest configuration performed best, achieving 80% test accuracy compared with the 73.75% baseline.
Also Read: Supervised vs Unsupervised Learning: Differences, Examples, and Applications
2. Grid Search
Grid search exhaustively evaluates every possible combination of hyperparameter values within a specified grid, guaranteeing that the best combination within that grid will be found, at the cost of testing every combination even when many of them are clearly unpromising.
Code:
from sklearn.model_selection import GridSearchCV
param_grid = {
"n_estimators": [100, 200, 300],
"max_depth": [5, 8, 10, None],
"max_features": ["sqrt", "log2"]
}
grid = GridSearchCV(
RandomForestClassifier(random_state=42),
param_grid, cv=5, n_jobs=-1
)
grid.fit(X_train, y_train)
print("Best params:", grid.best_params_)
print("Best CV accuracy:", grid.best_score_)
The grid tests 24 combinations across 5 folds, resulting in 120 model fits. It found the same best settings as manual search, with 0.7844 cross-validation accuracy and 0.80 test accuracy.
3. Random Search
Random search samples a fixed number of random combinations from specified distributions or ranges of hyperparameter values, rather than testing every possible combination. It is often more efficient than grid search in practice, since it can explore a wider range of values for the hyperparameters that matter most without wasting evaluations on every combination of the ones that matter less.
Code:
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint
param_dist = {
"n_estimators": randint(50, 400),
"max_depth": randint(3, 15),
"max_features": ["sqrt", "log2", None]
}
random_search = RandomizedSearchCV(
RandomForestClassifier(random_state=42),
param_dist, n_iter=20, cv=5, random_state=42, n_jobs=-1
)
random_search.fit(X_train, y_train)
print("Best params:", random_search.best_params_)
print("Best CV accuracy:", random_search.best_score_)
Random search tested 20 combinations across 5 folds, resulting in 100 model fits. It achieved a cross-validation accuracy of 0.7844 and a test accuracy of 0.8125, using fewer combinations than grid search.
4. Bayesian Optimization for Hyperparameter Tuning
Bayesian optimization for hyperparameter tuning builds a probabilistic model of how hyperparameter values relate to performance, using the results of previous trials to intelligently choose the next combination to try, rather than sampling randomly or exhaustively. This generally finds strong configurations in fewer total trials than grid or random search, which matters most when each individual model training run is slow or expensive.
Code:
import optuna
from sklearn.model_selection import cross_val_score
def objective(trial):
n_estimators = trial.suggest_int("n_estimators", 50, 400)
max_depth = trial.suggest_int("max_depth", 3, 15)
max_features = trial.suggest_categorical("max_features", ["sqrt", "log2"])
model = RandomForestClassifier(
n_estimators=n_estimators, max_depth=max_depth,
max_features=max_features, random_state=42
)
return cross_val_score(model, X_train, y_train, cv=5).mean()
study = optuna.create_study(direction="maximize")
study.optimize(objective, n_trials=25)
print("Best params:", study.best_params)
print("Best CV accuracy:", study.best_value)
Optuna’s Bayesian optimization for hyperparameter tuning tested 25 trials, with about 125 total model fits. It achieved 0.7844 cross-validation accuracy and 0.80 test accuracy, matching grid search. Its main advantage is handling larger search spaces with fewer trials.
Also Read: Classification vs Regression in Machine Learning
5. Automated Hyperparameter Optimization
Automated hyperparameter optimization, sometimes bundled under the broader label of AutoML, goes a step further than any single method above by automatically managing the entire search process, including choosing the search strategy, allocating a time or trial budget, and in some tools, stopping clearly unpromising trials early before they finish training.
Code:
import optuna
def objective(trial):
n_estimators = trial.suggest_int("n_estimators", 50, 400)
max_depth = trial.suggest_int("max_depth", 3, 15)
max_features = trial.suggest_categorical("max_features", ["sqrt", "log2"])
min_samples_split = trial.suggest_int("min_samples_split", 2, 10)
model = RandomForestClassifier(
n_estimators=n_estimators, max_depth=max_depth,
max_features=max_features, min_samples_split=min_samples_split,
random_state=42
)
return cross_val_score(model, X_train, y_train, cv=5).mean()
# Automated pruning stops unpromising trials early
study = optuna.create_study(
direction="maximize",
pruner=optuna.pruners.MedianPruner()
)
study.optimize(objective, n_trials=50, timeout=120) # time-budgeted search
print("Best params:", study.best_params)
Adding a 120-second time limit and pruning lets the search run automatically within a fixed time. It also adds min_samples_split to the search. Unlike the earlier methods, you don't need to set the number of trials or combinations manually.
Tools such as Optuna, Hyperopt, Ray Tune, and Keras Tuner provide different approaches to automated hyperparameter optimization. Now let’s explore these tools used for hyperparameter tuning in machine learning.
Hyperparameter Tuning Tools
Several tools are available for performing hyperparameter tuning in ML projects. The right tool depends on the model, framework, search strategy, and computing requirements.
1. GridSearchCV
GridSearchCV is available in scikit-learn and performs an exhaustive search over the specified parameter grid.
It is useful when you have a relatively small number of hyperparameters and want to test every defined combination.
from sklearn.model_selection import GridSearchCV
It also supports cross-validation, making it useful for evaluating different combinations more systematically.
2. RandomizedSearchCV
RandomizedSearchCV is another scikit-learn tool. Instead of testing every combination, it samples a specified number of combinations.
from sklearn.model_selection import RandomizedSearchCV
It is useful when the search space is large and you want to limit the number of model evaluations.
3. Optuna
Optuna is a framework designed for automated hyperparameter optimization. It can explore a search space and use results from earlier trials to guide future trials.
It can be used with different machine learning and deep learning frameworks and supports features such as pruning, which can stop unpromising trials early.
4. Keras Tuner
Keras Tuner is designed for tuning machine learning models built with Keras.
It can search parameters such as:
Number of layers
Number of units
Learning rate
Dropout rate
Optimiser
For example:
import keras_tuner as kt
It can be useful when building and optimising neural network architectures.
5. Ray Tune
Ray Tune is a framework for distributed hyperparameter tuning. It can run multiple trials across available computing resources. It is useful for larger experiments where many models or hyperparameter combinations need to be evaluated.
Ray Tune can work with several machine learning frameworks and supports different search and scheduling strategies.
6. Hyperopt
Hyperopt is a Python library for optimising hyperparameters. It supports search techniques such as random search and Tree-structured Parzen Estimator-based optimisation.
It can be used when you need to define a search space and automatically explore different combinations.
7. Weights & Biases
Weights & Biases, also known as W&B, provides tools for tracking machine learning experiments and hyperparameter sweeps.
It can help record:
Also Read: K Means Clustering in Machine Learning: Algorithm, Steps, Examples, and Applications
Hyperparameter Tuning in Machine Learning With Cross-Validation
Cross-validation is commonly used together with hyperparameter tuning in machine learning to obtain a more reliable estimate of model performance.
Instead of evaluating a hyperparameter combination on just one validation split, cross-validation evaluates it across multiple folds.
For example, consider three learning rates:
With 5-fold cross-validation, each learning rate is evaluated across five folds. The average validation score can then be used to compare the three choices which reduces the dependence on one particular train-validation split.
Also Read: Reinforcement Learning (RL) in Machine Learning: How It Works, Algorithms and Types
Now let’s see some of the advantages and disadvantages of hyperparameter tuning.
Advantages and Disadvantages of Hyperparameter Tuning
Hyperparameter tuning can improve model performance, but it also requires careful planning and computational resources.
Advantages of Hyperparameter Tuning
It can improve model performance.
It helps find suitable model configurations.
It can reduce overfitting when appropriate settings are selected.
It supports systematic model comparison.
It can make better use of cross-validation.
Automated tools can reduce manual experimentation.
It helps identify how different hyperparameters affect model behaviour.
Limitations of Hyperparameter Tuning
It can require significant computational resources.
Large search spaces can take a long time to explore.
Grid Search can become expensive as the number of combinations increases.
The results depend on the selected search space and evaluation metric.
Poor validation strategies can produce misleading results.
Excessive tuning can lead to overfitting on the validation process.
Different models may require different tuning strategies.
Conclusion
So, what is hyperparameter tuning meaning in simple terms, it is an important part of developing machine learning models. It involves testing different hyperparameter values and selecting the settings that help the model perform well on validation data.
Methods such as Manual Search, Grid Search, Random Search, Bayesian Optimisation, and automated optimisation provide different ways to explore the search space. Tools such as GridSearchCV, RandomizedSearchCV, Optuna, Keras Tuner, and Ray Tune can make this process easier.
