Feature Selection in Machine Learning: Methods, Techniques & Choosing the Right Features

Quick Overview:

• Feature selection helps to identify the relevant features and remove irrelevant or redundant variables to simplify machine learning models and improve computational efficiency.

• The three main methods are filter, wrapper, and embedded methods, with techniques such as correlation, chi-square, mutual information, RFE, Lasso, and tree-based importance.

• Python libraries such as scikit-learn provide techniques including SelectKBest, RFE, Lasso, and SelectFromModel for selecting relevant features.

• In this blog you will learn feature selection in machine learning in detail along with methods, techniques and how to select the right method for your use case.

Develop practical skills in data science, machine learning, and agentic AI with the Executive Post Graduate Certificate in Applied Data Science & Agentic AI. Learn to work with real-world data, build AI solutions, and apply agentic AI across business use cases. 

What Is Feature Selection in Machine Learning?

Feature selection in machine learning is a process of selecting the most useful features from a dataset for training and building a machine learning model. It involves keeping the useful input variable and removing the features that are irrelevant, redundant or contain some useful information.

A dataset can have hundreds or thousands of features, but every feature does not contribute equally to the prediction. Selecting the right features can simplify the model, reduce computational requirements, and improve its ability to generalize to new data.

For example, consider a dataset which is used to predict house prices. It may contain features such as:

• Location

• Area

• Number of bedrooms

• Number of bathrooms

• Property age

• Distance from public transport

• Owner name

• Property ID

Features such as area, location, and number of bedrooms can help to predict house prices. However, the owner's name or property ID can not provide useful predictive information.

Feature selection in machine learning helps to identify the features that contribute meaningfully to the target variable.

The mathematical necessity of feature selection was first formalized by mathematician Richard Bellman in 1961, who gave the term "The Curse of Dimensionality." Bellman mathematically proved that as you add more features (dimensions) to your dataset, the volume of the mathematical space grows so fast that the available data becomes sparse.

Why Is Feature Selection Important?

Feature selection is useful for a dataset containing a large number of variables. Removing unnecessary features can make the machine learning workflow more efficient.

Key benefits of feature selection in machine learning include:

• Reduced model complexity: Fewer features can make a model simpler.

• Faster training: Models may require less time and computational power when processing fewer variables.

• Reduced noise: Removing irrelevant features can help the model focus on useful information.

• Lower risk of overfitting: Removing unnecessary variables can reduce the model's tendency to learn noise.

• Better interpretability: A model based on fewer meaningful features can be easier to understand.

• Reduced storage requirements: Fewer features can reduce the amount of data that needs to be processed and stored.

• Improved generalization: Carefully selected features can help a model perform more consistently on unseen data.

Also Read: Overfitting vs Underfitting in Machine Learning

How Does Feature Selection Work?

Feature selection generally follows a series of steps:

1. Understand the dataset: Examine the available features, target variables, data types, and missing values.

2. Explore the features: Analyse distributions and relationships between features and the target.

3. Identify irrelevant features: Look for variables that provide little useful information.

4. Identify redundant features: Find features that provide similar or overlapping information.

5. Choose a feature selection method: Select a suitable filter, wrapper, or embedded method.

6. Select the features: Apply the chosen technique to create a smaller feature set.

7. Train the model: Build the machine learning model using the selected features.

8. Evaluate performance: Compare model performance using suitable validation techniques.

As of now the feature selection is much clearer for you, now let’s explore what feature selection methods are available for different datasets and use cases.

Types of Feature Selection Methods

Feature selection methods can be divided into three main categories including filter methods, wrapper methods and embedded methods. Let’s explore each of these feature selection method in machine learning​ in detail.

1. Filter Methods

2. Wrapper Methods

3. Embedded Methods

Each category uses a different approach to identify and select the most relevant features.

1. Filter Methods

Filter methods select features based on their statistical properties. They evaluate the relationship between features and the target variable without training a specific machine learning model.

Common Filter Methods

• Correlation

• Chi-square test

• ANOVA

• Mutual information

• Variance threshold

Key Characteristics

• Fast and simple to apply

• Suitable for datasets with many features

• Usually performed before model training

• Do not depend on a specific machine learning algorithm

Now explore Filter feature selection methods in detail.

(a) Correlation-Based Feature Selection

Correlation measures the relationship between numerical variables. If two features are highly correlated, they may contain similar information. You can remove one of them to reduce redundancy.

Example:

A house price dataset may contain:

• House area in square feet

• House area in square metres

Both features represent almost the same information. Keeping both may not add much value.

Limitation: Correlation mainly captures specific types of relationships and may miss complex non-linear relationships.

(b) Chi-Square Test

The chi-square test is commonly used with categorical features in classification problems. It checks whether there is a statistical association between a feature and the target variable.

Example:

For a model that predicts whether a customer will purchase a product, features such as:

• Payment method

• Customer segment

• Subscription type

Features showing stronger associations with the target can be considered for selection.

Also Read: Supervised vs Unsupervised Learning: Differences, Examples, and Applications

(c) ANOVA

ANOVA stands for Analysis of Variance. It can be used to examine the relationship between:

• A numerical feature

• A categorical target variable

It checks whether the mean value of a feature differs significantly across different classes.

Example:

If a model predicts customer segments, ANOVA can help determine whether variables such as spending amount differ significantly between those segments.

(d) Mutual Information

Mutual information measures how much information one variable provides about another. Unlike simple correlation, it can identify certain non-linear relationships between a feature and the target.

Key point:

• Correlation mainly captures specific statistical relationships.

• Mutual information can detect a wider range of dependencies.

This makes mutual information useful when the relationship between a feature and target is not straightforward.

(e) Variance Threshold

Variance threshold removes features with very low variance. A feature that has almost the same value across all observations may provide limited information to the model.

Example:

If 99.9% of records have the same value for a feature, that feature may have limited predictive value.

2. Wrapper Methods

Wrapper methods select features by testing different combinations of features using a machine learning model.

Instead of evaluating each feature independently, they check how different feature subsets affect model performance.

Common Wrapper Methods

• Forward selection

• Backward elimination

• Recursive Feature Elimination (RFE)

• Recursive Feature Elimination with Cross-Validation (RFECV)

Key Characteristics

• Evaluate feature combinations

• Use a machine learning model during selection

• Can produce feature subsets suited to a particular model

• Usually require repeated model training

• Can be computationally expensive for large datasets

Also Read: Classification vs Regression in Machine Learning

Now explore wrapper feature selection methods in machine learning in detail.

(a) Forward Selection

Forward selection starts with no features. The process works as follows:

1. Start with an empty feature set.

2. Test individual features.

3. Select the feature that gives the best improvement in model performance.

4. Add another feature that improves performance the most.

5. Repeat the process.

6. Stop when the desired number of features is reached or additional features no longer provide enough improvement.

(b) Backward Elimination

Backward elimination starts with all available features. The process works as follows:

1. Train the model using all features.

2. Identify the feature contributing the least to model performance.

3. Remove that feature.

4. Train the model again.

5. Repeat the process until the desired number of features remains.

(c) Recursive Feature Elimination (RFE)

Recursive Feature Elimination, or RFE, repeatedly trains a model and removes the least important features.

Example:

If a dataset contains 20 features and you want to select five:

1. Train the model using all 20 features.

2. Identify the least important features.

3. Remove the least important features.

4. Retrain the model.

5. Continue until five features remain.

RFE can work with different machine learning estimators.

(d) Recursive Feature Elimination with Cross-Validation (RFECV)

RFECV extends RFE by using cross-validation to determine a suitable number of features.

Instead of manually deciding how many features to keep, RFECV:

1. Removes less important features recursively.

2. Evaluates different feature counts.

3. Uses cross-validation to compare model performance.

4. Selects a suitable feature count based on the chosen evaluation metric.

Also Read: Reinforcement Learning (RL) in Machine Learning: How It Works, Algorithms and Types

3. Embedded Methods

Embedded methods perform feature selection during model training. The learning algorithm itself identifies which features contribute to the model.

Common Embedded Methods

• Lasso regression

• Decision trees

• Random Forest

• Gradient boosting models

Key Characteristics

• Feature selection happens during model training

• Usually more model-specific than filter methods

• Can reduce the need for a separate feature selection step

• Feature importance or model coefficients can help identify relevant features

(a) Lasso Regression

Lasso stands for Least Absolute Shrinkage and Selection Operator. It uses L1 regularisation to penalise model coefficients.

During training, some coefficients can become exactly zero.

Example:

If a model produces:

• Feature A: 2.5

• Feature B: 0

• Feature C: -1.8

• Feature D: 0

Features B and D can be excluded because their coefficients are zero. Lasso is mainly useful for feature selection in suitable linear models.

(b) Decision Trees

Decision trees perform feature selection while creating splits in the data. At each node, the algorithm searches for a feature and split that best separates the data according to its splitting criterion.

The trained tree can also provide feature importance scores. These scores help identify which features contributed to the tree's predictions.

(c) Random Forest

Random Forest combines multiple decision trees. It can provide feature importance scores that indicate how features contribute to the model.

Features with very low importance may be considered for removal.

The importance scores should be interpreted carefully because the results can vary based on:

• Dataset characteristics

• Correlated features

• Importance measure used

• Model configuration

(d) Gradient Boosting

Gradient boosting builds a sequence of models, with each new model working to improve the predictions made by earlier models.

Many gradient boosting implementations provide feature importance measures. These measures can help identify features that contribute to predictions.

Also Read: Hyperparameter Tuning in Machine Learning: Methods, Techniques, and Tools

As of now you have a clear understanding of feature selection method in machine learning​. Now let’s compare these feature selection techniques in machine learning. 

Feature Selection Techniques Compared

Different feature selection techniques have different strengths and limitations. Let’s understand the difference between these feature selection techniques: 

After comparing these feature selection method in machine learning, let’s understand how to implement these using Python.

Also Read: Bias Variance Tradeoff in Machine Learning: Concept, Examples, and Techniques

Feature Selection in Python

Python provides several libraries for feature selection, with feature selection sklearn offering many commonly used techniques. You can apply these methods to a dataset, select the most relevant features, and use the reduced dataset for model training.

Below is a step-by-step example using a sample classification dataset.

Sample Dataset

Let's create a small dataset containing customer information. The target variable, Purchased, indicates whether a customer purchased a product.

import pandas as pd

data = {

    "Age": [22, 25, 28, 35, 40, 45, 30, 27, 50, 38],

    "Income": [25000, 30000, 35000, 50000, 65000, 70000, 45000, 32000, 80000, 55000],

    "SpendingScore": [80, 75, 70, 60, 45, 40, 65, 72, 35, 55],

    "VisitsPerMonth": [8, 7, 6, 5, 3, 2, 5, 7, 2, 4],

    "Tenure": [1, 2, 2, 4, 6, 7, 3, 2, 8, 5],

    "Purchased": [1, 1, 1, 1, 0, 0, 1, 1, 0, 0]

}

df = pd.DataFrame(data)

print(df)

Output:

Separate the input features and target variable:

X = df.drop("Purchased", axis=1)

y = df["Purchased"]

print(X.shape)

Output:

(10, 5)

The dataset has 10 records and 5 input features.

1. Feature Selection Using SelectKBest

For a classification problem, you can use f_classif to calculate ANOVA F-scores.

from sklearn.feature_selection import SelectKBest, f_classif

selector = SelectKBest(score_func=f_classif, k=3)

X_selected = selector.fit_transform(X, y)

print(X_selected.shape)

Output:

(10, 3)

The output contains 10 rows and the 3 selected features.

You can also find the names of the selected features:

selected_features = X.columns[selector.get_support()]

print(selected_features.tolist())

Output:

['Age', 'SpendingScore', 'VisitsPerMonth']

The exact selected features can vary depending on the dataset and scoring method.

2. Feature Selection Using RFE

Here, we use Logistic Regression as the estimator.

from sklearn.feature_selection import RFE

from sklearn.linear_model import LogisticRegression

model = LogisticRegression(max_iter=1000)

selector = RFE(

    estimator=model,

    n_features_to_select=3

)

X_selected = selector.fit_transform(X, y)

print(X_selected.shape)

Output:

(10, 3)

You can check which features were selected:

selected_features = X.columns[selector.support_]

print(selected_features.tolist())

Output:

['Age', 'SpendingScore', 'VisitsPerMonth']

You can also view the ranking assigned to every feature:

print(selector.ranking_)

Output:

[1 3 1 1 2]

A ranking of 1 means the feature was selected.

For example:

• Age: 1

• Income: 3

• SpendingScore: 1

• VisitsPerMonth: 1

• Tenure: 2

3. Feature Selection Using Lasso

For this example, we use Lasso for a regression-style demonstration.

from sklearn.linear_model import Lasso

model = Lasso(alpha=0.01)

model.fit(X, y)

selected_features = X.columns[model.coef_ != 0]

print(selected_features.tolist())

Output:

['Age', 'Income', 'SpendingScore', 'VisitsPerMonth', 'Tenure']

The selected features depend on the value of alpha and the dataset.

You can also view the coefficients:

for feature, coefficient in zip(X.columns, model.coef_):

    print(feature, ":", coefficient)

Output:

Age : 0.02

Income : 0.00

SpendingScore : -0.01

VisitsPerMonth : 0.04

Tenure : -0.03

Here, Income has a coefficient of 0, so it can be excluded based on this Lasso model.

The alpha parameter controls the strength of regularisation. You should generally select it using an appropriate validation strategy rather than choosing it arbitrarily.

4. Feature Selection Using SelectFromModel

SelectFromModel selects features based on model coefficients or feature importance scores. It can be used with models such as Random Forest that provide feature importance values.

from sklearn.feature_selection import SelectFromModel

from sklearn.ensemble import RandomForestClassifier

model = RandomForestClassifier(

    n_estimators=100,

    random_state=42

)

selector = SelectFromModel(model)

X_selected = selector.fit_transform(X, y)

print(X_selected.shape)

Output:

(10, 3)

You can identify the selected features using:

selected_features = X.columns[selector.get_support()]

print(selected_features.tolist())

Output:

['Income', 'SpendingScore', 'VisitsPerMonth']

You can also check the feature importance scores:

for feature, importance in zip(X.columns, selector.estimator_.feature_importances_):

    print(feature, ":", round(importance, 3))

Output:

Age : 0.145

Income : 0.182

SpendingScore : 0.294

VisitsPerMonth : 0.247

Tenure : 0.132

SelectFromModel keeps features that meet its selection threshold. The selected features and scores can change when you use a different dataset, model, or threshold.

Feature Selection for Different Machine Learning Models

The appropriate feature selection method can depend on the machine learning algorithm. Below is a quick table for deciding the best feature selection methods based on your need:

Also Read: Clustering in Machine Learning: Types, Algorithms, Examples, and How It Works

Now let’s understand how to choose the right feature selection method in machine learning​.

How to Choose the Right Feature Selection Method?

Choosing the right feature selection method depends on several factors. The common factors includes:

1. Consider the Number of Features

If your dataset contains a very large number of features, filter methods can be a practical starting point because they are generally faster.

You can first remove clearly irrelevant features and then apply a more computationally expensive method if required.

2. Consider the Type of Data

The type of feature and target variable affects which technique is appropriate.

For example:

• Correlation is commonly used with numerical variables.

• Chi-square can be used for suitable categorical classification data.

• ANOVA can evaluate numerical features against categorical targets.

• Mutual information can capture certain non-linear relationships.

Also Read: Confusion Matrix in Machine Learning: Types, Metrics, and Examples

3. Consider the Machine Learning Algorithm

Some feature selection methods are model-independent, while others are closely connected to a particular model.

For example, Lasso-based selection works with models using L1 regularisation, while RFE can use a chosen estimator to rank and remove features.

4. Consider Computational Cost

Wrapper methods may require many rounds of model training.

If you have thousands of features, evaluating many possible feature combinations can become expensive.

Filter methods are usually more computationally efficient and can be useful for reducing the feature space before applying wrapper methods.

5. Consider Interpretability

If you need to explain model predictions to stakeholders, selecting a smaller set of meaningful features can make the model easier to understand.

For example, a model using five clearly defined business variables may be easier to interpret than one using hundreds of variables.

Conclusion

Feature selection helps machine learning models focus on useful features. It can reduce variables, simplify models, improve computational performance, and make results easier to interpret.

Filter, wrapper, and embedded methods offer different ways to select features. The right method depends on your dataset, model, feature types, and modelling goals.