Standardization & Normalization

Standardization & Normalization

So you've collected all your data and now it's time to run your machine learning project. In the data you have collected there will be the features which all have two important properties; the unit and the magnitude. For example, the feature 'age', has units of years and the magnitude is the value.

Each feature in your dataset will have both different magnitudes and units. Since algorithms that compute the distance between the features are biased towards numerically larger values it is important that you scale down this data for your ML algorithm.

The two most common techniques to do this are normalization and standardization. Let's take a closer look at the two.

Why Scaling Helps

When an algorithm uses distance, a feature with larger numbers can dominate the calculation even if it is not more important.

For example, suppose we compare two wines using alcohol and density. Alcohol may vary around values like 8 to 15, while density may vary around 0.99. Without scaling, the alcohol column can have a much larger effect on the distance calculation.

After scaling, both features are put on a more comparable scale, so the model can learn from the pattern rather than just the size of the numbers.

What is Normalization?

This method also known as Min-Max Scaling involves the rescaling of values into a common scale between a range of [0,1] or [-1,1]. It is best suited when there are no outliers. In Python we use a transformer from the Scikit-Learn package called MinMaxScaler for Normalization.

Firstly we will import our dataset as follows.

import pandas as pd
import numpy as np

df = pd.read_csv("/content/winequality-red.csv")

# Choosing 3 features for our analysis
wine_df = df.loc[:,['quality','alcohol','density']]

Next using MinMaxScaler we will perform Normalization on our data.

from sklearn.preprocessing import MinMaxScaler
scaling = MinMaxScaler()

scaling.fit_transform(wine_df[['quality','alcohol']])

Normalization is also known as Min-Max Scaling.

What is Standardization?

This method also known as Z-Score Normalization involves the rescaling based on standard normal distribution where the mean is 0 and the standard deviation is 1. Unlike normalization it is less affected by outliers since there is no predefined range of transformed features. In Python we use a transformer from the Scikit-Learn package called StandardScaler for standardization.

Using the same dataframe we will perform Standardization on our data using the StandardScaler.

from sklearn.preprocessing import StandardScaler
scaling = StandardScaler()

scaling.fit_transform(wine_df[['quality','alcohol']])

Fit Only Once

One important detail is that we should not learn scaling values from the test data. The scaler learns values like the minimum, maximum, mean, and standard deviation during fit().

So the correct process is:

1. Split the data into training and testing sets.
2. fit the scaler on the training data only.
3. transform both the training and testing data using that same scaler.

This prevents information from the test set from leaking into the training process.

```python
from sklearn.model_selection import train_test_split
from sklearn.preprocessing import StandardScaler

X = wine_df[['quality', 'alcohol']]

X_train, X_test = train_test_split(X, test_size=0.2, random_state=42)

scaling = StandardScaler()

X_train_scaled = scaling.fit_transform(X_train)
X_test_scaled = scaling.transform(X_test)
```

When scaling data for a machine learning model, which step should be done on the training data only?

Standardization involves the rescaling of data between a range of [0,1].

What Do I Choose?

Normalization is a good option when you don't know the distribution of your data. It is useful when your data has varying scales and the algorithm you are using does not make assumptions about the distribution of your data, such as k-nearest neighbors and artificial neural networks.

Standardization on the other hand is a good option when you have assumed your data has a Gaussian distribution. Although not strictly necessary, this technique is more effective with a Gaussian distribution. It is useful when your data has varying scales and the algorithm you are using does make assumptions about the distribution, such as linear regression, logistic regression, and linear discriminant analysis.

The Normalization method is most appropriate when your data has a Gaussian distribution.

Conclusion

Feature scaling is an important preprocessing step in ML. Choosing whether to use Normalization or Standardization all depends on your data and algorithm you are using.

One Pager Cheat Sheet

• Scaling numerical data by either normalizing or standardizing the features is important for many machine learning algorithms in order to account for differences in magnitude and unit.
• Normalization is the process of rescaling values into a common scale between a range of [0,1] or [-1,1] by using the MinMaxScaler transformer from the Scikit-Learn package.
• Normalization is a method of rescaling the values of a dataset so that they are within a given range, which is typically either between 0 and 1 or between -1 and 1, and is commonly referred to as Min-Max Scaling.
• Standardization, also known as Z-Score Normalization, involves rescaling the data based on standard normal distribution, and is performed using the StandardScaler transformer in Scikit-Learn.
• Standardization is the process of calculating the z-score relative to the mean and standard deviation of the dataset.
• When deciding between Normalization and Standardization, it depends on whether you know the distribution of your data or if you need an algorithm that makes assumptions about the distribution, such as the use of k-nearest neighbors or linear regression respectively.
• No, Normalization should not be used when data has a Gaussian distribution, instead Standardization should be used for data with a Gaussian distribution where the algorithm makes assumptions about it.
• Feature scaling is an important preprocessing step for machine learning models, with Normalization or Standardization being the two main techniques available.