Explain the difference between normalization and standardization. When would you use each, and which algorithms require scaling?

Question

Accepted Answer

Feature scaling ensures that all features contribute proportionally to the model and prevents features with larger magnitudes from dominating those with smaller ranges. Normalization and standardization are the two primary approaches, each with distinct characteristics and use cases.

Normalization, often called min-max scaling, transforms features to a fixed range, typically 0 to 1. The formula is: (x - min) / (max - min). This method preserves the original distribution shape and relationships between values. It is particularly useful when you need bounded values, such as for image pixel intensities or when working with neural networks that use activation functions expecting inputs in specific ranges. However, normalization is sensitive to outliers because a single extreme value can compress the rest of the data into a narrow band.

Standardization, also known as z-score normalization, transforms features to have a mean of 0 and a standard deviation of 1. The formula is: (x - mean) / standard deviation. This method centers the data around zero and scales based on variance. It handles outliers better than min-max scaling because it uses robust statistics. Standardization is preferred when features have different units or when the algorithm assumes normally distributed data.

Algorithms that require scaling include distance-based methods like K-nearest neighbors and K-means clustering, where features with larger scales would disproportionately influence distance calculations. Support vector machines are sensitive to feature scales, especially with RBF kernels. Neural networks typically require scaled inputs to ensure stable gradient descent and faster convergence. Principal component analysis and other dimensionality reduction techniques also benefit from standardization since they analyze variance.

Tree-based algorithms like random forests, gradient boosting, and decision trees generally do not require scaling because they make decisions based on thresholds rather than distances or magnitudes. The split points adapt to the feature values regardless of scale.

An important consideration is that you must fit the scaler on training data only and then apply the same transformation to validation and test data. Fitting on the entire dataset before splitting causes data leakage because the scaler parameters would incorporate information from the test set. For robust scaling with outliers, consider using median and interquartile range instead of mean and standard deviation.