What are the main techniques for encoding categorical variables, and when would you use each one?

Question

Accepted Answer

Encoding categorical variables is essential because most machine learning algorithms require numerical input. The choice of encoding method depends on the nature of the categorical variable and the algorithm being used.

One-hot encoding creates a binary column for each category. If a feature has values like red, blue, and green, you create three new columns, each containing 1 if the original value matches that category and 0 otherwise. This method is ideal for nominal variables with no inherent order and works well with algorithms that can handle sparse data. However, it can lead to high dimensionality when dealing with features that have many unique values, a problem known as the curse of dimensionality.

Label encoding assigns a unique integer to each category. Red might become 0, blue becomes 1, and green becomes 2. This method is compact and preserves memory, but it introduces an artificial ordinal relationship. The model might incorrectly interpret that green with value 2 is somehow greater than blue with value 1. Use label encoding only for ordinal variables where order matters, such as education level or satisfaction ratings, or with tree-based algorithms that make splits and are not affected by the numerical ordering.

Target encoding, also called mean encoding, replaces each category with the mean of the target variable for that category. For a binary classification problem, each category becomes the probability of the positive class within that category. This method captures the relationship between the feature and target but risks data leakage if not implemented carefully with proper cross-validation folds.

Frequency encoding replaces categories with their occurrence count or percentage in the dataset. This is useful when the frequency of a category carries meaningful information, such as popularity or commonality.

Binary encoding converts category integers to binary representation and splits digits into separate columns. This creates fewer columns than one-hot encoding while avoiding the ordinal assumptions of label encoding, making it useful for high-cardinality features.

For high-cardinality categorical variables with hundreds or thousands of unique values, consider combining rare categories into an "Other" group, using target or frequency encoding, or applying embedding techniques in deep learning contexts. The key is matching the encoding method to both the data characteristics and the model requirements.