What are word embeddings, and how do Word2Vec and GloVe differ in their approach to creating them?

Question

Accepted Answer

Word embeddings are dense vector representations of words in a continuous vector space where semantically similar words are mapped to nearby points. Unlike sparse one-hot encodings that treat each word as independent, embeddings capture semantic and syntactic relationships between words, enabling models to understand that "king" and "queen" are related even though they are different words.

The key insight behind word embeddings is that words appearing in similar contexts tend to have similar meanings. This distributional hypothesis forms the foundation for both Word2Vec and GloVe, though they implement it differently.

Word2Vec, developed by Google in 2013, learns embeddings by predicting words based on their local context using a shallow neural network. It offers two training approaches. The Continuous Bag of Words model predicts a target word given surrounding context words. Skip-gram does the opposite, predicting context words given a target word. Skip-gram typically performs better with rare words and smaller datasets, while CBOW is faster and works well with frequent words. Word2Vec focuses exclusively on local context windows, typically spanning five to ten words on each side of the target.

GloVe, developed by Stanford, takes a different approach by leveraging global corpus statistics. It first constructs a co-occurrence matrix capturing how frequently word pairs appear together across the entire corpus. Then it uses matrix factorization to learn embeddings that preserve these co-occurrence relationships. The key advantage is that GloVe captures both local context and global statistical information about word usage patterns.

In practice, both methods produce embeddings that capture meaningful relationships. The famous "king minus man plus woman equals queen" analogy works with both approaches. Word2Vec tends to excel at capturing syntactic relationships and performs well when you have large training corpora. GloVe often captures semantic relationships better and can be more efficient since the co-occurrence matrix is computed once.

A significant limitation of both methods is that they produce static embeddings. Each word gets exactly one vector regardless of context. The word "bank" has the same embedding whether it refers to a financial institution or a river bank. Modern contextual embeddings from models like BERT address this limitation by generating different representations based on surrounding context.

For practical applications, pre-trained embeddings from large corpora often work well as starting points. You can use them directly or fine-tune them on domain-specific data. Common dimensions range from 100 to 300, balancing expressiveness against computational cost.