Model Components
Contextual Embeddings
Contextual embeddings produce different vector representations for the same word depending on its context—"bank" near "river" and "bank" near "finance" get different vectors. ELMo (2018) introduced contextual embeddings; transformer models like BERT made them standard.
Contextual embeddings dramatically outperform static embeddings (Word2Vec, GloVe) on disambiguation-dependent tasks. Sentence transformers extend this to full sentence-level contextual embeddings optimized for semantic similarity tasks.
Authority Links
Related Terms
Model Components
Embeddings
Dense numerical vectors that represent text, images, or other content in a high-dimensional space where semantically similar items are geometrically close — the foundational data structure for semantic search and RAG retrieval.
Techniques & Methods
Word Embedding
Technique representing words as dense vectors that capture semantic similarity.
Model Components
Transformer
A neural-network architecture, introduced by Vaswani et al. in 2017, that uses self-attention and parallel computation across all sequence positions — the foundation under virtually every frontier language and multimodal model in production today.
Techniques & Methods
Semantic Similarity
Measure of how closely related two pieces of text are in meaning.