New Adaptive Dictionary Embeddings Scale LLM Representations

Researchers have introduced the Adaptive Dictionary Embeddings (ADE) framework, which successfully scales multi-anchor word representations for large language models. ADE addresses traditional limitations of word embeddings, offering three significant innovations: Vocabulary Projection transforms dual-stage anchor lookups into a single efficient operation, Grouped Positional Encoding enhances semantic coherence by sharing positional information among anchors, and context-aware anchor reweighting utilizes self-attention for dynamic anchor contributions. Evaluations against benchmarks show ADE significantly reduces trainable parameters while achieving high performance comparable to established models like DeBERTa.