Study Reveals Retrieval Bias in Large Language Models

The recent study published on arXiv addresses the issue of retrieval bias in large language models (LLMs), specifically under conditions of multiple in-context knowledge updates. Unlike previous research that focused on single updates, this work explores how multiple historically valid versions of facts can create competition during retrieval, leading to increased bias. The study introduces a Dynamic Knowledge Instance (DKI) evaluation framework, which models these multi-updates and assesses the LLM performance through endpoint probing of initial and current states. Results indicate that while early-state accuracy remains high, latest-state accuracy suffers significantly as updates accumulate.

The implications of this research are substantial for the development and deployment of LLMs in knowledge-intensive applications. As retrieval bias becomes more pronounced with the increase in updates, the ability of models to track and accurately follow knowledge revisions becomes critically challenged. Even with cognitive-inspired heuristic interventions, the study suggests that these biases are persistent and that fundamental adjustments may be necessary to improve LLM performance under such multi-update conditions. Understanding and mitigating retrieval bias is essential as LLMs continue to evolve as central tools in various knowledge-driven sectors.