Research Unveils Modality Gap in Multimodal LLMs

Recent research published on arXiv investigates the performance disparities of multimodal large language models (MLLMs) when processing text as images versus traditional text tokens. The study systematically examines seven MLLMs across diverse benchmarks, revealing that performance often declines when using image-based input, particularly in tasks like mathematics, where accuracy can drop significantly due to rendering choices like font and resolution. The authors conducted a thorough analysis, identifying specific errors exacerbated by image input while showing that reasoning capabilities largely remain intact under these conditions.

The implications of these findings are substantial for the development of MLLMs. The study proposes a self-distillation approach that boosts model accuracy significantly, enabling MLLMs to achieve 92.72% accuracy on certain benchmarks when trained on their text reasoning paired with images. This advancement could enhance AI's capacity for understanding and processing visual text data, addressing concerns around the 'modality gap' and paving the way for improved multimodal applications in various sectors including education and technology.