MMIE: MASSIVE MULTIMODAL INTERLEAVED COMPREHENSION BENCHMARK FOR LARGE VISION-LANGUAGE MODELS

The document describes the development of MMIE, a large-scale benchmark designed to evaluate the performance of Large Vision-Language Models (LVLMs) in interleaved multimodal comprehension and generation tasks. MMIE comprises a dataset of 20,000 meticulously curated multimodal queries across various domains, including mathematics, coding, and literature, which are designed to challenge LVLMs to produce and interpret both images and text in arbitrary sequences. The authors also propose a reliable automated evaluation metric for MMIE, leveraging a scoring model fine-tuned with human-annotated…