MMMU

MMMU (Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark) is a benchmark for evaluating multimodal large language models on tasks that require college-level subject knowledge and deliberate reasoning.

It was released in November 2023 by a team of 22 researchers led by Xiang Yue at Ohio State University and Wenhu Chen at the University of Waterloo, and was presented as an oral paper at CVPR 2024.

The benchmark comprises 11.5 thousand multimodal questions manually collected from college exams, quizzes, and textbooks, spanning six disciplines, Art & Design, Business, Science, Health & Medicine, Humanities & Social Science, and Tech & Engineering, covering 30 subjects and 183 subfields. The questions incorporate highly heterogeneous image types, including charts, diagrams, maps, tables, music sheets, and chemical structures, and are designed to test expert-level perception, knowledge, and reasoning beyond commonsense visual understanding.At the time of release, the best-performing models achieved only around 56% accuracy, well below the human expert range of 76-89%. MMMU has since become a standard evaluation for frontier multimodal models, with scores routinely reported in the technical reports of systems such as GPT-4o, Gemini, and Qwen-VL.