Music benchmarks are young. MuChoMusic appeared in 2024, and most of the benchmarks on the previous page appeared in 2025 or 2026. General LLM benchmarks have a longer history, and it shows what happens next: benchmarks get harder, age faster, and become valuable enough to keep private.
Benchmarks age faster than ever¶
General LLM benchmarks have climbed a ladder of who writes the questions (Figure 1).
Figure 1:Benchmarks climb from grade-school problems to graduate-level questions. For music, the next step is open.
Grade school. GSM1k is a new set of grade-school math problems “designed to mirror the style and complexity” of the widely used GSM8K 1. Some models score lower on the new set, with “accuracy drops of up to 8%”, while “many models, especially those on the frontier, show minimal signs of overfitting”. The models that drop most also tend to assign higher probability to GSM8K’s own examples (Spearman’s ), a sign that they may have seen them in training.
Graduate level. GPQA has 448 questions written by domain experts, and skilled non-experts with web access answer only 34% of them correctly 2 (Chapter 1). Humanity’s Last Exam, “a multi-modal benchmark at the frontier of human knowledge”, holds 2,500 questions across dozens of subjects in its current version 3, and on FrontierMath, problems crafted by expert mathematicians, models solved under 2% at release 4.
Music benchmarks are on the same climb. Questions derived from captions (MuChoMusic) gave way to questions from music-theory forums and college exams (WildScore, CSyMR), questions written by experts with musical training (HumMusQA), and complete scores (MSU-Bench). The next step has no benchmark yet: questions that only expert performers, composers, and analysts can write and grade, about why a passage works, how to play it, or what to change.
Good evaluation is becoming a moat¶
Each published benchmark leaks into the next round of training data, so the most trusted evaluations are increasingly private. Scale’s SEAL leaderboards rank models on “curated private datasets that can’t be gamed”, which “will remain private and unpublished” 5. FrontierMath was commissioned and funded by OpenAI: OpenAI owns 300 of its problems and has access to their statements and solutions, “except for a 50-question holdout set” 6. Private evaluation brings its own risks. Bansal & Maini (2025) point to the “conflict of interest arising from private data curators’ business relationships with their clients”, when one company both sells training data to a lab and evaluates that lab’s models.
MIR thankfully has an answer older than LLMs. From its first plenary at ISMIR 2005, MIREX kept test data with the organizers: because “moving the data to participants is problematic, MIREX has the participants submit their algorithms to IMIRSEL for running against the collections” 89. One could imagine a future with MIREX keeping its closed test tradition but embracing the modern scene of Agentic Music-Language systems.
What to measure next¶
The evidence in this chapter points to six directions.
Deeper analysis. Many questions ask for a label. Few ask why a progression works, how a form unfolds, or what an arrangement does, the questions of the reasoning level in the overview.
Answers tied to time. A model should say when something happens and be checked against the timestamp. Time-aligned positions, such as TMRoPE and RoTE, make this possible (Prefix, coarse, and fine interleaving).
Whole songs and stems. Context lengths now reach full songs, up to 20 minutes in Music Flamingo 10, but most benchmark clips are short.
More musical traditions. Most of the score and symbolic benchmarks in this chapter use Western staff notation and theory. NOTA, with score data from three regions of the world, is an exception 11.
Living benchmarks. Benchmarks generated on demand from symbolic data, as in MusICA-MetaBench 12 and SSMR-Bench 13, can be refreshed faster than models can memorize them.
Expert judges. Open-ended musical answers need graders who know music. LLM judges are not yet reliable on music (Can we trust the grader?), so expert panels, and judges validated against them, are part of the work.
From benchmarks to listeners¶
In this section, we’ve primarily considered Music LLMs in the context of broadly answering relatively simple questions about music. However, such a machine A is not yet particularly useful to anyone. A listener does not ask which of four options describes a track; they ask for “something for a late-night drive in the rain”. The next chapters put music understanding to work. Chapter 3 builds systems for listeners: annotation, search, and recommendation. Chapter 4 turns to creators, and Chapter 5 to learners and teachers.
- Zhang, H., Da, J., Lee, D., Robinson, V., Wu, C., Song, W., Zhao, T., Raja, P., Zhuang, C., Slack, D., Lyu, Q., Hendryx, S., Kaplan, R., Lunati, M., & Yue, S. (2024). A Careful Examination of Large Language Model Performance on Grade School Arithmetic. Advances in Neural Information Processing Systems (NeurIPS), Datasets and Benchmarks Track. https://arxiv.org/abs/2405.00332
- Rein, D., Hou, B. L., Stickland, A. C., Petty, J., Pang, R. Y., Dirani, J., Michael, J., & Bowman, S. R. (2024). GPQA: A Graduate-Level Google-Proof Q&A Benchmark. Conference on Language Modeling (COLM). https://arxiv.org/abs/2311.12022
- Phan, L., Gatti, A., Han, Z., Li, N., Hu, J., Zhang, H., Zhang, C. B. C., Shaaban, M., Ling, J., Shi, S., & others. (2025). Humanity’s Last Exam. arXiv Preprint arXiv:2501.14249. https://arxiv.org/abs/2501.14249
- Glazer, E., Erdil, E., Besiroglu, T., Chicharro, D., Chen, E., Gunning, A., Olsson, C. F., Denain, J.-S., Ho, A., de Oliveira Santos, E., Järviniemi, O., Barnett, M., Sandler, R., Vrzala, M., Sevilla, J., Ren, Q., Pratt, E., Levine, L., Barkley, G., … Wildon, M. (2024). FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI. arXiv Preprint arXiv:2411.04872. https://arxiv.org/abs/2411.04872
- Scale AI. (2024). Scale’s SEAL Research Lab Launches Expert-Evaluated LLM Leaderboards. https://scale.com/blog/leaderboard
- Besiroglu, T., & Sevilla, J. (2025). Clarifying the Creation and Use of the FrontierMath Benchmark. https://epoch.ai/blog/openai-and-frontiermath
- Bansal, H., & Maini, P. (2025). Peeking Behind Closed Doors: Risks of LLM Evaluation by Private Data Curators. ICLR 2025 Blogpost Track. https://arxiv.org/abs/2503.04756
- Downie, J. S. (2006). The Music Information Retrieval Evaluation eXchange (MIREX). D-Lib Magazine, 12(12). https://www.dlib.org/dlib/december06/downie/12downie.html
- Downie, J. S. (2008). The Music Information Retrieval Evaluation Exchange (2005–2007): A Window into Music Information Retrieval Research. Acoustical Science and Technology, 29(4), 247–255. 10.1250/ast.29.247
- Ghosh, S., Goel, A., Koroshinadze, L., Lee, S., Kong, Z., Santos, J. F., Duraiswami, R., Manocha, D., Ping, W., Shoeybi, M., & others. (2026). Music Flamingo: Scaling Music Understanding in Audio Language Models. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2511.10289
- Tang, M., Li, J., Yang, L., Zhang, Z., Tian, J., Li, Z., Zhang, L., & Wang, P. (2025). NOTA: Multimodal Music Notation Understanding for Visual Large Language Model. Findings of the Association for Computational Linguistics: NAACL 2025, 7160–7173. 10.18653/v1/2025.findings-naacl.399
- Sourada, T., Vendrame, K., & Hajič, J. (2026). Music I Care About: Automated Multimodal Benchmarking of LLM Music Perception Skills on (Almost) Any Music. arXiv Preprint arXiv:2607.06015. https://arxiv.org/abs/2607.06015
- Wang, Z., Yang, Z., Luo, Y., Li, Y., Qu, X., Qiao, Z., Zhang, H., Zhan, R., Wong, D. F., Zhou, J., & Cheng, Y. (2025). Towards an AI Musician: Synthesizing Sheet Music Problems for Musical Reasoning. arXiv Preprint arXiv:2509.04059.