How Music Gets into an LLM

If you’ve been around the ISMIR community long enough, music research often split models into “audio” and “symbolic”. For an LLM, a more useful demarcation is thinking what representation the model sees: a signal, a picture, or text. That choice impacts how we construct the interface between music and the LLM (Figure 1).

The same reel enters an LLM by three routes. Audio goes through an audio encoder and an adapter, about 25 tokens per second in Qwen2-Audio. A score image goes through an image encoder or patch projection and an adapter, as in any vision-language model. Text such as ABC notation needs only the tokenizer, so any text LLM can read it. All three routes end in the same decoder-only LLM, which writes the answer as text. — Figure 1:Three routes for the running example. Only the audio route needs an audio-specific component.

The shared recipe¶

For models that read audio or images, modality adaptation follows the same four steps: (Figure 2)

Four steps shared by audio and image routes. Encode: an optional fixed pre-transform such as a spectrogram, then a learned encoder that outputs continuous vectors or discrete codes, or no encoder at all. Adapt: map vectors to the LLM's embedding size with a projection or a Q-Former, or give each discrete code a new row in the LLM's embedding table. Position: put the music tokens in the LLM's input as a prefix, interleaved with text, or read them through cross-attention. Train: next-token loss on the text answer. Training usually runs in stages: align on captions, instruction-tune on questions and answers, and optionally reinforcement learning with verifiable rewards. — Figure 2:Encode, adapt, place, train.

Encode. A learned encoder turns the input into a much shorter sequence, one step per short frame of audio or patch of image. Its input can be the raw waveform or a fixed, optional pre-transform of it, such as a mel spectrogram (Optional pre-transforms). Its output is either continuous vectors or discrete codes. Qwen2-Audio’s encoder, for example, gives one vector per 40 ms of audio 1, while a neural audio codec gives a few discrete codes per frame 2. Note some recent models skip the encoder altogether.
Adapt. Most pretrained encoders emit vectors that have the wrong size for the LLM, and may be direclty hard to use. An adapter maps them into the LLM’s token embedding space. It can be a linear layer or a small MLP, as in LLaVA for images and LLark for music 34, or a small Transformer that compresses many frames into a few vectors, the Q-Former 56. Discrete codes need no projection: each one is simply added to the LLM’s vocabulary as a new row in its embedding table 78 (Adapters).
Sequence Position. The adapted vectors enter the LLM. Many models put them in the input sequence, before the text or interleaved with it, where the LLM’s self-attention reads them like any other token. Others keep them outside the sequence and add cross-attention layers that read them 910 (What sequence is the music in, and where?).
Train. The training target is the text answer, with the same next-token loss as any LLM. As in supervised fine-tuning, only the answer tokens carry loss; the music tokens and the question are context. This is particularly true if the model operates on general music embeddings rather than audio tokens, as the Music LLM then no longer needs to worry about producing audio.

Training usually runs in stages. The first stage aligns the new parts on paired audio and text, often with the encoder and the LLM frozen: BLIP-2 bridges “a frozen image encoder and a frozen LLM” 5, and MusiLingo keeps its music encoder and LLM completely frozen and trains only the adapter 11. Later stages tune on instructions and questions, and some models end with preference or reinforcement learning. Qwen2-Audio runs pre-training, supervised fine-tuning, and direct preference optimization 1. Music Flamingo ends with GRPO, with rewards for answer format, answer accuracy, and how well a caption matches structured metadata 12. The training page of Chapter 1 describes these stages for text LLMs.

What each representation route sees¶

Each modality hands the model different evidential forms of music. Table 1 sums up the three routes and one more: text that describes the music, such as a title, an artist, or tags, with no music at all.

Table 1:What the model sees on each route, and what it has to work out.

Route	What the model sees	Interface	Music-specific training?	Explicit in the input	Must be inferred or is missing
Audio	music as a time or frequency domain signal	audio encoder + adapter, or a light projection	yes	timbre, performance, mix, production	semantic information like notes, chords, and beats
Score image	symbolic music as images of sheet music	any vision-language model	no, but notation reading may be weak	symbols, layout, bar numbers	the symbols’ meaning; how it sounds, timbral features
Symbolic text	ABC, **kern, MusicXML	the tokenizer	no, but may be suboptimal	pitches, durations, voices, key signature	how it sounds; anything the format omits
Metadata	title, artist, tags	the tokenizer	no	general high level information about the music	the music itself, any particular fine-grained features

The last row is important to consider given the internet scale training of powerful frontier LLMs: A question about a famous recording can often be answered from its title alone, because the LLM has read about it. The evaluation page will return to this concept.

The next three pages take the representation routes in turn:

Music as audio: waveforms and spectrograms, where encoders come from, adapters, and how audio tokens sit next to text.
Music as images: scores and spectrograms read by vision-language models.
Music as text and code: notation formats, tools, and written procedures for symbolic music.

References¶

Chu, Y., Xu, J., Yang, Q., Wei, H., Wei, X., Guo, Z., Leng, Y., Lv, Y., He, J., Lin, J., Zhou, C., & Zhou, J. (2024). Qwen2-Audio Technical Report. arXiv Preprint arXiv:2407.10759. https://arxiv.org/abs/2407.10759
Défossez, A., Copet, J., Synnaeve, G., & Adi, Y. (2023). High Fidelity Neural Audio Compression. Trans. Mach. Learn. Res.
Liu, H., Li, C., Wu, Q., & Lee, Y. J. (2023). Visual Instruction Tuning. Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/2304.08485
Gardner, J. P., Durand, S., Stoller, D., & Bittner, R. M. (2024). LLark: A Multimodal Instruction-Following Language Model for Music. Proceedings of the 41st International Conference on Machine Learning, 235, 15037–15082. https://proceedings.mlr.press/v235/gardner24a.html
Li, J., Li, D., Savarese, S., & Hoi, S. (2023). BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models. Proceedings of the 40th International Conference on Machine Learning (ICML), 202, 19730–19742. https://arxiv.org/abs/2301.12597
Tang, C., Yu, W., Sun, G., Chen, X., Tan, T., Li, W., Lu, L., Ma, Z., & Zhang, C. (2024). SALMONN: Towards Generic Hearing Abilities for Large Language Models. The Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=14rn7HpKVk
Rubenstein, P. K., Asawaroengchai, C., Nguyen, D. D., Bapna, A., Borsos, Z., de Chaumont Quitry, F., Chen, P., El Badawy, D., Han, W., Kharitonov, E., Muckenhirn, H., Padfield, D., Qin, J., Rozenberg, D., Sainath, T., Schalkwyk, J., Sharifi, M., Ramanovich, M. T., Tagliasacchi, M., … Frank, C. (2023). AudioPaLM: A Large Language Model That Can Speak and Listen. arXiv Preprint arXiv:2306.12925. https://arxiv.org/abs/2306.12925
Zhan, J., Dai, J., Ye, J., Zhou, Y., Zhang, D., Liu, Z., Zhang, X., Yuan, R., Zhang, G., Li, L., Yan, H., Fu, J., Gui, T., Sun, T., Jiang, Y.-G., & Qiu, X. (2024). AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling. In L.-W. Ku, A. Martins, & V. Srikumar (Eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 9637–9662). Association for Computational Linguistics. 10.18653/v1/2024.acl-long.521
Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., Ring, R., Rutherford, E., Cabi, S., Han, T., Gong, Z., Samangooei, S., Monteiro, M., Menick, J., Borgeaud, S., … Simonyan, K. (2022). Flamingo: a Visual Language Model for Few-Shot Learning. Advances in Neural Information Processing Systems (NeurIPS). https://arxiv.org/abs/2204.14198
Kong, Z., Goel, A., Badlani, R., Ping, W., Valle, R., & Catanzaro, B. (2024). Audio Flamingo: A Novel Audio Language Model with Few-Shot Learning and Dialogue Abilities. ICML. https://openreview.net/forum?id=WYi3WKZjYe
Deng, Z., Ma, Y., Liu, Y., Guo, R., Zhang, G., Chen, W., Huang, W., & Benetos, E. (2024). MusiLingo: Bridging Music and Text with Pre-trained Language Models for Music Captioning and Query Response. In K. Duh, H. Gomez, & S. Bethard (Eds.), Findings of the Association for Computational Linguistics: NAACL 2024 (pp. 3643–3655). Association for Computational Linguistics. https://aclanthology.org/2024.findings-naacl.231
Ghosh, S., Goel, A., Koroshinadze, L., Lee, S., Kong, Z., Santos, J. F., Duraiswami, R., Manocha, D., Ping, W., Shoeybi, M., & others. (2026). Music Flamingo: Scaling Music Understanding in Audio Language Models. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2511.10289