Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Introduction

Music listening with LLMs

Take a listener, user 1024: female, aged 25–34, with Bill Evans and Nils Frahm among her favorites. She asks for “something for a late-night drive in the rain, no vocals, and not too old.” The request describes the music: a sound (“no vocals”), a release date (“not too old”), and a situation (“a late-night drive in the rain”). The profile describes the person, and it changes what a good answer is. For this listener, a late-night drive probably means sparse piano and slow jazz; for someone whose favorites are synthwave, the same words point somewhere else entirely.

The two halves correspond to the two tasks a listening system performs. Search/Retrieval maps a query to music: the listener says what they want, and the system returns tracks that match it. Recommendation maps a user to music: the system starts from the user, identified by an ID and described by a profile and listening history, and suggests the next track for that user. The request above needs both.

Neither task reaches the music directly. A system can match a query or a user only against what is recorded about each track, and producing that record is a third task. Annotation maps music to information: tags, captions, and structured fields such as genre, instrumentation, or release year. The listener never sees it, but it sets the limit on what search/retrieval and recommendation can answer.

Three columns, one per task. Annotation maps music, an audio track, to information: tags, captions, and fields. Search/retrieval maps a query, "late-night drive in the rain, no vocals, not too old", to music, the tracks that match it. Recommendation maps a user (ID 1024, female, aged 25–34, favorites Bill Evans and Nils Frahm) to music, the next track for that user.

Figure 1:The three tasks. Annotation maps music to information, search/retrieval maps a query to music, and recommendation maps a user to music.

TaskInput → outputFor the listener
Annotationmusic → informationwhat the system knows about each track
Search/Retrievalquery → musicthe listener says what they want
Recommendationuser → musicthe system suggests the next track for this user

Previous approaches

Search/retrieval improved by widening what could be written about a track. Early catalogs held a title, an artist, and a year, so a query was an exact match or nothing. Crowds then added free-form tags, which made “jazz” and “no vocals” searchable, at the cost of sparse and uneven coverage 1. Autotagging closed the coverage gap: a model trained on labeled tracks stamped the same vocabulary on every track in the catalog, and a keyword query was ranked against those stamps 234. The vocabulary was fixed, though, and “a late-night drive in the rain” was not in it. Joint audio-text embeddings removed the fixed vocabulary: any phrase could be a query, scored against the audio directly 56. But an embedding trained on audio and captions knows only the sound. “Not too old” needs a release date, and nothing in the audio carries one.

Recommendation took a different route. Collaborative filtering represents each user and each track by an ID and learns a vector for each from listening logs, so that a user’s vector scores highest on tracks that similar listeners played; matrix factorization made this the standard design 78. It needs no annotation at all, and it captures taste that no tag describes. Its limits follow from the same design. A new track has no plays and therefore no vector, which is why content-based models were brought in to fill the gap 9. A vector cannot say why a track was chosen. And there is no place to put a query: the system can predict what this listener usually plays, but not what they want tonight.

With these approaches, query and user lived in different representations. A query was a set of keywords or a text embedding; a user was an ID learned from logs. The two systems could not be combined, and each answered only half of the request. It was the listener who adapted, choosing keywords the tagger knew and expressing taste through the skip button. Annotation and search/retrieval in this setting, from tagging to captioning and from query-by-text to joint embeddings, are covered in depth in our earlier tutorial, Connecting Music Audio and Natural Language 10.

Three columns, one per task, as solved by previous approaches. Annotation by tagging: a tagger scores an audio track against a fixed vocabulary (jazz 0.91, piano 0.84, instrumental 0.78, vocals 0.05, rock 0.02) and writes the top tags as information. Search/retrieval by joint text-audio embedding: a text encoder and an audio encoder place the query "late-night drive in the rain" and every track in one shared space, and the nearest tracks are returned. Recommendation by matrix factorization: the user-by-track play matrix is approximated by user vectors times track vectors, and the missing cells in user 1024's row are scored to pick the next track.

Figure 2:Previous approaches. Annotation is tagging with a fixed vocabulary, search/retrieval is nearest-neighbor search in a joint text-audio embedding, and recommendation factorizes the user-by-track play matrix. Each task has its own model and its own representation.

What LLMs change

LLMs change all three tasks, and the change is the same in each: the input becomes language.

Annotation. An LLM can write a description for every track from whatever signals exist, such as tags, metadata, or the outputs of MIR models, so the catalog gets captions without anyone writing them 111213. The model also brings world knowledge about artists, scenes, and eras, so “jazz-ish” and “not too old” can be resolved against the same record as “no vocals” 14. That knowledge is also where the model is most likely to be confidently wrong, about an obscure artist or a release year.

Search/Retrieval. The query no longer has to fit a vocabulary. An LLM reads the request, separates it into a sound description, metadata constraints, and context, and sends each part to what can answer it: a text-to-music retriever for the sound, a filter over the record for the date 15. “Not too old” becomes a condition rather than a phrase the embedding cannot place.

Recommendation. The user no longer has to be an ID. A listening history can be written as a list of tracks and a stated taste as a sentence, and an LLM given both can return tracks and say why it chose them 161718. A profile written in text can be read and corrected by the listener 19, and a new track can be recommended as soon as it has a description.

Once query and user are both language, they can sit in the same context. Search/retrieval and recommendation then become one problem, (user, query) → music: the request supplies the query, the history supplies the user, and the same model answers both. In a conversation the line fades further. “Make it more upbeat” refines the query, a skip updates what the system knows about the user, and both enter the same loop 2015. This convergence is the main claim of the chapter.

What LLMs change. On the left, annotation: an audio-language LLM takes an audio track and its metadata row, where genre (jazz) and year (2019) come for free and mood and instruments are empty, and a system prompt naming the columns to add ("write a caption; add mood and instruments"). It goes beyond tagging, writing a caption ("A sparse piano trio, brushed drums.") and imputing the new columns (mood nocturnal, instruments piano trio). The information is stored in a database table with columns track, genre, year, mood, and instruments, where the mood and instruments cells are marked as added by the LLM; the table feeds the model on the right. On the right, search/retrieval and recommendation merge into conversational recommendation: the query and the user (ID 1024, female, aged 25–34, favorites Bill Evans and Nils Frahm) both feed into one LLM or agent that maps (user, query) to music and a response. It answers with slow piano tracks after 2015, close to the user's Nils Frahm plays; asked for something more upbeat, it swaps in brighter tracks that still have no vocals.

Figure 3:What LLMs change. Annotation extends from tagging to captioning and metadata imputation. Search/retrieval and recommendation merge: the query and the user enter one LLM or agent, which returns music and a response, and the two tasks become conversational recommendation.

The rest of the chapter follows the late-night-drive request through this shift: how LLMs annotate a catalog, how they search and recommend from it, how the two meet in conversation, and what does not work yet.

References
  1. Lamere, P. (2008). Social tagging and music information retrieval. Journal of New Music Research.
  2. Whitman, B. A. (2005). Learning the meaning of music [Phdthesis]. Massachusetts Institute of Technology.
  3. Turnbull, D., Barrington, L., Torres, D., & Lanckriet, G. (2008). Semantic annotation and retrieval of music and sound effects. IEEE Transactions on Audio, Speech, and Language Processing, 16(2), 467–476.
  4. Nam, J., Choi, K., Lee, J., Chou, S.-Y., & Yang, Y.-H. (2018). Deep learning for audio-based music classification and tagging: Teaching computers to distinguish rock from bach. IEEE Signal Processing Magazine.
  5. Huang, Q., Jansen, A., Lee, J., Ganti, R., Li, J. Y., & Ellis, D. P. (2022). Mulan: A joint embedding of music audio and natural language. arXiv Preprint arXiv:2208.12415.
  6. Manco, I., Benetos, E., Quinton, E., & Fazekas, G. (2022). Contrastive audio-language learning for music. arXiv Preprint arXiv:2208.12208.
  7. Hu, Y., Koren, Y., & Volinsky, C. (2008). Collaborative Filtering for Implicit Feedback Datasets. Proceedings of the IEEE International Conference on Data Mining (ICDM).
  8. Koren, Y., Bell, R., & Volinsky, C. (2009). Matrix Factorization Techniques for Recommender Systems. Computer, 42(8), 30–37.
  9. van den Oord, A., Dieleman, S., & Schrauwen, B. (2013). Deep Content-Based Music Recommendation. Advances in Neural Information Processing Systems (NeurIPS).
  10. Doh, S., Manco, I., Novack, Z., Kim, J. W., & Chen, K. (2024). Connecting Music Audio and Natural Language. Tutorial at the International Society for Music Information Retrieval Conference (ISMIR). https://mulab-mir.github.io/music-language-tutorial/
  11. Doh, S., Choi, K., Lee, J., & Nam, J. (2023). Lp-musiccaps: Llm-based pseudo music captioning. International Society for Music Information Retrieval (ISMIR).
  12. Gardner, J. P., Durand, S., Stoller, D., & Bittner, R. M. (2024). LLark: A Multimodal Instruction-Following Language Model for Music. Proceedings of the 41st International Conference on Machine Learning, 235, 15037–15082. https://proceedings.mlr.press/v235/gardner24a.html
  13. Bukey, I., Wang, Z., Donahue, C., & Bryan, N. J. (2026). Rethinking Music Captioning with Music Metadata LLMs. ICASSP 2026 – 2026 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). https://arxiv.org/abs/2602.03023
  14. Doh, S., Lee, M., Jeong, D., & Nam, J. (2024). Enriching Music Descriptions with a Finetuned-LLM and Metadata for Text-to-Music Retrieval. Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP).
  15. Doh, S., Choi, K., & Nam, J. (2025). TalkPlay-Tools: Conversational Music Recommendation with LLM Tool Calling. arXiv Preprint arXiv:2510.01698.
LLM-powered Music Understanding
Looking Ahead
Music Listening with LLMs
LLM-powered Music Annotation