Search and Recommendation

LLM-powered Music Search and Recommendation¶

Track #0381 now has a full row: jazz, 2019, 72 bpm, D♭ major, vocals: none, mood: nocturnal, instruments: piano trio, and a caption that says all of it in one sentence. User 1024 is still waiting for something for a late-night drive in the rain, with no vocals and nothing too old. Whether #0381 reaches her depends on two readings: of the request against the row, and of her (favorites Bill Evans and Nils Frahm, ten recent plays, one skip) against what the system knows about who listens to what. The introduction argued that once both are language, search and recommendation become one problem, (user, query) → music. This page asks where in that computation the LLM sits. It can be the recommender itself, a reranker over what a retriever found, or an agent that calls tools and keeps the conversation going. Each placement reads the request and the user differently, costs something different, and is measured, so far, by a benchmark that misses something.

What a request contains¶

Natural-language music requests mix several kinds of information at once. A study of real music-seeking questions found bibliographic details, lyrics, genre, mood, and intended use side by side in the same request 1. For user 1024 we collapse these into four kinds (Figure 1). The situation, “a late-night drive in the rain,” implies a sound without naming one. The constraints, “no vocals” and “not too old,” are facts the row holds. The taste, Bill Evans and Nils Frahm, is why the situation means sparse piano for this listener and dim synths for someone else. The behavior, her plays and skips and behind them the logs of everyone else, records who listened to what.

Each earlier system read one of these. Tag retrieval read a constraint if it happened to be a tag, such as instrumental, and nothing else 23. Joint audio-text embeddings read any phrase, including, loosely, the situation 456, and enriching their text side with metadata and LLM-written descriptions let them read an artist or a year as well 78. None of them handled “no”: contrastive embeddings tend to retrieve what is named rather than what is excluded, so “no vocals” pulls the query toward vocals. Collaborative filtering read behavior, and through it taste, without reading the music at all 910; it fails on a new track and cannot say why it chose one 11.

An LLM reads the first three directly, in the words the listener used. The fourth it reads only in part. It can read user 1024’s ten plays written out as a list, but not the co-plays of millions of other listeners that make collaborative filtering work. That asymmetry runs through the rest of this page: the LLM is good at reading this request and this listener, and needs something else for what everyone else did. The question is where to put it.

The LLM as recommender¶

The most direct design makes the model produce the recommendation itself. The profile, the history, and the request are written into the context, and the model continues the sequence with tracks (Figure 2). Prompting an off-the-shelf LLM this way already works surprisingly well on conversational recommendation benchmarks, and it fails in two characteristic ways: the model names items that do not exist, and it favors popular ones 12. Both follow from the model knowing the catalog only through text it has read.

The remedy is to make the catalog part of the vocabulary. In general recommendation this began with item IDs the model learns to generate 13. In music, Text2Tracks generates track IDs directly from a prompt 14, and TalkPlay builds each track token from audio, lyrics, metadata, and collaborative-filtering embeddings, so one model continues a listening history and a conversation in the same stream 15. MuseChat applies the idea to recommending music for a video 16. The dialogues these models train on were themselves written by LLMs, from real listening sessions and intent analyses 171819.

What this design buys is one model that reads the conversation and the collaborative signal together, with no retrieval stage to tune. What it costs follows from where the catalog lives. A new release is not a token until the model is retrained, and the context window bounds how much history the model can read. The row from the previous page is not consulted at query time either: the recommender answers from what it absorbed in training, so if the mood of #0381 was corrected yesterday, it does not know.

The LLM as reranker¶

The second design keeps retrieval as it was and puts the LLM on both sides of it (Figure 3). Before the retriever, the LLM rewrites the request into what each tool reads well. “A late-night drive in the rain, no vocals, not too old” becomes the query “slow, sparse piano, nocturnal” for the text-to-music retriever and vocals = none, year >= 2015 for a filter over the record. The profile shapes the rewrite: Bill Evans and Nils Frahm are why the query says piano and not synths. Here the record pays off. Once vocals is a field, “no vocals” stops being a fuzzy retrieval problem and becomes an exact filter, and the negation that embeddings could not handle is handled by the row instead.

After the retriever, the LLM reads the candidates’ rows and orders them against the whole request and the listener. The two components read differently. The retriever reads the audio, and may rank a lo-fi track with a Rhodes and a drum machine highest, because it sounds like late night. The LLM reads the rows and the profile, and moves the piano trio and the solo piano above it. Putting LLM-written descriptions into the retriever’s text side at indexing time was an earlier way to get the same knowledge into the score 8. Rewriting at query time and reranking afterwards is the more flexible way, the retrieval-augmented pattern of Chapter 1 20, because nothing has to be reindexed when the model improves.

The reranker’s reason is what it read. A collaborative recommender’s “because you listened to X” is attached after the fact; a reranker that placed #0381 first because its row says 2019, piano trio, no vocals can say so. Whether it actually does, rather than inventing a plausible reason afterwards, is a question for the evaluation section below.

Note where the audio is in this design. The LLM never hears it. It reads the retriever’s scores and the rows, and two tracks with the same row are the same track to it. If the row is thin, the reranker is ordering by names and years. The quality of search is bounded by the quality of annotation, which is why the previous page came first.

Agentic music recommendation¶

The third design gives the LLM the retriever, the filter, the collaborative model, and the records as tools, and lets it decide which to call, in what order, and when to stop (Figure 4). It is the agent pattern of Chapter 1 2122 applied to listening. In music it was introduced with TalkPlay-Tools, which moved recommendation from generating track tokens to calling a catalog and MIR models inside the conversation 23.

The agent contains the reranker: planning is the rewrite above, and the final ordering is the reranking above. It adds two things. The first is the collaborative model, back as a tool, which closes the asymmetry from the start of this page: the LLM reads the request and the listener, and the tool supplies what millions of others played. The second is the loop. The limits of single-turn retrieval were clear before LLMs could address them: context lost between queries, no way to learn from feedback, no “like this but more upbeat.” The Conversational Playlist Curation Dataset collected 917 human dialogues and showed what multi-turn curation looks like 24. An agent keeps the conversation as context, so “more upbeat” revises the plan, raising the tempo while keeping vocals = none and year >= 2015, instead of starting a new search, and a skip is evidence rather than noise.

The agent is also where the listener state lives. Over a conversation it accumulates what the listener said, what they skipped, and what it inferred. Scrutable user models have been proposed before, as short editable statements of taste 25; what is new is that the state is written in free text, by the same component that uses it, and can be carried into the next session if the system is built to keep it 26. And because the catalog stays outside the model, a new release is available the moment it has a row.

The three designs differ less in what the LLM can read than in where the catalog lives.

We expect the agent design to be the shape production systems converge on, because it is the only one that keeps both the record and the collaborative signal outside the model, where each can be updated on its own schedule. That is a prediction, not yet a survey result.

Evaluation¶

Each role is evaluated the way its predecessors were, and each inherits the blind spot.

Captions describe a sound. Real requests describe a situation, a constraint, and a taste, and no widely used benchmark scores relevance and constraint satisfaction at once. For agents, human-collected dialogues exist but are small 24, and the current answer is an LLM that plays the listener over several turns 28. A system trained on LLM-written dialogues 19 and evaluated by an LLM-played listener can look better than it is. Our own view is that synthetic dialogues are for training and the human-collected ones should be kept for evaluation.

What a listening evaluation should score, for all three roles, is shown in Figure 5: whether every track in the list satisfies the stated constraints, whether a revision (“more upbeat”) is handled as a revision that keeps the earlier constraints, and whether every field the reason cites is one the ranker actually read. None of these needs the held-out track, and all of them can be checked against the record.

With this page’s systems, user 1024 gets a ranked list that respects “no vocals” and “not too old,” a reason for each track, and a way to say “more upbeat” and be heard. What they cannot do yet is the subject of the next page.