Training Stages

An LLM is trained in stages (Figure 1). Each stage reads less data than the one before, and the data gets cleaner and closer to the conversations the model will have. Pre-training and mid-training produce the base model, which continues any text it is given. Post-training turns it into the model people talk to: first supervised fine-tuning (SFT), then reinforcement learning (RL). AI labs often repeat both in several rounds. An instruct model follows instructions; a reasoning model also writes out its steps before answering.

Each stage might read the same musical fact in a different form (the texts are illustrative):

Pre-training¶

Pre-training is like reading books without a purpose. There is no task and no human-written label, and the next token is the only target. Much of the work goes into choosing and cleaning the text.

Most of the text comes from web crawls. After the text is extracted from each page, a typical pipeline does three things 12:

• Remove duplicates. The web holds many copies of the same text, such as copied articles and text repeated on every page of a site. One sentence appeared more than 60,000 times in a widely used web dataset. Removing copies saves training time, and models trained on deduplicated text reproduce memorized passages about ten times less often 3.

• Filter by quality. A small classifier scores each document, for example on its educational value, and low-scoring documents are dropped. Models trained on text filtered this way did much better on knowledge and reasoning tests than models trained on the same amount of text without this filter 1.

• Set the data mix. The pipeline sets the share of text from each source, such as web pages, code, and math. Large labs report choosing the mix by training small models on candidate mixes and comparing the results 24.

Before post-training, recent models read tens of trillions of tokens 56. A 300-page book is about 100,000 words, or 130,000 tokens, so these models read the equivalent of well over 100 million books.

A base model has learned only to continue documents, like the web page and the textbook in the table above. Given “Why does G7 lead to C?”, it may continue with “Why does D7 lead to G?” and more questions like it, as if the request were one item in a list 9. The continuation is illustrative. A few examples in the prompt can still steer a base model (in-context learning).

Mid-training¶

Toward the end of pre-training, the data mix shifts to the cleanest, most information-rich material, such as textbooks, math, and code 5. This phase is called mid-training 10. Long documents are added around the same time, so that the model learns to use a longer context window 25.

Labs use mid-training to plant skills that post-training later strengthens, such as long step-by-step solutions in math and code 510. The stages overlap: one model family also puts question-and-answer text into its mid-training data 10.

Some music LLMs start from a released base model and continue its pre-training. One study trained such a model further on a few billion tokens of music text and ABC tunes, mixed with general text, then fine-tuned it 11. Audio LLMs (Chapter 2) start from a pre-trained LLM and go through similar stages 1213.

Post-training: supervised fine-tuning¶

Post-training starts with SFT, also called instruction tuning. It teaches the chat format, instruction following, multi-turn dialogue, and the format for calling tools (Reasoning, Tool Calling, and Agents). Each training example is a conversation written out as one token sequence in a chat template. Figure 2 shows one model family’s template 16; each family has its own. In it, the special tokens <|im_start|> and <|im_end|>, one vocabulary entry each, open and close each turn, and the word after <|im_start|> names the speaker. The conversation often opens with a system prompt: instructions from the app’s developer, such as “You are a music assistant.”

The loss is the next-token loss of pre-training, usually applied only to the assistant’s tokens. These include the end-of-turn token, so the model learns to end its turn. The second question, “What key is it in now?”, only makes sense if the model can see the first exchange. The first answer is both a training target and context for the second answer.

Training on answers narrows the base model’s continuations. Raising the probability of an answer after “Why does G7 lead to C?” leaves less probability for other continuations, such as more questions like it.

Early SFT data came largely from people: one early instruction-following model learned from answers written by hired labelers 17. A later study fine-tuned a large base model on only 1,000 carefully chosen examples and obtained a capable chat model. Its authors proposed that most of the knowledge is already in the base model and that SFT mostly teaches format and style 18. Recent public SFT datasets hold about one to two million examples, written by people or generated and then filtered, covering skills such as math, code, and safety 1910.

In one report, a lab ran post-training in rounds and built new SFT examples from its own model. In each round, it sampled many answers per prompt from the previous round’s model and kept the one rated highest by a reward model, a network trained to score answers the way people would. This is called rejection sampling, because the lower-rated samples are thrown away 2.

Open models let anyone download their weights, so you can fine-tune one with SFT on your own examples. Low-rank adaptation (LoRA) freezes the original weights and trains a small set of extra weights added to them, which makes fine-tuning much cheaper 20. Chapter 5, on music education, walks through LoRA fine-tuning of an audio LLM in a code notebook.

Post-training: reinforcement learning¶

In RL, the model writes its own answers to a prompt, and each answer gets a reward, a number that scores it. Training makes the higher-scoring answers more likely and the lower-scoring ones less likely. Many methods sample a group of answers to the same prompt and compare them.

SFT needs a written answer for every prompt. RL needs only a way to score the model’s own answers, so it also works where full answers are hard to write, such as long step-by-step solutions or descriptions of recordings. Because the model learns from its own answers, it can try ways of answering that nobody wrote down, and training makes the ones that score well more likely. Trying such answers is called exploration. In one lab’s run, RL was applied directly to a base model, with no SFT. Its answers grew steadily longer as it learned to write out more steps before answering 21 (Reasoning).

Start with a request that has one correct answer, the transposition from supervised fine-tuning: “Transpose C–Am–F–G up a whole step.” The model writes three answers, and a program compares each with the reference answer, D–Bm–G–A. A match earns a reward of 1, anything else 0 (the answers are illustrative). Each answer’s advantage is its reward minus the group mean, here 0.33:

Training makes answers with a positive advantage more likely and those with a negative advantage less likely 22. The model is never shown the reference answer. It learns only from comparing the scores of its own answers.

If all three answers had scored the same, for example 0 because none was right, every advantage would be zero and the prompt would teach nothing. RL can reward a skill only when some of the model’s own answers already show it. Labs therefore choose prompts that the model gets right only some of the time, and plant skills earlier, in mid-training 22510.

The next example asks an audio LLM to describe a recording: an audio encoder turns the recording into a sequence of vectors, which the LLM reads alongside the prompt’s tokens (Chapter 2).

In Figure 3, the model is asked to describe a funk track. Nobody has written a reference description of it to imitate. Its facts are known from annotations: genre, key, tempo, meter, and instruments. Each description’s reward is the fraction of those facts it gets right, a simplified form of a reward used to train one music audio LLM 23. B gets two facts right, a reward of 0.4. That is below the group mean of 0.47, so B becomes slightly less likely.

After SFT, the training signal usually comes from one of two sources:

• A check of each answer. In RL with verifiable rewards (RLVR), a program, the verifier, checks each answer against a reference or a rule, such as a known math answer or a test suite for code 1921. Both examples above work this way, and reasoning models learn to write long step-by-step solutions from such rewards.

• Choices between two answers. This signal is relative: it says which answer is better, not how good either one is. In RL from human feedback (RLHF), people make the choices, and a reward model learns from those choices to score each answer. A larger gap between two answers’ scores means a higher chance that the higher-scoring one is preferred. This is the Bradley–Terry model of paired comparisons 24. The LLM is then trained with RL to score high on that reward model 2517. Direct Preference Optimization (DPO) learns from the same pairs but trains no separate reward model and samples no new answers, so it is not RL in the strict sense. It makes the preferred answer in each pair more likely relative to the other 26.

Many MIR tasks already have verifiers: a key, a tempo, a list of beat times, or a timed chord sequence can be scored against annotations with the standard metrics in mir_eval 27.

Training on preferences between pairs of answers, by RLHF or DPO, is often called preference tuning. One early model was fine-tuned with SFT and then RLHF. Labelers preferred its answers to those of a model more than 100 times larger that had neither step 17. In music, one study applied RLHF to a text-to-music model that, like an LLM, generates audio tokens one at a time. Its reward model learned from 300,000 pairwise preferences given by the model’s users 28. The preferences can also come from automatic scores or from an LLM acting as a judge (Evaluating LLMs) 19. Another study applied DPO to a model that both understands and generates audio, and picked the preferred output in each pair by how well it matched the prompt and how good it sounded 29.

Labs often combine both sources. They use verifiers where answers can be checked, such as math problems, and preferences for open-ended requests 1921.