Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Foundation Models

A large language model (LLM) is pre-trained on trillions of words from the web, books, code, and papers to predict the next piece of text. Some of that text is about music: reviews, theory lessons, and folk tunes written in ABC notation, a plain-text format for tunes. Predicting all of it well forces the model to learn patterns behind the text. Those patterns give the model abilities that nobody explicitly trained it for. Bommasani et al. (2021) called such a model a foundation model, because it is trained once on broad data and then adapted to many tasks, by prompting or by further training.

Next-token prediction

A tokenizer splits text into tokens, words or pieces of words from a fixed vocabulary. Given the tokens so far, the model gives every token in the vocabulary a probability of being the next token (Figure 1). The model is a neural network, usually a decoder-only Transformer 23. In this kind of Transformer, each token sees only itself and the tokens before it. Predicting each token from the ones before it is called autoregressive modeling. Many symbolic music models are trained the same way, on sequences of note and timing events 45.

The context "In C major, G7 resolves to" is split into eight tokens and read by the LLM, a decoder-only Transformer in which each position sees only itself and earlier positions. It outputs next-token probabilities: C 0.58, the 0.21, a 0.09, Am 0.03, all other tokens 0.09. Generation samples one token, appends it to the context, and predicts again.

Figure 1:Next-token prediction for “In C major, G7 resolves to”. Token boundaries and probabilities are illustrative.

Generation runs the model in a loop: pick a token at random according to the next-token probabilities, append it, and predict again. The number of tokens the model can read at once, including those it has generated, is its context window.

Tokens and sampling

The vocabulary is usually built before training by byte-pair encoding, which merges frequent character sequences into single tokens 6. An early model’s vocabulary held about 50,000 tokens 7, and many newer ones hold several times more. In the ABC bar d2 fd A2 FA, where d2 lasts two eighth notes, a token may hold two neighboring notes, such as fd, or split a note from its length, such as d and 2. So tokens do not line up with notes, and the lengths that fill a bar are spread across them. A neural audio codec turns sound into a sequence of codes, a few per short slice. Some models add these codes to a pre-trained LLM’s vocabulary and train it further on audio and text, so that a single LLM can generate sound as well as text 89.

A temperature setting controls sampling. Near zero, the model almost always takes the most probable token. At higher values, less likely tokens such as Am, the deceptive resolution, are picked more often.

Training adjusts the model’s weights θ\theta (also called parameters), the billions of numbers inside the network. At each position tt, it raises the probability pp of the token xtx_t that actually came next, given the earlier tokens x<tx_{<t}. That token’s surprisal is −log⁡p-\log p. Suppose the training text continues “In C major, G7 resolves to” with C. If the model gives C a probability of 1% early in training, the surprisal of C is −ln⁡0.01≈4.6-\ln 0.01 \approx 4.6 nats (natural-log units); at 58%, it is 0.54 nats. Training lowers the loss, the average surprisal over all TT positions:

L(θ)=−1T∑t=1Tlog⁡pθ(xt∣x<t).\mathcal{L}(\theta) = -\frac{1}{T}\sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t}).
(1)

This is the same cross-entropy loss as in a classifier with one class per token in the vocabulary. IDyOM, a statistical model of musical expectation, computes the same surprisal for each note of a melody, given the notes before it, and calls it information content, measured in bits (4.6 nats is 6.6 bits). Listeners tend to rate notes with higher information content as more unexpected 10.

The patterns behind the text

Each blank below needs knowledge that its sentence does not state.

Text to continueKnowledge needed
In C major, G7 resolves to ___Harmony: the dominant seventh usually resolves to the tonic (V7 → I)
At 120 BPM, one beat lasts ___ secondsArithmetic: 60 / 120 = 0.5
An ABC reel whose header ends R:reel, M:4/4, L:1/8, K:D (rhythm, meter, unit note length, key), with first bar |: d2 fd A2 FA | ___ABC syntax, and the conventions of reels in D

The model never receives these rules as labels. Examples of each are scattered through the training text, phrased differently and mixed with mistakes. The model can store only a small part of its training text. One small model read about 1,900 tokens for every weight, far more text than its weights could store exactly 11. The more often a passage appears, the more likely the model is to memorize it 12. To lower the loss on text it sees rarely, the model has to capture patterns that many documents share.

Prediction as compression

A model with low loss can also compress files. An arithmetic coder, a standard lossless compression method, can turn the model’s predicted probabilities into a compressed file. In this file, a token predicted with probability pp costs about −log⁡2p-\log_2 p bits, so the file’s length is close to the sum of these costs: the loss in bits times the number of tokens. One large model was trained mostly on text. Given short chunks of 8-bit audio of read-aloud English speech as raw bytes in its context, it compressed them to about 16% of their raw size. FLAC reached about 30%. The comparison does not count the model itself, which anyone decompressing the file would also need 13.

In-context learning and emergent abilities

Pre-training also produces in-context learning: the model picks up a task from examples in its prompt, with no change to its weights 14. In the illustrative prompt below, two pairs of relative keys set the task.

C major → A minor
G major → E minor
F major →

Music theory pages often include lists of relative keys. The two examples show that this prompt is such a list, so the expected continuation of “F major →” is “D minor”. Examples of parallel minors (C major → C minor) would give F minor.

Nobody explicitly trained the model for in-context learning. Bommasani et al. (2021) called such abilities emergent: training induces them without anyone building them in. Many papers use the word in a second sense, for abilities that appear only at a large enough scale. Such abilities include solving certain tasks from a few examples in the prompt, and solving problems by writing out steps (Reasoning) 15. Scale here means the number of weights, the amount of training text, and the computation spent on training. As all three grow, the loss falls predictably 1617. Larger models are also better at in-context learning 14. Whether abilities appear suddenly with scale is debated: with scores that give partial credit to nearly right answers, many apparent jumps become gradual 18.

From base model to assistant

The model described so far is a base model. It continues whatever text it is given, so a question may get more questions back. Further training, called post-training and covered on the next page, adapts it into an assistant. The requests below show an assistant’s range, with the section that covers each one:

AbilityExample requestCovered in
Answer“Why does a ii–V–I sound resolved?”Knowledge from pre-training; answering as an assistant from supervised fine-tuning and reinforcement learning
Follow instructions“Transpose C–Am–F–G up a whole step. Reply with chord symbols only.”Supervised fine-tuning and reinforcement learning
Reason“Which chord in C–Am–Fm–G is borrowed, and why?”, answered after writing out the stepsReasoning
Act“What’s the tempo of track.wav?”, answered by calling a tempo estimatorTool calls and agents
ListenGiven a clip: “Which instruments do you hear?”Chapter 2: an audio encoder joined to the LLM and trained on paired audio and text makes an audio LLM 192021229
References
  1. Bommasani, R., Hudson, D. A., Adeli, E., Altman, R., Arora, S., von Arx, S., Bernstein, M. S., Bohg, J., Bosselut, A., Brunskill, E., & others. (2021). On the Opportunities and Risks of Foundation Models. arXiv Preprint arXiv:2108.07258. https://arxiv.org/abs/2108.07258
  2. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, Ł., & Polosukhin, I. (2017). Attention is All you Need. Advances in Neural Information Processing Systems, 30.
  3. Radford, A., Narasimhan, K., Salimans, T., & Sutskever, I. (2018). Improving Language Understanding by Generative Pre-Training [Techreport]. OpenAI. https://cdn.openai.com/research-covers/language-unsupervised/language_understanding_paper.pdf
  4. Huang, C.-Z. A., Vaswani, A., Uszkoreit, J., Simon, I., Hawthorne, C., Shazeer, N., Dai, A. M., Hoffman, M. D., Dinculescu, M., & Eck, D. (2019). Music Transformer: Generating Music with Long-Term Structure. International Conference on Learning Representations, ICLR.
  5. Thickstun, J., Hall, D. L. W., Donahue, C., & Liang, P. (2024). Anticipatory Music Transformer. Trans. Mach. Learn. Res., 2024. https://openreview.net/forum?id=EBNJ33Fcrl
  6. Sennrich, R., Haddow, B., & Birch, A. (2016). Neural Machine Translation of Rare Words with Subword Units. Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (ACL). https://arxiv.org/abs/1508.07909
  7. Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language Models are Unsupervised Multitask Learners [Techreport]. OpenAI. https://cdn.openai.com/better-language-models/language_models_are_unsupervised_multitask_learners.pdf
  8. Zhan, J., Dai, J., Ye, J., Zhou, Y., Zhang, D., Liu, Z., Zhang, X., Yuan, R., Zhang, G., Li, L., Yan, H., Fu, J., Gui, T., Sun, T., Jiang, Y.-G., & Qiu, X. (2024). AnyGPT: Unified Multimodal LLM with Discrete Sequence Modeling. In L.-W. Ku, A. Martins, & V. Srikumar (Eds.), Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) (pp. 9637–9662). Association for Computational Linguistics. 10.18653/v1/2024.acl-long.521
  9. Tian, J., Lee, S., Kong, Z., Ghosh, S., Goel, A., & others. (2026). UALM: Unified Audio Language Model for Understanding, Generation and Reasoning. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2510.12000
  10. Pearce, M. T. (2018). Statistical Learning and Probabilistic Prediction in Music Cognition: Mechanisms of Stylistic Enculturation. Annals of the New York Academy of Sciences, 1423(1), 378–395. 10.1111/nyas.13654
  11. Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., Yang, A., Fan, A., & others. (2024). The Llama 3 Herd of Models. arXiv Preprint arXiv:2407.21783. https://arxiv.org/abs/2407.21783
  12. Carlini, N., Ippolito, D., Jagielski, M., Lee, K., Tramèr, F., & Zhang, C. (2023). Quantifying Memorization Across Neural Language Models. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2202.07646
  13. Delétang, G., Ruoss, A., Duquenne, P.-A., Catt, E., Genewein, T., Mattern, C., Grau-Moya, J., Wenliang, L. K., Aitchison, M., Orseau, L., & others. (2024). Language Modeling Is Compression. International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2309.10668
  14. Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., & others. (2020). Language Models are Few-Shot Learners. Advances in Neural Information Processing Systems, 33, 1877–1901. https://arxiv.org/abs/2005.14165
  15. Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., & others. (2022). Emergent Abilities of Large Language Models. Transactions on Machine Learning Research. https://arxiv.org/abs/2206.07682
LLMs × Music: Scalable Language Modeling Unlocks Music Applications
Overview of Tutorial
Overview of Large Language Models and Agents
Training Stages