Music as Audio

Most music reaches listeners as sound, and most modern Music LLMs read audio. This page follows the reel from its waveform to the tokens an LLM reads, then runs it through an open audio LLM.

From waveform to tokens¶

A digital recording is a waveform: a list of air-pressure values, sampled tens of thousands of times per second. The reel file has 22,050 samples per second, so its eight seconds hold about 176,000 numbers (Figure 1). A three-minute stereo song at the CD rate of 44,100 Hz holds about 16 million. This direct approach would thus pose intractable to efficient language modeling, where an LLM would need to process millions of samples individually in its context. Every audio LLM thus first compresses the signal.

The compression has up to three parts (Figure 2). An optional pre-transform rearranges the waveform with a fixed signal processing formula, with nothing learned. A learned encoder turns the result into a much shorter sequence, of continuous vectors or of discrete codes. An adapter maps that sequence into the LLM’s token embedding space. Models differ in which parts they keep.

Optional pre-transforms¶

A pre-transform is a deterministic, generally non-learned function of the waveform. It builds an inductive bias into the model before learning starts.

Spectrograms. The short-time Fourier transform (STFT) cuts the signal into short overlapping windows and measures how much of each frequency each window contains. The result is a complex number per frequency and time: its magnitude says how strong the frequency is, and its phase says where in its cycle the wave is. While magnitude carries most of what listeners hear, phase looks like noise to the untrained eye, so models that use a spectrogram usually keep the magnitude only, which thus cannot be turned back into the exact waveform (Figure 3).

Two transformed versions are common. A mel spectrogram merges the frequency bins into bands that are narrow at low frequencies and wide at high ones, roughly following human pitch perception. Whisper’s encoder reads 80 mel bands computed on 25 ms windows every 10 ms, from audio resampled to 16 kHz 1, and Qwen2-Audio’s reads 128 bands with the same window and hop 2. A constant-Q transform (CQT) spaces its bins geometrically, so each bin can be one semitone. In the CQT panel, the reel’s notes sit on a piano-roll-like grid. A pre-transform can also serve as a training target on top of an input: MERT learns to predict a CQT of the audio it cannot see 3.

All four views take a few lines of librosa 4:

import numpy as np, librosa

y, sr = librosa.load("reel.mp3", sr=22050)        # 176,392 samples
S = librosa.stft(y, n_fft=2048, hop_length=512)   # complex, 1025 bins x 345 frames
magnitude_db = librosa.amplitude_to_db(np.abs(S), ref=np.max)
phase = np.angle(S)

mel = librosa.feature.melspectrogram(y=y, sr=sr, n_fft=2048, hop_length=512, n_mels=128)
mel_db = librosa.power_to_db(mel, ref=np.max)    # 128 bands x 345 frames

cqt = librosa.cqt(y, sr=sr, hop_length=512, fmin=librosa.note_to_hz("C2"),
                  n_bins=72, bins_per_octave=12)  # one bin per semitone, C2 to B7
cqt_db = librosa.amplitude_to_db(np.abs(cqt), ref=np.max)

Rearrangements that keep the waveform. Other pre-transforms keep (almost) all the information and only shorten the sequence by making it wider. A polyphase split sends every $M$-th sample to one of $M$ channels, so the sequence becomes $M$ times shorter and $M$ times wider, and nothing is lost. Cutting the signal into non-overlapping frames of $M$ samples is the same reshape: Gemma 4 12B’s 40 ms chunks of 16 kHz audio are frames of 640 samples, which become 640-dimensional vectors 5. A pseudo-quadrature mirror filter (PQMF) bank instead splits the signal into $M$ frequency bands, each decimated by $M$. PQMFs and polyphase filters are not common pre-transforms in music-language modeling, but they are of a similar class of signal-processing operations that are valid for pre-tranformation.

Whether to pre-transform is a design choice, not a requirement (Table 1). Many encoders read a mel spectrogram, but neural audio codecs such as EnCodec read the raw waveform directly 63, and so does Jukebox 7. A lossy pre-transform is fine for a model that only needs to understand audio. Models that must rebuild the waveform from their own representation, such as codecs, read the waveform or a rearrangement that keeps it, because a magnitude spectrogram has lost the phase.

The encoder¶

An audio encoder is a neural network that turns the waveform, or its pre-transform, into a much shorter, more semantically meaningful sequence. Each step of the output summarizes a short stretch of time. The output can be continuous vectors, or discrete codes from a fixed codebook.

Continuous encoders¶

Audio LLMs may get a continuous encoder in one of three ways (Figure 4).

Borrowed encoders. The first music LLMs reused an encoder trained for something else, often frozen.

• Whisper was trained to transcribe speech, on 680,000 hours of labeled audio 1. Qwen2-Audio initializes its encoder from Whisper-large-v3 and adds a pooling layer with stride two, so each output vector covers about 40 ms 2. SALMONN runs two encoders side by side, Whisper’s for speech and BEATs 8 for other sounds, both at 50 vectors per second 9.

• MERT is trained on music without labels. It learns to predict two targets for masked stretches of audio: codes from an acoustic tokenizer, such as EnCodec, and a CQT 3. It comes in sizes from 95M to 330M parameters. MusiLingo uses MERT-v1-330M 10, and MU-LLaMA uses a frozen MERT encoder 11.

• Jukebox is a music generation model that works on the raw waveform 7. LLark takes the output of layer 36 of Jukebox-5B’s encoder, 4,800-dimensional vectors at 345 Hz, and mean-pools them in 100 ms frames, so the LLM gets 10 vectors per second 12.

A borrowed encoder brings what it learned from its own training data, but it was not built for the LLM. Whisper’s training targets are transcripts, which say little about harmony or timbre. Such encoders’ training data may vary wildly from the data used to adapt an LLM to its music representation, potentially kneecapping the overall performance.

Encoders trained for the LLM. Newer models train the encoder as part of the LLM project. Audio Flamingo 2 trains its own version of CLAP, AF-CLAP, on 8,037,712 ten-second audio–caption pairs 13. Audio Flamingo 3 starts from Whisper large-v3, attaches a text decoder, and trains the pair to generate audio captions, to get AF-Whisper 14. Qwen3-Omni replaces Whisper with AuT, an encoder of about 0.6B parameters trained from scratch on 20 million hours of supervised audio. It downsamples its 128-band mel spectrogram eight times, to 12.5 vectors per second, about 80 ms each 15.

No encoder. In 2026, two model families dropped the audio encoder. Gemma 4 12B segments raw 16 kHz audio into 40 ms chunks, which gives 640-dimensional vectors that are “projected directly into the LLM embedding space” 5. Images get the same treatment: 48×48 RGB patches pass through a single matrix multiplication instead of a 550M-parameter vision encoder. Note that the smaller Gemma 4 models, E2B and E4B, still use a 305M-parameter audio encoder. Inkling, an open-weights mixture-of-experts model with 975B total and 41B active parameters, inputs audio as dMel spectrograms, a quantized mel representation, and passes them through “a light-weight embedding layer” processed jointly with text tokens 16. Without an encoder, the pre-transform is the only fixed choice left: a reshape for Gemma 4 12B, a quantized mel spectrogram for Inkling.

The three columns trade borrowed knowledge for flexibility. A borrowed encoder works with little paired data, but it fixes the sample rate, the frame rate, the clip length it was trained on, and its knowledge cutoff. An encoder-free model makes almost no such choices, but the LLM itself must learn to hear, which may take far more data and compute.

Discrete encoders: codecs and tokenizers¶

A neural audio codec is an encoder–decoder trained to compress a waveform into a discrete collections of codes and rebuild it. Its encoder reads the raw waveform and outputs a vector per downsampled frame, and residual vector quantization (RVQ) replaces each vector with a few codes: the first codebook picks the nearest entry, the next codebook quantizes what is left over, and so on. EnCodec outputs 75 frames per second at 24 kHz 6; as MERT uses it, each frame becomes 8 codes 3. DAC compresses 44.1 kHz audio about 90 times, at 86 frames per second 17. The decoder turns codes back into sound, so the same codes serve for listening and for generating.

Codes are tokens, so an LLM can read them with no continuous vectors at all. AnyGPT tokenizes music with an EnCodec model at 32 kHz and 50 frames per second, with four codebooks of 2,048 codes each, on a LLaMA-2 7B backbone 18. Discrete tokens need not come from a codec. AudioPaLM takes its tokens from speech models such as w2v-BERT, whose tokens come at 25 per second from a vocabulary of 1,024 19. Because the audio is ordinary tokens, the same LLM can also generate audio (Chapter 1 describes this briefly). The cost is length: a codec keeps the detail needed to rebuild the waveform (inherently, it is what one would call an acoustic representation), and if each of EnCodec’s 8 codes per frame became its own token, one second would take 600 tokens.

Adapters¶

The encoder’s output often does not match the LLM’s token embeddings in size or in meaning, so an adapter translates it. For continuous vectors there are two main designs (Figure 5); for discrete codes, the adapter can be just an embedding table.

Projection. The simplest adapter applies the same linear layer, or a small MLP, to every encoder vector. LLaVA connects an image encoder to an LLM with “a trainable projection matrix” 20, and LLark’s projection module “is parameterized by a single linear projection layer” 12. Qwen2-Audio’s adapter, as implemented in transformers, is a single linear layer too. MusiLingo adds a temporal compression step after its linear layer 10. With projection, the number of tokens grows with the length of the audio.

Q-Former. BLIP-2 introduced the Querying Transformer to connect a frozen image encoder to a frozen LLM 21. It is a small Transformer, initialized from BERT-base, that holds 32 learned query vectors of 768 dimensions. The queries attend to each other with self-attention and to the encoder’s outputs with cross-attention layers, inserted every other block. Only the 32 query outputs pass on, through a fully connected layer, to the LLM. The image, however many patches it has, becomes 32 tokens. SALMONN applies the same idea to audio in windows: one query per window of 17 encoder frames, about 0.33 s, which gives 88 tokens for 30 seconds of audio 9. Flamingo’s Perceiver Resampler works in a similar way and outputs a fixed 64 visual tokens 22.

A Q-Former controls the token budget, but it has to decide what to keep before the LLM sees the question. Projection keeps everything and leaves the choice to the LLM’s attention, at the cost of more tokens. Note that Q-Forming has largely fallen out of fashion, as the desire for more fine-grained detailed information for music-LLMs has taken precedence over length concerns.

New rows in the embedding table. For discrete codes, each code gets its own row in the LLM’s embedding table, like a new word. AudioPaLM “expand[s] the size of the embeddings matrix”, and the new audio embeddings “are freshly initialized and must be trained” 19. GLM-4-Voice expands GLM-4’s vocabulary with speech tokens in the same way 23, and MIDI-LLM does it for MIDI events (General LLMs and music LLMs). Practically, this is just a discrete version of the continuous projection.

What sequence is the music in, and where?¶

Once adapted, the music has to reach the LLM’s computation. There are two places to put it (Figure 6).

In the sequence. The adapted tokens are placed in the input next to the text, and the LLM’s ordinary self-attention reads them. Every layer can mix music and text, and nothing in the LLM changes except its inputs. LLaVA, LLark, and Qwen2-Audio work this way. The cost is growing self attention context.

Beside the sequence. Flamingo freezes the LLM and inserts new gated cross-attention layers between its layers 22. Text tokens stay in the sequence; the new layers let them attend to the visual (or audio) features. Each new layer multiplies its output by $\tanh(\alpha)$, where $\alpha$ is a learned scalar that starts at 0, so at the start of training the model is exactly the original LLM. Audio Flamingo uses these layers with a CLAP- based feature extractor and 7-second sliding windows, up to 16 windows or 33.25 seconds 24. Audio Flamingo 2 keeps the design and extends it to 30 windows, five minutes of audio 13. Note that like Q-Forming, cross-attention conditioning has fallen out of practice given the larger support (both in terms of growing scientific research and practical engineering developments) for pure decoder-only models.

Prefix, coarse, and fine interleaving¶

Inside the sequence, there is still a choice of order (Figure 7).

• Prefix. One clip comes first, then the question. This is the usual setup for single-turn question answering and captioning, as in Qwen2-Audio, SALMONN, and LLark, and the one the code below uses.

• Coarse interleaving. Whole clips sit between turns of text, as in a chat that compares two recordings, or a prompt with few-shot examples. Audio Flamingo 3 trains on “75k multi-turn, multi-audio chat instances” for this 14.

• Fine interleaving. Short chunks of text and audio alternate on a shared timeline, so that each piece of text sits next to the sound it describes or accompanies. CosyVoice 2 alternates 5 text tokens with 15 speech tokens at 25 speech tokens per second 25, GLM-4-Voice alternates 13 text and 26 speech tokens at 12.5 per second 23, and Spirit LM switches between speech and text at word boundaries 26. Qwen2.5-Omni does the same for video and its audio, cutting both “into chunks every 2 seconds according to the actual time” 27.

For fine interleaving, positions should follow time, not just token order. Qwen2.5-Omni’s TMRoPE position encoding gives each token a position tied to time, and Music Flamingo’s RoTE likewise sets the rotation angle “using absolute timestamps rather than token indices” 28. The speech models above use fine interleaving to speak while they write. For music, it fits time-aligned outputs, such as lyrics, chords, or captions for each section, and questions about when something happens in a song.

The token budget¶

The pre-transform, the encoder, and the adapter together set how many tokens a minute of music costs, and that sets how much music fits in a context window. Table 2 lists the rates of the models on this page.

Longer audio needs a longer context. Audio Flamingo 3 had a context of 8,192 tokens and about 10 minutes of audio; Music Flamingo extends it to about 24,000 tokens, enough for songs of up to 20 minutes 28. Whisper-based encoders also inherit Whisper’s 30-second training segments 1.

Asking an audio LLM about the reel¶

Qwen2-Audio-7B-Instruct is an open audio LLM on Hugging Face. It is a Qwen-7B LLM with a Whisper-large-v3-based encoder, 8.2B parameters in total 2. With the transformers library, its processor turns a chat into token IDs and audio features in one call:

# pip install transformers accelerate librosa
from transformers import AutoProcessor, Qwen2AudioForConditionalGeneration

model_id = "Qwen/Qwen2-Audio-7B-Instruct"
processor = AutoProcessor.from_pretrained(model_id)
model = Qwen2AudioForConditionalGeneration.from_pretrained(
    model_id, dtype="auto", device_map="auto")

conversation = [
    {"role": "user", "content": [
        {"type": "audio", "audio": "reel.mp3"},
        {"type": "text", "text": "What key is this piece in?"},
    ]},
]
inputs = processor.apply_chat_template(
    conversation, add_generation_prompt=True, tokenize=True,
    return_dict=True, return_tensors="pt",
).to(model.device)

n_audio = (inputs["input_ids"] == model.config.audio_token_index).sum().item()
print(n_audio)  # 200: eight seconds at 25 audio tokens per second

output = model.generate(**inputs, max_new_tokens=64)
answer = processor.batch_decode(
    output[:, inputs["input_ids"].shape[1]:], skip_special_tokens=True)[0]
print(answer)

Try the three questions from the overview on the reel, and then ask the same questions with no audio at all. The evaluation page explains why the second run matters.