Part 1: Music Making Through Suggestions

A musician might ask a generative model for a chord progression, a melody, or a drum loop, then keep only a small part of what it produces. The suggestion gives them something to try: they can listen, make changes, and see how it fits with their own ideas. This part examines systems designed for that kind of exchange, from generating new material to making existing recordings easier to edit.

What matters to creators¶

Drawing on studies of songwriters and musicians working with AI, we focus on four aspects of the creative experience: agency, control, exploration, and ownership 123. While these are not an exhaustive account of creativity, they offer a useful starting point for discussing how suggestion systems support creators.

Agency concerns the creator’s role in directing the work: deciding what to pursue, when to seek help, and which suggestions to use.

Control concerns the ability to carry out those decisions; for example, changing a harmony while keeping the melody, or revising only part of a phrase.

Exploration involves trying alternatives and discovering possibilities that were not part of the initial plan. A suggestion can be useful even when it is not used directly, if it prompts a new idea or helps the creator hear their material differently.

Ownership refers here to the feeling that the work is one’s own. This can come from contributing ideas, shaping the material, and recognising one’s own choices in the result.

Design considerations for suggestion systems¶

How can suggestion systems support these aspects of creativity while leaving room for creators to work in their own ways? There are many possible approaches. Here, we focus on four design considerations drawn from prior work: whether suggestions are steerable, modular and editable, diverse, and contextual 2413. We use these to examine how different systems help creators guide, explore, and revise musical ideas.

The rest of this part looks at three families of systems through the lens of these principles.

Overview: approaches to AI-assisted music creation¶

Compositional ideas¶

Symbolic suggestions (chords, melodies, lead sheets) are naturally modular and editable: a creator can drag a chord, change a note, or keep only the first two bars. The two systems below generate such suggestions from different kinds of input.

Amuse: from multimodal inspiration to chords¶

Songwriting is often driven by inspirations that are not necessarily musical: a photo, a line of a story, the feeling of a song someone else wrote. Amuse transforms multimodal (image, text, audio) inputs into chord progressions that can be seamlessly incorporated into songwriters’ creative process 1.

The key technical challenge is that there is no dataset of paired multimodal inputs and chords. Amuse addresses this in two steps (Figure 2):

1. Music keyword generation. A multimodal LLM converts a text or image input into music keywords (e.g., mellow, acoustic, indie pop).

2. Chord generation. An LLM proposes chord progressions for these keywords. Because LLM-generated chords are relevant but often musically incoherent, a unimodal chord model trained on real music is used to filter them (rejection sampling), keeping only progressions that are both relevant and musical.

Audio inputs take a separate path: a chord transcription pipeline extracts the progression directly from the recording.

Amuse was evaluated against Hookpad Aria, a generative assistant built into the Hookpad lead sheet editor 5. Aria suggests melody and harmony for the song being edited: it can continue existing material, fill in a missing span in the middle, or generate harmony from a melody and vice versa.

In a within-subjects study with 10 songwriters writing 8-bar choruses, Amuse was used alongside Aria. Participants felt more guided and better aligned with their creative goals than with Aria alone, and reported enhanced agency and creativity. Although Amuse was designed for the early stages of songwriting, participants used it throughout the process.

MIDI-LLM: text-to-MIDI by adapting an LLM¶

MIDI-LLM adapts a pretrained language model to generate editable MIDI from text descriptions 6. It adds MIDI tokens to the model’s vocabulary and trains it on music-related text, MIDI, and paired text–MIDI examples. The resulting model can generate multitrack music and be further trained for specific composition tasks.

For songwriting, the authors adapt MIDI-LLM to lead sheets (melodies and chord progressions) and evaluate it in Hookpad Aria. Creators can generate a new passage or fill in selected sections of the melody, chords, or both. They can guide suggestions with genre and mood tags, as well as musical attributes such as chord difficulty and chord–melody tension. The generated notes remain editable in Hookpad, allowing creators to revise the suggestions as they develop the song.

MIDI-LLM adapts a pretrained text LLM to music by extending its vocabulary and training it on text and MIDI.

Extending the vocabulary. The model starts from Llama 3.2 (1B parameters) and adds MIDI tokens while retaining the pretrained weights. Each note is represented by three tokens: onset time, duration, and instrument–pitch (). The embedding table combines the original text embeddings with the new MIDI embeddings:

where $\mathcal{V}_{\text{AMT}}$ is the MIDI vocabulary of the Anticipatory Music Transformer and $D$ is the LLM’s hidden size. The model retains the LLM architecture, allowing it to use optimized inference frameworks such as vLLM.

Training in two stages. Continued pretraining uses music-related text and standalone MIDI files to develop the model’s representations of musical language and structure. Supervised fine-tuning on paired text–MIDI examples then teaches it to generate music from text. Further fine-tuning on TheoryTab dataset adapts it to text-conditioned lead-sheet generation and infilling, with melody and chords as the output.

The MIDI-LLM paper reports that the model outperforms the recent Text2midi model in both text control and musical quality. An in-the-wild blind study with 58 Hookpad users and 4,002 generated outputs found that MIDI-LLM had the highest acceptance rate for lead sheet generation, compared with baselines without text control or without LLM pretraining 6.

Audio generation by time-varying controls¶

A text prompt can describe a genre or mood, but it is less convenient for specifying exactly when a melody should rise, a drum should hit, or the music should get louder. Time-varying controls let musicians specify these details along a timeline. Music ControlNet, Sketch2Sound, and TRIA use such controls to guide audio generation 789.

Although these models differ in architecture, they share a recipe:

1. Extract controls from the training audio. Compute features such as melody, loudness, or rhythm from each recording to create paired audio and controls without manual annotation.

2. Train a conditioned generator. The model learns to generate audio using these controls alongside a text or audio prompt.

3. Let the musician provide the controls. At inference time, extract compatible features from humming or tapping, or accept a directly drawn control curve.

This gives musicians a way to demonstrate an idea when words are awkward. They can tap the rhythm they want or hum a melodic contour, then use a separate prompt to guide the sound.

TRIA: The Rhythm In Anything¶

Tapping on a table or beatboxing can be enough to communicate a drum pattern. TRIA (The Rhythm In Anything) uses that recording to generate drums with the sound of a separate reference recording 9.

• Inputs: a tapped or beatboxed rhythm and a short drum recording with the desired timbre.

• Control: two feature streams that track low- and high-frequency activity over time.

TRIA follows the shared recipe step by step:

1. Extract rhythm features. TRIA computes a short-time Fourier transform (STFT) of the recording and divides the spectral energy equally between a low and a high band. The resulting features track activity in each band over time. This coarse representation reduces timbral detail, allowing sounds as different as a table tap and a kick drum to provide similar rhythm cues.

2. Train the generator. Each training recording provides acoustic tokens from the DAC neural audio codec and rhythm features from the audio. Augmentations such as noise, filtering, pitch and phase shifts, and equalization help the model handle rhythm inputs from different sound sources. A masked transformer learns to recover hidden acoustic tokens, using the visible tokens as a timbre reference and the rhythm features as timing cues.

3. Generate from the musician’s inputs. At inference time, the timbre reference is encoded as a fixed token prefix. The model fills a masked suffix using rhythm features extracted from the musician’s tapping or beatboxing. DAC then decodes the predicted tokens into audio.

The paper reports that TRIA, trained on fewer than 10 hours of publicly available drum recordings, can follow rhythmic gestures across a range of timbres without additional training on those gestures 9.

Recovering editable structure: music decomposition¶

Creators also need ways to revise material they already have. A drum recording may contain a useful groove, but moving one hit or replacing its sound is difficult while everything is mixed together. Decomposition systems recover parts that can be edited separately. Here, we look at two examples: turning drum audio into sequenced samples, and turning MIDI into program.

Both tasks need paired training examples: musical material and the components used to produce it. Since these pairs are scarce, both systems create them synthetically:

1. Synthesize training pairs by rendering known building blocks into material.

2. Train a language model to predict the building blocks from the material.

3. Run it on real material and let the creator edit the predicted blocks.

D-Composer: drum recordings → playable sound events¶

D-Composer converts a drum recording into a piano roll and a set of isolated drum sounds 10. A single language model predicts both the transcription (onset times and classes such as kick, snare, clap, and hi-hat) and the audio samples needed to play it. The result can be loaded into a DAW, where the musician can rearrange the pattern or change individual sounds.

Training starts with drum scores and samples whose contents are already known:

1. Synthesize. Render MIDI drum scores with one-shot samples and mix them into recordings. Each recording has a known score and source samples as its targets.

2. Train. An encoder–decoder language model learns to predict drum classes, onset times, and acoustic tokens for the one-shot samples from the synthetic recording.

Decomposer: MIDI → live coding programs (Strudel.js)¶

MIDI stores individual note events, but a passage may be easier to edit as a pattern: repeat this bass line four times, then transpose it. Decomposer converts MIDI into Strudel, a live-coding language, so musicians can work with those patterns in code 11.

The task has two difficulties. Strudel is a low-resource language with almost no naturally paired (MIDI, code) data, and optimizing only for faithful reconstruction can collapse to unreadable note-by-note transliteration. Decomposer handles both with the decomposition recipe:

1. Synthesize. A frontier LLM generates Strudel programs, which are cleaned with regular-expression and abstract-syntax-tree checks, then executed in the Strudel runtime to produce MIDI. The resulting dataset, Strudel-Synth, contains 21,174 program–MIDI pairs.

2. Train in two stages. Supervised finetuning on Strudel-Synth teaches the model to write valid, idiomatic Strudel. Reinforcement learning on unpaired MIDI then optimizes decompilation directly, rewarding sampled programs for both faithfulness (does the rendered MIDI match the input?) and readability (is the code structured rather than a note dump?).

The paper reports that the resulting 8B open-weight model reconstructs input MIDI more faithfully than the closed-weight frontier LLMs evaluated, while maintaining comparable code readability 11.

Takeaways¶

Creators may use only part of a suggestion, change it, or take it as inspiration for something else. Its value lies partly in how it helps them develop their own ideas. When evaluating these systems, it is worth asking not only whether creators accept a suggestion, but also what it helps them do next.

These systems also face a common challenge: suitable training pairs are often scarce. Amuse works around the lack of paired inspirations and chords by combining an LLM with a chord model. Time-varying control systems extract control signals from recordings, while decomposition systems render known components into audio or MIDI to create training pairs. These approaches show how useful supervision can be built from existing models, recordings, and musical representations.