Part 2: Real-Time Music Co-Creation

In Part 1, creators listen to suggestions and decide what to do next. During improvisation, those decisions happen while the music continues. A musician might change the melody, hold a note, or introduce a new rhythm, and an AI partner needs to respond in time. This part examines systems designed for that exchange and the technical choices that make it possible.

Making music through improvisation¶

Playing an instrument or singing can be a way to work out an idea before knowing exactly where it will go 1. An AI system that joins this process could provide:

• a partner to practise or improvise with when other musicians are unavailable,

• responses that reflect aspects of the musician’s own style, offering a way to hear it differently 2, and

• opportunities to explore forms of interaction beyond human performance capabilities, including unusually fast responses 3.

Applying LLMs to live performance remains an emerging area 3. Alongside musical quality, developers must consider when a response arrives and how it fits the ongoing performance.

Processing music as it arrives¶

Real-time audio software typically processes sound in short frames, also called blocks or buffers. In a simple pipeline, the system computes the next output frame while the current one plays. If that output is not ready in time, playback may contain a gap or glitch 4.

This example separates two timing constraints. Latency is the delay between input and its corresponding output; the illustrated pipeline introduces one frame of delay. Compute budget is the time available to prepare each output frame. To sustain playback, processing must keep pace with the frame duration. Smaller frames can reduce buffering delay, but leave less time to compute each one. Actual systems may introduce additional delay through audio hardware, buffering, and communication.

Generative models may work with audio blocks, codec tokens, or beat-level musical events. These units differ, but each system needs to deliver its output before the scheduled playback time.

What makes real-time generation hard?¶

1. Tight latency budget. Closely coordinated playing can require responses on the order of tens of milliseconds, with less than roughly 100 ms often used as a design target. The acceptable delay depends on the interaction. Server GPUs can support larger models but introduce network delay; local inference avoids that communication cost but has fewer computing resources 325.

2. No future context. The model must choose what to play before hearing everything it will accompany. A chord scheduled for the next beat may already be committed when the musician’s melody note arrives.

3. Responsiveness and throughput. Generating larger chunks can be more efficient, but the model then has fewer opportunities to adjust to new (and unexpected) input.

Designing for live interaction¶

Meeting these constraints involves both model design and the software around it. The examples below make changes at training, inference, and deployment 2678.

Overview: real-time music co-creation¶

We consider two groups of systems, based on whether they generate symbolic music or audio.

Symbolic systems generate notes or chords that can be played through a synthesizer or other MIDI-compatible instrument. Audio systems generate the sound itself, including timbre and performance details. This gives them different expressive possibilities, but also increases the amount of information they must produce in time.

Symbolic music accompaniment: ReaLchords and ReaLJam¶

ReaLchords generates chord accompaniment to a melody as it is played 6. ReaLJam builds on this work with an interface and communication protocol for live interaction 9. Together, they illustrate two parts of the problem: learning to accompany without future context, and delivering those predictions in a playable system.

Training ReaLchords: an online student and an offline teacher¶

ReaLchords must choose chords without knowing the upcoming melody. Its online model (the student) uses only past melody notes and chords. Learning from existing songs alone does not fully solve this problem: the harmonies in those songs were composed with knowledge of the melody, while the live model has only partial information.

To improve accompaniment under this constraint, ReaLchords fine-tunes the student with reinforcement learning using two sources of guidance

• An offline teacher has access to the complete melody. A KL-divergence term encourages the student’s chord predictions to stay close to the teacher’s distribution, even though the student has less information.

• Reward models score how well the generated chords fit the melody. Contrastive and discriminative models learn from real melody–chord pairs and mismatched examples.

Here $\pi_\theta$ is the online policy, $\phi$ the offline teacher, $x$ the melody, $y$ the chords, and $R$ the combined reward.

ReaLJam: making it playable¶

ReaLJam connects a server-side model (ReaLchords) to a musician playing in a browser. It needs to account for computation and communication delays while keeping the accompaniment responsive. Three features help manage this 9:

1. Plan ahead. The agent continually predicts four beats of melody and chords. The predicted melody is its estimate of what the musician will play, giving it a basis for preparing the accompaniment.

2. Commit near playback. Chords within a short commit window are fixed so they can be scheduled reliably. Later chords remain open to revision as new notes arrive.

3. Show upcoming chords. The interface displays the plan, helping the musician anticipate the accompaniment and respond to it.

Symbolic co-performance: jam_bot¶

Jam_bot supports live human–AI improvisation and was developed at the MIT Media Lab with keyboardist Jordan Rudess 2. It debuted in concert in 2024, bringing model design, personalisation, and performance software into the same project.

Its design combines several forms of interaction with training and inference choices tailored to live use:

• Interaction modes. The musician and agent can trade phrases, the agent can accompany a melody, or it can play a melody over the musician’s chords. These modes use different context and conditioning signals for the music language model.

• Personalization. Models are fine-tuned on Rudess’s recordings, which he structured to demonstrate the playing styles needed for particular pieces.

• Inference and scheduling. Separate threads handle listening, model requests, and playback scheduling. Notes are generated ahead of time so they are ready to play on schedule. A follow-up implementation moved from ONNX Runtime to ggml 10. For a 170M-parameter model with a 120-token context, it reduced generation time from about 4 to 2 ms per token on an RTX 4090; inference on an M3 Max laptop took about 10 ms per token.

Audio co-performance¶

Audio generation brings timbre and texture into the interaction, alongside notes and rhythm. Streaming models produce sound in chunks and allow users to influence the music as it plays. The examples here support three overlapping forms of interaction:

1. Text control. Change the prompt while the model plays, and the music morphs into the new style (e.g., Magenta RT, LMDM).

2. Respond to your playing. The model hears the musician’s live audio and answers or varies it (e.g., Magenta RT’s audio injection, LMDM’s generative delay).

3. Accompaniment. The model generates a matching part for another instrument’s live audio or MIDI (e.g., streaming accompaniment, LMDM, applications built on Magenta RT 2).

Approaches to streaming audio¶

Magenta RT / Live Music Models. Magenta RealTime generates audio continuously and accepts text and audio prompts 5. Its first version uses a 750M-parameter model. With audio injection, live input is mixed into the audio context used for generation, allowing the musician’s playing to influence what comes next. Magenta RealTime 2 includes 2.4B- and 230M-parameter models designed for local use, with MIDI, text, and audio controls 11.

Live Music Diffusion Models (LMDM). Diffusion-based music generators commonly produce a complete clip in one pass through the generation process. LMDM adapts Stable Audio Open Small for streaming 7, with changes to inference, training, and live interaction:

• Reuse earlier computation. Block-wise KV caching lets new audio blocks reuse computation from previous blocks. In the paper’s comparison, time to first frame was about 0.03 seconds for LMDM and 4 seconds for Magenta RT.

• Train on generated context. ARC-Forcing uses adversarial post-training on the model’s own rollouts to reduce error accumulation during long generations, without explicit reinforcement learning or reward models.

• Use delay as part of the interaction. LMDM runs on a consumer gaming laptop as a generative delay. It buffers $S$ seconds of input, computes for $\tau$ seconds, and produces an $S$-second output block. The resulting delay is $\Delta = S + \tau$. Sustained streaming requires $\tau < S$; the roughly one-second delay becomes an audible transformation of the musician’s playing, rather than an attempt at immediate accompaniment.

Streaming generation for accompaniment. Here, the model must generate an accompanying part while listening to a live input, such as a singer. Wu et al. (2026) study two variables: future visibility $t_f$, which specifies how far the available input extends relative to output playback, and chunk size $k$, the amount generated per call. Negative future visibility means the model must predict ahead to compensate for system delay.

Their experiments with transformer decoders show a trade-off. Access to later input improves coherence, but leaves less time to compute before playback. Larger chunks improve throughput, but reduce how often the accompaniment can respond. The study also finds that maximum-likelihood training alone struggles to produce coherent accompaniment without future context, motivating approaches that explicitly train anticipation.