Fine-tuning a performance coach¶
This notebook builds a query-based piano coach in the style of LLaQo (Zhang et al., 2025): a LoRA adapter on the Thinker of Qwen2.5-Omni-3B, trained with LLaMA-Factory on expert question-answer pairs over student recordings. A general audio model answers the same questions fluently without a correct diagnosis (Bang et al., 2026), so the data is crucial here.
The worked dataset is NeuroPiano (Zhang et al., 2024). The LLaQo adapter was trained on the full LLaQo instruction set, twelve piano datasets and about 34K QA pairs; this notebook trains on NeuroPiano alone, the one recorded and annotated for this purpose and released with its complete annotation schema. The notebook goes data, adapter configuration, training recipe, inference, failure modes.
What gets adapted¶
Qwen2.5-Omni splits into a Thinker and a Talker (Xu et al., 2025). The Thinker is a transformer decoder with audio and vision encoders in front of it: it consumes the modalities and produces text. The Talker takes the Thinker’s hidden states and emits speech tokens.
A coaching model needs the Thinker only. The adapter is trained on Qwen2.5-Omni-3B/thinker, and at inference the Talker is switched off. Load the adapter onto model and PEFT will not find its target modules, because they live one level down under model.thinker.
%pip install -q torch torchaudio transformers accelerate peft datasets soundfile qwen-omni-utils[decord] llamafactory==0.9.5The data: NeuroPiano¶
NeuroPiano (Zhang et al., 2024) is a set of 104 short recordings by 39 advanced student pianists, each playing one of six technical exercises (scales, arpeggios, third dyads, block chords, octaves) at a prescribed tempo and dynamic. 45 professional pianists rated each recording on a 0 to 6 scale and wrote free-text feedback, in Japanese, against a fixed list of 13 questions: two open ones (what is good, what is bad) and eleven closed ones, each naming one dimension such as tempo constancy, evenness of legato, or balance between the hands. The release on Hugging Face (anusfoil/NeuroPiano-data, MIT licence) adds English translations and span annotations that mark, inside each answer, the body part, the technique, the location in the excerpt, and the evaluative adjectives the teacher used.
The dataset is small in hours (every clip is 2 to 9 seconds) and dense in labels: 2,265 audio-question-answer rows over 104 recordings, each recording answered by one to three teachers.
from datasets import load_dataset, Audio
train = load_dataset("anusfoil/NeuroPiano-data", split="train")
evals = load_dataset("anusfoil/NeuroPiano-data", split="eval")
# keep the WAV bytes but skip decoding until we write them out
train = train.cast_column("audio_path", Audio(decode=False))
evals = evals.cast_column("audio_path", Audio(decode=False))
print(train)
print(f"train: {len(train)} rows, {len(set(train['fname']))} recordings, {len(set(train['subject']))} pianists")
print(f"eval: {len(evals)} rows, {len(set(evals['fname']))} recordings, {len(set(evals['subject']))} pianists")
print(f"pianists in both splits: {len(set(train['subject']) & set(evals['subject']))}")Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
WARNING:huggingface_hub.utils._http:Warning: You are sending unauthenticated requests to the HF Hub. Please set a HF_TOKEN to enable higher rate limits and faster downloads.
Dataset({
features: ['audio_path', 'subject', 'piece', 'user_id', 'playdata_id', 'fname', 'good_or_bad', 'mean', 'idx', 'ses', 'ses_pianist', 'question_id', 'question_source_id', 'question', 'answer', 'q_eng', 'a_eng', 'score', 'opinion', 'location', 'physical', 'technique', 'adjectives', 'suggested_location', 'suggested_physical', 'suggested_technique', 'suggested_adjectives', 'split'],
num_rows: 1167
})
train: 1167 rows, 52 recordings, 33 pianists
eval: 1098 rows, 52 recordings, 34 pianists
pianists in both splits: 32
import json
row = train[3]
show = {k: v for k, v in row.items() if k not in ("audio_path", "user_id", "playdata_id", "idx", "ses", "ses_pianist", "question_source_id", "split")}
print(json.dumps(show, ensure_ascii=False, indent=2)){
"subject": "sub_051",
"piece": "S-11_84_pp",
"fname": "63218a05c6190f3921803bc1_644634ef0d12a78974fb822d.wav",
"good_or_bad": "bad",
"mean": 2.5999999046325684,
"question_id": 5,
"question": " 響の芯があリますか",
"answer": "親指の音だけ芯があるように聞こえるがそのほかの音は芯が少し弱い",
"q_eng": "How solid is the sound?",
"a_eng": "Only the sound of the thumb seems to have a solid core, while the other notes are slightly weaker.",
"score": 3,
"opinion": "mixed",
"location": null,
"physical": "thumb",
"technique": null,
"adjectives": "solid sound, weaker sound",
"suggested_location": null,
"suggested_physical": null,
"suggested_technique": null,
"suggested_adjectives": null
}
One row is one teacher answering one question about one recording. piece encodes exercise, tempo and dynamic (S-11 at 84 bpm, pp). score is the teacher’s rating for that question; opinion collapses it to affirm, mixed, or deny. physical, technique, location and adjectives are spans copied from the answer. good_or_bad and mean are recording-level aggregates across all teachers and questions.
The training target is the English answer: one or two sentences that name a body part or a technique and attach an adjective to it. Nothing here is a paragraph.
from collections import Counter
counts = Counter((r["question_id"], r["q_eng"].strip()) for r in train)
for (qid, q), n in sorted(counts.items()):
print(f"{qid:2d} {n:4d} {q}")
print("\nscore distribution:", sorted(Counter(train["score"]).items())) 1 100 What's good about this performance?
2 102 What's bad about this performance?
3 74 Is the legato even?
4 101 Are the note values uniform?
5 102 How solid is the sound?
6 101 How clean is the attack?
7 42 Are the left and right hands balanced?
8 43 Are the timings onsets aligned on the left and right hands?
9 100 Is it played with the correct rhythm?
10 102 Is the tempo kept constant?
11 100 Are the lines connected?
12 101 Is it played with a sense of tonality?
13 99 Is the dynamics change natural?
score distribution: [(0, 39), (1, 91), (2, 181), (3, 315), (4, 184), (5, 194), (6, 163)]
Three properties of this table decide how the model is trained and evaluated.
Two question types. Questions 1 and 2 are open and want a description; questions 3 to 13 are closed and carry a rating. LLaQo trains both from the same rows by putting the rating into the answer text of closed questions, so one generation head serves both. Its rating error on the held-out half of NeuroPiano was a mean absolute error of 0.97 on the 6-point scale (Zhang et al., 2025).
A label-noise floor. Two teachers rating the same clip agree within 2 points about 65% of the time, and on the dynamics and legato questions only about 50% (Zhang et al., 2024). An MAE near 1 is therefore close to the agreement between teachers, and a smaller number would mean overfitting one annotator’s habits rather than learning the dimension.
Split by recording, not by pianist. The train and eval halves share no recordings but 32 of the 33 training pianists also appear in eval. A model can learn a pianist’s tone and be rewarded for it. For any claim about generalisation to new students, regroup the split on subject before training.
From rows to training examples¶
LLaMA-Factory (Zheng et al., 2024) reads multimodal SFT data as a list of conversations with a parallel audios list. The cell below writes each recording to disk once, then emits one conversation per row. Closed questions get the rating prepended to the answer so the model learns to produce it.
import json, os
os.makedirs("data/neuropiano/audio", exist_ok=True)
def to_example(r):
wav = f"data/neuropiano/audio/{r['fname']}"
if not os.path.exists(wav):
with open(wav, "wb") as f:
f.write(r["audio_path"]["bytes"])
answer = r["a_eng"].strip()
if r["question_id"] >= 3: # closed question: carries a rating
answer = f"Rating: {r['score']}/6. {answer}"
return {
"messages": [
{"role": "user", "content": "<audio>" + r["q_eng"].strip()},
{"role": "assistant", "content": answer},
],
"audios": [f"neuropiano/audio/{r['fname']}"], # relative to data/
}
examples = [to_example(r) for r in train]
with open("data/neuropiano/neuropiano_qa.json", "w") as f:
json.dump(examples, f, ensure_ascii=False, indent=1)
print(len(examples), "examples,", len(os.listdir("data/neuropiano/audio")), "wav files")
print(json.dumps(examples[3], ensure_ascii=False, indent=2))1167 examples, 52 wav files
{
"messages": [
{
"role": "user",
"content": "<audio>How solid is the sound?"
},
{
"role": "assistant",
"content": "Rating: 3/6. Only the sound of the thumb seems to have a solid core, while the other notes are slightly weaker."
}
],
"audios": [
"neuropiano/audio/63218a05c6190f3921803bc1_644634ef0d12a78974fb822d.wav"
]
}
Register the file in LLaMA-Factory’s data/dataset_info.json:
dataset_info = {
"neuropiano_qa": {
"file_name": "neuropiano/neuropiano_qa.json",
"formatting": "sharegpt",
"columns": {"messages": "messages", "audios": "audios"},
"tags": {"role_tag": "role", "content_tag": "content",
"user_tag": "user", "assistant_tag": "assistant"},
}
}
with open("data/dataset_info.json", "w") as f:
json.dump(dataset_info, f, indent=2)
print(json.dumps(dataset_info, indent=2)){
"neuropiano_qa": {
"file_name": "neuropiano/neuropiano_qa.json",
"formatting": "sharegpt",
"columns": {
"messages": "messages",
"audios": "audios"
},
"tags": {
"role_tag": "role",
"content_tag": "content",
"user_tag": "user",
"assistant_tag": "assistant"
}
}
}
The LoRA configuration¶
LoRA (Hu et al., 2022) freezes the base weights and trains a low-rank update on selected linear layers. The values below are the configuration of the LLaQo adapter, read from its adapter_config.json. The training run below uses the same values.
adapter_config = {
"base_model_name_or_path": "Qwen2.5-Omni-3B/thinker",
"peft_type": "LORA",
"task_type": "CAUSAL_LM",
"r": 16,
"lora_alpha": 32,
"lora_dropout": 0.05,
"bias": "none",
"use_rslora": False,
"use_dora": False,
}
for k, v in adapter_config.items():
print(f"{k:28s} {v}")
# LLaMA-Factory reports 29,933,568 trainable params and upcasts them to float32,
# so adapter_model.safetensors is about 120 MB at 4 bytes per param
print(f"\n{'trainable params':28s} {29_933_568 / 1e6:.1f}M of 4.73B (0.63%)")base_model_name_or_path Qwen2.5-Omni-3B/thinker
peft_type LORA
task_type CAUSAL_LM
r 16
lora_alpha 32
lora_dropout 0.05
bias none
use_rslora False
use_dora False
trainable params 29.9M of 4.73B (0.63%)
Rank 16 with alpha 32 is the common default. The target list is the part to read: that config enumerates every attention projection (q_proj, k_proj, v_proj, o_proj) and every MLP projection (gate_proj, up_proj, down_proj) across all Thinker layers, which is what LLaMA-Factory writes when asked for lora_target: all. Adapting attention alone is usually enough for a new voice; adapting the MLPs as well is what a new vocabulary needs, and “solid core of the sound” or “4th finger hard to hear” is new vocabulary for this model.
%%writefile llama_factory_coach.yaml
model_name_or_path: Qwen/Qwen2.5-Omni-3B
trust_remote_code: true
stage: sft
do_train: true
finetuning_type: lora
lora_rank: 16
lora_alpha: 32
lora_dropout: 0.05
lora_target: all
dataset: neuropiano_qa # registered in data/dataset_info.json
template: qwen2_omni
cutoff_len: 4096
preprocessing_num_workers: 8
output_dir: saves/piano-coach-lora
per_device_train_batch_size: 1
gradient_accumulation_steps: 8
learning_rate: 1.0e-4
num_train_epochs: 3.0
lr_scheduler_type: cosine
warmup_ratio: 0.1
bf16: true
logging_steps: 10
report_to: none
overwrite_output_dir: trueWriting llama_factory_coach.yaml
Rank, alpha, dropout and target set are read from the LLaQo adapter. Learning rate, schedule and epochs are conventional starting points for a 3B model on a few thousand short examples; with 1,167 rows, three epochs at an effective batch of 8 is about 440 optimiser steps. The 13 question strings are the only prompts the adapter ever sees, so a paraphrased question at inference time is already out of distribution.
Training¶
Training needs a GPU with bf16 support, such as an L4 or A100. A T4 does not support bf16. On Colab, choose the GPU under Runtime, Change runtime type. The adapter is written to saves/piano-coach-lora.
On a Colab A100 the cell runs for about 47 minutes. The base model downloads in under a minute, and tokenising the 1,167 examples takes about 2 minutes. The 438 optimiser steps take 44 minutes, at about 6 seconds per step. LoRA trains 29.9M parameters, 0.63% of the 4.73B parameters in the Thinker with its audio and vision encoders. The training loss falls from 3.2 at the first log to about 1.0 in the third epoch, with a mean of 1.30.
!llamafactory-cli train llama_factory_coach.yamlLoading the coach¶
Load the base model and the adapter from the training run. The base model is about 7 GB in bf16.
import torch
from transformers import Qwen2_5OmniForConditionalGeneration, Qwen2_5OmniProcessor
from qwen_omni_utils import process_mm_info
from peft import PeftModel
MODEL_ID = "Qwen/Qwen2.5-Omni-3B"
LORA_DIR = "saves/piano-coach-lora" # written by the training cell
processor = Qwen2_5OmniProcessor.from_pretrained(MODEL_ID)
model = Qwen2_5OmniForConditionalGeneration.from_pretrained(
MODEL_ID,
torch_dtype=torch.bfloat16,
device_map="auto",
attn_implementation="sdpa",
)
# The adapter targets the Thinker, not the top-level model.
model.thinker = PeftModel.from_pretrained(model.thinker, LORA_DIR)
model.thinker = model.thinker.merge_and_unload()
model.disable_talker() # no speech output; we want text feedback
model.eval()
print("ready")[transformers] Qwen2_5OmniForConditionalGeneration LOAD REPORT from: Qwen/Qwen2.5-Omni-3B
Key | Status | |
---------------------------------------------------+------------+--+-
token2wav.code2wav_dit_model.rotary_embed.inv_freq | UNEXPECTED | |
Notes:
- UNEXPECTED: can be ignored when loading from different task/architecture; not ok if you expect identical arch.
ready
SYSTEM = ("You are a piano performance coach. "
"Listen to the audio and provide helpful, specific feedback.")
def ask(audio_path, question, max_new_tokens=256):
conversation = [
{"role": "system", "content": [{"type": "text", "text": SYSTEM}]},
{"role": "user", "content": [
{"type": "audio", "audio": audio_path},
{"type": "text", "text": question},
]},
]
text_prompt = processor.apply_chat_template(
conversation, add_generation_prompt=True, tokenize=False
)
audios, images, videos = process_mm_info(conversation, use_audio_in_video=False)
inputs = processor(text=text_prompt, audio=audios, images=images, videos=videos,
return_tensors="pt", padding=True)
inputs = inputs.to(model.device).to(model.dtype)
input_len = inputs["input_ids"].shape[-1]
ids = model.generate(**inputs, max_new_tokens=max_new_tokens,
temperature=0.1, do_sample=True, top_p=0.9,
return_audio=False)
return processor.batch_decode(ids[:, input_len:],
skip_special_tokens=True)[0].strip()Ask the NeuroPiano questions of a held-out recording first, so the answers can be read against the teachers’ rows for the same clip. Then point AUDIO at your own recording.
r = evals[0]
AUDIO = f"data/neuropiano/audio/{r['fname']}"
with open(AUDIO, "wb") as f:
f.write(r["audio_path"]["bytes"])
QUESTIONS = [
"What's good about this performance?",
"What's bad about this performance?",
"How solid is the sound?",
"Is the tempo kept constant?",
"Are the left and right hands balanced?",
"Is the dynamics change natural?",
]
for q in QUESTIONS:
print(f"Q: {q}")
print(f"A: {ask(AUDIO, q)}")
teacher = [x for x in evals if x["fname"] == r["fname"] and x["q_eng"].strip() == q]
for t in teacher:
print(f" teacher ({t['score']}/6): {t['a_eng'].strip()}")
print()Q: What's good about this performance?
A: The sound is clear and resonant.
teacher (3/6): The note values and tempo are uniform and easy to listen to.
Q: What's bad about this performance?
A: The sound of the thumb is heavy.
teacher (3/6): The timing of the first four notes was inconsistent.
Q: How solid is the sound?
A: The sound has a solid core.
teacher (3/6): The thumb and 5th finger seem to have a solid core, but the 4th finger is weaker.
Q: Is the tempo kept constant?
A: Rating: 2/6. The tempo is slower than indicated.
teacher (6/6): The tempo was consistent and easy to listen to.
Q: Are the left and right hands balanced?
A: The left hand is a bit too loud.
Q: Is the dynamics change natural?
A: Rating: 2/6. The volume of the first note is too loud.
teacher (5/6): It was played at the specified pianissimo volume.
Where it breaks¶
Read the six answers together rather than one at a time.
It will not say “I don’t know.” Ask about dynamics on a clip that is pp throughout and the model rates the dynamic change anyway. The annotation contains no abstentions: a teacher asked to rate a dimension rates it.
The rating and the prose can disagree. A 3/6 next to a sentence with no fault named is common. The number and the sentence are the same next-token stream with no constraint tying them together, and the teachers’ own rows show the same looseness.
Localisation is a guess. The model heard 5 seconds of audio and consulted no score. When the answer names the 4th finger or the top of the scale, it is reproducing the training distribution of physical and location spans, not measuring anything. Claims about where in the excerpt something happened need an aligner (Chang et al., 2025), and this model has none.
Evaluate against the teacher rows, never against the fluency of the text.