Building a Simple Teaching Agent

Building a simple teaching agent¶

The previous three sections described the ingredients: MIR tools, learner memory, and pedagogy as the policy. This notebook assembles a minimal teaching agent (TA) from those ingredients: a piano transcription model, a score alignment, three learners, and an agent loop in about fifty lines.

The setting is a small practice studio. You log in as one of three learners, hand the TA that learner’s newest recording, and watch it transcribe the audio, compare it with the score, give feedback at the learner’s level, and write back what it learned for next week.

The learners and their recordings come from the Expert-Novice dataset (Jiang, 2023): 83 recordings of seven simple piano arrangements by 21 amateur pianists. Four piano instructors rated each recording from 1 to 5 and wrote a short comment on it.

Learner profiles come from the dataset. A profile holds the player’s self-identified level and the instructor comments on their other recordings, which the TA reads as notes from earlier sessions.
This week’s comments are held back. The instructor comments on the recording the TA analyses stay out of its context, so you can compare them with what the TA says.

What a domain-specific agent is¶

Strip away framework vocabulary and a tool-using agent is four artefacts (Chu et al., 2025):

A model, reached through an API.
A system prompt that encodes the domain policy, here the pedagogy of Pedagogy as the agent’s policy, plus the context the model must always have.
Tools: JSON schemas describing functions the model may request, plus your implementations of those functions.
A loop that sends the conversation, executes whatever tools the model requested, appends the results, and repeats until the model stops requesting.

Everything a framework advertises on top (planning, memory, orchestration) is built from these four.

We use the Gemini models that Colab offers free through google.colab.ai, so the notebook needs no API key. That library takes one text prompt and returns text, with no tool parameter, so the loop below does the tool calling itself: the prompt lists the tools, the model replies with a JSON object, and our code runs the tool it names. Provider APIs do the same parsing on their side under the name function calling.

%pip install -q transkun partitura parangonar remotezip "setuptools<81"

transkun runs on the GPU when one is available. On Colab, choose a GPU under Runtime, Change runtime type. The agent uses Colab’s built-in Gemini models, which are free within monthly limits and need no API key, so the agent cells run only in Colab.

The data¶

The Expert-Novice dataset is on Zenodo under CC BY-NC-SA 4.0: the recordings, the seven scores in MusicXML, each player’s self-identified level, and the ratings and comments. Its augmented version on GitHub adds, for 34 recordings, a MIDI track that marks the regions where the playing goes wrong, annotated from the raters’ comments (Morsi et al., 2024). The cell below clones the GitHub repository and fetches single recordings from the Zenodo archive without downloading all 590 MB of it.

import pathlib, shutil, subprocess
from remotezip import RemoteZip

EN = pathlib.Path("EN-augmented-data")
if not EN.exists():
    subprocess.run(["git", "clone", "-q", "--depth", "1",
                    "https://github.com/anusfoil/EN-augmented-data", str(EN)], check=True)

ZENODO = ("https://zenodo.org/records/8392772/files/"
          "Recordings%20and%20Alignments.zip?download=1")

def piece_key(piece):  # "Over the Waves" -> "OverTheWaves", the dataset's folder and score names
    return "".join(w[0].upper() + w[1:] for w in piece.replace("'", "").split())

def fetch_recording(piece, number, dest):
    member = f"Recordings and Alignments/{piece_key(piece)}/{piece}-{number:02d}.wav"
    with RemoteZip(ZENODO) as archive, archive.open(member) as src, open(dest, "wb") as out:
        shutil.copyfileobj(src, out)

The studio database¶

A first instinct for “give the model context” is retrieval-augmented generation: embed everything, retrieve by similarity. For learner state, that is not the best approach. A TA’s context is small, structured, and keyed by identity: one profile, one memory list, a handful of recordings, all belonging to the logged-in user. The natural access pattern is a lookup keyed by the login, not a nearest-neighbour search over documents.

Each learner file holds two of the four kinds of state that Memory for longitudinal coaching argued a TA needs: identity with level, and notes from earlier sessions. The dataset records no style preferences and no constraints from a teacher, so these profiles have none. The notes are the four instructors’ ratings and comments on each of the player’s other recordings. The dataset does not record the order of a player’s recordings, so we treat the other pieces as earlier sessions.

We picked three players at different levels with different problems:

Player 13, intermediate. The instructors found wrong rhythms in all four earlier pieces and a heavy left hand in two of them. This week: The Entertainer.
Player 19, advanced. The instructors call the earlier pieces accurate and comment on pedalling, left-hand articulation, legato, and the final ritardando. This week: The Entertainer.
Player 3, beginner. Played the right hand an octave too high in The Blues. This week: Over the Waves.

The learner profile¶

A profile is one JSON file per learner, studio_db/learners/<learner>.json. login puts the whole file into the system prompt, and end_session appends to its memory list. The recordings and scores sit in the same database but outside the profile: the TA reaches them only through the tools.

Field	What it holds	Source	Player 13
`name`	display name	player number in the dataset	`"Player 13"`
`instrument`	instrument	fixed, the dataset is piano only	`"piano"`
`level`	the level the player reported	`self-identified_skill_levels.txt`	`"Intermediate (self-identified)"`
`memory`	notes from earlier sessions, oldest first	see below	4 entries

A memory entry has one of two shapes, depending on who wrote it:

Written by	Fields	Example
the dataset, before the first session	`piece`, `from: "four piano instructors"`, `ratings_1_to_5`, `comments`	Lavender’s Blue, ratings `[3, 3, 1, 1]`, “LH too loud. Eighth note rhythms not correct. Pitches OK”
the TA, at `end_session`	`date`, `from: "TA"`, `note`	one consolidated note per session

The instructor comments on this week’s take do not go into the profile. They stay in HELD_OUT for the comparison after each session.

import json
import pandas as pd

DB = pathlib.Path("studio_db")
if DB.exists():
    shutil.rmtree(DB)
for sub in ["learners", "recordings", "scores"]:
    (DB / sub).mkdir(parents=True)

evals = pd.read_csv(EN / "evaluation_data_anonymous.csv")
levels = pd.read_csv(EN / "self-identified_skill_levels.txt", sep="\t",
                     index_col="Player_id")["Level"]

THIS_WEEK = {13: "The Entertainer", 19: "The Entertainer", 3: "Over the Waves"}

def instructors(row):
    return {"ratings_1_to_5": [int(row[f"Instructor{i}_rating"]) for i in range(1, 5)],
            "comments": [row[f"Instructor{i}_text"].strip() for i in range(1, 5)]}

HELD_OUT = {}
for player, piece in THIS_WEEK.items():
    uid = f"p{player:02d}"
    takes = evals[evals.Player_id == player]
    this = takes[takes.Piece_name == piece].iloc[0]
    HELD_OUT[uid] = {"piece": piece, "recording_number": int(this.Recording_number),
                     **instructors(this)}
    profile = {
        "name": f"Player {player}", "instrument": "piano",
        "level": f"{levels[player]} (self-identified)",
        "memory": [{"piece": row.Piece_name, "from": "four piano instructors", **instructors(row)}
                   for _, row in takes.iterrows() if row.Piece_name != piece],
    }
    (DB / "learners" / f"{uid}.json").write_text(
        json.dumps(profile, indent=2, ensure_ascii=False))
    fetch_recording(piece, int(this.Recording_number),
                    DB / "recordings" / f"{uid}_{piece_key(piece)}.wav")
    shutil.copy(EN / "Scores" / f"{piece_key(piece)}.mxl", DB / "scores")

profile = json.loads((DB / "learners" / "p13.json").read_text())
print(json.dumps({k: v for k, v in profile.items() if k != "memory"}, indent=1))
print(f"memory: {len(profile['memory'])} entries, the second one:")
print(json.dumps(profile["memory"][1], indent=1, ensure_ascii=False))

{
 "name": "Player 13",
 "instrument": "piano",
 "level": "Intermediate (self-identified)"
}
memory: 4 entries, the second one:
{
 "piece": "Lavender's Blue",
 "from": "four piano instructors",
 "ratings_1_to_5": [
  3,
  3,
  1,
  1
 ],
 "comments": [
  "LH too loud. Eighth note rhythms not correct. Pitches OK",
  "Need to keep rhythm steady. Work on playing eighth notes evenly. Nice solid chords.",
  "wrong rhythms in right hand.  Left hand too loud and heavy.",
  "Wrong rhythms 8th note measures. LH too loud."
 ]
}

The recordings¶

Each learner has one recording on disk, this week’s take, named <learner>_<piece>. The first 20 seconds of Player 13’s take:

import soundfile as sf
from scipy.signal import resample_poly
from IPython.display import Audio, display

audio, sr = sf.read(DB / "recordings" / "p13_TheEntertainer.wav")
display(Audio(resample_poly(audio[:20 * sr], 16000, sr), rate=16000))

Tools: transcription and score alignment¶

A tool has two halves. The schema is what the model sees: a name, a natural-language description, and typed parameters. The function is what runs when the model requests that name. Descriptions are prompt engineering: the model decides when to call each tool purely from that text.

Two of the three tools below do MIR work. transcribe_recording runs transkun (Yan and Duan, 2024), a piano transcription model that also estimates how hard each key was struck (MIDI velocity) and when the sustain pedal was down. The instructors comment on the balance between the hands and on pedalling, and a pitch-only transcription cannot see either. check_against_score aligns the transcribed notes to the score with the note matcher of parangonar (Peter et al., 2023) and reports per bar: wrong, missed and extra notes, how long the bar took compared with the median bar, the loudness of each hand, and the pedal.

make_tools(user) binds the three functions to the logged-in learner.

import subprocess, sys, warnings

import numpy as np
import partitura as pt
import parangonar as pa
import torch

warnings.filterwarnings("ignore", module="partitura")

DEVICE = "cuda" if torch.cuda.is_available() else "cpu"
NAMES = ["C", "C#", "D", "Eb", "E", "F", "F#", "G", "Ab", "A", "Bb", "B"]
BEAT = {2: "half note", 4: "quarter note", 8: "eighth note"}


def note_name(midi):
    return f"{NAMES[midi % 12]}{midi // 12 - 1}"


def transcribe(recording_id):
    """Run transkun once per recording and cache the MIDI next to the wav."""
    wav = DB / "recordings" / f"{recording_id}.wav"
    mid = wav.with_suffix(".mid")
    if not mid.exists():
        subprocess.run([sys.executable, "-m", "transkun.transcribe", str(wav), str(mid),
                        "--device", DEVICE],
                       check=True, capture_output=True)
    return pt.load_performance_midi(str(mid))


def compare(recording_id):
    """Align the transcription to the score and summarise each bar."""
    perf = transcribe(recording_id)
    played = perf.note_array()
    piece = recording_id.split("_", 1)[1]
    part = pt.load_score(str(DB / "scores" / f"{piece}.mxl")).parts[0]
    written = part.note_array(include_staff=True)

    alignment = pa.AutomaticNoteMatcher()(written, played)
    by_sid = {n["id"]: n for n in written}
    by_pid = {n["id"]: n for n in played}
    matched = [(by_sid[a["score_id"]], by_pid[a["performance_id"]])
               for a in alignment if a["label"] == "match"]
    missed = [by_sid[a["score_id"]] for a in alignment if a["label"] == "deletion"]
    extra = [by_pid[a["performance_id"]] for a in alignment if a["label"] == "insertion"]

    # score beat <-> seconds, from the matched notes
    beats = np.unique([s["onset_beat"] for s, _ in matched])
    secs = np.maximum.accumulate(
        [np.median([p["onset_sec"] for s, p in matched if s["onset_beat"] == b])
         for b in beats])
    spb = float(np.median(np.diff(secs) / np.diff(beats)))  # seconds per beat
    def beat_at(t):  # before the first or after the last match, continue at the median tempo
        if t < secs[0]:
            return beats[0] - (secs[0] - t) / spb
        if t > secs[-1]:
            return beats[-1] + (t - secs[-1]) / spb
        return float(np.interp(t, secs, beats))
    def sec_at(b):
        return float(np.interp(b, beats, secs))

    pedal_down = [(c["time"], c["value"] >= 64) for c in perf.performedparts[0].controls
                  if c["number"] == 64]

    def pedal_fraction(t0, t1):
        if t1 <= t0:
            return None
        grid = np.linspace(t0, t1, 50)
        state = [next((down for t, down in reversed(pedal_down) if t <= g), False)
                 for g in grid]
        return round(float(np.mean(state)), 2)

    bars = []
    for m in part.iter_all(pt.score.Measure):
        b0, b1 = part.beat_map(m.start.t), part.beat_map(m.end.t)
        in_bar = lambda b: b0 <= b < b1
        mt = [(s, p) for s, p in matched if in_bar(s["onset_beat"])]
        ms = sorted(int(s["pitch"]) for s in missed if in_bar(s["onset_beat"]))
        ex = sorted(int(p["pitch"]) for p in extra if in_bar(beat_at(p["onset_sec"])))
        wrong = []
        for w in list(ms):  # a missed and an extra note in one bar read as a wrong note
            near = [x for x in ex if abs(x - w) <= 12]
            if near:
                x = min(near, key=lambda x: (abs(x - w) % 12 > 0, abs(x - w)))
                ms.remove(w); ex.remove(x)
                wrong.append({"written": note_name(w), "played": note_name(x)})
        t0, t1 = sec_at(b0), sec_at(b1)
        rh = [p["velocity"] for s, p in mt if s["staff"] == 1]
        lh = [p["velocity"] for s, p in mt if s["staff"] == 2]
        bars.append({
            "bar": m.number,
            "notes_correct": f"{len(mt)}/{len(mt) + len(ms) + len(wrong)}",
            "wrong_notes": wrong,
            "missed_notes": [note_name(p) for p in ms],
            "extra_notes": [note_name(p) for p in ex],
            "start_s": round(t0, 2) if mt else None,
            "seconds": round(t1 - t0, 2) if mt and t1 > t0 else None,
            "rh_velocity": round(float(np.mean(rh))) if rh else None,
            "lh_velocity": round(float(np.mean(lh))) if lh else None,
            "pedal_down": pedal_fraction(t0, t1) if mt else None,
        })

    durations = [b["seconds"] for b in bars if b["seconds"]]
    median_bar = float(np.median(durations))
    for b in bars:
        b["length_vs_median_bar"] = round(b["seconds"] / median_bar, 2) if b["seconds"] else None
    ts = next(part.iter_all(pt.score.TimeSignature))
    opening = next(part.iter_all(pt.score.Measure))
    marking = next((d.text for d in part.iter_all(pt.score.TempoDirection, include_subclasses=True)
                    if d.start.t < opening.end.t), None)
    return {
        "piece": piece,
        "time_signature": f"{ts.beats}/{ts.beat_type}",
        "tempo_marking": marking,
        "played_beats_per_minute": round(ts.beats * 60 / median_bar),
        "beat": BEAT.get(ts.beat_type),
        "notes_correct": f"{len(matched)}/{len(written)}",
        "bars": bars,
    }


def make_tools(user):
    """Bind the three tools to one logged-in learner. The model never sees `user`."""
    def own(recording_id: str) -> bool:
        return recording_id.startswith(f"{user}_") and \
            (DB / "recordings" / f"{recording_id}.wav").exists()

    def list_recordings() -> str:
        return json.dumps([p.stem for p in sorted((DB / "recordings").glob(f"{user}_*.wav"))])

    def transcribe_recording(recording_id: str) -> str:
        if not own(recording_id):
            return json.dumps({"error": "no such recording for this learner"})
        notes = transcribe(recording_id).note_array()
        return json.dumps([{"onset_s": round(float(n["onset_sec"]), 3),
                            "duration_s": round(float(n["duration_sec"]), 3),
                            "pitch": note_name(int(n["pitch"])),
                            "velocity": int(n["velocity"])} for n in notes])

    def check_against_score(recording_id: str) -> str:
        if not own(recording_id):
            return json.dumps({"error": "no such recording for this learner"})
        return json.dumps(compare(recording_id))

    return {f.__name__: f for f in [list_recordings, transcribe_recording, check_against_score]}


TOOLS = [
    {
        "name": "list_recordings",
        "description": "List the logged-in learner's uploaded recordings by id. "
                       "An id is <learner>_<piece>.",
        "input_schema": {"type": "object", "properties": {}, "required": []},
    },
    {
        "name": "transcribe_recording",
        "description": "Run piano transcription (transkun) on one recording and return raw "
                       "note events: onset and duration in seconds, pitch name, and MIDI "
                       "velocity (loudness, 1 to 127). Use this to inspect raw evidence. "
                       "Prefer check_against_score for graded feedback.",
        "input_schema": {"type": "object",
                         "properties": {"recording_id": {"type": "string"}},
                         "required": ["recording_id"]},
    },
    {
        "name": "check_against_score",
        "description": "Align one recording to the score of its piece and return a per-bar "
                       "report: correct notes, wrong notes (written vs played), missed and "
                       "extra notes, bar start time and length in seconds, bar length "
                       "relative to the median bar (well above 1 means a hesitation or "
                       "added beats), mean velocity of the right hand (staff 1) and left "
                       "hand (staff 2), and the fraction of the bar with the sustain pedal "
                       "down. Also returns the written tempo marking and the played tempo in "
                       "beats per minute. The transcription and "
                       "the alignment both make errors, so single-note anomalies may be "
                       "artefacts.",
        "input_schema": {"type": "object",
                         "properties": {"recording_id": {"type": "string"}},
                         "required": ["recording_id"]},
    },
]

Before wiring these into an agent, look at what they actually return. This is the entirety of what the model will know about the playing:

report = json.loads(make_tools("p13")["check_against_score"]("p13_TheEntertainer"))
print({k: v for k, v in report.items() if k != "bars"})
pd.DataFrame(report["bars"]).set_index("bar")

{'piece': 'TheEntertainer', 'time_signature': '2/2', 'tempo_marking': 'moderato', 'played_beats_per_minute': 94, 'beat': 'half note', 'notes_correct': '162/169'}

The augmented dataset marks where this take goes wrong. The annotation is a copy of the transcription MIDI with one extra track: each mistake region is a single note of pitch 0 that lasts as long as the region. The annotator’s notes (mistake_notes.pdf in the repository) list this take under two rhythm mistakes: drag, where a hesitation lengthens a note and shifts everything after it, and misread, where notes are played at twice or half their written length. The cell below draws the regions over the transcription and colours the bar numbers that the report flags:

import mido
import matplotlib.pyplot as plt

def annotated_regions(piece, number):
    """Read the annotation track: each mistake region is one note of pitch 0."""
    name = f"{piece_key(piece)}-{number:02d}-annot.mid".lower()
    path = next(p for p in (EN / "Recordings_and_Alignments" / piece_key(piece)).iterdir()
                if p.name.lower() == name)
    t, start, regions = 0.0, None, []
    for msg in mido.MidiFile(path):
        t += msg.time
        if msg.type == "note_on" and msg.note == 0 and msg.velocity > 0:
            start = t
        elif msg.type in ("note_on", "note_off") and msg.note == 0 and start is not None:
            regions.append((start, t))
            start = None
    return regions

played = transcribe("p13_TheEntertainer").note_array()
regions = annotated_regions("The Entertainer", HELD_OUT["p13"]["recording_number"])
flagged = {b["bar"] for b in report["bars"]
           if b["wrong_notes"] or b["missed_notes"] or b["extra_notes"]
           or (b["length_vs_median_bar"] or 0) >= 1.3}

fig, ax = plt.subplots(figsize=(14, 4.5))
for s, e in regions:
    ax.axvspan(s, e, color="tab:red", alpha=0.2, lw=0)
ax.barh(played["pitch"], played["duration_sec"], left=played["onset_sec"],
        height=0.8, color="0.25")
top = played["pitch"].max() + 3
for b in report["bars"]:
    if b["start_s"] is None:
        continue
    ax.axvline(b["start_s"], color="0.8", lw=0.6, zorder=0)
    hit = b["bar"] in flagged
    ax.text(b["start_s"], top, str(b["bar"]), fontsize=8, ha="left",
            color="tab:orange" if hit else "0.5", weight="bold" if hit else "normal")
ax.set(xlabel="time (s)", ylabel="MIDI pitch", ylim=(played["pitch"].min() - 2, top + 3),
       xlim=(0, played["onset_sec"].max() + 1))
ax.set_title("Player 13, The Entertainer: transcribed notes, annotated mistake regions (red), "
             "bar numbers the report flags (orange)", loc="left", fontsize=10)
plt.show()

Nine of the ten annotated regions contain a bar the report flags. In eight of them the flag is a bar that took 1.3 to 1.9 times as long as the median bar: the hesitations and added beats the instructors complain about. The report also flags bars 1 and 2, where Player 13 played the opening an octave lower than written. The instructors noticed this, but the annotation does not mark it. Bars 8, 12 and 31 run long without an annotation, and the region in bars 29 to 30 has no flag. Real MIR evidence arrives noisy, which is why the pedagogy below tells the model to weight consistent patterns over isolated anomalies, the evidence-with-uncertainty point of Tool calling for a practice agent.

Structure, not tools: profile and memory¶

The profile and memory notes are injected into the system prompt when the learner logs in. That is the context engineering: the model starts every turn already knowing who it teaches.

The harness owns the memory: end_session makes one extra model call asking for a consolidated note, then appends it to the learner file itself. This is the assess, update state, act loop of ScaffoldLM (Li et al., 2026) with the update step guaranteed by code rather than requested by prompt.

PEDAGOGY = """\
You are a between-lessons teaching agent (TA) for one instrumental learner. You support \
their human teacher; you do not replace them.

Rules, in priority order:

1. Match the learner's level, given in the profile below.
2. Tool output is evidence and may contain artifacts. Transcription misses and invents notes, and the \
score alignment can misplace notes around a mistake; weight consistent patterns over \
isolated one-note anomalies. If the report looks unreliable overall, say so and ask for \
another take.
3. Give formative feedback: tied to specific bars, about the playing rather than the \
person. Lead with the single most important issue and mention at most one more. Never \
recite the full report.
4. Assign one concrete drill for the coming week, sized just beyond what the memory notes \
say the learner already manages. If the same issue recurs across sessions, change strategy \
instead of repeating the old advice.
"""

SYSTEM_PROMPT, TOOL_FUNCTIONS, current_user = None, None, None

def login(user_id: str) -> None:
    global SYSTEM_PROMPT, TOOL_FUNCTIONS, current_user
    current_user = user_id
    profile = json.loads((DB / "learners" / f"{user_id}.json").read_text())
    SYSTEM_PROMPT = (PEDAGOGY
        + "\nThe learner and your memory of past sessions:\n\n"
        + json.dumps(profile, indent=2, ensure_ascii=False))
    TOOL_FUNCTIONS = make_tools(user_id)
    print(f"Logged in as {profile['name']} ({profile['level']})")

The loop and the session boundary¶

This is the entire mechanism. generate_text is stateless and takes one string, so call_model writes the system prompt, the whole conversation, and the tool protocol into that string on every call. When the reply names a tool, run_turn runs it, appends the result, and calls the model again. When the reply is an answer, the turn is over. A reply that is not valid JSON, or that names a tool that does not exist, goes back to the model as an error message. end_session closes the loop from the memory side.

import json
from datetime import date
from google.colab import ai
from IPython.display import Markdown, display

PROTOCOL = """Reply with exactly one JSON object and nothing else:
{"tool": "<tool name>", "input": {<arguments>}} to call one of the tools below, or
{"answer": "<your reply to the learner, in Markdown>"} when you are ready to reply.

Tools:
""" + json.dumps(TOOLS, indent=1)

def call_model(messages: list, instruction: str = PROTOCOL) -> str:
    conversation = "\n\n".join(f"[{m['role']}]\n{m['content']}" for m in messages)
    prompt = f"{SYSTEM_PROMPT}\n\nConversation so far:\n\n{conversation}\n\n{instruction}"
    return ai.generate_text(prompt)  # Colab's default Gemini model; ai.list_models() shows others

def parse(reply: str):
    try:
        action = json.loads(reply[reply.find("{"):reply.rfind("}") + 1], strict=False)
    except json.JSONDecodeError:
        return None
    return action if isinstance(action, dict) else None

def run_turn(messages: list, max_steps: int = 8) -> str:
    for _ in range(max_steps):
        reply = call_model(messages)
        messages.append({"role": "assistant", "content": reply})
        action = parse(reply)
        if action is None:
            messages.append({"role": "harness", "content": "Not valid JSON. Reply with one JSON object."})
        elif "answer" in action:
            return action["answer"]
        elif action.get("tool") in TOOL_FUNCTIONS:
            print(f"  tool call: {action['tool']}({json.dumps(action.get('input', {}))})")
            try:
                output = TOOL_FUNCTIONS[action["tool"]](**action.get("input", {}))
            except TypeError as err:
                output = json.dumps({"error": str(err)})
            messages.append({"role": "tool", "content": f"{action['tool']} returned:\n{output}"})
        else:
            messages.append({"role": "harness", "content": f"There is no tool {action.get('tool')!r}."})
    return "The TA stopped after too many steps."

def ask_ta(user_text: str, messages: list) -> None:
    messages.append({"role": "learner", "content": user_text})
    display(Markdown(run_turn(messages)))

def end_session(messages: list) -> None:
    note = call_model(messages, "The session is over. Write one consolidated memory note for "
                      "the next session: what you observed, what you assigned, anything worth "
                      "carrying forward. Reply with the note text only.")
    path = DB / "learners" / f"{current_user}.json"
    profile = json.loads(path.read_text())
    profile["memory"].append({"date": date.today().isoformat(), "from": "TA", "note": note})
    path.write_text(json.dumps(profile, indent=2, ensure_ascii=False))
    print(f"memory saved for {current_user}: {note}")

Session one: Player 13¶

Player 13 rates their own level as intermediate, and the instructors found wrong rhythms in every one of their earlier pieces. Nothing in the message below tells the agent any of that; the profile arrived through login, the evidence arrives through the tools.

login("p13")
p13_session = []
ask_ta("I recorded The Entertainer this week, can you check how it went?", p13_session)

Watch the printed tool calls: the agent chose them itself, nothing in our loop scripts the order. A good reply leads with the hesitations, the long bars that recur through the piece, instead of reciting forty bars, and links them to the rhythm problems in the memory notes. Because the same problem appears in every earlier piece, it should change tactics instead of repeating “count carefully”. Model outputs vary run to run, so your transcript will read differently.

Compare the reply with what the four instructors wrote about this take, which the TA never saw:

def show_instructors(uid):
    held = HELD_OUT[uid]
    print(f"{held['piece']}, ratings {held['ratings_1_to_5']}")
    for comment in held["comments"]:
        print(" -", comment)

show_instructors("p13")

The Entertainer, ratings [2, 2, 1, 1]
 - Octave error m. 1-2. Meter errors throughout in LH, adding extra beats in both RH and LH. Pitches are correct but almost always at the wrong time
 - Notes are accurate. But rhythm needs reworking. Counting is incorrect.
 - Didn't begin in the proper octave.  Too many rhythmic inaccuracies and hesitations.  Poor.  Wrong notes.
 - No 8va. Rhythm wrong end line 2. Then lots of hesitations and rhythmic inaccuracies throughout.

Follow-ups work without re-analysis, since the conversation is stateful within a session:

ask_ta("It always falls apart when I try to keep going. What should I do this week?", p13_session)

Closing the session writes the note the next session will read:

end_session(p13_session)

for entry in json.loads((DB / "learners" / "p13.json").read_text())["memory"]:
    text = entry.get("note") or " / ".join(entry["comments"])
    print(f"{entry.get('piece') or entry['date']} ({entry['from']}): {text}")

The last entry was written by the model, but its persistence was guaranteed by the harness. Next week’s login folds it into the system prompt, which is the whole difference between a reactive session bot and a longitudinal teaching agent in the sense of Memory for longitudinal coaching. Note the compression choice: one curated note per session rather than the raw transcript, the inspectable-memory pattern from DeepTutor (Zhao et al., 2026).

Same architecture, different learner¶

Player 19 rates their own level as advanced. The report for their take of the same piece looks different: bars of even length, and a handful of single missed or extra notes, mostly in the left hand. They play about 96 half notes per minute. The score marks only Moderato, so the report cannot say whether that is too slow. The instructors’ comments on their earlier pieces are about pedalling, articulation and the ending, not about notes. The code path is identical; only the injected profile and the audio changed.

login("p19")
p19_session = []
ask_ta("Uploaded my take of The Entertainer. Be specific.", p19_session)
end_session(p19_session)

show_instructors("p19")

The Entertainer, ratings [4, 4, 2, 2]
 - Very close, errors in m.21-22  and also the last line
 - Though there were missed notes, good job at keeping the tempo going. Need to fix a few wrong notes. Some notes were missing from left hand as well.
 - A bit lumpy and heavy in approach, not feel the arc of the phrasing.  Wrong notes here and there.
 - Too slow ( not cut time). Wrong note line 4. No P page 2. Wrong notes p. 2 line 2 and 3 and 4.

The register should shift: one rhythm problem and a concrete drill for Player 13, detail on tempo and the left-hand slips for Player 19, from the same system prompt scaffold and the same loop. Personalisation lives in the data, not in the code, which is why deployed practice products spend their engineering on the profile and memory schema rather than on prompt variants per user.

Exercises¶

Player 3’s octave. Player 3 plays the right hand of Over the Waves an octave too high, the same mistake the instructors noted in The Blues. The report shows it as wrong notes exactly twelve semitones apart in two thirds of the bars. Log in as p03, ask for feedback, and check whether the TA names the octave error instead of listing thirty bars of wrong notes, and whether it connects it to the memory note. Then compare with show_instructors("p03").
A real human. Record yourself playing one of the seven pieces (scores in EN-augmented-data/Scores), save the wav as studio_db/recordings/yourname_TheBlues.wav, copy the score into studio_db/scores/, add a profile, and see how the pipeline copes with your room, your piano and your tempo.
Check the alignment. The dataset includes an audio-to-score alignment for every recording: one .txt file per recording with the audio time of each score position, in the Zenodo archive and in EN-augmented-data/Recordings_and_Alignments. Compare it with the bar start times in the report and find the bars where the automatic alignment goes wrong.
Grade the TA. Build a profile for every player who recorded one piece, run the TA on each recording, and compare its main point with the held-out instructor comments. This is a small version of the evaluation harness discussed below.
Forgetting. After five memory entries, have end_session consolidate them into one summary entry. Decide what survives, guided by Memory for longitudinal coaching.

What made it domain-specific¶

The agent code contains no music, but the agent is steeped in it: the model’s pretrained prior supplies the concepts and vocabulary, the system prompt injects the doctrine and the learner, and the tools return the evidence. Where the fit to the domain lives is more specific, and practitioner discussions of production agents converge on the same five places:

Data: the learner schema, the scores, and the recordings. The TA is exactly as good as the state you keep; most of the engineering in a real product goes here.
Tools: the three functions, which decide what the model can observe. The MIR stack of Tool calling for a practice agent enters the agent only through this surface.
Context engineering: what reaches the prompt each turn. The profile arrives via login, evidence arrives curated through tool returns rather than as raw audio frames, and end_session controls what gets written back.
Tone: the register shift between Player 13 and Player 19, driven by the level in the profile rather than separate prompts.
Guardrails: the login boundary enforced in code where the model cannot reach it, and abstention on unreliable evidence.

The same discussions rank a sixth element above fine-tuning: a verification loop against domain ground truth. The transcription and alignment stack plays that role here. The TA never trusts its own listening; it checks the take against note-level evidence, for the same reason a legal agent should verify a claim against the statute rather than recall it. Fine-tuning enters last, once knowledge access and verification are solved. The piece this notebook lacks is an evaluation harness that scores teaching quality across learners. The held-out instructor comments give such a harness a reference to score against; Pedagogy as the agent’s policy closed on the observation that longitudinal benchmarks for music practice barely exist.

How much can we trust the TA’s domain knowledge¶

Everything the TA said about rhythm, fingering, and practice strategy came from its pretrained prior, uncited and unverified. How far that prior extends is measurable. MusicTheoryBench, introduced with ChatMusician, tests college-level written theory in a knowledge split and a reasoning split (Yuan et al., 2024). GPT-4 answered a little under sixty percent of the knowledge split and stayed near chance on multi-step reasoning, far below trained musicians on both. And neither split measures pedagogical judgement, the capability this chapter actually needs: knowing the rules of counterpoint is not knowing how to fix Player 13’s hesitations in a ragtime.

The practical stance mirrors expert domains such as law and medicine: trust the prior for concepts and vocabulary, ground everything that could be looked up. In increasing order of cost:

Injected doctrine: rubrics, grade syllabi, and methodology playbooks placed in the system prompt or loaded on demand. The four rules above are this layer in miniature.
Retrieval over a pedagogy corpus: the canonical treatises are public domain and number in the dozens, so a TA can quote its sources with simple retrieval and no vector infrastructure. Clinical and legal agents live on this pattern at far larger scale, because a retrieved claim carries a citation the user can check and a recalled claim does not.
Fine-tuning and continued pretraining: reshaping the model itself on domain text, as ChatMusician does with theory texts and ABC notation (Yuan et al., 2024). It buys domain fluency and style more reliably than facts, and a tuned model still cannot cite its sources, so production agents reach for it only after the cheaper layers are exhausted.

Where a claim must be right, the TA should not be recalling it; it should be reading it.

Where frameworks fit¶

Nothing above needed a framework, and for understanding agents you should build the raw loop at least once: every trace you will ever debug is this message list. Provider APIs move part of run_turn to the server: you pass the tool schemas as a parameter and get structured tool calls back, so the JSON parsing and the error messages for broken replies disappear. Frameworks are the same loop plus the production concerns this notebook skipped: streaming, retries, rate-limit handling, tracing and evaluation, human approval gates, parallel tool execution, and prompt caching for the stable system prompt and schemas. Current options, roughly in order of distance from the raw loop: PydanticAI for typed tools, LangChain’s create_agent on the LangGraph runtime, and smolagents for agents that act by writing code. One tier further out sit visual orchestrators such as n8n and role-scripted multi-agent frameworks such as CrewAI: prototyping surfaces and product glue rather than a different mechanism, since everything they run is still this loop. The Model Context Protocol standardises the tool boundary itself, so a transcription tool exposed once as an MCP server becomes callable from any of these and from end-user products.

Building a Simple Teaching Agent