Perceiving Audio Performances

Perceiving audio performances¶

Audio performance analysis relies on a stack of MIR tasks applied to the waveform: automatic music transcription (AMT) for pitch and onset recovery Benetos et al. (2019), F0 estimation for continuous pitch trajectories, beat and tempo tracking, loudness envelope analysis, vibrato and timbre characterization, and source separation in ensemble contexts. Lerch et al. Lerch et al. (2019) survey the full landscape of performance analysis methods, from signal-level feature extraction through aesthetic judgment; their framework organizes the field into technical correctness, expressive deviation, and quality assessment.

Piano has the richest infrastructure. ASAP Foscarin et al. (2020) and ATEPP Zhang et al. (2022) provide score-aligned and transcribed corpora enabling studies of rubato, dynamics, and expressive deviation. Specialized systems address discrete assessment dimensions: PianoJudge Zhang et al. (2024) benchmarks performance understanding across multiple datasets using audio encoders; pedal technique detection Liang et al. (2019) identifies sustain use from audio; PISA Parmar et al. (2021) frames skill-level estimation as a graded task; CROCUS Matsubara et al. (2021) provides expert critique documents grounding automated assessment in professional language.

String instruments add bowing direction, bow pressure, contact point, vibrato, and unfretted intonation as parameters without discrete keyboard equivalents. URMP Li et al. (2019) provides multi-track ensemble recordings across eleven instruments with per-instrument audio stems and F0 annotations. Luo et al. Luo et al. (2015) detect pitch intonation and bowing errors in novice violin recordings using HMM-based models. The Violin Etudes dataset Tamer et al. (2022) provides F0-annotated recordings of standard etude repertoire for technique-aware pitch analysis. Giraldo et al. Giraldo et al. (2019) assess tone quality automatically from spectral features, targeting the timbral dimensions that adjudicators attend to. Tamer Tamer (2024) addresses the label scarcity problem through weak supervision, classifying playing techniques from partial annotations.

Guitar carries left-hand technique (hammer-on, pull-off, slide, vibrato) and right-hand technique (pick vs. finger, contact angle, palm muting) as timbral variables. GuitarSet Xi et al. (2018) provides six-string pitch annotations, plucking style labels, and beat positions across multiple playing styles. GAPS Riley et al. (2024) aligns classical guitar recordings by more than 200 performers to their scores at the note level. Its transcription model, trained on these pairs, sets the state of the art on GuitarSet. Chen et al. Chen et al. (2015) detect electric guitar playing techniques from F0 sequence patterns in real-world recordings, covering bends, vibrato, and slides. Stefani and Turchet Stefani & Turchet (2025) embed real-time technique recognition in a smart acoustic guitar, demonstrating on-device deployment for five main right-hand techniques. Timbral variation from contact angle and nail shape remain largely uncharacterized in the dataset literature.

Singing spans pitch accuracy, vibrato, breath management, tonal quality, and expressive dynamics. Gupta et al. Gupta et al. (2020) rank a pool of singers with no reference recording and no score. Pitch-histogram measures give each singer an absolute intonation and rhythm score, and statistics across singers give a relative one. A twin network then learns the same ranking by choosing the better of two recordings Gupta et al. (2020). Zhang et al. Zhang et al. (2021) train a triplet model that maps singing rated pleasant closer to the reference track than other takes, so the model learns what good singing has in common without hand-designed features. TG-Critic Sun et al. (2023) scores singing quality without a reference recording, using timbre features from the CQT spectrogram. Narang et al. Narang et al. (2024) address loudness and expressive shaping as learnable targets from audio. Singing mistake detection Kumar et al. (2026) identifies pitch and rhythm errors by comparing learner and teacher recordings. Audio-language models extend this to generated critique text, though the published work targets synthesised singing rather than learners Li et al. (2025).

Wind and band instruments carry breath timing, embouchure control, and air speed as expressive variables absent from piano or guitar. The longest line of work is the Florida Bandmasters Association (FBA) series from Lerch’s group. Its data are recordings of Florida all-state band auditions, each rated by one judge on musicality, note accuracy, rhythmic accuracy, and tone quality Wu et al. (2016). The models moved from hand-designed descriptors to deep networks on the spectrogram Pati et al. (2018), then to networks that also read the audition score Huang et al. (2020), then to contrastive training of the embedding Seshadri & Lerch (2021). The target never changed: the judge’s number per criterion. The models do not say where the wrong notes fall or what the embouchure did, so the student gets the rating without the judge’s comment. Trumpet tone quality assessment Knight et al. (2011) targets timbre and stability from audio alone. Flute beginner mistake detection Han & Lee (2014) identifies pitch and rhythm errors in novice recordings using a hierarchical classifier.

Non-Western instruments bring playing techniques, tunings, and styles that models trained on Western data do not cover. Guqin performance analysis Huang et al. (2020) classifies six playing techniques on the GQ39 dataset using modal and structural context. Guzheng technique detection Li et al. (2023) applies multi-scale networks with self-attention to multi-label frame-level classification on the GuzhengTech99 dataset. Tabla stroke classification Rohit et al. (2021) addresses timing deviation and loudness dynamics in Hindustani vocal accompaniment. These works confirm that performance assessment must be built instrument-by-instrument: feature sets, annotation categories, and evaluation criteria do not transfer from one instrument tradition to another.

Coverage is uneven by instrument:

Assessment is conditional on instrument, style, period, and developmental stage. Models that inherit this conditionality from annotated expert data generalize; models that do not conflate skill levels and style contexts.