Perceiving Motion Performances

Perceiving motion performances¶

Sound is not the only observable output of a performance. The performer’s body generates a parallel stream of visual and proprioceptive information: bow-arm trajectory, key-depression velocity, breath timing, left-hand vibrato motion, postural weight shifts. For instruments where body-to-sound coupling is direct, motion data reveals technique aspects that are invisible in audio alone. A pianist’s arm weight determines the onset envelope of a note in ways MIDI velocity cannot capture; a cellist’s bow contact point is the single most decisive variable for tone quality and is entirely absent from audio features.

Emerson and Harrison Emerson & Harrison (2025) survey 41 accessible multimodal corpora and organize the available modalities across three phases: before performance (score, instrument, venue, performer background), during performance (performer movement and physiological state, instrument state, sound), and after performance (performer and listener evaluation). The observation: Performer physiological state during performance appears in one dataset, listener eye tracking in one, performer physical attributes and personality inventories in none.

Optical motion capture gives millimetre precision at the cost of ecological validity. MOSA Huang et al. (2024) is the largest such corpus: 742 performances by 23 professional pianists and violinists, 30 hours, 570K notes, with 3D motion paired to audio and note-level annotation of pitch, beat, phrase, dynamic, articulation, and harmony. SKY-Piano Bae et al. (2026) adds hand and body motion, multi-view video, audio, MIDI, and scores from 7 professional and 12 amateur pianists on a shared core repertoire stratified by technique and difficulty, with motion released in both occlusion-flagged and imputed form. For bowed strings, TELMI Volpe et al. (2017) records four master violinists across 41 techniques with synchronized motion, video, audio, and EMG; QUARTET Papiotis et al. (2014) captures body and bow motion for string quartets to study ensemble interdependence; SPD Jin et al. (2024) provides cello and violin performances from up to 23 camera views with 3D annotation of body, hands, instrument, and bow. Percussion is served by the snare drum rudiment corpus Van Rooyen et al. (2015).

Video trades precision for scale and for access to performances that were never staged for a lab. PianoMotion10M Gan et al. (2025) annotates 10 million hand poses across 116 hours of bird’s-eye-view piano video collected from the internet. FürElise Wang et al. (2024) reconstructs 10 hours of 3D hand motion from 15 elite pianists using five calibrated cameras, refining pose estimates by inverse kinematics against Disklavier MIDI. PianoVAM Kim et al. (2025) records amateur daily practice with top-view video, hand landmarks, and semi-automated fingering labels; its reported gain from adding visual information to onset detection was smaller than expected, which is the most useful negative result in this literature. Fingering extraction from video and MIDI Moryossef et al. (2023) yields 150,000 tagged notes without any capture hardware. MusiQAL Christodoulou et al. (2025) shows that standard performance video supports multimodal reasoning rather than only kinematic measurement: 310 recordings spanning jazz, raga, djembe rhythms, and Polynesian dance, paired with 11,793 question-answer pairs grounded in musicology and music psychology.

The tradeoff is instrument-dependent. Markerless pose carries centimetre-level joint error, which is tolerable for gross posture and ensemble coordination and fatal for fingering, embouchure, or bow contact point. Emerson and Harrison nonetheless recommend video-based estimation as the default for new corpora Emerson & Harrison (2025), on the grounds that intrusive capture suppresses the behaviour being measured.

Instrument-embedded sensors and biosignals sit outside this corpus infrastructure. EMG appears in TELMI and the Violin Gestures set, breath and embouchure sensing exists only as lab instrumentation, and gaze is confined to sight-reading studies with no released data. EEG is the sharpest case: the MIR-facing EEG corpora are all listening datasets, and performance-side EEG lives in music psychology venues with few benchmark infrastructure. For a system that must reason about how someone plays, these channels are currently evidence that a question can be asked, not data on which a model can be trained.

Body motion is not instrument-agnostic or culture-neutral. A cellist’s bow arm path reflects centuries of Western conservatory pedagogy; a sarod player’s right-hand technique follows entirely different physical and sonic principles. The Hindustani case is now measured rather than asserted. Clayton et al. Clayton et al. (2023) classify singers from movement information alone and reach 38.2% mean accuracy on nine ragas from pose data, and gesture-melody correspondence has been studied on purpose-built alap corpora Nadkarni et al. (2023). Singers’ gesturing is idiosyncratic and tightly coupled to melodic movement at the same time, which is precisely the property that a universal body-motion feature space would destroy.

The practical consequence for LLM-based systems is that motion is the modality where data exists but the interpretation does not. Current work is either pose estimation with threshold rules, or audiovisual benchmarks that stop at description. A motion analyser belongs in the system as a tool the model calls, returning bow contact point or key-descent velocity as structured measurements. The interpretable measurement (labels) that corresponds raw coordinates to skill interpretation (wrist too high, leap on the left hand) is the key to enable instrumental pedagogy.