AUTOMATIC TRANSCRIPTION OF PIANO MUSIC
BY SPARSE REPRESENTATION OF MAGNITUDE SPECTRA
Cheng-Te Lee, Yi-Hsuan Yang, and Homer Chen
National Taiwan University
aderleee@gmail.com, affige@gmail.com, homer@cc.ee.ntu.edu.tw
ABSTRACT
Assuming that the waveforms of piano notes are pre-stored
and that the magnitude spectrum of a piano signal segment
can be represented as a linear combination of the magnitude
spectra of the pre-stored piano waveforms, we formulate the
automatic transcription of polyphonic piano music as a
sparse representation problem. First, the note candidates of
the piano signal segment are found by using heuristic rules.
Then, the sparse representation problem is solved by l1-
regularized minimization, followed by temporal smoothing
the frame-level results based on hidden Markov models.
Evaluation against three state-of-the-art systems using ten
classical music recordings of a real piano is performed to
show the performance improvement of the proposed system.
Index Terms— F0 estimation, multiple pitch estimation,
sparse representation, l1-regularized minimization.
1. INTRODUCTION
Music transcription is the process of converting a musical
recording to a musical score (i.e., symbolic representation).
This process in the strict music sense entails a number of
complicated tasks to extract musical attributes, including
pitch, starting time, duration, instrument type, and dynamics
of each note and tempo, meter signature, and tonality (key
signature) of the music. In its simplest form, music
transcription can be reduced to the estimation of the pitch
and timing of each individual note. Such information,
although only a small subset of the whole musical attributes,
is sufficiently useful for content-based music retrieval, such
as query by humming, query by example, and cover song
identification. This paper mainly deals with this simplified
version of music transcription.
Like the work described in [1]-[3], we limit the audio
source to piano (either real or synthesized). Although only
one single type of audio source is considered, the problem is
by no means trivial. In fact, Peeters [4] showed that the
___________________________
This work was supported by the National Science Council of
Taiwan under contract NSC 97-2221-E-002-111-MY3.
978-1-61284-350-6/11/$26.00 ©2011 IEEE
of piano music is more challenging than that of other
instruments because pianos have wide pitch range and high
polyphony. We are motivated to tackle the transcription of
piano music in this work also because piano is the most
popular musical instrument worldwide. The fact that there
are usually no more than six concurrent notes played from
the 88 notes of a piano inspires us to use sparse
representation to solve the transcription problem.
1.1 Previous work
Music signals can be monophonic or polyphonic. For
polyphonic music, one has to deal with issues such as the
source number ambiguity and the octave ambiguity that do
not exist in monophonic music; therefore, polyphonic music
transcription is more complicated than monophonic music
transcription. For this reason, monophonic music
transcription is relatively easy to deal with and mature [5],
but polyphonic music transcription still has much room for
improvement.
Moorer [6] made the first attempt to transcribe
polyphonic music. Although the system is limited to duets,
it has inspired many interesting ideas. For example, Marolt
[1] applied neural networks and oscillator networks to track
partials, Klapuri [7] adopted an iterative estimation and
cancellation scheme to estimate concurrent fundamental
frequencies (F0s), Bello et al. [2] applied a rule-based
framework to find meaningful groups of spectral peaks,
Poliner et al. [3] adopted a machine learning approach to
determine whether a particular note is present, and Duan et
al. [8] presented a probabilistic approach to fundamental
frequency estimation. These systems used the batch learning
technique to learn the relationships between the spectra of
the music signals and the underlying notes. Therefore, the
models cannot adapt to different types of music without re-
training. Moreover, generating the ground truth data for
learning is time-consuming [9]. Abdallah et al. [10] and
Cont [11] introduced a sparsity constraint into their
transcription system, and they only used one base element
for each note. As shown by Wright et al. [12], the
performance of sparse representation highly depends on the
overcompleteness of the dictionary of base elements. Here,

Fig. 1. The proposed automatic piano transcription system.
Fig. 2. The original and the tuned spectra of a sample of note C4
(MIDI number 60). The estimated tuning factor is 0.982.
Fig. 3. Illustration of the peak detection algorithm.
sparse representation refers to an expression of the input
signal as a linear combination of base elements, and the
resulting representation only has few non-zero terms.
To address these issues, we propose a system that
adapts to different pianos without retraining. It only requires
the WAVE [13] files of a dozen individual notes of a new
piano to be added to the piano sound database. In addition,
the system uses more than one base element for a note to
improve the transcription performance.
Table 1. Tuning algorithm
Input: magnitude spectrum M of a sample, tuning factor τ
Output: tuned magnitude spectrum Mt
1. For i = 1 to the dimension of M
2. Mt[i] = 0
3. For i = 1 to the dimension of M
4. n = round((i-1) × τ) + 1
5. /* round(x) rounds x to the nearest integer */
6. Mt[n] = Mt[n] + Mt[i]
1.2 Organization of the paper
The organization of this paper is as follows. In Section 2, we
propose a piano transcription system that is based on sparse
representation of magnitude spectra without the need for
model training. In Section 3, the proposed system is applied
to transcribe ten recordings of solo piano music [3] and
compared to previous approaches. Section 4 concludes this
paper.
2. PROPOSED SYSTEM
Fig. 1 shows the schematic diagram of our transcription
system. The system takes a piano WAVE file as input and
generates the transcription in MIDI file format [14]. The
system first normalizes the root-mean-squared (RMS)
amplitude of the input WAVE file and estimates the tuning
factor of the piano. Then the samples in the piano sound
database are tuned accordingly. During this preprocessing
stage, the system also decomposes the audio content of the
normalized WAVE file into short-time frames and applies
fast Fourier transform (FFT) to each frame. The note
candidates are selected according to the resulting spectrum
and the tuning factor. Then, the sparse representation of the
frame spectrum using the spectra of the note candidates is
created. Because we formulate the sparse representation as
an l1-regularized minimization problem, there are some
small non-zero coefficients equivalent to noise. The system
eliminates such noise by setting a threshold. Finally, the
system applies temporal smoothing to the resulting frame-
level sparse representation to generate the transcription. The
detail of each component is described below.
2.1 Data preprocessing
Before the transcription process starts, a data pre-processing
step is performed.
First, a database of piano note waveforms with standard
tuning is built. There should be at least one waveform for
each piano note in the database. Also, there can be several
waveforms generated by different pianos for the same note.
Each waveform is then divided into short-time samples, and
the energy of each sample is normalized.
Then, the RMS amplitude of the input piano WAVE
file is normalized, and the tuning of the piano is estimated
using

(a)
(b)
Fig. 4. Spectra of note D4 (MIDI number 62) of two different
pianos. The magnitude of the fundamental frequency is marked by
circle. (a) Strong-fundamental type. (b) Weak-fundamental type.
the method proposed in [1]. It uses adaptive oscillators to
find partials in the input WAVE file and calculates the
tuning factor by comparing the frequencies of the partials to
the frequencies of the notes of a piano with standard tuning
(i.e., A4=440 Hz). After the tuning factor is obtained, the
samples in the database are tuned using the algorithm shown
in Table 1. Note that the tuning factor is a real number and
that its value is set to 1.0 for standard tuning. Fig. 2 shows
the difference between the original and the tuned spectra of
a certain note sample. The spectra of the tuned samples
serve as the base elements of the sparse representation.
2.2 Note candidates selection
For each frame, the system selects a number of note
candidates to reduce the search space of the sparse
representation coefficients. In this way, we can reduce the
time complexity of sparse representation.
First, the system detects significant spectral peaks in a
manner similar to the one proposed in [15], which is
illustrated in Fig. 3. The smoothed spectrum is obtained by
convolving the original spectrum with a moving Gaussian
filter. If the magnitude of a frequency satisfies the two
requirements described below, it is considered a significant
spectral peak. The first requirement is that the original
magnitude of a significant spectral peak should be higher
than a predefined global threshold. The second requirement,
which is a local requirement, is that the original magnitude
of a significant spectral peak should be higher than the
magnitude of its smoothed version by a predefined margin.
In addition to peak detection, the system determines the
frequency of each note. The frequency f of the nth key of a
piano is defined by
n − 49
f = (440 × τ ) × 12 2 , (1)
where τ is the tuning factor.
The system at this point can select note candidates by
the following two heuristic rules:
• If the magnitude of a significant spectral peak at
frequency fp is higher than the magnitude of the spectrum
at integer multiples (harmonics) of fp, the note whose
frequency is nearest to the spectral peak is selected as a
candidate.
• Following the last rule, if the magnitude of the spectrum
at fp/2 is higher than a pre-defined threshold, the note
whose frequency is nearest to fp/2 is selected as a note
candidate. The same procedure is applied for fp/3.
The two rules are designed for two types, strong-
fundamental and weak-fundamental, of harmonic series of
piano notes. As shown in Fig. 4, the spectra of the same note
of different pianos may look different. Fig. 4(a) shows an
example note of strong-fundamental type. We can see that
the spectral envelop has maximum magnitude at the
fundamental frequency marked by circle. The first rule is
used to detect notes of the strong-fundamental type, while
the second rule is used to detect notes whose spectral
envelope has maximum magnitude at one of the harmonics,
see Fig. 4(b). Notes with such spectral envelope belong to
the weak-fundamental type. Yeh et al. have a similar
classification of the harmonic series [16].
2.3 Computation of sparse representation
We assume that the magnitude spectrum of a frame can be
represented as a linear combination of the base elements,
which are the spectra of samples of the note candidates in
the tuned database. However, the coefficients of such linear
combination may not be unique because of the mathematical
dependencies between the base elements. To address this
issue, we impose a sparsity constraint to require that the
number of non-zero coefficients should be minimal. The
sparsity constraint is reasonable [12] because “the sparsest
representation is naturally discriminative.” For the
transcription problem dealt with in this paper, the constraint
implies that the spectrum of a frame is most compactly
expressed using the base elements of notes present in the
frame. Fig. 5 illustrates the idea of sparse representation.
Denote the dictionary of base elements for the ith key
of a piano by Ai = [ai,1|ai,2|…|ai,Ni], where ai,k is a column
vector containing the magnitudes of the Fourier coefficients
of the kth short-time sample of that key and Ni is the number
of such samples. Let the column vector y be the magnitudes
of the Fourier coefficients of a frame, the problem now is to
find a sparse coefficient vector x* such that
x* = arg min || x ||0 subject to y = Ax, (2)
x
where A contains the dictionaries of the note candidates.
Because finding the solution of (2) is NP-hard, it is not
 (a) (b)
Fig. 5. Illustration of the sparse representation of frame spectrum for music transcription. (a) The magnitude spectrum of a frame. (b) The
coefficients of sparse representation. (c) The base elements. (d) The residues.
practical to solve it directly. Fortunately, Donoho [17]
demonstrated that if the solution of (2) is sparse enough, it is
equal to the solution of the following l1-minimization
problem:
x* = arg min || x ||1 subject to y = Ax.
x
It is shown in [18] that the solution of (3) is close to the
following l1-regularized minimization problem:
x = arg min || y - Ax || + λ || x ||1 ,
* 2
x significant spectral peaks
where the regularization parameter λ is a real non-negative
number. We use the method described in [19] to solve (4).
After obtaining the sparse coefficient vector x* of (4), we
consider that a note is present in the frame if the summation
of the coefficients corresponding to that note, its activation
index, is larger than a threshold.
2.4 Temporal smoothing
The FFT described above treats the short-time frames
independently, leaving the temporal structure of music
unexploited. To address this issue, we use two-state (on and
off) hidden Markov models (HMMs) to model the attack
and decay of each note [3]. For each note, we want to
maximize
∏ p( x | q ) p(q t t t | qt −1 ),
t [3]. The executable files of [1], [7] are provided by the
where qt is the state at time t, xt is the frame beginning at
time t, p(xt|qt) is the probability of xt given qt, and p(qt|qt-1) is
the transition probability between states. Although we do
not know p(xt|qt), from the conditional probability we have
p ( qt | xt ) ∝ p ( xt | qt ) p (qt ) .
Therefore, we can maximize
p ( qt | xt )
∏ p ( qt )
p( qt | qt −1 )
t 3.1 Experiment set-up
instead of (5). p(qt|xt) is obtained by dividing the activation
index of the note by the maximum activation index of xt.
Both the prior p(qt) and the state transition probability
p(qt|qt-1) can be learnt from the training data. Note that in
(c) (d)
Table 2. Parameter values used in the proposed system
Parameter Value
The RMS amplitude for the normalization of the
0.25
input WAVE file
(3) The length of each base element (|ai,k|) 1.0
The radius of the moving Gaussian filter 10
The global threshold for the selection of the
1.0
significant spectral peaks
(4) The minimal margin for the selection of the
0.75
Regularization parameter (λ) 100.0
this problem formulation the learning process is about the
music structure rather than the timbre characteristic of a
particular piano. We apply the Viterbi algorithm to find the
solution of (7).
After the temporal smoothing by HMMs, the system
generates the transcription of the input WAVE file in the
MIDI format.
3. EVALUATION
We evaluate the performance of the proposed system against
three state-of-the-art systems [1], [7], [20] using ten one-
minute long classical music recordings (in the form of
(5) WAVE files) of a Yamaha Disklavier playback grand piano
authors, and the evaluation result of [20] is provided by its
author. Each music recording comes with a MIDI file to
serve as the ground truth. There are 4952 notes in the
recordings, and the average polyphony of them is 3.5.
Klapuri’s system [7] achieved the best performance for
(6) Multiple Fundamental Frequency Tracking task of Music
Information Retrieval Evaluation eXchange (MIREX) in
2008, whereas Yeh’s system [20] achieved the best
performance for the same contest in 20101.
(7)
1
http://www.music-ir.org/mirex/
 Table 3. Result of the frame-based evaluation Table 4. Result of the note-based evaluation
F-measure Precision Recall F-measure Precision Recall
Proposed system 70.2% 74.4% 66.5% Proposed system 73.0% 74.6% 71.6%
Marolt’s system 66.1% 78.6% 57.1% Yeh’s system 67.1% 57.2% 81.1%
Klapuri’s system 62.2% 72.4% 54.6%
2 × Precision × Recall (10)
F-measure = ,
Precision + Recall
where Ntp is the number of correctly transcribed notes (true
positives), Nfp is the number of unvoiced notes transcribed
as voiced (false positives), and Nfn is the number of voiced
notes transcribed as unvoiced (false negative) of all frames.
Table 3 shows the average performance results for the
frame-based evaluation. We can see that the proposed
system achieves a better F-measure, which is a combined
(a)
metric for recall and precision. It is also observed that the
high recall of our system is obtained at the cost of a little bit
lower precision than Marolt’s system [1].
Fig. 6 shows the details of the transcription
performance with respect to polyphony. Both F-measure and
recall of the proposed system are consistently better than
that of Klapuri’s system [7] and Marolt’s system. Compared
with Klapuri’s system, the proposed system has better
precision for polyphony 2, 4, and 6 or more. Compared with
(b) (c) Marolt’s system, the proposed system has better precision
Fig. 6. Evaluation results in terms of the polyphony of the frames. for polyphony 2. As mentioned, there is a tradeoff between
The numbers in the parentheses are relative frequencies of the precision and recall, and the best tradeoff is achieved when
polyphony. (a) F-measure. (b) Precision. (c) Recall. the F-measure is maximized. To improve the F-measure
performance, a more sophisticated method that learns the
The waveforms in the database are synthesized by two
music structure, say, the relationship between concurrent
commercial software tools, and three different piano timbres
notes, should be investigated in future work.
are used2. There are 646 base elements in the database, and
each note has six to eight corresponding base elements. The
3.3 Note-based evaluation
testing WAVE files are monaural, and the sampling rate is 8
kHz. The frame size for FFT is 100 milliseconds, and the
In addition to the frame-based evaluation, we also evaluate
hop size between successive frames is 10 milliseconds. The
the performance of the proposed system in a note-based
training data used for learning the prior and the transition
manner and compare it with Yeh’s system [20]. A note
probability is also provided in [3]. Note that the training
output by the system is considered a correctly transcribed
data is disjoint with the testing data. Values of the
note if its MIDI number matches with the ground truth and
parameters of the proposed system are listed in Table 2.
if the system switches it on within 100 milliseconds from
the onset specified in the ground truth MIDI file. The
3.2 Frame-based evaluation
evaluation is performed for each of the ten music recordings
described earlier. At the end of the evaluation, the number
We use three typical metrics to evaluate the transcription
of false negatives is obtained by subtracting the number of
performance, namely precision, recall, and F-measure. The
correctly transcribed notes from the total number of ground
metrics are defined as follows:
truth notes, and the number of false positives is obtained by
N tp (8) subtracting the number of correctly transcribed notes from
precision = , the total number of notes output by the system. We do not
N tp +Nfp
take the offset of the notes into consideration because they
N tp (9)
recall = , have little perceptual importance [3]. Table 4 shows the
N tp +N fn result of the note-based evaluation. We can see that the F-
measure and precision of the proposed system are better
than Yeh’s system.
2 In summary, the performance of the proposed system is
Piano 1 and Piano 2 of Native Instrument’s Bandstand and
Acoustic Piano 1 of Arobas‘s Guitar Pro. consistent and unbiased, and the improvement over the three
state-of-the-art systems is significant under the one-tailed t-
test (p-value<0.05).
4. CONCLUSION
We have presented an automatic transcription system for
piano music. The system first performs a data pre-
processing step for volume normalization, tuning factor
estimation, and database tuning. Then, the system
decomposes the audio content of the input music file into
short-time frames. After the selection of note candidates
based on two types of harmonic series, the sparse
representation of each frame is computed by l1-regularized
minimization, followed by temporal smoothing the frame-
level results based on hidden Markov models. Evaluation
results show that the approach is viable.
The system can be improved in three aspects. First, we
estimate only one tuning factor for the entire piano keys in
the current system, but the fact is that each key has its own
tuning. Estimating the tuning factors for the treble, middle,
and bass sections of the piano may help improve the
accuracy of the estimation. Second, in addition to learning
the intra-note relationship, we should learn the inter-note
relationship to gain insight into the temporal structure of
music. Last but not least, onset detection can be
incorporated to the system to improve performance, for
which the sparse representation coefficients serve as a
valuable clue.
ACKNOWLEDGEMENT
The authors would like to thank A. Klapuri and C. Yeh
for generously providing source code and evaluation result
for comparison.
REFERENCES
[1] M. Marolt, “A connectionist approach to automatic
transcription of polyphonic piano music,” IEEE Trans.
Multimedia, vol. 6, no. 3, pp. 439–449, 2004.
[2] J. Bello, L. Daudet, and M. Sandler, “Automatic piano
transcription using frequency and time-domain
information,” IEEE Trans. Audio, Speech, Lang. Process.,
vol. 14, no. 6, pp. 2242–2251, Nov. 2006.
[3] G. Poliner and D. Ellis, “A discriminative model for
polyphonic piano transcription,” EURASIP J. Advances
Signal Process., vol. 8, pp. 1–9, 2007.
[4] G. Peeters, “Music pitch representation by periodicity
measures based on combined temporal and spectral
representations,” in Proc. Int. Conf. Acoust., Speech,
Signal Process., Toulouse, France, pp. 53–56, May 2006.
[5] A. de Cheveigné and H. Kawahara, “Yin, a fundamental
frequency estimator for speech and music,” J. Acoust. Soc.
Amer., vol. 111, pp. 1917–1930, 2002.
[6] J. A. Moorer, “On the transcription of musical sound by
computer,” Comput. Music J., vol. 1, no. 4, pp. 32–38,
1977.
[7] A. Klapuri, “Multiple fundamental frequency estimation
by summing harmonic amplitudes,” in Proc. Int. Conf.
Music Inform. Retrieval, Victoria, Canada, pp. 216–221,
Oct. 2006.
[8] Z. Duan, B. Pardo, and C. Zhang, “Multiple Fundamental
Frequency Estimation by Modeling Spectral Peaks and
Non-Peak Regions,” IEEE Trans. Audio, Speech, Lang.
Process., vol. 18, no. 8, pp. 2121–2133, Nov. 2010.
[9] C. Yeh, N. Bogaards, and A. Roebel, “Synthesized
polyphonic music database with verifiable ground truth
for multiple-F0 estimation,” in Proc. Int. Conf. Music
Inform. Retrieval, Vienna, Austria, pp. 393–398, 2007.
[10] S. A. Abdallah and M. D. Plumbley, “Polyphonic
transcription by non-negative sparse coding of power
spectra,” in Proc. Int. Conf. Music Inform. Retrieval,
Barcelona, Spain, pp. 318–325, Oct. 2004.
[11] A. Cont, “Realtime Multiple Pitch Observation using
Sparse Non-negative Constraints,” in Proc. Int. Conf.
Music Inform. Retrieval, Victoria, Canada, pp. 206-212,
Oct. 2006.
[12] J. Wright, A. Yang, A. Ganesh, S. Sastry, and Y. Ma,
“Robust Face Recognition via Sparse Representation,”
IEEE Trans. Pattern Anal. Mach. Intell., vol. 31, no. 2, pp.
210-227, Feb. 2009.
[13] IBM Corporation and Microsoft Corporation,
“Multimedia Programming Interface and Data
Specifications 1.0,” Aug. 1991.
[14] MIDI Manufacturers Association, “Complete MIDI 1.0
Detailed Specification,” Nov. 2001.
[15] Z. Duan, Y. Zhang, C. Zhang, and Z. Shi, “Unsupervised
single-channel music source separation by average
harmonic structure modeling,” IEEE Trans. Audio,
Speech, Lang. Process., vol. 16, no. 4, pp. 766–778, May
2008.
[16] C. Yeh, A. Roebel, and X. Rodet, “Multiple Fundamental
Frequency Estimation and Polyphony Inference of
Polyphonic Music Signals,” IEEE Trans. Audio, Speech,
Lang. Process., vol. 18, no. 6, pp. 1116–1126, Aug. 2010.
[17] D. Donoho, “For Most Large Underdetermined Systems
of Linear Equations the Minimal l1-Norm Solution Is
Also the Sparsest Solution,” Commun. Pure Appl. Math.,
vol. 59, no. 6, pp. 797-829, 2006.
[18] E. Candès, J. Romberg, and T. Tao, “Stable signal
recovery from incomplete and inaccurate measurements,”
Commun. Pure Appl. Math., vol. 59, no. 8, pp. 1207-1223,
Aug. 2006.
[19] S. Kim, K. Koh, M. Lustig, S. Boyd, and D. Gorinvesky,
“An interiorpoint method for large-scale l1-regularized
least squares,” IEEE J. Sel. Topics Signal Process., vol. 1,
pp. 606–617, 2007.
[20] C. Yeh and A. Roebel. (2010). Multiple-F0 estimation for
MIREX 2010. Music Information Retrieval Evaluation
eXchange. [Online]. Available:
http://www.music-ir.org/mirex/abstracts/2010/AR1.pdf