Intelligent algorithm for music playing robot

Applied to the anthropomorphic piano robot control

Yen-Fang Li* and Chi-Yi Lai
Dept. of Elec. Eng., Ming-Hsin University of Science and Technology, Hsin-Chu 30401, Taiwan
Email:yfli@must.edu.tw
AbstractSimple and friendly operation interface, intelligent
calculation, high performance control and driving circuit, and
precision and reliable plant are the necessary conditions for
intelligent robot design. In this paper, an intelligent algorithm is
proposed for the piano robot control with an interactive program
and an automatic control code generator. The interactive
program of man-machine interface makes the musical score input
is simple and friendly as the piano keyboard playing. This
interactive program can identify the musical beat and note
automatically while the user plays the keyboard on PC monitor.
The interactive program of man-machine interface makes the
musical score input is simple as the piano keyboard playing. This
program can identify the musical beat and note automatically
while the user plays the keyboard on PC monitor and then the
musical score is built. The control code generator, an intelligent
algorithm, will convert the musical score to generate a series of
optimum positions commands accordingly for the hands and
fingers of the piano robot to play a piano. The optimum positions
commands are programmed with crashing protection and
minimum movement for the hands and fingers to
anthropomorphize the robot. Via the algorithm operation, the
music control codes are generated automatically to replace the
situation of manual coding and give the ability of intelligent
thinking for the piano robot.
Keywords: Intelligent robot, piano robot, man-machine interface,
intelligent algorithm, crashing protection, PC-based
controller

I. INTRODUCTION
At the viewpoint of industry and economics, robots are
designed for industrial manufacture. Except the industrial and
military applications, the other applications of robotic design
[1][2] cant be paid much attention. In 2005 Japan's Aichi Expo
[3], the amazing designs of robotic applications triggers
peoples impression the robots are not only for industrial
manufacture that take the place of worker but also they can
further develop to service for human. In this Expo, Honda
robot Asimo takes over the spotlight [4]. She can serve a cup of
tea and take the slippers to master. Moreover, the traditional
cognition was broken while Sony introduced the AIBO, a
robotic dog [5]. The robot is not necessary human, it may be a
pet she can interact with people.
The development of the services and entertainment robots
changes the concepts about the robot. The robot is not
necessarily a rigid and heartless machine it can also be a life
accompanied partner. In the future, an era with aging
population, we need more served robots to take the housework
and serve the life for the elder. 2005 Aichi Expo has triggered
the development of robot for entertainment and service
application. Focusing on the unlimited business opportunities,
in the future, robot design and development has once again in
fashion [5] [7] [8] [9] [10][11].

978-1-4799-2399-1/14/$31.00 2014 IEEE

The playing robot, a kind of entertainment robot, is

designed specifically for playing a musical instrument. Toyota,
a Japan's motor manufacturer, has developed a series of playing
robot, namely Toyota Partner [5]. The series of playing robots
can play some common instruments including tuba, trombone,
trumpet, and drums and even the violin.
Best of our knowledge, the first two hands playing robot for
piano playing was proposed in [10]. Another piano robot was
presented by [11] where it claims the playing speed can exceed
human playing. The mechanism only has 19 fingers that these
cant be rotated without movable hands. In spite of the fixed 19
fingers, without rotatable fingers and movable hands, can highspeed playing it is difficult to call a piano robot since it is
unlike a human being.
Continuously, Li et al. proposed the hierarchical control
structure, PC as the host controller and FPGA as the target
controller, to implement the anthropomorphic piano robot
control with parallel controlling of two hands and ten fingers [12].
This system consists of three sub-systems, the host controller,
the target controller (or local controller) and the realistic
control plant. Moreover, the practical design method for the
target controller has been studied in [13] that proposed the
details about the controller design for the driving motors of
palms and fingers.
Besides the controller design for all of the driving motors, it
also needs a friendly man-machine interface as the input unit of
a music score and an action producer as the control code
generator of the driving motors for the music robot. These
works should be done by the host controller, in other words, it
needs an intelligent program to deal with the complicated
action scheduling.
Continuous the research of music playing robot, in this
paper, we focus on the developments of music score inputting
and robot action scheduling. Since the host controller is in
charge of the robot motions scheduling and control codes
commanding to target controller, an intelligent algorithm is
needed to generate the music control code with a friendly
interactive man-machine interface for music score inputting.
The man-machine interface provides a simple way to input
music notes on touch screen to simulate the piano keyboard.
While the music score has inputted, the scheduling program
will generate a series of optimum positions commands for the
hands and fingers of the piano robot to play the piano. The
series of optimum positions commands are programmed with
collision checking, crashing protection and movement
minimization for the hands and fingers to anthropomorphize
the robot. Finally, the design algorithm and controller have
been totally implemented to prove our design philosophy.

1538

II.
PRELIMINARY AND SYSTEM DESIGN
In general, an instrument is played by two hands and multifingers at the same time coordinately. It implies that the music
playing robot must actuate many actions in the meanwhile
coordinately. In other words, a piano playing robot should have
the capability of the coordinately action control. For the piano
playing robot, it must have two main functions, multi-task
operation and good coordinated control. Multi-task operation
implies the system can provide with parallel control capability
for several driving actuators of the robot. Good coordinated
control infers the system has central controller to respond the
overall motion planning. In our research, the proposed robot
has two hands and ten fingers to play a piano for
anthropomorphizing the human action as Fig. 1.

Fig. 1 The proposed piano robot

The developed system is a hierarchical control structure

with three sub-systems including the host controller, the target
controller (or local controller) and the realistic control plant (or
physical plant). The overall architecture for the piano robot is
shown as the diagram of Fig. 2.

Fig. 2 The overall architecture diagram of piano robot

The host controller is implemented by PC and it does not

actually execute the close loop control of each driving motor.
The host controller is in charge of the robot motions scheduling
and control codes commanding to target controller. Absolutely,
the host controller must have a friendly operation interface to
input music notes and beats code comes from a music score for
the music playing. Fig. 3 shows a simple music score for piano
playing.

Fig. 3 Part of a music score

Hence, first, the controller will input a series of music notes

and beats code including treble stave and bass stave. Next,
according to these notes and beats code, the controller will
program the path of hand movement with least displacement
and assign the most appropriate hand and finger to beat the
note. Then, according to these notes and beats code, the

controller will generate a series of commanding codes and

transmitting the code to target controller under the rules of data
format. The summary of the host controller working is
described in Fig. 4.

Fig. 4 The host controller function diagram of the piano robot

As Fig. 4, the host controller is responsible to get music

scores, plan the motion paths, schedule the hands and fingers
action, and generate the control codes for playing music of the
robot. In the controller, an interactive man-machine interface
and an intelligent algorithm are programmed to input music
score and generate the music playing control code.
Since the host controller is implemented by PC and then all
the controller works are realized with software program.
Basically, there are three tasks should be achieved by this
program:
(1) A friendly man-machine operation interface: It is required
for the user to input a playing music score with an
interactive man-machine operation panel. And, the
program can generate a specific music code file for the
next process working.
(2) Scheduling and control code generator: An intelligently
calculated algorithm is necessary for the transformation of
music score to control code. The function is the kernel of
the host controller to help the controller works for the
motions scheduling and control codes generating with
collision relieving and minimum hands displacement
while the robot playing music.
(3) Commanding code generator: Finally, the work is to
combine the motion control codes of hands and fingers
with action coordinating and crashing protection to get
actually commanding the target controller by transmitting
a sequence of command code through the parallel I/O
interface of PC.
III. DATA CODING AND TRANSMISSION FORMAT
For the service robot, an interactive man-machine interface
is necessary to serve human that implies the host controller
should have a friendly operation panel to easily input the music
score for the user. In the proposed system, on the interactive
man-machine interface, the user can input the music score by
playing a simulated keyboard on a touch screen or directly fills
the coding table with a certain data format. Before states the
interactive algorithm of the music score input, the music code
coding should be introduced. In the paper, the music note and
beat are coded with the following format:
(1) Music note coding (8bits):
b7
b6
b5
b4
b3
b2
b1
b0
scale
key
note
(a) Scale
000 lower the next octave; C2 B2

1539

index*1

001 lower octave; C3 B3

010 normal octave; C4 B4
011 higher octave; C5 B5
100 higher the next octave; C6 B6
101 liaison
other reserved
(b) Key
00
01
10
normal
sharp: #
flat; b
(c) Note
000
001 010 011 100
101 110 111
reserved Do
Re
Mi Fa
So
La Si
(2) Beat time coding (8bits):
0x00
sustain
0x10 1 beat
0x01
1/16 beat
0x18 1+1/2 beats
0x02
1/8 beat
0x20 2 beats
0x04
1/4 beat
0x30 3 beats
0x08
1/2 beat
0x40 4 beats
0x0C
3/4 beat
other reserved

Note:
1. Hand index: left hand = 0, right hand = 1
2. Hand position coding: 5 bits binary coding for 31 positions
3. Thumb: Angle coding for thumb moving around
4. Index finger: Angle coding for index finger moving around
5. Middle finger: Angle coding for middle finger moving around
6. Ring finger: Angle coding for ring finger moving around
7. Little finger: Angle coding for little finger moving around
TABLE 2 DATA FORMAT OF FINGERS RAPPING COMMAND
bit
port

Example 1: The below stave is part of the music score

proud of you.

Following the above coding rules, the treble stave can be

coded as:
Note code = {0x63,0xB0,0x63,0x64,0x65,0xB0,0x66,0x67}
Beat code={0x10,0x00,0x08,0x08,0x10,0x00,0x08,0x08}.
And the bass stave can be coded as:
Note code ={0x41,0x45,0x61,0xB0,0x27,0x45,0x47,0xB0};
Beat code ={0x08,0x08,0x10,0x00,0x08,0x08,0x10,0x00}.
After the music score input, the host controller, the
intelligent algorithm, will generate a series of optimum
positions commands for the hands and fingers of the piano
robot to play the piano. Hence, the command code should
follow the hardware specification and communication protocol
of the interface between the host controller and target
controller. Here, we introduce the command transmission unit
of the host controller. In the system, we use a PCI DI/DO
interface card manufactured by Advantech Technologies Corp.,
Taiwan with 48 bits DI/DO ports. Through the DI/DO interface
card, the series commanding codes will down load to the target
controller to control the two hands and ten fingers. The I/O
interface ports are programming with two bytes for the hands
and fingers position encoding and 18 bits for the fingers
rapping action. The details about the I/O ports arrangement are
described in the following.
TABLE 1 DATA FORMAT OF HANDS AND FINGERS POSITION COMMAND
bit
port
PA0
PB0

Index finger
coding*4
Hand

Middle finger Ring finger

coding*5
coding*6
Hand position coding*2

b
coding 3

6
*1

*1

NA

NA

NA

NA

NA

NA

PC0

RB2

PA1

LB1*2 LB2*2 LB3*2 LB4*2 LB5*2 RB5*1 RB4*1 RB3*1

PB1

LS2*3

RB1

LS3*3

LS4*3

LS5*3

RS5*4

RS4*4

RS3*4 RS2*4

Note:
1. RBx: Rapping command of finger no. x of right hand
2. LBx: Rapping command of finger no. x of left hand
3. LSx: Stretching command of finger no. x of left hand
4. RSx: Stretching command of finger no. x of right hand
5. Finger no. x: 1= thumb, 2= index finger, 3= middle finger, 4= ring
finger, 5= little finger

Table 1 and Table 2 show the data transmission format

between host computer and target controller. Table 1 expresses
the position command format of two hand driving motor which
is implemented by a permanent magnet synchronous linear
motor (PMSLM) with two movers in common stator track
(Hiwin-LMCB5 linear motor) and the angle command format
of ten fingers driving motors which are implemented by servo
motor sets. For the sake of data bus using effectively, the data
transmission for the two hands control command is design by
sharing the same I/O ports with the time division. Simply put,
the host controller only orders the position and angle command
instead of direct control the motor. From Table 1, the hand
position is coded with 6 bits which can address 64 positions
great than 52, the white key of piano. And, each finger is coded
with 2 bits that implies the finger can address 4 angles except
the thumb coding with 1 bit.
Table 2 describes the I/O mapping table of music tone
raping command. Comparing with position and angle control,
the raping action is easy to control since the raping actuators
are implemented by solenoid valves for each finger raping
action. The counterpart needs to process the close loop control
with a properly controller design. Although the raping action is
only simple control, it needs accurate control in timing
according to the music tones of playing song. In this design,
the raping action is controlled by host controller directly based
on the host computer is easier programming for the scheduling
of raping action.
Since the position commands transmission of the hand and
fingers of left and right shares the same I/O ports, thus the
commands transmission will have time slot for each hand. The
time slots dividing and control timing of the I/O data
transmission between host controller and target controller is
described in Fig. 5.

Little finger
coding*7
Thum

1540

stave code compatible problems.

Function 3; Measure score check and store: Step (7) ~ Step
(8) check the one music measure if it has completed.
Function 4; Music score check and database build: Finally,
Step (9) ~ Step (12) will build the database of music code
with a music list for the next process, the control code
generating process. In this step, the music code will be
checked if a score input has achieved completely.

Fig. 5 Data transmission timing between host controller and target controller

IV.

INTELLIGENT ALGORITHMS

Algorithm 1: The interactive man-machine interface.

As Fig. 4 shows that the host controller has an interactive
man-machine interface for music score input. This function is
achieved by an intelligent program to receive music code
including the notes and beats of the treble (right hand code)
and bass (left hand code) stave through the input from a
touching screen. Emulating the piano playing, user inputs the
music score and codes the music code only by touching the
console of the host controller while the simulated piano
keyboard is displaying on the screen of controller.
Without caring about the beat code, the program can
calculate the rapping time and convert it automatically to the
beat code by counting the touching time of the screen. Also, for
easy using, the input program can retrieve the existed music
code file to revise or modify the music code. The program
works can describe step by step as the flow chart which shows
in Fig. 6.

Algorithm 2: The scheduling and control code generator.

In the next, let me state about the control code generating.
While the music code of a song has been built, the music
database also has been founded. The control code generating
program is responsible for calculating the hands moved
position and the fingers rotated angle to make the finger
rapping the right key with minimizing movement while a
music note is translated to a control code. The intelligent
algorithm will assign the rapping mission to the most adequate
finger for the current hand position. Come from the mechanical
structure of the two palms, seeing the picture of the palms and
fingers in Fig. 7, the palms have different span for playing
white key and black key even while the palm position change.

Fig. 7 The span of palms and fingers

Table 3 gives the various situations about the span for white
and black key that the finger can rap the piano key while each
palm move to different position, namely state 1 to state 7. It
should be noted that Table 3 has included all situations though
it only gives the 7 state situations. It is no problem since the
situations will repeat again for each octave.
TABLE 3 SPAN OF PALMS FOR FINGER RAPPING
*1

Fig. 6 The algorithm flow chart of the interactive man-machine interface

Generally, following the flow chart of Fig. 6, the whole

function of the music score input program can be stated as
follows:
Function 1; Music code input: Step (1) ~ Step (3) deal with
the functions about the touching screen sensing and
existed code file retrieving.
Function 2; Code recognition and compatibility check: Step
(4) ~ Step (6) treat the rapping time counting, beat code
converting, coding format checking, and treble and bass

Span of left palm (key)

White key
Black key

Span of right palm(key)

White key
Black key

State
(base)

*2

*3

*4

*2

*3

*4

*2

*3

*4

*2

*3

*4

1 (4)
2 (5)
3 (6)
4 (7)
5 (1)
6 (2)
7 (3)

10
10
10
10
10
10
10

-5
-5
-5
-5
-5
-5
-5

+4
+4
+4
+4
+4
+4
+4

7
7
6
7
7
6
6

-3
-4
-3
-3
-4
-3
-3

+3
+2
+2
+3
+2
+2
+2

8
8
8
8
8
8
8

-3
-3
-3
-3
-3
-3
-3

+4
+4
+4
+4
+4
+4
+4

7
6
7
7
6
6
7

-3
-2
-2
-3
-2
-2
-2

+3
+3
+4
+3
+3
+3
+4

Remark: *1: Palm center position aligning to white key for a cycle of octave
*2: Span of palm for finger rapping
*3: maximum offset in left side for finger rapping
*4: maximum offset in right side for finger rapping

As Fig. 7 shown, a music note or piano key can play by

different finger, that will bring the finger scheduling has many
combinations and make it becomes a complicated process.
Based on minimum displacement of hand driving motors, the
control code generating algorithm will program a path for left
palm and right palm, called the base position calculation, to
assure the playing note has be covered with the span of each
palm. The algorithm will produce two paths, two series of base
position, for left hand and right hand. Here, the base position is

1541

corresponding to the palm positioning.

Also, the hand collision protection is a very important issue
for the robot control since the left hand and right hand driving
motors share the same stator track in the piano robot. How to
relieve the mover collision is big load for this algorithm. Fig. 7
may give a hint to relieve the collision by scheduling the
rapped finger skillfully. Hence, the algorithm will check and
adjust the base position again and again to fit the music note
playing and have no hand collision.
The whole function about the control code generating
program can be expressed as the following flow chart in Fig. 8.
The main works of this program can be stated as follows:

finger playing range of the two thumbs while the

aforementioned situation was happening.
Algorithm 3: The commanding code generator.
This algorithm is in charge of the command code
generating to command the target controller. Following the
data transmission format as Table 2, the program will generate
a series of hands position and fingers angle command. The
functions of the algorithm are stated as the following flow chart
in Fig. 10.

Fig. 9 The algorithm flow chart of the commanding code generator

Fig. 8 The algorithm flow chart of the scheduling and control code generator

Function 1; Base position calculation: Step (1) ~ Step (3)

calculate the base position to program the moving paths
for left hand and right hand. In this step, the hand
collision will be checked and relieved roughly.
Function 2; Base position optimization and scheduling:
Adding the sharp and flat notes consideration, Step (4) ~
Step (5) minimize the displacement of two hands by
adjusting the base positions to enhance the playing
performance. The hand collision should be checked again
precisely since the base positions have been changed.
Function 3; Rapping task assignment and error correction:
Based on the path programming, Step (6) ~ Step (8) deal
with the rapping task assignments for the ten fingers.
Sharp and flat notes will be corrected alternatively to
balance the rapping load of fingers. Also, the duplicated
assignment will be check to avoid the finger crashing and
crisscross.
Function 4; Finger rotation angle calculation and finger
crashing check: Step (9) ~ Step (11) achieve the finger
rotation angle calculation and finger crashing check. In
this step, the palm position will be considered with the
finger rapping key to check again for the finger crashing.
It may happen that the fingers crisscross between the left
hand and right hand though the palm doesnt collide yet,
since the fingers span is wide than the palms width. The
finger rapping task will be reassigned by shrinking the

1542

Function 1; Fingers rotated command combination and

rapping timing calculation: Step (12) ~ Step (13) focus on
the finger action control. The target controller is a parallel
control system. Each finger has individual driving system.
Then the robot can play chord music. In this step, the
algorithm will check the chord notes and combine the
fingers action in a control code for same time rapping
requirement. Also, the beat time of every note will be
decoded and the action timing will be precise calculated
in the step.
Function 2; Peripheral initialization, hardware wake up, and
actuators homing: Step (14) will link the host controller to
the target controller and physical plant. The program
should send the initializing codes to the controlled system
and sense the response from the target controller to make
sure the peripheral and physical system are ready.
Function 3; Rest and sustain note check: Step (15) ~ Step (16)
hand out the task of timing check and hand positioning
command calculation. Rest note means the robot doesnt
rap the piano just idle and wait the time passing away
without any mechanical action. The counterpart, sustain
note, denotes the robot continues the rapping but doesnt
change the palm and finger position in the interval of
sustain note. The hands displacement commands are
calculated in this step.
Function 4; Motion command execution: Step (17) ~ Step
(22) are the final action control execution. From the view
of robot control, every music note playing consists with
many mechanical actions which imply the corresponding
actuator control. In this step, the program will precise
calculate the each mechanical action time, including the

finger lifting, palm moving, finger rotating, and finger

rapping, and control the commanding timing. Absolutely,
all the actions should coordinate with the beats and
rhythm of the music score.
V.

PROGRAM EXECUTION

In fact, the above algorithms can be implemented by many

kind of the program language, such as Basic, C, C++, C++
Builder, etc., in this case, the three algorithms of host controller
have been implemented by the C++ Builder. To demonstrate the
algorithms work, the program executed outcome of each
algorithm is shown in the following.
(1) Music score input & music code encoding

VI.

CONCLUSION

In this paper, we focus on the host controller design that is

the intelligent algorithm development including the music
score input, music code encoding, control code generating, and
action commanding to achieve a music playing by the proposed
robot. All of the algorithms of host controller have been
realized by the C++ Builder with the friendly interactive
interface that leads the user easily to input a music score on the
touching screen like playing the piano keyboard. The executing
results by the developed program have been shown as above.
ACKNOWLEDGEMENT
The work was supported by the National Science Council of
the Republic of China under Grant NSC 102-2221-E-159 -009.
REFERENCES
[1] S. Shigeki, O. Toshio, T. Yoshiharu, and K. Ichiro, Autonomic limb
control of the information processing Robot - movement control system
of robot musician WABOT-2, in Journal of the Robotics Society,
Tsukuba, Japan, Vol.3, No.4, pp. 81-95, 1985.
[2] J. Solis, M. Bergamasco, S. Isoda, K. Chida, and A. Takanishi, Learning
to play the flute with an anthropomorphic Robot, in International
Computer Music Conference (ICMC), Miami, Florida, 2004.

Fig. 10 The picture of score input on touching screen of the host controller

[3] See http://www.expo2005.or.jp/tcn/robot/index, last visited Feb. 21, 2013.

[4] See http://world.honda.com/ASIMO/, last visited Feb. 21, 2013.
[5] See http://www.sonyaibo.net/home.htm, last visited Feb. 21, 2013.
[6] See http://www.toyota.co.jp/en/news/07/1206_2.html, last visited Feb. 21,
2013.
[7] G. Weinberg, and S. Driscoll, The design of a perceptual and
improvisational Robotic marimba player, in Proc. of the 16th IEEE
International Conference on Robot & Human Interactive Communication,
pp. 769-774, 2007.

Fig. 11 The generated file of music code after algorithm 1 execution

(2) Action scheduling & control code generating

[8] J. Solis, K. Taniguchi, T. Ninomiya, T. Yamamoto, and A. Takanishi,

Development of Waseda flutist Robot WF-4RIV: implementation of
auditory feedback system, in International Conference on Robotics and
Automation, Japan, pp. 3654-3659, 2008.
[9] C.T. Chao, and J.S. Huang, Implementation of an intelligent automatic
guitar accompaniment System, in 3rd Intelligent Living Technology
Conference, pp. 850-857, 2008.

Fig. 12 The control code generating after algorithm 2 execution

(3) Robot commanding & music playing

[10]J. C. Lin, H. H. Huang, Y. F. Li, J. C. Tai, L. W. Liu, Electronic Piano

Playing Robot, 2010 IEEE International Symposium on Computer,
Communication, Control and Automation, Tainan, Taiwan, pp.353-356.
May 5-7, 2010.
[11]See http://www.nownews.com/2011/10/20/11490-2750982.htm,
visited Feb. 21, 2013.

last

[12]Y.F. Li, L. C. Fan, and K. C. Cheng, FPGA-based distributed control

module design for music playing robot Applied to the anthropomorphic
piano robot control, 2012 The 43rd International Symposium on
Robotics, Taipei, Taiwan, pp.1065-1070. Aug. 29-31, 2012.
[13]Y.F. Li, L.L. Chuang, Controller design for music playing robot
applied to the anthropomorphic piano robot, 2013 The 10th IEEE
International Conference on Power Electronics and Drive Systems,
Kitakyushu, Japan, pp.968- 973, Apr. 22-25, 2013.
[14]Y.F. Li, FPGA-based module design for PM linear motor controlapplied to music playing robot, 2013 IEEE International Symposium on
Industrial Electronics, Taipei, Taiwan, pp.1-6. May 28-31, 2013.
Fig. 13 The picture while the music playing

1543

Powered by TCPDF (www.tcpdf.org)