Robot Gesture Generation from EnvironmentalSounds Using Inter-modality Mapping

Yuya Hattori∗ Hideki Kozima∗∗ Kazunori Komatani∗ Tetsuya Ogata∗ Hiroshi G.Okuno∗∗Graduate School of Informatics, ∗∗National Institute of Information and

Kyoto University, Japan Communication Technology, Japan{yuya, komatani, ogata, okuno}@kuis.kyoto-u.ac.jp xkozima@nict.go.jp

Abstract

We propose a motion generation model inwhich robots presume the sound source of anenvironmental sound and imitate its motion.Sharing environmental sounds between hu-mans and robots enables them to share envi-ronmental information. It is difficult to trans-mit environmental sounds in human-robotcommunications. We approached this prob-lem by focusing on the iconic gestures. Con-cretely, robots presume the motion of thesound source object and map it to the robotmotion. This method enabled robots to im-itate the motion of the sound source usingtheir bodies.

1. IntroductionBased on advances in information technologies, thewidespread use of robots is expected. Robots inter-acting with humans in the real world are supposedto do so using diverse modalities as humans do. Inparticular, we focus on various kinds of non-verbalsounds in our surroundings such as a door-openingsound and the cry of an animal, called “environmen-tal sounds.” Since environmental sounds are very im-portant clues to understanding the surroundings, itis very useful to share the information of the environ-mental sounds with humans and robots. Ishihara etal. developed a sound-to-onomatopoeia translationmethod for such interactions (Ishihara et al., 2004),for example. In this paper, we focus on gestures inorder to interact about environmental sounds.

2. Motion Generation using Inter-modality Mapping

2.1 Definition of Inter-modality MappingHumans ordinarily perceive events in the real worldas the stimuli of multiple modalities such as visionand audition. Accordingly humans can express thestimuli in each modality. In the real environment, in-formation from all modalities is not obtained prop-erly. Optical information can have occlusions, forexample. In such cases, humans can complementlosses from properly obtained information. We de-fined inter-modality mapping as mapping frominformation of properly obtained modalities to theinformation of modalities not obtained.

In this paper, we focus on mapping from inputsounds to motions. It is because visual and audi-tory information are remarkably important in hu-mans’ communications. It has been reported thatchildren often use onomatopoeia and a gesture simul-taneously and that the linkup of them is importantfor the development of multi-modality interactions(Werner and Kaplan, 1963).

2.2 Iconic Gesture GenerationWe aim to generate motions expressing environmen-tal sounds by imitating the motions of the soundsource objects. It is because the kinds of environ-mental sounds are closely related to the motion of thesound source. It is known as iconic gestures froman observer viewpoint to imitate the motion ofobjects (McNeill, 1992). Iconic gesture means imi-tating concrete circumstances or events using one’sbody.

In our model, robots memorize the motion of thesound source, which is captured by their camera,when they listen to a sound with looking the soundsource. After learning the correspondences betweenthe sound and the motion, they can imitate the mo-tion of the sound source when they listen to a soundwithout looking at the sound source. Namely, theycan perform iconic gestures.

3. System Implementation3.1 Tasks and System OverviewIn order to make iconic gestures, the system learnsconnections between the sound and the motion whena sound occurs. In interaction phase, the systemgenerates a motion when a sound is input.

We use the robot “Keepon” for implementationand experiments in this study. It is a creature-likerobot developed at NICT mainly for communicativeexperiments with infants. Its body is approximately12 cm high.

3.2 Learning ProcessIf object velocity, or the norm of the optical flowvector, is higher than the threshold when a sound isinput, the system interprets the motion of the objectas the reason why the sound occurred. The systemmemorizes the pair of the spectrogram of the sep-arated sound and the sequence of the optical flowvectors into the mapping memory. This process isshown in Fig. 1. The detailed process in each mod-ule is explained in the following paragraphs.

Figure 1: Motion and Sound Learning

Optical Flow Extraction Optical flows are con-stantly extracted from camera images. We adoptedthe block matching method for optical flow. Assum-ing that the camera captures only a sound source ob-ject, the system averages the flows of all of the blocks

139

Berthouze, L., Kaplan, F., Kozima, H., Yano, H., Konczak, J., Metta, G., Nadel, J., Sandini, G., Stojanov, G. and Balkenius, C. (Eds.)Proceedings of the Fifth International Workshop on Epigenetic Robotics: Modeling Cognitive Development in Robotic Systems

Lund University Cognitive Studies, 123. ISBN 91-974741-4-2

on every frame. The averaged vector is regarded asthe velocity vector of the sound source object on theframe.

Sound Separation In most physical phenomena,one sound is equivalent to one peak in the powerenvelope of an input audio signal. Therefore, eachevent is separated by extracting each peak. Weadopted the separation method developed by Ishi-hara et al. (Ishihara et al., 2004). Individual soundseparation is done by extracting each local maximumof power envelope which is not regarded as a part ofanother local maximum.

Learning Correspondences between Soundsand Motions Not only the motion during asound, but also the motions right before and rightafter the sound are also parts of the motion whichgenerates the sound. Therefore they are recordedinto the mapping memory. The time span Ti of thei-th sound stored in the memory is defined as follows:

Ti = [ mint{ t | Vs < |F (t)|, t ≤ minTs} ,

maxt{ t | Vs < |F (t)|, max Ts ≤ t} ]

where F (t) is the averaged optical flow vector at timet, Vs is the threshold by which the system decideswhether the object is moving or not, and Ts is thetime span that corresponds to the time span of thesound, which is given by the sound separation mod-ule.

3.3 Sound-to-Motion TranslationFig. 2 illustrates the process of a translation fromsounds to robot motions. The auditory distance frominput sound to each sound in the mapping memoryis computed. The motion learned as a pair with thesound which gives the smallest distance is picked up.Then the robot imitates the motion. If the smallestdistance is larger than the threshold, the system doesnothing. Motion generation uses the following mod-ules in addition to the above-mentioned modules.

Figure 2: Sound-to-Motion Translation

Auditory Distance Computation Auditorydistances are computed by dynamic time warp-ing (DTW) with mel filter bank output. DTWhas been reported as one of the most effectivemethods in recognition of environmental sounds(Cowling and Sitte, 2003).We adopt mel filter bankoutput, which is made to adapt to human perception,as a feature of each frame in DTW.

Motion Generation In imitating the motion ofthe object, the body position X(t′) at the elapsedtime t′ from beginning of the gesture is defined as

X(t′) = C∑t′

s=1Fi(s)

for the recorded optical flow sequence Fi(t) and aconstant C. Because the motion X(t′) is generatedon a 2-dimensional plane, 2 degrees of freedom, which

can reproduce the generated motion, must have beenselected from all of the degrees of freedom of therobot beforehand.

4. ExperimentsThe motion generation system was implemented onKeepon. We made Keepon learn 4 kinds of sounds,such as the sound of rubbing uneven plastic piecestogether and the sound of striking metal boards ver-tically. After the learning, we made Keepon gener-ate the imitation motion by giving a sound withoutlooking at the sound source. Based on the rubbingsound, Keepon made a right rotation, then a leftrotation as shown in Fig. 3 (a). According to thestriking sound, Keepon executed an anteflexion thena retroflexion as Fig. 3 (b). The coordinates of thesemotions are shown as Fig. 4. A video of these ex-periments can be viewed at http://winnie.kuis.kyoto-u.ac.jp/members/yuya/demo.avi.

Figure 3: Generated Motionsat the Experiment

Figure 4: Coordinates ofthe Generated Motions

5. ConclusionIn this paper, we proposed an interaction model,in which robots make motions to communicatethe sound. For this purpose, we developed aniconic gestures generation system on the real robotKeepon. The mappings from motions to soundsand motion generation for unknown sounds arethe subjects of our future work. We are alsointerested in the relation with bouba-kiki effect(Ramachandran and Hubbard, 2001).

Acknowledgments This study was partially supportedby the Grant-in-Aid for Scientific Research from the JapaneseMinistry of Education, Science, Sports and Culture, and bythe JPSP 21st Century COE Program.

ReferencesCowling, M. and Sitte, R. (2003). Comparison of techniques

for environmental sound recognition. Pattern Recogn.Lett., 24(15):2895–2907.

Ishihara, K., Nakatani, T., Ogata, T., and Okuno, H. G.(2004). Automatic sound-imitation word recognitionfrom environmental sounds focusing on ambiguity prob-lem in determining phonemes. PRICAI–2004.

McNeill, D. (1992). HAND AND MIND: What GesturesReveal about Thought. Univ. of Cicago Pr, Chicago.

Ramachandran, V. S. and Hubbard, E. M. (2001). Synaes-thesia — a window into perception, thought and lan-guage. Journal of Consciousness Studies, 8(12):3–34.

Werner, H. and Kaplan, B. (1963). Symbol Formation: AnOrganismic-Developmental Approach to the Psychologyof Language. John Wiley and Sons, New York.

140