Jean Baptiste Thiebaut, Samer Abdallah, Andrew Robertson, Nick Bryan Kinns, Mark Plumbley

Proc. of the 8th Int. Conf. on New Interfaces for Musical Ex-pression(NIME '08)

, pp. 215-217, Genova, Italy, June 2008

Real Time Gesture Learning and Recognition: Towards Automatic Cat-

egorization

Introduction • Domain

– Gesture as an integral part of music performance– Important issue to address is to categorize and recognize gestures

• Related works– Research by Cadoz and Wanderley [3]

• Stress of the importance of gesture classification and recognition– Research by Cadoz [2]

• Emphasis of the importance of haptic feedback for the design of interactive in-terface for sound production: the physical feedback given by the intermediary device

• Creation of memorizable gestures, and audio feedback rendered by the inter-face

– Kela et al. [5] • Using accelerometers for multi modal activities

• Contribution– Real-time classification for use in music performances

• Other approaches → pre-defined classification

Introduction • Contribution

– Development of learning of specific gestures – Creation of a database of recognizable gestures that shared between

performers

• Two methods– Recognition of a fixed length gesture– Dynamic and unsupervised recognition model to handle various length

gestures

Supervised Method With Haptic Feed-back

• Wii remote controller– Popular and pervasive device that detects 3-dimensional movements – Transmission of signals of the accelerometers via Bluetooth– Using Max/MSP to decode the transmissions from the controller

• Developed by Masayuki Akamatsu– Sampling rate of 50 Hz – Latency produced by bluetooth of approximate 50ms

• Example of data over a fixed period of time

– Vibration as feedback to the user when a gesture is recognized– Categorization of a gesture in real time with supervision

Euclidean distance for classification• Fragments of the incoming data

– L → length of samples depends on the sampling rate• E.g. 6 (x, y, z) triplets at 50 Hz for a gesture → lasting 120ms

– Capture of a gesture • Pressing a button (‘A’) on the controller at the end of the movement

• Variables– Vr : reference gesture, 3L-dimensional vector– Vi : similar vector constructed from the last L samples of the input signal

– 3 차원 벡터 내적의 합 (L 개의 샘플로부터 계산 )

Cosine similarity for classification• Using cosine of the angle

– Calculation of cosine between reference vector and input vector– Taking the dot product and dividing by the norms of the two vectors

• Discussion– Cosine method : detection of all 45 instances– Distance method : detection of 44 among 45 instances

Unsupervised method using information dynamics

• Requirements of above supervised method – Reference gesture with its label– Indication of the particular time point, relative to the reference

• Mark indicating the ‘perceptual center’ of the gesture

• Predictive information– Heart of gesture recognition → perception of discrete and punctual

events in a continuous signal– Predictive information rate of the signal as processed by the observer

• Hypothetical observer to be engaged in a continuous process of trying to pre-dict the future evolution of a signal as it unfolds

• Assignments of probabilities to the various possible future developments

Unsupervised method using information dynamics

• Predictive information– Statistical regularities in the signal

• E.g.) smoothness or any typical or repeated behavior for better predictions

– Kullback-Leibler divergence • a measure of distance between probability distributions• Calculation of the divergence between P(Y |Z =z) and P(Y |Z =z,X =x)

– where Z =z and X = x denote the propositions that past and present variables respec-tively were observed to have particular values z and x

– Many forms of predictive information rate• In some cases be relatively flat, while in others, more peaky or bursty• Arrival in concentrated ‘packets’ interspersed by longer periods of relatively

low predictive information• Identification of the ‘packets’ of information as the ‘events’

Unsupervised method using information dynamics

• HMMbased implementation– Implementation of a version of hypothetical observer

• Using a relatively simple predictive model (a Markov chain)• A vector with N = 3L components (L : consecutive samples)• The sequence of vectors → the continuous valued observation sequence• Hidden Markov model (HMM) with Gaussian state-conditional distributions• K possible states

– Training of parameters of HMM • Transition matrix and the mean and covariance for each of the K states• Using a variant of the Baum-Welch algorithm• After training, the most likely sequence of hidden states is inferred using the

Viterbi algorithm

– Variation in predictive information rate over time• Event detection by picking all transitions with a predictive information greater

than a fixed threshold

Conclusion• Development of efficient tools for real-time gesture recogni-

tion– Using Nintendo Wii remote– Both supervised and unsupervised algorithms to deal with signals

• Template matching system is based on well-known template matching methods• HMM based system uses novel information-theoretic criteria to enable unsu-

pervised identification of an initially unknown number of gestures

• Explicit probabilistic formulation– Handling the detection latency problem by predicting the future motion

of the controller– Estimating how accurate this prediction might be

Gonzalo Bailador, Daniel Roggen, Gerhard Tröster, and Gracián Triviño

Proc. of the ICST 2nd int. conf. on Body area networks table of contents,

Florence, Italy, 2007

Real time gesture recognition using Continuous Time Recurrent Neural

Networks

INTRODUCTION• Motivation

– Appearance of intelligent devices integrating in clothing or objects• “On the body” wearable computers

– Ideal location to detect important information about the “state” of the user

• Such as his position, his activities or gestures or even his social interactions

• Fields for context awareness– Personal health assistant of the user (e.g. by monitoring the physical ac-

tivity)– Delivering context-based information– Human-computer interactions– Detect social interactions– Insight into affective disorders or depression

• Challenge of gesture recognition in wearable computing– Good recognition accuracy on miniature wearable devices– Offering long battery life, and consequently limited computational power

INTRODUCTION• Technologies for gesture recognition

– Hidden Markov models (predominant approach)– Dynamic programming– Neural networks– Fuzzy expert systems

• Analyzing complex features of the signal like the doppler spectrum• Difficult to classify two different gestures with similar values for these features