RecoFit: Using a Wearable Sensor to Find, Recognize, and Count Repetitive Exercises

RecoFit: Using a Wearable Sensor to Find, Recognize, and Count Repetitive Exercises + CHI 2014- Dan Morris et al./ 맹욱재x 2016 Spring

UX Lab, Graduate School of Convergence Science and Technology, Seoul National Univ. 2

ABSTRACT

해결책 RecoFit: 관성 센서로 반복 운동 ( 근력 , 맨몸 ) 을 자동으로 추적하는 팔에 차는 장치목적 사용자가 운동 중 조작 / 운동 종류 선택 없이 , 실시간 , 운동 후 피드백 제공방법 1) 분할 (segmenting) 운동과 임시 멈춤 구간 구분

2) 인지 (recognizing) 어떤 운동을 하고 있는지 3) ( 횟수 ) 계산 counting 반복 운동

평가 최종 시스템 평가를 위해 114 명 참가자의 146 세션 training data 를 교차검증(cross-validation)

결과 정확도운동 / 비운동 구분 precision > 95% recall > 95%

운동 종류 구분 (순환 운동 구성 수 )

99% (4) 98% (7) 96% (13)

운동 횟수 측정 93%


INTRO

규칙적인 운동의 여러 장점이 있지만 [11], 꾸준히 지속하기 어렵다 .[20]-> 대부분 권장 운동량을 채우지 못한다 .[5] => 만보계로 걸음수를 자동 트래킹하면 더 걷게 된다 [4]

https://www.flickr.com/photos/neoadam/15837373658https://www.youtube.com/watch?v=4LpYyULx2T4

https://commons.wikimedia.org/wiki/

운동 데이터 전송

만보계 콘솔 액세서리 유산소 운동 기계

실내 운동 + TV걷기 & 뛰기

기회를 인식 가전제품 회사가 신체 활동 측정 기기를 출시했다 .


문제점상용화 , 보편화된 기기들이 놓친 2 가지 운동 영역1) 근력 운동 (weight training) 2) 맨몸 운동 (calisthenics) 일부 사용자에겐 생활 방식 , 취향으로 인해 근력 운동이 유산소 운동보다 지속하기 좋음균형 잡힌 운동을 위해 근력운동은 필수며 , 미 질병관리본부의 성인 최소 권장량은 주 2 회 [6]권장량에 대한 순응도 (compliance) 가 유산소 운동량보다 적음 [5]

INTRO

calisthenics : 기구 없이 하는 근력 운동 ex) 윗몸일으키키 , 팔굽혀펴기 , 팔벌려높이 뛰기Fitbit, Fuelband 가 운동중 사용되며 , 전체 활동에 대한 칼로리 소모량을 제공하지만횟수 , 세트수 , 시간 등의 근력 운동과 관련된 high-fidelity 정보를 주는 장치는 없음카메라 같은 센서는 동작의 다양성이나 근력 운동의 복잡성을 처리하기 어려움

웨어러블로 근력 운동을 자동 트래킹해서GPS watch 로 달리기를 시작할 때 “ set it and forget it” 된 것처럼근력 운동 중의 조작 없이 실시간 , 운동후 분석 피드백을 제공함

목적


운동 분석

REALTED WORK

RecoFit VS What can an arm holster worn smart phone do for activity recognition?smart phone worn on

arm RecoFit

공통점 1) segmentation, recognition, counting 로 나눠 접근 2) segmentation using autocorrelation features

# of participant 7 104thresholds for segmentation heuristic learned

sensor placement variation X

dimensionality reduction

orientation-invariant repetition counting X false peak rejection

real-time X O

결과segmentation 85% > 95%recognition(# of exercises)

94% (10)subject-independent training 96%(13)


myHealthAssistant

Reference, 나눔명조 OTF, Regular, 15pt

운동 분석

REALTED WORK

myHealthAssistant RecoFit

sensor(position) three accelerometers (hand, arm, leg)

inertia sensor(arm)

classifierBayesian classifier

trained on the mean and variance on each accelerometer axis

HMM, SVM

countingcombination of autocorrelation-based period estimation and peak counting on one of the

accelerometer axes

결과segmentation X > 95%recognition(# of exercises)

92% (13)subject-specific training 96%(13)

To handle non-axis- aligned movements and more complex temporal patterns (e.g. secondary peaks within repetitions, preparatory movements) that are common in natural exercise behavior


Significant contribution: exercise segmentation, finding exercise amidst periods of non-exercise. gesture recognition is similar problem. “gesture spotting”

https://upload.wikimedia.org/wikipedia/commons

Gesture Spotting

REALTED WORK

Autocorrelations and Periodicity

key contributions: autocorrelation function to find regions of self-similar, repetitive exercise. highly periodic signalsex) tracking the pitch of a musical signal, finding abnormalities in EKG signals


Exercise looks very similar to non-exercise

WHY IS AUTOMATIC EXERCISE ANALYSIS HARD?

The most challenging problem: actually exercising time 10% ~ 100% of workout session. Between exercises, one walks around the workout space, socializes, stretches, rests, drinks water, selects and retrieves equipment, etc. Distinction obvious to a human observing, but to a wearable sensor much less clear

Magnitude alone is rarely informativemost exercises are quite slowly, strive to avoid jerky movements - high acceleration.Amplitude of acceleration during exercise consistent with that during non- exercisenon-exercise stretches > the magnitude of most exercises

“easy cases” high-velocity exercises like jumping jacksmotion magnitude > typical non-exercise magnitude. exception

“normal cases”motion magnitude = typical non-exercise magnitude.




Limit of GyroscopesSlow pushups almost no translation of an arm-worn sensorslow rotation of a sensor = the noise of gyroscopes.

Primary observable phenomenon - not energy on any sensed axis, but repetitive change in the gravity axis by accelerometer, shoulder presses, non-rotational exercise:not observed at all

This fundamental diversity characterizing exercises motivates machine learning approach to segmentation

https://upload.wikimedia.org/wikipedia/commons/e/e2/3D_Gyroscope.png




exercise is typically more periodic(repetitive) than non-exercise-> features; autocorrelation derive metrics of repetitiveness.





Another challenge: walking is the most common non-exercise activity performed during a workout. Walking is extremely periodic, almost impossible to heuristically describe systematic differences between walking and exercise-> machine learning to segmentation


dynamic stretching – repetitive to loosen joints or muscles, not intended as exercises per set - is quite common during a workout. tremendous challenge to robust exercise segmentation: separating “exercise” from “dynamic stretching” is almost a question of semantics, but one that significantly impacts UX. it is extremely rare for an individual to consistently perform the same dynamic stretching movement – without changing orientation – for more than a few seconds, which supports our use of self-similarity as a core of our feature set, and motivates the temporal smoothing approach




robust segmentation critical to the practicality of automatic exercise analysis. At the highest level, false positives (when the system tracks an exercise that the user did not actually perform) or false negatives (when the system fails to “credit” the user for an exercise) are potentially disastrous from UX perspective.

when segmentation is “correct”, the boundaries need to be precise to enable robust performance at subsequent stages of our pipeline. reliable counting relies heavily on accurate segmentation to ignore preparatory and post-exercise movements, such as lying down to perform pushups, or putting weights down after biceps curls.



Variability in form

WHY IS AUTOMATIC EXERCISE ANALYSIS HARD?Aim to require no user-specific training, variability in users’ interpretations of exercise descriptions, and their ability to consistently execute a particular form, has a tremendous impact on recognition accuracy

Pushups, consistent definition to most potential users, exhibits wide variation in arm posture, pace of repetition, the temporal “shape” of the movement. And less familiar exercises often exhibit an entirely more challenging level of variability, where users interpret the fundamental form of the exercise differently.

End-users have some familiarity with the available exercises, not typically watch a proscriptive video or have access to a coach who would refine their form-> no way to address the problem of variation in form other than large-scale training data collection with enough flexibility to elicit such variation. users in both our training data collection and our evaluation study were given instructions

“reasonable familiarity”, but allow enough interpretation to elicit natural variation. Instructions contained an illustrative image and a high level description for each exercise, and experimenters did not coach or correct form during data collection


Temporal Irregularities


Simple peak-picking or zero-crossing approach will yield an accurate count

jumping jacks - high-amplitude activities

typical set of squats - “double peak” for each count, 1) autocorrelation-based period estimation2) peak counting

highlight very challenging cases.


Unpredictable device orientation


Form factor challenge: in real world use, an arm-worn sensor naturally rotate differently around different users’ arms, due to preference and natural fit.

“trust” the axis pointing along the arm (ex: watch always has its face pointed out, in a readable orientation), but that the device might rotate arbitrarily around the arm.


Hardware

TRAINING DATA COLLECTION

Right forearm, inertial sensor = 3-axis accelerometer + 3-axis gyroscope. battery, Bluetooth radio, transmitting to PC at 50Hz


“Natural” Environment and Procedure Design

TRAINING DATA COLLECTION

it quickly became clear that early success was the result of “robotic” behavior among training participants: segmentation dropped to precision/recall levels close to 50% when users were in a more natural environment it became clear that real-world deployment required a data set that was both larger and more natural. The practicalities of labeling activities and scaling to over 100 participants prevented us from operating in an actual gym, so we retro-fitted a large lab space to resemble a home gym, with appropriate décor (wallpaper, curtains, etc.), video and audio entertainment under participants’ control, a couch for rest periods, and no computers or experimenters visible to participants.

importance of encouraging natural variability in training data. smaller data set (30 participants), collected in a space-constrained laboratory environment that did not aesthetically resemble a gym, with clear instructions regarding sequencing and form.


Overview of Pipeline

THE RECOFIT SYSTEM

whether the user is exercising

RecoFit’s 6-axis data at 50Hz

Which particular exercise type?

segmentation recognition counting

How many times?


Segmentation

THE RECOFIT SYSTEM


Counting

THE RECOFIT SYSTEM

Counting algorithm depends on the label (e.g. “jumping jacks”), raw accelerometer data corresponding to an exercise. Empirically, gyroscope not helpful for counting.

first compute a set of candidate peaks (local maxima). Sort these peaks based on amplitude and loop through this sorted list,


RESULT


RESULT

Recognition accuracy is assessed in the context of a circuit, and inevitably the choice of circuit affects accuracy. A larger number of activities or high similarity among activities will reduce accuracy.


DISCUSSION AND FUTURE WORK

Intensive Strength-Training and Periodicity Breakdown

Mechanical and Form Factor Considerations

User Experience Considerations


Implications for Generalized Activity Recognition

(1)When analyzing periodic signals, the use of independent learned models for periodicity identification and activity recognition can increase robustness.

(2)Dimensionality reduction can increase robustness to variation in device placement and behavioral orientation.

(3)We provide specific novel features to capture self-similarity for human motion applications, relevant to fitness, pe- dometry, physical therapy, etc.

감사합니다

Healthcare

RecoFit: Using a Wearable Sensor to Find, Recognize, and Count Repetitive Exercises