Jürgen Scheible, Timo Ojala, and Paul CoultonProceeding of the 16th ACM international conference on Mul-

timedia

2010. 5. 29.

MobiToss: A novel gesture based interface for creating and sharing mobile multimedia art

on large public displays

2

Contents• Introduction

• MobiToss System

• User Evaluation– UI Issues– Motivation for participation– Immersion and perception issues– Commercial aspects– Collaboration aspects– Advertisement

• Conclusion

3

Introduction

• Creating and sharing mobile multimedia art– Standard interaction devices (mobile phone, large public display, web)– Interaction styles (mobile phone UI, gesture control)– Multimedia processing techniques

• MobiToss System– Mobile phone equipped with built-in accelerometer sensors– Throwing the clip onto a large public display for instant viewing– Manipulation of video on the public display by tilting the phone

• Related works– Accelerometer-based gesture control for personalization of gestures (Kela

et al.)– Throwing digital media in digital documents on walls (Bertelsen et al.)

• By throwing a ball at the wall or bouncing it onto the floor– Wii game console for interacting with games on a large screen– System to connect a mobile phone to large public game screens over

Bluetooth (Vajk et al.)

4

Introduction

• Related works– Mobile phones with attached miniature accelerometers for gesture based

group interaction with public displays (SQEAK)– Use of a mobile device as an artistic tool in public spaces (Clay et al.)

• Mobile music work 'China Gates‘ using wearable computing as an art tool

5

MobiToss System• Throwing a video to the large screen

– Capturing a photo or video with a personal mobile phone – Throwing it onto the large public display by making a throwing gesture– ‘Recognized’ throwing → matching sound provided by the phone as feed-

back

6

MobiToss System• Manipulation of the appearance of the video or photo on the

screen– Moving the phone up, down, left or right– Different video effects and graphical filters

• Matrix, kaleidoscope, spiral distortion and color saturation– Matrix effect → multiplication of the video over a number of rows and

columns• Tilting the phone forward or backward

– Spiral distortion effect → rotation around the centre of the screen

7

MobiToss System• Produced artifact

– Recording a 30-second video clip • Encoded with a commercial pop song, a logo and a web address as advertise-

ment– Uploading encoded mash-up clip to the users’ phone as a personal re-

ward– Uploading mash-up clip to a dedicated community website– Collection of the clips and sharing them with friends– Configuration for creating event and location based collections

8

MobiToss System• Suitable reasons of the mobile phone for MobiToss

– Personal trusted device for any proprietary application specific devices– Phones with built-in accelerometer sensors on the market– Providing functionality to create content by the users– Reliable channel for delivering confidential user specific content back to

the user• such as the music video including logos and web links as advertisement.

• Four components for MobiToss system– Symbian client application running on a mobile phone with built-in ac-

celerometer– Server running on a PC for creating the clips– Large public display showing the video– Dedicated community website for collecting the clips

9

MobiToss System

10

MobiToss System• Client

– Implementation in Python for S60 built-in accelerometer sensor data– Shooting of photos and videos by a single application– Uploading/downloading of content– Controlling the video manipulation on the server– Communication with the server using the HTTP protocol and sockets over

WLAN (IEEE 802.11) connection

• State diagram of the system– Choice from a menu to either take a photo or to record a video– Automatic invocation of camera’s viewfinder– Setting recording period by pressing the start and end buttons (A)– Sending video to the server– Selecting video effect for manipulating the video on the public display(B)

11

MobiToss System• State diagram of the system

– Notification to the user after sending of the video• Playing a sound and by displaying a ‘Ready to throw’ message

– Throwing motion of the phone → acceleration exceeding certain thresh-olds

• Sends an HTTP request to the server to make the video visible (‘video ON sig-nal’).

– Switching client immediately into gesture control mode• Using socket communication to transmit control signals for real-time manipula-

tion of the video

12

MobiToss System• Server

– Implementation in Max/MSP Jitter – Handling all video effects and adding brand icons and web links to the

video– Including ffmpeg software for adding the music into the clip– Python scripts for connectivity and server management

13

User Evaluation• Evaluation of MobiToss with real users

– Social event of an international conference– Goal

• First impressions on how the system is perceived by the users • Identification of the strengths and weaknesses as input for the next design cy-

cle– Settings

• Public display located on 2-5 meters in front of the users• Nokia N95 phone containing the client application• 25 people, most of them researchers on mobile multimedia• Feedback was collected from 11 persons via video interviews

• User’s comment– Question: How was your experience using MobiToss?– “The throwing is the best, I like it”. – “It is very reactive, I didn’t expect this reaction, it works well.”– “It is pretty nice tool for a social event like this because you can capture

the event live and you can have some fun. I like that it meshes the tech-nological stuff with the artistic effects”

– “It’s fun to recognize other people in the video.”

14

User Evaluation• UI issues

– Different people do their throwing movement in many different ways– Unsuccessful detection of all users’ throwing gesture– User feedback

• “Have on the phone screen an animated preview of the possible effects to see what they do”

• “Reduce the number of video effects, probably have only 2-3 very clear ones to explore the system, and maybe you are more immersed.”

– Need for careful UI and more clear relationship between effect and inter-action

• Motivation for participation– Successful tool for artistic expression

• “Having the feeling I can produce my own pieces of art”;• “If I could impress other people with doing the thing, than I would be ready to

pay e.g. 50 cent”• “I would put the faces of my friends and my pet there.”• “To get the music piece together with the ready art piece sent your phone is

nice”. – Need to get something out of it when using MobiToss

15

User Evaluation• Immersion and perception issues

– Feedback• “I didn't really feel immersed. For me it would mean that I am attracted to the

screen while I am interacting, so that I have the feeling that even I can contrib-ute to the evolving piece of Art”

• “It didn't have as much meaning as it would have if you would be able to iden-tify things more clearly in the video“

• “One has to try it out few times to figure it out.” – Indication of need for careful redesign of the video effects

• Commercial aspects– Paying money for downloading the client application

• “It depends, if I download it once and use as much as I want, then it is ok for me to pay. But if you would have to pay every time you go to a party, or if its time based payment, then probably not.”

• “It would be better if you can customize the filters and effects and give the people options to buy them separately. It would depend on the quality and how artistic they are on how much they cost.”

– Paying a small amount and need for exploring various payment models

16

User Evaluation• Collaboration aspects

– Participants’ feedback in favor of having more than one person to throw their video

• “Maybe you could have a competition with a split screen and you can compare the two players outcome in real-time.”

• “When several people throw their video - depending on the starting time of throwing - they can collide and depending on the crash a new thing is created.”

– Competition and group play → enriching for MobiToss

• Advertisement– Encoding branding content as well as music into the user-generated mo-

bile multimedia art clip • Phones and public sharing website.

– Great opportunities for placing advertisements

17

Conclusion• Summary

– Capturing and throwing mobile content onto a large screen– Manipulating it with gesture control into an art piece as a fun activity– Concepts of “me the artist” and “my art piece creation” – Community websites such as YouTube, Flickr or SeeMe TV– Need for better tools to create, to produce, and to participate– New types of collaborative tools for creating novel forms of input and

participation to community websites, possibly coupled with commercial content

• Future work – Further development for intended full-scale experience

• E.g. More balanced set of video effects, adding group interaction and a more intuitive UI

– Expansion in different ways• E.g. collaborative tool in meeting or class rooms• Everyone can just throw their content to the large screen and control it from

their own seat, or into a remote browser for photo, video or other content.