JackIn Head : Immersive Experience with First PersonOmnidirectional Video

Shunichi KasaharaSony CSL, The University of

TokyoShinagawa, Tokyo, Japankasahara@csl.sony.co.jp

Shohei NagaiThe University of TokyoShinagawa, Tokyo, Japan

shohei.nagai14@gmail.com

Jun RekimotoSony CSL, The University of

TokyoShinagawa, Tokyo, Japan

rekimoto@acm.org

ABSTRACTFully recording and sharing an immersive experience is oneof the ultimate goals of media technology. As extensive tech-nical evolution, omnidirectional video is one of promisingmedia to capture an immersive experience. First person om-nidirectional video provide an unique experience of worldthrough someone elses perspective. This will bring variousapplications such as entertainment, sports viewing, educationand simulation training, newscasting and therapy. However,difficulties in wearable camera design and cybersickness in-duced by shaky video has been obstacle to explore applica-tions of first person omnidirectional video. We introduce thedesign and implementation of ”JackIn Head” a system includ-ing a wearable omnidirectional camera that provide 360 de-grees spherical images of the first person’s surrounding en-vironment, and image stabilization to improve cybersickness.We’ve performe the series of workshops to explore user ex-perience and applications in actual use cases such as virtualtravel and virtual sports.

Author KeywordsFirst person; Omnidirectional video; Wearable camera;Image processing; immersive;

ACM Classification KeywordsH.5.1 Information Interfaces and Presentation (I.7): Multime-dia Information Systems- video; H.5.2 Information Interfacesand Presentation (I.7): User Interfaces - interaction styles,evaluation, User-centered design

INTRODUCTIONRecording and sharing own experience completely has beenan ultimate goal of media technology. As envisioned by theSF movie Brainstorm [3], recording and sharing immersivesensations will allow us to virtually experience what we couldnot experience by our own. Small wearable wide-angle cam-eras such as the GoPro1 enables to record a video of the ones1GoPro http://gopro.com/

Paste the appropriate copyright statement here. ACM now supports three differentcopyright statements:• ACM copyright: ACM holds the copyright on the work. This is the historical ap-proach.• License: The author(s) retain copyright, but ACM receives an exclusive publicationlicense.• Open Access: The author(s) wish to pay for the work to be open access. The addi-tional fee must be paid to ACM.This text field is large enough to hold the appropriate release statement assuming it issingle spaced.Every submission will be assigned their own unique DOI string to be included here.

First-person omnidirectional video

The first-person The spectator

Fir

Figure 1. JackIn Head Overview, immersive experience record and play-back architecture with a wearable omnidirectional camera.

perspective and then be shared with others. Whereas ordinarycameras capture only a limited field of view in front of them,omnidirectional video cameras are able to capture the entirespace. Such a Omni-directional video (ODV) provides spec-tator experience with impressive and immersive sensations asif they were present in [1, 2].

We introduce JackIn Head system including a wearable om-nidirectional camera and image processing for stabilizingFODV (Fig 1). We embed multiple cameras within a form-factor of a head band for comfortable wearability. Head-motion effect of omnidirectional images is compensated byanalyzing image rotation, and is recorded or transmitted toanother user. Spectators can look around the stabilized sharedscene of the first person.

We perform series of demos and workshops with many ac-tivities such as sports to prove actual benefit of FODV andcollect insight from participants

JACKIN HEADJackIn Head system includes a wearable headgear (Figure 3)with six camerasto capture omnidirectional video. The headgear was designed with two design considerations. One islower center of balance, which allows users to move theirbody and head dynamically. The high center of balance isdangerous for even usual activity. Another is that the cap-tured environment should be close to the first person view-point. This provides a realistic sensation of the first personperspective.

Video images from these cameras are stitched together intoa spherical omnidirectional video. Then image processing

tracking point p(t+1)

tracking point p(t)

the estimated rotation Q’(t+1) previous rotation Q(t)

Projected tracking point P(t)

Projected tracking point P(t+1)

estimated tracking point P(t)dQ(t+1)

projection

pn to Pn

transform image

by Q’(t+1)

the stabilized video sequence I’(t+1)the input video sequence I(t+1)

Figure 2. Image processing procedure for estimation of head rotation and stabilization.

(a) (b)

(c) (d)Figure 3. Variation of JackIn Head gear. (a) and (b) are lighter versionwith usb cameras. (c) and (d) are HD version with six HD cameras.

(a) (b) (c)

Figure 4. Actual use of the headgear type-d in several situation. (a)playing the squash, (b) track, (c) giant swing on the high bar

for stabilization is an important part of this system. Unlikea handheld camera, the first person video from the wearablecamera contains intense movement due to users physical mo-tion such as nodding in the conversation, turning head in thewalking, and sports activity, and these are unconscious andinevitable. This will produce serious bad experience for spec-tators. Rotational motion in omnidirectional video is esti-mated at each frame by tracking image feature points (Fig. 2).By performing inverted rotation for omnidirectional imagerendering, rotational motion can be canceled and the videoimage can be seen spatially registered in spherical environ-ment.

The stabilized omnidirectional video and the head ego-motion information of video are streamed to the viewer de-vice on the spectator side application. In the spectator side,there are various options for viewing device, example setup

(a) (b)

Figure 5. (a) : Example view of first person omnidirectional video ofthe paraglider. (b) HMD playback with a harnessfor practice greatlyin-crease realistic feeling and excitement.

is a HMD with head motion tracking (such as Oculus rift 2)that the spectator uses to look around the first person visualenvironment.

USECASEWe performed series of demo and workshop with various ac-tivities, a virtual travel, virtual gymnastics and virtual flyingexperience of paraglider. Our proposed headgear design wasfeasible for actual video shooting in active motion usecase(Fig. 4).

So far we’ve shoot many active sport FOVs includeing,squash, gymnastics, paraglider, hurdle race, bungee jumping,aerial skiing and martial arts. We’ve focus sport activities be-cause we found these physical activity will produce a lot ofinsight and challenges for system and user experience (Fig.5).

REFERENCES1. Kasahara, S., Nagai, S., and Rekimoto, J. Livesphere:

Immersive experience sharing with 360 degreeshead-mounted cameras. In Proceedings of the AdjunctPublication of the 27th Annual ACM Symposium on UserInterface Software and Technology, UIST’14 Adjunct,ACM (New York, NY, USA, 2014), 61–62.

2. Nagai, S., Kasahara, S., and Rekimoto, J. Livesphere:Sharing the surrounding visual environment forimmersive experience in remote collaboration. InProceedings of the Ninth International Conference onTangible, Embedded, and Embodied Interaction, TEI ’15,ACM (New York, NY, USA, 2015), 113–116.

3. Trumbull, D. Brainstorm, 1983.2Oculus rift DK-2 : http://www.oculus.com/