Upload
hisa
View
33
Download
0
Embed Size (px)
DESCRIPTION
A Framework for Multi-resolution and Multi-touch Systems . 多重解析度及多重觸控互動系統之開發. Presenter: 胡婷婷. Advisor: 洪一平 教授. Outline. Introduction Related Work i -m- Top System i -m- Top SDK Application Conclusion & Future Work. Outline. Introduction Related Work i -m- Top System - PowerPoint PPT Presentation
Citation preview
A Framework for Multi-resolution and Multi-touch Systems
Presenter: 胡婷婷多重解析度及多重觸控互動系統之開發
Advisor: 洪一平 教授
A Framework for Multi-Resolution & Multi-Touch Systems Page 2 of 41
Outline
Introduction Related Work i-m-Top System i-m-Top SDK Application Conclusion & Future Work
A Framework for Multi-Resolution & Multi-Touch Systems Page 3 of 41
Outline
Introduction Related Work i-m-Top System i-m-Top SDK Application Conclusion & Future Work
A Framework for Multi-Resolution & Multi-Touch Systems Page 4 of 41
Introduction
• Motivation Multi-resolution• Variable-resolution nature of human vision• Cost-effective large high-res display
Multi-touch• Intuitive interaction• Computer-vision based approach
Software Toolkit• Hide engineering challenges
A Framework for Multi-Resolution & Multi-Touch Systems Page 5 of 41
Introduction
• i-m-Top interactive multi-resolution tableTop• interactive: multi-touch• multi-resolution: fovea + peripheral projectors
• i-m-Top SDK Toolkit for rapid prototyping applications• Featured on multi-resolution/multi-touch
A Framework for Multi-Resolution & Multi-Touch Systems Page 6 of 41
• Overview
Introduction
ProjectorsTouch Detection
Other sensors
Cams
User Interface
i-m-TopSDK
Image Processing
Applications
Pan-tilt Unit
Multimedia
A Framework for Multi-Resolution & Multi-Touch Systems Page 7 of 41
Outline
Introduction Related Work i-m-Top System i-m-Top SDK Application Conclusion & Future Work
A Framework for Multi-Resolution & Multi-Touch Systems Page 8 of 41
Related Work
• Multi-Resolution Display Fovea Plus Context • LCD screen (fixed) + projector
Baudisch, P., Good, N., and Stewart, P. "Focus Plus Context Screens: Combining Display Technology with Visualization Techniques.“ In proceedings of ACM UIST '01
A Framework for Multi-Resolution & Multi-Touch Systems Page 9 of 41
Related Work
• Multi-Resolution Display Fovea-Tablett• Tablets as high-res displays
• Tracked with visual marker• Multiple high-res is allowed
• Cons: physical boundary
J. Geisler, R. Eck, N. Rehfeld, E. Peinsipp-Byma, C. Schutz, and S. Geggus, “Fovea-�tablette: A new paradigm for the interaction with large screens," in HCI (8), 2007
A Framework for Multi-Resolution & Multi-Touch Systems Page 10 of 41
Related Work
• Software Development Toolkit DiamondTouch SDK (C++)• Multi-touch, multi-user
Esenther, A.; Forlines, C.; Ryall, K.; Shipman, S., "DiamondTouch SDK: Support for Multi-User, Multi-Touch Applications", ACM Conference on CSCW, 2002
A Framework for Multi-Resolution & Multi-Touch Systems Page 11 of 41
Related Work
• Software Development Toolkit DiamondSpin (Java): • Around-the-table interaction
• Real-time polar to Cartesian transformation
public + personal work areas rotatable circular tabletop
Shen, C.; Vernier, F.D.; Forlines, C.; Ringel, M., "DiamondSpin: An Extensible Toolkit for Around-the-Table Interaction", CHI 2004
A Framework for Multi-Resolution & Multi-Touch Systems Page 12 of 41
Related Work
• Software Development Toolkit T3 (Java)• Multiple projectors → single high-res display• Mixed-presence collaboration
Philip Tuddenham and Peter Robinson. “T3: Rapid Prototyping of High-Resolution and Mixed-Presence Tabletop Applications”, TABLETOP 2007.
A Framework for Multi-Resolution & Multi-Touch Systems Page 13 of 41
Outline
Introduction Related Work i-m-Top System
• Hardware Configuration• Touch Detection
i-m-Top SDK Application Conclusion & Future Work
A Framework for Multi-Resolution & Multi-Touch Systems Page 14 of 41
i-m-Top System
Peripheral Projector
Fovea Projector
IR Camera
IR Illuminator
Mirror
Pan-tilt Unit
56”96 ppi
24 ppi
A Framework for Multi-Resolution & Multi-Touch Systems Page 15 of 41
i-m-Top System
A Framework for Multi-Resolution & Multi-Touch Systems Page 16 of 41
• Touch Detection
i-m-Top System
Background Subtraction
OpeningDown-scale
Opening
ConnectedComponent
Palm Tracking
ConnectedComponent
PCA Analysis
Palm-fingerAssociation
Finger TrackingConnected
ComponentPCA
AnalysisPalm
Association
FingerFusionBackground
Subtraction
-
-Opening
-
finger palm
A Framework for Multi-Resolution & Multi-Touch Systems Page 17 of 41
Outline
Introduction Related Work i-m-Top System i-m-Top SDK
• Design Principles• Architecture• Implementation
Application Conclusion & Future Work
A Framework for Multi-Resolution & Multi-Touch Systems Page 18 of 41
i-m-Top SDK
• Design Principles Efficient prototyping
• Hide engineering details• Use metaphors• Provide reusable components
Touch-detection module independent• Vision-based & electronic-based approaches
Real-time requirement• Touch event handling + UI rendering
Extensibility• Integrate with other components
A Framework for Multi-Resolution & Multi-Touch Systems Page 19 of 41
i-m-Top SDK
• ArchitectureVirtual Director
Scene
ViewNode
Scene
ViewNode
Scene Scene
Node Node Viewport /Virtual Camera
Position / Textures /
Event Handlers /Animation Effects
Global Coordinate
Local Coordinate
Node
Node Virtual Camera (Fovea View)
Virtual Camera (Peripheral View)
A Framework for Multi-Resolution & Multi-Touch Systems Page 20 of 41
i-m-Top SDK
• Architecture
SceneTouch
Detection
View
Application
Node
OtherSensors
Projector
Projector
PTU
Speaker
Socket
Window Message
COM Message Parser OpenGL c#
DeviceControl
Virtual Director
Event queue
LED LightCamNode
A Framework for Multi-Resolution & Multi-Touch Systems Page 21 of 41
i-m-Top SDK
• Loop of virtual director
Check Inputs
Update States
Render Scenes
Multimedia Output
Affiliated Timer Functions
Sleep1/fps
Time spent
1/fps-T
T
0
A Framework for Multi-Resolution & Multi-Touch Systems Page 22 of 41
i-m-Top SDK
• Sample Code A scene with a photo
InitScenes( ){ oScene scene = new oScene(); //construct a scene director.AddScene(scene); //add scene to director
oView view = new View(0, 0, 1280, 720); //construct a viewscene.AddView(view); //add view to scene
oPhoto photo = new oPhoto(0,0,-100,640,480,”sun.png”); //construct a photoscene.AddNode(photo); //add to scene
}
A Framework for Multi-Resolution & Multi-Touch Systems Page 23 of 41
i-m-Top SDK
• Implementation Multi-resolution Handling Touch-event Processing Reusable Components
A Framework for Multi-Resolution & Multi-Touch Systems Page 24 of 41
Implementation
• Multi-resolution Handling Calibration• Homography of fovea & peripheral projections• Run-time warping
Masking Steerable fovea projection• Region/Object oriented
Multi-presented Information
A Framework for Multi-Resolution & Multi-Touch Systems Page 25 of 41
• Calibration Homography of fovea & peripheral projections• Provide tools for manual calibration• Save homography/ center point /pan-tilt of PTU
Implementation
F1 F2
F3F4
V1 V2
V3 V4
Y
X
V2V1
V3 V4
P1
P4
P2
P3
Y
X
F1 F2
F3F4
V1 V2
V3 V4
A Framework for Multi-Resolution & Multi-Touch Systems Page 26 of 41
• Calibration Run-time warping• Adapt OpenGL rendering pipeline
Implementation
Projection Matrix
Perspective Division
Viewport Transformation
Model-View Matrix
Local coordinate
Global coordinate
Eyecoordinate
Clipping coordinate
Normalized device
coordinate
Windowcoordinate
Warping Homography
Matrix
A Framework for Multi-Resolution & Multi-Touch Systems Page 27 of 41
• Masking
Implementation
foveaperipheral fovea
A Framework for Multi-Resolution & Multi-Touch Systems Page 28 of 41
• Masking Find maximum region in fovea• Utilize Stencil buffer in OpenGL
Implementation
Applyarbitrary masks
Set ‘0’ in mask regions
Set ‘1’ in the fovea region
Invert pixels in fovea region
A Framework for Multi-Resolution & Multi-Touch Systems Page 29 of 41
Implementation
• Steerable Fovea Projection Region-oriented
• Spotlight on a specific region • Steered by finger gestures
Object-oriented• Spotlight on a specific object• Move with object
Real-time update • Steer the mirror• Retrieve homography• Apply mask
A Framework for Multi-Resolution & Multi-Touch Systems Page 30 of 41
• Multi-presented Information Different content in fovea and peripheral regions• Eg. Google Map• Automatically change
Implementation
oRectangle multi_presentation = new oRectangle(0, 0,-100, 200, 200); multi_presentation.LoadTexture(“context.png"); //load low-resolution image multi_presentation.LoadHighTexture(“detail.png"); //load high-resolution image
A Framework for Multi-Resolution & Multi-Touch Systems Page 31 of 41
• Touch-event Processing
VirtualDirector
Implementation
TouchDetection
Pass
MessageParser Node
i-m-Top SDKTouch events
Receive Check Dispatch Handle (by event handler)
A Framework for Multi-Resolution & Multi-Touch Systems Page 32 of 41
Implementation
• Touch-event Processing Receive & parse• Run on background threads
Pass to director• Gather all events in one detection frame before pass
• if rendering rate < detection rate» Keep vital events: ‘Up’, ‘Down’» Discard other events: ‘Move’
VirtualDirector
Pass
MessageParser Node
i-m-Top SDK
Receive
A Framework for Multi-Resolution & Multi-Touch Systems Page 33 of 41
Implementation
• Touch-event Processing Check• ‘Down’: Test if hit some node
• By SELECTION mode in OpenGL• If yes: add to TargetTable (event id → node)
• ‘Move’, ‘Up’: Check if there is a target• Hit TargetTable by event id• ‘Up’ : Remove from TargetTable
Dispatch• Dispatch events to each node
VirtualDirector
Dispatch
MessageParser Node
i-m-Top SDK
Check
A Framework for Multi-Resolution & Multi-Touch Systems Page 34 of 41
Implementation
• Touch-event Processing Define event handler• Node decides how to act• Default handler
• Move/Scale/Rotate under ‘Move’ events• Redefine handler
InitScenes( ){ //……
//add new event handlers to the photo object photo.MultiFingerEvent += new MultiFingerHandler(MyMultiFingerHandler); photo.FingerClick += new FingerClickHandler(MyFingerClickHandler);}
VirtualDirector
Handle
MessageParser Node
i-m-Top SDK
A Framework for Multi-Resolution & Multi-Touch Systems Page 35 of 41
P4
Rotation: R
Implementation
P1
P2
P3
P4
Translation: T Scaling: S
P12
P3
P4
P3
P34
P3
P4
P1
P2
A Framework for Multi-Resolution & Multi-Touch Systems Page 36 of 41
Implementation
• Reusable components Multimedia• Video/audio• Camera capture
UI components• Virtual keyboard• Text-pad• Button, scrollbar…
A Framework for Multi-Resolution & Multi-Touch Systems Page 37 of 41
Outline
Introduction Related Work i-m-Top System i-m-Top SDK Application Conclusion & Future Work
A Framework for Multi-Resolution & Multi-Touch Systems Page 38 of 41
Application
• Map Browser Different content in the fovea & peripheral
• Photo Browser Object/Region oriented fovea projection
• Emotion Player Interactive art
c
A Framework for Multi-Resolution & Multi-Touch Systems Page 39 of 41
Outline
Introduction Related Work i-m-Top System i-m-Top SDK Application Conclusion & Future Work
A Framework for Multi-Resolution & Multi-Touch Systems Page 40 of 41
Conclusion
• i-m-Top Multi-resolution
• Accommodating to human vision• Cost-effective large high-res display
Multi-touch• Intuitive interaction
• i-m-Top SDK Hide engineering challenges
• Rapid prototyping applications• Investigating new interfaces / interactions
A Framework for Multi-Resolution & Multi-Touch Systems Page 41 of 41
Future Work
Attentive computing• Combine with gaze-tracking
3D interaction• Surface to space
Auto calibration• Vision-based• Continuous pan-tilt / homography map
Thanks for your listening