Google home: Experience, support and re-experience of social home activities Anton Nijholt 소프트컴퓨팅연구실황주원

Google home: Experience, sup-port and

re-experience of social home ac-tivities

Anton Nijholt

소프트컴퓨팅연구실황주원

Overview

• Introduction

• Browsing, sharing, visiting, inhabiting, participating

• Ambient intelligence technology and environments– The meeting paradigm– Social and intelligent home environments: support and looking back– The role of autonomous and semi-autonomous embodied agents

• Smart and distributed meeting environments– General background and introduction– AMI: from signal processing to interpretation– Progress and research results– Visualization, virtual reality representation and replay

• Conclusions

2

Introduction (1)

3

• Ambient Intelligence (AMI)– Definition

• 일상 생활 속에 존재하는 모든 사물이 지능화되어 , 인간의 눈에 띄지 않으면서 언제 어디서든지 인간이 원하는 활동을 편하고 효율적으로 수행할 수 있도록 지원하는 것

– Feature• 다양한 공간 중 어느 곳으로 이동하더라도 끊임없이 사용자가 원하는 서비스가

제공된다는 것

Introduction (1)

4

Introduction (2)

5

• Ambient Intelligence (AMI)– Function

• 시스템의 주변 환경과 상황 정보를 파악할 수 있어야 함• 다양한 조건 하에서도 동적으로 시스템을 조건에 맞게 설정할 수 있어야 함• 주변 시스템과 효율적으로 상호작용할 수 있는 규칙을 찾고 생성할 수 있어야 함• 예외적인 사건에 대해서도 사용자에게 불편함을 주지 않고 스스로 복구할 수

있어야 함• In this paper

– Goal• Provide inhabitants or visitors of ambient intelligence environ-

ments with support in their activities.– Activities

• Interactions between inhabitants and between inhabitants and (semi-) autonomous agents

Introduction (2)

6

Introduction (3)

7

• In this paper– Googles’ search engine

• Search engines, tools for retrieval, searching and summarizing• This is familiar to the users and adapted to their preferences.

– We can build our own personalized and real-life web envi-ronment

• In daily activity, during our work, at home and during times of recreation.

Browsing, sharing, visiting, inhabiting, par-ticipating(1)

8

• Web environments and web tools– Retrieve pictures– Query results can be categorized according to relevance

and user’s interests.– Text, audio, pictures, video on the web can be queried.

• A individual homepage– Share their diaries, their photo albums, their music and

video– Individual life on the web– Share it with friends– MyLifeBits

• Virtual communities– In virtual 3D environments– ActiveWorlds

• You can build your own home and gave it visited by other members of the community.

• While these artificial worlds allow the display of personal in-formation through chat, choice of avatars and the design of buildings and rooms

Browsing, sharing, visiting, inhabiting, par-ticipating(2)

9

• Google Map & Google Earth• They allow interactive access to maps and satellite photos

and, although presently only for a limited number of loca-tions, 3D views of parts of cities.

• Modeling home environments in 3D virtual reality• Cameras, microphones and other sensors, and from the in-

formation• On-and off-line searching, browsing and participating

• What becomes possible if we can do this in real-time?– We can observe events taking place in reality in a virtual

reality representation– Access meta-information– Real-time generation allows real-time interaction with

human and virtual agents in these environments

Ambient intelligence technology and envi-ronments

10

• support activities of inhabitants in Ambient intelli-gence environments

• Cameras, microphones and other sensors can be used to detect and capture such activities

• In this section– Support to individuals and parties– Remote participation– Off-line access to the captured information

• This off-line access should also allow the replay of experiences.

• Sub section– The meeting paradigm– Social and intelligent home environments: support and

looking back– The role of autonomous and semi-autonomous embodied

agents


11

The meeting paradigm (1)

• Why we extend the usual viewpoint on ambient intelli-gence

- 1– Provide real-time support to activities taking place in a

smart environment– Memorized these activities– Manipulate and replay these activities

- 2 (meeting problems)– People who cannot be present to view what is going on– People to remotely participate– Provide access to captured multi-media information about

a previous meeting– People who were present and want to recall part of a

meeting– People who could not attend


12

The meeting paradigm (2)

• A first research– In order to be able to provide support, the environment is

asked to understand the interactions between its inhabi-tants and between inhabitants and the environment• The interaction that gas to be perceived does not only include

all aspects of focused interaction, but also aspects of unfo-cused interaction.

•A second research– The real-time monitoring of activities– The on-line access to information about activities– On-line remote participation in activities– Influencing activities in smart environments

• A third research– This concerns the off-line access to stored information

about activities in smart environments.– Retrieval, summarization, and browsing


13

Social and intelligent home environments: support and look-ing back

• Our viewpoint is that there are lots of reasons for wanting to look back on a previous activity•Look at events that involve multi-party interaction for which real-time support is useful


14

The role of autonomous and semi-autonomous embod-ied agents

• 3D embodied agents– These agents are real-time controlled by the behavior of

their human equivalents.– An agent can change from semi-autonomous behavior to

human-guided and human-controlled behavior.

• Maior-Domo– A domotic controller repre-

sented as avatar– Home lab situation : areal

kitchen living room– Prepare a meal, create a

shopping list, program the washing machine

– The user is wearing a wire-less microphone to have her conversation with the em-bodied agent.

Smart and distributed meeting environ-ments

15

“ What do people do at work? They go to meetings. How do we deal with meetings? What is it about sitting face-to-face that we need to capture?

We need software that makes it possible to hold a meeting

with distributed participants ㅡ a meeting with interactivity and feeling, such that, in the future, people will prefer being telepresent.”

Bill Gates, 1999.


16

General background and introduction

• The earlier AMI project– M4 project (Multi-Modal Meeting Manager)

• This projects are concerned with the design of a demonstration system that enables structuring, browsing and querying of archives of automati-cally analyzed meetings

• The meetings take place in a room equipped with multi-modal sensor. (microphones, cameras → multi-media meeting minutes)• The result of the M4 project was an off-line meeting browser.

• The Recently AMI project– Multi-modal events

• The verbal and nonverbal interaction between participants• Many events take place that are relevant for the interaction → communication content and form (someone enters the room, someone distributes a paper, a person opens

of closes the meeting…)

•cameras, circular microphone arrays, electronic paper, laper microphones and cameras


17

AMI: from signal processing to inter-pretation

• The meeting support application requires– The development of tools

• That take into account the meeting context• Bottom-up approach : more general observation on the ories of verbal

and nonverbal communication– Models

• This is needed for the integration of the multi-modal streams in order to be able to interpret events and interactions.

• These models include statistical models to integrate asynchronous mul-tiple streams and semantic representation formalisms that allow reason-ing and cross-modal reference resolution.

– Collected information• Person identification using face recognition• Current speaker recognition using multi-modal information• Speaker tracking


18

Progress and research results (1)

• Review of some more detail the research– Data recording and annotation– Meeting modeling– Audio-video processing– Access to multi-modal meeting data– Real-time support


19


• Data recording and annotation (1)– AMI Meeting Corpus consisting of 100 h of multi-modal

meeting data• The data allows empirical observations and the training of sta-

tistical models (for speech recognition, for gesture and body pose recognition,

the recognition of meeting activities and gaze and turn taking behavior of participants.)

– Machine learning techniques• This techniques are based on manually annotated meeting data.• Aim

At developing techniques for automatic recognition of proper-ties that have been annotated explicitly in the training sets.


20


• Data recording and annotation (2)– The rooms were equipped with microphones, both for close-talking

and far-field audio, and with cameras capturing close-ups of the participants and cameras that capture global room views.


21


• Data recording and annotation (3)– Tools have been developed to annotate the meeting data that has

been captured.– Interdependencies of annotated phenomena need to be explored in

order to allow us or an automatic extraction procedure to under-stand meeting activities.


22


• Meeting modeling– Develop technology

• Give real-time support to meeting participants.• These participants can be physically present in the same meeting room.• We can gave remote participants• We can gave a situation where all meeting participants are distributed.

– Structure and present meeting information• In such a way that it can be more easily accessed, in an off-line manner,

after a meeting, by both participants and others that are interested.• When the methods work in real-time, chairpersons and meeting assis-

tants can use this information about the meeting to improve their per-formance and the meeting process.


23


• Audio-video processing (1)– Various recognition algorithms

• These have been ported to the AMI meeting domain and evalu-ated.

– Automatic recognition from audio, video, audio & video• Recognize what is said by participants• Recognized what is done by participants (physical actions)• Recognize where each participant is, at each time• Recognize participants’ emotional states• Track what (person, object, or region) each participant is focus-

ing on• Recognize the identity of each participant


24


• Audio-video processing (2)– Speech recognition

• Verbal communication is the backbone of meetings.• Automatic transcription of this communication - meeting analysis, content analysis, browsing, retrieval and summariza-

tion - speech activity detection, evaluation, keyword spotting and phoneme

recognition– Localization and tracking

• Detecting and tracking of head, face and hands provides us with infor-mation about locations

• It is a first step towards identifying people, face recognition, facial ex-pression recognition and emotion recognition

– Actions and gestures• Recognized what is done by participants (physical actions)• Recognize where each participant is, at each time• Recognize participants’ emotional states• Track what (person, object, or region) each participant is focusing on• Recognize the identity of each participant


25


• Access to multi-modal meeting data


26

Visualization, virtual reality representation and replay


27

Visualization, virtual reality representation and replay

28

Conclusions

• Ambient intelligence in the home environment– Home automation is important– But providing real-time support to the inhabitants during

their activities is important as well– We have to deal with multi-party interaction– That is, there are verbal and nonverbal interactions be-

tween the human inhabitants of the environment.– The environment needs some understanding of such inter-

actions

Documents

Google home: Experience, support and re-experience of social home activities Anton Nijholt 소프트컴퓨팅연구실황주원