상 황 인 식 시 스 템Context-Aware Pervasive Systems

Architecture for a New Breed of Applications

Yoon, Yong Ik010-5091-0388

yiyoon@sm.ac.kr

1

Overview of this course

1주차 : context-awareness 소개 및 시스템 개요2 주차 ; 상황인식 시스템 구조와 구성요소3주-5주 : 모바일 환경에서의 상황인식 서비스6주 : 상황인식 Artifacts에서의 구성 요소 및 S/W 개요7주 : 지능형 agents와 Web 서비스와의 연동8주 : 중간 발표9주 : 상황인식을 이용한 address, communication10주 : context-aware sensor networks and automatic identification

technology11주 : 상황인식을 이용한 security 구축12주 : integration with spatial model (GIS)13주 : 상황인식을 이용한 Mirrored world의 구축14주 : 상황인식 시스템 구축 방법 및 미들웨어

– Cook &DAS smart environment chapter 5 Middleware for smart environment 참조

15주 : 상황인식 사례 및 과제 발표16주 : 학기말 고사 Chapter 1 2

What is Context-Behavior ?

3

What is Context-aware behavior ?

1. Current Computing Trends – From the Virtual to Physical

2. Context, Context-Awareness, and Situation

3. When Systems become Context-Aware ?

4. An Overview of this course ….

Chapter 1 4

What is Context-aware Behavior ?

Chapter 1 4

What is Context-aware Behavior ?

아침/침실에서 기상 – Coffee boilingE-Newspaper on TV or ScreenGreeting you into Kitchen.

Car-Seat – Adjusting for a driver : seats, mirror, steering wheel

Office – sit on the chair adjust the heightlog-in your computer

Lunch time – greeting messages from restaurants on your Phonemenu, your favorites, location near

To Beware of people’s situations, or their contexts, Do Things for them

Chapter 1 6

What is Context-aware Behavior ?

The Economist, June 19, 2003; ACM TechNews, June 26, 2003The Sentient office is coming “Sentient computing systems are likely to be everywhere within five years --

listening and watching, and ready to anticipate their users' every need."“using information about where you are in your home to control the lighting

or ensure that only your nearest telephone rings or that the television program you are watching follows you from screen to screen as you move between rooms.”– Not only robots, but also everything – such as coffee-cup, TV, bed, …

Implementing a Sentient Computing System , 2000, Mike Addlesee, etc, AT&TSentient computing systems, which can change their behavior based on a model of the environment they construct using sensor data, may hold the key to managing tomorrow’s device-rich mobile networks.

Chapter 1 7

Current Computing trends : from Virual to physical

Intelligent augmentation (Pattie Maes, 1998)- Physical world can be enriched with

intelligent behavior.- IA is not AIhttp://www.edge.org/3rd_culture/maes/maes_p2.html

Smart environment (Cook and Das, 2005)• link computers to everyday settings and

commonplace tasks.• pervasive computing, machine learning,

wireless and sensor networking• all kinds of smart devices are continuously

working to make inhabitants’ lives more comfortable

Chapter 1 8

Programmer’s work for augmenting the physical environments with IT.

Schematic view of smart environments.

Networked devices in smart environments

Chapter 1 9

.

User Expectations

All-in-one device: convergence or divergence of applications?Quality of Experience: devices with intuitive, flexible, interactive and enabling interfaces, active guidance, personalized services, context awareness, price not an issue if the service proposition is compelling.Life assistant services: citizens wish to remain independent and mobile even in advanced years.Privacy safeguarding: requirements will increase in reaction to the growing possibilities of information interception and user profiling (location awareness and identity).Security technology: is demanded by users to cope with tracking services and surveillance systems

Challenges: Bridging the gap from technological availability to actual market take-up, Creating opportunities for user-based innovation

Chapter 1 10

Current Computing trends : The world of Convergence

상황인식 2010 가을 Chapter 1 11

Current Computing trends : Mobile & Ubiquitous

Suppose that you could get a 0.1€ “computer”.What would you do with it?Play, Work, Sense, ActuateCommunicate, Compute, AccessSearch for info, Find directionsExtend your perception of the worldTrack content and location

How would you configure it?In a stand alone modeIn a fixed networked environmentIn ad-hoc networked environment

Should it be visible?How do you interface with it?How do you power it?

Chapter 1 12

Computer systems with Context-Behavior

Computer systems with a vision– which surround, pervade, and intelligently serve people in pleasant

and unobtrusive waysUbiquitous computingPervasive computingInvisible computing,The disappearing computerProactive computingAutonomic computingAmbient intelligenceSentient computing…

These are not merely phrases; much technology lies beneath the new vocabulary and are being developed under these topics

Chapter 1 13

Ubiquitous Computing

Ubiquitous Computing (Mark Weiser,1993)- Main-Frame – many people share a computer

PC – 1 People with 1 computerUbiquitous computing – each person served

by many computers- Put out computers into everyday living

environments, instead of representing the everyday living environments into the computer.

- 2 kinds of wireless is essentiallong-range wireless (Cellphone, Satellite,…)short-range wireless(Less than 50/5 m)dual mode (indoor/outdoor) devices

Chapter 1 14

Three ubiquitous system devices:Tabs: accompanied or wearable centimeter sized devices, e.g., smartphones, smart cardsPads: hand-held decimeter-sized devices, e.g., laptopsBoards: meter sized interactive display devices,

e.g., horizontal surface computers and vertical smart boards.

Pervasive computing

Chapter 1 15

Vision of devices or computers pervading lives (by Lou Gerstner IBM CEO)- “A billion people interacting with a million E-business with a trillion

intelligent devices interconnected.”- Another term for ubiquitous computing

- A combination of mobile computing and computers embedded in the fixed environment

(use of computers worn on or carried by users)

- The goal for Planet Blue is to create and deploy a technology-assisted immersive environment used by knowledge workers in their daily lives, in which individuals and teams can create, learn, use and share knowledge with few limitations or disruptions, regardless of physical location or context.

Invisible computing

The Invisible computer , 1998, NormanThe challenges to invisible computing, 2000, BorrielloUse of computers – Focusing is not tools, but tasks

many computers in the everyday environment are merely the tools to perform tasks

ubiquitous computing - we must transfer the picture data, direct where it needs to go, and index it appropriately

invisible computing - we simply snap the picture. The system takes care of the rest of the process

automatically

Chapter 1 16

Chapter 1 17

After the user takes a picture, (1)The camera forwards the data to a

body server (shown as a cell phone), or to an interim carrier (shown as a wristwatch) and then to the body server.

Along the way, (2) the picture picks up the photographer’s identity from a personal item the user is wearing (such as a ring).

(3) The cellular connection lets the picture data travel into the information infrastructure (shown as a mainframe computer),

where (4, 5) it uses computational resources to find the services (in this case, a photo album and printed pictures) to which the user is registered.

Finally, (6) an acknowledgment that the services have received the data goes to the user’s PC,

which (7) later informs the camera that it can reuse the space occupied by that photograph. asks

Digital photography in a world of invisible computing

Disappearing-computer

EU disappearing computer projects (2001 ~ 2003)mission - to see how information technology can be diffused into everyday

objects and settings, and to see how this can lead to new ways of supporting and enhancing people's lives that go above and

beyond what is possible with the computer today.

- Create information artifacts in everyday life with computational capabilities (H/W, S/W)

- Produce new behavior and functionality with collections of artifacts.- People’s experiences with new artifact environments- Wearable computers, e-gadgets , smart-its

Chapter 1 18

Proactive Computing

Proactive computing, C. ACM, 2000, Tennenhouse• Anticipating user needs and Taking action on users’ behalf. • The idea is that human attention devoted to interaction can be reduced so that

users can focus on higher level tasks. • Getting physical. Getting real. Getting out.

Chapter 1 19

Autonomic computing

Autonomic Computing: IBM’s Perspective on the State of Information Technology, October, 2001, Horn

자동화를 통한 농업생산성 증가(200년간) : 노동인구 90 % 2.6 %, 노동시간 300 h 30 h자동교환기를 통한 교환원 생산성증가: (120만명이상 필요한데 10만명 보다 적은 운영인력)

Design and build computing systems capable of running themselves, adjusting to varying circumstances, and preparing their resources to handle most efficiently the workloads we put upon them. These autonomic systems must anticipate needs and allow users to concentrate on what they want to accomplish rather than figuring how to rig the computing systems to get them there.

Share information with every other part and contribute to some overall system awareness and regulation

Building systems with self management capabilities– self-monitor, self-healing, self-configuration, self-protection, self-optimization

Adaptable Software Agents govern a system’s definition of itself and its interaction with I/T systems around it. These systems will merge automatically with others to form new ones

Chapter 1 20

Chapter 1 21

A large retail chain with hundreds of outlets, a network of warehouses, delivery fleets, employee services, customer service call centers, web interfaces and more—an autonomic computing system manages all these distinct (and quasi-independent) I/T systems as one and provides integrated time-sensitive functionality, as well as “always available” access through web interfaces

Comparing Autonomic and Proactive computing

Chapter 1 22

Comparing autonomic and proactive computing, IBM Systems Journal, 2003, Want

Both utilize context information about the environment of the system and users to operate or make decisions.

They differ in their emphasis on the kind of system behavior to be achievedAutonomic computing : addresses the problem of managing complexityProactive computing : exploring new application domains that require principles, enabling the transition from today’s interactive systems to proactive environments that anticipate our needs and act on our behalf

Ambient Intelligence

Chapter 1 23

The New Everyday View on Ambient Intelligence , 2003 Marzano and Aarts

Research topics and case studies of Design & Research lab in PhilipsBuilds on Ubiquitous computing and intelligent user interfaces until 2010

User friendliness & efficient services for usersPeople surrounded by intelligent and intuitive interfacesEmbedded in everyday objects around usAn environment recognizing and responding to the presence of individualsContext-aware computing overlaps with the ambient intelligence vision

다섯 가지의 핵심적 요소1. Embedded - 다양한 네트워크관련 장치가 환경과 자연스럽게 숨겨지거나 결합2. Context-Aware - 사용자를 인식하고 사용자의 상황에 따른 contexts를 이해3. Personalized- 개개인 사용자의 기호와 요구에 맞출 수 있는, 4. Adaptive – 사용자에 대해 응답하여 즉각적으로 바뀔 수 있는, 5. Anticipate - 사용자의 패턴을 인식하여 사용자의 다음 행위를 예견하여 반응

Sentient Computing

The Clifford paterson Lecture, 1999, Sentient Computing, Philosophical Transactions 2000, Hopper University Cambridge

“Using sensors and resource status data to maintain a model of world which is shared between users and applications”

Build a model of a part of world from sensory informationlocation and status data extends throughout the physical environmentmonitoring spatial relationship for environment objectsVirtual network computing : automatic control of the digital environment without user intervention, the personal desktop follows the user to any nearby device

Chapter 1 24

Context, Context-Awareness, and Situations

사전적 의미Context – the set of circumstances or facts that surround a particular

events, situations, etc맥락 - 사물 따위가 서로 이어져 있는 관계나 연관

Ex) Those events should be considered in[within] the historical context그 사건들은 역사적인 맥락에서 고려되어야 한다

Context – Observed in numerous areaslinguistics, Philosophy, knowledge representation, Problem solving, Theory of communication

– Key notion treated as first-class objects in a logic– logics have been developed to enable assertions to be made about

contexts and allow context to be reasoned about explicitly in knowledge-based systems.

Chapter 1 25

First Class Object 의 정의

First Class Object 의 정의– 한 언어에서 사용되는 제약없는(without restriction) 변수 부류– 값(valuable) 형식을 취하는 것들 first class object 라 정의할 수 있

다

Definition : An object is first-class when it– can be stored in variables and data structures– can be passed as a parameter to a subroutine– can be returned as the result of a subroutine– can be constructed at runtime– has intrinsic identity (independent of any given name)

The term "object" is used loosely here, not necessarily referring to objects in object-oriented programming.

The simplest scalar data types, such as integer and floating-point numbers, are nearly always first-class.

Chapter 1 26

Definition of Context

Context-aware computing applications, 1994, Proceedings of the Workshop on Mobile Computing Systems and Applications IEEE, Schilit, B., Adams, N., Want

3 important aspects of context : perspective of distributed and mobile computing– Where are you, – Who are you, – What resources are nearby

Mobile distributed computing system– Ubiquitous access information,– Communication, computation

Context – information about a person’s proximate environment, such as Location, lighting, noise level, network connectivity, communication costs, communication band width, social situation (whether you are with your manager or with a co-worker)

.Chapter 1 27

Definition of Context

Understanding and Using Context, 2001, Personal and Ubiquitous Computing Journal, Dey

an operational definition of contextContext is any information that can be used to characterize the situation of an entity.

Entity is a person, place, or object that is considered relevant to the interaction between a user and an application, including the user and applications themselves.

Ex) weather, presence of other peoplefor indoor tour guide APP – weather is not context, other people is context

Chapter 1 28

A system is context-aware if it uses context to provide relevant information and/or services to the user, where relevancy depends on the user’s task.

A context widget is responsible for acquiring a certain type of context information and it makes that information available to applications in a generic manner, regardless of how it is actually sensed. Applications can access context from widgets using traditional poll and subscribe methods, commonly available with graphical user interface (GUI) widgets.context widgets operate independently from theapplications that use them

The Context toolkit provides three abstractions: widgets, interpreters and aggregators

Categories of Context

Computing Context – computing informationNetwork context – networking informationUser Context – user’s informationPhysical Context – environmental informationTime Context – such as time of day, week, monthEtc, etc, etc

Context Attributes from a computational perspective:Location -an entity’s position, co-location, proximity, etc.Identity - each entity has an unique identifierActivity - Status Time – used for timestamps to accurately define situation, ordering events, etc.)

Answer the questions of who, what, when and where

Primary Context Types & which form the basis for determining other contextual information known as Secondary Context Types

Chapter 1 29

Context ModelsIdentifying & structure of the context information

– Time, location, execution state of application, Computational resources, Network bandwidth, activity, user intentions, user emotions , Conditions of the environment

3 Models for more complex kinds of contextual information :– Location model

location systems design, location representations and sensor fusion, compelling location-aware applications, and factors affecting the deployment of location-aware systems in everyday environments.

– World model (Lehmann, 2004)Spatial model represents context information from the real worldModel consists of a graph, with locations as vertices and relations as

edges– Activity model :

activity refers some action or operation undertaken by human being. activity – a type of contextual information that can be used to

characterize the situation of a person.Chapter 1 30

The way that context can be acquired and modeled

How to Represent, Structure, Manage, and Use context• Knowledge-representation formalism & techniques with ontology

(Chen, 2004; Wang et al., 2004; Matheus et al., 2003)• First-order logic theory (Katsiri, 2003 ; Ranganathan, 2003)• Conceptual graph (Peters, 2003)

Ontology provide a vocabulary of concepts with which for describing context and enable reasoning with and reuse of contextual information– Formal mathematical representations enable automatic reasoning with

knowledge.

Chapter 1 31

Context-Awareness in pervasive Computing

Context awareness – Pervasive computing utilize contextual information about the physical

world.– Understanding the context and possibly situations of a system,Ex) User activity, User Location

• Smart Space activity (a meeting, a presentation…)• Sentient Building, Tour Guides,• Context-aware network management

• Connection of sensor information to Context-Aware pervasive computing.– What can be feasibly sensed– The best way to acquire sensor information– How to reason with sensor information to infer context

• Any information including the emotional states of users and movements obtained via sensors can be used as context,

Chapter 1 32

Context-aware Artifacts

Context-aware artifacts : – Entity is an artifact instead of a person

– From buildings to dolls, one can add such sensors :Ability to act on sensed information about the physical worldAutomatic behaviors without users’ direct interventionEnhance the functions of artifacts or add aesthetic value.

Chapter 1 33

SituationsSituation : The combination of circumstances at a given moment: a state of

affairs상황 : 개인이 각기 이해관계를 가지고 밀접하게 관련되어 있는 현실

그것은 단순히 자연법칙적인 세계가 아니라 의미를 가지며, 물리적임과 동시에심리적이기도 한 구체적 ·역사적인 현실이다. 상황은 한편으로 개인의 존재를 제한함과 아울러, 한편으로는 그 활동공간을 이루고, 한편으로 우연적인 소여(所與)임과 동시에 다른 한편으로는 행위에 따라 바꿀 수 있는 면도 가진다

Explored in artificial intelligence (AI), philosophy, and linguistics, such as in situation theory (Barwise and Perry, 1983) and situation calculus

An agent (e.g., a human) is able to individuate a situation

A situation : Devlin (1991)– “structured part of reality that it (the agent) somehow manages to pick

out” by “direct perception of a situation, perhaps the immediate environment, or thinking about a particular situation,” and

– “individuation of a situation by an agent does not (necessarily) entail the agent being able to provide an exact description of everything that is and is not going on in that situation.”

Chapter 1 34

Situations

Understanding and Using Context, 2001, Personal and Ubiquitous Computing Journal, Dey

Description of the states of relevant entities– aggregate context information to determine the situation of the entities

relevant to an Application– situation as being at a higher level of abstraction than context

Chapter 1 35

Situation in pervasive computing

Situation abstraction allows the modeler or application designer – carve up the world into the manageable pieces– Recognized by the system via the sensors– Compose such pieces to construct more complex models of situations

Chapter 1 36

SYSTEM

Entity

context

EntityGuess the situation of those entities

Detecting a change of situationMachine understanding of situations -> Goal

1.개인안심 서비스 : 시나리오보호대상자 Situation Transition

비 보호상태

Unsafe 상태구조 중

상태

긴급상황상태

보호상태-정지-이동

잠재위험지역근접이동시간/경로 이상 발생신체 이상징후 발견단말신호포착 중단

Unsafe 상태시간 경과최종경로이탈발생긴급구조신호

잠재위험요소해제단말포착재개정상복귀신호

긴급상황해지경로 재복귀

최종목적상태도착

주기적보호시간긴급보호요청

긴급. 잠재위험해제단말정상모드복귀

긴급구조요청

위험재발생

구호작업실시중

37

신체이상/이상음성포착->주변 카메라 확인동영상 송신 (좌표포함)센터는 수신된 동영상으로 상태판단/해당 상태정보 알려줌

이상움직임 대상 포착 -->잠재위험정보 전송인접/경로 지정된 보호대상자에게 경고신호

카메라는 게속적인 위치추적및tracking보호자에게 동영상 정보제공범죄자에는 계속적인 alarm신호 발생하여 추가 범죄 예방

구조 진행 동영상 송신

대체경로확인

When Systems Become Context-Aware

A Context-Aware System Architecture for Mobile Distributed Computing, 1995, Phd. Thesis, Schilit, Colombia University

Context-aware software adapts according to the location of use, collection of nearby people and objects ,The accessible devices,Changes to those objects over time.

• Survey the computing environment and reacts to changes to that environment

Context-Aware Pervasive computing H/W + S/W • aware of context• adapt and respond to such context automatically• reduce the burden of excessive user involvement• Provide proactive intelligent assistance

Chapter 1 38

Pervasive computing- A paradigm for the 21st centry

Pervasive Computing: Paradigm for the 21st Century , IEEE computer 2003, Sahaand Mukherjee

Use of perceptual information about environment – essential ingredient ,distinguishing them from traditional computing

The mobile computing goal of “anytime anywhere” connectivity is extended to “all the time everywhere ”by integrating pervasiveness support technologies such as interoperability, scalability, smartness, and invisibility.

Middleware mediates interactions with the networking kernel on the user’s behalf and keeps users immersed in the pervasive computing space.

Chapter 1 39System view of Pervasive computing

Pervasive computing Framework

Pervasive computing applications

Chapter 1 40

Context-aware information retrieval, (2001, Brown)Reminder systems (1997, Rhodes)Context-aware mobile service & electronic tour guides (1997, abowd)Sentient objects(2002, Fitzpatric)Sentient carsSentient buildingsContext-aware response to emergenciesProactive automated healthcare(2004, badram)Effective E-commerce(2002, Jin and Miyazawa)Military system(2002, Vander poel)Security warning systems in home or public space

Traditional desktop Application enhancement – Media player with Context-awareness

Artifacts with context-aware capabilityMobile environment

Context based Retrieval

User: Personalization, Access control Module – Preference: 취향에 따라서 정보를 검색하여 제공. Set of Keyword, Data

Association – 이 책을 산 사람은 이 책도 산다. 특정 사용자를 구분하여 저장하지 않고 모든

사용자들의 통계를 기반으로 data association을 표현. – 'A'라는 사람이 이 책을 샀다. => – 전체 사용자들의 사용데이타 마이닝을 해서 표현. – 사용자의 weight를 두어 assocation mining을 해야 한다. => Weighted

Association Analysis• A 사용자가 산 것과 B 사용자가 산 것이 다름.

Location: Local Retrieval User's Device: PDA, Phone, Notebook etc. H/W dependency User's Mood: Climate: 날씨에 따라서 가볼 만한 곳 추천이 달라야 한다. Time: 특정한 날, 계절, 월, 시간대 등Access Log/Process 지금까지의 행적/작업 이력이 다음 작업 이력을 prediction

– 특정 event 하나가 다름 event로 연결되지 않고 series of event based retrieval – Sequential Analysis and Mining?

Chapter 1 41

References and further readingshttp://www.edge.org/3rd_culture/maes/maes_p2.html intelligence augmentation, 1998,

maesAutonomic Computing: IBM’s Perspective on the State of Information Technology,

October, 2001, Horn The Clifford paterson Lecture, 1999, Sentient Computing, Philosophical Transactions

2000, Hopper University Cambridge상황인식 컴퓨터 응용기술 동향, 2004, ETRIComparing Autonomic & Proactive Computing, 2002, WantContext-Aware Computing Applications, 1994, Schilit b.Implementing a Sentient Computing System, 2000, AddlessePervasive computing - 1 paradigm for the 21st centry, 2003, SahaProactive Computing, 2000, Tennenhouse, CMUWeiser Some computer science issues in ubiquitous computing, 1993, Weiser Munderstanding and using context ,2001, DeySmart Environments. Technologies, Protocols, and Applications. - Cook Diane J., Das

Sajal K. – 2005Sentient City survival kit - archaeology of the Near Future, 2009, Shepard

상황인식 2010 가을 Chapter 1 42

References and further readingshttp://www.edge.org/3rd_culture/maes/maes_p2.html intelligence augmentation, 1998,

maesAutonomic Computing: IBM’s Perspective on the State of Information Technology,

October, 2001, Horn The Clifford paterson Lecture, 1999, Sentient Computing, Philosophical Transactions

2000, Hopper University Cambridge상황인식 컴퓨터 응용기술 동향, 2004, ETRIComparing Autonomic & Proactive Computing, 2002, WantContext-Aware Computing Applications, 1994, Schilit b.Implementing a Sentient Computing System, 2000, AddlessePervasive computing - 1 paradigm for the 21st centry, 2003, SahaProactive Computing, 2000, Tennenhouse, CMUWeiser Some computer science issues in ubiquitous computing, 1993, Weiser Munderstanding and using context ,2001, DeySmart Environments. Technologies, Protocols, and Applications. - Cook Diane J., Das

Sajal K. – 2005Sentient City survival kit - archaeology of the Near Future, 2009, Shepard

상황인식 2010 가을 Chapter 1 43