Situational Awareness in Emergency Response

Dr. Sharad MehrotraDr. Sharad MehrotraProfessor of Computer ScienceProfessor of Computer Science

Director, RESCUE ProjectDirector, RESCUE Projecthttp://www.itr-rescue.orghttp://www.itr-rescue.org

Crisis Response

• A massive, multi-organization operationA massive, multi-organization operation

• Many layers of governmentMany layers of government FederalFederal: FEMA, FBI, CDC, national : FEMA, FBI, CDC, national

guard, .guard, ... StateState: Governor’s Office of Emergency : Governor’s Office of Emergency

Services (OES), highway patrol, …Services (OES), highway patrol, … CountyCounty: county EOC, police, fire : county EOC, police, fire

personnel, …personnel, … CityCity: city emergency offices, police, : city emergency offices, police,

firefighters, …firefighters, …

• Volunteer OrganizationsVolunteer Organizations Red cross, organized citizen teamsRed cross, organized citizen teams

• IndustryIndustry Gas, electric utilities, telecommunication Gas, electric utilities, telecommunication

companies, hospitals, transportation companies, hospitals, transportation companies, media companies ….companies, media companies ….

FEDERAL

STATE

LOCAL

EMC C2

Incident command C2

FIRST RESPONDERS

VICTIMS

SYSTEMLEVELS

POLICY

LAW

AUTHORITY

RESOURCE COORD

OPERATIONS

Operational Area Emergency Operations Center

Cities of Los Angeles County (87)

Disaster ManagementArea Coordinators

Other Entities

Los Angeles County Emergency Management

Organization

Board of Supervisors

Chair of the BoardOperational Area Coordinator

Director of Emergency OperationsSheriff

LA County EmergencyManagement Council

Sheriff Contact Stations

Emergency MgmtInformation System

3

Operational View of Response

• Crisis ManagementCrisis Management Field level operationField level operation Command and control Command and control Usually local government in-chargeUsually local government in-charge

• Consequence ManagementConsequence Management Gather informationGather information

• Field, Cities, Special districts, County departments, Other EOC Field, Cities, Special districts, County departments, Other EOC sections/branchessections/branches

Analyze consequences with focus on the futureAnalyze consequences with focus on the future Develop plan of actionDevelop plan of action

• Life safety, Property loss, Environment, ReconstructionLife safety, Property loss, Environment, Reconstruction Establish who is responsibleEstablish who is responsible

Operations- Consequence Analysis

Potential need for:Potential need for:• Security for damaged/evacuated Security for damaged/evacuated

structuresstructures• Route managementRoute management• Civil disturbance controlCivil disturbance control• Casualty/Fatality collection pointsCasualty/Fatality collection points• Fire fighting/HAZMAT support Fire fighting/HAZMAT support

..

• Shelter requirementsShelter requirements• Impact on poorImpact on poor• Language, other cultural Language, other cultural

needsneeds• Food/water distributionFood/water distribution• Impact on schoolsImpact on schools• Impact on non-profit Impact on non-profit

agenciesagencies

Public Safety

Care/Shelter

Operations – Consequence Analysis

• Need for building inspectionsNeed for building inspections• Removal of hazardous materialsRemoval of hazardous materials• Demolition/debris removalDemolition/debris removal• Transportation network – impact Transportation network – impact

and restorationand restoration• Water/sewage/flood control Water/sewage/flood control

system impactssystem impacts

Construction

Logistics

CONSTRUCTION & ENGINEERING CONSTRUCTION & ENGINEERING

•Impact of utility outages•Priorities for restoration•Impact on purchasing system•Impact on transportation •Priorities for transportation restoration•Other support

Role of Information in Response

HypothesisHypothesis: Right InformationRight Information to the to the Right PersonRight Person at the at the Right TimeRight Time can result in dramatically better response can result in dramatically better response

Response Effectiveness• lives & property saved • damage prevented• cascades avoided

Quality & Timeliness of

Information

Situational Awareness• incidences• resources• victims• needs

Quality of Decisions• first responders• consequence planners• public

Challenges in Situational Awareness

RESCUE Project

The mission of RESCUE is to The mission of RESCUE is to

enhance the ability of emergency enhance the ability of emergency

response organizations to rapidly response organizations to rapidly

adapt and reconfigure crisis adapt and reconfigure crisis

response by empowering first response by empowering first

responders with access to responders with access to

accurate & actionable evolving accurate & actionable evolving

situational awarenesssituational awareness

• Privacy• Security• Trust

• Natural Hazards Center• Social Science

• Data Management• Security and Trust

• Disaster Analysis • Earthquake Engineering• GIS

• Civil Engineering• Data Analysis & Mining• Data Management• Middleware & Distributed Systems

• Civil Engineering• Transportation Engineering

• Computer Vision• Networking• Multimodal Speech

Research Team

• Transporation Modeling• Urban Planning

• Privacy• Social Science • Transportation Science

• Wireless

Funded by NSF through its large ITR program

RESCUE Partners

Industrial Partners5G Wireless

Broad-ranged IEEE 802.11 networking

AMDCompute Servers

Apani NetworksData security at layer 2

Asvaco1st responder (LAPD), and threat

analysis software

BoeingCommunity Advisory Board

Member

CanonVisualization equipment SDK

ConveraSoftware partnership

Cox CommunicationsBroadcast video delivery

D-LinkCamera Equipment and SDK

Ether2Next-generation ethernet

IBMSmart Surveillance Software (S3)

and 22 e330 xSeries servers

ImageCat, Inc.GIS loss estimation in emergency

response

MicrosoftSoftware

PrintronixRFID Technology

The School Broadcasting Company

School based dissemination

Vital Data TechnologySoftware partnership

Walker WirelessPeople-counting technology

Government PartnersCalifornia Governor’s Office of Emergency

Services

California Governor’s

Office of Homeland Security

City of Champaign City of Dana Point

City of Irvine City of Los Angeles

City of Ontario

Fire DepartmentCity of San Diego

Department of Health and Human Services – Centers

for Disease Control

Lawrence Livermore

National Laboratory

Los Angeles CountyNational Science

Foundation

Orange CountyOrange County Fire

Authority

U.S. Department of

Homeland Security

RESCUE Research

Networking & Computing systemsComputing, communication, & storage systems under extreme situations

Information Centric Computingenhanced situational awareness from multimodal data

Social & Disaster Science

context, model & understanding of process, organizational structure,

needs

Engineering & Transportationvalidation platform for role of IT research

Secu

rity

, Pri

vacy

& T

rust

C

ross

cutt

ing iss

ue a

t every

level

Situational Awareness Research in RESCUE

Extraction, synthesis,

Interpretation

SituationalData

Management

Decn. Support Tools

Approach

• Multimodal multi-sensor signal processingMultimodal multi-sensor signal processing Robustness to noise – noise affecting one modality may be Robustness to noise – noise affecting one modality may be

independent of the others.independent of the others.• E.g., multimicrophe speech recognition with background noiseE.g., multimicrophe speech recognition with background noise

Complementary information in different modalities – certain events Complementary information in different modalities – certain events easier to detect in some modalities than others. By combining easier to detect in some modalities than others. By combining modalities we can build systems that detect complex eventsmodalities we can build systems that detect complex events

• E.g., E.g., Tracking people is easier in video whereas speaker Tracking people is easier in video whereas speaker identification is easier in audio.identification is easier in audio.

• Exploit semantics & context for signal interpretationExploit semantics & context for signal interpretation Knowledge of domain can help interpret data, fill missing values, Knowledge of domain can help interpret data, fill missing values,

disambiguate. disambiguate.

Exploiting Semantics for Situational Awareness

• How does the system obtain & represent semantics?How does the system obtain & represent semantics? User specified User specified

• Language for specification of semantics, expressibility, completenessLanguage for specification of semantics, expressibility, completeness learnt from datalearnt from data

• expressibility, training set might not be available for supervised learning, noise in data expressibility, training set might not be available for supervised learning, noise in data may skew unsupervised learningmay skew unsupervised learning

• Principled approach to exploiting semantics to interpret dataPrincipled approach to exploiting semantics to interpret data Probabilistic models?Probabilistic models?

• Efficiency Efficiency Most such problems are NP-hardMost such problems are NP-hard

• Generalizability of the approachGeneralizability of the approach Can we design a generalized approach that can be used to work across diverse Can we design a generalized approach that can be used to work across diverse

types of data and for diverse situational awareness tasks.types of data and for diverse situational awareness tasks.

Event Detection from sensors

• 2300 Loop sensors in LA 2300 Loop sensors in LA and OCand OC

• Goal: Detect events such Goal: Detect events such as “baseball game” from as “baseball game” from loop sensor count data.loop sensor count data.

• Semantics:Semantics: Historical traffic data both Historical traffic data both

during game night and non-during game night and non-game nightgame night

Data is, however, Data is, however, unlabelled.unlabelled.

• Smyth et. al. -- TRBC 06, Smyth et. al. -- TRBC 06, SIGKDD 06, ACM TKDD, AAAI SIGKDD 06, ACM TKDD, AAAI 07, UAI 0707, UAI 07

Detecting Unusual Events

Unsupervised learning faces a “chicken and egg” dilemma (and others)

Ideal model

car

coun

t

Baseline model

car

coun

t

Inference over Time

Event

TrueCount

ObservedCount

SensorState

Time,Day

Time t Time t+1

Event

TrueCount

ObservedCount

SensorState

Time,Day

Note how many hidden variables are in this model

Detecting Real Events: Baseball Games

Total Number

Of Predicted Events

Graphical

Model

Detection of the 76 known events

Baseline

Model

Detection of the 76 known events

203 100.0% 86.8%

186 100.0% 81.6%

134 100.0% 72.4%

98 98.7% 60.5%

Remember: the model training is completely unsupervised,no ground truth is given to the model

Entity Resolution Problem

"J. Smith"

Raw Dataset

...J. Smith ...

.. John Smith ...

.. Jane Smith ...

MIT

Intel Inc.

?

Normalized Dataset(now can apply data analysis techniques)

Extraction(uncertainty,

duplicates, ...)

John Smith Intel

Jane Smith MIT

... ...

John SmithJane Smith

Intel

MIT

=

Attributed Relational Graph (ARG)

The problem:

(nodes, edges can have labels)(for any objects, not only people)

TODS 2005, IQIS 05, SDM 05, JCDL 07, ICDE 07, DASFAA 07, TKDE 07

April 19, 2023 DASFAA 2007, Bangkok, Thailand 20

Two Most Common Entity-Resolution Challenges

...J. Smith ...

.. John Smith ...

.. Jane Smith ...

MIT

Intel Inc.

Fuzzy lookup

– reference disambiguation– match references to objects

– list of all objects is given

Fuzzy grouping

– group together object repre-sentations, that correspond to the same object

Example of the problem: Disambiguating locations

DASFAA 2007, Bangkok, Thailand 22

Web Disambiguation

Music Composer

Football Player

UCSD Professor

Comedian

Botany Professor @ Idaho

ifif reference reference rr, made in the context of entity , made in the context of entity xx, refers to an , refers to an

entity entity yyjj but, the description, provided by but, the description, provided by r,r, matches multiple matches multiple

entities: entities: yy11,…,,…,yyjj,…,,…,yyNN, ,

thenthen xx and and yyjj are are likelylikely to be more strongly connected to to be more strongly connected to

each other via chains of relationships each other via chains of relationships

than than xx and and yykk ( (kk = 1, 2, = 1, 2, … … , , NN; ; kk jj). ).

Context Attraction Principle (CAP)

“J. Smith”publication P1

John E. SmithSSN = 123

Joe A. SmithP1

John E. Smith Jane Smith

Can be translated into a graph connectivity analysis which can be interpreted using aprobabilisitic interpretation.

25

Experiments: Quality (web disambiguation)

By Artiles, et al. in SIGIR’05 By Bekkerman & McCallum in WWW’05

26

GDF vs. Traditional (Robustness)

27

GDF vs. Context (Bhattarya & Getoor)

Co

nte

xt

Co

nte

xt

GD

F

GD

F

0.83

0.84

0.85

0.86

0.87

0.88

0.89

Publications Dataset Movies Dataset

Fp

me

asu

re

Semantics in IE

• Extracting relations from free / semi-Extracting relations from free / semi-structured text (slot-filling)structured text (slot-filling)

• Exploiting semantics in IEExploiting semantics in IE declaratively specifieddeclaratively specified

• Specified as (SQL) integrity constraintsSpecified as (SQL) integrity constraints On the relation (s) to be extractedOn the relation (s) to be extracted

Learnt from dataLearnt from data• Mine patterns and associations from the dataMine patterns and associations from the data

Declarative Constraints

create table researcher-bios (name: persontitle: thingemployer: organizationemployer-joined: datedoctoral-degree: degreedoctoral-degree-alma: organizationdoctoral-degree-date: datemasters-degree: degreemasters-degree-alma: organizationmasters-degree-date: datebachelors-degree: degreebachelors-degree-alma: organizationbachelors-degree-date: dateprevious-employers: organization awards: thing

CHECK employer != doctoral-degree-almaCHECK doctoral-degree-date > masters-degree-date)

Pattern mining over data

• Represent data as graph (RDF)Represent data as graph (RDF)• Mine interesting patterns Mine interesting patterns

Including “graph associations”Including “graph associations”

• Example aboveExample above Mostly people who have a PhD degree from a school outside the US Mostly people who have a PhD degree from a school outside the US

also have their bachelors degree from a school out side the US.also have their bachelors degree from a school out side the US.

Stanford CSU Tsinghua

1989 2002

Top10 med unranked in US OUT

PI PD MI MD BI

PI PD MI MD BI

PI PD MI MD BI

T1

T2

T3

Constraints in Action

John Smith, PhD, UCI, 2000, MS, MIT, 1997, BS, UCI, 1995

John Smith, PhD, MIT, 1997, MS, MIT, 2000, BS, UCI, 1995

John Smith, PhD, MIT, 2000, MS, MIT, 1997, BS, UCI, 1995

TUPLE (POSSIBLE) INSTANCES

CONSTRAINTS

1. Order of degree dates2. No “toggling” of schools

John Smith, PhD, UCI, 2000, MS, MIT, 1997, BS, UCI, 1995

John Smith, PhD, MIT, 1997, MS, MIT, 2000, BS, UCI, 1995

John Smith, PhD, MIT, 2000, MS, MIT, 1997, BS, UCI, 1995

Experimental Results: Improvement

accuracy (F-measure) against constraints

00.10.20.30.40.50.60.70.8

none S T CT1 CT2 CD1 CD2 CD3 CD4

constraints

CONSTRAINTS

ATTRIBUTE LEVELCD1. All (CS) PhDs awarded after 1950CD2. Current position is from among a fixed listCD3. PhD awarded only by a PhD awarding school

TUPLE:CT1. People do not “toggle” between schoolsCT2. Dates of doctoral, masters, and bachelors degrees are in orderCT3. People do not work at the same place they graduate fromCT4. More likely that the grad school is US and the undergrad school is outside US (vs other way around)CT5. The grad school rank is at least as good (or better) than the undergrad school rank

researcher-bios domain (upto) 300 training documents (Web bios) Test set > 2000 documents

Use RAPIER + Schema (type) information as baseline Add several constraints Improvement in both precision and recall

Challenges

• Language for specifying constraints.Language for specifying constraints.

• Principled approach to exploiting constraints/ patterns for extraction.Principled approach to exploiting constraints/ patterns for extraction.

• Scalability/efficiency Scalability/efficiency Naïve approach of enumerating all possible worlds leads to exponential complexity. Naïve approach of enumerating all possible worlds leads to exponential complexity. Problem NP hard even with a single FD (e.g., Problem NP hard even with a single FD (e.g., Year Year BestMovie) BestMovie)

Crash, 2005Crash, 2006

Million Dollar Baby, 2005

The Lord of the Rings, 2004The Lord of the Rings, 2005

Crash, 2005Million Dollar Baby, 2005The Lord of the Rings, 2004

Crash, 2006Million Dollar Baby, 2005The Lord of the Rings, 2005

Crash, 2006Million Dollar Baby, 2005The Lord of the Rings, 2004

Possible “worlds” (exponential !!)X

X

Summary

• Situational Awareness research in RESCUESituational Awareness research in RESCUE Event detection, extraction, and interpretation from multimodal sensor dataEvent detection, extraction, and interpretation from multimodal sensor data Situational data management (R. Jain, S. Mehrotra)Situational data management (R. Jain, S. Mehrotra) Tools for decision support (S. Mehrotra)Tools for decision support (S. Mehrotra)

• Two approaches:Two approaches: Exploiting multimodal and multisensor inputExploiting multimodal and multisensor input

• Multimodal speech, multi-microphone recog. Multimodal speech, multi-microphone recog. B. Rao, B. Rao, • Speech enhanced video Speech enhanced video M Trivedi M Trivedi• Bayesian framework for Multi-sensor event detection Bayesian framework for Multi-sensor event detection P Smyth, P Smyth,

Exploiting semantics for interpretationExploiting semantics for interpretation• Text, entity disambiguation Text, entity disambiguation S Mehrotra S Mehrotra

• Sensor data Sensor data P Smyth P Smyth• Dynamic recalibration of video based event detection system exploiting semantics Dynamic recalibration of video based event detection system exploiting semantics

[MMCN 08] [MMCN 08] S. Mehrotra, N. Venkatasubramanian S. Mehrotra, N. Venkatasubramanian• Automated tagging of images using speech input exploiting context and Automated tagging of images using speech input exploiting context and

semantics [Tech. Report 08] semantics [Tech. Report 08] S, Mehrotra S, Mehrotra

Summary

• Situational awareness applications requires techniques to translate raw Situational awareness applications requires techniques to translate raw multimodal signals into higher level events. multimodal signals into higher level events.

• Extensive research on signal processing but much of it studies different Extensive research on signal processing but much of it studies different modalities in isolationmodalities in isolation

• Multimodal event detection and exploiting semantics to interpret data is Multimodal event detection and exploiting semantics to interpret data is a promising direction.a promising direction.

• A principled, generalizable, and a comprehensive approach represents A principled, generalizable, and a comprehensive approach represents a major challenge and an opportunity. a major challenge and an opportunity.

• Situational awareness tools built on such tools could bring Situational awareness tools built on such tools could bring transformative changes to the ability of first responders and response transformative changes to the ability of first responders and response organizations to respond to crisis. organizations to respond to crisis.

Connection to Cyber SA

Physical systems

Cyber Systems

Situational AwarenessOf physical

Systems

interdependencies

Situational Awareness

Of underlying cyber systems

Adaptation, refinement

Adaptation, Security intercepts

Awareness of state of physical systemhelps gain cyber situational awarenessand vice versa. I.e., State of physical systems can serve as sensors for cyber systems and vice versa

Most of this talk focussed on here.Techniques could translate for cyber awareness.Also, through monitoring physical systems they directly could impact cyber SA.