Situational Awareness in Emergency Response
Dr. Sharad MehrotraDr. Sharad MehrotraProfessor of Computer ScienceProfessor of Computer Science
Director, RESCUE ProjectDirector, RESCUE Projecthttp://www.itr-rescue.orghttp://www.itr-rescue.org
Crisis Response
• A massive, multi-organization operationA massive, multi-organization operation
• Many layers of governmentMany layers of government FederalFederal: FEMA, FBI, CDC, national : FEMA, FBI, CDC, national
guard, .guard, ... StateState: Governor’s Office of Emergency : Governor’s Office of Emergency
Services (OES), highway patrol, …Services (OES), highway patrol, … CountyCounty: county EOC, police, fire : county EOC, police, fire
personnel, …personnel, … CityCity: city emergency offices, police, : city emergency offices, police,
firefighters, …firefighters, …
• Volunteer OrganizationsVolunteer Organizations Red cross, organized citizen teamsRed cross, organized citizen teams
• IndustryIndustry Gas, electric utilities, telecommunication Gas, electric utilities, telecommunication
companies, hospitals, transportation companies, hospitals, transportation companies, media companies ….companies, media companies ….
FEDERAL
STATE
LOCAL
EMC C2
Incident command C2
FIRST RESPONDERS
VICTIMS
SYSTEMLEVELS
POLICY
LAW
AUTHORITY
RESOURCE COORD
OPERATIONS
Operational Area Emergency Operations Center
Cities of Los Angeles County (87)
Disaster ManagementArea Coordinators
Other Entities
Los Angeles County Emergency Management
Organization
Board of Supervisors
Chair of the BoardOperational Area Coordinator
Director of Emergency OperationsSheriff
LA County EmergencyManagement Council
Sheriff Contact Stations
Emergency MgmtInformation System
3
Operational View of Response
• Crisis ManagementCrisis Management Field level operationField level operation Command and control Command and control Usually local government in-chargeUsually local government in-charge
• Consequence ManagementConsequence Management Gather informationGather information
• Field, Cities, Special districts, County departments, Other EOC Field, Cities, Special districts, County departments, Other EOC sections/branchessections/branches
Analyze consequences with focus on the futureAnalyze consequences with focus on the future Develop plan of actionDevelop plan of action
• Life safety, Property loss, Environment, ReconstructionLife safety, Property loss, Environment, Reconstruction Establish who is responsibleEstablish who is responsible
Operations- Consequence Analysis
Potential need for:Potential need for:• Security for damaged/evacuated Security for damaged/evacuated
structuresstructures• Route managementRoute management• Civil disturbance controlCivil disturbance control• Casualty/Fatality collection pointsCasualty/Fatality collection points• Fire fighting/HAZMAT support Fire fighting/HAZMAT support
..
• Shelter requirementsShelter requirements• Impact on poorImpact on poor• Language, other cultural Language, other cultural
needsneeds• Food/water distributionFood/water distribution• Impact on schoolsImpact on schools• Impact on non-profit Impact on non-profit
agenciesagencies
Public Safety
Care/Shelter
Operations – Consequence Analysis
• Need for building inspectionsNeed for building inspections• Removal of hazardous materialsRemoval of hazardous materials• Demolition/debris removalDemolition/debris removal• Transportation network – impact Transportation network – impact
and restorationand restoration• Water/sewage/flood control Water/sewage/flood control
system impactssystem impacts
Construction
Logistics
CONSTRUCTION & ENGINEERING CONSTRUCTION & ENGINEERING
•Impact of utility outages•Priorities for restoration•Impact on purchasing system•Impact on transportation •Priorities for transportation restoration•Other support
Role of Information in Response
HypothesisHypothesis: Right InformationRight Information to the to the Right PersonRight Person at the at the Right TimeRight Time can result in dramatically better response can result in dramatically better response
Response Effectiveness• lives & property saved • damage prevented• cascades avoided
Quality & Timeliness of
Information
Situational Awareness• incidences• resources• victims• needs
Quality of Decisions• first responders• consequence planners• public
Challenges in Situational Awareness
RESCUE Project
The mission of RESCUE is to The mission of RESCUE is to
enhance the ability of emergency enhance the ability of emergency
response organizations to rapidly response organizations to rapidly
adapt and reconfigure crisis adapt and reconfigure crisis
response by empowering first response by empowering first
responders with access to responders with access to
accurate & actionable evolving accurate & actionable evolving
situational awarenesssituational awareness
• Privacy• Security• Trust
• Natural Hazards Center• Social Science
• Data Management• Security and Trust
• Disaster Analysis • Earthquake Engineering• GIS
• Civil Engineering• Data Analysis & Mining• Data Management• Middleware & Distributed Systems
• Civil Engineering• Transportation Engineering
• Computer Vision• Networking• Multimodal Speech
Research Team
• Transporation Modeling• Urban Planning
• Privacy• Social Science • Transportation Science
• Wireless
Funded by NSF through its large ITR program
RESCUE Partners
Industrial Partners5G Wireless
Broad-ranged IEEE 802.11 networking
AMDCompute Servers
Apani NetworksData security at layer 2
Asvaco1st responder (LAPD), and threat
analysis software
BoeingCommunity Advisory Board
Member
CanonVisualization equipment SDK
ConveraSoftware partnership
Cox CommunicationsBroadcast video delivery
D-LinkCamera Equipment and SDK
Ether2Next-generation ethernet
IBMSmart Surveillance Software (S3)
and 22 e330 xSeries servers
ImageCat, Inc.GIS loss estimation in emergency
response
MicrosoftSoftware
PrintronixRFID Technology
The School Broadcasting Company
School based dissemination
Vital Data TechnologySoftware partnership
Walker WirelessPeople-counting technology
Government PartnersCalifornia Governor’s Office of Emergency
Services
California Governor’s
Office of Homeland Security
City of Champaign City of Dana Point
City of Irvine City of Los Angeles
City of Ontario
Fire DepartmentCity of San Diego
Department of Health and Human Services – Centers
for Disease Control
Lawrence Livermore
National Laboratory
Los Angeles CountyNational Science
Foundation
Orange CountyOrange County Fire
Authority
U.S. Department of
Homeland Security
RESCUE Research
Networking & Computing systemsComputing, communication, & storage systems under extreme situations
Information Centric Computingenhanced situational awareness from multimodal data
Social & Disaster Science
context, model & understanding of process, organizational structure,
needs
Engineering & Transportationvalidation platform for role of IT research
Secu
rity
, Pri
vacy
& T
rust
C
ross
cutt
ing iss
ue a
t every
level
Situational Awareness Research in RESCUE
Extraction, synthesis,
Interpretation
SituationalData
Management
Decn. Support Tools
Approach
• Multimodal multi-sensor signal processingMultimodal multi-sensor signal processing Robustness to noise – noise affecting one modality may be Robustness to noise – noise affecting one modality may be
independent of the others.independent of the others.• E.g., multimicrophe speech recognition with background noiseE.g., multimicrophe speech recognition with background noise
Complementary information in different modalities – certain events Complementary information in different modalities – certain events easier to detect in some modalities than others. By combining easier to detect in some modalities than others. By combining modalities we can build systems that detect complex eventsmodalities we can build systems that detect complex events
• E.g., E.g., Tracking people is easier in video whereas speaker Tracking people is easier in video whereas speaker identification is easier in audio.identification is easier in audio.
• Exploit semantics & context for signal interpretationExploit semantics & context for signal interpretation Knowledge of domain can help interpret data, fill missing values, Knowledge of domain can help interpret data, fill missing values,
disambiguate. disambiguate.
Exploiting Semantics for Situational Awareness
• How does the system obtain & represent semantics?How does the system obtain & represent semantics? User specified User specified
• Language for specification of semantics, expressibility, completenessLanguage for specification of semantics, expressibility, completeness learnt from datalearnt from data
• expressibility, training set might not be available for supervised learning, noise in data expressibility, training set might not be available for supervised learning, noise in data may skew unsupervised learningmay skew unsupervised learning
• Principled approach to exploiting semantics to interpret dataPrincipled approach to exploiting semantics to interpret data Probabilistic models?Probabilistic models?
• Efficiency Efficiency Most such problems are NP-hardMost such problems are NP-hard
• Generalizability of the approachGeneralizability of the approach Can we design a generalized approach that can be used to work across diverse Can we design a generalized approach that can be used to work across diverse
types of data and for diverse situational awareness tasks.types of data and for diverse situational awareness tasks.
Event Detection from sensors
• 2300 Loop sensors in LA 2300 Loop sensors in LA and OCand OC
• Goal: Detect events such Goal: Detect events such as “baseball game” from as “baseball game” from loop sensor count data.loop sensor count data.
• Semantics:Semantics: Historical traffic data both Historical traffic data both
during game night and non-during game night and non-game nightgame night
Data is, however, Data is, however, unlabelled.unlabelled.
• Smyth et. al. -- TRBC 06, Smyth et. al. -- TRBC 06, SIGKDD 06, ACM TKDD, AAAI SIGKDD 06, ACM TKDD, AAAI 07, UAI 0707, UAI 07
Detecting Unusual Events
Unsupervised learning faces a “chicken and egg” dilemma (and others)
Ideal model
car
coun
t
Baseline model
car
coun
t
Inference over Time
Event
TrueCount
ObservedCount
SensorState
Time,Day
Time t Time t+1
Event
TrueCount
ObservedCount
SensorState
Time,Day
Note how many hidden variables are in this model
Detecting Real Events: Baseball Games
Total Number
Of Predicted Events
Graphical
Model
Detection of the 76 known events
Baseline
Model
Detection of the 76 known events
203 100.0% 86.8%
186 100.0% 81.6%
134 100.0% 72.4%
98 98.7% 60.5%
Remember: the model training is completely unsupervised,no ground truth is given to the model
Entity Resolution Problem
"J. Smith"
Raw Dataset
...J. Smith ...
.. John Smith ...
.. Jane Smith ...
MIT
Intel Inc.
?
Normalized Dataset(now can apply data analysis techniques)
Extraction(uncertainty,
duplicates, ...)
John Smith Intel
Jane Smith MIT
... ...
John SmithJane Smith
Intel
MIT
=
Attributed Relational Graph (ARG)
The problem:
(nodes, edges can have labels)(for any objects, not only people)
TODS 2005, IQIS 05, SDM 05, JCDL 07, ICDE 07, DASFAA 07, TKDE 07
April 19, 2023 DASFAA 2007, Bangkok, Thailand 20
Two Most Common Entity-Resolution Challenges
...J. Smith ...
.. John Smith ...
.. Jane Smith ...
MIT
Intel Inc.
Fuzzy lookup
– reference disambiguation– match references to objects
– list of all objects is given
Fuzzy grouping
– group together object repre-sentations, that correspond to the same object
Example of the problem: Disambiguating locations
DASFAA 2007, Bangkok, Thailand 22
Web Disambiguation
Music Composer
Football Player
UCSD Professor
Comedian
Botany Professor @ Idaho
ifif reference reference rr, made in the context of entity , made in the context of entity xx, refers to an , refers to an
entity entity yyjj but, the description, provided by but, the description, provided by r,r, matches multiple matches multiple
entities: entities: yy11,…,,…,yyjj,…,,…,yyNN, ,
thenthen xx and and yyjj are are likelylikely to be more strongly connected to to be more strongly connected to
each other via chains of relationships each other via chains of relationships
than than xx and and yykk ( (kk = 1, 2, = 1, 2, … … , , NN; ; kk jj). ).
Context Attraction Principle (CAP)
“J. Smith”publication P1
John E. SmithSSN = 123
Joe A. SmithP1
John E. Smith Jane Smith
Can be translated into a graph connectivity analysis which can be interpreted using aprobabilisitic interpretation.
25
Experiments: Quality (web disambiguation)
By Artiles, et al. in SIGIR’05 By Bekkerman & McCallum in WWW’05
26
GDF vs. Traditional (Robustness)
27
GDF vs. Context (Bhattarya & Getoor)
Co
nte
xt
Co
nte
xt
GD
F
GD
F
0.83
0.84
0.85
0.86
0.87
0.88
0.89
Publications Dataset Movies Dataset
Fp
me
asu
re
Semantics in IE
• Extracting relations from free / semi-Extracting relations from free / semi-structured text (slot-filling)structured text (slot-filling)
• Exploiting semantics in IEExploiting semantics in IE declaratively specifieddeclaratively specified
• Specified as (SQL) integrity constraintsSpecified as (SQL) integrity constraints On the relation (s) to be extractedOn the relation (s) to be extracted
Learnt from dataLearnt from data• Mine patterns and associations from the dataMine patterns and associations from the data
Declarative Constraints
create table researcher-bios (name: persontitle: thingemployer: organizationemployer-joined: datedoctoral-degree: degreedoctoral-degree-alma: organizationdoctoral-degree-date: datemasters-degree: degreemasters-degree-alma: organizationmasters-degree-date: datebachelors-degree: degreebachelors-degree-alma: organizationbachelors-degree-date: dateprevious-employers: organization awards: thing
CHECK employer != doctoral-degree-almaCHECK doctoral-degree-date > masters-degree-date)
Pattern mining over data
• Represent data as graph (RDF)Represent data as graph (RDF)• Mine interesting patterns Mine interesting patterns
Including “graph associations”Including “graph associations”
• Example aboveExample above Mostly people who have a PhD degree from a school outside the US Mostly people who have a PhD degree from a school outside the US
also have their bachelors degree from a school out side the US.also have their bachelors degree from a school out side the US.
Stanford CSU Tsinghua
1989 2002
Top10 med unranked in US OUT
PI PD MI MD BI
PI PD MI MD BI
PI PD MI MD BI
T1
T2
T3
Constraints in Action
John Smith, PhD, UCI, 2000, MS, MIT, 1997, BS, UCI, 1995
John Smith, PhD, MIT, 1997, MS, MIT, 2000, BS, UCI, 1995
John Smith, PhD, MIT, 2000, MS, MIT, 1997, BS, UCI, 1995
TUPLE (POSSIBLE) INSTANCES
CONSTRAINTS
1. Order of degree dates2. No “toggling” of schools
John Smith, PhD, UCI, 2000, MS, MIT, 1997, BS, UCI, 1995
John Smith, PhD, MIT, 1997, MS, MIT, 2000, BS, UCI, 1995
John Smith, PhD, MIT, 2000, MS, MIT, 1997, BS, UCI, 1995
Experimental Results: Improvement
accuracy (F-measure) against constraints
00.10.20.30.40.50.60.70.8
none S T CT1 CT2 CD1 CD2 CD3 CD4
constraints
CONSTRAINTS
ATTRIBUTE LEVELCD1. All (CS) PhDs awarded after 1950CD2. Current position is from among a fixed listCD3. PhD awarded only by a PhD awarding school
TUPLE:CT1. People do not “toggle” between schoolsCT2. Dates of doctoral, masters, and bachelors degrees are in orderCT3. People do not work at the same place they graduate fromCT4. More likely that the grad school is US and the undergrad school is outside US (vs other way around)CT5. The grad school rank is at least as good (or better) than the undergrad school rank
researcher-bios domain (upto) 300 training documents (Web bios) Test set > 2000 documents
Use RAPIER + Schema (type) information as baseline Add several constraints Improvement in both precision and recall
Challenges
• Language for specifying constraints.Language for specifying constraints.
• Principled approach to exploiting constraints/ patterns for extraction.Principled approach to exploiting constraints/ patterns for extraction.
• Scalability/efficiency Scalability/efficiency Naïve approach of enumerating all possible worlds leads to exponential complexity. Naïve approach of enumerating all possible worlds leads to exponential complexity. Problem NP hard even with a single FD (e.g., Problem NP hard even with a single FD (e.g., Year Year BestMovie) BestMovie)
Crash, 2005Crash, 2006
Million Dollar Baby, 2005
The Lord of the Rings, 2004The Lord of the Rings, 2005
Crash, 2005Million Dollar Baby, 2005The Lord of the Rings, 2004
Crash, 2006Million Dollar Baby, 2005The Lord of the Rings, 2005
Crash, 2006Million Dollar Baby, 2005The Lord of the Rings, 2004
Possible “worlds” (exponential !!)X
X
Summary
• Situational Awareness research in RESCUESituational Awareness research in RESCUE Event detection, extraction, and interpretation from multimodal sensor dataEvent detection, extraction, and interpretation from multimodal sensor data Situational data management (R. Jain, S. Mehrotra)Situational data management (R. Jain, S. Mehrotra) Tools for decision support (S. Mehrotra)Tools for decision support (S. Mehrotra)
• Two approaches:Two approaches: Exploiting multimodal and multisensor inputExploiting multimodal and multisensor input
• Multimodal speech, multi-microphone recog. Multimodal speech, multi-microphone recog. B. Rao, B. Rao, • Speech enhanced video Speech enhanced video M Trivedi M Trivedi• Bayesian framework for Multi-sensor event detection Bayesian framework for Multi-sensor event detection P Smyth, P Smyth,
Exploiting semantics for interpretationExploiting semantics for interpretation• Text, entity disambiguation Text, entity disambiguation S Mehrotra S Mehrotra
• Sensor data Sensor data P Smyth P Smyth• Dynamic recalibration of video based event detection system exploiting semantics Dynamic recalibration of video based event detection system exploiting semantics
[MMCN 08] [MMCN 08] S. Mehrotra, N. Venkatasubramanian S. Mehrotra, N. Venkatasubramanian• Automated tagging of images using speech input exploiting context and Automated tagging of images using speech input exploiting context and
semantics [Tech. Report 08] semantics [Tech. Report 08] S, Mehrotra S, Mehrotra
Summary
• Situational awareness applications requires techniques to translate raw Situational awareness applications requires techniques to translate raw multimodal signals into higher level events. multimodal signals into higher level events.
• Extensive research on signal processing but much of it studies different Extensive research on signal processing but much of it studies different modalities in isolationmodalities in isolation
• Multimodal event detection and exploiting semantics to interpret data is Multimodal event detection and exploiting semantics to interpret data is a promising direction.a promising direction.
• A principled, generalizable, and a comprehensive approach represents A principled, generalizable, and a comprehensive approach represents a major challenge and an opportunity. a major challenge and an opportunity.
• Situational awareness tools built on such tools could bring Situational awareness tools built on such tools could bring transformative changes to the ability of first responders and response transformative changes to the ability of first responders and response organizations to respond to crisis. organizations to respond to crisis.
Connection to Cyber SA
Physical systems
Cyber Systems
Situational AwarenessOf physical
Systems
interdependencies
Situational Awareness
Of underlying cyber systems
Adaptation, refinement
Adaptation, Security intercepts
Awareness of state of physical systemhelps gain cyber situational awarenessand vice versa. I.e., State of physical systems can serve as sensors for cyber systems and vice versa
Most of this talk focussed on here.Techniques could translate for cyber awareness.Also, through monitoring physical systems they directly could impact cyber SA.