Upload
nasapmc
View
14.523
Download
3
Embed Size (px)
Citation preview
1
NASA PM ChallengeFeb 2011
Multidimensional RISKRISK INTEGRATED WITH SCHEDULE, COST, PERFORMANCE, AND ANYTHING ELSE YOU CAN THINK OF
2
NASA Uses Two Complementary Processes For Risk Management Risk-Informed Decision Making (RIDM)
– Emphasizes the proper use of risk analysis to make risk-informed decisions that impact all risk dimensions including safety, technical, cost, schedule, etc…
– Acknowledges the role that subject matter experts (SMEs) play in decisions. Emphasizes that the cumulative wisdom provided of SMEs is essential for integrating technical and nontechnical factors to produce sound decisions due to the availability of technical data and the complexity of missions
– Source: NASA/SP-2010-576 NASA Risk-Informed Decision Making Handbook
Continuous Risk Management (CRM)– To manage those risks associated with the performance levels that drove selection of a
particular alternative (from RIDM)– A systematic and iterative process that efficiently identifies, analyzes, plans, tracks,
controls, and communicates and documents risks associated with implementation of designs, plans, and processes
– Source: NPR 8000.4A Agency Risk Management Procedural Requirements
3
RIDM Selects Alternatives & CRM Addresses The Implementation Of Alternatives
* Source: NASA/SP-2010-576 NASA Risk-Informed Decision Making Handbook
Continuous Risk Management (CRM)
Risk-Informed Alternative Selection
Deliberate and Select an Alternative and Associated Performance Commitments Informed by (not solely based on) Risk Analysis
Risk Analysis of AlternativesRisk Analysis (Integrated Perspective) and Development of the
Technical Basis for Deliberation
Identification of AlternativesIdentify Decision Alternatives (Recognizing Opportunities) in the
Context of Objectives
Risk-Informed Decision Making (RIDM)
4
CRM Uses The Identify Step To Document Risks In The Form of Risk StatementsRisk Statements have 3 distinct elements1. Scenario
– A sequence of credible events that specifies the evolution of a system or process from a given state to a future state. In the context of risk management, scenarios are used to identify the ways in which a system or process in its current state can evolve to an undesirable state
2. Likelihood– Probability of occurrence
3. Consequence– The possible negative outcomes of the current conditions that are creating uncertainty
Given SCENARIOthere is a
LIKELIHOOD that,
CONSEQUENCE will occur
5
Risk Is Typically Measured As Likelihood Times Consequence
Likelihood
Estimation of the likelihood that the risk
event will occur
Likelihood
Estimation of the likelihood that the risk
event will occur
Definitions DefinitionsRISKS
Consequence
Estimation of the impact to the
program if the risk event
occurs
Consequence
Estimation of the impact to the
program if the risk event
occursLikelihood x Consequence
4.79 10.73 48.9429.30
Quantitative Risk Score
Samples
6
The Identify Step of CRM Documents Risk In Multiple Dimensions To Get A Complete Risk Picture
Safety
CostConfiguration Management
People
ScheduleEnvironment
Technical
On-Orbit operations risk
7
CONSEQUENCE 1 Very Low 2 Low 3 Moderate 4 High 5 Very HighTechnical Negligible or no impact to
achievement of Subsonic Transport System Level
Metrics, Technical Deliverables, Technology Maturation, or KPP Goals
Minor impact to achievement of Subsonic Transport System
Level Metrics, Technical Deliverables, Technology Maturation, or KPP Goals
Some impact to achievement of Subsonic Transport System
Level Metrics, Technical Deliverables, Technology Maturation, or KPP Goals
Moderate impact to achievement of Subsonic
Transport System Level Metrics, Technical Deliverables,
Technology Maturation, or KPP Goals
Major impact to achievement of Subsonic Transport System
Level Metrics, Technical Deliverables, Technology Maturation, or KPP Goals
ScheduleLevel 2 Milestone(s):
< 1 month impact
Level 3,4 Milestone(s): ≤ 1 month impact
Level 2 Milestone(s): ≥ 1 month impact
Level 3,4 Milestone(s): ≤ 2 month impact
Level 1 Milestone(s):≤1 month impact
Level 2 Milestone(s): ≤ 2 month impact
Level 3,4 Milestone(s):≤ 3 month impact
Level 1 Milestone(s): > 1 month impact
Level 2 Milestone(s):> 2 month impact
Level 3,4 Milestone(s): >3 month impact
Level 1 Milestone(s): > 2 month impact
Level 2 Milestone(s):≥ 3 month impact
Cost Between 0% and 5% increase over allocated
budget (Sub-Project, Element or Task level)
Between 5% and 10% increase over allocated budget (Sub-Project, Element or Task
level)
Between 10% and 15% increase over allocated budget (Sub-Project, Element or Task
level)
Between 15% and 20% increase over allocated budget (Sub-Project, Element or Task
level)
Greater than 20% increase over that allocated budget (Sub-
Project, Element or Task level)
Safety Negligible or no impact Could cause the need for only minor first aid treatment
May cause minor injury or occupational illness or minor
property damage
May cause severe injury or occupational illness or major
property damage
May cause death or permanently disabling injury or destruction of
property
7
5
4
3
2
1
Very Likely
Likely
Possible
Unlikely
HighlyUnlikely
5
4
3
2
1
1 2 3 4 5
Expected to happen
Could happen. Controls have significant limitations or uncertainties.
Could happen. Controls exist, with some limitations or uncertainties.
Not expected to happen. Controls have minor limitations or uncertainties.
Extremely remote possibility that it will happen. Strong controls in place.
Likelihood Rating RISK MATRIX
LIKE
LIHO
OD
CONSEQUENCES
Level Probability
Managers Use Customized Criteria To Bin Risks Into A Risk Matrix
8
The Risk Matrix Provides The Framework For CRM Risk Analysis
Effective analysis makes it possible to move total project risk from red to green
But how do you know you are focused on the right project risks?
Focusing on the wrong risks may keep total project risk in the red?
5
4
3
2
1
1 2 3 4 5
RISK MATRIX
LIKE
LIHO
OD
CONSEQUENCES
9
5
4
3
2
1
1 2 3 4 5
LIKE
LIHO
OD
CONSEQUENCES
5
4
3
2
1
1 2 3 4 5
LIKE
LIHO
OD
CONSEQUENCES
5
4
3
2
1
1 2 3 4 5
LIKE
LIHO
OD
CONSEQUENCES
Current Risk Matrix Development Methods Often Fail To Give A Complete Risk Picture
Notional Representation Of Risks In Three Dimensions
Why are we looking at only one dimension at a time?
Should we call pt3(3,3,3) a Cost Risk, a Schedule Risk, or a Performance Risk?
Is pt2(1,4,1) more risky than the other points just because it has a high schedule severity?
Is pt1(3,2,3) just as risky as pt3(3,3,3)?
What if we have risk across four dimensions? Or five? Or Six?
How do we know we are focusing on the right risks?
Cost Risk Schedule Risk
=pt1=pt2=pt3
Performance Risk
10
MRisk Makes Use Of Anchor Points And Multidimensional-Distance Measure To Determine Total Risk
The anchor points (1,1,1) and (5,5,5) come from our definition of the consequence scale
Distance for each point is defined by the distance of that point from the minimum over the sum of the distance from the minimum and the maximum
The distance value explains the precise consequence for each risk regardless of the number of dimensions
The greater the distance the greater the consequence and vice versa
This procedure is scalable to infinite dimensions of consequence, i.e. (1,1,…,1n) (5,5,…,5n)
min (1,1,1) max (5,5,5)
dmin
dmin
dmax d =
+
0102030405060708090
100
L M1 M2 M3 H1 H2 H3 C1 C2 C3
Consequence Scale
Low Medium High Critical
5
4
3
2
1
1 2 3 4 5
LIKE
LIHO
OD
CONSEQUENCES
5
4
3
2
1
1 2 3 4 5
LIKE
LIHO
OD
CONSEQUENCES
5
4
3
2
1
1 2 3 4 5
LIKE
LIHO
OD
CONSEQUENCES
Cost Risk Schedule Risk Performance Risk
11
Anchor Points & Mahalanobis Distance Make Risk Analysis Objective & Logically Consistent
The anchor points make it possible for us to know relative risk – Anchor points allow us to make the distinction between a (3,3,3) and a (4,2,2)– A cost consequence of 3, schedule consequence of 3, and safety consequence of 3 has a
distinct distance away from no consequence (1,1,1) and disaster (5,5,5)
Mahalanobis Distance keeps decision makers consistent in their thinking. By calculating risk based on the relationship between costs, schedule, safety, etc… MRisk identifies when violations of known relationships occur in the risk ranking process– For example, cost and schedule have a known relationship in the PM world
Schedule Cost
Scope
12
MRisk Provides A Complete Risk Picture
MRisk addresses several shortcomings in the current methods
1. MRisk deals with all of the dimensions of Risk simultaneously to provide a complete risk picture
2. MRisk makes risk analysis objective and consistent with SME judgment
3. MRisk provides more advanced statistical algorithms to Risk Management without changing the current processes or products
5
4
3
2
1
1 2 3 4 5
RISK MATRIX
LIKE
LIHO
OD
CONSEQUENCES
Schedule Cost
Scope
13
5
4
3
2
1
1 2 3 4 5
LIKE
LIHO
OD
CONSEQUENCES
Mahalanobis Distance Mapping Tells Us How Far Each Risk Is From All Of The Other Risks, Thus Highlighting Outliers
Point Sched Cost Perf …Dimn
1 4 3 3 0
2 2 1 5 3
… … … … …
m 2 4 1 4
Notional Data Set From Risk Scoring
Using traditional distance measures the outlier point in the above scenario could be masked by its proximity to the other points
Mahalanobis distance highlights the point as an outlier because of its relative distance away from the group
Mahalanobis distance accounts for the relationship of each risk to another and highlights the risks that are uncorrelated, thus detecting extreme risks more efficiently
dmin = (x-xmin)S-1(x-xmin)
dmax = (x-xmax)S-1(x-xmax)
where
S-1 is the Inv(Covariance Matrix)
xmin = [1,1,1] xmax = [5,5,5]
Outlier
Typical Point
14
Mahalanobis Distance Is Based On The Interdependencies Of Dimensions Consider two, random variables X and Y that consist of risk observations for some project or
program Those observations will have a variance and covariance Any set of random variables will have a Variance-Covariance matrix
𝑉𝑎𝑟 ( 𝑋 )= 1𝑛−1
∑𝑖=1
𝑛
(𝑥𝑖−𝑥 )2
Obs1
Obs2
Obs3
…
Obsn
X=
Obs1
Obs2
Obs3
…
Obsn
Y=
¿2𝑉𝑎𝑟 (𝑌 )= 1
𝑛−1∑𝑖=1
𝑛
(𝑦 𝑖− 𝑦 )2
[ 𝑉 𝑎𝑟 (𝑋 ) 𝐶𝑜𝑣 (𝑋 ,𝑌 )𝐶𝑜𝑣 (𝑋 ,𝑌 ) 𝑉𝑎𝑟 (𝑌 ) ]
𝐶𝑜𝑣 ( 𝑋 ,𝑌 )=∑𝑖=1
𝑛
(𝑥𝑖−𝑥)(𝑦 𝑖− 𝑦¿)
𝑛−1¿
=
¿1 ¿1
15
Two Dimensional Mahalanobis Distance Example Consider again our two, random variables X and Y that consist of risk observations for some
project or program with the derived variance-covariance matrix The distance between two points in XY-plane depends on the inverse of the variance-
covariance matrix It’s simple to expand this case to Schedule Risks(X) vs Cost Risks(Y) vs Technical Risks(Z) or
any other type of risk comparison
5
4
3
2
1
1 2 3 4 5
Y
X
X0=(2,1)
Y0=(2,4)
Distance = (X-Y)(X-Y)`= (0,3)(0,3)`=18
X-Y = (2,1)-(2,4)=(0,3) Inv(Var-Cov)= =
16
Legacy Methods By Contrast Assume Independence Of The Dimensions Of Risk
Consider again our two, random variables X and Y that consist of risk observations for some project or program with the derived variance-covariance matrix
In Euclidean Measure the distance between two points in XY-plane depends on the inverse of the Identity matrix
5
4
3
2
1
1 2 3 4 5
Y
X
X0=(2,1)
Y0=(2,4)
Distance = (X-Y)(X-Y)`= (0,3)(0,3)`=9
X-Y = (2,1)-(2,4)=(0,3) Inv(Identity)= =
17
The MRisk Metric Calculates Distance While Accounting For The Point To Point Relationship
Mahalanobis D2 is a multidimensional version of a z-score. It measures the distance of a case from the centroid (multidimensional mean) of a distribution, given the covariance (multidimensional variance) of the distribution.
A case is a multivariate outlier if the probability associated with its D2 is 0.001 or less. D2 follows a chi-square distribution with degrees of freedom equal to the number of variables included in the calculation.
Mahalanobis' distance identifies observations which lie far away from the center of the data cloud, giving less weight to variables with large variances or to groups of highly correlated variables (Joliffe, 1986).
This distance has advantages to other distance measures like the Euclidean distance which ignores the covariance structure and thus treats all variables equally
18
Case In Point: Multiple Risks Consider a risk scoring session involving
5 risks
SMEs vote on the probability and consequence (1,5) for five events across three dimensions: Performance, Cost, & Schedule
The score for each event is recorded in the table below
Event Prob Perf Cost Sched1 2 1 2 12 4 4 4 33 4 3 4 54 3 5 1 35 4 4 2 1
19
Using MRisk All The Events Fit Onto One Scale
Event Prob dmin dmax d dscaled
1 2 0.75 14.00 0.05 1.202 4 3.69 1.59 0.70 3.793 4 2.85 4.37 0.39 2.584 3 2.02 9.99 0.17 1.675 4 2.46 7.00 0.26 2.04
min (1,1,1) max (5,5,5)
Event 1 Event 3
Event 5
Event 2
MRisk answers the question regarding highest project risk
A (4,4,3) is more consequential than a (3,4,5) or a (5,1,3)
The current methods would have us focus our attention on the (3,4,5) and the (5,1,3) despite the fact that they are not the most consequential
Event Prob Perf Cost Sched1 2 1 2 12 4 4 4 33 4 3 4 54 3 5 1 35 4 4 2 1
Event 4
20
MRisk Provides A Clear Picture Of The Risk Profile Regardless Of The Number Of Dimensions Involved
Thi document contains
Booz Allen Hamilton
Inc. Proprietary
and Confidential
Business Information.
Event Prob Perf Cost Sched dscaled
1 2 1 2 1 1.202 4 4 4 3 3.793 4 3 4 5 2.584 3 5 1 3 1.675 4 4 2 1 2.04
5
4
3
2
1
1 2 3 4 5
LIKE
LIHO
OD
CONSEQUENCES
21
Traditional Multivariate Methods Like Euclidean Distance May Not Be As Clear Because They Don’t Consider Relationships
Thi document contains
Booz Allen Hamilton
Inc. Proprietary
and Confidential
Business Information.
Event Prob Perf Cost Sched Escaled1 2 1 2 1 1.102 4 4 4 3 4.143 4 3 4 5 4.414 3 5 1 3 3.005 4 4 2 1 2.11
5
4
3
2
1
1 2 3 4 5
LIKE
LIHO
OD
CONSEQUENCES
Lumping on severity despite
differences
Possible collusion of extreme risks
22
MRisk Deals With Several Shortcomings In Risk Analysis
Just because we cannot visualize risk in multiple dimensions doesn’t mean it’s not there. We all realize that Risk Management is a multi-dimensional problem that requires a multi-dimensional solution.
MRisk does not seek to change Risk Management from its current practices and procedures. It just revolutionizes Risk Analysis.
MRisk does not require any change to current data collection techniques for implementation
MRisk takes the data from the current risk methods and allows for interpretation of risks through a multidimensional lens
The use of Mahalanobis Distance as a measure of consequence takes into account the relationships that risk events have across dimensions, i.e. cost, schedule, etc…– Since we know cost relates to schedule, schedule relates to performance, performance
relates to safety, etc… MRisk is most appropriate for measuring risk as it emphasizes the relationships among risks to calculate distance