Upload
buinga
View
216
Download
1
Embed Size (px)
Citation preview
seari.mit.edu © 2008 Massachusetts Institute of Technology 1
RESEARCH PROFILE
Systems Engineering EconomicsOctober 21, 2008
Dr. Ricardo Valerdi
Massachusetts Institute of Technology
mit.edu/~rvalerdi/www
seari.mit.edu © 2008 Massachusetts Institute of Technology 2
Outline
• Description of systems engineering economics– Reuse
– Human Systems Integration
– Optimism
– Adoption
– Integrating systems and software
– Heuristics for cost estimation
seari.mit.edu © 2008 Massachusetts Institute of Technology 3
Research Description
SYSTEMS ENGINEERING ECONOMICS
This research area aims at developing a new paradigm that encompasses an economics view of systems engineering to achieve measurable and predictable outcomes while delivering
value to stakeholders. Examples include:
• Measurement of productivity and quantifying the ROI of systems engineering
• Advanced methods for reuse, cost modeling, and risk modeling
• Application of real options in systems and enterprises • Leading indicators for systems engineering
effectiveness
seari.mit.edu © 2008 Massachusetts Institute of Technology 4
Why measure systems engineering?Cost Overrun as a Function of SE Effort
NASA Data
Honour, E.C., “Understanding the Value of Systems Engineering,” Proceedings of the INCOSE International Symposium, Toulouse, France, 2004.
seari.mit.edu © 2008 Massachusetts Institute of Technology 5
Feasibility Plans/Rqts. Design Develop
and TestPhases and Milestones
Relative
Size
Range
Operational
Concept
Life Cycle
Objectives
Life Cycle
Architecture
Initial
Operating
Capability
x
0.5x
0.25x
4x
2x
Boehm, B. W., Software Engineering Economics, Prentice Hall, 1981.
Cone of Uncertainty
seari.mit.edu © 2008 Massachusetts Institute of Technology 6
COSYSMO
SizeDrivers
EffortMultipliers
Effort, $
Calibration
# Requirements# Interfaces# Scenarios# Algorithms
- Application factors-8 factors
- Team factors-6 factors
Valerdi, R., The Constructive Systems Engineering Cost Model (COSYSMO): Quantifying the Costs of Systems Engineering Effort in Complex Systems, VDM Verlag, 2008.
seari.mit.edu © 2008 Massachusetts Institute of Technology 7
Reuse Terminology
New:
Items that are completely new
Managed:
Items that are incorporated and require no added SE effort other than technical management
Adopted: Items that are incorporated unmodified but require verification and validation
Modified: Items that are incorporated but require tailoring or interface changes, and verification
and validation
Deleted: Items that are removed from a legacy system, which require design analysis, tailoring
or interface changes, and verification and validation
Notes:• New items are generally unprecedented• Those items that are inherited but require architecture or implementation changes should be
counted as New
seari.mit.edu © 2008 Massachusetts Institute of Technology 8
Modified vs. New Threshold
Reuse Continuum
Modified
Adopted
New 1.0
0
Deleted
Managed
Reu
se w
eig
ht0.65
0.51
0.43
0.15
Wang, G., Valerdi, R., Fortune, J., “Reuse in Systems Engineering,” aimed for IEEE Systems, Man and Cybernetics – Part: C. (under review)
seari.mit.edu © 2008 Massachusetts Institute of Technology 9
Human Systems Integration
• The overall objective of the research is to contribute to the goals of making acquisitions more cost effective and to better support the warfighter by effectively meeting HSI requirements.
• The proposed project will develop approaches to validate two hypotheses: (1) The percentage of HSI effort can be estimated as a function of the total
systems engineering effort and used for predictive purposes on future programs; and
(2) Current systems engineering leading indicators can be extended for HSI and used to improve the predictability of HSI programmatic and technical performance on a program.
• Specific Aims: The study will focus on tools and methods to incorporate human systems integration into new and existing systems engineering economic models and indicators.
seari.mit.edu © 2008 Massachusetts Institute of Technology 10
Design task(tools/methods)Design accomplished through:
• Requirements analysis
• Quality function
deployment
• Feasibility analysis• Operational requirements & maintenance concept
• Functional analysis
• Design trade-off studies
• Simulation & modeling
• Requirements allocation
• Reliability & maintainability
analyses
• Human system integration
• Supportability analysis
• Test and evaluation
• Risk analysis
• Other supporting analyses
FunctionalGrouphardware
FunctionalGroupsoftware
FunctionalGrouphuman
EquipmentComputerSoftwareunits
HumanActivities/duties
HumanTasks/Subtasks
HardwareStructure
SoftwareStructure
MPRequirements
ComponentIntegration &prototypes
SoftwareComponentintegration
PersonnelDevelopment &Training
EquipmentTesting
SoftwareTesting
PersonnelTesting
Requirements Analysis
Functional analysis(systems level)
Evaluation(system integrationAnd testing)
Equipment& Accessories
SoftwareConfiguration∫ ∫
Day-to-day designIntegration activities
Modified from Blanchard & Fabrycky, Systems Engineering and Analysis, 2006, pp. 106
Integration of Hardware, Software, & Human Life Cycles
Design Requirements(criteria)*
Design for:• Performance
• Cost-system effectiveness
• Reliability
• Maintainability
• Political, Social, & Tech
Feasibility
• Human Factors
• Safety
• Environment
• Occupational Health
• Manpower
• Personnel
• Training
• Survivability
• Habitability
• Vulnerability
• Supportability
• Producibility
• Reconfigurability
• Affordability
• Disposability
• Flexibility (growth)
* applicable to all levels in the system structure and tailored to specific program needs
seari.mit.edu © 2008 Massachusetts Institute of Technology 11
Systems Engineering Defined
• Acquisition and Supply
– Supply Process– Acquisition Process
• Technical Management
– Planning Process– Assessment Process– Control Process
• System Design
– Requirements Definition Process– Solution Definition Process
• Product Realization
– Implementation Process– Transition to Use Process
• Technical Evaluation
– Systems Analysis Process
– Requirements Validation Process
– System Verification Process
– End Products Validation Process
EIA/ANSI 632, Processes for Engineering a System, 1999.
seari.mit.edu © 2008 Massachusetts Institute of Technology 12
Cost Driver Clusters
UNDERSTANDING FACTORS
– Requirements understanding
– Architecture understanding
– Stakeholder team cohesion
– Personnel experience/continuity
COMPLEXITY FACTORS
– Level of service requirements
– Technology Risk
– # of Recursive Levels in the Design
– Documentation Match to Life Cycle Needs
OPERATIONS FACTORS
– # and Diversity of Installations/Platforms
– Migration complexity
PEOPLE FACTORS
– Personnel/team capability
– Process capability
ENVIRONMENT FACTORS
– Multisite coordination
– Tool support
Valerdi, R., The Constructive Systems Engineering Cost Model (COSYSMO): Quantifying the Costs of Systems Engineering Effort in Complex Systems, VDM Verlag, 2008.
seari.mit.edu © 2008 Massachusetts Institute of Technology 13
Optimism in Systems Engineers
seari.mit.edu © 2008 Massachusetts Institute of Technology 14
Koehler, D. J., Harvey, N. (1997). Confidence judgments by actors and observers. Journal of Behavioral Decision Making. 10, 221-242. Russo, J. E. and Schoemaker, P. J. H. (1992). Managing
overconfidence. Sloan Management Review, 33(2), 7-17.
Why are other disciplines better at estimating?
seari.mit.edu © 2008 Massachusetts Institute of Technology 15
Quantifying Optimism Bias
Confidence (fi)
Acc
urac
y (d
i)
0 0.5 1
1
0.5 fi > di optimistic
fi = di calibrated
fi < di pessimistic
( )∑=
−
=N
i
ii dfN
scoreBrier1
21_
fi = respondent’s probability that their judgment is correctdi = outcome of the respondent’s accuracyN = total number of judgments
Where fi is a subjective probability di is an objective (empirical) probability
seari.mit.edu © 2008 Massachusetts Institute of Technology 16
Cox, W. M., Alm R., “You Are What You Spend,” NY Times, Feb 10, 2008.
seari.mit.edu © 2008 Massachusetts Institute of Technology 17
What Makes an SE Tool Adoptable?
(supply side)
• Well documented
• Trialabilty
• Low barrier of entry
• Transparency
• Demonstrates value
• Variety of incentives
• Tailorable
• Information freshness
• Relative advantage
• Compatibility
• On-going peer support
• Credibility
• Agility
• Flexibility
• Failure modes
• Enabled by IT
seari.mit.edu © 2008 Massachusetts Institute of Technology 18
Ranking of Adoption Attributes (n=35)
Adoption Attributes
2.63
2.49
2.4
2.29
2.29
2.23
2.23
2.11
1.97
1.89
1.77
1.57
0 0.5 1 1.5 2 2.5 3
Well_Documented
Credibility
Demonstrates_value
Low_Barrier_of_Entry
Information_Freshness
Transparency
Compatibility
Tailorable
On-going_Peer_Support
Variety_of_Incentives
Relative_Advantage
Trialability
Att
rib
ute
Score
Must-be
One-dimensional
Attractive
Valerdi, R. “Cultural Barriers to the Adoption of Systems Engineering Research,”2nd Asia-Pacific Conference on Systems Engineering, Yokohama, Japan, September 2008.
seari.mit.edu © 2008 Massachusetts Institute of Technology 19
Dimensions of Organizational Culture(demand side)
Social Science
• Power distance – the extent to which a society accepts the unequal distribution of power in the organization
• Uncertainty avoidance – the extent to which people are comfortable or uncomfortable with uncertainty and little structure
• Individualism – the extent to which individuals are supposed to be self-reliant and look after themselves, versus being more integrated into a group
• Masculinity or Femininity – hardness vs. softness; toughness vs. tenderness
• Long term or short term orientation –the culture’s members having a stance on delayed, or immediate, gratification
Management
• Innovation and risk taking – willing to experiment, take risks, encourage innovation
• Attention to detail – paying attention to being precise vs. saying its “good enough for chopped salad”
• Outcome orientation – oriented to results vs. oriented to process
• People orientation – degree of value and respect for people. Are people considered unique talents, or is an engineer an engineer an engineer?
• Individual vs. Team orientation – are individuals most highly noted, or are collective efforts
• Aggressiveness – taking action, dealing with conflict
• Stability – openness to change
Hofstede, G., Culture and organizations: Software of the mind. London: McGraw-Hill, 1991.
O’Reilly, C., Chatman, J., & Caldwell, D., People and organizational culture: A profile comparison approach to assessing person-organization fit. Academy of Management Journal, 34, 487-516, 1991.
seari.mit.edu © 2008 Massachusetts Institute of Technology 20
Culture of Technology
TechnologyPractice
Cultural AspectGoals, values, and ethical codes, beliefin progress, awarenessand creativity
Organizational AspectEconomic and industrial activity,professional activity, users andconsumers, trade unions
Technical AspectKnowledge, skill, and technique,tools, machines, chemicals, resources, products and wastes
General meaningof “technology”
Restricted meaningof “technology”
Pacey, A., The Culture of Technology, MIT Press, 1983.
seari.mit.edu © 2008 Massachusetts Institute of Technology 21
COCOMO II COSYSMO
Software Systems Engineering
Wang, G., Valerdi, R., Roedler, G., Ankrum, A., Gaffney, J., “Harmonizing Systems and Software Estimation,” working paper.
seari.mit.edu © 2008 Massachusetts Institute of Technology 22
COCOMO II COSYSMO
Rele
van
t ta
sks
n
ot
co
vere
d
Software Systems Engineering
seari.mit.edu © 2008 Massachusetts Institute of Technology 23
Contract Engineering WBS Based On Standards
1.0 – System/Project1.1 – Integrated Project Management (IPM)
1.2 – Systems Engineering
1.3 – Prime Mission Product (PMP)1.3.1 – Subsystem / Configuration Item (CI) 1…n (Specify Names)
1.3.2 – PMP Application Software
1.3.3 – PMP System Software
1.3.4 – PMP Integration, Assembly, Test & Checkout (IATC)
1.3.5 – Operations/Production Support
1.4 – Platform Integration
1.5 – System Test & Evaluation (ST&E)
1.6 – Training
1.7 – Data Management
1.8 – Peculiar Support Equipment
1.9 – Common Support Equipment
1.10 – Operational / Site Activation
1.11 – Industrial Facilities
Product-oriented construct, by tailoring MIL-HDBK 881A and
ANSI/EIA 632
Six Functions:
1. Systems Engineering
2. Software Engineering
3. Electrical Engineering
4. Mechanical Engineering
5. Support Engineering
6. Project Engineering Management
Six Functions:
1. Systems Engineering
2. Software Engineering
3. Electrical Engineering
4. Mechanical Engineering
5. Support Engineering
6. Project Engineering Management
seari.mit.edu © 2008 Massachusetts Institute of Technology 24
Rechtin’s
Systems
Architecting
Heuristics
COSYSMO
Model
COSYSMO
Tool
Model
Development
Heuristics
Model
Calibration
Heuristics
Model
Usage
Heuristics
Cost
Estimation
Heuristics
inspired implemented in
experiencelead to
confirmed
Valerdi, R. “Zen in the Art of Cost Estimation,” 2nd Asia-Pacific Conference on Systems Engineering, Yokohama, Japan, September 2008.
Experiential Closed Loop
seari.mit.edu © 2008 Massachusetts Institute of Technology 25
Elements of Successful Systems Architects
1. Know the engineering fundamentals on which each architecture is based
• Common sense, derived from specific engineering fundamentals and the experience of other architects, is needed to reduce the search to practical dimensions
2. Experience and judgment are necessary• Hands-on system problem solving is mandatory
3. Acquire the insights gained from experience in the design laboratories on the job
Rechtin, E. 1991. Systems Architecting: Creating & Building Complex Systems, Upper Saddle River: Prentice Hall.
seari.mit.edu © 2008 Massachusetts Institute of Technology 26
Development-related Heuristic
Heuristic #5: Some system characteristics are
more likely to be cost penalties than cost savings.
Migration Complexity
Nominal High Very High Extra High
Legacy
contractor
Self; legacy system is well documented. Original team largely available
Self; original development team not available; most documentation available
Different contractor; limited documentation
Original contractor out of business; no documentation available
Effect of
legacy
system on
new system
Everything is new; legacy system is completely replaced or non-existent
Migration is restricted to integration only
Migration is related to integration and development
Migration is related to integration, development, architecture and design
seari.mit.edu © 2008 Massachusetts Institute of Technology 27
Calibration-related Heuristic
Heuristic #6: All calibrations are local.
Before local calibration
After local calibration
seari.mit.edu © 2008 Massachusetts Institute of Technology 28
Implications for Systems Engineering
� Modeling of socio-technical phenomena• Architecture understanding, process
capability, stakeholder team cohesion, requirements understanding
� Quantification of emergent properties• Impacts of “ilities” on system costs
• Drivers of complexity
• Reuse
• Diseconomies of scale
seari.mit.edu © 2008 Massachusetts Institute of Technology 29
Agenda
9:00 Welcome and Introductions Dr. Donna Rhodes, SEAri Director
9:30 SEAri – Overview of the SEAri Research Program Dr. Donna Rhodes, Research Director
10:00 Research Profile: Socio-Technical Decision Making and Designing for Value Robustness
Dr. Adam Ross, Research Scientist
10:45 Break
11:00 Research Report: Designing Systems for Survivability Matt Richards, Doctoral Research Assistant
11:30 Research Report: Real Options in Enterprise Architecture Tsoline Mikaelian, Doctoral Research Assistant
noon LUNCH
1:00 Research Profile – Systems Engineering in the Enterprise Dr. Donna Rhodes, Principal Research Scientist
1:30 Research Report: Leveraging Organizational Culture, Standard Process, and Team Norms to Enable Collaborative Systems Thinking
Caroline Lamb, Doctoral Research Assistant
2:00 Stretch Break
2:10 Research Profile: Systems Engineering Economics Dr. Ricardo Valerdi, Research Associate
2:50 Research Poster Session with Refreshments SEAri Research Assistants
4:15 Participant Feedback and Recommendations for Research SEAri Leadership
4:55 Closing Remarks Dr. Donna Rhodes
5:00 Adjourn