OMNIMARKETS
MODEL RISK MANAGEMENT
Science for Finance2017
AGENDA
Model Risk Management at OmniMarkets
Regulatory Guidance & Requirements
Lessons Learnt15
2
7
Model Risk Managementat
OmniMarkets
2
Overview
Focus: Model Risk Management
OmniMarkets provides a range of services to the financial industry on valuation, trading and risk management with effective and
customized solutions covering these topics. By working with global leaders on a broad range of issues, we have developed a
unique global perspective on the needs and the challenges they face and their practical solutions. We combine our industry
insights with quantitative skills and proven methodologies to help our clients transform their processes and systems and meet the
ever increasing regulatory and business requirements.
Models have been at the core of OmniMarkets from the very beginning, and more recently, responding to a growing need in
the industry and leveraging our experience, we have started a Model Risk Management ("MRM") practice. Our model
coverage includes: Valuation, Risks, IM/SIMM, CCAR, DFAST, Liquidity, SVF/WSVF, FRTB, Structured finance, CECL.
The need for MRM is only growing (more automation) and the scrutiny from Regulators (OCC & FRS) is only increasing.
At the same time MRM is still maturing and evolving and more progress is to be expected in the future.
To promote this progress, OmniMarkets has started the Model Risk Management Exchange (non-profit organization) and we
are building an MRM app.
3
Leadership
Mr. Leignadier-Fahlströmlaunched OmniMarkets in 2004. Prior to OmniMarkets, he was the head of Scandinavian interest-rate derivatives trading at SvenskaHandelsbanken in London. He also headed the Securitized Credit Analytics team at CalyonAmericas and was a derivatives trader and quantitative developer for Banque Indosuez in Paris and Stockholm.
Mr. Zhang is a Managing Partner at OmniMarkets specializing in structured finance and quantitative research and development. Prior to joining OmniMarkets, Mr. Zhang was a statistical researcher at the Centre for Economic Development Research in China and an equity analyst at CITIC Securities in Wuhan, China.
Mr. Stoikov is a Quantitative Finance Advisor at OmniMarkets. He is currently the Head of Research at Cornell Financial Engineering Manhattan (CFEM), a satellite campus of Cornell University. Previously, he was a Senior Vice President of High Frequency Trading at Cantor Fitzgerald.
Mr. Tian is a Model Risk Management Advisor at OmniMarkets. He is currently the director of risk modeling at a regional bank in New England, managing a team of quants to develop all credit risk models to support operation, reporting and compliance purposes. Previously, he was a senior manger of MRM at GE Capital, and a senior analyst of MRM at Freddie Mac Internal Audit.
Pierre Leignadier Kevin Zhang Sasha Stoikov Robert Tian
4
Existing Assets
5
• Deep understanding
of what governance means for MRM
• Deep expertise on models in general;
Body of work on some of the trickiest models
• Methodologies for the whole model lifecycle
(which we use ourselves in our work)
• In-house development and replication
capacity
• Established network of seasoned
professionals (we have worked with)
available if needed
• Up-to-date knowledge of the MRM field
Templates and documentation guidance
Checklists and governance
Flexible reporting modules
Statistical libraries for results analysis
Computing power (in-house & on cloud)
Back testing libraries
Models code
Data analysis and reporting
Data and data collection libraries
A strong team
Policies, Procedures & Governance Adapted to You
Model Development and Implementation
Model Validation of Internal and Vendor Models
Inventory of All Your Models and Readiness Assessment
Holistic MRM Procedures and Management Tool
1
2
3
4
5
What We Can Offer
6
Customized, Practical and Interactive Training6
Regulatory Guidance & Requirements
7
Key Idea 1: HOLISTIC viewKey Idea 2: a PROCESS, not an event
DevelopmentValidation
Ongoing Monitoring
Model Risk Management
Internal AuditModel Risk Management (“MRM”) is still a relatively new concept.
The landmark bulletin OCC 2000-16 focused mainly on mitigating MRM by means of Model Validation. OCC 2011-12 (SR 11-7) presented a more comprehensive approach.However, this holistic view is often not yet fully adopted and implemented. This presentation briefly reviews the key elements of MRM and how they can be brought together to help the real management of model risk.
8
1st LoD(Level of Defense)
Development and Implementation
• Specify Model Requirements:
Element #1
Collaboration between stakeholders to specify model requirements
Assess methodology and determine
modeling approach
Data quality check and initial data processing
Formulate model, testing alternatives
Review with stakeholders
• Data Analysis • Modeling & Test • Document Finalization
• Literature Review• Specify Model Requirements
Loop may be necessary, depending on feedback from stakeholders and test results.
Start documentation from the beginning Finalize documentation
• Literature Review:
• Data Analysis:
• Modeling & Test:
• Document Finalization:
Joint effort between Modeler and primary stakeholders, where model requirements are defined in accordance with business needs and intended model uses.
Determining high-level modeling approach, which will guide data collection, model framework selection, implementation and documentation.
Data are collected by Modeler in accordance with the chosen modeling approach, and necessary data cleaning, review, transformation, and sampling are performed.
The model is formulated through model segmentation, variable selection, functional form specification and parameter estimation. Alternative model formulations are tested. Testing includes outcomes analyses, benchmarking analyses, and sensitivity analyses. Model limitations should also be identified.
The model should be reviewed with primary stakeholders, all sub-stages of the model development stage should be sufficiently documented to allow for outside validation processes to be performed, and Modeler should initiate steps to implement the model. 9
Element #2-1: Validation – Internal Models
Evaluation of Conceptual Soundness
The conceptual soundness is
evaluated based on the
documentation completeness,
the model effectiveness and
model stability and robustness.
Evaluation of Technical Soundness
Technical soundness is assessed
by proofreading the original
code, or by comparing to well-
validated benchmark model
with similar theories and inputs,
or by doing model replication.
This process is performed by
comparing model outputs to
corresponding actual outcomes,
and analyzing the
discrepancies found in
between.
Outcome Analysis Document Finalization
Document should articulate
model aspects that were
reviewed, highlighting potential
deficiencies over a range of
conditions, and determining
whether adjustments are
warranted.
STEP 1 STEP 2 STEP 3 STEP 4
DocumentationDocumentationDocumentation
10
Element #2-2: Validation – Vendor Models
Evaluation of Compatibility
STEP 2-2
Vendor models are
subject to the common
validation process as
internal models. Limited
access to computer coding
and implementation
details may lead the
validator to rely more on
sensitivity analysis and
benchmarking.
The input data and assumptions provided by the vendor are
needed to assess whether they are representative of the
bank’s current situation. The customization choices used by
the bank should also be documented and justified as part of
the validation.
11
Element #3: Ongoing Monitoring
BenchmarkingCompare model outputs to estimates from alternative
data or models
Override Analysis
Override performance should be tracked and
analyzed
Keep track of code changesKeep track of system integration changes
Technical Changes Verification
Input Verification
Monitor data inputMonitor risk factor Documentation
from beginning to end
Input Verification: Data input should continue to be
accurate, complete and consistent; risk factors should
consistently fall within their effective boundaries
regarding the model
Technical Changes Verification: All changes in code
should be approved, logged and can be audited;
system may need to be updated to reflect any
changes in the data or its use
Override Analysis: Overrides may be an indication
of model limitations. If the rate of overrides is high, or
if the override process consistently improves model
performance, the underlying model may need revision
or redevelopment
Benchmarking: Discrepancies between the model
output and benchmarks should trigger investigation
into the sources and degree of the differences, and
examination of whether they are within an expected
or appropriate range given the nature of the
comparison
12
Element #4: Internal Audit
Verify General Policies
• Documentation standards• Model inventory
• Model updating procedures• Clear responsibility assignment
Review Model Development
Review Ongoing Monitoring
Review Model Validation
• Methodologies are recorded specifying merits and limitation
• Data is clean and representative• Models have been tested properly
• Evaluate process for monitoring limits on model usage
• Assess supporting operational systems
• Evaluate the reliability of data
• Independency of validators• Effective challenge is being
carried out• Validations are performed in a
timely manner
Internal audit should ensure
that the model development
is conducted in a rigorous
manner and is well-
documented
Internal audit should verify
that acceptable policies are
in place and that model
owners and control groups
comply with those policies
The review should be
focused on ensuring that the
ongoing monitoring process
is comprehensive and
effective.
Internal audit should review
validation activities conducted
by internal and external
parties with the same rigor to
see if those activities are being
conducted in accordance with
the guidance of SR 11-7
4-01 4-02 4-03 4-04
13
Element #5:Bringing Everything Together
The inventory of models, policies and procedures are all mandated by bulletin OCC 2011-12.
We suggest bringing them all together with the core elements of MRM in a workflow/alert based system.Such a system greatly improves the ‘management’ in MRM and it can also be utilized as the basis for a rudimentary quantification of MRM (questionnaire & scorecards).
By doing this, you will:1. Bring your
organization in compliance with the regulations.
2. Have the necessary framework in place for model risk measurement, monitoring and tracking.
3. Be able to have an impact and “see” it.
14
Lessons Learnt
15
Some random lessons & thoughts on MRM
16
Governance:
• Evaluate current MRM practices (gap analyses) and develop remediation plans
• Firm-wide committee and governance of MRM with better integration across LoBs
• Greater attention to MRM by board • Reports appropriately scaled for board vs
senior management • Firm pays attention to model materiality,
scales MRM accordingly
Model Overlays and Vendor Models: Both are in scope for MRM
Policies and Procedures:
• Firm has policies that apply to models everywhere in the organization
• P&P tie to supervisory expectations• Evidence that P&P serve as key guiding
documents• Clear to all in the firm which P&P are the
“law of the land” • P&P regularly updated• Firm has mechanisms to assess conformance
with P&P
Special attention:
• Good policies without compliance do not help • Uneven application of better practices, even within a bank • How to keep it going will be a challenge – ensure sustainability and regular updates • Good to demonstrate within the firm that MRM is worth the substantial costs (beyond simple compliance)
Internal Audit:
• Audit makes a full assessment of the whole MRM framework
• Planning and scopes are clearly defined and communicated ex ante
• Audit focuses on evaluating processes and controls
• Audit has sufficient standing in the organization to feel comfortable pointing out shortcomings and to have them addressed
• Audit specifies clearly what was covered and what was not
• Regular reports provided to senior management and the board (or its delegate)
Parallel review process allows validation to be completed shortly after the development ends
Ongoing feedback from validators to developers, while maintaining independence
“Observation and Request Log” documents interactions
Any “showstopper” issues Identified early
Validation finished shortly after development
PreliminaryAnalysis
Model Build Final ModelDataset Construction
Kick off
Development Documentation
Preliminary Analysis
Preliminary Model Review
Final Model Approval
Data Review
Model Design
Validation Report
This optimized approach requires additional safeguards:
Preserve effective challenge Validation staff not involved in making any decisions about
the model Policy states clearly that validation’s role is to determine if
there are model limitations Model should then be reviewed on an ongoing basis 17
The three lines of defense
1st line of defense
2nd line of defense
3rd line of defense
Model owners, Model developers, Model sponsors, Model users
Independent model reviewModel oversight committee
Internal audit
Ideal state: Appropriate oversight on 1st LODCurrent state: Significant oversight on 1st LOD
Quality of execution by 1st LoD:• Understanding and adhering to regulatory
expectation and industry best practices.
Stature/Influence of 2nd and 3rd LoD:• Ability to influence the culture of the
organization.• Ability to influence senior management
and board of directors.
Qualification and incentives provided to 2nd
and 3rd LoD:• Familiarity across a broad spectrum of
technical and business expertise.• Up to date knowledge of evolving
regulatory landscape.• Ability to communicate and influence all
stakeholders.• Organization’s focus to provide
appropriate incentive to hire right talent.
A strong and competent 1st LOD results in an efficient 2nd and 3rd LOD
Increasing amount of resources dedicated
18
Aggregating risk for interconnected models
• Dependency risk (downstream models)
• Contagion risk (upstream models)
• Measure such as model centrality can be employed
Model risk needs to be captured not only in individual
models but also in the stream of interconnected models:
Critical to manage the risk of interconnected models in
its life cycle:
• Model identification and tiering
• Model development and validation: understanding upstream and downstream risk
• Risk mitigation during model usage
• Monitoring and change control
19
Contact Us
(212)784-6487 160 Broadway, Suite 700
New York, NY 10038