Federal Acquisition Service U.S. General Services Administration Performance Metrics Chris Hamm Operations Director (acting) GSA FEDSIM

Federal Acquisition Service

U.S. General Services Administration

Performance Metrics

Chris HammOperations Director (acting)

GSA FEDSIM

Performance Metrics

Chris HammOperations Director (acting)

GSA FEDSIM


2

Topics

Quote PBSA Perspectives Contract Types Definitions

Performance Metrics SLAs

Implementation Details Pre-Solicitation Solicitation Pre-Award Kickoff Post-Award

Measurements Exceptions & Pitfalls


3

If you can’t measure it, you can’t manage it


4

Performance Based Services Acquisition

FY08 – 50% of all eligible services acquisitions should be PBSA

FAR Subpart 37.6 (b) Performance-based contracts for services shall include—

(1) A performance work statement (PWS); (2) Measurable performance standards (i.e., in terms of quality,

timeliness, quantity, etc.) and the method of assessing contractor performance against performance standards; and

(3) Performance incentives where appropriate. When used, the performance incentives shall correspond to the performance standards set forth in the contract (see 16.402-2).

http://www.acquisition.gov/far/current/html/Subpart%2016_4.html


5

Government Perspective

1. Compliance – We have to do it

2. Funding – Less $ to do more work

3. Cost – Best Value for the taxpayer

4. Performance – Use of commercial best practices

5. Schedule – Congressional & Executive mandated dates

But . . . PBSA is hard, especially when we are overwhelmed withwork and are starting with the old SOW


6

Contractor Perspective

Profitability – Companies are in business to make money first. Project Success – Delivering success typically results in more

follow-on business Cash Flow – Regular payments and meeting corporate

profitability is essential for publicly traded companies. Missing deliveries and failing SLAs may result in lower stock prices and replacement of personnel

Personal Services – Use of performance metrics reduces the ability for the Government to dictate the solution


7

Cost Reimburseable to Fixed Price continuumResearch Development

Government assumes

more cost riskContractor assumes

more cost risk

Production/Sustainment

As you move to the right, the more detailed and meaningful the performance measures become. . . .

Higher risk, less-defined requirements

CPFF CPFF / CPAF CPIF/CPAF

R&D Development

FPAF/FPIF/ FFP FPIF /FFPEnhancements / Production Operations &

Maintenance

Lower Risk, well-defined requirements


8

Definitions

Performance Metric A measure collected by the Contractor that may or

may not have financial incentives & disincentives

SLA – Service Level Agreement Part of a contract between the Government and

the Contractor to provide services reliably and at a specified performance level. SLAs have financial incentives and/or disincentives

FYI: There are lots of other names, like performance goals or SLOs – Service Level Objectives.


9

Performance Metrics are. . . .

1. A communications tool. The value of an agreement is not just in the final product; the very process of establishing metrics helps to open up communications.

2. A conflict-prevention tool. An agreement helps to avoid or alleviate disputes by providing a shared understanding of needs and priorities. And if conflicts do occur, they tend to be resolved more readily and with less gnashing of teeth.

3. A living document. This is one of its most important benefits. The agreement isn't a dead-end document consigned to the Forget Forever file. On a predetermined frequency, the parties to the review the metric to assess service adequacy and negotiate adjustments.

4. An objective basis for gauging service effectiveness. An metric ensures that all parties use the same criteria to evaluate service quality.


10

Definitions

Cost – The fully loaded cost of performing the service (DCAA cost accounting system). This comes into play on CPAF & CPIF, but does not apply to FFP types

Base Fee – Some Incentive & Award Fees have two components. The Base Fee is like percentage of fee that is automatically earned, while the rest is part of the award/incentive fee pool. Contractors like base fee for two reasons: (1) They automatically earn it, and (2) it helps their cash flow. Otherwise, fee isn’t earned until the end of the period

Rollover – If any fee is not earned at the end of a period, it can be lost or “rolled over”.


11

How GSA FEDSIM implements Incentives

Primarily, we have SLAs on our larger task orders There are about 30 active projects with SLAs

Most incentives at FEDSIM are part of task orders with hybrid contract types (CPIF & CPAF) with a few notable exceptions (FFP)

SLAs tend to be implemented more on our Operations & Maintenance projects, but also exist on Development projects


12

Examples of SLAs

• Network Availability• Server Availability• Circuit Installation• Move / Add / Change

• called a MAC• Trouble Ticket

• Tier 1/2/3/4• Premium / Standard

• Call Center• 1st Call Resolution• Speed of Answer• Abandonment Rate

• Customer Satisfaction• Cost

• CPI from EVMS• Budget / Savings

• Schedule• SPI from EVMS

• Small Business Subcontracting• Time to fill vacancies• Backups• COOP• Invoicing Accuracy


13

Examples of Measurements

Percentages See Chart

Hours Indicate normal business

hours for response! Days

Always clarify business vs. calendar days!

Yes / No

Measure Downtime in one year

99% 3.7 days

99.90% 8.8 hours

99.99% 53 minutes

99.999% 5.3 minutes

Most clients do not need Five 9s and could not afford it if they wanted it. The cost is prohibitive!NEW EXCEPTION - CLOUD availability


14

Philosophy

Why have incentives? Setting performance standards, measuring performance,

and understanding cost impacts is difficult. Without SLAs, all actions are inherently subjective

Incentives improve project performance only if contract administration and quality assurance functions are implemented. Otherwise, they just add cost.

SLAs should be “end to end” as much as possible. Don’t measure email server availability if you can measure

the time from when an email is sent to when it is received


15

Philosophy

SLAs should align with the client and project objective AND motivate good contractor behavior Do not design an SLA that, when missed, motivates the

contractor to give up on that measure for the rest of the period

– Example: Doing an “average” over a six month period. If the contractor fails in the 1st month, there is no incentive to maintain service levels or improve over time.

Just because something can be measured doesn’t mean it should be an SLA Example: It is easy to measure how fast someone picks up a

phone, but a better measure is how long it takes to actually solve the problem.


16

Process (Pre-Award)

• Draft Incentive/ Award Fee Plan and SLA Format included in Section J• Evaluation Decision:

• Option 1: Specify SLAs & have all vendors bid the same• Option 2: Let vendors chose SLAs & targets

• Review requirements with the client before release and see if tasks are sufficiently defined to create notional SLAs to include as recommendations

• Remember that the Government’s strongest negotiation position is pre-award.• Fee Plans are unilateral before the period begins, bi-lateral

during the period (if the CO allows any changes at all)

Option # 2 is harder to evaluate but better post award: Better measures, more leverage after award.


17

Process (immediate Post-Award)

Ensure Contract Award includes any contractor proposed SLAs and/or changes from Negotiation

Set the percentage of award fee pool that is SLA based Set up separate meeting from Kickoff Meeting with key

stakeholders to review SLAs. Ensure that all stakeholders agree to the SLAs Set the “weight” of each SLA Make sure the SLA is entirely within the control of the

contractor (no mixed responsibility) Add further definition of the SLA and all exclusions


18


Develop the methodology of how to measure performance Periods (monthly, quarterly, bi-annually) Set Levels (Stretch, primary, secondary, tertiary) Set Measurement Scale with Incentive / Disincentive

– For Example: Stretch: 100% + rollover Primary: 100% of available fee Secondary: 90% Tertiary: 75% Failure: 0 % Fee

Not all levels are required. Some only have one target. It depends on the client requirements.


19


Award Fee Plan scale:

– Outstanding (95-100)– Excellent (90-95)– Good (80-90)– Average (70-80)– Poor (0-70)

SLA Measurement Scale:

– Outstanding 99.99+

– Excellent 99.9 – 99.99

– Good 99 – 99.9

– Average 98 – 99

– Poor >98


20

Process (a few months post award)

Hold a regular review of the metrics to ensure that the levels are set correctly

Create a dashboard for simplified reporting to executive management

Set up a quality assurance function to check the contractor’s self reporting metrics

Insist on access to the raw data from the system tools used to generate the SLA reports

Review all trends and focus on failures Root cause analysis can identify process issues


21

Process (a few months post award)

Set a structure for gradual improvement of SLAs over time

For particularly difficult SLAs to negotiate, consider benchmarking them for a period without making them SLAs to obtain buy-in and set the appropriate target

Gradually increase the overall weight of SLAs in the award fee plan overtime, with the goal of eliminating subjectivity as much as possible


22

Exceptions

Scheduled Downtime Government Intervention in Contractor Response Other Contractor Intervention Acts of God / Force Majure

Make sure someone from the Government is aware of and approves all scheduled downtime or deviations from SLAs. No fox in the hen house!

Another benefit of outsourcing your infrastructure to the “cloud” . . . No more scheduled downtime!


23

Measurement

• Recommend use of Contractor owned automated tools• If not owned by the contractor and proposed at time of

award, be prepared to pay for the tool• Some clients have their own tools. Make sure the client can

extend the license to contractors• Use an IV&V contractor to sample the data from the contractor• Send the monthly results of the SLAs to stakeholders so there

are no surprises when it comes time to pay the contractor


24

Structure & Rollover

Design your SLAs to motivate improved performance on SLAs overtime.

If one SLAs is missed during a period, put some rollover on the table to improve the performance against that measure for the future and earn back the lost fee. That will motivate the contractor to achieve a higher standard for the remaining period and result in better service to your client. Example: If a contractor misses a measure in Period 1, put

some rollover towards that measure (and that measure only) in Period 2 if they meet a stretch goal. You can build this into your award fee plan’s structure to happen automatically

Note: This is not always appropriate. Some failures need to have financial disincentives to change behavior.


25

Pitfalls

Having too many measures Example: Too many SLAs, some of which were redundant

and worth less than 1% of the pool Example: Aligning SLAs to Government staff. At least one

SLA (and sometimes more) for each of 25 tasks to appease stakeholders

Targets Just like performance reviews, if you set a target, the

contractor will plan and staff to meet that target. If the target is an average, make sure that they plan to make the average, not to always beat the target.


26

Pitfalls

Funding If the SLAs has an incentive, that amount must be fully

funded to prevent any anti-deficiency situation– This is what makes really good incentives difficult level of

oversight

Oversight Clients and Contractor may initially balk at the level of effort

required to stand up a SLA management process


27

Pitfalls

Reliance on Industry Standards Benchmarks are great, but that does not mean they are right

for your client’s infrastructure. Just because Gartner says five 9s is normal for a call center doesn’t mean that your requirement needs that.

Customer Satisfaction This one is difficult. It will take time to reach agreement on

what scale to use (1-5, Y/N, etc), what survey to use, how to count the no-response pool.

At the end of the day, customer satisfaction may be the most important measure but the hardest to get right


28

Pitfalls Tendency to award very high scores on Award fee (Government wide)

94.6%

93.8%

96.1%

95.7%

92.5%

93.0%

93.5%

94.0%

94.5%

95.0%

95.5%

96.0%

96.5%

FY02 FY03 FY04 FY05

% Awarded on DOD


29

Questions

Documents

Federal Acquisition Service U.S. General Services Administration Performance Metrics Chris Hamm Operations Director (acting) GSA FEDSIM