SCO Payroll Assement

7/27/2019 SCO Payroll Assement

1/10

August 2013

California State Controllers Office

Preliminary Internal Review

of the

21st Century Project


2/10

August 12, 2013 1

Preliminary Internal Review

The State Controllers Office (SCO) is responsible for operating the States existing payroll systems. KPMG

Peat Marwick audited the States existing payroll systems as part of a 1995 assessment of the SCOs operations.

KPMGs audit report concluded the State should replace the payroll systems because they are obsolete. The

audit confirmed the systems are decades old, written in an outdated computer language and do not have the

flexibility and functionality to meet the needs of todays state government.

The 21st Century Project became a focus of the States attention after the issuance of the 1995 KPMG audit

report. In 2004, after delays due primarily to budget constraints, the SCO received funding to work with other

state stakeholders to purchase and implement a software suite to replace the States existing employment,

payroll, timekeeping, leave, and position control functionalities.

The SCO and other stakeholders spent considerable time on developing the project procurement process, both

because of the fundamental importance of payroll to the States work, and because the State recognized that

successful statewide technology implementation changes require thoughtful planning, procurement and contract

strategies, coupled with practical business knowledge, technological expertise, implementation methodology

and project management.

The SCO brought its business expertise and knowledge associated with successfully operating the States $20

billion annual payroll to the 21st Century Project, including a practical understanding of the complexities of

ensuring the States payroll is compliant with federal and state laws and regulations, the terms of state collective

bargaining agreements and memorandums of understanding, and related business requirements.

After the first system integrator was unable to perform, the SCO engaged in an extensive re-procurement

process and ultimately selected SAP to be the system integrator. The objective in selecting a new system

integrator was to hire the best-qualified candidate to successfully complete the job of configuring SAPs

commercial off-the-shelf (COTS) payroll software to ensure that Californias payroll continued to be properly

determined. Payroll is properly determined when it is consistent with federal and state laws and regulations, the

terms of state collective bargaining agreements and memorandums of understanding, and related business

requirements. Unfortunately, over time, SAP proved itself unable to meet the requirements of the 21st Century

Project, and SAPs failure led to its termination.

As the system integrator and per contract requirements, SAP was responsible for planning, implementing, and

executing a successful data conversion strategy. Their strategy did not include a data conversion tool that was

sufficient to perform data validation or to convert the volume and complexity of data required. Without that

tool, SAP was unable to perform its responsibilities and the project was delayed. After a cure notice was servedupon SAP on August 19, 2011, SAP took steps to secure an effective tool.

After resolving the data conversion issue, a new plan was developed for the project with a small pilot of only

1,400 employees scheduled for June 2012. After go-live significant errors surfaced during the first payroll


3/10

August 12, 2013 2

cycle and persisted through each of the subsequent eight monthly payroll cycles. SAP was never able to

stabilize the system even for this small set of employees.

SAPs termination resulted from its delivery of an unstable payroll system plagued with errors. SAP could not

explain or fix many of the errors, and was unresponsive to the States request that it focus on identifying the

root causes of these errors and then create a corrective action plan to demonstrate how SAP would address them

so the State could have confidence that the 21st Century Project should go forward.

SAPs lack of responsiveness to the States significant concerns lead the SCO to issue a cure notice demanding

that SAP address the payroll systems defects. SAP responded to the cure notice with an unequivocal denial

that the payroll system had any issues. SAP then went on to allege that if there are issues with the payroll

system, they are all the fault of the State. As a consequence, and taking into account contractual remedy

limitations and other considerations, the SCO terminated SAPs contract in February 2013. SCO employees

being paid through the SAP-built payroll system were rolled back to being paid on the States legacy payroll

systems.

What Should Happen Next?

The SCO is focused on completing the following tasks:

Complete payroll reconciliation (also known as Do the Math) to ensure the SCOs 1,400 employeesand its business partners are made whole;

Support the States legal pursuit of full contractual restitution from SAP for its failed work; and Reduce MyCalPays maintenance costs.

In addition, the SCO believes the State should move forward with an independent assessment(s) to answer the

following questions:

What project work to date is salvageable and could be leveraged as part of a go-forward California plan? Can the SAP system meet the States requirements? Can the State simplify and modernize its payroll practices? What options and alternatives exist in terms of completing the replacement of the payroll system, along

with a cost-benefit analysis for each?

What lessons can be learned from the project to improve the States information technology projectmanagement practices?

What is the States best path forward to achieve its original goal of modernizing its human resources andpayroll systems?

SCO is working with the Department of Finance and Department of Technology to ensure the independent

assessment is completed. To start this process, below the SCO provides its preliminary internal review of the

project and what can be learned from it.


4/10

August 12, 2013 3

Preliminary Review of What Went Right

1) The project followed state guidelines, best practices and industry standards related to projectmanagement.

In managing MyCalPAYS, the project adhered to the general framework and processes contained in the

Californias Statewide Information Management Manual (SIMM) guidelines and Project Management

Methodology (PMM). The PMM is consistent with the Project Management Body of Knowledge(PMBOK) standard issued by the Project Management Institute.

The 21st Century Project Management Plan consists of nine components: Risk, Issue, Schedule, Change

Control, Document, Status Reporting, Quality, Requirements Traceability Management plans and the

projects Governance plan. Each of these components was managed pursuant to the PMM. In addition,

the States Project Management Office (PMO) maintained internal Deliverable, Contract and Cost

management plans for overseeing the contract work.

2) The project established strong formal governance and four levels of oversight.Consistent with the States best practices, the project had formal governance and oversight in place.

Project governance included the PMO, Project Leadership Team, Operational Steering Committee,

Executive Steering Committee, and Project Sponsor in the decision-making process.

The project is governed by an Executive Steering Committee, chaired by the State Controllers Office

(SCO) and composed of voting members from the Business Transportation and Housing Agency (BTH),

California Environmental Protection Agency (CalEPA), California Natural Resources Agency, State and

Consumer Services Agency (SCSA), Labor and Workforce Development Agency, Department of

Human Resources (CalHR), Department of Finance (DOF), California Health and Human ServicesAgency (CHHS), and Department of General Services (DGS) and non-voting members from the

California Technology Agency and SCO project leadership.

The Legislative Analysts Office (LAO) also provided periodic oversight reporting to the Legislature.

In addition, general project oversight was provided by the Independent Validation and Verification

(IV&V) and the Independent Project Oversight Reporting (IPOR) processes, both with contractors and

consultants embedded in the project.

These leadership, decision-making, and oversight bodies and organizations were kept abreast of project

status, issues and risks on a regular basis through defined governance, meetings, and status reporting.

3) The project identified critical system defects in Pilot 1 and stopped deployment, therebypreventing additional impact to employees.


5/10

August 12, 2013 4

The two-pilot and three-wave deployment strategy was designed to limit the risks of any state payroll

issues to as small a group of employees as possible by ensuring the project did not progress unless the

current pilot was complete and stable.

This mitigation strategy prevented harm to additional state employees. It limited the risks of the payroll

disruption experienced by many of the 1,400 SCO employees in Pilot 1. Further, SAP was not permitted

to start Pilot 2 despite its repeated requests to do so when it could not remedy the issues in the Pilot 1

payroll phase. SAP was ultimately terminated because it could not or would not address the Pilot 1errors.

4) The project used an FDA-like gate approach to ensure that risk is contained and that keyfoundational activities are completed before advancing.

To manage the risk associated with a statewide implementation, the MyCalPAYS system was designed,

configured, developed and tested to help assure scalability by tackling the each design issue early in the

project lifecycle. The system would be tested at each stage, and then rolled out to incrementally larger

populations in Pilot 1 and Pilot 2 to limit the impact to users and employees should there be significant

issues. After stabilizing Pilot 1 and moving to Pilot 2, there was a planned six-month stabilization effort

before Wave 3 and deployment to an estimated 51 departments and 78,992 employees. A three-month

stabilization period was provided for Wave 3, and then Wave 4 would be implemented to five

departments and 79,003 employees. Another three-month stabilization period was provided for Wave 4

before deploying the system to Wave 5, which would comprise the final deployment wave of 118

departments and 61,910 employees. Another three-month stabilization period was provided for Wave 5

and then the system would be accepted, assuming that it met all of the acceptance criteria.

Figure 1

Figure 2

Conceptual

Vision

FSR

Gate

Pre Live

Research and

Development

Small Scale

Live Pilots

Large Scale

Deployment Gate

State Officially

Accepts and

Replaces Legacy

Gate Gate

,&

, IVV, ,MyCalPAYS Project Development and Approval Process

Drug

Conceptual

Vision

Gate

Pre-Clinical

Research and

Development in

Laboratory

Clinical Trial

With Live

Subjects

Large Scale

Clinical TrialGate

FDA Approval and

Manufacturing

Begins

Gate Gate

,&

, IVV, ,FDA Pharmaceutical Development and Approval Process


6/10

August 12, 2013 5

Figure 1 is the MyCalPAYS Project Development and Approval Process and Figure 2 is the FDA

Pharmaceutical Development and Approval Process. These high-level processes are similar in that they both

require initial research and approval, and a structured and defined development process with significant

laboratory testing before the product or system can move to a small-scale live pilot. Success along the way by

meeting defined acceptance criteria means that the product or system can move to the next phase. The Small

Scale Live Pilots (Pilot 1 in the MCP example) serves to validate that all of the laboratory testing is correct and

accurate in a limited live environment before introducing it into a larger environment (Pilot 2). This serves toreduce the risk or exposure in a larger population. Both processes apply management disciplines designed to

minimize the risk of implementation to a larger population. Failure at any part of the process requires stopping

and analyzing what has gone wrong, if it can be fixed, and if so, a plan to move forward.

Preliminary Review of What Went Wrong and Lessons Learned

When appropriate, this section also offers preliminary lessons learned that can be applied to the States $5

billion information technology portfolio in order to help other projects avoid similar pitfalls and increase the

probability of success.

1) The project needed more robust contractual carrots and sticks to manage the vendor.The contract lacked effective tools to manage vendor performance, such as liquidated damages,

performance related penalties, or a performance bond.

Lesson Learned: These provisions can be useful in focusing a vendors attention on its performance.

2) Public Contract Code section 6611 procurement guidelines were not in place in 2009.The SCO piloted a two-stage procurement using this section as an essential and key component of

completing a major procurement in an accelerated time frame (one year). Due to the fact that PCC

section 6611 had never been used in an information technology procurement, the Department of General

Services (DGS) had no specific guidelines or procedures for conducting a procurement and negotiation

under this section. As a result, the State was limited in its ability to thoroughly vet and communicate

with the potential bidders for fear of protests accusing favoritism.

Lessons Learned: Guidelines could have helped the State achieve a better balance between protecting

against protests and claims of favoritism and being able to adequately assess bidder qualifications. In

April 2013, DGS adopted guidelines for this code section.

3) The contract payment structure was not designed to align the States and the vendors goals.


7/10

August 12, 2013 6

Because payments were made based on the completion of individual tasks, such as completing a test or

crafting a schedule, SAP keyed in on these activities, rather than focusing on the States overall goal of

delivering a reliable and accurate payroll system.

Lessons Learned: Restructure payments so they are tied to system progress at rationally-defined

milestones (e.g., accurately paying 1,400 employees in Pilot 1). This would likely focus the vendor on

the States project goals, rather than incremental billing.

4) SAPs testing methodology was inadequate.This factor above all others played very heavily into SAPs termination. System testing was

insufficient, as evidenced by the number and types of system defects in Pilot 1. The responsibility for

testing rested squarely with SAP, as identified in the Statement of Work, which requires SAP to, among

other things, provide the test team, tools, methodology, proper test coverage, and comprehensive test

plans for the MyCalPAYS (MCP) solution, including thorough testing of all interfaces. The State

trusted SAPs representations about its qualifications and unique integration experience and believed it

would provide quality work. Yet, SAPs testing proved substandard. The deficiencies in testing only

became apparent when Pilot 1 went live.

Payroll comparison testing was also inadequate. At the completion of Payroll Comparison Test 3,

SAP assured the SCO that employees pay was correct within the range of $1 and that differences in the

test outside of this tolerance were accounted for by known differences that could be addressed. This set

the expectation that payroll would be accurate within $1 with the exception of known differences. At

the time, the Payroll Comparison Test results were delivered and accepted, SAP assured the SCO that

the payroll would be accurate and the SCO relied upon SAP as being uniquely qualified as the developer

of the software to perform sufficient testing. These expectations were not met.

Lessons Learned: To ensure better checks and balances, require a skilled independent party to design,

execute, and validate sufficient testing. The use of a separate team to review the system functionality

adds rigor and a formal testing discipline which can lead to improved quality. An independent

assessment can ensure the system, people, and processes are ready to move into production as a

mandatory gate to moving the system into a production environment.

5) SAP couldnt keep pace with fixing the system defects.Once Pilot 1 went live, far more system errors occurred than were ever anticipated. SAP executives

seemed perplexed and could not explain why the errors were not identified in the numerous test cycles.

SAP could not keep pace with fixing the system errors, which it was required to do under the contract.

SAP could not determine the root causes of the system errors or implement a corrective action plan to

address the problems encountered in Pilot 1.

The chart in Figure 3 depicts the volume and type of new system defects incurred every month in Pilot 1


8/10

August 12, 2013 7

plotted against the cumulative number of open (unresolved) defects. Thus, every month a new baseline

was established which included the current and the preceding months unresolved defects.

Figure 3

6) SAP could not or would not disclose root causes of Pilot 1 defects.SAP informed the State that a special team of experts deployed to do a root-cause analysis of the

problems in Pilot 1 had gathered 200 pages of findings. However, it would not provide those documents

to the State. When the State pursued this request, SAP provided an abbreviated document that contained

insufficient information to determine the root cause of Pilot 1 errors, or how they could be resolved.

Lessons Learned: Without its own escalation team, the State relied on SAP to do the root-cause analysis

for Pilot 1 problems. However, SAPs team appeared to be focused on asserting that its software was

not at fault and did very little to resolve outstanding issues and get the project back on track. The State

could have benefited from commissioning its own analysis instead.

7) SAP failed to provide an operations framework for Pilot 1.SAP failed to deliver its assessment of the SCO support model, provide an operations design framework,

provide effective assistance to the State with setting up incident management, provide the knowledge


9/10

August 12, 2013 8

transfer required, and the production support processes required to operate the system. SAP failed to

staff production support sufficiently to keep pace with resolving the volume of system defects identified,

thus causing additional payroll errors every time payroll was executed.

8) SAPs blueprint validation process proved to be insufficient to design a system to produceaccurate paychecks.

The statement of work required SAP to validate the States business requirements and confirm theircompleteness. In its proposal, SAP confirms that it is responsible for this role: During blueprint

validation, SAP consultants work with the various state business owners to define business requirements

fully so appropriate designs can be engineered that will provide a foundation to avoid future problems.

Because this validation effort proved to be insufficient, SAP came out of this important phase with an

incomplete understanding of the States business requirements to design the system.

Consequently, SAP apparently assumed that a number of States business requirements would fit within

the standard software functionality. Only through later deeper dives into the States requirements and

analysis did SAP discover that its standard functionality could not comply with Californias legal and

labor agreement requirements. This necessitated customizations and changes to the software. SAP

claims that this is a result of attempting to build a system just like the States legacy payroll systems.

However, SAP misses the point. The new payroll system must meet applicable legal and contractual

compliance requirements.

9) SAP failed to provide the staffing needed to successfully deliver a project of this size andmagnitude.

Out of the 17 key positions identified in the SAPs procurement proposal, only one key resource

remained on the project from the time the bid was let in February 2010 through the suspension inFebruary 2013. The staff that was provided failed to perform required project roles and work activities,

and failed to manage their respective teams, pursuant to the contract. SAP's resources should have

leveraged the experience SAP represented it had on other large-scale public sector projects, and

provided expertise on best practices and key lessons to reduce risk and ensure success. Moreover, SAP

failed to consistently maintain the continuity of staffing with the requisite experience required in the

SOW.

Lessons Learned: The State needs additional means to ensure the vendor provides the level and quality

of staffing proposed in its bid.

Conclusion

The SCO believes SAP failed because it was not committed to the same objectives as the State, despite a

uniquely transparent procurement process, jointly-agreed upon contract deliverables, SAPs assurances that it

understood the 21st Century Project better than any other potential integrator and SAPs assurances that it had

never failed to successfully complete a troubled project.


10/10

August 12, 2013 9

The State believed and relied on SAPs representations about its qualifications and unique expertise to be

successful throughout the pre-production work on the project in areas requiring special COTS software

expertise. Ultimately, however, SAP delivered an unstable payroll system that is plagued with material errors

and proved to be a non-responsive contractor that could not fix the errors, determine the root cause of the errors,

or create a corrective action plan to go forward. The combination of all these factors prevented success and,

ultimately, the SCO had no option but to terminate SAP due to its lack of performance.

Documents

SCO Payroll Assement