Upload
vudiep
View
214
Download
0
Embed Size (px)
Citation preview
1
R A U L S O T O , M S C , C Q E
I V T A P R I L 2 0 1 7 S A N D I E G O
SPREADSHEET VALIDATION 101: CREATE AND VALIDATE FDA-COMPLIANT
MS EXCEL SPREADSHEETS
ABOUT THE AUTHOR
• 20+ years of experience in the medical devices, pharmaceutical, biotechnology, and consumer electronic industries
• MS Biotechnology, emphasis in Biomedical Engineering
• BS Mechanical Engineering
• ASQ Certified Quality Engineer (CQE)
• I have led validation / qualification efforts in multiple scenarios:
• High-speed, high-volume manufacturing and packaging equipment
• Laboratory : information systems and equipment / instruments
• Enterprise resource planning applications (i.e. SAP)
• IT network infrastructure
• Software applications
• Product improvements, material changes, vendor changes
• Contact information:
• Raul Soto [email protected]
2
TABLE OF CONTENTS
1. REGULATORY PERSPECTIVE
2. SPREADSHEET VALIDATION
3. BUILDING A COMPLIANT SPREADSHEET
4. HANDS-ON EXERCISES
SCOPE
SPREADSHEET
• Microsoft Excel file that may contain:
• mathematical calculations
• Statistical analysis
• Graphing
• Outside our scope:
• Macros or custom code
• Database lookups
4
5R E G U L A T I O N S G O V E R N I N G S P R E A D S H E E T S
PART 1: REGULATORY
PERSPECTIVE
RAUL SOTO
IVT APRIL 2017 SAN DIEGO
WHICH SPREADSHEETS NEED TO BE
VALIDATED?
• Companies have received observations from FDA for not
validating spreadsheets used for:
• In–process analytical calculations
• Finished Product analytical calculations
• Tracking and trending quality metrics
• Databases / list
6
FDA 483
Pharmaceuticals (Eli Lilly):
FDA WARNING LETTERS
http://www.contractormisconduct.org/ass/contractors/18/cases/305/221/f
da-warning-letter.pdf
Pharmaceuticals (Cardinal Health):
FDA WARNING LETTERS
http://www.fda.gov/ICECI/EnforcementActions/WarningLetters/2013/ucm
369409.htm
Pharmaceuticals (Aarti Drug Limited):
FDA WARNING LETTERS
http://www.fda.gov/ICECI/EnforcementActions/WarningLetters/2006/u
cm075766.htm
Biologics (National Genetics Institute):
FDA WARNING LETTERS
11
http://www.fda.gov/ICECI/EnforcementActions/WarningLetters/2011/ucm2
56498.htm
Medical Devices (Aubrey Inc.):
FDA WARNING LETTERS
http://www.ofnisystems.com/media/g3667d.pdf
Medical Devices (B. Braun Medical Ltd):
FDA WARNING LETTERS
http://www.ofnisystems.com/Resources/Warning_Letters/g4452d.pdf
Medical Devices (Eldon Biologicals A/S):
FDA WARNING LETTERS
http://www.ofnisystems.com/media/g1483d.pdf
Medical Devices (EP Medsystems):
MEDICAL DEVICES
• 21 CFR 820.70(i): Automated processes
• When computers or automated data processing systems are
used as part of production or the quality system, the
manufacturer shall validate computer software for its intended
use according to an established protocol.
• All software changes shall be validated before approval and
issuance.
• These validation activities and results shall be documented.
15
MEDICAL DEVICES
• Let’s take this apart:
• Applies to computer / data processing systems used for
production or quality
• The manufacturer is responsible for validation
• Validation must be for the intended use(s)
• Must have an established protocol
• Changes must be validated before implemented
• Document all validation activities and results
16
MEDICAL DEVICES
• Section 6.0
• “Many other commercial software
applications, such as word processors,
spreadsheets, databases, and
flowcharting software are used to
implement the quality system. All of
these applications are subject to the
requirement for software validation, but
the validation approach used for each
application can vary widely.”
17
MEDICAL DEVICES
• Section 6.1
• “Numerous commercial software
applications may be used as part of the
quality system (e.g., a spreadsheet or
statistical package used for quality system
calculations, a graphics package used for trend analysis, or a commercial database
used for recording device history records or
for complaint management). The extent of
validation evidence needed for such
software depends on the device
manufacturer's documented intended use of that software.”
18
PHARMACEUTICALS
• 21 CFR 211.165(e): Testing and Release for Distribution
• (e) The accuracy, sensitivity, specificity, and reproducibility
of test methods employed by the firm shall be established
and documented. Such validation and documentation
may be accomplished in accordance with 211.194(a)(2).
19
BIOLOGICS
• 21 CFR 606.140: Laboratory Controls
• Laboratory control procedures shall include:
• (a) The establishment of scientifically sound and appropriate
specifications, standards and test procedures to assure that
blood and blood components are safe, pure, potent and
effective.
• (b) Adequate provisions for monitoring the reliability,
accuracy, precision and performance of laboratory test
procedures and instruments.
• (c) Adequate identification and handling of all test samples so
that they are accurately related to the specific unit of product
being tested, or to its donor, or to the specific recipient, where
applicable.
20
ISO 13485DESIGN AND MANUFACTURE OF MEDICAL DEVICES
• Clause 7.5.2.1
• The organization shall establish documented procedures for
the validation of the application of computer software (and
changes to such software and/or its application) for
production and service provision that affect the ability of the
product to conform to specified requirements.
• Such software applications shall be validated prior to initial
use.
21
ISO 13485 : 2015DESIGN AND MANUFACTURE OF MEDICAL DEVICES
• Will explicitly mandate:
• That software used as part of the quality system must be
validated and subject to change control.
• That we must have procedures to govern this
• That validation must be documented and the documentation
maintained.
22
FDA LAB MANUAL OF QUALITY PROCEDURES
• http://www.fda.gov/ScienceResearch/FieldScience
/LaboratoryManual/ucm174286.htm
23
24
4.5.3 VALIDATION OF SPREADSHEETS
SUMMARY
• Lock all non-input cells
• Make spreadsheets read-only, with password protection
• Automatically reject data outside valid ranges
• Positive and negative testing: Test by entering expected and
extreme values, and nonsensical data
• Keep record of all cell formulas
• Document changes made to the spreadsheet
• Change control with version numbers
• Periodic re-validation spreadsheets, including
• verification of cell formulas
• manual verification of spreadsheet calculations25
26V A L I D A T I O N P L A N , E X E C U T I O N , R E P O R T I N G
PART 2: VALIDATING THE
SPREADSHEET
RAUL SOTO
IVT APRIL 2017 SAN DIEGO
VALIDATING YOUR SPREADSHEET
• validation, software. (NBS) Determination of
the correctness of the final program or
software produced from a development
project with respect to the user needs and
requirements. Validation is usually
accomplished by verifying each stage of the
software development life cycle. See:
verification, software.
• Source: FDA Glossary of Computer System Software
Development Terminology
http://www.fda.gov/iceci/inspections/inspectionguides
/ucm074875.htm
27
GOVERNANCE
• Spreadsheet Validation Procedure• Outlines HOW spreadsheets will be validated
• What document(s) need to be produced
• Roles and responsibilities, Authors and Approvers, etc.
• Templates (i.e. Spreadsheet template)
• Change Control Procedure
• May be a separate procedure, or part of the main
Spreadsheets Validation Procedure
• Validated Spreadsheet Inventory
• Training28
VALIDATED SPREADSHEETS INVENTORY
• You need to know which validated spreadsheets are being
used, and where.
• Inventory document should contain:
• Spreadsheet Name and number
• Current Version
• Area(s) where used
• Owner
• Location
• Protected network folder address
• eDMS
29
TYPICAL VALIDATION ELEMENTS
• Validation Plan
• Scope
• Requirements Specification
• Design Specification
• Traceability Matrix
• Testing
• Development
• Installation Qualification
• OQ / PQ -or-
• System / User Acceptance
• Validation Report
• Procedures & Governance
• Templates
30
You can create individual
documents for each deliverable
-OR-A single Validation Protocol that includes ALL validation
elements
VALIDATION PLAN SCOPE
• Spreadsheet name
• Description and Purpose
• What the spreadsheet will be used for
• Process(es) supported
• Where it will be used
• References
• Analytical methods, SOPs, etc.
32
REQUIREMENTS
• Describes, with detail, WHAT you want the spreadsheet to do.
• Requirements should be
• Numbered
• Specific
• Testable
• Traceable
• MS Excel version required
• Excel Add-Ins required, if any
33
REQUIREMENTS
• Input / output cells and formula cells requirements
• Formatting (colors, font size and type, etc.)
• Data validation (field type, size, etc.)
• Expected input data and ranges, per input cell
• Expected output result, per output cells
• Any unit conversions required
• Descriptions for every formula
• Source (USP, literature, SOP, etc.)
• Page formatting
• Titles
• Headers
• Footers 34
REQUIREMENTS
• Charts and Graphs
• Title and Description of each
• Required labels for each axis
• Security requirements
• Password protection
• Output requirements
• Color printer, PDF-creating software
• Specific folder in a specific shared drive
• Automatic eMail to specific people
• Watermarks
35
DESIGN SPECIFICATION
• Describes, with details, HOW the spreadsheet will
comply with the Requirements
• Address all requirements, individually or by groups
• Describes the actual spreadsheet as written
• Shows actual EXCEL version of formulas
• Describes algorithms used, if any
• Describe conditional cell formatting used
• Describe data validation used
• Describe outputs produced (i.e. graphs, tables, etc.)
• OS in which the spreadsheet was developed
• i.e. MS Windows 7 SP1, Mac OS X 10.10
36
TRACEABILITY MATRIX
• Traces every requirement to its
corresponding design
element, and to the test
script(s) where the
requirement is challenged and
tested.
37
DEVELOPMENT TESTING & DRY RUNS
• Recommended!
• Typically not documented formally
• Dev Testing helps discover bugs and fix them while we’re not
yet in formal validation.
• Dry Runs help discover issues with the validation test scripts,
and allows us to fix these before we move into formal
document change control.
• Use formula auditing!
38
INSTALLATION QUALIFICATION (IQ)
• IQ includes Installation and Configuration
• State all prerequisites, ensure they are met prior to
execution
• List all spreadsheet file(s) to be installed
• List network location and path where spreadsheet files will
be installed
• Development environment – does not have to be
validated
• Test environment – may need to be validated
• Production environment – must be validated
39
OPERATIONAL QUALIFICATION (OQ)
• State all prerequisites, ensure they are met prior to
execution
• Verify that all requirements documented in the
Requirements section are met
• Ensure all outputs have the correct formatting
• Provide printouts of all formulas, input values used, output
results.
• Positive v Negative testing !!!
• Security testing
• Performance testing
• Simulate output issues
40
OPERATIONAL QUALIFICATION (OQ)
• Ensure all non-input cells are locked
• Ensure conversion factors are correct
• Ensure spreadsheets are read-only, with password protection
• Ensure the spreadsheet automatically rejects data outside valid
ranges
• Test for rounding errors
• Ensure Excel formulas match the source formula
• Ensure formula input / output cells are correct
• Compare formula results vs manual calculations
• Ensure units and spec limits are present and correct
41
OPERATIONAL QUALIFICATION (OQ)
42
• use the INFO command to provide information about the execution
environment.
OQ: POSITIVE VS NEGATIVE TESTING
• Positive testing : test and verify every single formula using
values that should produce correct results
• Ensure the spreadsheet produces correct results for
various input data sets
• i.e. at least two data sets
• Data set 1: known to result in a PASS decision
• Data set 2: known to result in a FAIL decision
• Include test cases with borderline data
• Verify spreadsheet results against results from manual
calculations
43
OQ: POSITIVE VS NEGATIVE TESTING
• Negative testing :
• Use incorrect inputs to ensure the
spreadsheet is robust, able to
handle them without crashing or
producing false or incorrect
outputs, and that it is not possible to
bypass security
44
OQ: POSITIVE VS NEGATIVE TESTING
Negative testing examples:
• Required data entry : ensure input fields cannot be left blank
• Field type test: ensure data entered is of the correct type (i.e. dates, alphanumeric, etc.)
• Enter alpha or special (@%^#) characters on a numeric field
• Enter a decimal number in an integer field
• Enter negative numbers in a positive numbers field
• Enter nonsensical inputs (i.e. ASCII CRTL codes)
• http://academic.evergreen.edu/projects/biophysics/technotes
/program/ascii_ctrl.htm
• Field size test: ensure user can only enter the specified number of characters in a field
45
OQ: POSITIVE VS NEGATIVE TESTING
Negative testing examples:
• Date bounds test: test upper and lower bounds for dates.
• Numeric bounds test: for example, if the number in a field should be between 10 and 50, the system should not accept numbers
lower than 10, and higher than 50.
• Numeric limits test: make sure the spreadsheet doesn’t crash or produce incorrect output when entering a number outside of
Excel’s limits:
• Excel 2013 limits:
• https://support.office.com/en-za/article/Excel-specifications-and-
limits-ca36e2dc-1f09-4620-b726-67c00b05040f
46
OQ: POSITIVE VS NEGATIVE TESTING
• For a good discussion, with
examples, of Positive vs Negative
software testing, please refer to:
• http://www.softwaretestingclass.co
m/positive-and-negative-testing-
in-software-testing/
48
OQ SECURITY TESTING
• Security testing:
• Ensure it is not possible to:
• Unlock locked cells / worksheets
• Alter formulas, output results, output
graphs /charts/ tables
• Add or remove worksheets
49
OQ PERFORMANCE TESTING
• Output issues : ensure the spreadsheet behaves as expected
when you
• Disconnect printer
• Force-close the PDF-rendering application
• No data loss even if it crashes
• Performance :
• Can it handle multiple users simultaneously?
• Open multiple instances of the spreadsheet at the same
time
• Execute spreadsheet while various other applications are
running
50
OPERATIONAL QUALIFICATION
• Formula Auditing
• Excel’s Formula Auditing tools allows us to demonstrate that
the inputs come from the expected cells
51
TEST EXECUTION
• Ensure test scripts are executed in
the appropriate sequence.
• Document deviations when actual
results do not match expected
results
• Take screenshots and printouts as
documented evidence
52
TEST EXECUTION
• Include all executed test scripts
• Include all screenshots, printed
outputs, manual calculations
• Post-approvals
53
CHANGE CONTROL AFTER VALIDATION
• Update Validation documentation
• Additional or changed requirements
• Corresponding design elements changes
• Update Testing
• New or updated test scripts
• Test new functionality (i.e. new/updated formula)
• Regression testing: ensure areas not changed still work
correctly
• Update traceability matrix
• Upon release, increase the spreadsheet Version number
• REMOVE ACCESS to the old version of the spreadsheet
54
CHANGE CONTROL
• Spreadsheets used for GxP processes should be subjected to
formal version control and validation.
• The organization’s change management system should be
used to document whenever we:
• Create a new spreadsheet for regulated / validated uses
• Make changes to a validated spreadsheet
• Retire/decommission a validated spreadsheet
• Move a validated spreadsheet to a different version of
Excel (i.e. from Excel 2007 to Excel 2010) from the one used
in development and validation
• Move a validated spreadsheet to a different operating
system (i.e. from Windows to Linux or MacOS)
55
VALIDATION: HOW MUCH IS TOO MUCH?
•Depends on the level of risk
56
57D E S I G N A C O M P L I A N T S P R E A D S H E E T
PART 3: BUILDING THE SPREADSHEET
RAUL SOTO
IVT APRIL 2017 SAN DIEGO
FDA’S INTERNAL GUIDELINES
FOR SPREADSHEET DESIGN AND VALIDATION
• Spreadsheet Design and Validation for the Multi-User
Application for the Chemistry Laboratory: Part I
• www.spreadsheetvalidation.com/pdf/LIB_Design_Multi-
User2A.pdf
• Spreadsheet Design, Verification and Validation, Use and
Storage of Single-User Workbook Files in the US FFDA
Laboratories: Part II
• www.spreadsheetvalidation.com/pdf/LIB_Part_II_Single-
User.pdf
58
COMMON SPREADSHEET ISSUES FOUND BY
FDA IN INDUSTRY
• Spreadsheet files not maintained, some deleted
• Rounding errors
• Formulas don't match (Excel vs source, Excel vs manual
calculation)
• Conversion factors issues
• Spreadsheet doesn't show specs or limits
• Unclear documentation
• Units not labeled
• Units: Weights described as volumes, areas as ratios
• Security and data integrity issues
• Cells not locked
• Regressions that inverted X and Y axis
59
IMPORTANT !!!
• Locking / unlocking cells
• Protect spreadsheets /worksheets
• Cell formatting:
• Data types
• Look and feel : Font, Colors
• Validation criteria definitions for input cells
• Define formulas
• Use of logical statements in formulas
• Use of Page Set-up
60
CREATING THE SPREADSHEET
• Verify formulas against sources
• Use correct units of measurement
• Lock all non-input cells
• Disable column/row add/remove
• Unlock input cells
• Distinguish input, calculation (formula) vs output cells
• Cell background color
• Font : bold, italics, size, color
• Use conditional formatting on output cells
• Different cell background color and/or font color for results
that pass vs results that fail
61
CREATING THE SPREADSHEET
• Worksheets
• Main
• Revision History
• Any data tables needed
62
CREATING THE SPREADSHEET
• The following information (at minimum) should be included on
the main worksheet:
• Spreadsheet Name
• Version number
• Date of last modification
• Every printout should contain
this information, plus
• Date / time of use
• User
63
CREATING THE SPREADSHEET
• Add a field where the user will enter his/her name every time
the spreadsheet is used.
• Use Page Setup to print the following information:
• Spreadsheet filename
• Correct date and time
• Page numbers and total pages
• Make the spreadsheet read-only
64
REVISION HISTORY
• Each spreadsheet should have a Change History worksheet
(a.k.a. Revision History)
• Every time the spreadsheet is updated, an entry should be
made in this Change History:
• New version number
• Date of change
• Change control number
• Name of person(s) who made the change
• Itemized description of ALL changes made (with DETAILS)
65
UNITS CONVERSION
• You can use the Excel CONVERT function to convert values
from one unit to another in your formulas
• Click HELP in Excel for a list of all units included.
66
SECURITY LAYERS
• Lock all cells except user input cells.
• HOME tab/ CELLS group/ Format / Protection
• Password-protect the worksheet
• File / permission / encrypt with password
• Install the validated spreadsheet in a protected folder on a
networked shared drive
• Access to the folder should only be given to authorized
personnel
• OR save the validated spreadsheet as a controlled document
in your EDMS
67
QUESTIONS
68
69
Thank You !
RAUL SOTO
IVT APRIL 2017 SAN DIEGO