Reading for Fun Official History VistA*/U.S. Department of
Veterans Affairs national- scale HIS Steven H. Brown, Michael J.
Lincoln, Peter J. Groen, Robert M. Kolodner International Journal
of Medical Informatics 69 (2003) 135/156 VistA Document Library
(VDL) www4.va.gov/vdl Slide 2 (With Pharmacy Additions) Richard
Pham, PharmD Enterprise Architect OI&T Corporate Data Warehouse
Architecture [email protected] Slide 3 The Health Care System Is
MUCH MORE COMPLICATED Than Other Business Processes (~7% of VistA)
Slide 4 Slide 5 Each Process Is A World Unto Itself Slide 6
DHCP/VistA/CPRS VistA Veterans Health Information Systems and
Technology Architecture - Refers both to the architecture and the
database which the architecture supports DHCP - Decentralized
Hospital Computer Program The DOS (Unix-like) system where many of
VistAs non-clinical entries take place CPRS - Computerized Provider
Record System A user- friendly GUI providing access to clinical
order entry functions Slide 7 Objectives The main objective is to
understand the data lifecycle of VAs VistA/CPRS and the user
experience of VistA/CPRS A high-level overview of VistA Internals
Learn about data structures and outputs in VistA Learn where data
enters and travels throughout the VA Try to make sense of data
resources within the VA and how they are accessed Slide 8 The VA
Data Lifecycle Slide 9 Slide 10 Core Patient Care Functionality
VistA is first and foremost an Electronic Medical Record. The
architecture design supports veteran health care. Slide 11 Core
Patient Care Functionality VistA Internals DHCP CPRS Slide 12 VistA
Internals 101 MUMPS Server and Operating System Kernel Three Wise
Men (Managers) TaskMan MailMan FileMan Modules Slide 13
Massachusetts General Hospital Utility Multi-Programming System
(MUMPS or M) My definition in English: M is a programming language
designed for hierarchical databases that is convenient for medical
applications or anything else where speed and data storage upkeep
are a problem and programmer intelligence/organization is not My
technical definition: M is a Turing-complete, low and high-level,
imperative, machine-compiled (no longer interpreted) programming
language utilizing a hierarchical global array file structure Used
commonly in healthcare and financial industry settings Slide 14
Structure of The Veterans Administration Data Efforts (Late 1970s)
VHA Ancestor Department of Medicine and Surgery (DMAS) VHA-OI
Ancestor Computer Assisted System Staff (CASS) OI&T Ancestor
Office of Data Management & Telecommunications (ODM&T)
Slide 15 Comparing The Two Offices CASS ODM&T Decentralized
design philosophy Rapid, agile development SME-involved development
Centralized design philosophy Bureaucratic, process-focused
development Development without SMEs Slide 16 Highlights of
ODM&T Development Took 6 years to deploy APPLES Pharmacy at 10
sites A 1980 paper detailing ODM&Ts transactional patient
treatment file (PTF) system promised an interactive national
solution by 1990. Navigating the mandated 17 steps between system
specification and deployment alone is said to have required at
least 3 years. Slide 17 Beginnings of DHCP There were subject
matter experts that believed that they could put out useful
applications faster than the ODM&T sloth Development of the
testing and principles was done unofficially throughout the late
1970s The Second Slide 18 Original DHCP Design Principles A
commitment to rapid prototype development All use ANSI MUMPS
Modular Design Actively Maintained Data Dictionary Code
Sharing/Portability Involve the SMEs Slide 19 DHCP Kernel Functions
as both an operating system for VistA applications and an M virtual
machine Kernel shields DHCP modules from needing to know hardware
and OS configurations on the server Isolates M to the ANSI standard
(1995) Provides a toolbox of standard functions for most
programmers Slide 20 VistA to Relational Database Terminology VistA
(Example)Relational Database (Example) Namespace (VHAFRE)Database
(VA Fresno) Package Not hardcodedSchema (RxOutpatient) File (50.68
VA PRODUCT)Table (NationalDrug) Field (.01 NAME)Column
(DrugNameWithDose) Domain (cardinal/decimal, setofcodes,
freetext/wordprocessing) Field Type (numeric, boolean, varchar)
Internal Entry Number (IEN or.001)~Key (9722) RecordTuple/Row
(ISOSORBIDE MONONITRATE 120MG TAB,SA) Slide 21 MUMPS Classic
Database One Data Type String (Text) Other types Cardinal Numbers
Float Numbers $H Dates One Data Storage Type Multidimensional Array
aka Globals Dynamic (duck) typing Slide 22 VistA Data Organization
Namespace File Field Record 654 (VAMC Reno) File 120.5 (GMR Vitals)
Field 0.1 (DATE/TIME VITALS TAKEN) IEN-1, BP, 140/90 Most Files
have an entry at the 0.001 Field called IEN or Internal Entry
Number as an identity key to mark the record as unique Slide 23
Upside of Using Globals Faster - No joins Faster All parameter
pointers built in Faster Direct and planned programmatic access to
database (Look at SQL execution plans) Less Data Storage Overhead
and faster paging If the data point does not exist in the array,
there does not need to be a fixed point like in relational Slide 24
Downside of Using Globals No Intrinsic Structure and No
Enforcement* - M believes whatever you put into the globals (most M
programmers view this as an advantage while relational programmers
have an MI) ACID-compliance not mandated (Il)logical data
structures guaranteed There are many interesting* ways that the M
programmers modeled the data that does not make sense to later
viewers Slide 25 MUMPS Quirks Whitespace (Space) matters Requires
knowledge of kernel and sometimes lower- level concepts Programming
Without Type or Structure Enforcement VA programming standards and
conventions Slide 26 Slide 27 The Three Wise Men (Managers) TaskMan
The man(anger) that schedules tasks to the kernel MailMan The
man(anger) that messages between the user, TaskMan, and any other
two-way communication between packages FileMan The man(anager) that
controls internal file (data structure) interactions Slide 28
TaskMan TaskMan handles application processing: Creation of
application processing tasks Scheduling these tasks Monitoring
health/statistics of these tasks If kernel is the brain, then
TaskMan is the body of the operation If programming, NEVER EVER use
the TaskMan global. This subverts TaskMans scheduling queue, and
can cause a system memory leak. Use the calls instead Slide 29
MailMan VistA needs a way to pass and receive data from the
database to other areas MailMan fulfills this function in the
pre-TCP/IP days Electronic mail doesnt mean just email Practically
any message between the database and anyone else (the end-user,
another site, or application, etc.) can be moved this way Gives
programmers methods to both receive and return data to the database
Slide 30 FileMan A higher-level method to access the VistA database
without exposing a programmer interface Mostly menu-driven One can
use limited programming Serves as the model for all other modules
that interact with the VistA database Slide 31 ODM&T Initial
Action Plan To DHCP Development (1980) Ordered that development
stop Fired the developers Removed the hardware Cut the DMAS budget
so it would never happen again Slide 32 The official history Slide
33 Development Goes Underground Developers that survived the
ODM&T purge continued their work as a black project in DMAS
During 1980 and 1981, the survivors (Underground Railroad)
continued work on developing modules for system integration Slide
34 Modules Modules are programmed to interact with the VistA
database Most use Fileman as a model for programming Slide 35 Some
of the Many Modules MedicineSurgeryDentistryNursingPharmacy
LaboratoryCare Management Patient Care Encounters ADTMental Health
EDISOncologyNutrition and Food Service Imaging/PACSProsthetics Not
really in the scope of this presentation to cover each module. Try
the VistA Documentation Library: http://www4.va.gov/vdl/
http://www4.va.gov/vdl/ Or VHA eHealth University (VeHU):
http://www.vehu.va.gov/http://www.vehu.va.gov/ Slide 36 Acceptance
and DHCP 1.0 Once there was a critical mass of packages that were
shown to be useful, the tide turned and the project was blessed
Initial testing done in 1980-83 1.0 installation was in 1985 Most
of the underlying packages can still be recognized by the original
programmers Slide 37 Special Topic The Pharmacy Package (File 50
Series) Where are the files? File 50 Series is the main line,
though there are other places How many files are there? - 437 Look
at screen capture on next side for File 50 series. How many
columns? - 3174 Slide 38 How Many File 50 Tables Are There? (1096
659 = 437 Tables!!) Slide 39 How Many File 50 Series Columns Are
There? (8526 5352 = 3172!!) Slide 40 Further Information On The
Background For the VA Base M Training
http://vaww.vistau.med.va.gov/VistaU/MTraining/Def ault.htm
http://vaww.vistau.med.va.gov/VistaU/MTraining/Def ault.htm For the
VA Programming Standards and Conventions
http://vista.med.va.gov/sacc/ For the VA Document Library
http://vista.med.va.gov/vdl/ Slide 41 Computerized Patient Record
System (CPRS) A Real-Time Order Checking System that alerts
clinicians during the ordering session that a possible problem
could exist if the order is processed A Notification System that
immediately alerts clinicians about clinically significant events A
Patient Posting System, displayed on every CPRS screen, that alerts
clinicians to issues related specifically to the patient, including
crisis notes, warning, adverse reactions, and advance directives
The Clinical Reminder System, which allows caregivers to track and
improve preventive health care for patients and ensure timely
clinical interventions are initiated Remote Data View functionality
that allows clinicians to view a patients medical history from
other VA facilities to ensure the clinician has access to all
clinically relevant data available at VA facilities CPRS DOES NOT
STORE DATA!!! Slide 42 CPRS Internals Written in Embarcadero Delphi
(NOT in MUMPS) Connects from the Graphic User Interface to the
VistA database using a Remote Procedure Call (RPC) Broker This
Remote Procedure Call Broker translates instruction sets from other
languages into M Slide 43 Present State of VistA Large MUMPS
database Over 50+ Main Clinical Packages Over 10,000 + Tables Each
medical center runs somewhere between 2-4 TB worth of data over 30
years (mostly imaging) Many processes 300+ MB of running executable
at any given time Over 20,000 subroutines (VDL) Many simultaneous
users Slide 44 The VA Data Lifecycle Slide 45 National Analytic
Systems A list of systems that support policy, planning, and
congressional needs There are more extracts than this, but I have
chosen the most common ones Slide 46 46 Slide 47 Systems to Support
Planning Decision Support System (DSS) Supports accounting and
costing for the OIG, GAO, CBO, and other auditing agencies
Allocation Resource Center Supports personnel and resource
allocation at the medical center level Workload capture, resource
allocation Basis for the VERA (VAs Fund Control Point) Model Slide
48 VistA DSS Data Feeds ADMAdmissionsCLIClinic Visits DEN
DentalECSEvent Capture System IVPPharmacy: IVLABLab LARLab
ResultsMOVPatient Movement Inpatient MTLMental Health
TestNURNursing PASPatient AssessmentPREPharmacy OP
PROProstheticsRADRadiology/Nuclear Med SUR SurgeryTRTPatient
Treatment Specialty UDPPharmacy Unit Dose IP May be transmitted by
site at any time of the month, ideally around the 25 th of the
month prior to the processing month. Slide 49 Systems to Support
Research National Patient Care Database An integrated set of data
that captures a patients care encounter with the VA Corporate Data
Warehouse A near real-time accumulation of much of the same data
The result of the Health Data Repository process Slide 50 NPCD Data
Flow Diagram Data extracted & backed up nightly M-F DMI Data
received in DMI 24x7 NPCD data is sent from the facilities to the
AAC via MailMan messaging MailMan Message VistA MailMan NPCD and
other applications retrieve their respective data from DMI for use
Data Stream Once a message reaches the AAC MailMan server, It
automatically moves to the Data Management Interface System (DMI)
Austin MailMan Server Acknowledgement messages are sent to
facilities HL7 data to Oracle DB Acknowledgement message z900 NPCD
Data extracted by application Slide 51 NPCD Processing DSS data
extracted Flat files are indexed and loaded into the database daily
Data is checked for duplicates bi- monthly Data is extracted and
filtered for reporting twice a month Oracle on Unix NPCD UNIX
Master Extract File (MEF) SAS z900 (MAINFRAME) VSSC/ KLF Menu
WINDOWS Daily Data Loading Slide 52 Resources for Further
Information VA Information Resource Center (ViREC)
http://www.virec.research.va.gov National Patient Care Database
(Internal) http://vaww.aac.va.gov/npcd/http://vaww.aac.va.gov/npcd/
National Data Systems (NDS) (Internal)
http://vaww4.va.gov/NDS/DataAccess.asphttp://vaww4.va.gov/NDS/DataAccess.asp
Slide 53 Secrets of the VA Data Universe This was an extremely
brief introduction to a complicated area I have another
presentation on the availability of databases in the VA and how to
access them for operations and/or research Slide 54 The VA Data
Lifecycle Slide 55 Regional Remote Data Processing Center Shadow
Systems A offsite backup process to ensure continuity of operations
for VistA Patient Care Slide 56 Regional Data Processing Centers
(RDPCs) Read only backup VistA systems are set up to take
journaling files When a record is written or altered to a local
medical centers VistA, a journal file with that entry is prepared
and sent to a Regional Data Processing Center This maintains an
active backup in case the local medical centers VistA goes down
VistA goes down OpenVMS goes down Slide 57 Regions and RDPCs Region
I RDPC Sacramento (SAC) and Denver (DEN) Region IV RDPC
Philadelphia (PHI) and Brooklyn Slide 58 The Day VistA Died Slide
59 Northern Cal Without VistA The medical staff was forced to write
discharge instructions and notes on paper. The electronic lists of
instructions and of medications were not available for the patients
being discharged. Patients being discharged could not be given
follow-up appointments at the time of discharge. The appointments
had to be made later and the patient notified by phone. There were
delays in obtaining discharge medications and patients remained on
the wards longer than would normally be required. The nurses
administered medications to the patients and used the paper
Medication Administration Record, or MAR, to record the
administration events. Initial medication passes were interrupted
and delayed until the paper copies of the (MAR) could be printed.
Slide 60 RDPC Denver and Brooklyn Slide 61 The VA Data Lifecycle
Slide 62 Business Intelligence Slide 63 Business Intelligence in
the VA Making the Data Work For Us VistA has a wealth of clinical
and administrative data available In the past, giving a
value-added, timely VistA dataset was hard Querying the active
system with minimal impact Needed an interface between M and
analyst languages (SAS, SQL, etc.) Easy to read reports was hard to
build Slide 64 Slide 65 Corporate/Regional Data Warehouse Takes a
copy of the journal file that goes into the backup shadow system
Translated from the M array to a relational database format using
Intersystems Caches class mapping program Staged in a
Feeder-Collector system for collection Indexed and value-added
columns produced and loaded to an VISN RDW Server Slide 66 CDW
Governance VHA Business Owners/SMEs VHA-OI Data Quality OI&T
Corporate Data Warehouse 10N, OIA, VBA CDW Governance Board
Communicates Organizational Priorities Organizes SMEs and Data
Stewards Provides Documentation and Clarification of Business Logic
Sets and monitors domain, work priorities, and timelines for
completion. Slide 67 CDW Governance Is In VHAs Hands Ordered By VHA
Domain and Work Prioritization By CDW Governance Board Chair KLF
(OIA) Vice-Chair Larry Mole (Public Health SHG) Monitored and
Accountable To VHA Project management provided by John Quinn
(National Data Systems) and KLF (OIA) Supported By VHA OI Data
Quality Business Owners PBMs Data Steward is Rob Silverman Slide 68
As the number of eyes goes up, the number of bugs goes down.
Writing documentation about the business logic of the files and
fields Answering end user questions about the data Data validation
Preferably before Inpatient Pharmacy ADR/Allergy Package Slide 69 1
st category models are simple V Health Factor Source Mapping
FMFileFMFieldResolveFldDWTableNameDWFieldName V HEALTH
FACTORSHEALTH FACTORHealthFactorHealthFactorTypeIEN V HEALTH
FACTORSHEALTH FACTOR0.01HealthFactorHealthFactorType V HEALTH
FACTORSPATIENT NAMEHealthFactorPatientIEN V HEALTH FACTORSEVENT
DATE AND TIMEHealthFactorEventDateTime V HEALTH
FACTORSVISIT0.01HealthFactorVisitVistaDate V HEALTH
FACTORSVISIT0.01HealthFactorVisitDateTime V HEALTH
FACTORSLEVEL/SEVERITYHealthFactorLevelSeverity V HEALTH
FACTORSVISITHealthFactorVisitIEN V HEALTH FACTORSENCOUNTER
PROVIDERHealthFactorEncounterStaffIEN V HEALTH FACTORSCOMMENTS
HealthFactorComments Slide 70 2 nd category models require
transformation Prescription Prescription and 1 st fill Refill
Prescription Only All Fills Partial Fill Fileman Data Warehouse
Slide 71 3 rd category models not usable without transformation -
PCMM Slide 72 Levels of Data National Corporate Data Warehouse
(CDW) Region Regional Data Warehouse (RDW) VISN VISN Data Warehouse
(VDW) Medical Center Local Data Slide 73 Entities Who Produce
Business Intelligence Products National VSSC, PSSG, DMDC, HEC, ARC,
DSS, BIPL, OQP, PCS, PBM Region Regional BISL Teams VISN VISN Data
Warehouse, VISN PBM Local DSS Bolded are ones that have substantial
resources in clinical business intelligence PSSG handles much of
the GIS and Statistical Demography for the VA Slide 74 Data Access
VISN and Station Level Contact Your VISN Database Manager
Regional/Corporate Access Contact NDS for the 9957 Permissions
Slide 75 Operational Challenges of VistA System Resources $8
Billion investment over 20 years New needs for new domains MUMPS
Programmers must be internally trained (and many of them are
retiring or dying) Communication with Other Systems HIMISS
compliance with data interchange E-functions (billing, prescribing,
verification) Interagency Cooperation DoD and NHIN Business
Intelligence Closing the data lifecycle and bringing back clinical
data for knowledge discovery Slide 76 Why Is Pharmacy ALWAYS Picked
As An IT Test Case Project? Pharmacy is data savvy Local data
quality control from ADPACs Federated data quality control from
VISN PBM, CMOP, PBM SHG Pharmacy is one of the few domains that
have active business logic SMEs Pharmacy did not contract
institutional memory out the door Pharmacy gets things done
Pharmacy has more technology success stories as a collective than
any other PCS office (BCMA, Automation, Central Fills) Pharmacy
actively mines data Slide 77 Acknowledgments Kernel Jack Schram
(Oakland OIFO) SQLI Ellen Zufall (SF IRMS) RPC Broker and MUMPS
coding Perry Richmond (VISN 18 BI) Regional Data Process Vincent
Bui and Ken Koenig (Region I SQL Back Office Team) Slide 78
Acknowledgments OI&T Business Intelligence Product Line (BISL)
Jack Bates Manager, OI&T BIPL Stephen Anderson Lead Data
Architect Mike Baker Lead ETL Architect Denver Griffith/Ken Fuchsel
Server Administrators Dave Fackler Ron Talmage Dan Hardan, Jeff
King, Jeff Price Slide 79 Questions
