Evaluation of Health Technologies-

A Systematic Mapping Study

Priyanka Vegad

Student ID: 13107224

Supervised by: Dr. Pdraig O'Leary

Submitted to the University of Limerick, August 2014 in the partial fulfillment of

the requirements for the Degree of Masters of Science in Software Engineering.

Abstract

Introduction: Objective is to provide an overview of the evaluation techniques used for

assessing health technologies.

Methods: We have applied a systematic mapping study focusing on the evaluation of health

technologies.

Results: Several frameworks were identified in the literature. This research was not restricted to

a particular domain and included Telemedicine, Connected health, Information systems, Health

informatics, e-Health, m-Health, etc. As a result of this mapping study, it was found that there is

little evidence of work done in the evaluation of connected health technologies. Also, there is a

need to focus on technical evaluation of systems and developing a framework which is accepted

universally for the evaluation of any particular area of health technologies.

The research question is: How are Health Technologies assessed?

This research question is broken into the following four sub-questions:

a) What are the frameworks for assessing health technologies?

b) What are the challenges in evaluating health technologies?

c) What are the main evaluation domains?

d) What are the different types of research methods used for evaluation?

Information technology companies and healthcare teams are uniting in the goal of giving their

patients the best service and setting up an improved connection between health care technology

and the patient(s). Technology used in healthcare enables the patient to get updates regarding

their health and allows them to manage their health. But, there are some issues which if ignored

might have a negative effect on the users of the technology. Therefore, there is a need to evaluate

health technologies and the challenges involved in evaluating them.

Conclusions: In order to evaluate Health technologies, the assessment of information technology

is a must. During Information systems evaluation, not only the technology should be assessed but

also the way in which the technology is used by the users/ stakeholders should be taken into

consideration. The emergence of new areas in this field such as connected health is significant

and underlines the importance of adaptive evaluation frameworks.

Declaration

Title: Evaluation of Health Technologies- A Systematic Mapping Study

I hereby declare that this thesis is entirely my own work and does not contain material previously

published by any other author, except where due reference or acknowledgement has been made.

Furthermore, I declare that it has not previously been submitted for any other academic award.

_________________

Priyanka Vegad

August 2014

Acknowledgements

I would like to sincerely thank my supervisor Dr. Pdraig O'Leary for his guidance, enthusiastic

encouragement and valuable suggestions during the planning and development of this research

work. His willingness to give his time so generously has been very much appreciated.

I would also like to thank my parents, Mr. Hitesh Vegad and Mrs. Heena Vegad and my

grandfather, Mr. Mohan Lal Devji Vegad for their love, support and encouragement throughout

to keep me going. I would also like to thank my sister, Payal Vegad for her encouragement and

support throughout to help me complete the research and the dissertation.

Finally, I would like to thank all the lecturers from the CSIS department for all their support

throughout the Masters course.

List of Figures

Figure 1 Research Structure 2

Figure 2 D&M IS Success Model (Delone and McLean, 1992) 9

Figure 3 D&M IS Success Model (Delone and McLean, 2003) 10

Figure 4 IT- Organizational Fit Model (Yusof, 2006) 11

Figure 5 Human-Organization-Technology Fit (HOT-fit) Framework (Yusof et al., 2006) 12

Figure 6 TAM Model (Davis et al., 1989) 13

Figure 7 Systematic Mapping Process (Petersen et al., 2008) 17

Figure 8 Research Questions 18

Figure 9 Search Process workflow (da Silva et al., 2011) 20

Figure 10 Phases of screening process 23

Figure 11 Research Domains 30

Figure 12 Research Domain Vs Study type 32

Figure 13: Research Domain Vs Research method 32

Figure 14: Research Domain Vs Year of Publication 38

List of Tables

Table 1 Search String 19

Table 2 List of digital databases searched 22

Table 3 Research Type Facet (Petersen et al., 2008) 24

Table 4 Data Extraction 26

Table of Contents

Introduction ..................................................................................................................................... 1

1.1 Introduction ............................................................................................................................... 1

1.2 Research Objective ................................................................................................................... 1

1.3 Research Structure .................................................................................................................... 2

Literature Review............................................................................................................................ 3

2.1 Introduction ............................................................................................................................... 3

2.2 Connected Health ...................................................................................................................... 4

2.3 Health Information Systems ..................................................................................................... 5

2.4 Evaluating clinical Outcomes of Health Information Systems ................................................. 6

2.5 Information System Success ..................................................................................................... 8

2.6 Clinical Success ...................................................................................................................... 13

Research Method .......................................................................................................................... 16

3.1 Protocol Development ............................................................................................................ 17

3.1.1 Definition of research Questions ...................................................................................... 17

3.1.2 Conduct the search ........................................................................................................... 19

3.1.3 Selection Criteria .............................................................................................................. 20

3.1.3.1 Inclusion criteria ........................................................................................................ 21

3.1.3.2 Exclusion criteria ....................................................................................................... 21

3.1.4 Screening of papers .......................................................................................................... 21

3.1.5 Classification Scheme ...................................................................................................... 24

3.1.6 Data Extraction ................................................................................................................. 24

Results ........................................................................................................................................... 26

Discussion ..................................................................................................................................... 36

5.1 Frameworks............................................................................................................................. 36

5.2 Research Domains .................................................................................................................. 37

5.3 Research Methods ................................................................................................................... 39

5.4 Challenges ............................................................................................................................... 39

5.5 Limitation of this Research ..................................................................................................... 40

Conclusion and Future Works ...................................................................................................... 41

6.1 Conclusion .............................................................................................................................. 41

Key findings .............................................................................................................................. 41

6.2 Future works ........................................................................................................................... 42

References ..................................................................................................................................... 43

Appendix A ................................................................................................................................... 45

Appendix B ................................................................................................................................... 45

1

Chapter 1

Introduction

1.1 Introduction

Information technology companies and healthcare teams are uniting to give their patients the best

of service and to set up an improved connection between health care technology and the

patient(s). Technology is considered as the backbone of health technologies. Innovations in the

field of technology has not only changed the face of healthcare but also has helped to bring some

of the major transformations in the healthcare industry. Innovations include both hardware as

well as software. Hardware innovations include some of the wearable medical devices such as

sensors, and software innovations include electronic personal health record systems such as

Google health and Microsoft Health Vault (Bushko, 2009).

Health technologies had been studied for effectiveness, safety, cost, and various other concerns

(Goodman, 2004). Health technologies includes terms such as e-Health, m-Health, Tele-health,

Digital Health, Telemedicine, Home based care, etc. The fusion of healthcare and technology

and its impact has been a topic of discussion for decades. Research is done and literature related

to it exists. Some of the studies and reports by Leonard and Winkelman 2002; Lorenzi and Riley

1995; Lorenzi and Riley 2000; Lorenzi et al. 1997; Stiell et al. 2003, have focused on identifying

ways to ensure information technology (IT) implementation (Leonard, 2004).

As defined by (Goodman, 2004) Health technology assessment is the systematic evaluation of

properties, effects or other impacts of health technology. Assessment of health technology

includes various properties like safety, economic attributes or impacts, effectiveness, social

impacts, ethical impacts, legal impacts, technical properties, etc. (Goodman, 2004).

1.2 Research Objective

The objectives of this research are listed below:

2

Identify the gaps in literature to be filled by the researchers in future related to evaluation of

Health Technologies.

Review the studies which are conducted till date (no specific year range) to get a better

understanding of the domain.

Deliver some suggestions for future works in the area of health technologies.

1.3 Research Structure

A Structure for this thesis is described in the figure below:

Figure 1.Research Structure

Chapter 2 includes the literature review of the evaluation of health technologies like Connected

Health, telemedicine, etc. It also includes Information Systems success which is considered

necessary for evaluating health technologies.

Chapter 3 presents the Research method which consists of several phases like Definition of the

research question, conducting a search, selection criteria, screening of papers, the classification

scheme and Data Extraction. This chapter also introduces the research questions and the

screening process followed for this mapping study.

Chapter 4 presents the results for the main research question including the answers to the sub-

research questions.

Chapter 5 presents the discussion of the results section in detail. Discussion about the identified

gaps is presented in this chapter.

Chapter 6 presents the conclusion for this mapping study and provides some recommendations

for the future works related to the evaluation of Health Technologies.

3

Chapter 2

Literature Review

2.1 Introduction

Evaluating Information system success is one of the important steps in evaluating health

technologies. Existing research gives an insight into several frameworks and models for

evaluating Information Systems (IS) Success to name a few: Information Systems Success

Model (Delone, 2003) and HOT-fit framework (Human, Organization and Technology- fit)

(Yusof et al., 2006).

Previous research and the ongoing research suggests that the purpose of the system used in an

organization can only be entirely successful in delivering its services only when all the

participants using the system are satisfied and are aware of the purpose for which the system is

developed. Information Technology is a key enabler to improving healthcare processes due to its

potential of providing rapid and comprehensive access to information at the point of care (Lenz

and Kuhn, 2004). However, there are some concerns related to the use of technology in

healthcare, which if ignored can have an adverse effect on the users of the technology. Priority is

to provide an overview of the evaluation techniques used for assessing health technologies. With

different interrelated dimensions and variables which will be the base for assessing the quality, it

is necessary to find a common approach to success measurement. In order to determine the

different set of dimensions for assessing the health technologies, review of the already existing

models like Delone and McLean Model of IS Success Model, Human- Organization-

Technology- fit (HOT- fit) Model, TAM model was done. The different attributes and its

respective dimensions were analyzed and the limitation of each of the models was noted.

Reviewing the existing literature for evaluating health technologies has highlighted some of the

issues which should not be neglected. Some of the frameworks like TEAM framework which

was developed for the evaluation of Health Information system is considered as a global

framework and is too broad for carrying out the IS evaluation. The HOT-IT fit organizational

4

model proposed by (Yusof et al., 2006) needs to be tested in clinical settings in order to validate

its usefulness.

2.2 Connected Health

Connected Health encompasses terms such as wireless, digital, electronic, mobile, and

telehealth and refers to a conceptual model for health management where devices, services or

interventions are designed around the patients needs, and health related data is shared, in such a

way that the patient can receive care in the most proactive and efficient manner possible. All

stakeholders in the process are connected by means of timely sharing and presentation of

accurate and pertinent information regarding patient status through smarter use of data, devices,

communication platforms and people (Caulfield and Donnelly, 2013).

Advancements in the technologies used in the past few decades has helped evolve healthcare

enabling remarkable advances in connected health (Bushko, 2009). Current healthcare systems

are concerned with only one part of the health continuum for diagnosing and treating patients

(Saranummi, 2013).

The aim of connected health technologies is to deliver health care and monitoring from the

hospital and into the customers homes. A further step in the advancement in the technology will

assist in monitoring the patient irrespective of where the patient is. Software installed on devices

like mobile phones with the help of internet enables continuous monitoring of parameters related

to patients health such as heart rate, ECG, sleep monitoring, etc. by receiving data or signals

(Bushko, 2009). The technology used in healthcare makes the patient aware of their health

related issues. In connected health, individuals are equally responsible and are partners with

healthcare professionals in the involvement for managing their health and wellness (Saranummi,

2013). The primary goal for adopting health technologies is to provide patients the best service

possible by gathering and interpreting accurate information which will help them to take correct

decisions on time which reduces the cost, time and effort thereby resulting in the timely

treatment of the patient.

5

2.3 Health Information Systems

Health information systems (HIS) is a patient-centered approach that aims at contributing a high

quality; efficient patient cares (Haux, 2006). Health care is beneficial in many ways by the

adoption of Health Information Systems. HIS consists of a collection of different processes

which are implemented to assist and give support to the health care organization to achieve their

objectives with accuracy which can be done by enhancing the efficiency and effectiveness

(Yusof et al., 2006). The barriers related to the use of HIS should also be considered while

evaluating HIS because these obstructs will decide whether the HIS is successful or a failure

depending on the particular context. Two common barriers identified regarding the use of

healthcare systems are culture and process changes. Other barriers include low levels of expertise

existing in the healthcare organization, lack of medical staff, changes in the traditional workflow

patterns (Yusof et al., 2006).

Because of the power of economic forces and community, patient-centered health care

information system is accepted by people these days. This patient-centered health information

system allows providers to keep the patients record updated and easily accessible when needed

(Menp et al., 2009). It is seen that while developing a system (Information system),

continuous change in the conditions or requirements can be expected. Healthcare institutions too

represent an environment of changing requirements. It is not fair to believe that all the

information and knowledge management process in healthcare are systematically mapped to an

IT system. Rather, the physicians, nurses and patients also integrate knowledge and

communicate it with each other, while IT systems complements and support these processes

(Lenz and Kuhn, 2004). Building a safer healthcare system is suggested where healthcare

professional should increase their understanding about information technology and how it could

be applied to deliver safer healthcare (Schrenker, 2006). To achieve the expected results,

information systems should perform accordingly. Moreover, to ensure the information system

performs correctly, the quality of the software used should be assessed. This determines whether

the information system is successful or not.

The use of IS can achieve remarkable benefits for an organization by not only gaining

competitive advantage, increasing productivity, automation of operational decision, shorter

6

product cycle and supporting of strategic and tactical decisions but also helps in enhancing the

overall impact on organizational forms and management paradigms (Mohd Al-adaileh, 2009).

However, in order to achieve these valuable benefits, success of ISs is a must. Several measures

had been taken to evaluate HIS (Health Information Systems). It is noted that attempts made to

adopt health care technologies that require data entry by the healthcare professionals have not

been entirely successful (Van Der Meijden et al., 2003).

2.4 Evaluating clinical Outcomes of Health Information Systems

Evaluation is performed to improve HIS by learning from the experiences and identifying more

effective ways to improve the systems (Yusof et al., 2006). Factors that affect the evaluation

include organization factors and human factors. Human factors include a range of issues like

training, personnel attitudes, ergonomics (the study of people's efficiency in their working

environment) and regulations affecting employment (Yusof et al., 2006).

For healthcare information system to work appropriately it requires an advanced health

information network, that supports clinical care, personal health management, the reduction of

avoidable mistakes in population health and research, and evidence-based medicine (Menp et

al., 2009). Different approaches to HIS evaluation developed based on a number of factors like

technical, economic, human, organizational, etc. are discussed by (Yusof et al., 2006). Research

works shows that there are a number of frameworks, specially developed for evaluating Health

information systems.

To name a few includes 4Cs framework, CHEATS framework, TEAM framework, ITAM

framework, etc. (Yusof et al., 2006). The 4Cs stands for (a) Communication (Communication

between the departments), (b) Care (Medical care delivery), (c) Control (Control in the

organization and (d) context (Clinical settings). The model developed for the evaluation of IT in

healthcare was CHEATS which includes characteristics like Clinical, Human and Organization,

Administrative, Technical and Social. This model helped in assessing and providing more

definite measures for clinical settings. Total Evaluation and Acceptance methodology (TEAM)

consists of three dimensions i.e. Role, Time and Structure. TEAM is considered as a global

framework and is too broad for carrying out a particular IS evaluation. The IT Adoption Model

(ITAM) primarily focuses on an individual users point of view. ITAM is insufficient when it

7

comes to evaluating organizational impact. But, they complement each other and help in

evaluating different aspects of HIS (Yusof et al., 2006).

There is a growth in the building of Regional Health Information infrastructure to provide the data

accessibility anytime and anywhere keeping the data secure and safe. Its underlying idea is to

improve the healthcare through quality, completeness and timeliness of public health data. This will

help the timely detection of disease (Menp et al., 2009). Regional Health Information Systems

(RHIS) focuses on five main areas i.e. flow of information, collaboration, process redesign, system

usability and organization culture. Based on their study they highlighted some of the outcomes

which include timely information, clinical data exchange and improved communication between the

professionals (Menp et al., 2009). The reason behind the existence of RHIS was to improve the

quality of care, the health and safety of individuals and improves the effectiveness. RHIS offers

better-patient-centered care which reduces the risk to patients' life. During the late nineties, the

concept of Integrated Electronic health records and patient-centered healthcare integrated with RHIS

has been suggested for a period of more than 30 years (Menp et al., 2009). According to the

authors, the outcomes of Regional Health Information Systems were better collaboration and

communicating resulting in better flow of information, process design, usability, etc. It was noted

that the RHIS was also responsible for enhancing the data access, allowing the clinical data

exchange between the different systems in distinct environments and providing the real-time patient

information (Menp et al., 2009). Because of the presence of diverse systems, the healthcare

industry faces some issues which include lack of integration between the different Health

Information Systems (HISs) used lack of communication also results in making the task difficult for

the healthcare professionals and the absence of a standardized Electronic health records (EHR). The

existing problem of integrating all the patients' data from all the different systems into one and was

not completely efficient in making the data accessible by the healthcare professionals. It was noted

that there was no standardized Electronic Health Record and because of the different EHR

structures, it was difficult to accumulate the vast data records of the patients and use it for treatment

purpose. This problem leads to the duplicity of records, unnecessary treatment, and inaccuracy in

medical records and increase in effort and cost. Maintaining an EHR also allows the patients to enter

their details manually that require the patient to input valid details related to them. Incorrect data

input might result in risk to the patient's information.

8

This problem of integrating all the patient records into one was overcome with the advancement in

technology. Google Health is a new platform which is an example of an electronic personal health

record system. It allows the patients to store their personal health information in an account that is

accessible anytime and from anywhere. Not only it allows entering the information manually, but

also allows the users (healthcare professionals and patients) to import their medical records from

their hospitals. Another example is the Microsoft HealthVault, which like Google Health integrates

personal health information. This platform needs to set up a free Microsoft account which permits

the patients to gather, store and search health information online. Microsoft Health Vault is not only

confined to personal health record, it acts as an online storage of information allowing applications

to interact with the existing data, analyzing it and providing solutions (Bushko, 2009). Active Health

is an enhanced personal health record which uses the Microsoft health vault to extract patient's

records (Bushko, 2009).

2.5 Information System Success

Initial step in evaluating the success of the information systems is to define what success means.

According to (Van Der Meijden et al., 2003) Determination of success depends on the setting, the

objective, and the stakeholders. Moreover, to check whether the system was successful or not, a

detailed study should be carried out based on a specific setting (Van Der Meijden et al., 2003).

Evaluating the success of Information Systems (ISs) is identified as one of the most crucial

issues in IS field. Information system infuses almost all facet of human life. Because of its

continuous use and presence, high investments are done which demands for success and quality

of the systems (Iivari, 2005). However, in order to achieve these valuable benefits, success of ISs

is a must. To date much research has been conducted to identify the factors that contribute to

Information System Success. Many researchers suggested different aspects of IS Success, which

demands a more integrated view of the concepts of IS Success (DeLone and McLean, 1992).

There is no good in measuring various independent or input variables, if the dependent or output

variable I/S success cannot be measured with a similar degree of accuracy (DeLone and

McLean, 1992).

9

The model developed by Delone and McLean is one of the most cited models for IS Success.

Ultimately, the different aspects of IS success was drawn together into a single descriptive model

referred to as D&M IS Success Model (DeLone and McLean, 1992) which included six

interdependent variables to measure the success of IS which includes: system quality,

information quality, user satisfaction, systems use, organizational impact and individual impact

(DeLone and McLean, 1992).

Figure 2. D&M IS Success Model (DeLone and McLean, 1992)

According to (Van Der Meijden et al., 2003), the characteristics with which the information

processing system is assessed includes the system quality attributes (accessibility, usability, etc.),

Information quality attributes (accuracy, completeness, legibility), User satisfaction can include

both system itself or its information. According to Delone and McLean, their paper presented an

IS Success model which attempted to capture the interdependent nature of IS Success (Delone,

2003). In their original model, Delone and McLean introduced the model as a casual-explanatory

model of how information quality and system quality affect user satisfaction and use, how use

and user satisfaction are correlated and how they affect each other , how individual impact leads

to organizational impact and how use and user satisfaction are direct antecedents of individual

impact (Iivari, 2005). A huge discussion continues pertaining to the use of the appropriate set of

10

variables that can be used to measure ISs Success. The original model has been reviewed and

modified by many researchers. Some of them even introduced frameworks for measuring the IS

Success effectively like the framework proposed by Grover et al., used an alternative,

theoretically based perspective to build a theoretically based construct space for IS effectiveness

that compliments and extends the [ Delone and McLean] IS Success Model (Delone, 2003).

However, the success model does not serve as a basis for the selection of suitable IS measures

(Delone, 2003). In 2003, Delone and McLean proposed The Updated D&M IS Success model

which consists of six major categories of I/S success: Service Quality, System Quality, System

Use, Information Quality, User satisfaction and Net Benefits.

Figure 3. Updated D&M IS Success Model (Delone, 2003)

(Seddon and Kiew, 2007) critically examined the meaning of these four constructs and the

support of relationships between the different variables. They considered D&M IS Success

Model as their foundation for their research and listed the difference between Delone and

McLean's model and the model tested. These differences are as follows: (a) One of the variables

Use in Delone and McLeans model was replaced by Usefulness (b) introduction of a new

variable System Importance which according to the authors will help to clarify the variations in

users view of usefulness and user satisfaction (c) the two- way dependency between user and

11

User satisfaction in Delone and McLeans model was replaced by a one- way dependency

between usefulness and user satisfaction (Seddon and Kiew, 2007).

Human- Organization- Technology Fit (HOT-fit) Framework

By considering the strengths and weaknesses of IS Success model and the IT-fit Model, and

integrating the best features of both the models into one, the HOT-fit framework is developed.

IT -fit Model

The concept of fit between technical, organizational and social factors is analyzed in order to

identify the gap between the present healthcare systems and features of the new system (Yusof et

al., 2006). IT-fit model includes both internal and external elements and the concept of fit

between the main organizational elements. Internal-fit is achieved by a dynamic equilibrium of

organizational components like business strategy, organizational structure, management

processes, and roles and skills. External fit is attained by considering organizational strategies

based on environment which includes: market, industry and technology (Yusof et al., 2006)

Figure 4. IT- Organizational Fit Model

Human-Organization-Technology Fit model as stated by the author is capable of identifying the

organizational elements which effects the IS Success and also provides a fit between them and

12

includes factors like Human (Roles and skills), Organization (Strategy, Structure and

Management process) Technology (IT). This model complements the Delone and McLean

Model of IS Success. The factors of HOT- fit are again categorized into several different

dimensions (as illustrated in figure 5) based on the dimensions described in D&M IS Success

Model. These three factors consists of eight interrelated dimensions of HIS success namely:

System Quality, Information Quality, Service Quality, System Use, User Satisfaction,

Organizational Structure, Organizational Environment and Net Benefits (Yusof et al., 2006).

Figure 5. Human-Organization-Technology Fit (HOT-fit) Framework

(Yusof et al., 2006)

TAM Model

Technology Acceptance Model (TAM) (Mohd Al-adaileh, 2009) is a model IT adoption and

operation that has been customized to explain computer usage. Figure 6 Illustrate the model

elements such as perceived usefulness, behavioral intention to use, perceived ease of use and the

relationships among them.

13

Figure 6. Source Davis et.al. (1989), Venkatesh et.al. (2003)

The above described model emphasizes the importance of perceived usefulness. Author believed

that the assumptions regarding the perceived usefulness might be true based on users level of

proficiency (Mohd Al-adaileh, 2009).

It is observable that so far there is no agreement on the use of a set of measures of IS Success. It

is not easy to choose some particular variables from the list of variables available. In reassessing

these variables, no single measure is essentially better than another; so to make a choice of a

success variable depends on the objective of the study, the aspect of the information system

which is addressed by the study, the organizational context, the independent variable under

investigation, the level of analysis, i.e. individual, organization, or society and the research

method (DeLone and McLean, 1992).

2.6 Clinical Success

The adoption of new technologies from a wide range of industries and sectors is not new but it is

still considered as a challenge to integrate technology with healthcare (Leonard, 2004).

Evaluation is considered as an important factor as it serves a number of purposes. Given the

unpredictable characteristics of IS in general and the aim of improving clinical performance and

patient outcomes in particular, evaluation is undertaken to understand system performance

(Yusof et al., 2006). While evaluating the clinical success the clinical evidence is collected,

analyzed and compared. Moreover, this raises some important issues which need to be taken into

14

consideration which includes: what to measure and how to measure (Jen-Hwa Hu, 2003). Many

papers discuss the factors that affect both information technology and systems. However, none of

the studies so far provides a description on how to operationalize (Leonard, 2004). The solution

to this is to summarize a brief list of critical success factors (CSFs) which will provide guidelines

to set the framework for achieving the success in designing, developing and implementing new

technology in healthcare. The challenge faced is to present these listed CSFs within a framework

that is both informative and illustrative (Leonard, 2004).

(Jen-Hwa Hu, 2003) introduced a new model to access the success of the systems used in

telemedicine. For developing this new model, he followed Delone and McLean Model of IS

success as the basis of his new model. This new revised model was designed considering the

various aspects of healthcare and the technologies used in it. A new dimension was added to the

model known as the input data quality. The idea behind adding this dimension for assessing the

success of the system was to focus on the importance of inputting the data onto the system in

connected health settings. The quality of input from connected clinical perspective is crucial

which has a strong impact on the resulting services along with the participants of the system and

the organization. In this model, several feedbacks loops are also integrated which are different

from the original Delone and McLean model.

One of the many solutions which is capable of dealing with significant healthcare problems is

the involvement of the service providers (of primary, secondary and tertiary care) and the

patients who are receiving treatment, the electronic data exchange allows the creation of virtual

service networks which are capable of dealing with significant problems in modern healthcare;

e.g., service accessibility, quality, utilization effectiveness/efficiency, and costs (Jen-Hwa Hu,

2003). (Leonard, 2004) Discuss some of the case studies to elaborate on the issues relating to the

CSFs (Critical success factors) of technology adoption. He discussed the acceptance of electronic

images of x-rays. During the conversion process from traditional film- based radiology process

to electronic images of x-rays, doctors were notified that their patients x-ray reports will be

emailed to them. Many doctors complained about not receiving the e-mails, but the case was

they did not check their e-mails. To overcome this problem, the patients reports were sent to the

doctors using computer diskette or compact discs (depending on the size of the data to be sent).

Even after making minor changes in order to help doctors getting acquainted with this

15

transformation, doctors complained about the delay in getting the results. Because they found it

difficult to match the arrival of these hardware supplies containing information to their

information needs. This does not mean that the doctors had no experience in identifying the

connection between the Compact discs and patient x-rays. Technology, they just were not

familiar to it. Taking into consideration this finding, the imaging department decided to continue

the use of CDs but in a different way. They finalized on a solution of sending the CDs via

internal hospital delivery system in traditional large x-ray envelopes. This helped the doctors to

recognize the type of packaging in which the CDs were placed in (Leonard, 2004).

Other factors, apart from clinical success that relates to the overall success in health technologies

includes Economic, operational, strategic and technological (Duarte and Costa, 2012). These are

considered as the four dimensions along the information system life cycle. These dimensions are

discussed in brief in the following section (Duarte and Costa, 2012):

Technological Dimension: It involves product, its characteristics and its alignment with the

business strategy. Companies follow this these days to acquire a place in the competitive market.

Operational Dimension: It includes people and the processes. This dimension is believed to be

an important one because it is related to all the people in the organization. Not only this, IS

system involves huge investment by the organizations.

Economic Dimension: It involves the cost and benefits for an investment project.

Strategic Dimension: It involves the business vision and the organizational strategy.

16

Chapter 3

Research Method

A systematic literature review (SLR) is a form of secondary study (A study that reviews all the

primary studies which is focused on a definite research question with the aim of

integrating/synthesizing evidence related to a specific research question) which deals with

identifying, evaluating and interpreting all available research with respect to a particular research

question, or phenomenon of interest, or topic area. (Kitchenham and Charters, 2007). Another

type of reviews other than systematic literature review includes systematic mapping studies.

Even before carrying out systematic review focusing on a particular domain, if very little

evidence exists regarding the subject or the chosen topic of research is very broad, in that case a

systematic mapping study might be an appropriate solution (Kitchenham and Charters, 2007).

A mapping study focuses on classifying the relevant literature based on the particular topic and

aggregate studies concerning the defined categories. These categories usually are categorized

depending upon the publication information such as authors name, authors affiliation,

publication source, publication date etc. (Kitchenham et al., 2011).

The reason behind conducting a systematic literature review is that it helps identify gaps in

current research in order to suggest areas which need further investigation. It also helps in

providing a framework or background to suitably position novel research activities (Kitchenham

and Charters, 2007). The early stages of systematic literature review and systematic mapping

study are similar which includes: (a) Identification of primary studies which may contain

significant research results (b) Selection of appropriate primary studies based on the inclusion

and exclusion criteria (c) if required, quality assessment of the selected studies is performed

(Budgen et al., 2008). Two stages where mapping studies differs strongly from systematic

literature review are during the classification of the papers and during data extraction and

aggregation(Kitchenham et al., 2011)

17

According to (Kitchenham and Charters, 2007), systematic literature review consists of three

different phases: Planning the Review, Conducting the Review, Reporting the Review. In the

following section, the process used for this mapping study is discussed in detail.

3.1 Protocol Development

We have applied systematic mapping study focusing on the evaluation of health Technologies.

Various process steps for systematic mapping study which includes the definition of research

questions, conducting a search for relevant papers, screening of papers, key wording of abstracts

and data extraction and mapping (Petersen et al., 2008). Each and every step has an outcome

depending on how the subsequent steps are performed. The outcome of this process is the

systematic map.

Figure 7: The systematic Mapping Process (Petersen et al., 2008).

3.1.1 Definition of research Questions

The research question that directs this mapping study is: How are health technologies assessed?

This primary research question was identified for carrying out the mapping study. Sub- questions

were derived from the primary questions which are illustrated in Figure 8.

18

Primary Question

Figure 8 Research Questions

Q1. What frameworks exist for assessing Health Technologies? This question identifies the

frameworks and models for evaluating information system success. To check whether the system

is successful or not, a detailed study should be carried out based on a specific setting(Van Der

Meijden et al., 2003). Evaluating the success of Information systems (ISs) is identified to be the

most crucial issues in IS field. Information system infuses almost all facet of human life.

Because of its continuous use and presence, high investments are done which demands for

success and quality of the systems (Iivari, 2005).

Q2. What are the challenges in evaluating health Technologies? This question includes the

barriers and the challenges involved in adopting and evaluating health technologies.

Q3. What are the main evaluation domains? Rather than focusing on just one domain, different

domains were considered for this study. Broadening the research domain might result in finding

a more general solution to the problem of the absence of study in a particular domain.

Q4. What are the types of research methods used for evaluation? Studies selected for this

research are categorized on the basis of research method. Studies included for this research

How are Health Technologies

assessed?

What are the types of

research methods used

for evaluation?

What are the challenges

in evaluating health

technologies?

What are the frameworks

for assessing health

technologies?

What are the main

evaluation domains?

19

includes both qualitative and quantitative which involves questionnaires, interviews, case study

etc.

3.1.2 Conduct the search

The terms selected for conducting the search were defined based on the approaches described by

(Kitchenham et al.), which followed a strategy with the below-mentioned stages:

(i) Derive major terms from the main research question as well as the sub- questions.

(ii) Identify alternative spellings and synonyms for major terms.

(iii) Checking the keywords.

(iv) Use of Boolean OR operator

(v) Use of Boolean AND operator to merge the different keywords.

Table 1. Search Strings

connected health OR health information technologies OR HIT OR information technology

OR healthcare OR telemedicine OR telehealth OR digital health OR health information

systems OR hospital information systems OR health services OR health informatics OR

medical informatics OR biomedical informatics OR assisted living, OR e-Health OR

mHealth.

AND

assessment OR evaluation OR success factors OR assessment frameworks OR

challenges OR outcomes OR key performance indicators OR success OR assess

variables OR performance measurement OR systematic review.'

OR and AND operators are used to merge different keywords which make the search process

simple. The search process used for conducting this mapping study was taken from (da Silva et

al., 2011) and is shown in Figure 9. This process mainly consists of three steps: (a) search strings

shown in Table 1 are entered in each of the digital databases (e.g. IEEE Explore, ACM Digital

library etc.). The entire list of the digital databases searched is illustrated in Table 2. (b) Manual

20

search was also conducted based on the year of publication, the area, etc. (c)Snow- balling

which means following the references of the papers already found for the mapping study.

Figure 9. Search Process workflow (da Silva et al., 2011).

The focus was to search the area of research for the relevant papers. Terms such as telemedicine,

tele-health, digital health, e-Health, etc. were used to find relevant literature based on the health

technologies. Both automatic and manual search were conducted. The set of inclusion and

exclusion criterias made the selection process easier and helped in narrowing down the research

domain which is discussed in the following section:

3.1.3 Selection Criteria

In this section, the relevant papers are searched using the search string (Table 1) based on some

inclusion and exclusion criteria which helped to eliminate articles which were not appropriate for

the study.

Manual Search Automatic Search

Inclusion/

Exclusion

Criteria

Snow-Balling

Primary Studies

21

3.1.3.1 Inclusion criteria

Below mentioned are the inclusion criterions set for this study:

The study must investigate the approaches, methods, challenges involved related to the

assessment of the health technologies.

Papers reflecting various works depicting different assessment frameworks/ models are

included.

Terms such as telemedicine, telehealth, digital health, eHealth, etc. were considered.

Peer-reviewed papers, i.e. conference, journal, book chapters, and workshop proceedings.

Only papers written in English were accepted for the study.

No specific year range was selected for conducting the search.

3.1.3.2 Exclusion criteria

Studies that are only available in PowerPoint presentations are excluded.

Studies which only deals with healthcare but lacks IT integration is excluded.

Studies relating to the domain and relating to the research questions were only included.

3.1.4 Screening of papers

Screening of the papers for the selection of the relevant paper was done in phases. Figure 10

illustrates the different phases of the screening process.

In stage 1, electronic sources were used to search studies related to the topic. List of all the

sources is given in Table 2. No specific year range was selected for this study. A specific number

of papers were selected at the end of the first stage by quickly reviewing the title of the papers

found. Papers were searched automatically by inputting the search string and by manually

inputting the keywords but were irrespective of the year of publication.

In stage 2, Papers that satisfied the inclusion and exclusion criteria are included. Numbers of

papers were found associated with the topic and the research questions. So, n= 1003 papers were

considered for conducting this study at the end of the second stage.

22

In stage 3, In order to determine whether the papers were related and answer to the research

questions or not, title and the keywords of the papers were reviewed. Based on the exclusion

criteria, the number of papers selected was narrowed down to n= 89 papers. Primary focus was

to identify the appropriate papers for the study. Many papers related to the integration of IT and

healthcare were obtained as a result of the conducted search but, only the papers discussing

assessment methods and frameworks involved in assessing health technologies and the success

factors associated are included. As a result, out of 1003 papers, 914 papers were excluded.

In Stage 4, abstracts of the papers were read to get a deeper understanding about the content of

the paper. At the end of this stage, papers were chosen for an in-depth study. Results were

extended by going through all references of each article and inspecting them for relevant studies.

This process of reviewing the references of the selected articles was done to make sure that none

of the important papers were left out. At the end of this stage, n= 40 papers were downloaded in

PDF format.

In stage 5, an in-depth study of all the papers from stage 4 was done and the papers that do not

justify the research area were excluded.

List of Digital databases

IEEE Explore

Google scholars

Science Direct

ACM Digital Library

Scopus

Wiley InterScience

Table 2.List of digital databases searched.

23

Figure 10. Phases of screening process

Stage 1

Conduct search using

the search strings

Stage 2

Include the relevant

papers

Stage 3

Eliminate non-relevant

papers based on the

exclusion criteria by

reading the keywords

and title.

Stage 4

Detail review of the papers

based on the abstract and

extending results

Stage 5

In-depth study of the

final selected papers

Library search (n=1003)

Titles considered

relevant (n=89)

No. of papers selected

after review based on

abstracts (n=40)

24

3.1.5 Classification Scheme

Results of the studies were classified based on (Petersen et al., 2008) idea of categorizing.

Studies were classified by reconsidering the abstracts, titles, keywords and sometimes even

reviewing the introduction and the conclusion. This classification scheme help identify the

different paper type across two different facets i.e. Research Facet and Context Facet. A

description of each research category is presented in Table 3.

Research Facet: Validation Research, Evaluation Research, Solution Proposal, Philosophical

Papers, Opinion Papers, and Experience Papers.

Context Facet: Connected Health, Telemedicine, Information system, Health information

system, e-Health, m-Health, Tele-health, Assisted Living, and Health Informatics.

Table 3 .Research Type Facet (Petersen et al., 2008)

3.1.6 Data Extraction

To address the objectives of this study, data was gathered by reviewing the papers in detail. The

following information was extracted from the papers selected for the study:

Authors name

publication year

25

Concept

Context Facet

Type of study

Research method (Analysis)

Measures

Models/frameworks evaluated,

Models/ frameworks identified in the study.

Gathering all the data in an excel sheet helped us to put down all the information in one place.

By analyzing the data, categorizing the types of papers, detecting the approach used for the

studies and identifying the frameworks for evaluation of IT in healthcare were represented using

bubble plot, bar chart and pie-chart. These types of representations are used for representing all

the data in a pictorial way that makes it easy for the readers to understand.

26

Chapter 4

Results

Main findings of this research are listed in this chapter based on the research questions. This

research was primarily focused on evaluating the connected health technologies but due to the

lack of evidence, the research area was extended to health technologies. The research question

including the sub research question is illustrated in Figure 8.

Connected Health is one of the emerging paradigms in healthcare domain, and this area needs to

be explored carefully in order to get the best out of it. This study not only focuses to the

evaluation of Health Technologies covers domains like telemedicine, e-Health, m-Health, Tele-

health, Bio-medical informatics, Assisted living, Health Information systems, etc. It also

emphasis on the challenges involved in evaluating health technologies.

Overviewing the systematic literature in IT healthcare, many models were identified for

evaluating health technologies. Evaluation frameworks are not only confined to one domain but

several which gives an insight on the different parameters on which each domain is evaluated

differently.

4.1 RQ1- What are the frameworks for assessing Health Technologies? Table 4, shows the list of

models that have been evaluated by different authors for their respective studies. Among the

models discussed, Delone and McLean model (both original and updated) are the most popular.

Out of 40 papers analyzed, 7 papers referred to Delone and McLean model for IS Success

evaluation as the basis for their research or evaluated the IS Success model. Different authors

made an effort to add different dimensions and variables to the already existing Delone and

McLean model of IS Success.

Studies also considered other frameworks as a basis of proposing a new framework for

telemedicine evaluation, IS Success etc. After gathering data and considering all the evaluation

models, Delone and McLean model of IS Success (original and updated) (1992 and 2003) was

evaluated by 7 authors.

27

Table 4

(DeLone and McLean, 1992) Proposed a framework for evaluating IS Success with dimensions

System Quality, Information Quality, Use, User satisfaction, Individual Impact, Organizational

impact. This model captured the multidimensional and interdependent nature of IS Success.

Being one of the most famous model, hundreds of research have challenged, validated and

applied the original model discussed and stated the need to find a common approach to success

Title Year Author Concept Model Evaluated by

Information systems success: the

quest for the dependent variable.

1992 William H. DeLone,

Ephraim R. McLean

The objective is to present an

integrated view of the concept of

IS Success and because of which

a taxonomy is introduced in this

paper. The taxonomy presents

some major dimensions and

variables for evaluating the IS

Success. The focus was to bring

awareness and provide structure to

the "dependent variable" of IS

Success.

IS Success

Model

William H. DeLone

Ephraim R. McLean-2003,

Paul Jen-Hwa Hu -2003,

Raid Mohd Al-adaileh -

2009,

William H. DeLone

Ephraim R. McLean-2002

The DeLone and McLean Model

of Information Systems Success:

A Ten-Year Update

2003 DeLone, William H

McLean, Ephraim R

In this paper, research contribution

made regarding IS Success and

the efforts made to update the

original DeLone and McLean

Information Systems (IS) Success

Model is discussed.

IS Success

Model- Updated

Raid Mohd Al-adaileh - 2009

Maryati Mohd. Yusof, Ray J.

Paul, Lampros K. Stergioulas -

2006,

Ana Isabel Mendes Duarte,

Carlos J. Costa- 2012,

Maryati Mohd. Yusof,

Anastasia Papazafeiropoulou,

Ray J. Paul, Lampros K.

Stergioulas- 2008

An evaluation of information

systems success: A user

perspective-The case of Jordan

telecom.

2009 Mohd Al-adaileh,

Raid

This paper focuses on reviewing

the available Information system

success evaluation studies. A new

framework for evaluation was

proposed and the model was

verified based on the research

findings.

A model for evaluation of

information system

success

Mohd Al-adaileh, Raid-2009.

Towards a Framework for

Health Information Systems

Evaluation

2006 Mohd, Maryati

Paul, Ray J

Stergioulas,

Lampros K

This paper combines the

approaches of existing models of

IS Evaluation mostly on two

models IT-Organization Fit and IS

Success Model. The proposed

framework was evaluated and was

suggested that it can be used as a

tool to conduct enhanced HIS

evaluation. A framework for

evaluating Information System in

healthcare setting inorder to ensure

effective implementation and

positive impact of HIS on

healthcare delivery.

HOT-Fit framework Mohammed, Siti Asma

Yusof, Maryati Mohd-2013

28

measurement (Delone, 2003). Due to the increasing impact of IS on society, and an increase in

the IS research, the authors felt the necessity to review, evaluate, and update the original success

model. Later, this model was self-evaluated and the dimensions modified for this model were

Information Quality, system quality, service quality, use, user satisfaction, net benefits. The

authors of the paper, suggested that intention of use might result into one of the worthwhile

alternatives for the updated Delone and McLean model of IS Success. They even substituted the

measure Use to Intension of use because the former is a behavior and latter being an attitude

(Delone, 2003). Dimensions like Individual Impact and Organizational impact were

combined into one single variable called Net benefits.' Net benefits refers to both the measures

i.e. Use and User satisfaction.'

A revised model of Delone and McLean model of IS Success was proposed by (Jen-Hwa Hu,

2003) in which the measures used for the model differed from the original Delone and McLean

model. They were: Input data quality, system quality, information quality, system use, User

satisfaction, service impacts, Individual impacts, organizational impacts. Comparing both the

models, the new IS Success dimension i.e. Input data quality was included. In the original model,

Delone and McLean only considered the quality of the information provided by the system.

Quality of input data is also one of the important factors in evaluating IS Success because the

resulting services will get effected by the quality of input data (Jen-Hwa Hu, 2003). The revised

model also includes service quality as an additional dimension.

Organizations major concern is the failure of IS Success. To increase the rate of IS success, IS

evaluation should be given importance. (Mohd Al-adaileh, 2009) Proposed a conceptual model

based on the Delone and McLean model of IS Success (original and updated) and TAM

(Technology acceptance model). This model was used to test an evaluation model of IS Success

from the users perspective. The dimensions used for the proposed model are Information

quality, perceived ease of use, perceived usefulness, user technical capabilities, management

support, user's perception of IS success. Systems ease of use was excluded from this model.

HOT-Fit model (Yusof et al., 2006) of HIS evaluation was developed using Delone and McLean

model for evaluation of IS Success and IT- fit model. This new model complements the Delone

and McLean model and consists of seven dimensions of HIS success: System Quality, Service

29

Quality, System Use, Net benefits. Information Quality, Structure, environment and User

satisfaction Authors modified the above-listed frameworks for their studies in order to derive

new frameworks from these existing models. These frameworks also helped as validation of the

various existing models. From this discussion, it is clear that Delone and McLean model of IS

Success and several other models which used this model have some dimensions in common such

as Information quality, system quality, service quality and user satisfaction. This states that these

dimensions were considered necessary for the evaluation purpose. On the other hand, dimensions

like input data quality, individual impact, organizational impact, user technical capabilities, etc.

were added considering the different aspects for which the model is used.

Using the snowballing technique, other relevant frameworks/models were identified. (Mohd Al-

adaileh, 2009) TAM (Technology acceptance model) is considered as one of the popular model

of IT adoption. TAM including other models like its evolution TAM2, the Diffusion of

Innovation (DOI), the Unified Theory of Acceptance and Use of Technology (UTAUT), an

Extension of the UT AUT Model and the Delone and McLean Model of Information Systems

Success are developed by the researchers to examine various aspects like acceptance, diffusion,

adoption and success of Information Systems (Srur and Drew, 2012).

TRA (Theory of Reasoned Actions) (Topacan et al., 2008) is considered as one of the models

that explain the human behavior in the adoption process. Later, an additional construct was

added to the model and the new model was named Theory of Planned Behavior (TPB). For

predicting technology acceptance in the medical sector, Daviss TAM model is considered as

more suitable as compared to TRA to predict technology acceptance in the medical sector.

(Glasgow, 2007) discussed two of the evaluation frameworks, one is RE-AIM which stands for

Reach, Effectiveness, Adoption, Implementation or consistency of program delivery, and

Maintenance or sustainability. RE-AIM model focuses on both individual and setting level

factors which are critical

for public health impact (Glasgow, 2007) and the other model discussed is PRECEDE-

PROCEED model.

Other model discussed in the study are TDQM- Total data quality management, TIQM- total

information quality management, PRISM, The Canadian Institute for Health Information (CIHI)

30

IQ framework, AHIMA data quality management model. HOT-fit model proposed by (Mohd et

al., 2006) combines the approaches of existing models of IS Evaluation mostly on two models

IT-Organization Fit and IS Success Model. The proposed framework was evaluated and was

suggested that this framework can be made into use as a tool to conduct enhanced HIS

evaluation. A framework for evaluating Information System in healthcare setting in order to

ensure effective implementation and positive impact of HIS on healthcare delivery.

4.2 RQ2- What are the main evaluation domains? Analyzing the results based on the

classification of publications according to the domains, majority of the studies were based on

telemedicine evaluation. Reviewing the studies selected for this research, studies were grouped

based on the different domains. A pie-chart representation is used to illustrate the distribution of

studies indifferent domains. Figure 11, demonstrates about 40% of the studies are based on

Telemedicine, 22% covers HIS (Health information system) domain, 17% focuses on IS

evaluation. Few studies discuss about the remaining domains. Out of 40 studies considered,

which were searched using the search string (Table 1), only 5% of the studies were related to

connected health evaluation.

Figure 11 Research Domains

5%

40%

17%

22%

7%

3% 3% 0% 3%

Domains

Connected Health

Telemedicine

IS

Health information systems

e-Health

m-Health

Telehealth

Assisted Living

Health informatics

31

A total of 40 papers were used for in-depth analysis. Papers were grouped together based on the

classification scheme discussed in Table 3. Bubble plot representation was used to represent the

different domains and the type of paper identified. Figure 12 presents the two dimensions i.e.

Context Facet, which includes the different domains (e.g. connected health, telemedicine, Health

information system, health informatics etc.) and Research Facet which includes the type of

papers (e.g. Evaluation research, validation research, solution proposal, etc.)

The highest number of papers focuses on the telemedicine domain and IS research and their

research types are Evaluation Research and Philosophical paper respectively. Figure 12 gives a

clear picture that none of the study described an opinion paper. Insignificant number of study

described validation research. The most common type of the research dimension is philosophical

papers and Evaluation Research.

Among the list of selected studies, some were a combination of more than one type of research

facet. Similarly, some studies covered more than one domain in their study e.g. (Le Gldic et al.,

2011) discussed more than one domain i.e. both telemedicine and e-health systems.

Discussing about Connected Health Technologies evaluation, (Mathur et al., 2007) and (O'Neill

et al., 2012) frameworks used for evaluating connected health. Discussed a functional

framework for the analysis of diabetes care improvement programs that used communication

technology and provides a systematic approach to the problem involved was proposed by

(Mathur et al., 2007). The framework used for analysis consisted of 3 stages: 1) Identification of

functional components 2) Identification of structural components, 3) Identification of the level of

automation in program design. Functional components used for this framework includes: Remote

monitoring, dynamic feedback, motivation support, coordination of care, structured Education.

32

Figure 12

4.3 RQ3- What are the types of research methods used for evaluation? The types of analysis that

helped us categorize the selected articles for the study are:

i) Qualitative Analysis

ii) Quantitative analysis

Figure 13, it is evident that about 90% of the studies followed the qualitative approach for

analysis whereas 15% adopted quantitative approach. At times, combinations of both qualitative

and quantitative methods are used. In some case qualitative methods are used to prepare

quantitative studies and sometimes to support qualitative argumentation, quantitative

measurements are used (Ammenwerth et al., 2009). In this study, a small number (10%) used

both the approaches.

Figure 13

33

In order to specify the type of methodology used and classify the studies, different research

methods such as interview, questionnaires, meetings, case study etc. that are adopted by studies

included in this research.

Majority (12%) studies undertook Questionnaire and interviews as part of their research

methodology. The second major research method which were identified in the study was

participant observation or patient interventions which covered 10% of the all the research

methods used in the identified studies. It needs to be noted that some studies did not specify the

type of the survey method used, those studies were included under Survey (7%). Meetings and

case studies were also a part of this evaluation with 3% and 5% respectively.

4.4 RQ4- What are the challenges in evaluating health Technologies? As stated by (Jen-Hwa Hu,

2003), challenges relating to technology and management should be addressed by healthcare

organizations in order to achieve the ultimate success of telemedicine. During IT evaluation, not

only the technology but also the approach in which the technology is used by the users should be

taken into account (Ammenwerth et al., 2003). Telemedicine evaluation should not be confined

to examine only the factors which lead to the effectiveness of the telemedicine application but

also evaluation should determine why was it not effective (Brear, 2006).

(Ammenwerth et al., 2003) discuss the problems, its consequences and the possible solutions for

the problems in evaluation of IT in healthcare. They are:

1. Complexity of the evaluation object

2. Complexity of the evaluation project

3. Motivation for evaluation

Problems related to the complexity of the evaluation object are while evaluating the information

systems. Evaluation will not only focus on the hardware and software but also on the users

interaction with the system. Not only this, evaluation also deals with other aspects such as the

way the technology or system is introduced and used in the organization, how useful it is for the

users, etc.

Consequences (Ammenwerth et al., 2003) to the problem of complexity of the evaluation

includes 1) IT usability changes during the first stage of evaluation. As a result, a considerable

34

amount of time is required to implement the technology and to get the evaluation results. 2) Due

to changes in work organization and staff, the use of information technology might get affected

and change with time. The authors referred to this problem as the moving evaluation target. 3)

Type of users based on their system knowledge, the workflow, and the functionalities of the

information system used in different departments might differ even if the information systems

are similar.

Complexity of the evaluation project is also considered as one of the other problem areas in

evaluating connected health technologies. Evaluation is usually performed in real and complex

health care environment and should consider all kinds of stakeholders of the technology which

includes different professional groups like nurses, doctors, patients, etc. Evaluation methods will

differ depending on the different view point such as economic, sociologic, psychological,

organizational, technical, information logistical, or clinical (Ammenwerth et al., 2003).

Consequences to the problem of complexity of the evaluation project includes: 1) Different

stakeholder has different evaluation questions which make it difficult to form a definite

evaluation criteria. 2) Evaluation question may change depending on the intermediate evaluation

results, changes in stakeholders opinion or the information technology itself. 3) Selection of a

suitable evaluation design or evaluation method is often regarded as a problem during evaluation

studies.

Third problem area includes motivation for evaluation (Ammenwerth et al., 2003). Evaluation is

only possible if the stakeholders agree to take part in this evaluation process. Evaluation study is

conducted only when there sufficient number of participants and funding available.

Consequences to the above-mentioned problem include: 1) Due to the lack of participants

willingness to be a part of the evaluation study, often there is a shortage of getting sufficient

resources. 2) The problem of external validity can be minimized by including the participants for

the study who are happy to volunteer.

Challenges related to evaluation of telemedicine include identifying many techniques and

sources for collection of health information (Brear, 2006). Communication between telemedicine

applications between two sites may differ because of the different ways in which the information

35

is recorded and stored. Different sites may have a different context, individual perspective and

different ways of using the technology (Brear, 2006) which makes the evaluation result to differ

from one organization to other.

(Heathfield et al., 1998) discuss the challenges involved in evaluating information technology in

healthcare. According to the author, clinical systems are embedded social systems with different

people, providers, settings, etc. There is a need to recognize and predict the behavior of systems

and deliver essential knowledge to notify further developments. Successful interdisciplinary

working requires overcoming cultural and methodological division which involves setting up of

trust and mutual respect (Pagliari, 2007).

36

Chapter 5

Discussion

This research was previously planned to identify the evaluation techniques for connected health

evaluation but because of the unavailability of much literature related to connected health.

Connected health domain is considered as an emerging paradigm and a lot needs to be explored.

We broadened the research area to Health technologies. The results of this systematic mapping

study indicate that there is still little information on the evaluation of connected health

technologies.

Results obtained from this mapping study could be put into use by other researchers as evidence

and can continue this research based on the results obtained.

5.1 Frameworks

Among the list of selected papers, the most referred model/framework for Information system

success assessment is Delone and McLean model of IS Success. Studies used this model for their

research and for developing a new model. Almost all the studies which used IS Success model as

their basis for research initially evaluated the model and then argued on the addition or

elimination of specific dimensions based on the context. To date, there is no particular

framework which is accepted universally for the evaluation of any particular area of health

technologies. The solution to this problem is by developing a model with different dimensions

which can be customized for evaluating the different health technologies. Analyzing all the

selected papers for this study it is noticed that not many evaluation frameworks were actually

evaluated (either self-evaluated or evaluated by others). This study identified only 4 different

frameworks which were evaluated.

Connected Health also involves technology and assessment of information system success is

required. Delone and Mclean model which is used for evaluating the IS Success can be used for

evaluating connected health technologies. Dimensions can be selected from a set of dimensions

by referring various evaluation frameworks.

37

Some of the dimensions of the IS Success model for telemedicine evaluation which were

considered important were: Information quality, system quality, service quality and user

satisfaction. Out of 40 studies, 7 studies evaluated the original (1992) and updated (2003) Delone

and McLean IS Success model. For the successful implementation of IT in healthcare, IS

Success is necessary. Because of the popularity of IS Success model by Delone and McLean,

many research were conducted in this domain, the popularity is also seen in the results obtained

from this study. Results of this study show that more focus is on one model i.e. Delone and

McLean model which is because of the popularity of the model but there will be no good when

concentrating on just one evaluation model. Other models related to IS Success and Health

technologies evaluation needs to be evaluated in order to extract new dimensions or variables.

5.2 Research Domains

This research work presented in the form of research work is not only confined to a specific

domain of health technologies rather it includes several other domains like Telemedicine, e-

Health, m-Health, Health Information systems, telehealth, Connected health, etc. Because of the

large involvement of Information technology in healthcare, making it more effective also

involves some risks. This makes evaluation of the health technologies a necessity.

Telemedicine being one of the oldest technologies used in healthcare, a good number of papers

could be found related to this domain. A lot of Research has been conducted in this field of

telemedicine in the past few decades and from this study we found that about 40% of the papers

discussed telemedicine. Recent works are related to connected health technologies and a few

papers can be found which shows the progress in research in this domain. Figure 14 shows the

domain and the year in which the different studies were published.

38

Figure 14. Research Domain Vs Year of Publication

Primary focus of this mapping study is to identify the various frameworks used for the

assessment of Health technologies. When conducting the search for relevant papers, many

studies focusing on telemedicine assessment methods and approaches were identified.

Data is represented by classifying the papers according to research facets discussed in Table 3

and the context facet which includes different domain of IT in healthcare. It is noted that

considering those 40 selected papers, the most common type of research facet used for the study

is Evaluation Research and Philosophical Paper. Only 2 studies used validation research for their

study which includes techniques which were new and were not yet implemented (Petersen et al.,

2008). Philosophical papers provide a new approach of looking at the already existing models by

organizing the field in the form of a conceptual framework (Petersen et al., 2008). Possibly,

developing a framework/ model right from the beginning is considered difficult as compared to

evaluating already existing models. It is much easier to evaluate and validate the existing model

rather than developing a new model. In this study, none of the papers was opinion papers as

these type of paper expresses opinion of the authors and usually do not depend on earlier works

related to it (Petersen et al., 2008). The reason for selecting evaluation paper type perhaps is that

previous research works related to a specific area gives the new research base or a direction for

further developments/research in the field. Systematic mapping studies could be very useful and

39

can help provide support for the new research because starting a research in a specific domain

which is entirely new requires much time and effort.

5.3 Research Methods

In order to conduct research in any of the domains in healthcare, we need to investigate a wide

range of topics, identify the gaps in the literature which gives an idea about the different issues

and how further research can provide a solution which can be beneficial to society.

Quantitative research method narrows down the result to numbers. Result obtained from

quantitative research is easy for other researchers to start a discussion because it is too direct.

Qualitative research, on the other hand, gives an in-depth understanding on a specific topic and

gives a clear picture of what needs to be done. For the in-depth analysis, qualitative research

method takes time and effort.

Sometimes, both qualitative and quantitative approaches are used for a study. A qualitative

method used along with quantitative research justifies the response for the study. This study

followed both qualitative and quantitative research as we intended to review the literature and

carried out a mapping study based on the research conducted. A huge difference in the use of

research methods can be seen from the results obtained (Figure 12). It was found that among the

list of selected studies, 90% of the studies followed qualitative method for research. Majority of

the papers identified in this study followed qualitative research despite of the fact that it takes

more time, effort and it not suitable for obtaining statistical data.

5.4 Challenges

To increase the rate of successful implementation of the health technologies, challenges should

be taken into consideration. Much research includes challenges in health technologies, but very

few challenges were identified in the evaluation of health technologies.

While there is emerging evidence of the challenges involved in evaluating some new areas of

health technologies like connected health. Evaluation needs to be conducted based on the context

of implementation. Evaluation results may vary depending on the type of participants,

stakeholders, sites, context in which it is implemented, etc.

40

5.5 Limitation of this Research

This research might have experienced some limitations. These are discussed below:

Not all the studies are included in this research. Some studies might not have appeared in

the search conducted because of the keywords in the search strings and could be missed.

Scientific libraries selected for the selection of studies related to the evaluation of health

technologies were kept minimum keeping time into account.

To make sure all the major studies related to the topic were included, the snow- balling

technique was used which took additional time than finding the relevant papers using the

search string.

During data extraction stage, classification of the study was done based on authors

judgement. Regardless of carefully extracting the data, there might be a possibility of

classifying it incorrectly.

41

Chapter 6

Conclusion and Future Works

6.1 Conclusion

In this paper we presented a systematic mapping study on the evaluation of health technologies.

This study provides an initial evidence for evaluation in the field IT healthcare.

This research includes a review of the existing frameworks for the evaluation of health

technologies regardless of the healthcare IT domain. Among the frameworks identified in the

study, some were evaluated and others were theoretical model (yet to be implemented).

Dimensions of different frameworks were also analysed. A comparison of some of the

dimensions of the identified frameworks were done in order to identify the reason for adopting

some dimensions/ variables over others. Health technologies evaluation model can be applied to

several context depending on the requirements.

Inorder to accept a technology in a hospital setting, focus should be on the evaluation of

technology. The success of the piece of technology depends on how well the technology is used

by its user, the context in which it is applied, and how effective the technology will be after

implementing it to hospital settings. Evaluating technology will help reduce the risks.

Assessment of Information systems is a must for evaluating Health technologies. During IT

evaluation, not only the technology but also the approach in which the technology is used by the

users/ stakeholders should be taken into consideration. The emergence of new areas in this field

such as connected health is significant and underlines the importance of adaptive evaluation

frameworks.

Key findings

6.1.1 Frameworks for assessing health technologies:

A good number of frameworks were identified while reviewing the literature. Delone and

Mclean model is evaluated by many researchers for their study which provide a

framework for the evaluation of Information system success.

http://www.bing.com/search?q=theoretical&FORM=AWRE

42

6.1.2 Main evaluation Domains:

Highest numbers of studies were identified in telemedicine. It is foun