PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY

Podejścia mieszane – przyczynek do modelu „piątej generacji” badań ewaluacyjnych

Some Visions for Fifth Generation Evaluation

VI KONFERENCJA EWALUACYJNAWARSZAWA, 13-14.12.2010

Some background information• My interest in research methodology goes back to the beginning of

1990s. • Since 1994 I have been a lecturer in data analysis;

since 2004 also lecturer in research methods. • My (5y+2y) Master’s thesis, defended in 1996, was about teaching

statistics in the framework of research methods courses. • Got interested in mixed methods research in 1998 during my one-year

study in Cambridge University• In the doctoral dissertation published in 2004 I focused on the combined

use of quantitative and qualitative approaches in educational research

• Belonging to the editorial board of Journal of Mixed Methods Research and International Journal of Multiple Research Approaches

Further reading related to the topic of speech• Niglas, K. (2010). The multidimensional model of research methodology.

An integrated set of continua. Tashakkori, A., Teddlie, C. (Eds). Handbook of Mixed Metods Research. 2nd Ed. Sage Publications Ltd

• Niglas, Katrin (2008). How the novice researcher can make sense of mixed methods designs. International Journal of Multiple Research Approaches, 1, 13 - 33.

• Niglas, K.; Kaipainen, M.; Kippar, J. (2008). Multi-perspective exploration as a tool for mixed methods research. . M.M. Bergman (Ed). Advances in Mixed Methods Research: Theories and Applications. Sage Publications Ltd

• Niglas, Katrin (2007). Spreadsheet Software Can Facilitate Mixed Methods Research – Using Old Tools in a New Context! Journal of Mixed Methods Research, 1(3), 297 – 29

• Niglas, K. (2004) The Combined Use of Qualitative and Quantitative Methods in Educational Research. Tallinn Pedagogical University. Dissertations on Social Sciences. Short version: http://www.tlulib.ee/files/arts/95/nigla32417030233e06e8e5d471ec0aaa32e9.pdfPS! Published version of the dissertation (book) is available by request (e-mail: katrin@tlu.ee)!

• Several papers available at Education Line http://www.leeds.ac.uk/educol/

Change in the aims and forms of evaluation

Guba and Lincoln (1989) describe 4 generations in evaluation of social and political programs:

1. Evaluation ≈ measurement, testing2. Evaluation ≈ description, mapping3. Evaluation ≈ substantive evaluation, judgement,

advise4. Evaluation ≈ collaboration, negotiation, initiation

and guidance of change

From positivism to constructivism …Critique of the first three generations by G & L (1989)

1. A tendency towards managerialism

managers who organise evaluation as well as evaluators are outsiders and their relationship with/to stakeholders is disempowering

2. A failure to accommodate value pluralismscientific mode comes with the claim of the value-freedom, but as evaluation is essentially about valuing, the value-free in practice means imposing the values of one group and not accommodating value differences

3. An overcommitment to the scientific paradigm of inquirycontext stripping; overdependence on quantitative measurement; belief in scientific truth leaves no room for negotiation or alternative explanations and frees evaluators from moral responsibility

From positivism to constructivism …Main principles for 4th generation evaluation by G&L (1989)

4th Generation evaluation 1. is organised by the claims, concerns, and issues of

stakeholding audiences 2. utilises the methodology of the constructionist

paradigm3. is a process whereby evaluators and stakeholders

jointly and collaboratively create (or move towards) a consensual valuing construction of some evaluand This process is: local, sociopolitical, emergent, continuous, recursive, and divergent, involves teaching/learning of all parties and sets them into hermeneutic dialectic relationship

From positivism to constructivism …Where are we 20 years after manifestation of 4th Generation?

1. Are we there?

(read: has constructivist paradigm won the paradigm wars and we all work within this framework?)

2. Do we want to be there? (read: are we convinced that constructivism as The paradigm is a desired end for our journey on conceptualising evaluation and it’s methodology?)

3. Do we have an alternative?(read: is our choice constrained by positivism and constructivism or …?)

From positivism to constructivism …My answers to the three questions set on the last slide:

1. Are we there?

(read: has constructivist paradigm won the paradigm wars and we all work within this framework?)

No! – a quick glance at the evaluation practice tells us that wealth of evaluation studies are not conceptualised and carried out in this framework

From positivism to constructivism …My answers to the three questions set on the last slide: 2. Do we want to be there?

(read: are we convinced that constructivism as The paradigm is a desired end for our journey on conceptualising evaluation and it’s methodology?)

No, not all of us! – Even though there is a lot to learn from constructivist tradition many of us resist the forced choice between positivism and constructivism and do not accept all the assumptions that constructivism brings with it (i.e relativist ontology, strong belief in and either or stance on paradigms, overemphasized belief in impossibility of representativeness and generalizations as well as that change cannot be engineered, etc.)

From positivism to constructivism …My answers to the three questions set on the last slide: 3. Do we have an alternative?

(read: is our choice constrained by positivism and constructivism or …?)

Yes! We can:– replace the bi-polar paradigmatic view by the conceptualization of research methodology as a multidimensional integrated set of continua– reason that the research/evaluation practice is inevitably lead by sophisticated and changing personal worldviews or mental models of the parties rather than well defined and incommensurable paradigms– set facilitating change as the sovereign aim and the staple framework for the evaluation without having to declare either the stakeholders’ or the managers/experts’ position, needs, views, etc. being always paramount

The idea of incommensurable paradigms

The Conventional and Constructivist Belief Systems (Adapted from Guba and Lincoln 1989)Fundamentalquestions

Conventionalbeliefs

Constructivistbeliefs

What is there that can beknown? –ONTOLOGY

REALISM RELATIVISM

What is relationship ofthe knower to the known(or knowable)? –EPISTEMOLOGY

OBJECTIVIST SUBJECTIVIST

What are the ways offinding out knowledge? –METHODOLOGY

INTERVENTIONIST HERMENEUTIC

qualitative quantitativesubjective objectiveinductive deductive

participant observation survey techniquesanthropology sociology

naturalism anti-naturalismart science

hermeneutics positivismaristotelian galileanteleological causal

finalistic mechanisticunderstanding explanation

Verstehen Erklarenphenomenological logical positivism

* bad gooddescriptive predictiveempiricism rationalism

atheoretical theoretical

Common dichotomies in methodological literature

Arguments to support rejection of the idea of paradigms and moving towards the idea of continuum

• view which divides social research into limited number of paradigms, whether based on philosophical categories or methodological distinction, is misleading

• research practice is much more variegated and complicated than that proposed by a paradigmatic view

• misguided attribution of two distinct and often opposite sets of qualities to the two large families of methods

Arguments to support rejection of the idea of paradigms and moving towards the idea of continuum

• misrepresentation of complex philosophical ideas• major differences in philosophical as well as

methodological preferences within the camp of qualitative researchers as well as within the ranks of quantitative researchers

• it is rather a complex and multifaceted individual mental model, which is formed by many factors like education and training, personal values, disciplinary perspectives, philosophy of science, etc., than formalized paradigm that guides the work of social inquirers and evaluators

On the way towards the idea of continuum

• Extension of the paradigm model to 5 paradigms by Guba and Lincoln (2001/2005/2008)

• The idea of mixed methods as an interactive continuum

by Ridenour & Newman (2008) • QUAL-MM-QUAN continuum • Multidimensional continuum of research projects by Teddlie & Tashakkori (2009)

On the way towards the idea of continuum

Integrated Multidimensional Continuum of Research Methodology (adapted from Niglas 2001)

• not static or “all-inclusive” • functions as an illustration supporting an argument

against validity of paradigmatic approach to methodology

• helps to organize our thinking about methodology and thereby serves educative purposes

Original figure available at: www.tlu.ee/~katrin/mmrm/

Experiences from the field ...

“[...] experience in conducting research to bring about social change and influence social policy indicates that the most persuasive policy research includes both elements: numbers that define the scope and patterns of the problem, and a story that shows how the problem works in daily life and provides for emphatetic understanding. These two elements stem from quantitative and qualitative research. Integrated approaches are effective for [...] analysis of programs, policies, or actions that helped to overcome persistent inequalities.”

Spalter-Roth, 2000

The choice of methodology and methods for a study depends on the purposes and research questions

epistemological,

philosophical level:

methodological

level:

technical

level:

data:

quantitative approach

qualitative approach

quantitative methods

qualitative methods

quantitative data

qualitative data

RESEARCH AIMS & PROBLEM(S)

different schools of thought;

different research traditions

or or

or or

or or

Needs of life and practice; goals of funding bodies; …

(Methodological) skills & commitments; resources; …

Best fit between aims/problems and designs/strategies(adapted from Creswell & Plano Clark 2007)

Need to ...Usually best suited research design/strategy

see if treatment is effective Experimental design

identify trends/attitudes/... in a population

Survey design

learn about and describe a culture shared by a group

Ethnography design

generate a theory of a process Grounded theory design

imporve the subsistence and/or empower certain group

Action research design

develop or imporve some policies/artifacts

Design research approach

Facilitating change as sovereign aim for evaluation - practice and change oriented research approaches

Design research

Cyclical nature of the research process: Diagnose / Analyze Plan action / design & develop Act / test / implement Evaluate / Learn

Need for variegated empirical data –> best results will be often achieved by utilizing mixed methods approach

Monitor

Action research

Traditional model of methodological aspects is here incorporated into more general framework! • Research problem

(question, hypothesis, purpose, …)• Strategy

(case study, survey, experiment, grounded theory, action research …)• Sampling

(random sample, one case, purposefully chosen cases, … )• Data collection

(structured questionnaire, unstructured/open interview, …)• Data analysis

(statistical methods, open coding, discourse analysis,… )• Interpretation and conclusions

(descriptions, empirical generalisations, …)

What is design research?

• Design research investigates the process of designing with the overall aim to develop an accessible, robust body of knowledge that enhances our understanding of design processes, applications, methods and contexts. Often, this knowledge helps to define best practice and workable methods in dealing with design and design related problems.

• Alternative approaches/terms used: design-based research, design-science research, design study, design experiment, developmental research, development study, formative research, ...

What is design research?

• Design research is distinguished from design by the production of interesting (to a community) new knowledge• Research driven• Systematic documentation• Formative evaluation• Generalization

• DR strives to combine the creativity of design communities with appropriate adherence to standards of traditional research

Characteristics of good design research

• Development and research takes place through continuous cycles of design, enactment, analysis, and redesign

• The central goals of designing artifacts or environments and developing theories about design process but also about the related phenomena are intertwined

• DR must lead to sharable theories that help communicate relevant implications to practitioners and other designers

• Research must account for how design functions in authentic settings

• The development relies on methods that can document and connect processes of enactment to outcomes of interest

Design research – the methodological cycle

Indicating the need for development and related knowledge -> -> initial plan of the study

• Analysis of the problem (needs, goals, pre-existing knowledge, ... )

• Designing / monitoring / documenting – planning of the design&development process (work allocation, schedule, applicable methods, ... ) – monitoring the d&d process (memos, team discussions, …) – describing the result of d&d process (sketches, alternatives, resulting design and/or artifact, …)

• Evaluation (incl initial implementation) (testing the design result, evaluation according to standards, feedback from users and/or experts, ... )

• Generalisations (suggestions, requirements, standards, concepts, models, ontologies, theories,...)

Our design research project:Induction year for novice teachers in Estonia

Novice teacher

0. Need to design a support system

1. Analysing problem, collecting and syntezising theoretical information

2.Design process, constructing/ improving the model

3. Applying the model (being reflective)

4 Evaluating the model and theoretical standpoints

5. Shortcomings of the model and Generalizations

Induction year for novice teachers in EstoniaPartners and implementation activities:

• Schools (mentors, novie teachers, school leaders)creating propitious environment for professional developmentmentoring beginning teachers

• Universities(re-)developing theoretical model for mentoringmentor trainingsupport programme for beginning teachers

• Ministrydeveloping policyensuring resources

Joint expert group(re-)designing implementation modelfacilitating networkingquality assurance

Induction year for novice teachers in EstoniaDesign research cycles

Design cycles of our project can be divided into two general phases:

• Cycles I and II - emphasis on planning, designing and re-designing implementation model for induction programme Analysis of theoretical and secondary materialsFocus group interviews with experts, novice, mentors and school

leadresSurvey for beginning teachersSemi-structured questionnaire for mentors Open interviews for the school leadersMonitoring and self-evaluation (diaries, peer visits, peer groups, ...)

• Cycles III, IV, etc – emphasis on implementation of the induction programme- Constant evaluation and improvement of the model => monitoring

and research activities continued

Experiences from our design research project

Well established cyclical process of design research where QUAL and QUAN data is systematically collected and analysed in order to inform design & development process, empovered by strong partnership links provide good basis for evidence based educational restructuring ensuring

• Continuity of the process - ongoing evaluation and development- ongoing partnership

• Sustainability of the project- acceptance by the practitioners- wide-scale implementation

Towards Fifth Generation Evaluation …some conclusions (read: visions)

The 5th Generation Evaluation can be characterised by:

Conceptualising evaluation as a tool for facilitating development and therefore utilizing cyclical methodological designs which place evaluation into the process of continuous pursuit for improvement

(evaluation not as a purpose of the project per se but implemented within a project to serve wider purposes)

Valuing initiation from stakeholders as well as from experts to start and lead systematic process of evidence-based development

(utilizing designs from action research to design research with various hybrids in between)

Towards Fifth Generation Evaluation …some conclusions (read: visions)

Open and inclusive stance towards various worldview positions and methodological approaches where paradigmatic confrontation is commuted for continuum model accommodating overlaps between and being enriched by the differences there are between traditions

(dignifying personal mental model instead of paradigm, following the principle of informed creativity, choosing methods and techniques in concord with specific aims and questions considering QUAN, QUAL as well as MM approaches)

Acceptance that evaluation can generate results that apply to the unique local settings as well as results that serve more general shared needs for development

(accepting and utilizing different forms of generalizations)

Katrin NiglasPhD, Professor

Institute of Informaticskatrin@tlu.ee; www.tlu.ee/~katrin/

Thank you !