PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej...

34
PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu „piątej generacji” badań ewaluacyjnych Some Visions for Fifth Generation Evaluation VI KONFERENCJA EWALUACYJNA WARSZAWA, 13-14.12.2010

Transcript of PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej...

Page 1: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY

Podejścia mieszane – przyczynek do modelu „piątej generacji” badań ewaluacyjnych

Some Visions for Fifth Generation Evaluation

VI KONFERENCJA EWALUACYJNAWARSZAWA, 13-14.12.2010

Page 2: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

Some background information

• My interest in research methodology goes back to the beginning of 1990s.

• Since 1994 I have been a lecturer in data analysis; since 2004 also lecturer in research methods.

• My (5y+2y) Master’s thesis, defended in 1996, was about teaching statistics in the framework of research methods courses.

• Got interested in mixed methods research in 1998 during my one-year study in Cambridge University

• In the doctoral dissertation published in 2004 I focused on the combined use of quantitative and qualitative approaches in educational research

• Belonging to the editorial board of Journal of Mixed Methods Research and International Journal of Multiple Research Approaches

Page 3: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

Further reading related to the topic of speech

• Niglas, K. (2010). The multidimensional model of research methodology.

An integrated set of continua. Tashakkori, A., Teddlie, C. (Eds). Handbook of Mixed

Metods Research. 2nd Ed. Sage Publications Ltd

• Niglas, Katrin (2008). How the novice researcher can make sense of mixed methods

designs. International Journal of Multiple Research Approaches, 1, 13 - 33.

• Niglas, K.; Kaipainen, M.; Kippar, J. (2008). Multi-perspective exploration as a tool for

mixed methods research. . M.M. Bergman (Ed). Advances in Mixed Methods Research:

Theories and Applications. Sage Publications Ltd

• Niglas, Katrin (2007). Spreadsheet Software Can Facilitate Mixed Methods Research

Using Old Tools in a New Context! Journal of Mixed Methods Research, 1(3), 297 – 29

• Niglas, K. (2004) The Combined Use of Qualitative and Quantitative Methods in

Educational Research. Tallinn Pedagogical University. Dissertations on Social Sciences.

Short version:

http://www.tlulib.ee/files/arts/95/nigla32417030233e06e8e5d471ec0aaa32e9.pdf

PS! Published version of the dissertation (book) is available by request (e-mail:

[email protected])!

• Several papers available at Education Line http://www.leeds.ac.uk/educol/

Page 4: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

Change in the aims and forms of evaluation

Guba and Lincoln (1989) describe 4 generations in evaluation of social and political programs:

1. Evaluation ≈ measurement, testing

2. Evaluation ≈ description, mapping

3. Evaluation ≈ substantive evaluation, judgement, advise

4. Evaluation ≈ collaboration, negotiation, initiation and guidance of change

Page 5: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

From positivism to constructivism …Critique of the first three generations by G & L (1989)

1. A tendency towards managerialismmanagers who organise evaluation as well as evaluators are

outsiders and their relationship with/to stakeholders is

disempowering

2. A failure to accommodate value pluralismscientific mode comes with the claim of the value-freedom, but as

evaluation is essentially about valuing, the value-free in practice

means imposing the values of one group and not accommodating

value differences

3. An overcommitment to the scientific paradigm of

inquirycontext stripping; overdependence on quantitative measurement;

belief in scientific truth leaves no room for negotiation or

alternative explanations and frees evaluators from moral

responsibility

Page 6: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

From positivism to constructivism …Main principles for 4th generation evaluation by G&L (1989)

4th Generation evaluation

1. is organised by the claims, concerns, and issues of

stakeholding audiences

2. utilises the methodology of the constructionist

paradigm

3. is a process whereby evaluators and stakeholders

jointly and collaboratively create (or move towards) a

consensual valuing construction of some evaluand This process is: local, sociopolitical, emergent, continuous,

recursive, and divergent, involves teaching/learning of all parties

and sets them into hermeneutic dialectic relationship

Page 7: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

From positivism to constructivism …Where are we 20 years after manifestation of 4th Generation?

1. Are we there? (read: has constructivist paradigm won the paradigm wars and

we all work within this framework?)

2. Do we want to be there? (read: are we convinced that constructivism as The paradigm is

a desired end for our journey on conceptualising evaluation and

it’s methodology?)

3. Do we have an alternative?(read: is our choice constrained by positivism and

constructivism or …?)

Page 8: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

From positivism to constructivism …My answers to the three questions set on the last slide:

1. Are we there? (read: has constructivist paradigm won the paradigm wars and

we all work within this framework?)

No!

– a quick glance at the evaluation practice tells us that wealth

of evaluation studies are not conceptualised and carried out in

this framework

Page 9: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

From positivism to constructivism …My answers to the three questions set on the last slide:

2. Do we want to be there? (read: are we convinced that constructivism as The paradigm

is a desired end for our journey on conceptualising evaluation

and it’s methodology?)

No, not all of us!

– Even though there is a lot to learn from constructivist

tradition many of us resist the forced choice between

positivism and constructivism and do not accept all the

assumptions that constructivism brings with it (i.e relativist

ontology, strong belief in and either or stance on paradigms,

overemphasized belief in impossibility of representativeness and

generalizations as well as that change cannot be engineered, etc.)

Page 10: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

From positivism to constructivism …My answers to the three questions set on the last slide:

3. Do we have an alternative?(read: is our choice constrained by positivism and

constructivism or …?)

Yes! We can:

– replace the bi-polar paradigmatic view by the conceptualization of

research methodology as a multidimensional integrated set of continua

– reason that the research/evaluation practice is inevitably lead by

sophisticated and changing personal worldviews or mental models of

the parties rather than well defined and incommensurable paradigms

– set facilitating change as the sovereign aim and the staple framework

for the evaluation without having to declare either the stakeholders’ or

the managers/experts’ position, needs, views, etc. being always

paramount

Page 11: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

The idea of incommensurable paradigms

The Conventional and Constructivist Belief Systems (Adapted from Guba and Lincoln 1989)Fundamentalquestions

Conventionalbeliefs

Constructivistbeliefs

What is there that can beknown? –ONTOLOGY

REALISM RELATIVISM

What is relationship ofthe knower to the known(or knowable)? –EPISTEMOLOGY

OBJECTIVIST SUBJECTIVIST

What are the ways offinding out knowledge? –METHODOLOGY

INTERVENTIONIST HERMENEUTIC

Page 12: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

qualitative quantitativesubjective objectiveinductive deductive

participant observation survey techniquesanthropology sociology

naturalism anti-naturalismart science

hermeneutics positivismaristotelian galileanteleological causal

finalistic mechanisticunderstanding explanation

Verstehen Erklarenphenomenological logical positivism

* bad gooddescriptive predictiveempiricism rationalism

atheoretical theoretical

Common dichotomies in methodological literature

Page 13: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

Arguments to support rejection of the idea of paradigms and moving towards the idea of continuum

• view which divides social research into limited number of paradigms, whether based on philosophical categories or methodological distinction, is misleading

• research practice is much more variegated and complicated than that proposed by a paradigmatic view

• misguided attribution of two distinct and often opposite sets of qualities to the two large families of methods

Page 14: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

Arguments to support rejection of the idea of paradigms and moving towards the idea of continuum

• misrepresentation of complex philosophical ideas

• major differences in philosophical as well as methodological preferences within the camp of qualitative researchers as well as within the ranks of quantitative researchers

• it is rather a complex and multifaceted individual mental model, which is formed by many factors like education and training, personal values, disciplinary perspectives, philosophy of science, etc., than formalized paradigm that guides the work of social inquirers and evaluators

Page 15: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

On the way towards the idea of continuum

• Extension of the paradigm model to 5 paradigms

by Guba and Lincoln (2001/2005/2008)

• The idea of mixed methods as an interactive continuum

by Ridenour & Newman (2008) • QUAL-MM-QUAN continuum

• Multidimensional continuum of research projects

by Teddlie & Tashakkori (2009)

Page 16: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

On the way towards the idea of continuum

Integrated Multidimensional Continuum of Research Methodology (adapted from Niglas 2001)

• not static or “all-inclusive”

• functions as an illustration supporting an argument against validity of paradigmatic approach to methodology

• helps to organize our thinking about methodology and thereby serves educative purposes

Page 17: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

Original figure available at: www.tlu.ee/~katrin/mmrm/

Page 18: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.
Page 19: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

Experiences from the field ...

“[...] experience in conducting research to bring about social

change and influence social policy indicates that the most

persuasive policy research includes both elements: numbers

that define the scope and patterns of the problem, and a story

that shows how the problem works in daily life and provides

for emphatetic understanding. These two elements stem from

quantitative and qualitative research. Integrated approaches

are effective for [...] analysis of programs, policies, or actions

that helped to overcome persistent inequalities.”

Spalter-Roth, 2000

Page 20: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

The choice of methodology and methods for a study depends on the purposes and research questions

epistemological,

philosophical level:

methodological

level:

technical

level:

data:

quantitative

approach

qualitative

approach

quantitative

methods

qualitative

methods

quantitative

data

qualitative

data

RESEARCH AIMS &

PROBLEM(S)

different schools of thought;

different research traditions

or or

or or

or or

Needs of life and practice; goals of funding bodies; …

(Methodological) skills & commitments; resources; …

Page 21: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

Best fit between aims/problems and designs/strategies(adapted from Creswell & Plano Clark 2007)

Need to ...Usually best suited research design/strategy

see if treatment is effective Experimental design

identify trends/attitudes/... in a population

Survey design

learn about and describe a culture shared by a group

Ethnography design

generate a theory of a process Grounded theory design

imporve the subsistence and/or empower certain group

Action research design

develop or imporve some policies/artifacts

Design research approach

Page 22: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

Facilitating change as sovereign aim for evaluation - practice and change oriented research approaches

Design research

Cyclical nature of the research process:

Diagnose / Analyze Plan action / design & develop Act / test / implement Evaluate / Learn

Need for variegated empirical data –> best results will be often achieved by utilizing mixed methods approach

Monitor

Action research

Page 23: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

Traditional model of methodological aspects is here incorporated into more general framework!

• Research problem (question, hypothesis, purpose, …)

• Strategy (case study, survey, experiment, grounded theory, action research …)

• Sampling (random sample, one case, purposefully chosen cases, … )

• Data collection (structured questionnaire, unstructured/open interview, …)

• Data analysis (statistical methods, open coding, discourse analysis,… )

• Interpretation and conclusions (descriptions, empirical generalisations, …)

Page 24: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

What is design research?

• Design research investigates the process of designing with the overall aim to develop an accessible, robust body of knowledge that enhances our understanding of design processes, applications, methods and contexts. Often, this knowledge helps to define best practice and workable methods in dealing with design and design related problems.

• Alternative approaches/terms used: design-based research, design-science research, design study, design experiment, developmental research, development study, formative research, ...

Page 25: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

What is design research?

• Design research is distinguished from design by the production of interesting (to a community) new knowledge

• Research driven

• Systematic documentation

• Formative evaluation

• Generalization

• DR strives to combine the creativity of design communities with appropriate adherence to standards of traditional research

Page 26: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

Characteristics of good design research

• Development and research takes place through continuous cycles of design, enactment, analysis, and redesign

• The central goals of designing artifacts or environments and developing theories about design process but also about the related phenomena are intertwined

• DR must lead to sharable theories that help communicate relevant implications to practitioners and other designers

• Research must account for how design functions in authentic settings

• The development relies on methods that can document and connect processes of enactment to outcomes of interest

Page 27: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

Design research – the methodological cycle

Indicating the need for development and related knowledge -> -> initial plan of the study

• Analysis of the problem (needs, goals, pre-existing knowledge, ... )

• Designing / monitoring / documenting – planning of the design&development process (work allocation, schedule, applicable methods, ... ) – monitoring the d&d process (memos, team discussions, …) – describing the result of d&d process (sketches, alternatives, resulting design and/or artifact, …)

• Evaluation (incl initial implementation) (testing the design result, evaluation according to standards, feedback from users and/or experts, ... )

• Generalisations (suggestions, requirements, standards, concepts, models, ontologies, theories,...)

Page 28: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

Our design research project:Induction year for novice teachers in Estonia

Novice teacher

0. Need to design a support system

1. Analysing problem, collecting and syntezising theoretical information

2.Design process, constructing/ improving the model

3. Applying the model (being reflective)

4 Evaluating the model and theoretical standpoints

5. Shortcomings of the model and Generalizations

Page 29: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

Induction year for novice teachers in EstoniaPartners and implementation activities:

• Schools (mentors, novie teachers, school leaders)creating propitious environment for professional developmentmentoring beginning teachers

• Universities(re-)developing theoretical model for mentoringmentor trainingsupport programme for beginning teachers

• Ministrydeveloping policyensuring resources

Joint expert group(re-)designing implementation modelfacilitating networkingquality assurance

Page 30: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

Induction year for novice teachers in EstoniaDesign research cycles

Design cycles of our project can be divided into two general phases:

• Cycles I and II - emphasis on planning, designing and re-designing implementation model for induction programme Analysis of theoretical and secondary materials

Focus group interviews with experts, novice, mentors and school leadres

Survey for beginning teachers

Semi-structured questionnaire for mentors

Open interviews for the school leaders

Monitoring and self-evaluation (diaries, peer visits, peer groups, ...)

• Cycles III, IV, etc – emphasis on implementation of the induction programme- Constant evaluation and improvement of the model => monitoring

and research activities continued

Page 31: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

Experiences from our design research project

Well established cyclical process of design research where QUAL and QUAN data is systematically collected and analysed in order to inform design & development process, empovered by strong partnership links provide good basis for evidence based educational restructuring ensuring

• Continuity of the process - ongoing evaluation and development- ongoing partnership

• Sustainability of the project- acceptance by the practitioners- wide-scale implementation

Page 32: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

Towards Fifth Generation Evaluation …some conclusions (read: visions)

The 5th Generation Evaluation can be characterised by:

Conceptualising evaluation as a tool for facilitating development and therefore utilizing cyclical methodological designs which place evaluation into the process of continuous pursuit for improvement

(evaluation not as a purpose of the project per se but implemented within a project to serve wider purposes)

Valuing initiation from stakeholders as well as from experts to start and lead systematic process of evidence-based development

(utilizing designs from action research to design research with various hybrids in between)

Page 33: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

Towards Fifth Generation Evaluation …some conclusions (read: visions)

Open and inclusive stance towards various worldview positions and methodological approaches where paradigmatic confrontation is commuted for continuum model accommodating overlaps between and being enriched by the differences there are between traditions

(dignifying personal mental model instead of paradigm, following the principle of informed creativity, choosing methods and techniques in concord with specific aims and questions considering QUAN, QUAL as well as MM approaches)

Acceptance that evaluation can generate results that apply to the unique local settings as well as results that serve more general shared needs for development

(accepting and utilizing different forms of generalizations)

Page 34: PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej generacji badań ewaluacyjnych Some Visions for Fifth Generation.

Katrin NiglasPhD, Professor

Institute of [email protected]; www.tlu.ee/~katrin/

Thank you !