PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej...
-
Upload
magdalena-keal -
Category
Documents
-
view
218 -
download
0
Transcript of PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY Podejścia mieszane – przyczynek do modelu piątej...
PhD, Professor KATRIN NIGLAS TALLINN UNIVERSITY
Podejścia mieszane – przyczynek do modelu „piątej generacji” badań ewaluacyjnych
Some Visions for Fifth Generation Evaluation
VI KONFERENCJA EWALUACYJNAWARSZAWA, 13-14.12.2010
Some background information
• My interest in research methodology goes back to the beginning of 1990s.
• Since 1994 I have been a lecturer in data analysis; since 2004 also lecturer in research methods.
• My (5y+2y) Master’s thesis, defended in 1996, was about teaching statistics in the framework of research methods courses.
• Got interested in mixed methods research in 1998 during my one-year study in Cambridge University
• In the doctoral dissertation published in 2004 I focused on the combined use of quantitative and qualitative approaches in educational research
• Belonging to the editorial board of Journal of Mixed Methods Research and International Journal of Multiple Research Approaches
Further reading related to the topic of speech
• Niglas, K. (2010). The multidimensional model of research methodology.
An integrated set of continua. Tashakkori, A., Teddlie, C. (Eds). Handbook of Mixed
Metods Research. 2nd Ed. Sage Publications Ltd
• Niglas, Katrin (2008). How the novice researcher can make sense of mixed methods
designs. International Journal of Multiple Research Approaches, 1, 13 - 33.
• Niglas, K.; Kaipainen, M.; Kippar, J. (2008). Multi-perspective exploration as a tool for
mixed methods research. . M.M. Bergman (Ed). Advances in Mixed Methods Research:
Theories and Applications. Sage Publications Ltd
• Niglas, Katrin (2007). Spreadsheet Software Can Facilitate Mixed Methods Research
–
Using Old Tools in a New Context! Journal of Mixed Methods Research, 1(3), 297 – 29
• Niglas, K. (2004) The Combined Use of Qualitative and Quantitative Methods in
Educational Research. Tallinn Pedagogical University. Dissertations on Social Sciences.
Short version:
http://www.tlulib.ee/files/arts/95/nigla32417030233e06e8e5d471ec0aaa32e9.pdf
PS! Published version of the dissertation (book) is available by request (e-mail:
• Several papers available at Education Line http://www.leeds.ac.uk/educol/
Change in the aims and forms of evaluation
Guba and Lincoln (1989) describe 4 generations in evaluation of social and political programs:
1. Evaluation ≈ measurement, testing
2. Evaluation ≈ description, mapping
3. Evaluation ≈ substantive evaluation, judgement, advise
4. Evaluation ≈ collaboration, negotiation, initiation and guidance of change
From positivism to constructivism …Critique of the first three generations by G & L (1989)
1. A tendency towards managerialismmanagers who organise evaluation as well as evaluators are
outsiders and their relationship with/to stakeholders is
disempowering
2. A failure to accommodate value pluralismscientific mode comes with the claim of the value-freedom, but as
evaluation is essentially about valuing, the value-free in practice
means imposing the values of one group and not accommodating
value differences
3. An overcommitment to the scientific paradigm of
inquirycontext stripping; overdependence on quantitative measurement;
belief in scientific truth leaves no room for negotiation or
alternative explanations and frees evaluators from moral
responsibility
From positivism to constructivism …Main principles for 4th generation evaluation by G&L (1989)
4th Generation evaluation
1. is organised by the claims, concerns, and issues of
stakeholding audiences
2. utilises the methodology of the constructionist
paradigm
3. is a process whereby evaluators and stakeholders
jointly and collaboratively create (or move towards) a
consensual valuing construction of some evaluand This process is: local, sociopolitical, emergent, continuous,
recursive, and divergent, involves teaching/learning of all parties
and sets them into hermeneutic dialectic relationship
From positivism to constructivism …Where are we 20 years after manifestation of 4th Generation?
1. Are we there? (read: has constructivist paradigm won the paradigm wars and
we all work within this framework?)
2. Do we want to be there? (read: are we convinced that constructivism as The paradigm is
a desired end for our journey on conceptualising evaluation and
it’s methodology?)
3. Do we have an alternative?(read: is our choice constrained by positivism and
constructivism or …?)
From positivism to constructivism …My answers to the three questions set on the last slide:
1. Are we there? (read: has constructivist paradigm won the paradigm wars and
we all work within this framework?)
No!
– a quick glance at the evaluation practice tells us that wealth
of evaluation studies are not conceptualised and carried out in
this framework
From positivism to constructivism …My answers to the three questions set on the last slide:
2. Do we want to be there? (read: are we convinced that constructivism as The paradigm
is a desired end for our journey on conceptualising evaluation
and it’s methodology?)
No, not all of us!
– Even though there is a lot to learn from constructivist
tradition many of us resist the forced choice between
positivism and constructivism and do not accept all the
assumptions that constructivism brings with it (i.e relativist
ontology, strong belief in and either or stance on paradigms,
overemphasized belief in impossibility of representativeness and
generalizations as well as that change cannot be engineered, etc.)
From positivism to constructivism …My answers to the three questions set on the last slide:
3. Do we have an alternative?(read: is our choice constrained by positivism and
constructivism or …?)
Yes! We can:
– replace the bi-polar paradigmatic view by the conceptualization of
research methodology as a multidimensional integrated set of continua
– reason that the research/evaluation practice is inevitably lead by
sophisticated and changing personal worldviews or mental models of
the parties rather than well defined and incommensurable paradigms
– set facilitating change as the sovereign aim and the staple framework
for the evaluation without having to declare either the stakeholders’ or
the managers/experts’ position, needs, views, etc. being always
paramount
The idea of incommensurable paradigms
The Conventional and Constructivist Belief Systems (Adapted from Guba and Lincoln 1989)Fundamentalquestions
Conventionalbeliefs
Constructivistbeliefs
What is there that can beknown? –ONTOLOGY
REALISM RELATIVISM
What is relationship ofthe knower to the known(or knowable)? –EPISTEMOLOGY
OBJECTIVIST SUBJECTIVIST
What are the ways offinding out knowledge? –METHODOLOGY
INTERVENTIONIST HERMENEUTIC
qualitative quantitativesubjective objectiveinductive deductive
participant observation survey techniquesanthropology sociology
naturalism anti-naturalismart science
hermeneutics positivismaristotelian galileanteleological causal
finalistic mechanisticunderstanding explanation
Verstehen Erklarenphenomenological logical positivism
* bad gooddescriptive predictiveempiricism rationalism
atheoretical theoretical
Common dichotomies in methodological literature
Arguments to support rejection of the idea of paradigms and moving towards the idea of continuum
• view which divides social research into limited number of paradigms, whether based on philosophical categories or methodological distinction, is misleading
• research practice is much more variegated and complicated than that proposed by a paradigmatic view
• misguided attribution of two distinct and often opposite sets of qualities to the two large families of methods
Arguments to support rejection of the idea of paradigms and moving towards the idea of continuum
• misrepresentation of complex philosophical ideas
• major differences in philosophical as well as methodological preferences within the camp of qualitative researchers as well as within the ranks of quantitative researchers
• it is rather a complex and multifaceted individual mental model, which is formed by many factors like education and training, personal values, disciplinary perspectives, philosophy of science, etc., than formalized paradigm that guides the work of social inquirers and evaluators
On the way towards the idea of continuum
• Extension of the paradigm model to 5 paradigms
by Guba and Lincoln (2001/2005/2008)
• The idea of mixed methods as an interactive continuum
by Ridenour & Newman (2008) • QUAL-MM-QUAN continuum
• Multidimensional continuum of research projects
by Teddlie & Tashakkori (2009)
On the way towards the idea of continuum
Integrated Multidimensional Continuum of Research Methodology (adapted from Niglas 2001)
• not static or “all-inclusive”
• functions as an illustration supporting an argument against validity of paradigmatic approach to methodology
• helps to organize our thinking about methodology and thereby serves educative purposes
Original figure available at: www.tlu.ee/~katrin/mmrm/
Experiences from the field ...
“[...] experience in conducting research to bring about social
change and influence social policy indicates that the most
persuasive policy research includes both elements: numbers
that define the scope and patterns of the problem, and a story
that shows how the problem works in daily life and provides
for emphatetic understanding. These two elements stem from
quantitative and qualitative research. Integrated approaches
are effective for [...] analysis of programs, policies, or actions
that helped to overcome persistent inequalities.”
Spalter-Roth, 2000
The choice of methodology and methods for a study depends on the purposes and research questions
epistemological,
philosophical level:
methodological
level:
technical
level:
data:
quantitative
approach
qualitative
approach
quantitative
methods
qualitative
methods
quantitative
data
qualitative
data
RESEARCH AIMS &
PROBLEM(S)
different schools of thought;
different research traditions
or or
or or
or or
Needs of life and practice; goals of funding bodies; …
(Methodological) skills & commitments; resources; …
Best fit between aims/problems and designs/strategies(adapted from Creswell & Plano Clark 2007)
Need to ...Usually best suited research design/strategy
see if treatment is effective Experimental design
identify trends/attitudes/... in a population
Survey design
learn about and describe a culture shared by a group
Ethnography design
generate a theory of a process Grounded theory design
imporve the subsistence and/or empower certain group
Action research design
develop or imporve some policies/artifacts
Design research approach
Facilitating change as sovereign aim for evaluation - practice and change oriented research approaches
Design research
Cyclical nature of the research process:
Diagnose / Analyze Plan action / design & develop Act / test / implement Evaluate / Learn
Need for variegated empirical data –> best results will be often achieved by utilizing mixed methods approach
Monitor
Action research
Traditional model of methodological aspects is here incorporated into more general framework!
• Research problem (question, hypothesis, purpose, …)
• Strategy (case study, survey, experiment, grounded theory, action research …)
• Sampling (random sample, one case, purposefully chosen cases, … )
• Data collection (structured questionnaire, unstructured/open interview, …)
• Data analysis (statistical methods, open coding, discourse analysis,… )
• Interpretation and conclusions (descriptions, empirical generalisations, …)
What is design research?
• Design research investigates the process of designing with the overall aim to develop an accessible, robust body of knowledge that enhances our understanding of design processes, applications, methods and contexts. Often, this knowledge helps to define best practice and workable methods in dealing with design and design related problems.
• Alternative approaches/terms used: design-based research, design-science research, design study, design experiment, developmental research, development study, formative research, ...
What is design research?
• Design research is distinguished from design by the production of interesting (to a community) new knowledge
• Research driven
• Systematic documentation
• Formative evaluation
• Generalization
• DR strives to combine the creativity of design communities with appropriate adherence to standards of traditional research
Characteristics of good design research
• Development and research takes place through continuous cycles of design, enactment, analysis, and redesign
• The central goals of designing artifacts or environments and developing theories about design process but also about the related phenomena are intertwined
• DR must lead to sharable theories that help communicate relevant implications to practitioners and other designers
• Research must account for how design functions in authentic settings
• The development relies on methods that can document and connect processes of enactment to outcomes of interest
Design research – the methodological cycle
Indicating the need for development and related knowledge -> -> initial plan of the study
• Analysis of the problem (needs, goals, pre-existing knowledge, ... )
• Designing / monitoring / documenting – planning of the design&development process (work allocation, schedule, applicable methods, ... ) – monitoring the d&d process (memos, team discussions, …) – describing the result of d&d process (sketches, alternatives, resulting design and/or artifact, …)
• Evaluation (incl initial implementation) (testing the design result, evaluation according to standards, feedback from users and/or experts, ... )
• Generalisations (suggestions, requirements, standards, concepts, models, ontologies, theories,...)
Our design research project:Induction year for novice teachers in Estonia
Novice teacher
0. Need to design a support system
1. Analysing problem, collecting and syntezising theoretical information
2.Design process, constructing/ improving the model
3. Applying the model (being reflective)
4 Evaluating the model and theoretical standpoints
5. Shortcomings of the model and Generalizations
Induction year for novice teachers in EstoniaPartners and implementation activities:
• Schools (mentors, novie teachers, school leaders)creating propitious environment for professional developmentmentoring beginning teachers
• Universities(re-)developing theoretical model for mentoringmentor trainingsupport programme for beginning teachers
• Ministrydeveloping policyensuring resources
Joint expert group(re-)designing implementation modelfacilitating networkingquality assurance
Induction year for novice teachers in EstoniaDesign research cycles
Design cycles of our project can be divided into two general phases:
• Cycles I and II - emphasis on planning, designing and re-designing implementation model for induction programme Analysis of theoretical and secondary materials
Focus group interviews with experts, novice, mentors and school leadres
Survey for beginning teachers
Semi-structured questionnaire for mentors
Open interviews for the school leaders
Monitoring and self-evaluation (diaries, peer visits, peer groups, ...)
• Cycles III, IV, etc – emphasis on implementation of the induction programme- Constant evaluation and improvement of the model => monitoring
and research activities continued
Experiences from our design research project
Well established cyclical process of design research where QUAL and QUAN data is systematically collected and analysed in order to inform design & development process, empovered by strong partnership links provide good basis for evidence based educational restructuring ensuring
• Continuity of the process - ongoing evaluation and development- ongoing partnership
• Sustainability of the project- acceptance by the practitioners- wide-scale implementation
Towards Fifth Generation Evaluation …some conclusions (read: visions)
The 5th Generation Evaluation can be characterised by:
Conceptualising evaluation as a tool for facilitating development and therefore utilizing cyclical methodological designs which place evaluation into the process of continuous pursuit for improvement
(evaluation not as a purpose of the project per se but implemented within a project to serve wider purposes)
Valuing initiation from stakeholders as well as from experts to start and lead systematic process of evidence-based development
(utilizing designs from action research to design research with various hybrids in between)
Towards Fifth Generation Evaluation …some conclusions (read: visions)
Open and inclusive stance towards various worldview positions and methodological approaches where paradigmatic confrontation is commuted for continuum model accommodating overlaps between and being enriched by the differences there are between traditions
(dignifying personal mental model instead of paradigm, following the principle of informed creativity, choosing methods and techniques in concord with specific aims and questions considering QUAN, QUAL as well as MM approaches)
Acceptance that evaluation can generate results that apply to the unique local settings as well as results that serve more general shared needs for development
(accepting and utilizing different forms of generalizations)