Qual Quant (2013) 47:659676 [625381]

Qual Quant (2013) 47:659–676
DOI 10.1007/s11135-011-9538-6
Mixed methods research synthesis: definition,
framework, and potential
M. Heyvaert ·B. Maes ·P. Onghena
Published online: 9 July 2011
© Springer Science+Business Media B.V . 2011
Abstract Literature on the combination of qualitative and quantitative research compo-
nents at the primary empirical study level has recently accumulated exponentially. However,
this combination is only rarely discussed and applied at the research synthesis level. The
purpose of this paper is to explore the possible contribution of mixed methods research tothe integration of qualitative and quantitative research at the synthesis level. In order to
contribute to the methodology and utilization of mixed methods at the synthesis level, we
present a framework to perform mixed methods research syntheses (MMRS). The presentedclassification framework can help to inform researchers intending to carry out MMRS, andto provide ideas for conceptualizing and developing those syntheses. We illustrate the use ofthis framework by applying it to the planning of MMRS on effectiveness studies concerninginterventions for challenging behavior in persons with intellectual disabilities, presentingtwo hypothetical examples. Finally, we discuss possible strengths of MMRS and note someremaining challenges concerning the implementation of these syntheses.
Keywords Mixed methods research ·Mixed methodology ·Systematic review ·
Research synthesis ·Intellectual disability ·Challenging behavior
Mixed methods research, the research paradigm that encourages the combined use of qual-
itative and quantitative research elements to answer complex questions, is recently gain-ing enormous popularity ( Creswell 2003 ;Greene 2007 ;Johnson and Onwuegbuzie 2004 ;
M. Heyvaert
Research Foundation Flanders (FWO), Brussels, Belgium
M. Heyvaert ( B)·P. Onghena
Methodology of Educational Sciences Research Group, Faculty of Psychology and Educational Sciences,Katholieke Universiteit Leuven, Andreas Vesaliusstraat 2, P.O. Box 3762, 3000 Leuven, Belgiume-mail: [anonimizat]
B. Maes
Parenting and Special Education Research Group, Faculty of Psychology and Educational Sciences,Katholieke Universiteit Leuven, Leuven, Belgium
123

660 M. Heyvaert et al.
Onwuegbuzie and Leech 2005 ;Tashakkori and Creswell 2007 ;Tashakkori and Teddlie
2003b ). Mixed methods research can be applied at the primary empirical study level as
well as at the synthesis level. In a primary level mixed methods study a researcher collects
qualitative and quantitative data directly from the research participants, for example throughinterviews, observations, and questionnaires, and combines these diverse data in a singlestudy. A synthesis level mixed methods study is a systematic review that applies the prin-
ciples of mixed methods research. We refer to this type of systematic review by the notion‘mixed methods research synthesis’ (MMRS). In such a synthesis, the data to be included inthe review are findings extracted from several published qualitative, quantitative, and mixedprimary level articles. A mixed methods approach combining qualitative and quantitativeresearch elements is used to integrate these qualitative and quantitative research findingswithin a single systematic review.
Literature concerning mixed methods research at the primary level has accumulated expo-
nentially ( Hanson 2008 ;Leech and Onwuegbuzie 2009 ). In comparison, very little attention
is paid to the possibilities of mixing qualitative and quantitative methods at the synthesislevel, although we could expect that the synthesis of qualitative and quantitative researchelements could lead to a more integrated and differentiated understanding and insight atthis level as well ( Creswell and Tashakkori 2007b ;Dellinger and Leech 2007 ;Harden and
Thomas 2005 ,2010 ;Hart et al. 2009 ;Sandelowski et al. 2006 ;V oils et al. 2008 ).
Accordingly, over the last two decades several authors have proposed typologies for
designing mixed methods designs at the primary level (Creswell 2003 ;Creswell and Plano
Clark 2007 ;Tashakkori and Teddlie 2003a ). The motives behind the articulation of these ty-
pologies are diverse, and include (1) presenting a flexible organizational structure for mixedmethods research, (2) developing conceptual frameworks that inform and guide the practiceof mixed methods inquiry, (3) offering credibility to the mixed methods field by providingsuccessful examples, (4) providing a common language for this field, and (5) facilitating andenhancing the instruction of courses in mixed methods research ( Collins and O’Cathain 2009 ;
Greene et al. 1989 ;Leech and Onwuegbuzie 2009 ;Teddlie and Tashakkori 2006 ). These argu-
ments for articulating mixed methods typologies are likewise applicable to the primary- as to
thesynthesis level . However, to date there exists no such typology framework for the synthesis
level . As will be argued in this paper, there are some fundamental differences between mixed
methods studies at the primary- and synthesis level, and it would not suffice to simply appeal toexisting typologies for mixed methods studies at the primary level when designing an MMRS.In order to fill this gap, this paper develops and introduces a typology framework for MMRS.
Since the intent of this paper is to explore the possible contribution of mixed methods
research to the integration of qualitative and quantitative research at the synthesis level, westart by defining mixed methods research at the synthesis level. Second, the framework tocarry out an MMRS is presented. Third, this framework is illustrated by applying it to the plan-ning of MMRS on effectiveness studies concerning interventions for challenging behavior inpersons with intellectual disabilities (ID). Two hypothetical examples are presented. Fourth,we discuss possible strengths of MMRS and note some remaining challenges concerning theimplementation of these syntheses.
1 Mixed methods research at the synthesis level
1.1 Synthesizing research evidence
During the last decades, the need to synthesize research evidence in order to inform policy
makers, practitioners, and fellow scientists concerning the most recent developments on a
123

Mixed methods research synthesis 661
Fig. 1 Qualitative, quantitative, and mixed methods research synthesis
certain topic has been recognized ( Chalmers et al. 2002 ;Mays et al. 2005 ). Especially due
to the Evidence-Based Practice Movement (EBP), systematic reviews are nowadays highly
valued as they often form the basis for evidence-based clinical practice guidelines. As aresult, several methods and techniques to systematically aggregate evidence have been (fur-ther) developed. Various terms (e.g., systematic review, integrative review, research synthesis,realist synthesis, qualitative review, narrative review, meta-analysis) are used to describe dif-ferent variants of the methods and techniques developed to synthesize empirical evidence(Forbes and Griffiths 2002 ;Major and Savin-Baden 2010 ;Pluye et al. 2009 ;Suri and Clarke
2009 ;Whittemore and Knafl 2005 ;Zimmer 2006 ).
Historically, two major approaches of research synthesis have been applied. First, a
variety of qualitative synthesis methods—‘systematic review’, ‘narrative review’, ‘meta-study’, ‘meta-synthesis’, ‘meta-summary’, ‘meta-ethnography’, ‘grounded formal theory’,‘aggregated analysis’—is used to generate new insights and understanding from interrelatedqualitative research findings. Second, several statistical models and techniques (e.g., fixedand random effects models, and varying techniques to address heterogeneity and bias) areapplied to conduct meta-analyses of quantitative research evidence. In addition to these twoapproaches, recently some pioneering work has been done concerning the mixed synthe-sis of various types of qualitative, quantitative, and mixed primary level research evidence(Harden and Thomas 2005 ,2010 ;Pluye et al. 2009 ;Sandelowski et al. 2006 ;V oils et al.
2008 ).
As depicted in the left rectangle of Fig. 1,qualitative methods for research synthe-
sis are applied to bring together data collected, analyzed and interpreted in qualitative,and sometimes also in quantitative and mixed primary level studies ( Jensen and Allen
1996 ;Johnson and Onwuegbuzie 2004 ;Mays et al. 2001 ;Paterson et al. 2001 ;Rice 2008 ;
Sandelowski et al. 1997 ;Walsh and Downe 2005 ). As shown in the middle rectangle of
Fig. 1,aquantitative synthesis particularly includes data from quantitative primary level
studies ( Cooper 1998 ;Hampton 2002 ;McKenna et al. 1999 ;Mitchell 1999 ), although some-
times data from qualitative studies are transformed to be incorporated in these synthesesas well ( Johnson and Onwuegbuzie 2004 ;Sandelowski et al. 2009 ). In addition, quantita-
tive data (or data fragments) investigated in primary level mixed studies can be included ina quantitative synthesis. As can be seen in the right rectangle of Fig. 1,amixed methods
research synthesis can investigate data coming from qualitative, quantitative, and mixed pri-mary level studies ( Harden and Thomas 2005 ,2010 ;Sandelowski et al. 2006 ;V oils et al.
2008 ).
123

662 M. Heyvaert et al.
2 Defining mixed methods research at the synthesis level
Amixed methods research synthesis is a systematic review applying the principles of mixed
methods research. As discussed by Creswell and Tashakkori (2007a ), the latter implies that
the study is not only expected to have two well-developed distinct strands, one qualitativeand one quantitative, each complete with its own questions, data, analysis, and inferences,it must also integrate, link, or connect these strands in some way (see Bryman 2007 ). It
is asystematic review, which means that it reviews available research data that has been
systematically searched for, studied, assessed, and summarized according to predetermined,transparent, and rigorous criteria. In an MMRS the data that are integrated in the review arefindings extracted from qualitative, quantitative, and mixed primary level articles. So, wherein a primary level study the participants are people, in a synthesis level study the partici-pants are primary level studies. Following the general definition of mixed methods researchproposed by Johnson et al. (2007 ), we define an MMRS as a synthesis in which researchers
combine qualitative, quantitative, and mixed methods studies, and apply a mixed methodsapproach in order to integrate those studies, for the broad purposes of breadth and depth ofunderstanding and corroboration.
Other suggestions concerning terminology for this synthesizing of qualitative and quan-
titative primary level studies are ‘mixed research synthesis’ ( Sandelowski et al. 2006 ;V oils
et al. 2008 ), ‘mixed studies review’ ( Pluye et al. 2009 ), and ‘mixed methods synthesis’
(Harden and Thomas 2005 ).
Supported by the work of other authors ( Harden and Thomas 2005 ,2010 ;Hart et al. 2009 ;
Sandelowski et al. 2006 ), we believe that the integration of qualitative and quantitative stud-
ies at the synthesis level has promising utility for research and practice, since the rationalefor conducting mixed methods synthesis research lies in combining the strengths of qualita-tive and quantitative techniques and studies, which are jointly available in many domains ofresearch ( Pluye et al. 2009 ;Sandelowski et al. 2006 ).
3 A framework for MMRS
3.1 A classification framework for MMRS
Our framework for MMRS was developed through a stepwise process. First, we studied exist-
ing classifications for mixed methods designs at the primary level . We found that the designs
described by Creswell (2003 ),Creswell and Plano Clark (2007 ),Greene et al. (1989 ),Johnson
and Onwuegbuzie (2004 ),Leech and Onwuegbuzie (2009 ),Mertens (2005 ),Morgan (1998 ),
Morse (1991 ), and Tashakkori and Teddlie (1998 )(rows in Table 1) are often referred to by
other mixed methods authors, and are often mentioned in mixed methods studies (for exam-ple, see Andrew and Halcomb 2006 ;Bazeley 2004 ;Bryman 2006 ;
Collins and O’Cathain
2009 ;Doyle et al. 2009 ).
Second, we explored the dimensions on which their frameworks are based. An analysis of
these influential typologies showed that mixed methods designs are usually classified accord-ing to some of the following five dimensions: emphasis of approaches, temporal orientation,integration, purpose of the study, and theoretical framework of the study ( columns in Table 1).
The cells of Table 1indicate which design types the above named researchers ( row heads )
differentiate within these five dimensions ( column heads ).
The third step included questioning which dimensions that had been identified in mixed
methods studies at the primary level are relevant for distinguishing mixed methods designs
123

Mixed methods research synthesis 663
Table 1 Classification frameworks for mixed methods designs at the primary level
Emphasis of approaches Temporal orientation Integration Purpose of the study Theoretical framework
of the study
Creswell (2003 ) QUAL dominant Concurrent Stage of integration Triangulation Transformative
QUAN dominant SequentialEmbedded design
Creswell and Plano Clark (2007 ) Embedded design Triangulation
ExplanatoryExploratory
Greene et al. (1989 ) Triangulation
ComplementaryDevelopmentInitiationExpansion
Johnson and Onwuegbuzie (2004 ) Equal status Concurrent
Dominant status Sequential
Leech and Onwuegbuzie (2009 ) Equal status Concurrent Partially mixed
Dominant status Sequential Fully mixed
Mertens (2005 ) Parallel Pragmatic
Sequential Transformative
Morgan (1998 ) QUAL dominant Complementary method
QUAN dominant preliminary or
follow-up
Morse (1991 ) QUAL dominant Simultaneous
QUAN dominant Sequential
Tashakkori and Teddlie (1998 ) Equal status Parallel Multilevel use
Dominant status Sequential of approaches
123

664 M. Heyvaert et al.
at the synthesis level . Because it is plausible that not all the dimensions identified at the
primary level are equally relevant at the synthesis level, we carefully compared severalprimary level and synthesis level mixed methods studies and their designs. The developmentof our classification framework was guided by three questions: (1) Which design-distinguish-ing dimensions found at the primary level are particularly relevant at the synthesis level?(2) Which design-distinguishing dimensions found at the primary level are less relevant atthe synthesis level? (3) Which differences exist between mixed methods studies at the primaryand synthesis level?
(1) Which design-distinguishing dimensions found at the primary level are particularly
relevant at the synthesis level?
‘Emphasis of approaches’ and ‘Temporal orientation’ are dimensions that are often applied
to distinguish mixed methods designs at the primary level (see Table 1). We believe that both
dimensions are key to distinguish between mixed methods designs at the synthesis level aswell. We add a third dimension that especially distinguishes our framework for designingmixed methods studies at the synthesis level from frameworks concerning the primary-studylevel: the ‘Integration’. We will briefly describe what these dimensions stand for.
First, the dimension ‘Emphasis of approaches’ indicates whether a qualitative or quantita-
tive approach has the priority with regard to the study’s purpose and its research questions, thedata collection and analysis, and the interpretation of the findings, or whether both approacheshave an approximate equal weight and influence.
Second, the dimension ‘Temporal orientation’ indicates whether qualitative and quantita-
tive research phases or sub-phases occur simultaneously or sequentially. In a simultaneousdesign the qualitative and quantitative data are collected concurrently, and parallel analyzedin a complementary manner. When both methods are implemented simultaneously and inter-actively within a single study, the interpretability of the results can be enhanced ( Greene et al.
1989 ). On the contrary, in a sequential design the quantitative and qualitative research phases
are conducted separately. The results of the method first implemented can help to identifyand refine the review question and/or the relevant outcomes of interest, to select the data, todevelop a theory or hypothesis, or to inform the analysis of the other method ( Dixon-Woods
et al. 2001 ;Greene et al. 1989 ). The two-phase design is the most elementary sequential
design. However, the sequencing quantitative and qualitative phases can go through severalcycles within a single study as well.
A third dimension that appears to be of major importance when combining qualitative,
quantitative, and mixed primary level studies in an MMRS, is ‘Integration’. The retrievedqualitative and quantitative data might be integrated at several stages in the research process:at the data collection, the data analysis, the interpretation phase, or a combination of phases(Creswell 2003 ).Morse and Niehaus (2009 ) define the position in which the qualitative and
quantitative components meet during the conduct of the research as the point of interface .
The dimension ‘Integration’ particularly distinguishes this framework for designing mixedmethods studies at the synthesis level from frameworks concerning the primary study level.As will be further explained in (3), a researcher engaging in an MMRS is limited in his choicesto design the synthesis due to the number of primary level studies including qualitative, quan-titative, or mixed data that are available on a certain research topic. At the synthesis level,the dimension ‘Integration’ indicates the difference between a synthesis that involves allthe
retrieved qualitative and quantitative articles in all the analyses and stages, and a synthesis that
analyzes ( parts of ) the retrieved qualitative and quantitative articles separately . The former
approach can for example be applied to examine the degree of accordance between qualitativeand quantitative elements found in the same group of studies, in order to corroborate results
123

Mixed methods research synthesis 665
or identify discrepancies within this group of studies. For example, a certain treatment for
challenging behavior in persons with ID may result in high effect scores (quantitative data),while the family of these persons report several negative side effects of this treatment (quali-tative data), or experience a contextual misfit of the treatment. Here, mixed methods could beused to identify discrepancies within a group of treatment studies by comparing qualitativeand quantitative ‘experience’ and ‘effect’ data concerning the applied treatments. The latterapproach for example applies when the synthesis is conducted separately on the qualitativeand quantitative studies before being combined in the results and conclusion section. Thisapproach can be used to expand or explain findings based on several primary level articlesby another set of articles (e.g., a set of deviant case articles in confrontation with a set ofcontextual background articles and a set of national surveys). In addition, it is also possiblethat primary level mixed articles are included in the review, each containing qualitative and
quantitative data. In that case, the qualitative and quantitative data reported in those articlescan be included in the qualitative and quantitative analysis respectively (cf. ‘ parts of ’).
(2) Which design-distinguishing dimensions found at the primary level are less relevant at
the synthesis level?
The dimensions ‘Purpose of the study’ and ‘Theoretical framework’ do not warrant a separate
place in our framework. Although it is at the utmost importance for a researcher to reflecton the objective of the study at hand, the dimension ‘Purpose of the study’ is not includedin our framework since differences between ‘triangulation’, ‘explanatory’ and ‘exploratory’designs (see Table 1) can be translated into differences between equal versus dominant sta-
tus of, and simultaneous versus sequential use of qualitative and quantitative approaches(Creswell 2003 ;Creswell and Plano Clark 2007 ;Greene et al. 1989 ), two dimensions that
are already incorporated in our framework. On the other hand, when considering the func-tions of a mixed methods study described by Greene et al. (1989 ), the decision whether to
select a complementary-, development-, initiation- or expansion-function for your MMRSdepends on the number of retrieved primary level studies including qualitative, quantitative,or mixed data (see: (3)), and can be related to the dimension ‘Integration’.
The dimension ‘Theoretical framework’ often distinguishes between ‘pragmatic’ and
‘transformative-emancipatory’ mixed methods designs ( Creswell 2003 ;Greene and Caracelli
1997 ;Mertens 2005 ). However, the majority of mixed methods studies have pragmatism as the
paradigmatic basis for methodologically combining qualitative and quantitative approaches(Feilzer 2010 ;Johnson and Onwuegbuzie 2004 ;Morgan 2007 ;Tashakkori and Creswell
2007 ). When an individual researcher applies an alternative theoretical framework (e.g.,
transformative-emancipatory perspective, dialectical worldview, communities of practice) itis desirable to make this explicit ( Denscombe 2008 ;Mertens 2007 ,2010 ;Plano Clark et al.
2008 ).
(3) Which differences exist between mixed methods studies at the primary- and synthesis
level?
We found one major difference between published mixed methods studies at the synthesis-
and primary level: the ‘range of choices’ of the researcher to design the study. A researcherconducting a primary level mixed study chooses which qualitative and quantitative data hecollects in order to answer the research question(s), and with which qualitative and quanti-tative methods he analyzes those data. So, he decides whether qualitative or quantitative (orboth) data and methods are dominant in the study.
However, a person conducting an MMRS can only work with the available primary level
studies on the research topic, containing a certain amount of reported qualitative and quan-titative data material. Although he can for example decide which inclusion and exclusion
123

666 M. Heyvaert et al.
criteria are used to select the primary level studies, and with which qualitative and quanti-
tative methods he analyzes those data, he still is strongly limited by the kind and amount ofqualitative and quantitative information reported in the available primary level studies in thechosen research domain. As a consequence, at the synthesis level the choice for an MMRSdesign not only depends on the posed research question, but also on the available qualitativeand quantitative information that is reported in the primary level articles.
This ‘range of choices’ for a researcher to design a mixed methods synthesis study is not
included as a fourth dimension in our classification framework, since it applies to allMMRS,
and only presents a difference between published mixed methods studies at the synthesis-versus primary level.
Summarizing, from the five dimensions that are often applied to distinguish mixed meth-
ods designs at the primary level (see Table 1), especially the dimensions ‘Emphasis of
approaches’, ‘Temporal orientation’, and ‘Integration’ are relevant at the synthesis level. Amatrix derived by combining these three dimensions (presented as column heads in Table 2)
yields 18 types of MMRS (indicated in bold and italics in Table 2). Guided by his research
question, and by the amount of available qualitative and quantitative data that are reportedin primary level studies, a researcher may choose one of these 18 designs.
Table 2 A classification framework for MMRS
Emphasis of
approachesIntegration (a) Temporal orientation
Concurrent Qual and
Quan approachSequential Qual and Quan
approach
Equal status of qualitative
and quantitative
approachesAll Qual and Quan data
involved in all research
stagesA−QUAL +QUAN (1)A−QUAL →QUAN (3)
A−QUAN →QUAL (4)
(Parts of) Qual and Quan
data involved separatelyin some/all researchstagesS−QUAL +QUAN (2)S−QUAL →QUAN (5)
S−QUAN →QUAL (6)
Dominant status of
qualitative orquantitative approachesAll Qual and Quan data
involved in all researchstagesA−QUAL +quan (7) A−QUAL →quan (11)
A−quan→QUAL (12)
A−QUAN +qual (8) A−QUAN →qual (13)
A
−qual→QUAN (14)
(Parts of) Qual and Quan
data involved separatelyin some/all researchstagesS−QUAL +quan (9) S−QUAL →quan (15)
S−quan→QUAL (16)
S−QUAN +qual (10) S−QUAN →qual (17)
S−qual→QUAN (18)
Notations: The plus sign (+) indicates that the qualitative and quantitative approaches are conducted simulta-
neously; the arrow ( →) indicates that the qualitative and quantitative approaches are conducted sequentially;
the uppercase indicates the dominant method of synthesis; the lowercase indicates the not-dominant method ofsynthesis; the A-sign indicates that all the qualitative and quantitative data are involved in all research stages;the S-sign indicates that (parts of) the qualitative and quantitative data are involved separately in some or allresearch stagesNote : (a) As pictured in Fig. 1, the qualitative and quantitative data that are incorporated in an MMRS can
come from qualitative, quantitative, and mixed primary level articles
123

Mixed methods research synthesis 667
3.2 Implementing MMRS
Frameworks for systematically synthesizing research evidence incorporate the following
stages: (1) the problem identification and question formulation stage, (2) the developmentof a review protocol and the literature search, (3) the selection of an appropriate designand method, (4) the data extraction and evaluation stage, (5) the data analysis and inter-pretation stage, and (6) the reporting and discussing of research findings ( Cooper 1998 ;
Cooper and Hedges 1994 ;Furlan et al. 2009 ;Gelo et al. 2008 ;Khan et al. 2001 ;Major and
Savin-Baden 2010 ;Onwuegbuzie and Leech 2005 ;Oxman and Guyatt 1988 ;Whittemore
and Knafl 2005 ). We will present a general elaboration of each of these stages involved in
doing an MMRS by giving two hypothetical illustrations of the classification frameworkin the domain of effectiveness studies concerning interventions for challenging behavior inpersons with ID.
4 Illustrations of the classification framework
4.1 Illustration of MMRS about the effect of interventions for challenging behavior in
people with intellectual disabilities
Challenging behaviors are culturally abnormal behavior of such an intensity, frequency or
duration that the physical safety of the person or others is likely to be placed in seriousjeopardy, or behavior which is likely to seriously limit use of, or results in the person beingdenied access to, ordinary community facilities ( Emerson 1995 ). Since challenging behav-
iors are highly prevalent among persons with ID and generate negative consequences for theindividual and his/her family, divergent biological, psychological, behavioral, and contextualinterventions are developed to reduce these behaviors ( Antonacci et al. 2008 ;Bouras 1999 ;
Didden et al. 1997 ;Dösen and Day 2001 ;Grey and Hastings 2005 ;Matson and Neal 2009 ;
McGillivray and McCabe 2006 ).
There exist several quantitative and qualitative reviews on the effects of different interven-
tions for challenging behavior in people with ID ( Balogh et al. 2008 ;Brylewski and Duggan
1999 ;Chan et al. 2010 ;Deb et al. 2007 ,2008 ;Didden et al. 1997 ,2006 ;Gustafsson et al.
2009 ;Harvey et al. 2009 ;Heyvaert et al. 2010 ;Lang et al. 2010 ;Shogren et al. 2004 ;Sohanpal
et al. 2007 ), while MMRS are not available for the time being. We believe that MMRS could
contribute to the development of this research domain by the integration of and confrontationbetween the many empirical qualitative and quantitative studies that are available. Thereby,MMRS could answer a broader and more complete range of research questions, and addinsights and understanding that might be missed when only a single method is used ( Johnson
and Onwuegbuzie 2004 ).
In our classification framework, 18 designs have been distinguished. We will illustrate
the distinction between these designs by describing two ‘extreme’ cells, namely cell 1(A−QUAL + QUAN) and cell 18 (S −qual→QUAN), pictured as the left above and
right bottom design in Table 2. As such, we present two diverging hypothetical applications
of MMRS. Elaborations of the other cells of Table 2can be deduced from these two exam-
ples. For both cells, we will hypothetically work out a synthesis on the effects of differentinterventions for challenging behavior in persons with ID, through the six above-describedresearch stages.
123

668 M. Heyvaert et al.
4.2 Hypothetical application 1
First of all, we should clearly identify the problem that the review is addressing, and the review
purpose ( Maxwell 2005 ;Onwuegbuzie and Leech 2005 ;Whittemore and Knafl 2005 ). Let’s
suppose that we work for a governmental service that studies treatments for challengingbehavior in persons with ID in order to give advice to centers and institutions working withthese persons. Our problem could be that we have, based on reading on this topic, the pre-sumption that although some high effect scores (quantitative data) favorable to a certaintreatment for challenging behavior in persons with ID (called treatment X) are published,this treatment should not be implemented on a large scale in his current form, since weheard several negative reactions to treatment X from staff claiming that this treatment onlyproduces good outcomes under very specific conditions. Our aim could be to systematicallyreview all the available primary level studies on treatment X for this target group, in orderto thoroughly answer the question ‘ what is it about this intervention that works (and does
not work), for whom ,i nw h a t circumstances ,i nw h a t respects ,a n d why?’ (see also Pawson
et al. 2005 ). Since we are not interested in the quantitative effect of treatment X an sich ,b u t
especially in its relation to several ‘black box’ intervention characteristics and participants
features, as well as in features of the context wherein each of the published interventions isembedded, we intend to perform an MMRS.
The second stage involves the development of a review protocol and the literature search.
In the review protocol, we describe and justify our search strategy, the selection of inclu-sion and exclusion criteria, the quality assessment, the data extraction strategy, and theway in which we synthesize the extracted findings ( Furlan et al. 2009 ;Khan et al. 2001 ;
Major and Savin-Baden 2010 ;Oxman and Guyatt 1988 ;Whittemore and Knafl 2005 ).
We search electronic databases, screen reference lists of retrieved articles, hand search,look for grey literature and conference proceedings, search research registers, and contactindividual researchers ( Cooper 1998 ;Cooper and Hedges 1994 ;Furlan et al. 2009 ;Khan
et al. 2001 ;Oxman and Guyatt 1988 ). So, we systematically retrieve primary level stud-
ies, and document this search in detail. If we would choose to perform a meta-analysis forthe quantitative part of the MMRS, all the selected articles should contain data that makethe application of meta-analytic techniques possible. Because we want to study this sta-tistical effect in its relation to several context, participants, and intervention features, thearticles that will be included in our review have to contain descriptions of these features aswell.
Third, we should select an appropriate research design, and provide a rationale for its
implementation. We would select an ‘A −QUAL + QUAN’ research design, since we want
to study all(‘A’) the included articles (containing quantitative effect data on treatment X
andqualitative data on context, participants, and intervention features) by equally applying
qualitative and quantitative research techniques (‘QUAL QUAN’) that are ‘ in an interactive
dialogue ’ (‘+’) with one another in order to understand and explain the differential effects
for treatment X in persons with ID.
Fourth, as a rule of thumb at least two reviewers should independently extract the arti-
cles, documenting their search in detail by using data extraction forms ( Furlan et al. 2009 ;
Khan et al. 2001 ). In addition, these articles should be evaluated with a quality assessment
instrument (e.g., the instrument presented in Pluye et al. (2009 )).
Fifth, the data analysis stage would involve concurrent qualitative and quantitative analyses
that are in dialogue with one another. We would record context, participants, and interven-tion features, and intervention effects for each included article (see Heyvaert et al. 2010 ).
Since we are interested in the relation between the quantitative effect data on treatment X
123

Mixed methods research synthesis 669
and several contextual, ‘black box’ intervention, and participants features, we would step-
wise navigate between the qualitative and quantitative information that is available in thepublished studies, and stepwise analyze this information by qualitative and quantitative tech-niques. We could start with generating comparable effect size measures for all the includedprimary level studies and statistically testing the general hypothesis that treatment X pro-duces positive quantitative outcome effects for challenging behavior in persons with ID bymeans of a random-effects meta-analysis of all the included studies ( Borenstein et al. 2009 ;
Cooper and Hedges 1994 ; see for example Heyvaert et al. (2010 ) for a detailed descrip-
tion of a random-effects meta-analysis on this topic). Concurrently, we could systematicallycollect all the available qualitative information on intervention, context, and participantsfeatures, and meanwhile systematically identify possible relations between the interventioneffects on the one hand, and ‘black box’ intervention, context, and participants features onthe other hand. The identified possible relations could be tested by statistical analysis andbe afterwards looped back to the qualitative systematic analysis. The qualitative systematicanalysis could adjust the former hypotheses or generate new hypotheses based on the resultsof this statistical analysis. These adjusted or new hypotheses could again be tested by sta-tistical analysis and be looped back to the qualitative systematic analysis, and so on. So, thestatistical meta-analytic and qualitative descriptive and analytical results could be stepwiseintegrated by identifying matches, mismatches, and gaps (see for an example Harden et al.
2004 ;Oliver et al. 2005 ). In the end, the qualitative and quantitative data should be integrated
in answering the question what is it about the reviewed intervention that works (and doesnot work), for whom, in what circumstances, in what respects, and why. By additionallyimplementing sequential meta-analytic analyses, the sufficiency of the retrieved cumulativeknowledge could be determined and the question could be answered whether there yet existsenough cumulative knowledge on treatment X implemented under certain conditions to yieldconclusive statistical evidence, or whether additional research on this treatment is needed(Kuppens and Onghena 2010 ).
Finally, we should describe our methods and results, and the implications for practice,
research, and policy in a research report, and communicate our research conclusions to centersand institutions working with persons with ID.
4.3 Hypothetical application 2
Another research problem could be that we intend to perform a meta-analysis of articles
on different interventions for challenging behavior among persons with ID, and that we areespecially interested in variables moderating the intervention effects, but that we do not knowwhich possible influencing variables (moderators) should be included in our analysis, andhow these variables are related to one another. Our proposed review purpose would then beto determine which interventions for challenging behavior among persons with ID producewhich effects, moderated by which variables.
Second, the development of a review protocol, the selection of the studies, and the litera-
ture search would be identical to the procedure described in Application 1. We could includedifferent (and possibly more) studies in our preceding qualitative analysis than in the domi-nant statistical meta-analysis, in order to generate a thematic network between the variablesinvolved in the intervention process that is as comprehensive as possible.
Third, we could select an ‘S −qual→QUAN’ research design, because we want to
study this topic by implementing a meta-analysis (‘QUAN’), but first need to explore whichvariables should be included in our analysis and how these variables are related, for example
123

670 M. Heyvaert et al.
through a qualitative thematic analysis sub-study (‘qual →’) that is based on possibly dif-
ferent primary articles than the articles included in the main meta-analysis (‘S’).
Fourth, parallel to Application 1, at least two reviewers should independently extract
primary level articles, and assess the quality of these articles.
Fifth, we could start by conducting a thematic analysis (‘ qual→’) in order to structure
and depict all the variables involved in the intervention process. We could prefer thematicanalysis because our research questions ask for a flexible analysis technique that clearlysummarizes key features of a large body of data and still offers a ‘thick description’ of thedata set ( Attride-Stirling 2001 ;Braun and Clarke 2006 ;Dixon-Woods et al. 2005 ;Harden
and Thomas 2005 ;Walsh and Downe 2005 ). The use of visual representations (thematic net-
works; Attride-Stirling 2001 ) could assist us in organizing relationships between variables,
and between different levels and groups of variables (main variables and sub-variables). Wewould analyze several primary articles on this topic until a saturation point is reached. Theresult of this preliminary analysis would consist of a thematic network incorporating allvariables connected with this intervention process. Afterwards, we could perform a random-effects meta-analysis (‘ →QU AN ’) on the effects of various interventions for challenging
behavior in people with intellectual disability (see H e y v a e r te ta l . (2010 ) for a more detailed
description of all the suggested meta-analytic procedures on this topic). Effect sizes and vari-ances should be computed for all included studies’ intervention effects. We could generatea summary effect with a 95% confidence interval, and measures of heterogeneity (Q-value,Tau-squared, I-squared). Next to that, we could assess the impact of the possible moderatingvariables (detected by the preceding thematic analysis) through subgroup and meta-regressionanalysis. Furthermore, a sensitivity analysis could be performed, each time removing onestudy, in order to show each study’s impact on the combined effect. In addition, we couldanalyze the possible impact of publication bias by a funnel plot-, a fail-safe N-, and Duval’sand Tweedie’s trim and fill-analysis. So, we could obtain answers to the question which inter-ventions for challenging behavior among persons with ID generate which effects, moderatedby which variables.
In our last research step, we should write the research report, containing a detailed descrip-
tion of our methods and results, and the implications for practice, research, and policy.
5 Future issues for MMRS
This paper intended to introduce the fascinating and promising, but still relatively new,
domain of MMRS. In addition, we wanted to stimulate a thoughtful designing of MMRS bydiscussing various dimensions that shape this type of research. That is why we presented aframework to carry out MMRS. By doing this, our manuscript adds to the mixed methodsliterature through filling a salient gap in the methodology of mixed methods research. Afterall, although several typologies for mixed methods designs at the primary level have been
proposed in order to inform and guide the practice of mixed methods inquiry, there existedno such typology framework for the synthesis level . As we argued that the ‘range of choices’
of the researcher to design the study constitutes a fundamental difference between mixedmethods studies designed at the primary- and synthesis level, simply referring to existingprimary level typologies for mixed methods designs would not suffice when designing anMMRS. Answering this void, we introduced a framework for designing such MMRS. Inaddition, the manuscript fills a salient gap in the methodology of research synthesis by intro-ducing the mixed methods perspective. The presented classification framework can help toinform researchers planning to carry out MMRS, and to provide ideas for conceptualizing
123

Mixed methods research synthesis 671
and developing those syntheses. Note however, that the framework is not intended as a rigid
or formulaic labeling for research.
Our framework was illustrated by two hypothetical MMRS applications on effectiveness
studies concerning interventions for challenging behavior in persons with ID. In this researchdomain several authors carry out qualitative syntheses, because there are not so many primarylevel articles that contain (enough) data to allow statistical meta-analysis. Here, an MMRScould be a good alternative: qualitative as well as quantitative primary level articles could beincorporated in the study. So, the researcher is not forced to exclude any available data fromthe synthesis. A more varied and possibly more nuanced palette of information on the topicat hand can be included in the analysis.
To conclude, we note several strengths of MMRS concerning the mixing of qualitative and
quantitative primary level findings andof qualitative and quantitative synthesis techniques.
First, the main advantage of the mixing of findings from qualitative and quantitative
primary level articles is that—compared to ‘unmixed’ syntheses—more complete, concrete,and nuanced answers can be given to complex research questions. For example, in the MMRSofThomas et al. (2004 ) the combining of ‘quantitative’ controlled-trial articles describing
the effects of interventions that promoted healthy eating with ‘qualitative’ studies that exam-ined the perspectives and understandings of children concerning barriers to and facilitatorsof fruit and vegetable intake, increases the policy relevance of the review since it can lead tothe development of more effective and appropriate interventions ( Harden and Thomas 2005 ).
As such, an MMRS can answer multiple aspects of the question ‘what is it about this kind ofintervention that works, for whom, in what circumstances, in what respects, and why?’ (seealso Pawson et al. 2005 ).
Second, the combination of qualitative and quantitative synthesis approaches holds the
possibility to help confirm or refute a theory to a greater degree than either one method can doon its own ( Risjord et al. 2002 ), or to uncover and profoundly explain discrepancies between
the findings of the included studies. For example, in the study of Thomas et al. (2004 )t h e
insights gained from the qualitative synthesis allowed an in-depth and nuanced explorationof the detected statistical heterogeneity.
Other advantages of mixed methods research are its attempts to fully respect the contribu-
tion and wisdom of both the qualitative and quantitative viewpoints, and to seek a workablemiddle solution for addressing divergent research problems ( Johnson et al. 2007 ;Leech and
Onwuegbuzie 2009 ;Niaz 2008 ;Tashakkori and Teddlie 2003b ). After all, when research-
ers collect multiple data using different strategies, approaches, and methods in such a waythat the resulting mixture or combination is likely to result in complementary strengths andcounterbalancing weaknesses, a mixed methods study has the potential to produce a morerobust understanding of a complex phenomenon, which is unavailable in a qualitative or aquantitative study undertaken in isolation ( Gelo et al. 2008 ;Greene et al. 1989 ;Johnson and
Onwuegbuzie 2004 ;Morgan 1998 ;O’Cathain et al. 2007 ;Onwuegbuzie and Johnson 2006 ;
Plano Clark et al. 2008 ;Robins et al. 2008 ).
However, there remain several challenges concerning the implementation of an MMRS.
First of all, although most researchers agree that the quality-quantity dichotomy and the‘incommensurability’-position is restricted, sterile, or even misleading ( Morgan 2007 ;
Newman and Benz 1998 ;Niglas 2006 ), various paradigmatic assumptions are still being
debated when conceptualizing, implementing, and interpreting mixed methods studies ( Greene
2008 ;Jang et al. 2008 ;Mertens 2010 ). Combining quantitative and qualitative studies and
methods with traditionally different viewpoints concerning ontology (single vs. multiplereality), epistemology (objectivism vs. subjectivism), and axiology (value bound vs. value
123

672 M. Heyvaert et al.
free) can turn out quite challenging ( Bryman 2007 ;Johnson et al. 2007 ;Onwuegbuzie and
Johnson 2006 ).
Second, there exist several methodological pitfalls generated by the diversity between and
within the mixed qualitative and quantitative methods ( Greene 2006 ). After all, MMRS not
only imply the integration of divergent qualitative and divergent quantitative studies withinseparate qualitative and quantitative strands of a synthesis, they most importantly involvethe integration of the conclusions from the qualitative and quantitative strands (for examplein the form of comparing, contrasting, building on, or embedding one type of conclusionwith the other) in order to provide a fuller understanding of the phenomenon under study(Creswell and Tashakkori 2007a ).
Third, answers to the questions whether it makes sense to perform an MMRS on a certain
topic, and which primary level studies can be combined within a single synthesis, depend onthe research domain and the topic at hand, the goal(s) of the synthesis, and the posed researchquestion(s). Remembering Eysenck (1978 ) comments on combining apples and oranges ,w e
have to annotate that a synthesis only gains credibility when the data in the included primaryarticles are comparable enough to be combined to answer a single research question. Althoughthis comment counts for MMRS as much as for meta-analyses and qualitative meta-syntheses,the former is in a more delicate position since it has to deal with combining more divergentprimary evidence. Concerning the question whether it makes sense to perform an MMRS,it is possible that a researcher intends to perform an MMRS on a certain topic (see Pawson
2008 p. 120: ‘method mix’ is the new methodological Holy Grail ), but that it turns out that
a mono-method approach is the only appropriate or feasible way. Ultimately, the researchquestion and the available evidence in the literature remain the key drivers for choosing amixed methods approach or not.
References
Andrew, S., Halcomb, E.J.: Mixed methods research is an effective method of enquiry for community health
research. Adv. Contemp. Commun. Family Health (2006). doi: 10.5172/conu.2006.23.2.145
Antonacci, D.J., Manuel, C., Davis, E.: Diagnosis and treatment of aggression in individuals with develop-
mental disabilities. Psychiatry Q. (2008). doi: 10.1007/s11126-008-9080-4
Attride-Stirling, J.: Thematic networks: an analytic tool for qualitative research. Qual. Res. (2001). doi: 10.
1177/146879410100100307
Balogh, R., Ouellette-Kuntz, H., Bourne, L., Lunsky, Y ., Colantonio, A.: Organising health care services for
persons with an intellectual disability. Cochrane Database Syst. Rev. (2008). doi: 10.1002/14651858.
CD007492
Bazeley, P. : Issues in mixing qualitative and quantitative approaches to research. In: Buber, R., Gadner, J.,
Richards, L. (eds.) Applying Qualitative Methods to Marketing Management Research, pp. 141–156.Palgrave Macmillan, UK (2004)
Borenstein, M., Hedges, L.V ., Higgins, J., Rothstein, H.: Introduction to Meta-Analysis. Wiley, Chichester
(2009)
Bouras, N.: Psychiatric and Behavioural Disorders in Developmental Disabilities and Mental Retarda-
tion. Cambridge University Press, Cambridge (1999)
Braun, V ., Clarke, V .: Using thematic analysis in psychology. Qual. Res. Psychol. (2006). doi: 10.1191/
1478088706qp063oa
Brylewski, J., Duggan, L.: Antipsychotic medication for challenging behaviour in people with intellectual
disability: a systematic review of randomized controlled trials. J. Intellect. Disabil. Res. (1999). doi: 10.
1046/j.1365-2788.1999.043005360.x
Bryman, A.: Integrating quantitative and qualitative research: how is it done? Qual. Res. (2006). doi: 10.1177/
1468794106058877
Bryman, A.: Barriers to integrating quantitative and qualitative research. J. Mix. Methods Res. (2007). doi: 10.
1177/2345678906290531
123

Mixed methods research synthesis 673
Chalmers, I., Hedges, L.V ., Cooper, H.: A brief history of research synthesis. Eval. Health Prof. (2002). doi: 10.
1177/0163278702025001003
Chan, S.W.C., Thompson, D.R., Chau, J.P.C., Tam, W.W.S., Chiu, I.W.S., Lo, S.H.S.: The effects of multi-
sensory therapy on behaviour of adult clients with developmental disabilities. A systematic review. Int.J. Nurs. Stud. (2010). doi: 10.1016/j.ijnurstu.2009.08.004
Collins, K., O’Cathain, A.: Ten points about mixed methods research to be considered by the novice
researcher. Int. J. Multiple Res. Approaches 3, 2–7 (2009)
Cooper, H.: Synthesizing Research: A Guide for Literature Reviews. 3rd edn. Sage, London (1998)
Cooper, H., Hedges, L.V .: The Handbook of Research Synthesis. Russell Sage Foundation, New York (1994)
Creswell, J.W.: Research Design: Qualitative, Quantitative, and Mixed Methods Approaches. 2nd edn. Sage,
Thousand Oaks (2003)
Creswell, J.W., Plano Clark, V .L.: Designing and Conducting Mixed Methods Research. Sage, Thousand
Oaks (2007)
Creswell, J.W., Tashakkori, A.: Developing publishable mixed methods manuscripts. J. Mix. Methods Res.
(2007a). doi: 10.1177/1558689806298644
Creswell, J.W., Tashakkori, A.: Differing perspectives on mixed methods research. J. Mix. Methods Res.
(2007b). doi: 10.1177/1558689807306132
Deb, S., Sohanpal, S.K., Soni, R., Lenotre, L., Unwin, G.: The effectiveness of antipsychotic medication in
the management of behaviour problems in adults with intellectual disabilities. J. Intellect. Disabil. Res.(2007). doi: 10.1111/j.1365-2788.2007.00950.x
Deb, S., Chaplin, R., Sohanpal, S.K., Unwin, G., Soni, R., Lenotre, L.: The effectiveness of mood stabiliz-
ers and antiepileptic medication for the management of behaviour problems in adults with intellectualdisability: a systematic review. J. Intellect. Disabil. Res. (2008). doi: 10.1111/j.1365-2788.2007.00965.x
Dellinger, A.B., Leech, N.L.: Toward a unified validation framework in mixed methods research. J. Mix.
Methods Res. (2007). doi: 10.1177/1558689807306147
Denscombe, M.: Communities of practice: a research paradigm for the mixed methods approach. J. Mix.
Methods Res. (2008). doi: 10.1177/1558689808316807
Didden, R., Duker, P.C., Korzilius, H.: Meta-analytic study on treatment effectiveness for problem behaviors
with individuals who have mental retardation. Am. J. Ment. Retard. 101, 387–399 (1997)
Didden, R., Korzilius, H., van Oorsouw, W., Sturmey, P.: Behavioral treatment of challenging behaviors in
individuals with mild mental retardation: meta-analysis of single-subject research. Am. J. Ment. Retard.(2006). doi: 10.1352/0895-8017(2006)111[290:BTOCBI]2.0.CO;2
Dixon-Woods, M., Fitzpatrick, R., Roberts, K.: Including qualitative research in systematic reviews: opportu-
nities and problems. J. Eval. Clin. Pract. (2001). doi: 10.1046/j.1365-2753.2001.00257.x
Dixon-Woods, M., Agarwal, S., Jones, D., Young, B., Sutton, A.: Synthesising qualitative and quantitative
evidence: a review of possible methods. J. Health Serv. Res. Policy 10, 45–53 (2005)
Dösen, A., Day, K.: Treating mental illness and behaviour disorders in children and adults with mental retar-
dation. American Psychiatric Publications, Washington, DC (2001)
Doyle, L., Brady, A.-M., Byrne, G.: An overview of mixed methods research. J. Res. Nurs. (2009). doi: 10.
1177/1744987108093962
Emerson, E.: Challenging Behaviour. Analysis and Intervention in People with Learning Difficulties.
Cambridge University Press, Cambridge (1995)
Eysenck, H.J.: An exercise in mega-silliness. Am. Psychol. (1978). doi: 10.1037/0003-066X.33.5.517.a
Feilzer, M.Y .: Doing mixed methods research pragmatically: implications for the rediscovery of pragmatism
as a research paradigm. J. Mix. Methods Res. (2010). doi: 10.1177/1558689809349691
Forbes, A., Griffiths, P.: Methodological strategies for the identification and synthesis of ‘evidence’ to support
decision-making in relation to complex health-care systems and practices. Nurs. Inq. (2002). doi: 10.
1046/j.1440-1800.2002.00146.x
Furlan, A.D., Pennick, V ., Bombardier, C., Van Tulder, M.: 2009 Updated method guidelines for systematic
reviews in the Cochrane Back Review Group. Spine (2009). doi: 10.1097/BRS.0b013e3181b1c99f
Gelo, O., Braakmann, D., Benetka, G.: Quantitative and qualitative research: beyond the debate. Integr. Psy-
chol. Behav. Sci. (2008). doi: 10.1007/s12124-008-9078-3
Greene, J.C.: Toward a methodology of mixed methods social inquiry. Res. Sch. 13, 93–98 (2006)
Greene, J.C.: Mixed Methods in Social Inquiry. Jossey-Bass, San Francisco (2007)Greene, J.C.: Is mixed methods social inquiry a distinctive methodology? J. Mix. Methods Res. (2008). doi: 10.
1177/1558689807309969
Greene, J.C., Caracelli, V .J.: Advances in Mixed-Method Evaluation: The Challenges and Benefits of Inte-
grating Diverse Paradigms. Jossey-Bass, San Francisco (1997)
Greene, J.C., Caracelli, V .J., Graham, W.F.: Toward a conceptual framework for mixed-method evaluation
designs. Educ. Eval. Policy Anal. (1989). doi: 10.3102/01623737011003255
123

674 M. Heyvaert et al.
Grey, I.M., Hastings, R.P.: Evidence-based practices in intellectual disability and behaviour disorders. Curr.
Opin. Psychiatry 18, 469–475 (2005)
Gustafsson, C., Ojehagen, A., Hansson, L., Sandlund, M., Nyström, M., Glad, J., et al.: Effects of psychosocial
interventions for people with intellectual disabilities and mental health problems. Res. Soc. Work Pract.(2009). doi: 10.1177/1049731508329403
Hampton, J.R.: Evidence-based medicine, opinion-based medicine, and real-world medicine. Perspect. Biol.
Med. (2002). doi: 10.1353/pbm.2002.0070
Hanson, B.: Wither qualitative/quantitative? Grounds for methodological convergence. Qual. Quant. (2008).
doi:10.1007/s11135-006-9041-7
Harden, A., Thomas, J.: Methodological issues in combining diverse study types in systematic reviews. Int. J.
Soc. Res. Methodol. (2005). doi: 10.1080/13645570500155078
Harden, A., Thomas, J.: Mixed methods and systematic reviews. In: Tashakkori, A., Teddlie, C. (eds.) Sage
Handbook of Mixed Methods in Social & Behavioral Research, pp. 749–774. Sage, Thousand Oaks (2010)
Harden, A., Garcia, J., Oliver, S., Rees, R., Shepherd, J., Brunton, G., et al.: Applying systematic review
methods to studies of people’s views: an example from public health. J. Epidemiol. Commun. Health(2004). doi: 10.1136/jech.2003.014829
Hart, L.C., Smith, S.Z., Swars, S.L., Smith, M.E.: An examination of research methods in mathematics edu-
cation (1995–2005). J. Mix. Methods Res. (2009). doi: 10.1177/1558689808325771
Harvey, S.T., Boer, D., Meyer, L.H., Evans, I.M.: Updating a meta-analysis of intervention research with
challenging behaviour: treatment validity and standards of practice. J. Intellect. Dev. Disabil. (2009).doi:10.1080/13668250802690922
Heyvaert, M., Maes, B., Onghena, P.: A meta-analysis of intervention effects on challenging behaviour among
persons with intellectual disabilities. J. Intellect. Disabil. Res. (2010). doi: 10.1111/j.1365-2788.2010.
01291.x
Jang, E.E., McDougall, D.E., Pollon, D., Herbert, M., Russell, P.: Integrative mixed methods data analytic
strategies in research on school success in challenging circumstances. J. Mix. Methods Res. (2008).doi:10.1177/1558689808315323
Jensen, L.A., Allen, M.N.: Meta-synthesis of qualitative findings. Qual. Health Res. (1996). doi: 10.1177/
104973239600600407
Johnson, R.B., Onwuegbuzie, A.J.: Mixed methods research: a research paradigm whose time has come. Educ.
Res. (2004). doi: 10.3102/0013189X033007014
Johnson, R.B., Onwuegbuzie, A.J., Turner, L.A.: Toward a definition of mixed methods research. J. Mix.
Methods Res. (2007). doi: 10.1177/1558689806298224
Khan, K.S., Riet, G., Popay, J., Nixon, J., Kleijnen, J.: Undertaking systematic reviews of research on effec-
tiveness. Centre for Reviews and Dissemination report no. 4, pp. 1–20 (2001)
Kuppens, S., Onghena, P.: Are there enough pieces to unravel the puzzle? A method to determine sufficiency
in single-case research synthesis. Paper presented at the Annual Meeting of the American EducationalResearch Association, Denver, CO (2010)
Lang, R., Koegel, L.K., Ashbaugh, K., Regester, A., Ence, W., Smith, W.: Physical exercise and individuals
with autism spectrum disorders: a systematic review. Res. Autism Spectr. Disord. (2010). doi: 10.1016/
j.rasd.2010.01.006
Leech, N.L., Onwuegbuzie, A.J.: A typology of mixed methods research designs. Qual. Quant. (2009). doi: 10.
1007/s11135-007-9105-3
Major, C.H., Savin-Baden, M.: An Introduction to Qualitative Research Synthesis: Managing the Information
Explosion in Social Science Research. 2nd edn. Routledge, London (2010)
Matson, J.L., Neal, D.: Psychotropic medication use for challenging behaviors in persons with intellectual
disabilities: an overview. Res. Dev. Disabil. (2009). doi: 10.1016/j.ridd.2008.08.007
Maxwell, J.: Qualitative Research Design: An Interactive Approach. 2nd edn. Sage, Thousand Oaks (2005)Mays, N., Roberts, E., Popay, J.: Synthesising research evidence. In: Fulop, N., Allen, P., Aileen, C., Black, B.
(eds.) Studying the Organisation and Delivery of Health Services: Research Methods, pp. 188–219.Routledge, London (2001)
Mays, N., Pope, C., Popay, J.: Systematically reviewing qualitative and quantitative evidence to inform
management and policy-making in the health field. J. Health Serv. Res. Policy (2005). doi: 10.1258/
1355819054308576
McGillivray, J.A., McCabe, M.P.: Emerging trends in the use of drugs to manage the challenging behaviour
of people with intellectual disability. J. Appl. Res. Intellect. Disabil. (2006). doi: 10.1111/j.1468-3148.
2005.00251.x
McKenna, H., Cutliffe, J., McKenna, P.: Evidence-based practice: demolishing some myths. Nurs. Stand.
14, 39–42 (1999)
123

Mixed methods research synthesis 675
Mertens, D.M.: Research and Evaluation in Education and Psychology: Integrating Diversity with Quantitative,
Qualitative, and Mixed Methods. 2nd edn. Sage, Thousand Oaks (2005)
Mertens, D.M.: Transformative paradigm: Mixed methods and social justice. J. Mix. Methods Res. (2007).
doi:10.1177/1558689807302811
Mertens, D.M.: Divergence and mixed methods. J. Mix. Methods Res. (2010). doi: 10.1177/
1558689809358406
Mitchell, G.J.: Evidence-based practice: critique and alternative view. Nurs. Sci. Q. 12, 30–35 (1999)
Morgan, D.L.: Practical strategies for combining qualitative and quantitative methods: applications for health
research. Qual. Health Res. (1998). doi: 10.1177/104973239800800307
Morgan, D.L.: Paradigms lost and pragmatism regained: methodological implications of combining qualitative
and quantitative methods. J. Mix. Methods Res. (2007). doi: 10.1177/2345678906292462
Morse, J.M.: Approaches to qualitative-quantitative methodological triangulation. Nurs. Res. 40, 120–123
(1991)
Morse, J.M., Niehaus, L.: Mixed Method Design: Principles and Procedures. Left Coast Press Inc., Walnut
Creek (2009)
Newman, I., Benz, C.R.: Qualitative-Quantitative Research Methodology: Exploring the Interactive Contin-
uum. Southern University of Illinois Press, Carbondale (1998)
Niaz, M.: A rationale for mixed methods (integrative) research programmes in education. J. Philos. Educ.
(2008). doi: 10.1111/j.1467-9752.2008.00625.x
Niglas, K.: Introducing the qualitative-quantitative continuum: an alternative view of teaching research meth-
ods courses. Paper and diagrams presented at the 2006 European Conference on Educational Research,Geneva, Switzerland (2006)
O’Cathain, A., Murphy, E., Nicholl, J.: Integration and publications as indicators of “yield” from mixed
methods studies. J. Mix. Methods Res. (2007). doi: 10.1177/1558689806299094
Oliver, S., Harden, A., Rees, R., Shepherd, J., Brunton, G., Garcia, J. et al.: An emerging framework for inte-
grating different types of evidence in systematic reviews for public health. Evaluation 11, 428–466 (2005)
Onwuegbuzie, A.J., Johnson, R.B.: The validity issue in mixed research. Res. Sch. 13, 48–63 (2006)
Onwuegbuzie, A.J., Leech, N.L.: Taking the “Q” out of research: teaching research methodology courses
without the divide between quantitative and qualitative paradigms. Qual. Quant. (2005). doi: 10.1007/
s11135-004-1670-0
Oxman, A.D., Guyatt, G.H.: Guidelines for reading literature reviews. Can. Med. Assoc. J. 138, 697–703 (1988)
Paterson, B.L., Thorne, B.L., Canam, C., Jillings, C.: Meta-Study of Qualitative Health Research. Sage,
Thousand Oaks (2001)
Pawson, R. : Method mix, technical hex, and theory fix. In: Bergman, M.M. (ed.) Advances in Mixed Methods
Research: Theories and Applications, pp. 120–137. Sage, Thousand Oaks (2008)
Pawson, R., Greenhalgh, T., Harvey, G., Walshe, K.: Realist review—a new method of systematic
review designed for complex policy interventions. J. Health Serv. Res. Policy (2005). doi: 10.1258/
1355819054308530
Plano Clark, V .L., Creswell, J.W., O’Neil Green, D., Shope, R.J. : Mixing quantitative and qualitative
approaches: an introduction to emergent mixed methods research. In: Hesse-Biber, S.N., Leavy, P.
(eds.) Handbook of Emergent Methods, pp. 363–387. Guilford Press, New York (2008)
Pluye, P., Gagnon, M.P., Griffiths, F., Johnson-Lafleur, J.: A scoring system for appraising mixed methods
research, and concomitantly appraising qualitative, quantitative, and mixed methods primary-level stud-ies in mixed studies reviews. Int. J. Nurs. Stud. (2009). doi: 10.1016/j.ijnurstu.2009.01.009
Rice, M.J.: Evidence-based practice in psychiatric and mental health nursing: qualitative meta-synthesis.
J. Am. Psychiatr. Nurses Assoc. (2008). doi: 10.1177/1078390308326661
Risjord, M.W., Dunbar, S.B., Moloney, M.F.: A new foundation for methodological triangulation. J. Nurs.
Scholarsh. (2002). doi: 10.1111/j.1547-5069.2002.00269.x
Robins, C.S., Ware, N.C., dosReis, S., Willging, C.E., Chung, J.Y ., Lewis-Fernandez, R.: Dialogues on mixed-
methods and mental health services research: anticipating challenges, building solutions. Psychiatr. Serv.(2008). doi: 10.1176/appi.ps.59.7.727
Sandelowski, M., Docherty, S., Emden, C.: Qualitative metasynthesis: issues and techniques. Res. Nurs. Health
(1997). doi: 10.1002/(SICI)1098-240X(199708)20:4<365::AID-NUR9>3.0.CO;2-E
Sandelowski, M., V oils, C.I., Barroso, J.: Defining and designing mixed research synthesis studies. Res.
Sch. 13, 29–40 (2006)
Sandelowski, M., V oils, C.I., Knaff, G.: On quantitizing. J. Mix. Methods Res. (2009). doi: 10.1177/
1558689809334210
Shogren, K.A., Faggella-Luby, M.N., Bae, S.J., Wehmeyer, M.L.: The effect of choice-making as an
intervention for problem behavior: a meta-analysis. J. Posit. Behav. Interv. (2004). doi: 10.1177/
10983007040060040401
123

676 M. Heyvaert et al.
Sohanpal, S.K., Deb, S., Thomas, C., Soni, R., Lenotre, L., Unwin, G.: The effectiveness of antidepressant
medication in the management of behaviour problems in adults with intellectual disabilities: a systematicreview. J. Intellect. Disabil. Res. (2007). doi: 10.1111/j.1365-2788.2006.00935.x
Suri, H., Clarke, D.: Advancements in research synthesis methods: from a methodologically inclusive per-
spective. Rev. Educ. Res. (2009). doi: 10.3102/0034654308326349
Tashakkori, A., Creswell, J.W.: The new era of mixed methods. J. Mix. Methods Res. (2007). doi: 10.1177/
2345678906293042
Tashakkori, A., Teddlie, C.: Mixed Methodology: Combining Qualitative and Quantitative Approaches.
Sage, Thousand Oaks (1998)
Tashakkori, A., Teddlie, C.: The past and future of mixed methods research: from data triangulation to mixed
model designs. In: Tashakkori, A., Teddlie, C. (eds.) Handbook of Mixed Methods in Social and Behav-ioral Research, pp. 671–701. Sage, Thousand Oaks (2003a)
Tashakkori, A., Teddlie, C. (eds.): Handbook of Mixed Methods in Social and Behavioural Research. Sage,
Thousand Oaks (2003b)
Teddlie, C., Tashakkori, A.: A general typology of research designs featuring mixed methods. Res. Sch. 13, 12–
28 (2006)
Thomas, J., Harden, A., Oakley, A., Oliver, S., Sutcliffe, K., Rees, R., et al.: Integrating qualitative research
with trials in systematic reviews. Br. Med. J. (2004). doi: 10.1136/bmj.328.7446.1010
V oils, C.I., Sandelowski, M., Barroso, J., Hasselblad, V .: Making sense of qualitative and quantitative findings
in mixed research synthesis studies. Field Methods (2008). doi: 10.1177/1525822X07307463
Walsh, D., Downe, S.: Meta-synthesis method for qualitative research: a literature review. J. Adv. Nurs. (2005).
doi:10.1111/j.1365-2648.2005.03380.x
Whittemore, R., Knafl, K.: The integrative review: updated methodology. J. Adv. Nurs. (2005). doi: 10.1111/
j.1365-2648.2005.03621.x
Zimmer, L.: Qualitative meta-synthesis: a question of dialoguing with texts. J. Adv. Nurs. (2006). doi: 10.1111/
j.1365-2648.2006.03721.x
123

Reproduced withpermission ofthecopyright owner. Further reproduction prohibited without permission.