Upload
others
View
5
Download
0
Embed Size (px)
Citation preview
VU Research Portal
De professionele jeugdzorgwerker
de Swart, J.J.W.
2011
document versionPublisher's PDF, also known as Version of record
Link to publication in VU Research Portal
citation for published version (APA)de Swart, J. J. W. (2011). De professionele jeugdzorgwerker: Kenmerken van jeugdzorgwerkers in relatie totkwaliteit van de jeugdzorg.
General rightsCopyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright ownersand it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights.
• Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal ?
Take down policyIf you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediatelyand investigate your claim.
E-mail address:[email protected]
Download date: 22. Mar. 2021
1
HOOFDSTUK 4. HOOFDSTUK 4. HOOFDSTUK 4. HOOFDSTUK 4.
THE EFFECTIVENESS OF INSTITUTIONAL YOUTH CARE OVER THE PAST
THREE DECADES: A META-ANALYSIS1
1 De Swart, J. J. W., Van den Broek, H., Stams, G. J. J. M., Asscher, J. J., Van der Laan, P. H., Holsbrink - Engels, G. A, & Van der Helm, G. H. P. (2011). The effectiveness of institutional youth care. Manuscript submitted for publication.
2
AbstractAbstractAbstractAbstract
A meta-analysis was conducted of 27 controlled studies, including 17.038 youth, to examine the
effectiveness of institutional youth care over the past three decades. We compared institutional
Evidence-Based Treatment (EBT) with non-institutional EBT, institutional Care As Usual (CAU)
with non-institutional CAU, institutional CAU with non-institutional EBT, and institutional EBT with
institutional CAU. Only the last comparison yielded a significant and small-to-medium effect (d =
.36), which not only shows that institutional youth care can be equally effective as non-
institutional youth care, but also demonstrates that it seems more effective to provide youth
with EBT during their stay in the institution. Furthermore, moderator analyses indicated that only
cognitive behavior therapy showed a significant and medium effect (d = .50), whereas skills
training and care as usual showed no effect. Age, gender, type of outcome measure
(delinquency, skills, problem behavior, and overall problems) and study design characteristics
(randomized, matched or non-matched controls) did not influence the results.
keywords: keywords: keywords: keywords: institutional care, group homes, evidence-based treatment, incarcerated, meta-
analysis, effectiveness
IntroductionIntroductionIntroductionIntroduction
Although there are only a small percentage of children in institutional care, about three to eight
percent of the children who seek treatment according to Hair (2005), the interest in the
effectiveness of institutional youth care has grown substantially over the years (Lyons, Woltman,
Martinovich, & Hancock, 2009). Institutional youth care is carried out in 24-hour group living
facilities that are not licensed as hospitals and offer care or mental health treatment for
children and adolescents with serious behaviour problems, including law violations. The daily
living environment is often used as a therapeutic means (Bates, English, & Kouidou-Giles, 1997;
Hair, 2005). There is a wide variety of institutional services, with juveniles living in groups of
different sizes, different treatment methods (French & Cameron, 2002) and specialized services
for different types of problem behaviour (Barth, 2002).
The increased interest in the outcomes of institutional youth care can be explained by
two factors. First, scientific knowledge about effective interventions in youth care in general and
in juvenile justice increased considerably and promoted evidence-based working in
(institutional) youth care as well (Lyons, Woltman, Martinovich, & Hancock, 2009). Second,
institutional youth care is the most drastic and the most expensive type of youth care (Barth,
2002; Hoagwood, Burns, Kiser, Ringweisen & Schoenwald, 2001; Knorth, Harder, Zandberg &
Kendrick, 2008), which resulted in putting even more pressure on this sector to demonstrate
effective outcomes than in non-institutional youth care (Lyons et al., 2009).
3
Equivocal results of institutional care interventions (Scherrer, 1994), high costs and the
general scepticism that institutional care has a negative effect on children and society turned
institutional care into a treatment of ‘the last resorts’ (Bates et al., 1997; Frensch & Cameron,
2002; McLeigh & O’Neill Bridell, 2011). However, the question is whether institutional youth
care is indeed less effective when compared to non-institutional care. Anglin (2004) even
argued that for specific juveniles, institutional youth care can be more effective than non-
institutional care, as institutional youth care may be the preferred option for juveniles who need
a structured and (highly) supervised environment. Moreover, for those juveniles for whom there
is no alternative to non-institutional youth care, an important question is whether there is a way
to treat effectively within institutional care.
A number of meta-analyses examined the effectiveness of institutional youth care in
general (Garret, 1984; Knorth et al., 2008; Scherrer, 1994). These meta-analyses yielded
positive effect sizes, which indicated that institutional care can be an effective intervention.
Although all meta-analyses reached the same general conclusion that institutional youth care
can be effective, there was quite some variability in effect sizes due to differences in outcome
measures, sample characteristics (age, gender), and study design characteristics. In the study
of Garret (1984) the effect size for the outcome measure ‘recidivism’ was only small, but
positive, whereas the effect size for ‘academic improvement’ was rather large. Scherrer (1994)
did not find any significant differences between outcome measures. Knorth et al. (2008)
concluded that treatment was much more effective for externalizing and general problem
behavior than for internalizing problems. Garret (1984) showed that age of the juvenile was
inversely related to the effectiveness of treatment. The explanation given for this result is that
younger juvenile delinquents are more open to change and less rooted in their deviant behavior
(see also: Garrido & Morales, 2008). Garret (1984) demonstrated that females were more
positively affected by treatment than males, whereas Scherrer (1994) did not find a significant
difference between male or female programs, but concluded that programs which mixed males
and females together were more effective. Garret (1984) examined how different research
designs affected differences in effect sizes. Non-controlled studies with a pre- and post-test
yielded a large overall effect size, but dropped to a small effect size in controlled studies.
We considered it timely to conduct a new meta-analysis, 17 and 27 years after the
comprehensive meta-analyses that were performed by Garret (1084) and Scherrer (1994),
respectively. Knorth et al. (2008) conducted a (selective) meta-analysis of the outcomes of
institutional child and youth care, but only examined non-controlled studies that were published
between 1990 and 2005, without performing moderator analyses due to lack of (statistical)
information in the primary studies. Knorth et al. could therefore not explain the heterogeneity of
the overall mean effect sizes for internalizing, externalizing and total behaviour problems.
Knorth et al. discussed the controlled studies in a (selective) narrative review. They concluded
4
that institutional care shows the most promising short term outcomes when treatment is based
on behaviour-therapeutic methods or family involvement, whereas there is not much evidence
of positive long term outcomes.
The present meta-analysis examines the effectiveness of institutional youth care over
the past three decades, and only focuses on the quasi (experimental) studies that were
published in this period. Heterogeneity of effect sizes is examined by means of analyses of
factors (moderators) that may impact on the outcome of effectiveness studies, including sample
characteristics (age, gender, cultural background of the children), characteristics of youth care
workers (e.g., education, age, and sex), type of setting (secure versus non-secure), study design
characteristic (e.g., experimental versus quasi-experimental, time to follow up), study quality,
outcome (delinquency, behaviour problems, skills), judicial context of the intervention, type of
intervention (skills, cognitive behaviour therapy or care as usual), and type of comparison
(comparing treatment within institutional care or between institutional and non-institutional
care). This final moderator was neglected in previous meta-analyses of institutional youth care,
but seems of crucial importance for weighting the effectiveness of institutional youth care. It is
plausible to suggest that the degree to which youth care is effective depends on the particular
comparison that is made. In the present meta-analysis, institutional evidence-based treatment
(EBT), which is structured and manual-guided treatment that is based on empirical evidence
(Weisz, Jensen-Doss, & Hawley, 2006), is compared to institutional care as usual (CAU), that is,
regularly group care. We also compared institutional EBT to non-institutional EBT, institutional
CAU with non-institutional CAU, such as regular foster care, and institutional CAU with non-
institutional EBT, such as multi systemic treatment.
MethodMethodMethodMethod
Selection of studies
The present study included all published studies since 1980 that address the effectiveness of
institutional youth care. First, we searched in the following electronic data-bases: psychINFO,
MEDLINE, ERIC, Picarta, International Bibliografy of Social Sciences, Adlib, Sciencedirect,
Springerlink and Google Scholar, using the keywords in various combinations: child, youth
adolescent, inpatient, hospit* Institutional youth care, institutional care, comparison, group
care, group homes, incarcerated. Second, we inspected the reference sections of systematic
reviews and meta-analyses concerning institutional youth care in order to find more studies that
had not been included yet. This search yielded 118 studies of which 27 studies (N= 17.038
youth) eventually met the inclusion criteria of our meta-analysis.
5
Inclusion criteria
Criteria for inclusion of studies into the meta-analysis were as follows: (1) the experimental
group had to receive CAU or EBT in an institutional setting; (2) studies had to include control
groups (youth receiving CAU or regular group care in an institutional setting, or youth receiving
EBT or CAU in non-institutional settings); (3) studies had to be published in a peer reviewed
journal (4) the participants had to be enrolled in the study from 1980 on; (5) Studies had to
provide at least post test scores; (6) the average age of the youth in institutional care had to be
under age 21. We decided to exclude studies if institutional treatment was based on external
control and coercion, such as boot camps, instead of therapeutic interventions that engage
juveniles in a supportive and constructive process of change (Lipsey, 2009a). Wilson,
Mackenzie and Mitchell (2008) showed in a comprehensive meta-analysis that correctional boot
camps were not effective. The literature search resulted in 27 articles (see the Appendix A) that
met the inclusion criteria.
Coding the studies
Each study included in the meta-analysis was coded for publication characteristics, sample
characteristics, the continent where the study was conducted, study design characteristics,
study quality, judicial context of the intervention, type of setting, type of intervention, the
particular outcome measure, and type of comparison that was made. No studies could be coded
for length of stay, IQ, and characteristics of youth care workers due to lack of information.
Publication characteristics were publication year and impact factor of the journal.
Sample characteristics were gender (male versus mixed sample), mean age of the youngsters,
and the percentage of juveniles with an ethnic minority background. The continent where the
study was carried out was coded as North America or Western Europe. The study design was
coded as Randomized Controlled Trial (RCT), quasi-experimental design with a matched control
group, and quasi-experimental design with a non-matched control group (a study was
considered matched if the experimental and comparison group were comparable on at least
gender, age, and cultural background). The methodological quality of the studies was assessed
using a study quality index (QI) (Downs & Black, 1998), which evaluates the study quality (is
their sufficient information provided to enable the reader to make an unbiased judgement
concerning the results of the study); external validity; internal validity (bias and confounding)
and power, with a maximal QI score of 32. QI scores of > 20 were considered good, 11 to 20
moderate, and < 11 poor. All studies proved to be of moderate quality (M = 15,48, SD = 1,847).
We coded the nature of the particular comparison that was made in each study by
contrasting institutional EBT with non-institutional EBT, institutional CAU (i.e., regular group care)
with non-institutional CAU (i.e. foster care), institutional CAU with non-institutional EBT, and
institutional EBT with institutional CAU. In addition, we coded the type of setting (secure versus
6
non-secure) and whether or not institutional youth care was being delivered in a judicial-
penitentiary context. . . . The type of intervention was coded as skills, cognitive behaviour therapy
(CBT) and care as usual, and the outcome measure was coded as delinquency, behaviour
problems, skills and a mixed category of miscellaneous problems.
Calculation and analysis of effect sizes
Cohen’s d was calculated to index the effectiveness of institutional youth care on the basis of
differences between juveniles receiving institutional and non-institutional care and difference
between evidence-based treatment (EBT) and care as usual (CAU) in institutional care settings.
If available, follow-up data were used instead of post-test data. Follow-up time was included as
both a categorical moderator (post-test or follow-up assessment) and a continuous moderator
(number of months between end-of-treatment and follow-up assessment). To account for initial
group differences that may impact on post-test group differences, Cohens’s d was adjusted for
pre-test group differences, if such information was available, by subtracting the pre-test effect
size from post-test effect size. Subsequently, a moderator was created in order to examine
whether adjustment for initial group differences affected the combined overall effect size, and if
so, to which degree.
In most instances, Cohen’s d was calculated on the basis of mean scores and standard
deviations. In some cases, however, the calculation of Cohen’s d was based on reported test
statistics, such as F, p or t values, or on differences between percentages. The reported values
were transformed into Cohen’s d with an effect size computation program (Wilson, 2001 a, b).
After having calculated the effect sizes for each individual study, the overall combined
effect size was calculated and moderator analyses were conducted using SPSS macros (Hox,
2002; Wilson, 2001), based on the random effect model. In random effect models, significance
testing is based on the total amount of studies in the meta-analysis and results can be
generalized to the population of studies from which the current set of studies was drawn
(Rosenthal, 1995).
Homogeneity analysis was conducted at p<0.05 with SPPS-macros (Hox, 2002; Wilson,
2001) to establish whether the sample of the studies was homogeneous, that is, whether effect
sizes were constant across studies, or if the differences between effect sizes could have been
the result of another cause then subject-level sampling error. If so, differences in effect sizes
between studies may be explained by study characteristics (Lipsey & Wilson, 2001, pp. 115-
119). In case of heterogeneity, an analysis of variance (ANOVA) for the categorical variables can
be performed to examine which moderators significantly account for variability in effect sizes
among the studies included in the meta-analysis. To investigate the effects of continuous
moderator variables, regression analyses can be performed. Outlying effect sizes were identified
7
on the basis of z values larger than 3.3 or smaller than -3.3 (p < .005; Tabachnick & Fidell,
2001).
Publication bias
In this meta-analysis we only used published studies in order to have some under limit of study
quality. However, this choice may increase the risk that studies with non-significant findings,
which are less likely to be published, are not included in the meta-analysis. This phenomenon
has been designated as the ‘file drawer problem’ (Rosenthal, 1995). A way to find out if the
meta-analytic results may be affected by publication bias is to examine a funnel plot of the
distribution of effect sizes. Each individual study’s effect size is plotted on the horizontal axis
against its sample size, standard error or precision (the reciprocal of the standard error) on the
vertical axis. The distribution of effect sizes should be shaped as a funnel if no publication bias
is present, since the more numerous studies with small sample sizes are expected to show a
larger variation in the magnitude of effect sizes than the less numerous studies with large effect
sizes. A violation of funnel plot symmetry reflects publication bias, that is, a selective inclusion
of studies showing positive or negative outcomes (Sutton, Duval, Tweedie, Abrams, & Jones,
2000). Funnel plot asymmetry can be tested by regressing the standard normal deviate, defined
as the effect size divided by its standard error, against the estimate’s precision (the inverse of
the standard error), which largely depends on sample size (see Egger, Smith, Schneider, &
Minder, 1997). If there is asymmetry, the regression line does not run through the origin and the
intercept significantly deviates from zero.
ResultsResultsResultsResults
The Appendix A provides an overview of all 27 studies (N= 17,038 youth) included in this meta-
analysis, with the effect size of each individual study and the respective study characteristics. In
line with generally accepted principles for the interpretation of effect sizes, an effect size of d =
.20 was considered small, an effect size of d = .50 was considered medium, and an effect size
of .80 was considered large (Cohen, 1988).
The individual study effect sizes ranged from d = -.690 to d = 1.806. Possible
publication bias was examined by testing funnel plot asymmetry. The standard normal deviate
was regressed against the estimate’s precision. As the intercept did not significantly deviate
from zero (t = 1.405, p = .172), there was no indication of funnel plot asymmetry and therefore
no indication of publication bias.
Because the set of effect sizes proved to be heterogeneous, Q (26) = 1355.838, p <
0.001, a moderator analysis was conducted in order to explain the variation in effect sizes
among studies (Mullen, 1989). As can be derived from Table 1, a series of ANOVA’s yielded two
significant moderators: comparison – Q (3) = 10.7234, p < .05 – and type of intervention – Q
8
(2) = 17.211 p < .0001. A significant and positive small-to-medium effect size of d = 0.337 was
found for the comparison between evidence-based treatment and care as usual (general group
care) in institutional settings, whereas other comparisons yielded negative but non-significant
effect sizes. This was also true when the comparison moderator was dichotomized into a set of
studies contrasting institutional EBT with institutional CAU or general group care (d = 0.337, p <
.001) and a set of studies contrasting residential care with non-residential care (d = -0.199, ns.):
Q (1, 25) = 10.530, p < .01. Furthermore, only CBT yielded a significant effect size of d = 0.520,
whereas skills training and care as usual had no significant effect. In addition, we analyzed the
continuous moderators (see Table 2), which only yielded a significant effect size for publication
year (Beta = -.453, t = -2,723, p < 0.01).
Table 1. Categorical Moderator Analyses (ANOVA’S): Cohen’s d, 95% CI, Q Statistics
*p < .05. ** p < .01. *** p < .001
Moderator variables
N
K
d
95% confidence interval
Q statistic between studies
Q statistic within studies
Overall 17038 27 0.129 -0.02 to 0.259 1355.838*** Gender 2.554 Male 4334 10 0.329* 0.033 to 0.626 14.261 Mixed 12704 17 0.026 -0.199 to 0.251 14.048 Continent 1.617 North America 11985 21 0.200 -0.007 to 0.408 27.755 Western Europe 5053 6 -0.084 -0.470 to 0.302 0.425 Criminal justice intervention 2.116 Yes 4757 13 0.271* 0.011 to 0.533 18.903 No 11620 12 -0.006 -0.276 to 0.264 8.881 Mixed 661 2 0.124 -0.534 to 0.781 0.594 Incarcerated 3.288 Yes 4198 12 0.321 0.055 to 0.588 16.000 No 12840 15 -0.008 -0.244 to 0.228 12.453 Comparison 10.723* ebt-inst vs. ebt-noninst 65 1 -0.338 -1.164 to 0.488 0.000 cau-inst vs. ebt-noninst 1493 6 -0.196 -0.525 to 0.133 1.921 cau-inst vs. cau-noninst 10060 3 -0.161 -0.620 to 0.298 0.350 ebt-inst vs. cau-inst 5420 17 0.337*** 0.138 to 0.537 26.736* Design 0.147 Randomized 1469 9 0.099 -0.224 to 0.422 6.078 Matched 14368 9 0.186 -0.136 to 0.509 13.475 Non-matched 1201 9 0.126 -0.202 to 0.455 8.622 Adjusted for pre-test differences 0.011 no 13543 13 0.148 -0.123 to 0.418 18.451 yes 3495 14 0.128 -0.133 to 0.389 9.757 Follow-up 0.054 yes 16411 20 0.150 -0.067 to 0.368 25.240 no 627 7 0.100 -0.272 to 0.471 2.956 Outcome 0.130 Delinquency 12555 9 0.100 -0.226 to 0.419 16.984* Behavior 2829 6 0.177 -0.220 to 0.573 6.057 Skills 1003 4 0.180 -0.303 to 0.663 1.100 Mixed 651 8 0.133 -0.214 to 0.480 3.225 Type of intervention 17.211*** Skills 1158 8 0.017 -0.252 to 0.286 6.960 CBT 4327 10 0.520*** 0.281 to 0.759 19.382* CAU 11553 9 -0.184 -0.430 to 0.061 0.783
9
Table 2. Continuous moderator analyses: Standardized Regression Coefficients (Beta), t-Values, and Significances (p)
Moderator variables
N
K
Beta
t
p Time Follow Up 16411 20 -.036 0.167 .868 Publication year 17038 27 -.453 -2.723 .007 Impact factor journal 17038 27 .033 -0.176 .860 Study quality 17038 27 -.106 -0.564 .573 Ethnic minority 13795 23 -.148 -0.729 .466 Age juveniles 17008 26 .246 1.321 .187
All significant moderators were included in a multiple regression analysis in order to examine
their unique contribution to the overall combined effect size. Comparison and type of
intervention were transformed into a set of dummy variables in order to be able to include these
moderators as dichotomous variables in the multiple regression analysis. We found a significant
regression equation – Q (5, 21) = 36.34, p < .001 – with two significant moderators, namely
CBT versus skills and care as usual (b = .426, t = 3.253, p < .01) and the comparison that
contrasts EBT with CAU in institutional settings versus all other possible comparisons (b = .269,
t = 2.058, p < .05).
The results of the multiple regression analysis indicate that the effect of year of
publication on the overall combined effect size is an artifact explained by an association
between the comparison that is made and treatment type on the one hand and year the
publication on the other hand. The multiple regression results also indicate that applying
evidence-based treatment in institutional settings, in particular cognitive behavior therapy,
uniquely accounts for the effectiveness of institutional youth care.
DiscussionDiscussionDiscussionDiscussion
The first purpose of this study was to replicate previous meta-analyses of effectiveness of
institutional youth care and to test the hypotheses about the effectiveness of residential care,
but only including studies with a (quasi)experimental design in contrast to some of these earlier
reviews. A total of 27 published studies (N = 17,038 youth) were included in this meta-analysis.
As the overall combined effect size proved to be heterogeneous, we conducted moderator
analyses that yielded two significant moderators uniquely explaining the effectiveness of
institutional youth care: type of comparison and type of intervention.
We compared institutional Evidence-Based Treatment (EBT) with non-institutional EBT,
institutional Care As Usual (CAU) with non-institutional CAU, institutional CAU with non-
institutional EBT, and institutional EBT with institutional CAU. Only the last comparison,
institutional EBT versus institutional CAU, yielded a significant and small-to-medium effect (d =
.34), which not only shows that institutional care can be equally effective as non-institutional
care, but also demonstrates that it is more promising to provide youth with EBT during their stay
in the institution.
10
This conclusion supports some of the findings of previous meta-analyses, showing that
youngsters react positively to treatment in institutional care and in general improve in their
psychological wellbeing (Garret, 1984; Knorth et al., 2008; Scherrer, 1994). Our meta-analysis
also supports the conclusion of the meta-analysis of evidence-based youth psychotherapies
versus usual clinical care by Weisz et al. (2006) in which EBT proved to be more effective than
CAU (d = 0.30), independent of treatment modality. The results of the narrative review by Knorth
et al. (2008), suggest that non-institutional evidence-based treatment might perform better than
institutional group care (CAU). The present meta-analysis included six studies comparing non-
institutional EBT with institutional group care, yielding a small and negative but non-significant
effect size for institutional group care (CAU) when compared to non-institutional EBT. However,
more research is needed to examine if this effect-size is still non-significant over a larger
number of studies.
Moderator analyses indicated that only cognitive behavior therapy (CBT) had a medium
effect (d = .50), whereas skills training and care as usual had no effect. In other meta-analyses
and narrative reviews, CBT also proved to be an effective intervention (Armelius &Andreassen,
2006; Garrido & Morales, 2007; Weisz et al., 2006), even though Armelius and Andreassen
found a rather small effect for CBT in comparison with CAU and no significant difference
between CBT and alternative evidence-based treatment.
It might come as a surprise that social skills training did not have a significant effect.
However, there is an abundant body of research showing that the effect of social skills training
for juveniles, in general, tends to be small. Meta-analyses by Ang and Hughes (2002), Cook et
al. (2008), Durlak, Weissberg, and Pachan (2010), Losel and Beelmann (2003), Schneider
(1992) and Quinn, Kavale, Mathur, Rutherford and Forness (1999) report small to medium
effects. Moreover, these effects generally do not persist long after the intervention has ended.
As social skills training usually does not take place in a naturalistic setting, the effects of social
skills training have limited generalizability (Maag, 2005, 2006). If social skills training is
delivered in institutional settings, it may be the case that the social skills cannot be generalized
to other settings outside the institution.
Age, gender, study design characteristics (randomized, matched versus non-matched
controls) and type of outcome measure (delinquency, skills, problem behavior, and overall
problems) did not influence the effectiveness of institutional youth care in our meta-analysis.
The conclusion of Knorth et al. (2008) that institutional care would be especially effective for
teaching specific skills could therefore not be confirmed. More recent studies showed smaller
effect sizes. This effect, however, disappeared after controlling for type of treatment and the
EBT-CAU comparisons in the multiple regression analysis. Earlier studies more often inspected
the effects of CBT than recent studies did.
11
Unfortunately, we were not able to test a number of relevant moderators, such as IQ,
length of stay, and characteristics of youth care workers, because there was not sufficient
information available to code these moderators. It is particularly remarkable that even recent
studies do not report on intelligence of the sample under study or examine intelligence as a
moderator of treatment effects, as there is a vast body of research showing that treatment
should be responsive to the offender’s cognitive abilities and learning style (Andrews & Bonta,
2010; Koning & Collin, 2007; Van Nieuwenhuijzen, Orobio de Castro, Wijnroks, & Vermeer,
2009). Although length of stay seems important to consider from the perspective of possible
institutionalization effects, which dates back to the seminal work of Goffman (1961), it is
difficult to examine the unique effect of length of stay on treatment outcomes in residential
youth care, as it might be confounded with a host of client and treatment characteristics that
cannot be accounted for in a single study. It is therefore imperative that studies describe length
of stay of the sample under study in order that future meta-analyses of institutional youth care’s
effectiveness can adequately examine the impact of length of stay.
The studies used for this meta-analysis do not report on characteristics of youth care
workers. This is remarkable, because it is assumed that characteristics of group care workers,
such as professionalism, education and training and relationship skills constitute an important
moderator of positive treatment outcomes (e.g. Bickman et al., 2004; Duncan, Miller, Wampold,
& Hubble, 2009; Van der Helm, Boekee, Stams, & van der Laan, 2011).
There are some limitations of this meta-analysis that need to be mentioned. To preserve
some minimum level of study quality we only included published studies, which might increase
the risk of publication bias. Analyses of publication bias, however, by means of a funnel-plot of
the distribution of effect sizes across studies, proved that there was no indication of publication
bias. A second limitation was that, although we decided to use published studies only, we could
not prevent that all studies included in this meta-analysis were of moderate quality. On the other
hand, there were no published studies that should be considered as low quality studies. A third
limitation is that the overall effect size for CBT was based on a heterogeneous set of effect
sizes. But homogeneity might not be a realistic expectation, because of the diverse populations
that are found within institutional youth care.
In spite of these limitations, this meta-analysis provides us with new and better insights
in the effectiveness of institutional youth care than any of the previous meta-analyses. First, we
demonstrated that evidence-based treatment delivered in institutional youth care does not have
to be less effective than evidence-based treatment in non-institutional youth care. Notably, we
replicated the findings from the meta-analysis of non-institutional youth care by Weisz et al.
(2006), showing that evidence-based treatment performs better than regular group care (i.e.
care as usual) provided in institutional settings, which has often been designated as treatment
of ‘the last resorts’. Nevertheless, the present meta-analysis also showed that providing regular
12
group care to youngsters who tend to have serious behavioral and developmental problems
does not seem to have a positive effect and might even increase problems. Therefore, it
remains important to always first consider non-institutional natural interventions, including
foster care, as an alternative to regular group care, in particular in the case of young children.
However, if non-institutional care is not a realistic option – for example, when children need a
structured and (highly) supervised environment – it seems imperative to offer these juveniles
evidence-based treatment. In the end, the question is not whether institutional or non-
institutional care should be preferred, but how to optimize the effects of EBT for children who
receive institutional and non-institutional youth care.
The availability of aftercare can be considered important for the generalization of
treatment effects after institutional care. A recent meta-analysis showed that aftercare can have
a small and positive short-term effect (James, Stams, Ascher, De Roo, & Van der Laan, 2011).
Aftercare, however, is currently not common in institutional youth care, but it is plausible to
suggest that aftercare that builds on positive outcomes of institutional care may considerably
increase the short and long term effects of evidence-based treatment delivered in institutional
youth care.
13
LiteratureLiteratureLiteratureLiterature
* Abrams, L.S., Shannon, S.K., & Sangalang, C. (2007). Transition services for incarcerated
youth: a mixed methods evaluation study. Child and Youth Services Review, 30, 746-
757.
Andrews, D.A., & Bonta, J. (2010). The Psychology of Criminal Conduct (Fifth Edition). New
Providence, NJ: Matthew Bender & Company Inc.
Ang, P. & Hughes, J. N. (2001). Differential benefits of skills training with antisocial youth based
on group composition: a meta-analytic investigation. School Psychology Review, 31,
164-185.
Anglin, J. (2004). Creating "well functioning" residential care and defining its place in a system of
care. Child and Youth Care Forum, 33, 175-192.
Armelius, B-Å & Andreassen, T.H. (2007). Cognitive-behavioral treatment for antisocial behavior
in youth in residential treatment. Cochrane Database of Systematic Reviews 2007,
Issue 4, Art.No. CD005650, DOI: 10.1002/14651858. CD005650.pub2.
* Armstrong, T.A. (2003). The effect of moral reconation therapy on the recidivism of youthful
offenses. Criminal Justice and Behavior, 30, 668–687.
Barth, R.P. (2002). Institutions vs. foster homes: the empirical base for the second century of
action. Chapel Hill, N.C.: School of Social Work, Jordan Institutes for Families.
* Barth, R.P., Greeson, J.K., Green, R.L., Hurley, S., & Sisson, J. (2007). Outcomes for youth
receiving intensive in-home therapy or residential care: a comparison using propensity
scores. American Journal of Orthopsychiatry,77, 497-505.
Bates, B.C., English, D.J., & Kouidou-Giles, S. (1997). Institutional treatment and its alternatives:
a review of the literature. Child & Youth Care Forum, 26, 7-51.
* Berger, L.M., Bruch, S.K., Johnson, E.I., James, S., & Rubin, D. (2009). Estimating the “impact”
of out-of-home placement on child well-being: approaching the problem of selection
bias. Child Development, 80, 1856-1876.
Bickman, L., Vides de Andrade, A.R., Lambert, E.W., Doucette, A., Sapyta, J., Boyd, A.S.,
Rumberger, D.T., Moore-Kurnot, J., McDonough, L.C., & Rautkis, M.B. (2004). Youth
Therapeutic Alliance in Intensive Treatment Settings. The Journal of Behavioral Health
Services & Research, 31, 134-148.
* Brugman, D., Bink, M.D., Nas, C.N., & Van den Bos, J.K. (2007). Kunnen delinquent jongeren
elkaar helpen in hun sociale ontwikkeling? Effecten peer-hulpprogramma EQUIP op
denkfouten en recidive. Tijdschrift voor Criminologie, 47, 153-169.
* Caldwell, M.F., & Van Rybroek, G.J. (2001). Efficacy of a decompression treatment model in
the clinical management of violent juvenile offenders. International Journal of Offender
Therapy and Comparative Criminology, 45, 469-477.
14
* Caldwell, M.F., & Van Rybroek, G.J. (2005). Reducing violence in serious juvenile offenders
using intensive treatment. International Journal of Law and Psychiatry, 28, 622-636.
* Cann, J., Falshaw, L., & Friendship C. (2003). Understanding ‘What Works’: Accredited
Cognitive Skills Programmes for adult men and young Offenders. Youth Justice, 5, 165-
179.
* Chamberlain, P., & Reid, J. B. (1998). Comparison of two community alternatives to
incarceration for chronic juvenile offenders. Journal of Consulting and Clinical
Psychology, 66, 624−633.
Cook, C.R., Gresham, F.M., Kern, L., Barreras, R.B., Thornton, S & Crews, S.D., (2008). Social
skills training for secondary students with emotional and/or behavioral disorders: A
review and analysis of the meta-analytic literature. Journal of Emotional and Behavioral
Disorders, 16, 131-144.
Dishion, T.J., McCord, J., & Poulin, F. (1999). When interventions harm. American Psychologist,
54, 755-764.
Downs S. H., & Black, N. (1998). The feasibility of creating a checklist for the assessment of the
methodological quality both of randomised and non-randomised studies of health care
interventions. Journal of Epidemiological and Community Health, 52, 377-384.
Duncan, B.L., Miller, S.D., Wampold, B.E., & Hubble, M.A. (2009). The heart and soul of Change.
Delivering What Works in Therapy. Washington D.C.: American Psychological Association.
Durlak, J.A., Weissberg, R.P, & Pachan, M. (2010). A meta-analysis of after-school programs that
seek to promote personal and social skills in children and adolescents. American
Journal of Community Psychology, 45, 294-309.
Egger, M., Smith, G.D., Schneider, M., & Minder, C. (1997). Bias in meta-analysis detected by a
simple graphical test. British Medical Journal, 315, 629-634.
French, K.M. & Cameron, G. (2002). Treatment of choice or a last resort? A review of
institutional mental health placements for children and youth. Child and Youth Care
Forum, 32, 307-339.
* Frensch, K., Cameron, G., & Preyde, M. (2009). Community adaptation of youth accessing
residential programs or a home-based alternative: School attendance and academic
functioning. Child & Youth Care Forum. 38, 287-303.
Garret, C.J. (1984). Meta-analysis of the effects of institutional and community residential
treatment on adjudicated delinquents. University of Colorado at Boulder: Unpublished
dissertation.
Garrido, V., & Morales, L.A. (2007). Serious (violent and chronic) juvenile offenders: a
Systematic Review of treatment effectiveness in secure corrections. In: The Campbell
Collaboration Reviews of Intervention and Policy Evaluations (C2-RIPE), July 2007.
Philadelphia, Pennsylvania: Campbell Collaboration.
15
Goffman, E. (1961). Asylums: Essays on the social situation of mental patients and other
inmates. New York: Doubleday.
* Gordon, J.A., Moriartry, L.J., & Grant, P.H. (2000). The impact of a juvenile residential
treatment center on minority offenders. Journal of Contemporary Criminal Justice, 16,
194-208.
* Guerra, N.G., & Slaby, R.G. (1990). Cognitive mediators of aggression in adolescent offenders:
2. Intervention. Developmental Psychology, 26, 269−277.
Hair, H. J. (2005). Outcomes for children and adolescents after residential treatment: A review
of research from 1993-2003. Journal of Child and Family Studies, 14-4, 551-575.
* Hawkins, J.D., Jenson, J.M., Catalano, R.F., & Wells, E.A. (1991). Effects of skills training
intervention with juvenile delinquents. Research on Social Work Practice, 1, 107-121.
* Hengeler, S.W., Rowland, M.D., Randall, J., Ward, D.M., Pickrel, S.G., Cunningham, P.B., Miller,
S.L., Edwards, J., Zealberg, J.J., Hand, L.D., & Santos, A.B. (1999). Home-based
multisystemic therapy as an alternative to the hospitalization of youths in psychiatric
crisis: clinical outcomes. Journal of the American Academy of Child & Adolescent
Psychiatry, 38, 1331-1339.
* Henggeler, S.W., Rowland, M. D., Halliday-Boykins, C., Sheidow, A. J., Ward, D. M., & Randall,
J. (2003). One-year follow-up of multisystemic therapy as an alternative to the
hospitalization of youths in psychiatric crisis. Journal of the American Academy of Child
and Adolescent Psychiatry, 42, 543−551.
Hoagwood, H., Burns, B.J., Kiser, L., Ringeisen, H., & Schoenwald, S.K. (2001). Evidence-based
praktijk in kinderen en adolescenten diensten voor geestelijke gezondheidszorg.
Psychiatric Services, 52, 1179-1189.
Hox, J. (2002). Multilevel Analysis. Techniques and Applications. London: Lawrence Erlbaum
Associates, Publishers.
James, C., Stams, G.J.J.M., Ascher, J.J., De Roo, A.R., & Van der Laan, P.H. (2011)
Aftercare programs for reducing recidivism among juvenile and young adult
offenders: A Meta Analytic Review. Manuscript submitted for publication.
* Katz, L.Y., Cox, B.J., Gunasekara, S., & Miller, A.L. (2004). Feasibility of dialectical
behavior therapy for suicidal adolescent inpatients. Journal of the American
Academy of Child and Adolescent Psychiatry, 43, 276−282.
* Kazdin, A.E. (1987). Treatment of antisocial behavior in children: current status and future
directions. Psychological Bulletin, 10, 187-203.
Koning, N.D. de, & Collin, P.J.L. (2007). The treatment of children and adolescents with a
psychiatric disorder and mental retardation. Kind en Adolescent, 28, 215-229.
16
Knorth, E.J., Harder, A.T., Zandberg, T., & Kendrick, A.J. (2008). Under one roof: A review and
selective meta-analysis on the outcomes of institutional child and youth care. Children
and Youth Services Review, 30, 123-140.
* Kolko, D. J., Loar, L. L., & Sturnick, D. (1990). Inpatient social–cognitive skills training groups
with conduct disordered and attention deficit disordered children. Journal of Child
Psychology and Psychiatry, 31, 737−748.
* Leve, L.D., & Chamberlain, P. (2005). Association with delinquent peers: Intervention effects
for youth in the juvenile justice system. Journal of Abnormal Child Psychology, 33,
339−347.
Losel, F. & Beelmann, A. (2003). Effects of child skills training on preventing antisocial behavior:
A systematic review of randomized evaluations. Annals of the American Academy of
Political and Social Science, 587, 84-109.
Lyons, J.S., & McCulloch, J.R. (2006). Monitoring and managing outcomes in institutional
treatment: Practice-based evidence in search of evidence-based practice. Journal of the
American Academy of Child and Adolescent Psychiatry, 45, 247-251.
Lyons, J.S., Woltman, H., Martinovich, Z., & Hancock, B. (2009). An outcomes perspective of
residential treatment in the system of care. Residential Treatment For Children & Youth,
26, 71-91.
Lipsey, M.W. (2009). The primary factors that characterize effective interventions with juvenile
offenders: a meta-analytic overview. Victims and Offenders, 4,124-147.
Lipsey, M.S. & Wilson, D.B. (2001). Practical Meta-Analysis. Applied Social Research Methods
Series (vol. 49). Thousand Oaks, CA Sage publications.
Maag, J.W. (2005). Social skills training for youth with emotional and behavioral disorders and
learning disabilities: Problems, conclusions, and suggestions. Exceptionality, 13, 155-
172.
Maag, J.W. (2006). Social skills training for students with emotional and behavioral disorders: A
review of reviews. Behavioral Disorders, 32, 4-17.
* Mattejat, F., Hirt, B.R., Wilken, J., Schmidt, M. H., & Remschmidt, H. (2001). Efficacy of
inpatient and home treatment in psychiatrically disturbed children and adolescents.
European Child and Adolescent Psychiatry, 10, 71−79.
* McCrae, J.S., Bethany, R.L., Barth, R.P., & Rauktis, M.E. (2010). Comparing three of well-being
outcomes for youth in group and nonkinship foster care. Child Welfare, 89, 229-249.
McLeigh, J.D., & O’Neill Briddell, L. (2011). Where Have All the Children Gone? The Effects of the
Justice System on America’s Children and Youth. The Community, 1, 334-341.
* Moody, E. E. (1997). Lessons from pair counseling with incarcerated juvenile delinquents.
Journal of Addictions and Offender Counseling, 18, 10−25.
Mullen, B. (1989). Advanced basic meta-analysis. New Jersey: Laurence Erlbaum Associates.
17
Quinn, M.M., Kavale, K.A., Mathur, S.R., Rutherford, R.B. & Forness, S.R. (1999). A meta-
analysis of social skills interventions for students with emotional or behavioral disorders.
Journal of Emotional or Behavioral disorders, 7, 54-64.
* Rohde, P., Jorgerson, J. S., Seeley, J. R., & Mace, D. E. (2004). Pilot evaluation of the coping
course: A cognitive behavioral intervention to enhance coping skills in incarcerated
youth. Journal of the American Academy of Child and Adolescent Psychiatry, 43,
669−676.
Rosenthal, R. (1995). Writing meta-analytic Reviews. Psychological Bulletin, 118, 183-192.
* Ryan, J.P., Marshall, J.M., Herz, D., & Hernandez, P.M. (2008). Juvenile delinquency in child
welfare: investigating group home effects. Children and Youth Services Review,
30, 1088-1099.
Scherrer, J.L. (1994). A Meta-analysis of the effectiveness of residential treatment programs for
children and adolescents. University of Illinois at Chicago: unpublished dissertation.
Schneider, B.H. (1992). Didactic Methods for enhancing children’s peer relations: a quantative
review, Clinical Psychology Review, 12, 363-382.
* Scholte, E. M., & Van der Ploeg, J. D. (2003). Effectiveness of residential treatment methods
for youngsters with severe behavioral problems: Findings from a one year follow-up
study. International Journal of Child and Family Welfare, 6, 85−197.
Stams, G.J., Brugman, D., Deković, M., Van Rosmalen, L., Van der Laan, P.H., & Gibbs, J.C.
(2006). The moral judgment of delinquents: a meta-analysis. Journal of Abnormal Child
Psychology, 34, 692-708.
Sutton, A.J., Duval, S.J., Tweedie, R.L., Abrams, K.R., & Jones, D.R. (2000). Empirical
assessment of effect of publication bias on meta-analyses, British Medical Journal,320,
1574-1577.
Tabachnick, B.G. & Fidell, L.S. (2001). Using multivariate statistics. New York: Harper & Row.
* Thompson, R.W., Smith, G. L., Osgood, D.W., Dowd, T.P., Friman, P. C., & Daly, D.L. (1996).
Residential care: A study of short- and long-term educational effects. Children and Youth
Services Review, 18, 221−242.
Van der Helm, G.H.P., Boekee, I., Stams, G.J.J.M., Van der Laan, P.H. (in press). Fear is the key:
keeping the balance between flexibility and control in a Dutch youth prison. Journal of
Children’s Services.
Van der Helm, G.H.P., Klapwijk, M., Stams, G.J.J.M., & Van der Laan, P.H. (2009). ‘What Works’
for juvenile prisoners: The role of group climate in a youth prison. Journal of Children’s
Services, 4, 36-48.
Van Nieuwenhuijzen, M., Orobio de Castro, B., Wijnroks, L., & Vermeer, A. (2009). Social
problem-solving and mild intellectual disabilities: Relations with externalizing behavior
18
and therapeutic context. American Association on Intellectual and Developmental
Disabilities, 114(1), 42-51.
Weisz, J.R., Jensen-Doss, A., & Hawley, K.M. (2006). Evidence-based youth psychotherapies
versus usual clinical care. A meta-analysis of direct comparisons. American Psychologist,
61, 671-689.
* Wilmshurst, L.A.(2002). Treatment Programs for Youth with Emotional and Behavioral
Disorders: An Outcome Study of Two Alternate Approaches. Mental Health Services
Research, 4, 85-96.
Wilson, D.B. (2001). Effect size determination program.
Wilson, D.B., MacKenzie, D.L., & Mitchell, F.N. (2005). Effects of correctional boot camps on
offending. New York: Campbell Collaboration.
19
Appendix Appendix Appendix Appendix AAAA Study characteristics
Study Publication year
Impact factor journal
Study quality rating
Worldpart of origin
Comparison Design Intervention residential
N Sex Avarage Age
Ethnic minority
Crime justice intervention
Open or incarcerated
d
Abrams et al. 2008 1,17 14 North America
TreatRes – Groupcare
non matched
skills 61 boys 15,90 73,77% yes incarcerated -0.563
Armstrong 2003 1,88 16 North America
TreatRes – Groupcare
RCT skills 212 boys 20,21 68,87% yes incarcerated 0.005
Barth et al. 2007 2,21 14 North America
GroupCare – TreatNotres
matched control
TAU 786 mixed 11,42 39,35% no open -0.146
Berger et al. 2009 3,63 16 Western Europe
GroupCare - NoTreatNoRes
matched control
TAU 1710 mixed 9,78 52,00% no open -0.054
Brugman et al.
2007 0,00 15 Western Europe
TreatRes – Groupcare
non matched
skills 77 boys 16,40 unknown yes incarcerated -0.083
Caldwell & Rybroek
2005 1,10 14 North America
TreatRes – Groupcare
matched control
moderate 248 boys 16,00 62,00% yes incarcerated 0.537
Caldwell & Rybroek
2001 1,09 18 North America
TreatRes – Groupcare
matched control
CBT 30 boys unknown unknown yes incarcerated 1.806
Cann et al. 2005 0,00 16 Western Europe
TreatRes – Groupcare
matched control
CBT 3068 boys 17,60 unknown yes incarcerated 0.113
French, Cameron & Preyde
2009 0,00 15 North America
GroupCare – TreatNotres
non matched
TAU 210 mixed 11,55 1,00% no open 0.027
Gordon 2000 1,05 19 North America
TreatRes – Groupcare
RCT CBT 480 boys 16,62 42,71% yes open 0.260
Guerra & Slaby
1990 3,56 16 North America
TreatRes – Groupcare
matched control
CBT 120 mixed 17,20 60,00% yes incarcerated 0.345
Hawkins et al.
1991 1,04 14 North America
TreatRes – Groupcare
RCT CBT 150 mixed 15,75 49,65% yes incarcerated 0.610
Henggeler et al.
1999 4,98 13 North America
GroupCare – TreatNotres
RCT TAU 116 mixed 13,00 66,00% no open -0176
Henggeler et al.
2003 4,98 15 North America
GroupCare – TreatNotres
RCT TAU 160 mixed 12,90 67,00% no open -0.149
Katz et al. 2004 4,98 16 North America
TreatRes – Groupcare
non matched
CBT 62 mixed 15,40 27,40% no open 0,410
Kazdin 1987 4,46 15 North America
TreatRes – Groupcare
CBT 37 mixed 10,90 23,20% no open 1.246
Kolko et al. 1990 4,98 16 Western Europe
TreatRes – Groupcare
matched control
CBT 56 mixed 10,40 50,32% no open -0,317
Leeman et al.
1993 1,70 18 North America
TreatRes – Groupcare
RCT skills 54 boys 16,00 33,33% yes incarcerated 0.629
Leve & Chamberlain
2005 2,97 17 North America
GroupCare – TreatNotres
RCT TAU 153 mixed 14,88 21,57% yes open -0.690
Mattejat et al.
2001 2,38 15 Western Europe
GroupCare – TreatNotres
RCT TAU 68 mixed 11,90 unknown no open -0.039
20
Mc Crae 2010 0,67 18 North America
GroupCare - NoTreatNoRes
matched control
TAU 124 mixed 12,20 67,70% no open -0.069
Moody 1997 0,00 11 North America
TreatRes – Groupcare
non matched
skills 28 boys 14,20 89,29% yes incarcerated 0.266
Rohde 2004 4,98 14 North America
TreatRes – Groupcare
RCT CBT 76 boys 16,30 41,70% yes incarcerated 0.469
Ryan et al. 2008 1,17 18 North America
GroupCare - NoTreatNoRes
matched control
TAU 8226 mixed 8,50 83,00% no open -0.354
Scholte & Van der Ploeg
2003 0,00 13 Western Europe
TreatRes – Groupcare
non matched
skills 74 mixed 14,90 35,00% mixed incarcerated -0.142
Thompson 1996 1,12 15 North America
TreatRes – Groupcare
unknown skills 587 mixed 14,55 29,60% mixed open 0.375
Wilmshurst 2002 0,00 17 North America
TreatRes - TreatNotRes
non matched
skills 65 mixed 11,12 5,00% no open -0.338