Evidence-Based Practices in Addiction Treatment: Review and Recommendations for Public Policy

Best Research Evidence

Although it has been appropriately noted that the definition of best research evidence depends upon the nature of the clinical question (e.g., etiological questions versus identification of the most efficacious treatment for a particular disease) [7], descriptions of this concept to date uniformly acknowledge a variety of sources from which data can be brought to bear on clinical decision making. These sources include randomized clinical trials (RCTs), quasi-experimental investigations, correlational studies, field studies, case reports, and clinical guidelines based upon professional consensus [8]. Moreover, it has been argued that the rigor of the study design may correspond to a hierarchical ranking denoting experimental integrity; on this basis, the value of the evidence for a particular approach could be “graded.” [9]. Evidentiary value encompasses the concepts of (a) evidence quality, or the extent to which bias is minimized in the context of the experimental design, and (b) evidence strength, which is inferred collectively on the basis of evidence quality, the size of treatment effects, the extent to which the outcomes reflect valid information about the populations and settings in which the study was conducted (i.e., internal validity), and the clinical utility and generalizability (i.e., external validity) of the findings [6,9]. Within this framework, RCTs are considered to be the least subject to bias, and therefore the most empirically “sound” source of evidence (i.e., that of the highest evidence quality). Table 1 depicts a hierarchical model of research evidence, drawing upon common elements of previously described systems for grading study quality [9–11].

As an alternative to weighting individual studies and drawing corresponding conclusions regarding the evidence basis for particular interventions, clinicians may draw upon published syntheses of study findings, typically in the form of systematic reviews and meta-analyses. Hierarchical models of research evidence place both of these methods in the highest tier alongside RCTs. Systematic reviews evaluate research evidence based upon pre-defined objective criteria. Over the past decade, the Cochrane Collaboration and the Agency for Healthcare Research and Quality have accelerated the dissemination of synthesized information concerning health care practices through systematic reviews [12,13]. Although these reviews may or may not use meta-analytic techniques, the standard methodology for evaluating the strength of evidence in meta-analyses involves the calculation of an effect size, or summary statistic depicting the magnitude of the treatment effect, averaged across studies. While this method provides a useful metric from which to infer the utility of a treatment approach for a population or subgroup, some disadvantages of this technique warrant consideration. Specifically, the use of aggregate estimates of effect size may obscure qualitative differences between the individual studies, including variations in experimental integrity, subject characteristics, and study endpoints. Moreover, meta-analyses are subject to publication bias (also known as the “file drawer problem”), in that studies showing an effect of a treatment are more likely to be published than those showing no effect, thereby biasing the pool of clinical data from which meta-analyses are conducted. Nevertheless, synthesized reports on treatment effectiveness, whether in the form of meta-analyses or systematic reviews, remain a valuable resource to inform clinical decision-making.

Among the many constituencies advocating for the use of EBP in behavioral health, some have proposed highly specified criteria that reflect the number and types of trials required to establish a treatment as “evidence-based.” To this end, in 1995, the American Psychological Association’s Division of Clinical Psychology published criteria for identifying evidenced-based treatments, formally labeled “empirically supported treatments.” Highlighting the distinction between treatment efficacy (i.e., clinical benefit produced by the intervention in the context of controlled research) and effectiveness (i.e., clinical benefit produced by the intervention in a clinical setting under naturalistic conditions), the criteria for empirically supported treatments require demonstration of efficacy in at least two investigations conducted by independent research teams [14]. Using a graded system, the criteria for empirically supported treatments specify that an intervention with evidence in favor of its use from a single study or from multiple studies conducted by the same research group is considered possibly efficacious pending replication. While studies demonstrating clinical utility of an intervention outside of the experimental setting are considered important for the translation of manualized interventions studied in RCTs into real-world clinical settings, the designation of a treatment as an empirically supported treatment does not require evidence of effectiveness; rather, efficacy is considered paramount [15].

Critics of empirically supported treatments’ dissemination have argued that interventions established through efficacy research are unlikely to generalize to “real world” clinical settings [16,17]. Contrary to this argument, however, an emerging body of effectiveness research has yielded promising evidence for psychotherapy approaches previously established through efficacy research [18,19]. Effectiveness studies are not without their limitations, however; clients in some effectiveness studies receive more treatment than is routinely provided in efficacy trials and achieve treatment effects of a smaller magnitude than that observed in efficacy studies [20]. Moreover, though high in external validity, the research designs employed are often problematic with respect to internal validity.

Clinical Expertise

The notion that the clinician’s expertise and experience are key components of the science-to-practice translation is well accepted [14,15]. Numerous elements constitute clinical expertise, including scientific expertise to guide evaluation and use of research evidence, awareness of individual patient characteristics as they influence treatment needs, interpersonal ability, awareness of the limits of one’s clinical skill set, and clinical decision-making [15].

Standards and regulations regarding the level of training required of clinicians delivering addiction treatment directly impact the extent to which clinicians in such settings evidence these elements of expertise, however, and recent studies suggest that the current educational requirements for substance abuse counselors fall short as a means of preparing them to adopt EBPs. Specifically, the minimum educational requirements in most states fail to support the development of skills to review and understand research evidence; according to a recent study of state requirements for the training of addiction counselors in 31 states, only one state (3%) mandated coursework in research and evaluation [21]. Moreover, the association between level of training (i.e., as indicated by degrees and certification) and innovation adoption is well-established [22,23]; thus, the potential success of EBP dissemination efforts in addiction treatment settings will depend, in part, on the pre-existing level of education and training of the workforce. Nevertheless, the majority of U.S. states require less than college education for entry to the field [21]. Increasing the rigor of training, certification, and licensing requirements for the addiction treatment workforce is therefore an important consideration as a means of facilitating successful technology transfer [22,24].

A wealth of evidence suggests that interpersonal skill is a particularly important aspect of clinical expertise [25], and individual therapist effects account consistently and significantly for variance in outcomes [26–28]. As such, scientific experts recommend that psychosocial EBP dissemination efforts in addiction focus on a limited set of core change principles with corresponding skill sets that can be widely applied to clinicians with varying levels of experience [29]. Arguably, a key component of these principles would involve skills to establish and maintain a therapeutic alliance, along with techniques that facilitate use of the alliance to promote behavior change.

Variations in the concept of EBP

Although several aspects of the guidelines set forth by the APA Division of Clinical Psychology have been sources of controversy [30], their initiative to disseminate information on empirically supported treatments was followed by numerous ongoing efforts to develop EBP guidelines. In the process of accelerated efforts towards EBP implementation, however, a variety of terms have been utilized to describe documented sets of guidelines, some of which have meaningful distinctions. Apart from the term EBP, two additional broad categories of documents that summarize recommendations for translating research evidence into clinical practice include: practice guidelines and best practices. Although numerous other terms have been used, they generally fall within the scope one of these three categories of recommendations.

Treatments that Don’t Work

It has been cogently argued that identifying the interventions for which there is the strongest empirical evidence of efficacy and/or effectiveness is equally as important as knowledge of which treatments are ineffective or perhaps even detrimental to the clinical course and outcomes of addicted patients. As such, to the extent that lists of EBPs are utilized to drive the development and refinement of addiction treatment programs, lists of discredited treatments may help providers avoid ineffective ones. To this end, a recent Delphi survey delineated a set of intervention approaches that are contraindicated treatments for addiction [52]. A list of these treatments is provided in Table 3.

While the treatments that are considered to be effective are established as such through empirical research, the absence of efficacy studies does not render an intervention approach ineffective [8]. This important argument underscores one of the problems with reliance upon well-studied interventions in shaping the treatment system. This may, in effect, exclude interventions that have not had the opportunity to accumulate evidence in support of their use. Certain easily or already well-standardized approaches to addiction treatment are more likely to be tested in RCTs (e.g., pharmacotherapies, cognitive behavioral therapy), thereby biasing the pool of available interventions by including only those that have generated the most scientific interest.

Workforce Barriers

Workforce characteristics are important determinants of EBP adoption. Providers’ familiarity with EBPs, perceptions of their effectiveness, and attitudes towards them are each associated with the likelihood of successful implementation [53–55]. Spreading awareness of EBPs and their effectiveness is a complex process which requires support from the addiction treatment system’s infrastructure. Nevertheless, surveys of the addiction treatment system, such as the National Survey of Substance Abuse Treatment Services [56], consistently reveal gross inadequacies in this infrastructure, particularly in the leadership, workforce, and information systems fundamental to supporting quality evidence-based care. System-level change has therefore been an emphasis of numerous discussions in recent years concerning EBP dissemination strategies [56–59].

Intervention Fidelity

In the face of nonstandard implementation, the transfer of EBPs into clinical settings can be problematic. Research consistently shows that accurate implementation of EBP protocols is associated with positive clinical outcomes [60,61]. As such, fidelity measurement is used to assess the extent to which an EBP is being implemented as intended. As yet, there is no consensus as to how to optimize fidelity assessment of EBPs for substance use disorders. Typically, survey research methods aimed at providers are utilized to examine the extent of EBP implementation in treatment programs. This approach is not without drawbacks, however. When surveyed, providers may over-estimate the extent to which they utilize EBPs, including those for which they have received no formal training [62]. Inaccuracies in reporting are even more likely in the context of pressures arising from mandates regarding EBP delivery. Likewise, although direct observation of clinical activities might overcome some of the limitations of survey research methodology, such observation at a single time point may capture the practitioner’s ability to conduct an EBP, while leaving the nature of their routine clinical practices largely unknown.

Implementation researchers are nevertheless making strides in developing fidelity assessment methods that overcome some of these problems. One such approach was described in a recent report from the National Implementing Evidence-Based Practices Project [63], from which the first systematic study of the fidelity of EBP implementation across a large number of sites (N=53) was performed. In this longitudinal study of dissemination of 5 EBPs for mental health and co-occurring disorders, trained fidelity assessors conducted site visits before and repeatedly after a 1-year EBP implementation phase. Fidelity was assessed using a multimethod approach that included interviews with site practitioners, observation of clinical activities, interviews with clients, and chart reviews. The integration of these evaluation methods, coupled with repeated measurements over time, provides a rigorous model for fidelity assessment, one that is consistent with prior recommendations based upon a review of fidelity measurement methods in psychosocial treatment research [64].

Practice-based Evidence

Given that the evidence base for psychosocial treatments for addiction is acquired largely from RCTs, EBP dissemination efforts focus on transporting specific theoretically based approaches with a relative de-emphasis on the level of competence of the individual therapist. Variability in clinicians’ level of competency is minimized in psychosocial RCTs through the selection of highly qualified and educated therapists coupled with rigorous training, supervision, and use of manuals to inform practice. Nevertheless, within the psychotherapy literature a number of studies and meta-analyses have reported moderate to large effects of individual therapists on clinical outcomes [65], often in the absence of observed differences between psychosocial treatment approaches of varying orientations [66]. These observations have formed the basis of a rationale for an approach that emphasizes practice-based evidence.

Using practice-based evidence, client outcome data are gathered from routine practice and used to provide therapists with real-time feedback regarding the impact of their interventions on client functioning. The delivery of feedback to therapists, when coupled with suggestions to improve clinical performance, has been touted as a complementary and effective means to improve the quality of care and outcomes [67,68]. This model, which historically has been implemented in psychological treatment settings in the United Kingdom [69], is currently the focus of a statewide mental health services initiative in Utah, in which “mental health vital signs” are monitored routinely using a 5-minute self-report instrument [67]. These “vital signs,” which correspond to several domains of functioning, are analyzed using a software system that tracks clinical trajectories. Using this system, patients who are at risk for “treatment failure” are flagged, with corresponding feedback and recommendations delivered to clinicians. The practice-based evidence model can also be used at the systems level; for example, in the context of performance-based contracting, administrative and clinical practices that are linked with the achievement of targeted clinical performance indices can be adopted by the system, an approach that was successfully adapted and instituted recently in Delaware [70].

Evidence-based Caveats

Given that the primary goal of implementing EBPs is to improve client treatment outcomes, treatment-related procedures that are thought to positively affect such outcomes are worthy of consideration. NIATx, for example, developed process improvement strategies that affect important outcome dimensions for substance abusers, including treatment retention and access to care [71,72]. Although the NIATx approach is, like EBPs, a manualized approach to improving treatment process and outcome, it does not lend itself to testing in RCTs and thus does not fit squarely into the EBP model. Other such treatment processes include urine monitoring, a common practice in addiction treatment that has not been subjected to testing in controlled research. Conceptually, it is unclear where these procedures fall in the continuum of EBPs. Despite their apparent utility, it is unclear how such processes will be viewed and to what extent they will be supported at this time, given the emphasis of current policy on EBP implementation.