A systematic review of evidence-based practice implementation in drug and alcohol settings: applying the consolidated framework for implementation research framework

Criteria for considering studies for this review were classified by:

The following electronic databases were searched (from inception to April 2020): PubMed/MEDLINE, Cochrane Library, PsycINFO, Web of Science, and CINAHL. Reference searches of relevant reviews and articles were also conducted. Similarly, a grey literature search was done with help of Google and the Grey Matters tool which is a checklist of health-related sites organised by topic. The tool is produced by the Canadian Agency for Drugs and Technologies in Health (CADTH) [34].

The search included all relevant peer-reviewed studies. The search was conducted across 4 relevant concepts (see draft strategy in Additional file 2): (1) implementation, (2) evidence-based practices, (3) drug and alcohol service setting and (4) eligible research designs. The MEDLINE search strategy is available in Additional file 2.

Two reviewers independently screened all articles identified from the search. First, titles and abstracts of articles returned from initial searches were screened based on the eligibility criteria outlined above. Second, full texts were examined in detail and screened for eligibility. Third, references of all considered articles were hand-searched to identify any relevant report missed in the search strategy by two reviewers independently. Any disagreement between reviewers was resolved by discussion to meet a consensus. EndNote version X9 (Clarivate Analytics) was used to manage all records.

Two researchers extracted data and organised it into variables based on the Cochrane Effective Practice and Organisation of Care (EPOC) Data Abstraction Form (e.g. clinical interventions, strategies, outcomes, and results), the conceptual model of Proctor et al. [9] (implementation, service system and clinical outcomes), information about any specific implementation frameworks used and a checklist of items aligned with the domains and subdomains of the CFIR (i.e. subdomains associated with intervention characteristics, outer setting, inner setting, characteristics of individuals, and the implementation process; see Table 1). This method was used effectively in two previous reviews [18, 35] as a means of categorising the types of implementation strategies addressed by each of the studies included in the review.

All included studies were critically evaluated by two researchers independently using the Revised Cochrane risk-of-bias tool (RoB 2) [22]. The RoB 2 provides a systematic assessment across five domains of bias (the randomisation process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of the reported results) to assess quality of the article per outcome. For cluster-randomised studies, an additional domain was used when assessing the randomisation process. Trial registries were also checked to determine the integrity of reported outcome measures and statistical methods. The grey literature search also assisted with identifying publication bias.

Included studies did not have sufficient characteristics for a meta-analysis and therefore a narrative synthesis was performed. The main methods of synthesis involved tabulation using “meta-matrices” [36], textual descriptions, qualitative synthesis of themes [37] and content analysis to determine the frequency of categorised data [38]. The findings from the included articles were synthesised using the CFIR framework.

As displayed in the flowchart (Fig. 1), the database search identified 2965 studies. After titles were screened, 159 studies were found to be relevant (103 of which were replicas). Abstracts of the remaining studies were screened and 26 were found to meet inclusion criteria. Finally, full-text articles of these studies were assessed for eligibility and 19 were included in the review. An additional, identical search was conducted to capture any further relevant studies conducted between the time the first search was conducted until April 2020. This search identified 91 studies, one of which met eligibility criteria and was included in the review. An outline of the main features of included studies is provided in Table 2, including the type of innovation, guiding theories, strategies employed, study design, treatment setting, participant characteristics, study outcomes, CFIR domains evaluated, and the effectiveness of the implementation.

The majority of studies (16, 80% [39, 42, 45, 47, 54, 59‐62, 65, 66, 68‐71]) were conducted in the United States of America (USA), outpatient, not-for-profit drug and alcohol services. Alternate settings included one USA adolescent day programme affiliated with the University of Miami Medical School and Jackson Memorial Hospital [55], one outpatient drug and alcohol service affiliated with a university hospital in Switzerland [46], one drug abuse treatment organisation in Peru funded by a US Department of State contract [50], and one involved outpatient addiction treatment centres in South Africa [67]. Study participants were most often female (50–82%) drug and alcohol clinicians, with a mean age ranging from 37 to 48 years. Participants were also mostly Caucasian (50–100% in US studies) and were otherwise African American (14 to 40%), Hispanic (7 to 50%) or some other type of ethnicity (1 to 12.6%). In the South African study participants were also mainly Caucasian (36.4%), with Africans representing 30.8%, 12.6% identifying as “mixed-race”, and 14% Other. Participants commonly held bachelor’s degrees or higher (54 to 100%) and had 3+ to 9.5 years of experience.

Nine (45%) of included studies were randomised controlled trials [59, 61, 62, 65‐69], eight (40%) were randomised trials [39, 42, 43, 50, 54, 60, 70, 71](one of which was a subject-by-trial split plot design with repeated measures, [50]), one was a cluster randomised trial [45], one was an interrupted time series design [55], and one was a controlled before-and-after study [47]. Studies varied in terms of the number of participants, the length of follow-up period, the number of addiction services clinicians were sourced from, and the levels of intervention in the approach.

All included studies were concerned with training as an implementation strategy. Approximately one third (n = 7) used multiple strategies that involved both passive (e.g. manuals and seminars) and active (e.g. supervision, workshops and champions) approaches to training [47, 59, 62, 65, 66, 68, 71], while 20% (n = 4) focused on discrete strategies (e.g. supervision [61], financial incentives [45], booster sessions [50], and workshop only [46]). Another third (n = 6) used technological strategies such as teleconferencing and web-based training [42, 54, 67‐70]. Three studies (15%) focused on the influence of the intervention context on the uptake of the intervention [39, 55, 60].

Fixsen and colleagues’ [16] conceptualisation of the implementation literature was the most frequently cited (3 of the 20 studies). These studies [59, 60, 67] incorporated Fixsen et al.’s recommendations regarding the importance of training in evidence-based practices through establishing i) program-based advocates, ii) providing adequate feedback and supervision, and iii) developing cost-effective approaches to training and coaching treatment providers. Suggestions from Carroll and Rounsaville [72] were also incorporated in one study [59] specifically in regards to the lack of effective program-based supervision in empirically supported treatments being one of the largest barriers to the implementation of these approaches in clinical practice. While only two studies were guided by Rogers’ [44, 48] argument that individuals are more likely to adopt an intervention after they have an increased knowledge about it and then develop a more favourable attitude towards it, eight (40%) adopted the notion that clinician factors may mitigate the relationship between fidelity to an intervention and patient outcomes [39, 42, 43, 46, 47, 54, 60, 62]. Clinician factors of interest included demographics (e.g. gender, age, experience, education; measured in all of the studies, although only sixteen (80%) reported an intention to evaluate these factors in relation to the implementation, [39, 42, 43, 46, 47, 50, 54, 59, 62, 65‐71]), knowledge (3 studies, [67, 69, 70]) and attitudes (6 studies, [39, 43, 46, 54, 59, 65], e.g. beliefs about the origins of addictive behaviour, beliefs about evidence-based treatments (EBTs) or about the intervention itself; learning, confidence and commitment). Factors related to the context of the intervention were the focus of five studies [39, 43, 55, 59, 60], and included organisational factors, organisational readiness for change, and the importance of the context and multilevel approaches. Only two of these studies [55, 60] adopted Simpson’s [56] recommendations about “systemically-oriented” dissemination models, and the evaluation of these efforts in multiple domains, including organisational, clinician and client outcomes. However, two studies [39, 47] used a comprehensive implementation framework. One was entitled “Context-Tailored Training” [39], which is a method of training tailored to the unique challenges of a work setting and the other was the CFIR [47].

The remaining studies drew upon general research or theories that provided a rationale for the training strategies employed. For instance, some identified specific barriers to implementation such as the barrier of limited resources and the challenge of developing cost-effective approaches, (e.g. [5, 73, 74], others presented evidence for the potential uses of technology (e.g. [75])and two studies referenced psychological theories that inform approaches to learning (e.g. [76‐80].

As can be seen in Table 3, of the 36 subdomains of the CFIR, 32 were evaluated by included studies, although one study [39] mainly contributed to the breadth of coverage. Missing constructs included Intervention Characteristics related to evidence strength and quality, Outer Setting constructs including peer pressure and external policies and incentives, and the Inner Setting construct related to the relative priority of the implementation climate. While sixteen (80%, of studies) evaluated Characteristics of Individuals, less than half (9, 45%) measured Intervention Characteristics, and even fewer (4, 20%) measured Inner Setting constructs, with only one study [39] measuring Outer Setting constructs and the Process domain.

Almost all implementation outcome measures were concerned with fidelity to the intervention (17, 85%), although three studies measured knowledge [67, 69, 70], two studies measured self-efficacy [70, 81], two studies measured the cost of the intervention [61, 67], two studies measured adherence to the training [39, 55], one study measured supervision integrity [61], and one study measured adoption [47]. Predictors of implementation including clinician characteristics were measured by sixteen (80%) studies and clinician evaluation of the training was measured by four (20%) studies [39, 47, 50, 66]. The most frequently measured clinician characteristics were demographics (such as age, gender, ethnicity, education, experience, prior exposure to the intervention, counselling style or techniques, knowledge and attitudes towards evidence-based practices or the intervention itself, and recovery status). Predictors of implementation related to organisational level factors were measured by five studies [39, 47, 50, 54, 55], covering categories including acceptability, appropriateness, feasibility and penetration. Clinician evaluations of the training largely related to satisfaction with the format, methods, attributes and overall experience of the training, as well as the clinical utility of the training, and one question addressed any ideological conflict experienced. Further questions assessed clinicians’ views of the classroom environment (1) and cultural sensitivity of the training material (1).

Service system outcomes such as location, entry criteria, types of services offered, client to staff ratio, staff turnover, and record data quality, were measured by one study [50].

Clinical outcomes were measured by eight (40%) studies. Substance use was the most frequently measured outcome, followed by retention in treatment. Other clinical outcome measures included emotional and behavioural symptoms, change in motivation and treatment utilisation. In terms of patient characteristics, two studies measured age and gender [46, 71], one of which [71] also measured additional demographics, employment status, admission to the legal system, prior treatment and type of substance.