Contributions to the literature
-
Awareness of what constitutes rigorous multilevel implementation research is essential for theory generation and refinement across the diverse contexts in which implementation research is conducted.
-
The methodological standards explained and recommended here are critical for planning, evaluating, and replicating multilevel implementation research.
-
This manuscript articulates eight characteristics of rigorous, high-quality multilevel implementation research and provides prompts, topic-specific references, and implementation examples to help readers incorporate these ideas into their studies.
Background
When are multilevel methods necessary in implementation science?
Challenges of multilevel research
Current literature
The predicament of the implementation scientist interested in conducting multilevel research
Our approach will be to see the ‘forest’ rather than some particular ‘trees.’ We examine the big picture, indicating the main elements of multilevel research. An exhaustive analysis of all the elements of multilevel research goes beyond the purpose of this methodological insight, but we provide key references in the literature that could be used…[with the hope that]…multilevel research brings us closer to the reality of [implementation] practice. pg. 2
Road map for this paper
Characteristic | Recommendations | In-paper resources |
---|---|---|
To conduct rigorous, high-quality multilevel implementation research... | ||
(1) Map and operationalize the specific multilevel context for defined populations and settings | 1a. Create and include a list or map of contextual levels most salient to the research question(s) and population(s) under study | ▪ Table 2 ▪ Additional File 1 ▪ Glossary |
(2) Define and state the level of each construct under study | 2a. For each construct, define its substantive meaning and the level at which it resides/population unit with which it is associated 2b. For each construct, provide an explanation or “mini theory” that explains why the construct is assigned to its specific level/population unit | ▪ Additional File 2 ▪ Glossary |
(3) Describe how constructs relate to each other within and across levels | 3a. Include a figure or narrative that describes the study’s theoretical model, including the level of each construct and the hypothesized relationships between constructs 3b. When hypothesized relationships cross levels, researchers should describe the processes through which higher-level antecedents influence lower-level consequents (i.e., top-down processes) or how lower-level antecedents shape higher-level consequents (i.e., bottom-up processes) 3c. Clarify each construct’s location in the study theoretical model relative to other constructs (e.g., is it an antecedent, mediator, consequent, primary or secondary endpoint, etc.) | ▪ Fig. 1 ▪ Additional File 3 ▪ Glossary |
(4) Specify the temporal scope of each phenomenon at each relevant level | 4a. Provide a detailed explanation of the expected temporal dynamics within the study at each level, using visual aids as needed, to include the following: i. When investigators expect to observe change in each relevant outcome at each relevant level (e.g., of system- or organization-level implementation strategies) ii. How frequently and when constructs will be measured to capture these changes iii. How changes in outcomes at different levels align with each other in the research design iv. The theoretical rationale for these choices | ▪ Additional File 4 |
(5) Align measurement choices and construction of analytic variables with the levels of theories selected (and hypotheses generated, if applicable) | 5a. Align the levels of theory and measurement; for unit-level constructs, determine whether the construct is a global, shared, or configural property of the unit and use this to align measures with theory 5b. For shared constructs, address the following: i. Include a specific referent that indicates who and/or what is being rated ii. Effectively communicate these referents to participants in measurement instruments iii. Ensure respondents who are asked to report on shared constructs can report on them, and that they are the appropriate persons to ask iv. Provide evidence that individuals within a unit reflect (and can report on) a shared phenomenon or experience v. When shared constructs are measured quantitatively using individual responses, aggregate the individual responses into unit-level scores of shared constructs | ▪ Additional File 5 ▪ Glossary |
(6) Use a sampling strategy consistent with the selected theories or research objectives and sufficiently large and variable to examine relationships at requisite levels | 6a. Design and justify a multilevel sampling plan, ensuring there is the following: i. A large enough sample at each level to rigorously test hypotheses or make theory-based inferences ii. Adequate variability within the sample at each level to rigorously test hypotheses or make theory-based inferences iii. Adequate representativeness of the achieved sample at each level (for quantitative) 6b. When reporting study findings for quantitative studies, include the following: i. The distribution and range of within-unit sample sizes ii. The distribution and range of within-unit response rates iii. A comparison of the characteristics of unit members who responded versus those who did not respond iv. The theoretical or empirical rationale for exclusion of units (as applicable) | ▪ Additional File 6 ▪ Glossary |
(7) Align analytic approaches with the chosen theories (and hypotheses, if applicable), ensuring that they account for measurement dependencies and nested data structures | 7a. Directly acknowledge dependencies (i.e., correlated observations/nesting) within the proposed study design, articulate what analytic method has been selected to account for those dependencies, and provide a rationale for the choice of analytic method with reference to specific characteristics of the data and strengths of the selected method/model 7b. For quantitative, ensure that variables enter statistical models at the level warranted and scrutinize choices related to centering, standardization, and calculation of effect sizes to confirm they reflect the study’s multilevel design; for randomized studies, the variable representing randomization to condition (i.e., exposure) should enter the statistical model at the level at which randomization occurs 7c. Be transparent and thorough in reporting details of the analytic approach 7d. Consider developing and sharing crosswalks that specify research questions and justify the use of data collection tools and their accompanying analytic techniques, defining their multilevel purpose and (anticipated) contributions, including “explicit connections” or “intentional redundancies” among quantitative and qualitative approaches 7e. Consider making final analytic tools accessible to end users of multilevel research reports (e.g., qualitative interview guides, statistical code) | ▪ Additional File 2 ▪ Additional File 7 ▪ Glossary |
(8) Ensure inferences are made at the appropriate level | 8a. Carefully craft and check language within research reports and presentations to ensure atomistic and ecological fallacies are not present | ▪ Additional File 8 ▪ Glossary |
1. Map and operationalize the specific multilevel context for defined populations and settings
Rationale
Our recommendation for implementation researchers
CFIR / EPIS | Level name | Definition | Examples |
---|---|---|---|
Outer setting / Outer Context | Countrya | A nation with its own government, occupying a specific territory (Oxford languages dictionary) | Denmark, USA, Honduras |
Health systemb | An organization that includes at least one hospital and at least one group of physicians that provide comprehensive care (including primary and specialty care) who are connected with each other and with the hospital through common ownership or joint management (U.S. Agency for Healthcare Research and Quality’s Compendium of US Health Systems, 2016) | National Health System (UK), Kaiser Permanente (USA) | |
Regionc | An area, administrative district, or division of a country with definable characteristics or boundaries | District, county, parish, province, state | |
Locality | An area or neighborhood that constitutes a subdivision of a larger social or political entity | Neighborhood, city, town, census tract | |
Inner setting / Inner context | Organization/agency | A group of people arranged within a formal legal structure (e.g., for-profit or not-for-profit) for the specific purpose of delivering healthcare. Exercises authority over departments/wards, clinics/practice sites, program/units, clinical teams, and/or providers | Hospital network; multistate organization delivering mental health services |
Department/ward | A subdivision of an organization with a specialized purpose, capacity, or workgroup(s) | Oncology unit within a hospital | |
Clinic/practice/site | A single, specific, geographically distinct location in which, or from which, providers deliver health-related or behavior change interventions to a target population or populations. Serves as the operational center of programs/units, clinical teams, and/or providers | Primary care practice, outpatient mental health clinic, school | |
Program/unit | A group of providers or clinical teams that deliver a specific, clearly defined health-related or behavior change intervention to a target patient population; programs are sub-units of higher-level entities such as clinics or organizations | Pediatric oncology ward in a children’s hospital | |
Individuals involved / Individual characteristics | Clinical team | One or more providers (including dyads) who directly co-deliver, or coordinate or share responsibility for delivery of, a health-related or behavior change intervention to a patient or patient population | Assertive Community Treatment team, Multisystemic Therapy team |
Provider | A single individual who delivers a health-related or behavior change intervention to the target patient | Clinician, prescriber, home visitor, peer-to-peer specialist, volunteer | |
Patientd | The individual, or smallest group of individuals, targeted by the clinical intervention or policy to be implemented | Child with depression, adult experiencing HIV, family group | |
Process / timing within EPIS Phases | Observation/time | A single, specific point in time at which an implementation-relevant measurement is taken for either research or clinical purposes | Baseline status, posttreatment status, 3-month follow-up status |