Advertisement

Improving the Replication Success of Evidence-Based Interventions: Why a Preimplementation Phase Matters

      Abstract

      Purpose

      Careful scrutiny of the literature reveals that the preimplementation phase is often overlooked by researchers interested in understanding the portability of evidence-based interventions to other settings. In this paper we document the importance of preimplementation and the planning year in enabling adopters to identify and resolve potential implementation barriers.

      Methods

      Roger's diffusion of innovation theory and tenets of technology transfer models are the heuristic frameworks used to guide the analysis of the preimplementation phase of an abstinence replication study.

      Results

      The planning year allowed for the securing and consolidation of stakeholder support; preparing the organization for implementation; redressing issues with the intervention packet; responding to the cultural norms of the adopting community; fine tuning the training approach and addressing emergent challenges.

      Conclusions

      Preimplementation provides adopters with opportunities to test the intervention before full implementation; and to identify and address potential threats to successful adoption.

      Keywords

      Implications and Contribution
      This paper demonstrates the importance of preimplementation and planning in replication studies. The preimplementation phase provides adopters with opportunities to make the necessary organizational changes; solidify stakeholder support and ensure that intervention packets are culturally consonant and medically accurate before full implementation.
      Approaches to the prevention of adverse sexual health outcomes for youth are increasingly based on the adoption and implementation of evidence-based interventions (EBIs). EBIs are programs with a strong base in both theory and scientific research that can be transferred to settings other than the ones in which their efficacy was established. However, replicating EBIs is often challenging. The many difficulties faced by adopters include balancing fidelity with adaptation, tailoring the intervention to meet the unique cultural needs of the implementing context and prospective participants, garnering and sustaining support for the intervention, and adjusting organizational structures to accommodate the specific requirements of the replicated program. In one study of organizations implementing HIV EBIs, Veniegas, Kao, Rosales, and Arellanes (2009) [
      • Veniegas R.C.
      • Kao U.
      • Rosales R.
      • Arellanes M.
      HIV prevention technology transfer: Challenges and strategies in the real world.
      ] show that organizations encounter challenges at almost every phase of the process: preimplementation, implementation, and maintenance and evolution. Thus, given these well-documented challenges in attaining high-quality replication, researchers and theorists argue for sound planning and the ability to test the intervention before moving into full implementation [
      • Kegeles S.M.
      • Rebchook G.M.
      • Hays R.B.
      • et al.
      From science to application: The development of an intervention package.
      ,
      • Lewis K.M.
      • Lesesene C.A.
      • Zahniser S.C.
      • et al.
      Developing a prevention synthesis and translation system to promote science-based approaches to teen pregnancy, HIV and STI prevention.
      ,
      • Eke A.N.
      • Neumann M.S.
      • Wilkes A.L.
      • Jones P.L.
      Preparing effective behavioral interventions to be used by prevention providers: The role of researcher during HIV research trials.
      ]. In fact, technology transfer models and theories on the diffusion of innovation—two conceptual approaches that are prevalent in the replication literature—emphatically underscore the importance of the preimplementation phase for subsequent implementation success [
      • Kraft J.M.
      • Mezoff J.S.
      • Sogolow E.D.
      • et al.
      A technology transfer model for effective HIV/AIDS interventions: Science and practice.
      ,
      • Rogers E.M.
      Diffusion of innovations.
      ,
      • Greenberg M.R.
      The diffusion of public health innovations.
      ].
      Most research on implementing EBIs focuses on issues related to adaptation and implementation fidelity during full implementation [
      • Morrison D.M.
      • Hoppe M.J.
      • Wells E.A.
      • et al.
      Replicating a teen HIV/STD preventive intervention in a multicultural city.
      ,

      Solomon J. Fidelity and adaptation challenges in replication and going to scale. NIH conf understanding reducing health disparities: Contrib behav soc sci. Bethesda, MD: October 23–24, 2006.

      ]. Fewer studies aim to understand whether successfully navigating the planning and “trialability” phases facilitates successful replication [
      • Fagan A.A.
      • Mihalic S.
      Strategies for enhancing the adoption of school-based prevention programs: Lessons learned from the blueprints for violence prevention replications of the life skills program.
      ]. In this article, we describe how the preimplementation phase was critical in identifying and redressing potential barriers to the successful replication of Promoting Health Among Teens-Abstinence Only (PHAT-AO).
      PHAT-AO is one of 31 EBIs identified by the U.S. Department of Health and Human Services (HHS) in the Pregnancy Prevention Research Evidence Review [

      U.S. Department of Health and Human Services: Office of Adolescent Health. TPP Replication of Evidence-Based Programs. 2013. Available at: http://www.hhs.gov/ash/oah/oah-initiatives/tpp.

      ], an independent systematic review of the evidence base of programs aimed at reducing teen pregnancy, sexually transmitted infections, and associated sexual risk behaviors. The review focuses on program models that demonstrate a statistically significant reduction (p < .05) in sexual activity, risky sexual behavior, sexually transmitted disease infection rates, or teen pregnancy rates [

      U.S. Department of Health and Human Services: Office of Adolescent Health. TPP Replication of Evidence-Based Programs. 2013. Available at: http://www.hhs.gov/ash/oah/oah-initiatives/tpp.

      ]. Thirty-one program models that met the rigorous HHS evidence review criteria were identified in the initial review. In 2010, through a competitive cooperative agreement process, the Office of Adolescent Health (OAH) awarded $75 million to 75 organizations to replicate these interventions in settings around the country as part of the Teen Pregnancy Prevention (TPP) Program.
      PHAT-AO is an 8-hour abstinence-only intervention that was originally developed with African-American youth in Philadelphia, Pennsylvania. PHAT-AO has demonstrated evidence of reducing sexual intercourse and the subsequent negative health consequences associated with early initiation of sex, including unintended pregnancy and sexually transmitted infection acquisition. PHAT-AO is based on the theory of planned behavior and is thus designed to (1) increase participant knowledge; (2) strengthen behavioral beliefs about the benefits of abstinence; and (3) increase the skills needed to negotiate abstinence. The intervention is being replicated and evaluated as a randomized controlled effectiveness study with sixth and seventh graders in Yonkers, New York. This program model was selected because of its compatibility with the adopting organization's values, culture, and mission. Furthermore, epidemiologic and behavioral data for the targeted youth suggested a strong need for prevention services. For example, the birth rate for girls under the age of 20 years residing in Yonkers was 43.3 per 1,000, three times higher than the county's rate [

      Westchester County Community Health Assessment. 2010. Annual data Book vital Statistics Report. Westchester county Department of health. Available at: http://health.westchestergov.com/images/stories/pdfs/databook_vital_statistics2010.pdf.

      ]. To date, since the first year of full implementation, over 800 students have participated in the intervention. Students in the treatment group receive 8 hours of instruction on the benefits of abstinence and how to negotiate abstinence. In comparison, the control group students receive an 8-hour health-promotion intervention, which focuses on behaviors related to diet, exercise, drugs, and alcohol consumption. Both programs are delivered on two consecutive Saturdays.

      Importance of the preimplementation stage in replicating evidence-based interventions

      Theories on the diffusion of innovations have been applied to understand how innovative ideas and practices in public health, most notably within the field of HIV prevention, are disseminated and effectively implemented [
      • Bowen S.K.
      • Saunders D.L.
      • Hussey J.
      • et al.
      Assessing levels of adaptation during implementation of evidence-based interventions: Introducing the Rogers-Rutten Framework.
      ,
      • Collins C.
      • Harschbarger V.
      • Sawyer R.
      • Hamadallah M.
      The diffusion of effective behavioral intervention projects, development, implementation and lessons learned.
      ]. Theories on the diffusion of innovations also have been used to form the theoretical basis for pregnancy prevention programs. Diffusion theories describe five identifiable stages in the adoption process, one of which is the trialability phase [
      • Rogers E.M.
      Diffusion of innovations.
      ]. Trialability refers to the ability for potential adopters to test the innovation before full adoption and is considered a critical deciding factor in the adoption process. Similarly, technology transfer models, which Kraft, Mezoff, Sogolow, Neumann, and Thomas (2000) defined as “the translation, dissemination, and acquisition of information about interventions, the process of deciding whether to use that intervention, the tailoring of the intervention and the provision of training and technical assistance to providers for planning and implementation” (p 8), stress the need for preimplementation [
      • Kraft J.M.
      • Mezoff J.S.
      • Sogolow E.D.
      • et al.
      A technology transfer model for effective HIV/AIDS interventions: Science and practice.
      ]. The preimplementation phase allows for the EBI to be tested in a new setting before full implementation and facilitates evaluations of how well the intervention fits the new context, the changes that may be necessary for successful replication, and potential constraints or barriers to full implementation. Preimplementation is considered a phase in which learning about the intervention can occur and adaptations and adjustments can be made; preimplementation thus allows for more purposeful and informed planning for full implementation. Regarding the TPP Program, the funder requires a planning and piloting year. In this particular replication project, the first year was used to test the intervention on a limited basis with youth in three of the target schools and two community sites. The following sections describe the major issues that occurred during the first year and how we were able to address them to ensure that the program would be implemented with increased fidelity.

      Securing and consolidating stakeholder support

      Generally, the planning year provides an opportunity for adopters to mobilize and consolidate interest and support for implementing an EBI. Without such support, implementation may be thwarted. However, as we learned in this particular study, stakeholder support and interest are not invariant over the life of an intervention. They can be easily eroded by myriad factors. For example, the school district's central office was initially a key supporter of adopting PHAT-AO and collaborated on applying for grant funds. However, shortly after being awarded a cooperative agreement grant by OAH, the primary liaison between the district and the adopting organization retired. Support for the intervention waned, and the district's institutional review board rescinded, in writing, the permission to replicate the study with students enrolled in public schools. The reasons varied, but two issues were of concern. First, the intervention's curriculum touched upon a number of sensitive health topics and the evaluation tools specifically included questions about contraceptive usage and oral sex. These topics were thought to be too mature for students in sixth and seventh grade, who were likely between 10 and 12 years old. Second, there were fears of a possible legal backlash against the district if the intervention were to proceed. Thus, it became challenging to balance the tension between implementing a strict replication project and adhering to the specific concerns of the school administration.
      To resolve the impasse, the adopting agency's executive director emerged as the program's champion; she courted the support of the city's political establishment, the school district, civic groups, and the model's developers. Her long-standing relationship with the school district and years of service to the community created a platform for bringing the key stakeholders together and successfully mediating the conflicting differences so that the replication could proceed. Equally important, the adopting agency was able to formalize a mechanism (regularly scheduled meetings) for communicating with the district once full implementation began. In anticipating that the EBI would be controversial because of the issues discussed previously, creating channels of communication with the district's central office was critical so that problems could be quickly resolved.
      As another challenge, the school district's reorganization adversely affected the number of youth who were potentially eligible for inclusion in the intervention. This reorganization involved a reconfiguration in the grade level composition of potential schools for study. The planning/preimplementation year gave us the time to find and develop new partnerships to meet the targeted numbers of youth to be served. Specifically, a series of meetings and presentations were made to the heads of the parochial schools, who subsequently agreed to participate in the replication study. Finally, the year for planning allowed collaborative relationships with other key strategic partners to be solidified. For instance, we were able to successfully fashion a partnership with New York Medical University that permitted its graduate students to serve as the intervention's facilitators.

      Preparing the organization for implementation

      Fixsen, Naoom, Blasé, Friedman, and Wallace (2005) [
      • Fixsen D.
      • Naoom S.F.
      • Blasé K.A.
      • et al.
      Implementation research: A synthesis of the literature.
      ] posit that organizational structures and processes have a determinative influence on how well the core elements of an EBI are implemented. Therefore, an important question to be answered was, “What planned organizational changes by the adopting agency, if any, were necessary for full implementation?” This particular project was fairly complex, because it involved implementing the intervention at multiple sites with a low youth-to-facilitator ratio (8:1). The organization required major restructuring for optimal implementation to ensure a good fit with the EBI. The preimplementation phase allowed for this restructuring to occur, the effectiveness of the restructuring in terms of furthering the goals of a successful full implementation to be assessed, and any further organizational adjustments to be made. For example, to accommodate the EBI, the staff had to be quadrupled, with new positions created and additional office space secured. With an expanded staff, the organization changed from a relatively flat structure to a more hierarchical and functionally differentiated structure, with the addition of a middle management supervisory layer. Two lead facilitator positions were added: one responsible for supervising the delivery of the abstinence curriculum and another responsible for the health curriculum. At the end of the planning year, the results from a performance assessment unveiled concerns about some staff members' capacity to effectively supervise the facilitators responsible for delivering the intervention. To avoid compromising the implementation of the program, the staffing positions had to be realigned with the Executive Director assuming the project director's role.

      Readiness for dissemination: information/intervention packaging of the EBI

      Numerous studies demonstrate the importance of creating intervention packages that are ready for dissemination [
      • Kegeles S.M.
      • Rebchook G.M.
      • Hays R.B.
      • et al.
      From science to application: The development of an intervention package.
      ,
      • Fixsen D.
      • Naoom S.F.
      • Blasé K.A.
      • et al.
      Implementation research: A synthesis of the literature.
      ]. Moreover, some authors recommend that intervention developers work closely with the field in developing these packets. Although one would expect such packets to predate the adoption of an intervention program, the preimplementation stage is useful for assessing how well materials and information about the intervention are packaged for end users. The opportunity to use the first year to pilot the intervention allowed for both the developer and the implementing agency to discover and remediate major issues with the intervention package. For example, the funder's medical accuracy review of the curriculum determined that the package contained information about HIV/AIDS and other health facts that were not medically accurate or current. Both the developer and the adopting site used the preimplementation year to revise the curriculum so that it was medically accurate. The intervention packet was also missing critical/core materials for the control curriculum and contained videos that were outdated. Because these discoveries were made during preimplementation, the developer had time to make corrections before the launch to full implementation.

      Responding to cultural norms and differences through adaptations and modifications

      Communities and families vary regarding their normative structures in general and, and more particularly, in their mores about sexual behaviors and what constitutes appropriate information to be shared with youth at different stages in their development. Most EBIs thus invariably undergo some degree of adaptation and modification during implementation as adoption contexts are likely to differ. Preimplementation permits implementers to work through potential implementation barriers arising from differences in cultural norms and intervention settings. With the PHAT-AO replication, a number of issues dealing with cultural responsiveness were resolved during the planning year: First, facilitators were provided with special training regarding the history of the community, one of which had been marked by a long and bitter legal battle over desegregation, the scars of which still lingered. Second, the initial evaluation design called for using race as one of the stratifying variables in the randomization procedure. However, race was excluded from the design at the request of the school district in deference to the city's racial history. Third, adaptations were made to non-core curriculum elements, including some video content, music, and character names in role-plays, because youth in the pilot phase were unable to connect with the characters portrayed in the videos. Working in concert with the developers, in-house complementary materials were substituted for the culturally dissonant ones that came with the original intervention package. Fourth, a Spanish translation was provided in the parent-child homework assignments to facilitate effective youth interaction for predominantly Spanish-speaking parents. Fifth, some evaluation tools (e.g., the student questionnaire) were revised. Because of concerns about the level of sexual maturity related to topics such as condom usage and oral sex, several items on these topics were deleted from the final instrument. In addition, based on the district's concerns about the instrument and at the district's urging, parents were given the option to review the instrument prior to providing informed consent. Sixth, elicitation research with youth in the pilot led to changes that addressed their concerns about the program. Specifically, the youth complained about the early start of the program day (which was subsequently changed to a later start during implementation) and the lack of “fun activities” (which was resolved by instituting ice breakers). Finally, to improve the enrollment rates among the eligible youth, staff members with cultural backgrounds similar to those of the youth were hired as recruitment and retention specialists.

      Training staff to deliver the intervention

      The planning year provided an opportunity to develop and test the most efficacious approach for building the facilitators' capacity to deliver the intervention. The initial model was based on turnkey training. Lead staff members at the adopting agency were first trained on how to deliver the curriculum. This training was conducted by the intervention developer's team. The lead staff, in turn, provided a 24-hour intensive training session for the facilitators. The training focused on providing facilitators with an overview of the theoretical basis of the program, outlining the delivery model, and giving each facilitator a chance to teach a section of the curriculum to their peers in a simulated intervention group model. Lead staff trainers at the agency observed the practice sessions using an internally developed observation tool and provided immediate feedback to the facilitators. At the end of the training, the facilitators completed a training evaluation that assessed their preparedness to deliver the curriculum. The evaluation was analyzed internally, and key problem areas were identified to shape future training and staff development sessions. Facilitators identified the following areas as particularly challenging: (1) reading from the manual while maintaining an interactive and engaging environment; (2) completing all activities in each module within the allotted time; (3) anticipating questions from students and responding with answers that do not compromise fidelity to the intervention's message. In addition to these concerns, the facilitators observed that classroom management was a key issue.
      The training model was redesigned accordingly. Specifically, the following changes were made: (1) coaching on classroom management was provided by a veteran educator; (2) workshops on understanding teens were provided by outside teen health experts; (3) workshops on the scientific method and the importance of fidelity in replication studies were provided; and (4) peer-led workshops were offered.

      Unanticipated emergent challenges

      The preimplementation phase also allows adopters to confront and address unanticipated challenges that could later adversely affect implementation. Similar to the original intervention model, the PHAT-AO replication was delivered on a Saturday. However, the replication design allowed parents to select the date and site that was most convenient for the family. During the pilot, because of transportation problems, some youth were not able to attend the program, despite their parents' consent for them to do so. Thus, transportation to and from the program site became an emergent need. To ensure adequate enrollment, the adopting agency was compelled to provide busing for students who lived more than one mile from the intervention site. The adopting agency was able to procure busing services with the transportation company that the school district used and at the district's rate. This not only saved the adopting agency valuable funds but also strengthened the collaboration between the adopting agency and the school district administration, which would prove vital to the continuation of the program.

      Lessons learned

      Although the rate of teen pregnancies in the United States is declining, the effective dissemination of proven practices and interventions in the field will continue to be a public health priority. However, the facilitative mechanisms for achieving successful replication remain a conundrum for providers, policy makers, researchers, and intervention developers. Although no two social contexts will be exactly the same and not all eventualities can be predicted, the ability to anticipate and plan for challenges to minimize suboptimal results during replication is nevertheless a useful and valuable implementation strategy. To that end, the preimplementation phase provides adopters with the opportunity to learn about an intervention by testing it in the adopting context and to identify and address potential implementation barriers before moving to full implementation. The insights gleaned from the preimplementation year in the present study provide several instructive lessons.
      Situating the EBI in the Prospective Context: Potential adopters should be cognizant of how contextual factors can affect the successful replication of EBIs. Contexts are dynamic and fluid social systems, and adopters should be prepared to respond to changes when they occur. For example, earlier in the paper, we noted that although the school district had provided support in advance of funding, changes in the district's personnel threatened to jeopardize our ability to move forward with the study. The ability to understand the community context and to use that knowledge to build support from other civic and political groups helped save the study. Thus, when implementing EBIs that may not be welcomed by some in the community, building a broad base of support among key stakeholders is important. Indeed, successful replication requires ongoing collaboration between adopters, key stakeholders, and developers at every stage of the process. As Fixsen, Naoom, Blasé, Friedman, and Wallace (2005) note, implementation is a process, not an event, and every step of the process requires diligent attention to maximize success [
      • Fixsen D.
      • Naoom S.F.
      • Blasé K.A.
      • et al.
      Implementation research: A synthesis of the literature.
      ]. The cultural and normative structures of the prospective setting are equally important contextual variables. The successful implementation of an EBI is partially contingent upon how consonant it is with the values and mores of the community. Thus, the challenge for implementers is to balance implementation fidelity with adaptations that are sensitive and appreciative of the community's culture.
      Organizational Flexibility and Agency: Most EBIs require some degree of internal organizational changes by the adopting organization. Organizations must be ready to make these changes to accommodate the EBI. Moreover, organizations should have mechanisms in place during the planning year to assess the effectiveness of these changes and should plan other activities germane to ensuring the success of full implementation. We reported that findings from an assessment of organizational structure changes that were made to accommodate PHAT-AO resulted in further modifications to the staffing plan. Implementing an EBI also requires a certain degree of organizational nimbleness, that is, the ability to respond quickly and effectively to unexpected emergent needs. This need became apparent when transportation problems prevented youth from participating in the PHAT-AO intervention.
      Assessing Dissemination Readiness: Identifying an intervention as effective does not necessarily reflect whether the intervention is packaged and ready for dissemination to the field. The preimplementation phase gave us a critical opportunity to assess whether the selected intervention was applicable to our specific social context. We found that the intervention was not fully prepared for dissemination. Thus, before choosing an evidence-based model for implementation, organizations should engage in exploration to determine the implementation readiness of the program and the quality of the implementation materials. Moreover, for prevention interventions with efficacy trials that are more than a decade old, adopters/programmers should review the intervention packages for currency, medical accuracy, and relevance.
      In summary, because replication is such an important element in diffusing effective evidence-based teen pregnancy prevention programs, studies such as the present one can be instrumental in helping to clarify the factors that contribute to successful adoption, implementation, and ultimately replication by highlighting the importance of preceding full implementation with strong upfront planning.

      Funding sources

      This publication was made possible by Grant Number TP1AHA000032 from the Office of Adolescent Health.

      References

        • Veniegas R.C.
        • Kao U.
        • Rosales R.
        • Arellanes M.
        HIV prevention technology transfer: Challenges and strategies in the real world.
        Am J Public Health. 2009; 99: S124-S130
        • Kegeles S.M.
        • Rebchook G.M.
        • Hays R.B.
        • et al.
        From science to application: The development of an intervention package.
        AIDS Educ Prev. 2000; 12: 62-74
        • Lewis K.M.
        • Lesesene C.A.
        • Zahniser S.C.
        • et al.
        Developing a prevention synthesis and translation system to promote science-based approaches to teen pregnancy, HIV and STI prevention.
        Am J Community Psychol. 2012; 50: 553-571
        • Eke A.N.
        • Neumann M.S.
        • Wilkes A.L.
        • Jones P.L.
        Preparing effective behavioral interventions to be used by prevention providers: The role of researcher during HIV research trials.
        AIDS Education Prev. 2008; 18: 44-58
        • Kraft J.M.
        • Mezoff J.S.
        • Sogolow E.D.
        • et al.
        A technology transfer model for effective HIV/AIDS interventions: Science and practice.
        AIDS Education Prev. 2000; 12: 7-20
        • Rogers E.M.
        Diffusion of innovations.
        4th edition. Free Press, New York1995
        • Greenberg M.R.
        The diffusion of public health innovations.
        Am J Public Health. 2006; 96: 209-210
        • Morrison D.M.
        • Hoppe M.J.
        • Wells E.A.
        • et al.
        Replicating a teen HIV/STD preventive intervention in a multicultural city.
        AIDS Education Prev. 2007; 19: 258-273
      1. Solomon J. Fidelity and adaptation challenges in replication and going to scale. NIH conf understanding reducing health disparities: Contrib behav soc sci. Bethesda, MD: October 23–24, 2006.

        • Fagan A.A.
        • Mihalic S.
        Strategies for enhancing the adoption of school-based prevention programs: Lessons learned from the blueprints for violence prevention replications of the life skills program.
        J Community Psychol. 2003; 31: 235-253
      2. U.S. Department of Health and Human Services: Office of Adolescent Health. TPP Replication of Evidence-Based Programs. 2013. Available at: http://www.hhs.gov/ash/oah/oah-initiatives/tpp.

      3. Westchester County Community Health Assessment. 2010. Annual data Book vital Statistics Report. Westchester county Department of health. Available at: http://health.westchestergov.com/images/stories/pdfs/databook_vital_statistics2010.pdf.

        • Bowen S.K.
        • Saunders D.L.
        • Hussey J.
        • et al.
        Assessing levels of adaptation during implementation of evidence-based interventions: Introducing the Rogers-Rutten Framework.
        Health Education Behav. 2010; 37: 815https://doi.org/10.1177/1090198110366002
        • Collins C.
        • Harschbarger V.
        • Sawyer R.
        • Hamadallah M.
        The diffusion of effective behavioral intervention projects, development, implementation and lessons learned.
        AIDS Education Prev. 2006; 18: 5-20
        • Fixsen D.
        • Naoom S.F.
        • Blasé K.A.
        • et al.
        Implementation research: A synthesis of the literature.
        National Implementation Network. University of South Florida, Tampa, FL2005