Advertisement

Implementing Three Evidence-Based Program Models: Early Lessons From the Teen Pregnancy Prevention Replication Study

      Abstract

      Purpose

      This article describes some of the early implementation challenges faced by nine grantees participating in the Teen Pregnancy Prevention Replication Study and their response to them.

      Methods

      The article draws on information collected as part of a comprehensive implementation study. Sources include site and program documents; program officer reports; notes from site investigation, selection and negotiation; ongoing communications with grantees as part of putting the study into place; and semi-structured interviews with program staff.

      Results and Conclusions

      The issues faced by grantees in implementing evidence-based programs designed to prevent teen pregnancy varied by program model. Grantees implementing a classroom-based curriculum faced challenges in delivering the curriculum within the constraints of school schedules and calendars (program length and size of class). Grantees implementing a culturally tailored curriculum faced a series of challenges, including implementing the intervention as part of the regular school curriculum in schools with diverse populations; low attendance when delivered as an after-school program; and resistance on the part of schools to specific curriculum content. The third set of grantees, implementing a program in clinics, faced challenges in identifying and recruiting young women into the program and in retaining young women once they were in the program. The experiences of these grantees reflect some of the complexities that should be carefully considered when choosing to replicate evidence-based programs. The Teen Pregnancy Prevention replication study will provide important context for assessing the effectiveness of some of the more widely replicated evidence-based programs.

      Keywords

      Implications and Contribution
      Is it possible to address the needs of program participants without compromising essential elements of evidence-based programs? Experiences of grantees replicating evidence-based programs with fidelity while, at the same time, working to ensure that the interventions meet the needs of their target populations offer important lessons for policymakers and service providers.
      The Teen Pregnancy Prevention (TPP) program, administered by the Office of Adolescent Health (OAH), seeks to address high rates of teen pregnancy by (1) replicating evidence-based program models and (2) testing innovative approaches. Funding was structured to maximize investment in programs with strong evidence of effectiveness (Tier 1: Replication grants) while providing support for new approaches that could add to the existing knowledge base (Tier 2: Research and Demonstration grants). Funding for both types of grantees was accompanied by requirements for evaluation activities. All grantees in both groups are required to assess and report on the fidelity with which they are implementing the program model. Adaptations are allowed with OAH approval, providing they do not compromise any of the program's core components.
      The TPP Replication Study, funded through OAH and the Office of the Assistant Secretary for Planning and Evaluation, is designed to supplement existing evaluation efforts by focusing on replication and expanding on the grantee-led evaluations that others are implementing. For this study, three program models were selected, with three replications of each model, to address the question “Do replications of evidence-based models, implemented with fidelity, produce impacts on teen pregnancy and births to teens, across different settings and populations?”

      The replication challenge

      Efforts to replicate an evidence-based program in real-world settings bring into sharp focus the tension between adherence to the core elements of the program (fidelity) and adaptation in response to implementation challenges. Some prevention researchers believe that adaptation always reduces the effectiveness of the program model [
      • Elliot D.S.
      • Mihalic S.
      Issues in disseminating and replicating effective prevention programs.
      ]. Berkel et al. (2011) [
      • Berkel C.
      • Mauricio A.M.
      • Schoenfelder E.
      • Sandler I.W.
      Putting the pieces together: An integrated model of program implementation.
      ] cite other researchers in support of the idea that adaptation can be defined as addition to the program model, rather than lack of fidelity. They suggest that this might help in distinguishing what might be a positive addition to the program from an inability to implement the program with fidelity to the original design. Because the context in which the program is delivered changes over time and may be offered to populations that differ in important ways from those on which it was originally tested, adaptation may be necessary to preserve program effectiveness [
      • Castro F.G.
      • Barrera M.
      • Martinez C.R.
      The cultural adaptation of prevention interventions: Resolving tensions between fidelity and fit.
      ,
      • Rogers E.M.
      Diffusion of innovations.
      ].
      Bumbarger and Perkins (2008) argue that, when evidence-based programs are replicated in natural conditions, complete fidelity is unlikely and that much of the adaptation is not intentional (innovation, a positive addition that strengthens the intervention without eroding its core elements) but rather reactive in response to barriers [
      • Bumbarger B.K.
      • Perkins D.F.
      After randomized trials: Issues related to dissemination of evidence-based interventions.
      ]. Their research provides some support for this contention. In a survey of organizations participating in the Pennsylvania Commission on Crime and Delinquency initiative, almost 40% of the respondents reported that they had not implemented the program model as it was originally designed, and in more than half of these, the adaptations could reasonably be expected to reduce program effectiveness. Examples included shortening or deleting lessons and changing pedagogic strategy, both in response to unanticipated barriers, and both directly affecting core program elements. In contrast, Greenberg and et al. (2005) [
      • Greenberg M.T.
      • Domitrovich C.E.
      • Graczyk P.A.
      • Zins J.E.
      The study of implementation in school-based preventive interventions: Theory, research and practice.
      ] suggest that adaptation can coexist with fidelity. OAH's emphasis on fidelity of implementation required grantees to find that balance and to develop solutions to unanticipated challenges that enhanced rather than eroded the program model.

      The TPP replication study

      The 5-year federally funded TPP Replication Study evaluation includes an impact evaluation study and a comprehensive implementation study. The evaluation incorporates nine rigorous experimental evaluations of grants awarded by OAH in 2010. Three of the nine grantees selected for the study are replicating Reducing the Risk, a widely used curriculum-based sexuality education program, whose 16 sessions are usually delivered in schools with students aged 14–19 years. Three other grantees are replicating ¡Cuídate!, an HIV/AIDS prevention program, culturally tailored to Latino adolescents aged 13–19 years and delivered over six sessions in small groups that may be either single sex or mixed gender. The third set of grantees are replicating Safer Sex, a clinic-based program to prevent sexually transmitted disease (STD) that targets sexually active females aged 14–19 years. The program is delivered individually to participants by a trained health educator using a motivational interviewing process. Participants complete an initial 1-hour session and then three subsequent booster sessions over a 6-month period. Table 1 lists key characteristics of the grantee, the model tested, and the population targeted.
      Table 1Key features of program replications in the evaluation by program model and replication site
      Program model, granteeProgram descriptionStudy locationTarget populationProgram duration and intensityProgram settingProgram delivered by
      AgeDemographics (from proposal description)
      Reducing the Risk
      This study found no effects after 6 months, but after 18 months, female, but not male, adolescents in the program who were sexually inexperienced at baseline were significantly less likely to report having had unprotected sex. No significant effects were found on sexual initiation, recent sexual activity, or pregnancy [7].
      Sexual health and risk prevention curriculum delivered to groups in schools or in community settings13 high schools throughout CA (46 classes)High school students62% white, 20% Hispanic, 9% Asian, 2% African-American, 2% Native American16 45-minute sessions, which can be doubled upHigh schoolsTeachers
      Better Family LifeSt. Louis and East St. Louis, MONinth graders98% African-American; low SES (75% eligible for free/reduced-price lunch in St. Louis City); high risk for teen births and STDs16 sessions delivered over 8–16 weeks, depending on school scheduleNoncore classes in six high schoolsHealth educators trained and employed by BFL
      LifeWorksAustin, TXNinth graders (with small numbers of tenth and eleventh graders)75% minority youth, almost all below poverty level; teen pregnancy rates are increasing (37 pregnancies per 1,000 female high school students in 2008–2009); high rate of STDs16 sessions delivered over 8 weeksHealth classes in four high schoolsHealth educators trained and employed by Planned Parenthood (grant partner)
      San Diego Youth ServicesSan Diego County, CANinth graders (one school with eighth graders)Very diverse population; youth at risk for involvement with the juvenile justice system or mandated to receive services by a judge or probation officer; “teen pregnancy hotspots” identified by the state16 sessions delivered over 8–16 weeks depending on school schedulePE/health classes in seven high schoolsHealth educators trained and employed by five agency grant partners
      San Diego Youth ServicesSan Diego County, CAYouth 13–19 years of age enrolled in community agency programs (some diversion by juvenile justice system)Very diverse population; youth at risk for involvement with the juvenile justice system or mandated to receive services by a judge or probation officer; “teen pregnancy hotspots” identified by the state16 sessions delivered over 2–3 weeksFive community agenciesHealth educators trained and employed by five agency grant partners
      ¡Cuídate!
      This study found that adolescents in the program were significantly less likely to report having had sexual intercourse and multiple partners in the previous 3 months; they reported significantly fewer days of unprotected sex and more consistent condom use. No significant effects were found on condom use at last sex or the proportion of days of sexual intercourse that were condom protected [8].
      HIV/AIDs prevention program for small groups with emphasis on Latino cultural valuesSaturday program serving neighborhoods in northeast PhiladelphiaAdolescents 13–18 years of age, mixed genderAll Latino, 85% Puerto RicanSix 1-hour sessions that can be delivered over 2 days to 6 weeksAfter-school programs or community-based organizationsTrained facilitators
      Touchstone Behavioral HealthApproved adaptation to deliver in classes of 20–24 students with two facilitatorsPhoenix, AZEighth graders61% Hispanic, 29% white, 7% African-American; 18.5% below Federal poverty lineApproved adaptation added one session on pregnancy prevention.

      Seven sessions once a week for 7 weeks
      Noncore classes in 10 middle schoolsFacilitators trained and hired by TBH
      La Alianza HispanaBoston, Chelsea and Lawrence, MANinth graders (some 10th and 11th graders)62%–78% Hispanic, 9%–20% white, .4%–25% African-American; 68%–88% free/reduced-price lunchSix sessions once a week for 6 weeksNoncore classes in two high schools, after school program in two high schoolsFacilitators trained and hired by LAH
      Community Action Program of San Luis ObispoSLO county, CANinth graders29%–47% Hispanic, 47%–64% white, 1%–3% African-American; 35%–50% free/reduced-price lunchApproved adaptation added two sessions on STDs and pregnancy prevention. Eight sessions over 8 weeksPullout sessions during school day in three high schoolsFacilitators trained and hired by CAPSLO
      Safer Sex
      This study found no effects 1 month after the program, but 6 months after the program, adolescents who participated in the program were significantly less likely to report having had another sexual partner, aside from their main partner, in the previous 6 months [9].
      HIV/AIDS prevention program for high-risk females 13–19 years of ageUrban children's hospital; adolescent clinicAdolescent females who are not pregnant49% African-American, 18% Hispanic, 14% Non-Hispanic, white; all sought treatment for an STD at health clinicInitial 1-hour face-to-face session with three 30-minute booster sessions over 6-month periodHealth clinicsFemale health educator
      Planned Parenthood of Greater OrlandoOrange County and adjacent counties, FLSexually active females 15–19 years of age, who are not pregnant72% white, 21% African-American, 25% Hispanic, 5% Asian; 41% of children living in economic hardship; high rates of STDsTwo PPGO reproductive health clinics in OrlandoHealth educators trained and hired by PPGO
      Knox County Health DepartmentKnox County and adjacent counties, TNSexually active females 14–19 years of age who are not pregnant89% white, 9% black, 19% females aged 15–19 years are Latina; poverty rates up to 34% for children under 18 years of age; many teens from high-risk situations; serve children in state custody16 reproductive health, adolescent health clinicsHealth educators trained and hired by Knox County Health Department and grant partners
      Hennepin County Health DepartmentHennepin County, MNSexually active females 14–19 years of age who are not pregnant32% African-American, 10% Latino, 46% Caucasian; large disparities in family income by race/ethnicity; sites selected for program implementation have teen birth rates approaching or exceeding the national teen birth rate20 reproductive health, adolescent health, school-based health clinicsHealth educators trained and hired by Hennepin County and grant partners
      PE = planning and evaluation; SES = socio-economic status.
      a This study found no effects after 6 months, but after 18 months, female, but not male, adolescents in the program who were sexually inexperienced at baseline were significantly less likely to report having had unprotected sex. No significant effects were found on sexual initiation, recent sexual activity, or pregnancy
      • Kirby D.
      • Barth R.P.
      • Leland N.
      • Fetro J.V.
      Reducing the risk: Impact of a new curriculum on sexual risk-taking.
      .
      b This study found that adolescents in the program were significantly less likely to report having had sexual intercourse and multiple partners in the previous 3 months; they reported significantly fewer days of unprotected sex and more consistent condom use. No significant effects were found on condom use at last sex or the proportion of days of sexual intercourse that were condom protected
      • Villarruel A.M.
      • Jemmott J.B.
      • Jemmott L.S.
      A randomized controlled trial testing an HIV prevention intervention for Latino youth.
      .
      c This study found no effects 1 month after the program, but 6 months after the program, adolescents who participated in the program were significantly less likely to report having had another sexual partner, aside from their main partner, in the previous 6 months
      • Shrier L.A.
      • Ancheta R.
      • Goodman E.
      • et al.
      Randomized controlled trial of a safer sex intervention for high-risk adolescent girls.
      .

      The implementation study

      The goals of the implementation study are as follows:
      • To provide an in-depth description of the intervention as planned and implemented in each of the replication sites for the three models;
      • To document the extent to which program models are implemented with fidelity and are able to meet their performance goals;
      • To examine barriers and challenges to implementation in each of the sites to arrive at a qualitative understanding of why replication efforts did or did not reproduce the impacts reported in the original study;
      • To identify and describe the services available to and used by youth in the control groups; and
      • To explore linking aspects of program implementation to variation in program impacts, in the event that the impact study identifies such variation.
      Underlying these goals are two questions that arise in many research fields, namely the extent to which multiple high-quality replications of a program model are feasible, and the factors internal and external to the program that affect replication.
      To guide the specification of data needed to address these goals and the research questions that flow from them, the study uses a framework that builds on the work of Berkel and et al. (op.cit.) and others, to identify aspects of implementation that have been shown to affect program outcomes, as well as the factors internal and external to the grantee that affect implementation. Figure 1 shows the proposed framework.

      Methods

      The information presented below is drawn from a much larger body of data that are being collected for the implementation study. Table 2 lists the data needs, sources, and data collection strategies for the study as a whole. Collection of these data is ongoing and will continue through spring 2014. This article relies on a small subset of this information collected early in the first implementation year. Sources of information for the article include grantee and program documents and reports; notes made by study staff during site recruitment; ongoing communication with grantee and partner staff; and semi-structured telephone interviews with program staff.
      Table 2Framework elements, data constructs and sources, and data collection strategies
      Implementation framework elements and constructsData sourcesData collection strategy
      Readiness/preparation
       Grantee and partner capacityProposal and other program documents

      Grantee and partner staff
      Review and extract information

      Conduct semi-structured interviews (1) by telephone and (2) in person
       Staff selection and preparationGrantee and partner staffConduct semi-structured interviews (1) by telephone and (2) in person
       Specificity of program modelInstructions/guidance provided by developer and OAH

      Grantee staff
      Review and extract information

      Conduct semi-structured interviews (1) by telephone and (2) in person
       Site-specific replication planProposal, annual report after pilot yearReview and extract information
       Stakeholder supportGrantee and partner staff

      Stakeholders
      Conduct semi-structured interviews (1) by telephone and (2) in person
      Implementation of the intervention
       Administrative and supervisory supportsGrantee and partner supervisory and frontline staffConduct semi-structured interviews (1) by telephone and (2) in person
       Extent to which the intervention was implemented as plannedGrantee and partner supervisory and frontline staff

      Fidelity checklists
      Conduct semi-structured interviews (1) by telephone and (2) in person

      Obtain data reported to OAH and re-analyze
       Service qualityRequired observations

      Program participants
      Obtain data reported to OAH and re-analyze.

      Focus group discussions with youth
       AdaptationsAdaptation requests submitted to OAH

      Grantee and partner supervisory and frontline staff
      Review and extract information

      Conduct semi-structured interviews (1) by telephone and (2) in person
      Community context
       Level of community riskProposal

      Local and national survey data

      Grantee and partner supervisory and frontline staff
      Review and extract information

      Conduct semi-structured interviews (1) by telephone and (2) in person
       Community resourcesGrantee and partner supervisory and frontline staffConduct semi-structured interviews (1) by telephone and (2) in person
       Participant responsivenessAttendance data

      Grantee and partner supervisory and frontline staff

      School staff

      Program sessions

      Program participants
      Obtain data reported to OAH and re-analyze.

      Conduct semi-structured interviews (1) by telephone and (2) in person

      Conduct in person semi-structured interviews

      Conduct observations during site visits

      Conduct focus group discussions
      OAH = Office of Adolescent Health.
      The information was compiled within site and then reviewed to extract model-specific themes that were identified by multiple sites implementing the model.

      Results

      Early implementation challenges

      Intervention goals and strategies varied by program model, as did the challenges encountered as grantees attempted to replicate each program model with fidelity. Below, we discuss challenges encountered and solutions devised for each of the three models.

      Reducing the Risk

      All three grantees are implementing the program in schools with eighth and ninth graders [
      • Kirby D.
      • Barth R.P.
      • Leland N.
      • Fetro J.V.
      Reducing the risk: Impact of a new curriculum on sexual risk-taking.
      ]. In addition, one grantee is delivering the program in community-based settings with youth involved in the juvenile justice system. In schools, the length of the program poses a challenge as does its suitability for younger (eighth grade) students. In community settings, the length of the program affected program delivery somewhat differently—by making it difficult to retain participants after they left the jurisdiction of the juvenile justice system. In both settings, some homework assignments present a challenge.

      School schedules and class sizes

      The curriculum is designed to be delivered in 16 lessons that last 50–60 minutes but can be compressed into eight sessions covering two lessons each. Some schools have moved to block scheduling, which makes it possible to deliver the program comfortably over one semester, even with interruptions in the school schedule, and also to cover the materials of two sessions. One grantee was able to negotiate delivery of the program twice a week, rather than on the more common once weekly schedule. Again this made it possible to fit the program into a single semester. However, the length of the class session was often 45 minutes rather than the required 50–60 minutes. This compressed schedule made it difficult to cover all the materials and activities required for the session. The fidelity checklists provided by OAH helped facilitators keep track of the materials covered, but they needed to develop strategies to move more quickly through specific sessions.
      An additional challenge was presented in some schools by the expansion of class sizes, as a consequence of budget cuts and subsequent teacher layoffs. The program is designed to be delivered in classes of approximately 15–20 students; in reality, some classes contained 40 or more students. In response, the number of facilitators was doubled in those classes.

      Age of students

      The curriculum is advertised as appropriate for youth 14–19 years of age, and, in general, facilitators found this to be correct. However, those who delivered the program to eighth graders found that students often lacked basic information about reproductive anatomy and therefore lacked the context for understanding material on contraceptive methods. Given the tight schedules that they faced, facilitators found it difficult to fit additional materials into the curriculum; they often used posters and a brief explanation at the beginning of the session.

      Retaining participants

      Retention was generally not a challenge in school settings. In community-based settings, in contrast, retaining participants once they left the jurisdiction of the juvenile justice system was a serious issue. Grantee staff used incentives to draw youth back to the program (e.g., offering door prizes, t-shirts). Each of these strategies had some limited success, but program staff acknowledged that a major problem remained because of the extreme mobility of many youth once they were no longer under the auspices of the juvenile justice system.

      Homework assignments

      An important homework assignment given to students was to identify and visit a clinic that could provide advice on contraceptives and condoms. Characteristically, this homework assignment was completed by a minority of students, either because of their own reluctance or opposition from their parents. When students did attempt the assignment, they sometimes had an unpleasant experience at the clinic (staff were busy, not receptive to the assignment)—the opposite of what the program intended. Some facilitators responded by identifying and preparing specific clinics ahead of time, to ensure a positive reception. Others are exploring the possibility of a “virtual” visit, via interactive video.

      ¡Cuídate!

      The program was originally tested in a community-based setting with almost entirely Puerto Rican Latino youth [
      • Villarruel A.M.
      • Jemmott J.B.
      • Jemmott L.S.
      A randomized controlled trial testing an HIV prevention intervention for Latino youth.
      ]. The grantees replicating this program faced the challenge of offering the program in schools that serve a variety of ethnicities without appearing to target a single ethnic group. All are implementing it in schools, usually as part of the school day. The one grantee who attempted to implement it as an after-school program faced difficulties in recruiting and retaining participants. Finally, the curriculum focused rather narrowly on HIV/AIDS prevention; the goals of the OAH initiative are broader and include prevention of other STDs as well as pregnancy prevention. As in Reducing the Risk, younger participants often lacked the knowledge of reproductive anatomy necessary to comprehend some of the materials on contraception, and a frequent adaptation request was to add material to address this issue.

      Targeting an ethnic group in school settings

      In one of the three replications, the overwhelming majority of the students in the school are Latino, so the issue of minority targeting did not arise. However, in other sites, it has been politically difficult to provide an intervention that is intended solely for Latino youth during the regular school day. The developer makes clear that the messages delivered by the program apply across ethnic groups; the program simply uses Latino cultural references and materials. The solution in one of the sites was to offer as an alternative the program from which ¡Cuídate! was derived (Be Proud! Be Responsible!). Students are free to choose either program, removing any potential stigma. However, potential implementers need to be aware that this essentially doubles the cost of curricula and training.
      Another solution tried was to offer the program as an optional after-school activity specifically for Latino youth. Although this was acceptable to schools, it proved unsatisfactory: the grantee had difficulties in recruiting youth and then in retaining them. In addition to transportation problems, many students had after-school responsibilities (e.g., caring for younger siblings). Attendance was inconsistent even for those students who initially agreed to participate, resulting in unacceptably small groups. Before implementing the program as an after-school offering, it would be important to investigate the barriers that students may face in specific schools.

      Gaps in the curriculum

      Both ¡Cuídate! and Be Proud! Be Responsible! are designed as HIV/AIDS prevention programs. The two obvious gaps in the curriculum are other STDs and pregnancy prevention. In California, the state mandates specific coverage of these two topics, so the grantee in San Luis Obispo needed to add two sessions to the original six to comply with state requirements. OAH encouraged other grantees to add a session or expand the coverage of the topics in the existing sessions.
      A less obvious gap in the curriculum is the absence of information on reproductive anatomy, which grantees found essential for any discussion of contraception, particularly with younger participants. Usually grantees solved this problem by incorporating appropriate information at the beginning of the relevant session.

      Safer Sex

      The three grantees replicating Safer Sex faced challenges identifying, recruiting, and retaining participants. The original implementation and evaluation of the program conducted by the model developer targeted females below the age of 24 who were being treated for an STD at a hospital clinic and who were required to return to the clinic three times over a 6-month period for follow-up treatment. The three booster sessions, which followed the initial session, were timed to coincide with the follow-up treatments [
      • Shrier L.A.
      • Ancheta R.
      • Goodman E.
      • et al.
      Randomized controlled trial of a safer sex intervention for high-risk adolescent girls.
      ].

      Identifying the target population

      All three of the replications of Safer Sex proposed to broaden the target population for the intervention from females diagnosed with an STD to sexually active females and to narrow the age range to 14- to 19-year-olds [

      As in the original study (Schrier et al., ibid), pregnant teens were not eligible for the intervention.

      ]. The ideal targets for services, from the developer's perspective, were young women who had not yet made the decision to change their behavior in ways that would protect them from STDs and unplanned pregnancy. However, identifying these young women in a systematic way was not an easy task. At the outset, program staff relied on clinicians to make the initial identification and referral for the program. In some instances, the requisite relationships and processes were not yet in place—buy-in on the part of the clinicians who didn't always have knowledge or understanding of how the intervention could be integrated into the array of clinic services, a way to screen out the highly motivated young women who were not the ideal targets for the intervention. Program staff worked very hard to integrate the identification and referral process into standard clinic practice and to cultivate relationships with clinic staff (a task that became easier once the intervention was up and running, and clinicians could, themselves, see a benefit). Beyond that, programs looked externally for referrals, either to existing partner organizations or to other youth serving agencies in the community.
      Once a referral was made, the health educator or program staff would begin outreach and recruitment. Over time, program staff were more comfortable with recruiting/serving the broader population, regardless of perceived motivation level at the time of enrollment. This was in part due to experience with implementing the intervention over time and the recognition that sexually active teens are highly labile and likely to switch back and forth between risky behavior and protective behavior, depending on their current sexual relationship.

      Recruiting and retaining participants

      Once a young woman was identified, there was an additional struggle to get them to participate. Young women came into the clinic seeking treatment or services, and recruitment often took place immediately following service receipt, requiring an extra time commitment (almost always unanticipated). Beyond the logistics, there was not always a perceived need on the part of potentially eligible young women, and health educators had to work hard to make the program enticing—through the use of creative media strategies and advertisements or incentives (such as condom key chains or books).
      Retaining young women in the program for the 6-month duration was also a challenge. Health educators constantly struggled with the chaotic and unpredictable schedules of adolescents, limited transportation options, and limited clinic service hours. The need for flexibility in scheduling was critical. In many instances, clinics offered extended hours. In at least one of the program sites, the use of social media and technology was an integral part of keeping participants connected to the intervention and program messages. During the initial session, health educators assessed a participant's available technology and interest in conducting booster sessions via video-conferencing (with software such as Skype, FaceTime, or other video chat and instant messaging software) and then offered that as an option if it was feasible.

      Monitoring and feedback

      Because of the individualized nature of the intervention, fidelity observations were not required by OAH, and for the most part, program staff deemed them not feasible. The grantees each relied heavily on the fidelity monitoring log as well as on the mechanisms they developed themselves for assessing the qualities and skills of the health educators. This was often through observing roleplays (using health educators, nonprogram participants, or young women who had been through the pilot and were no longer receiving the intervention).

      Summary of Lessons Learned

      The implementation experiences of this set of grantees offer a unique opportunity to learn, systematically and in depth, how these three evidence-based program models are being implemented and the challenges that are likely to be encountered by others in the field. Each of the grantees made planned adaptations to the program model almost always to improve the fit with either the population or the community. For the ¡Cuídate! programs, this took the form of adding material to address gaps in knowledge of program participants and, in some instances, modifying specific units to address concerns of key constituents (or offering an alternative curriculum). Reducing the Risk grantees modified roleplays or homework assignments to bolster the learning experience of participants. Safer Sex grantees broadened the population served, allowing clinics to offer the program to more young women. Beyond trying to improve the fit with program participants, grantees contended with internal and external constraints that influenced the ability to implement the program, such as school schedules, lack of support for or understanding of the program within the sponsoring organization, or political climate.
      The discussion here reflects some of the complexities that should be carefully considered when choosing to implement evidence-based programs and retain fidelity to the program model. Each of the grantees highlighted here struggled in some way with making the program fit, either within the community, the organization, or with program participants. In response to implementation challenges, grantees made changes to the programs that adhered to requirements for fidelity while strengthening the programs to fit different sets of needs. Investing time upfront for planning goes a long way to increasing the likelihood of success in implementing the program with fidelity. At the same time, it is important to recognize that not everything that may influence implementation can be anticipated ahead of time, and there is a need for communication and monitoring throughout the period of implementation. The information that will be generated from this evaluation effort as well as the numerous TPP program grantee-led evaluations is advancing the understanding of what it takes to implement evidence-based programs for teenage pregnancy prevention.

      Acknowledgments

      The authors thank the project team at Abt and the participating grantee project staff.

      Funding Sources

      This research is supported by the Office of Adolescent Health and the Office of the Assistant Secretary for Planning and Evaluation in the U.S. Department of Health and Human Services under contract number HHSP23320095624WC Order No. HHSP23337011T awarded in September 2011.

      References

        • Elliot D.S.
        • Mihalic S.
        Issues in disseminating and replicating effective prevention programs.
        Prev Sci. 2004; 5: 47-52
        • Berkel C.
        • Mauricio A.M.
        • Schoenfelder E.
        • Sandler I.W.
        Putting the pieces together: An integrated model of program implementation.
        Prev Sci. 2011; 12: 23-33
        • Castro F.G.
        • Barrera M.
        • Martinez C.R.
        The cultural adaptation of prevention interventions: Resolving tensions between fidelity and fit.
        Prev Sci. 2004; 5: 41-45
        • Rogers E.M.
        Diffusion of innovations.
        5th edition. The Free Press, New York1995
        • Bumbarger B.K.
        • Perkins D.F.
        After randomized trials: Issues related to dissemination of evidence-based interventions.
        J Children's Serv. 2008; 2: 53-61
        • Greenberg M.T.
        • Domitrovich C.E.
        • Graczyk P.A.
        • Zins J.E.
        The study of implementation in school-based preventive interventions: Theory, research and practice.
        Center for Mental Health Services, Substance Abuse and Mental Health Services Administration, Rockville, MD2005
        • Kirby D.
        • Barth R.P.
        • Leland N.
        • Fetro J.V.
        Reducing the risk: Impact of a new curriculum on sexual risk-taking.
        Fam Plann Perspect. 1991; 23: 253-263
        • Villarruel A.M.
        • Jemmott J.B.
        • Jemmott L.S.
        A randomized controlled trial testing an HIV prevention intervention for Latino youth.
        Arch Pediatr Adolesc Med. 2006; 160 (A program offered on a weekend day to youth recruited from schools and community settings. Non-Latino youth were not precluded from participating in the program, but they were excluded from the analysis.): 772-777
        • Shrier L.A.
        • Ancheta R.
        • Goodman E.
        • et al.
        Randomized controlled trial of a safer sex intervention for high-risk adolescent girls.
        Arch Pediatr Adolesc Med. 2001; 155: 73-79
      1. As in the original study (Schrier et al., ibid), pregnant teens were not eligible for the intervention.