If you don't remember your password, you can reset it by entering your email address and clicking the Reset Password button. You will then receive an email that contains a secure link for resetting your password
If the address matches a valid account an email will be sent to __email__ with instructions for resetting your password
The Volunteers of America Greater Los Angeles (VOALA) Girls Inc. program is implementing and rigorously evaluating its Preventing Adolescent Pregnancy curriculum as part of a demonstration grant to identify effective teen pregnancy prevention programs sponsored by the U.S. Department of Health and Human Services Office of Adolescent Health (OAH). A total of 517 participants from Title I urban middle and high schools were randomly assigned to either Preventing Adolescent Pregnancy (treatment) or Economic Literacy (control) in two cohorts. Programming occurred after school weekly at middle and high schools. Low attendance and loss of sample (attrition) are common challenges in after-school programming, negatively affecting both the ability of a program to be successful and the integrity of a randomized controlled trial. The current article discusses challenges encountered with recruitment, incentives, and school factors during a first cohort of youth and innovative implementation changes during a second cohort that resulted in increased attendance rates and decreased attrition rates. Commentary is provided by the OAH Project Officer as well as lessons learned after 2 years of implementing the program.
Implementing a randomized control trial (RCT) study as part of an after school program is daunting due to additional requirements when after school programming is already challenging because of competing obligations. Successful strategies are identified to counter common problems in implementing an RCT as part of an after school program.
After-school programming is recognized as a vital time to provide structured services to students who would otherwise be unsupervised [
].Yet, after-school programming is a challenge to implement due to many competing after-school activities. To prove an after-school program is effective through a randomized control trial (RCT) adds greatly to the existing challenge. Volunteers of America, Greater Los Angeles (VOALA)—Girls Inc. learned many lessons along the way while implementing a teenage pregnancy prevention after-school program as part of a demonstration grant for the Office of Adolescent Health (OAH), a division of the U.S. Department of Health and Human Services. The current article reviews the major problems encountered during program implementation of the VOALA-Girls Inc. curriculum as well as the lessons learned as a result of strong collaboration between the technical assistance contractor, Mathematica Policy Research (MPR), OAH, and VOALA-Girls Inc. The topics include recruitment, incentives, attendance rates, postprogram survey completion rates, and communication.
The Office of Adolescent Health (OAH) and Volunteers of America, Greater Los Angeles—Girls Inc. program share a mission of preventing teen pregnancy. Beginning in fiscal year (FY) 2010, grant funding was provided annually to VOALA-Girls Inc. for the implementation and rigorous evaluation of their innovative teen pregnancy prevention (TPP) program, Preventing Adolescent Pregnancy (PAP). The intent of the program is to provide students with accurate information regarding healthy choices on sexual relations to prevent or delay sexual activity, prevent contracting a sexually transmitted disease, and prevent becoming pregnant. The program aims to empower students with knowledge and through activities such as role playing to practice how to be assertive.
The VOALA-Girls Inc. Preventing Adolescent Pregnancy study is challenging as a randomized controlled trial (RCT) in an after-school setting [
]. Students need to fill out forms and receive consent from parents prior to program participation, which may not be required by other after-school programs. Other programs may be more flexible with attendance. Therefore, a great deal of the discussion among OAH, MPR, and VOALA-Girls Inc. involved problem solving around these important issues. The challenges and lessons learned have been documented, which will help our understanding of implementing after-school programs [
The VOALA-Girls Inc. team includes staff from VOALA, the Girls Inc. program, and Advanced Empirical Systems (AES). VOALA is a large nonprofit organization that provides needed services to over 30,000 Angelenos through various programs. Girls Inc., which is one of VOALA's programs, provides after-school services to Title I middle and high school female students across the metropolitan Los Angeles area. Title I schools are schools that receive Title I funds due to being attended by a large percentage of students who qualify for free or reduced lunches indicating a large population with financial need. AES is the third-party evaluator of Girls Inc. program implementation and effectiveness. The VOALA internal evaluation department assists in the TPP study and ensures that program fidelity occurs and that Girls Inc. meets performance expectations. OAH supports VOALA-Girls Inc.'s implementation and evaluation activities by assigning an OAH Project Officer and a federally sponsored evaluation technical assistance contractor, Mathematica Policy Research (MPR), to the project. The strength of the implementation team and project management structure is critical to the success of the project. A multidisciplinary team that is inclusive and consists of program and evaluation specialists, administration, external evaluators, organizational leadership, and frontline staff helps to assure that program monitoring along with lessons learned are taken into account with reference to overall program goals and objectives. Each representative on the implementation team contributes a different aspect to the project; more importantly, each member constructively explores their differences to find workable solutions to the problem.
At the national level, Girls Inc. is an organization that promotes the development of curricula designed to empower girls to be “strong, smart, and bold” [
]. VOALA-Girls Inc. (the Los Angeles affiliate) is evaluating the effectiveness of the Preventing Adolescent Pregnancy curriculum when implemented in an after-school format. Preventing Adolescent Pregnancy provides age-appropriate lessons on sexual health and sexuality, assertiveness, decision-making and future planning. Program specialists were trained staff with at least a bachelor's degree, preferably in the social sciences, and previous experience working with second-decade youth. The program specialists conducted weekly 90-minute sessions over an 11-week period for 5 to 20 students. Interactive activities included debates, role playing, group discussions, and games to provide participants with opportunities to learn and integrate acquired knowledge and assertive skills into their lives. The control group received Economic Literacy curriculum, which focused on financial independence, wants versus needs, budgeting, and future planning. Regardless of curriculum, each session built upon material from previous sessions; therefore, it was imperative that participants attend consistently. The study is being implemented over multiple years. Cohort 1 occurred during the 2011–2012 school year. Cohort 2 took place during the 2012–2013 school year.
For successful implementation and evaluation, VOALA-Girls Inc. must achieve consistent attendance and high completion of postprogram surveys. OAH has a number of performance measures on which all TPP grantees must report biannually, including attendance. OAH has a 75% attendance benchmark for all of their funded TPP programs [
]. If students do not receive a sufficient portion of the curriculum, program effectiveness cannot be determined due to insufficient exposure to material. In addition, postprogram surveys are necessary to determine the level of immediate growth from participating in the program. If too many students do not complete the postprogram survey, then one must question whether there are differences among those students who do and do not complete the survey. The results may not be representative of all students who were enrolled in the study. Participants received 11 weeks of the Preventing Adolescent Pregnancy curriculum in a weekly, after-school setting. During program implementation for Cohort 1, attendance was poor. Factors influencing poor attendance and postprogram-survey completion rates included school factors such as changing school staff, incentives, and program structure. For example, participant recruitment took over a month. Participants first recruited became involved with other activities by the time actual program implementation took place. Of those students who participated, many were involved in other after-school activities. Thus, not only was attendance negatively affected, postprogram-survey turnout was as well. As a result of program improvements, overall attendance and postprogram-survey completion rates improved during program implementation for Cohort 2 (all data are presented in the Methods section).
Students were recruited from urban Title I schools throughout the metropolitan Los Angeles area across multiple school districts. A total of 517 students participated across two cohorts in middle schools (sixth and seventh grades, ages 10–12 years) and high schools (ninth and tenth grades, ages 14–16 years) (Table 1). Program implementation for Cohort 1 (373 participants) occurred from October 2011 to May 2012, with 202 participants from six middle schools and 171 from six high schools. Program implementation for Cohort 2 (144 students) occurred from October 2012 to March 2013, with 95 participants from four middle schools and 49 from three high schools.
Table 1Participants by treatment and control within middle and high schools across cohorts
Third-party evaluators, namely AES, conducted the data collection. For the purpose of the evaluation, baseline data were collected the week prior to implementation of the curriculum. Post-survey data were collected following the 11 weeks of the curriculum as well as at 6- and 12-month follow-up to determine short-term and long-term program impacts on participants. For the current article, data included in the analyses will be from baseline and the postprogram surveys that occurred immediately after completion of programming.
Every 2 weeks, program specialists would meet with the program manager and internal evaluation team to review fidelity logs during a fidelity meeting. The fidelity logs kept a record of proper coverage of curriculum as well as provided the opportunity to note challenges. The fidelity meetings provided opportunities for program specialists to note observations they made during program implementation as well as provide support to each other. Information from these meetings would be taken to the monthly meetings with the major stakeholders.
Major stakeholder meetings
To ensure the program and evaluation is implemented in a manner consistent with OAH requirements and the high standards of the U.S. Department of Health and Human Services Pregnancy Prevention Research Evidence Review [
], monthly calls have been held over the past 3 years with the major stakeholders, which included the funder, its technical assistance contractor, our program staff, our implementers, and our third-party evaluator. Using real-time reports from the VOALA-Girls Inc. database, the team discussed project barriers, accomplishments, modifications, and solutions to real-world implementation and evaluation challenges. VOALA-Girls Inc. successfully monitored and documented the intervention with sufficient detail to facilitate future replication. VOALA-Girls Inc. developed a program manual to adequately document efforts testing our innovative program model. This included documenting any adjustments made during the program implementation period, significant adaptations, and the identification of the core components. Program changes based on these meetings occurred during the summer prior to the next program implementation.
Challenges of implementing an after-school program randomized clinical trial
As mentioned earlier, two important elements of a randomized clinical trial (RCT) are participant retention and low attrition rates for data collected across time points. For Cohort 1, average attendance rates for both treatment and control were 50% for high school and 48% for middle school (Table 2). The attendance rates were well below OAH's goal of 75%. Only 79% of high school students and 47% of middle school students among students who completed the preprogram survey also completed the postprogram survey.
Table 2Comparison of attendance rates and postprogram survey completion of preprogram-survey group by cohort
The original recruitment goal was 20 participants per classroom to ensure a large overall sample size to meet the evidence standards required by the grant, a total of 1,000 participants by the end of the grant. A small sample size diminishes power, which helps detect an effect [
], and effects determined to be significant could be questionable. For example, small samples can result in extreme effect sizes, either positive or negative, as a result of differing teacher styles such as one teacher being more highly experienced than another, which would balance out in a larger sample [
]. Many identified effective recruitment practices were in place, including attending Back to School Night, a parent event provided by the schools to introduce parents to their students' new teachers as well as new student orientations, which were provided by schools for new students [
]. Recruitment took place at the schools during lunch and in classrooms. Recruiters were armed with flyers and snacks as well as program registration packets. During our first year, we had some delays prior to commencing implementation. While VOALA-Girls Inc. was in the early stages of program implementation, OAH was in the early stages of determining how the study would be evaluated. One of the requirements toward the end of the first year was to have a medical review of the curriculum. Programs funded under the TPP grant needed to ensure that information provided was age appropriate, as well as scientifically and medically accurate. VOALA-Girls Inc. was required to submit all core curricula and related educational materials to the OAH for review and approval prior to use in the project. The medical review took a few months to be completed due to multiple programs' curricula needing to be reviewed. VOALA-Girls Inc. was unable to implement programming until the medical review determined the curriculum was medically accurate. We took advantage of this time to extend our recruitment period because older second-decade youth (10–16 years old) in higher needs areas are difficult to recruit into programming [
]. Those students who were already recruited began participating in presession activities. Delays to programming lasted up to 6 weeks for some school sites, resulting in long periods of presession. Although we increased our recruitment numbers during the extended period, this negatively impacted the study. As a result of the extended recruitment period, we lost many girls through attrition. A portion of participants did not participate in programming, because there was too much time between recruitment and program implementation. The participants were in one large group and not yet randomized into treatment and control. Some students became attached to a specific program specialist before students were randomized into classes with potentially different program specialists. In addition, the students formed bonds among themselves but were then assigned to different classrooms once the study began. Of those girls who participated in the presessions, we had some burnout after the sessions started but prior to the completion of actual programming. Other activities toward the end of the year, such as tutoring for tests or tryouts for sports, began to conflict with the time of Preventing Adolescent Pregnancy sessions, and the students were no longer available to participate in Girls Inc.
Another recruitment challenge was that program specialists were only at the schools once or twice a week. Therefore, up to a week would pass between when students received registration packets and the next time they saw program specialists. Packets were forgotten or lost, resulting in a waste of materials and time to complete the recruitment and randomizing of the students.
Establishing connections at the school are important and yet challenging. Although we located a primary contact person in the spring for the following school year, in the fall, the contact person was no longer at the school. High turnover rates are a common struggle when working with schools [
]. Therefore, a presence on campus and establishing relationships with staff are all the more important. During program implementation for Cohort 1, the program specialists prepared for the day's curriculum at the VOALA- Girls Inc. office and arrived at the school prior to implementation of curriculum. When the program specialists arrived, they often discovered that there had been a last minute classroom change. The program specialists would have to search for a new place to hold the session. Late-arriving students would not know where to go. The session itself would be delayed, and specialists would have difficulty getting through the session. Not being present on campus caused other difficulties as well. Specialists did not always know about upcoming events or schedule changes. Because Girls Inc. is only once a week, students were more likely to forget that they were supposed to stay after school that day. Yet, better attendance was observed when program specialists were present at school for lunch recruitment. The students would see the program specialists at lunch and have a reminder of the day's session.
Incentives are a necessity in RCT studies, and it is highly recommended that incentives be included in budgets [
] to assist with the challenge of the necessary 75% attendance rate when competing with other after-school activities. During Cohort 1, three items were raffled off weekly at the end of a session; incentives included movie tickets, $5 gift cards, and cash. Program specialists noted during the fidelity meetings that they felt their students were more excited about the incentives in the smaller schools, because they were receiving more incentives as a result of the raffle item per student ratio. During the summer following program implementation for Cohort 1, we conducted analyses to determine the effectiveness of this incentive plan. Attendance was highly correlated to frequency of receiving incentives, r(301) = .56, p ≤ .01. However, we were not sure whether the correlation was purely due to the fact that students with high attendance were more likely to receive an incentive or whether student attendance rates improved due to the incentives. Therefore, we looked at descriptive data to further examine differences in reception of raffle items as observed by the program specialists. Attendance rates and frequency of receiving incentives for the two largest recruited schools were compared with the two smallest recruited schools in Cohort 1. As shown in Figure 1, the average attendance rate for the two largest schools was 51%; whereas, for the two smallest schools it was 70%. As demonstrated in Figure 2, students in the smaller schools were more likely to receive an incentive than students in the larger schools. Students were likely to receive almost three incentives during the 10 weeks in the smaller schools whereas students in larger schools were likely to only receive one incentive.
In addition to the raffle, pizza was provided at each session. Pizza parties are another effective strategy that some studies utilize to keep students interested [
]. However, providing pizza on a weekly basis involved a great deal of time and money. When we stopped providing pizza, program specialists noted that attendance rates immediately dropped off. When examining attendance rates for the 10 weeks, we noted a drop in attendance once we stopped providing pizza (Figure 3). Once we brought back the pizza, the attendance rates increased again.
Changes to program implementation
Although an extended recruitment period did help VOALA-Girls Inc. meet recruitment numbers, there were many resulting impacts on the study. For Cohort 2, recruitment lasted for only 3 weeks. Program implementation started promptly regardless of the numbers. By quickly implementing the program sessions, we achieved higher attendance and completion rates as well as overall retention. However, this strategy provided additional challenges. At times, the classrooms ended up being too small for effective group projects and, if the classroom was already small, then even a few absent students greatly impacted the day's instruction. Without enough students to break into groups, some students opted to no longer attend because of a lack of other participants. Therefore, for the second half of the OAH study, program implementation for the third cohort during the fall 2013 will occur only at sites where at least eight participants are enrolled in each classroom. Schools with low recruitment will not receive programming.
Girls Inc. provided a drop box to the school. Drop boxes were located either in the main office or with the main contact person at the school. Students placed the completed registration packets in the drop box on the day they brought them back. Program specialists picked them up when stopping by or during the next week of implementation. As a result, students were able to bring back packets and turn them in immediately. Turn-in rates increased, and loss of packets decreased.
For Cohort 2, the program specialists prepared on site, and at lunch they reminded students of programming. By doing so, the program specialists were able to establish stronger bonds with their participants and create a stronger rapport. After lunch, the program specialists ensured that classrooms were available. If not, another room was secured prior to beginning the session and an announcement was made over the public address system to inform students of the room change. Program specialists also went to the main office to find out about upcoming events and any possible bell-schedule changes. They were able to secure the classrooms, which ensured less disruption. Program specialists engaged school sites to assist with program recruitment and retention while not placing an additional burden on them [
]. This helped promote sustainability and foster buy-in among school staff. They provided school sites with clear expectations regarding the program delivery process and needs. They also emphasized to school staff the importance of girls returning to the program weekly and the need for their support. Providing school principals, administrators, and staff with regular program updates ensured there was a feedback loop to share implementation status, challenges, and accomplishments. The program specialists built stronger relationships with school personnel and, as a result, school personnel displayed greater buy-in and assisted in recruitment and providing communication to students for implementation days as well as later data collection.
During Cohort 2, we implemented a new incentive program: the Swag Bag. Attendance charts were posted for girls to see how many days each attended, which is a practice identified as being very effective for incentives [
]. For every week attended, one more item would be placed in the Swag Bag. Focus groups identified some of the most popular incentives as being Girls Inc.-branded materials. A gift card to a very popular store often frequented by second-decade youth was added if nine or more classes were attended. Swag bags were passed out after the postprogram survey to increase attendance for the surveys. In addition to the Swag Bags, pizza parties were provided on days when key curricula were being delivered.
As a result of our many implemented strategies from lessons learned, attendance and retention improved tremendously (Table 2). For Cohort 2, average attendance rates were 66% for high school students (a 16% increase) and 64.5% for middle school students (a 16% increase); for Cohort 2, 83% of high school students completed the postprogram survey (a 4% increase) who had also completed the preprogram survey, and 100% of middle school students did so (a 53% increase). The large increase in completed postprogram surveys may be due to the new Swag Bag incentive, which was handed out on the day of the party and postprogram survey.
This success would not have been possible without the strong collaboration that occurred throughout the project across the many contributing parties. Therefore, the final lesson learned we wish to address is the importance of communication.
There were frequent challenges during implementation of an after-school program RCT. Delays occurred in programming that forced the program specialists to adjust many events on the calendar. A well-structured communication system, in place across teams, was imperative in order to be ready for unexpected problems. All members of the VOALA-Girls Inc. team met weekly to review the latest events and determine multiple plans for problem solving. They considered back-up plans as contingencies if initial plans did not work. The team would brainstorm potential solutions with each member, which contributed to the overall plan. The third-party evaluator, AES, allowed VOALA-Girls Inc. to determine the best solution for the program while providing guidance to ensure that all potential solutions did not veer from the overall design of the evaluation and federal study. The ongoing communication between Girls Inc. programming and AES allowed for fluidity despite the many changes that had to take place in response to challenges. For example, because students have many different competing interests after school, recruitment and program implementation must occur quickly before students need to shift their attention to yet another after-school activity. Changes to after-school program implementation based on ongoing assessment have actually been recognized as a strong component of effective programming within a community where adjustments need to be made to meet the needs of the participants [
]. VOALA-Girls Inc. and AES carefully balanced the two needs through ongoing communication and problem solving.
Along with close communication among the immediate teams, monthly communication occurred with OAH and MPR in which the VOALA-Girls Inc. team would share their recent struggles. OAH and MPR would then provide ideas and support to the program. Having the perspective of members of the OAH and MPR teams who were not deep within our study, but who have the perspective of working across all 43 of the TPP evaluations, helped the team see options and think through innovative ideas.
Through continuous communication and collaboration, VOALA-Girls Inc. and the major stakeholders of the RCT were able to make effective changes to program implementation that resulted in an increase in attendance rates as well as the number of postprogram surveys collected. The lessons learned resulted in four major recommendations for implementing an after-school program RCT:
Recruitment should occur for 3 weeks with immediate program implementation the following week.
Drop-off locations for completed registration packets improved recruitment numbers.
Increasing program staff visibility improves after-school program attendance.
Connect incentives to attendance and hand out earned incentives on day of postprogram survey.
To be effective in implementing an after-school program for a demonstration grant, it was necessary to structure the program to include an on-site presence and meaningful incentives linked to participant attendance. Collaboration among all team partners was a must to be ready for any challenges and initiating responsive timely changes. Substantial federal involvement in the program, including OAH's close collaboration with VOALA-Girls Inc., helped to ensure adherence to project goals and a balanced approach to implementing any changes to delivery of the program, as well as participant recruitment and retention methods, all of which impacted the evaluation.
Interventions and demonstration grants in after-school programming are challenging. However, the identified lessons learned are helpful in overcoming some of the obstacles relevant to the given context. VOALA-Girls Inc. will continue to identify and implement new lessons during the next 2 years as the TPP grant concludes so that strong, smart, and bold strategies can continue to improve attendance, retention, and overall program quality.
This publication was made possible by grant number TP2AH000019 from the Office of Adolescent Health in the Department of Health and Human Services.
Maximizing retention in community-based clinical trials.
Conflicts of Interest: The authors declare no conflicts of interest.
Disclaimer: Publication of this article was supported by the Office of Adolescent Health, U.S. Department of Health and Human Services. The opinions or views expressed in this paper are those of the authors and do not necessarily represent the official position of the Office of Adolescent Health, U.S. Department of Health and Human Services.