Strong, Smart and Bold Strategies for Improving Attendance and Retention in an After-School Intervention

The VOALA-Girls Inc. team includes staff from VOALA, the Girls Inc. program, and Advanced Empirical Systems (AES). VOALA is a large nonprofit organization that provides needed services to over 30,000 Angelenos through various programs. Girls Inc., which is one of VOALA's programs, provides after-school services to Title I middle and high school female students across the metropolitan Los Angeles area. Title I schools are schools that receive Title I funds due to being attended by a large percentage of students who qualify for free or reduced lunches indicating a large population with financial need. AES is the third-party evaluator of Girls Inc. program implementation and effectiveness. The VOALA internal evaluation department assists in the TPP study and ensures that program fidelity occurs and that Girls Inc. meets performance expectations. OAH supports VOALA-Girls Inc.'s implementation and evaluation activities by assigning an OAH Project Officer and a federally sponsored evaluation technical assistance contractor, Mathematica Policy Research (MPR), to the project. The strength of the implementation team and project management structure is critical to the success of the project. A multidisciplinary team that is inclusive and consists of program and evaluation specialists, administration, external evaluators, organizational leadership, and frontline staff helps to assure that program monitoring along with lessons learned are taken into account with reference to overall program goals and objectives. Each representative on the implementation team contributes a different aspect to the project; more importantly, each member constructively explores their differences to find workable solutions to the problem.

At the national level, Girls Inc. is an organization that promotes the development of curricula designed to empower girls to be “strong, smart, and bold” []. VOALA-Girls Inc. (the Los Angeles affiliate) is evaluating the effectiveness of the Preventing Adolescent Pregnancy curriculum when implemented in an after-school format. Preventing Adolescent Pregnancy provides age-appropriate lessons on sexual health and sexuality, assertiveness, decision-making and future planning. Program specialists were trained staff with at least a bachelor's degree, preferably in the social sciences, and previous experience working with second-decade youth. The program specialists conducted weekly 90-minute sessions over an 11-week period for 5 to 20 students. Interactive activities included debates, role playing, group discussions, and games to provide participants with opportunities to learn and integrate acquired knowledge and assertive skills into their lives. The control group received Economic Literacy curriculum, which focused on financial independence, wants versus needs, budgeting, and future planning. Regardless of curriculum, each session built upon material from previous sessions; therefore, it was imperative that participants attend consistently. The study is being implemented over multiple years. Cohort 1 occurred during the 2011–2012 school year. Cohort 2 took place during the 2012–2013 school year.

For successful implementation and evaluation, VOALA-Girls Inc. must achieve consistent attendance and high completion of postprogram surveys. OAH has a number of performance measures on which all TPP grantees must report biannually, including attendance. OAH has a 75% attendance benchmark for all of their funded TPP programs []. If students do not receive a sufficient portion of the curriculum, program effectiveness cannot be determined due to insufficient exposure to material. In addition, postprogram surveys are necessary to determine the level of immediate growth from participating in the program. If too many students do not complete the postprogram survey, then one must question whether there are differences among those students who do and do not complete the survey. The results may not be representative of all students who were enrolled in the study. Participants received 11 weeks of the Preventing Adolescent Pregnancy curriculum in a weekly, after-school setting. During program implementation for Cohort 1, attendance was poor. Factors influencing poor attendance and postprogram-survey completion rates included school factors such as changing school staff, incentives, and program structure. For example, participant recruitment took over a month. Participants first recruited became involved with other activities by the time actual program implementation took place. Of those students who participated, many were involved in other after-school activities. Thus, not only was attendance negatively affected, postprogram-survey turnout was as well. As a result of program improvements, overall attendance and postprogram-survey completion rates improved during program implementation for Cohort 2 (all data are presented in the Methods section).

Students were recruited from urban Title I schools throughout the metropolitan Los Angeles area across multiple school districts. A total of 517 students participated across two cohorts in middle schools (sixth and seventh grades, ages 10–12 years) and high schools (ninth and tenth grades, ages 14–16 years) (Table 1). Program implementation for Cohort 1 (373 participants) occurred from October 2011 to May 2012, with 202 participants from six middle schools and 171 from six high schools. Program implementation for Cohort 2 (144 students) occurred from October 2012 to March 2013, with 95 participants from four middle schools and 49 from three high schools.

Third-party evaluators, namely AES, conducted the data collection. For the purpose of the evaluation, baseline data were collected the week prior to implementation of the curriculum. Post-survey data were collected following the 11 weeks of the curriculum as well as at 6- and 12-month follow-up to determine short-term and long-term program impacts on participants. For the current article, data included in the analyses will be from baseline and the postprogram surveys that occurred immediately after completion of programming.

Every 2 weeks, program specialists would meet with the program manager and internal evaluation team to review fidelity logs during a fidelity meeting. The fidelity logs kept a record of proper coverage of curriculum as well as provided the opportunity to note challenges. The fidelity meetings provided opportunities for program specialists to note observations they made during program implementation as well as provide support to each other. Information from these meetings would be taken to the monthly meetings with the major stakeholders.

To ensure the program and evaluation is implemented in a manner consistent with OAH requirements and the high standards of the U.S. Department of Health and Human Services Pregnancy Prevention Research Evidence Review [], monthly calls have been held over the past 3 years with the major stakeholders, which included the funder, its technical assistance contractor, our program staff, our implementers, and our third-party evaluator. Using real-time reports from the VOALA-Girls Inc. database, the team discussed project barriers, accomplishments, modifications, and solutions to real-world implementation and evaluation challenges. VOALA-Girls Inc. successfully monitored and documented the intervention with sufficient detail to facilitate future replication. VOALA-Girls Inc. developed a program manual to adequately document efforts testing our innovative program model. This included documenting any adjustments made during the program implementation period, significant adaptations, and the identification of the core components. Program changes based on these meetings occurred during the summer prior to the next program implementation.

As mentioned earlier, two important elements of a randomized clinical trial (RCT) are participant retention and low attrition rates for data collected across time points. For Cohort 1, average attendance rates for both treatment and control were 50% for high school and 48% for middle school (Table 2). The attendance rates were well below OAH's goal of 75%. Only 79% of high school students and 47% of middle school students among students who completed the preprogram survey also completed the postprogram survey.

The original recruitment goal was 20 participants per classroom to ensure a large overall sample size to meet the evidence standards required by the grant, a total of 1,000 participants by the end of the grant. A small sample size diminishes power, which helps detect an effect [], and effects determined to be significant could be questionable. For example, small samples can result in extreme effect sizes, either positive or negative, as a result of differing teacher styles such as one teacher being more highly experienced than another, which would balance out in a larger sample []. Many identified effective recruitment practices were in place, including attending Back to School Night, a parent event provided by the schools to introduce parents to their students' new teachers as well as new student orientations, which were provided by schools for new students []. Recruitment took place at the schools during lunch and in classrooms. Recruiters were armed with flyers and snacks as well as program registration packets. During our first year, we had some delays prior to commencing implementation. While VOALA-Girls Inc. was in the early stages of program implementation, OAH was in the early stages of determining how the study would be evaluated. One of the requirements toward the end of the first year was to have a medical review of the curriculum. Programs funded under the TPP grant needed to ensure that information provided was age appropriate, as well as scientifically and medically accurate. VOALA-Girls Inc. was required to submit all core curricula and related educational materials to the OAH for review and approval prior to use in the project. The medical review took a few months to be completed due to multiple programs' curricula needing to be reviewed. VOALA-Girls Inc. was unable to implement programming until the medical review determined the curriculum was medically accurate. We took advantage of this time to extend our recruitment period because older second-decade youth (10–16 years old) in higher needs areas are difficult to recruit into programming []. Those students who were already recruited began participating in presession activities. Delays to programming lasted up to 6 weeks for some school sites, resulting in long periods of presession. Although we increased our recruitment numbers during the extended period, this negatively impacted the study. As a result of the extended recruitment period, we lost many girls through attrition. A portion of participants did not participate in programming, because there was too much time between recruitment and program implementation. The participants were in one large group and not yet randomized into treatment and control. Some students became attached to a specific program specialist before students were randomized into classes with potentially different program specialists. In addition, the students formed bonds among themselves but were then assigned to different classrooms once the study began. Of those girls who participated in the presessions, we had some burnout after the sessions started but prior to the completion of actual programming. Other activities toward the end of the year, such as tutoring for tests or tryouts for sports, began to conflict with the time of Preventing Adolescent Pregnancy sessions, and the students were no longer available to participate in Girls Inc.

Another recruitment challenge was that program specialists were only at the schools once or twice a week. Therefore, up to a week would pass between when students received registration packets and the next time they saw program specialists. Packets were forgotten or lost, resulting in a waste of materials and time to complete the recruitment and randomizing of the students.

Establishing connections at the school are important and yet challenging. Although we located a primary contact person in the spring for the following school year, in the fall, the contact person was no longer at the school. High turnover rates are a common struggle when working with schools [, , ]. Therefore, a presence on campus and establishing relationships with staff are all the more important. During program implementation for Cohort 1, the program specialists prepared for the day's curriculum at the VOALA- Girls Inc. office and arrived at the school prior to implementation of curriculum. When the program specialists arrived, they often discovered that there had been a last minute classroom change. The program specialists would have to search for a new place to hold the session. Late-arriving students would not know where to go. The session itself would be delayed, and specialists would have difficulty getting through the session. Not being present on campus caused other difficulties as well. Specialists did not always know about upcoming events or schedule changes. Because Girls Inc. is only once a week, students were more likely to forget that they were supposed to stay after school that day. Yet, better attendance was observed when program specialists were present at school for lunch recruitment. The students would see the program specialists at lunch and have a reminder of the day's session.

Incentives are a necessity in RCT studies, and it is highly recommended that incentives be included in budgets [] to assist with the challenge of the necessary 75% attendance rate when competing with other after-school activities. During Cohort 1, three items were raffled off weekly at the end of a session; incentives included movie tickets, $5 gift cards, and cash. Program specialists noted during the fidelity meetings that they felt their students were more excited about the incentives in the smaller schools, because they were receiving more incentives as a result of the raffle item per student ratio. During the summer following program implementation for Cohort 1, we conducted analyses to determine the effectiveness of this incentive plan. Attendance was highly correlated to frequency of receiving incentives, r(301) = .56, p ≤ .01. However, we were not sure whether the correlation was purely due to the fact that students with high attendance were more likely to receive an incentive or whether student attendance rates improved due to the incentives. Therefore, we looked at descriptive data to further examine differences in reception of raffle items as observed by the program specialists. Attendance rates and frequency of receiving incentives for the two largest recruited schools were compared with the two smallest recruited schools in Cohort 1. As shown in Figure 1, the average attendance rate for the two largest schools was 51%; whereas, for the two smallest schools it was 70%. As demonstrated in Figure 2, students in the smaller schools were more likely to receive an incentive than students in the larger schools. Students were likely to receive almost three incentives during the 10 weeks in the smaller schools whereas students in larger schools were likely to only receive one incentive.

In addition to the raffle, pizza was provided at each session. Pizza parties are another effective strategy that some studies utilize to keep students interested []. However, providing pizza on a weekly basis involved a great deal of time and money. When we stopped providing pizza, program specialists noted that attendance rates immediately dropped off. When examining attendance rates for the 10 weeks, we noted a drop in attendance once we stopped providing pizza (Figure 3). Once we brought back the pizza, the attendance rates increased again.