Economic Evaluation of Adolescent Addiction Programs: Methodologic Challenges and Recommendations

Methodologic challenges posed by adolescent SAT 

The small number of economic evaluations of adolescent SAT conducted may reflect the methodologic challenges inherent to this type of research. These complexities primarily stem from the characteristics of adolescent clients and the far-reaching consequences of SAT.

Adolescence is a period of dramatic emotional, physical, familial, and economic change [27]. Substance misuse may represent one of many developmental adolescent problems, which complicates the delivery of appropriate and effective services. Adolescents may have low internal motivation to attend SAT, particularly if they do not view their behavior as a problem, and many are required to attend treatment by the courts or their parents.

Unlike most health care interventions that primarily target a single person, with benefits accruing to the patient alone, the outcomes of adolescent SAT extend to numerous sectors of society beyond the patient, family, and health care system. Aarons et al [28] estimated the prevalence of substance use disorders among adolescents in five sectors of care. Rates of substance use disorders were high in the following service sectors: alcohol and drug (82.6%), juvenile justice (62.1%), mental health (40.8%), school based services for youths with serious emotional disturbance (23.6%), and child welfare (19.2%). Based on this study, an adolescent participating in SAT funded by their insurance company would likely produce benefits for the health care system as well as the criminal justice system, school system, and child welfare programs. In economics jargon, the improvements in the criminal justice, school, and child welfare sectors would be considered positive externalities, because the benefits are experienced by parties other than the consumer (adolescent in SAT) or producer (insurance company) of a good or service (SAT) [29]. Ignoring the positive externalities in an economic evaluation may underestimate the program's benefits, while overlooking any negative externalities may underestimate the program's costs. One challenge associated with evaluating adolescent SAT is capturing all the relevant externalities.

Although some methodologic issues are specific to the type of evaluation (e.g., CEA, BCA) being conducted, several underscore all economic evaluations of adolescent SAT. The next sections will discuss key methodologic challenges related to study design, the choice of perspective, the estimation of costs and outcomes, and the generalizability of study results.

Selecting a study design and follow-up period 

As noted earlier, economic evaluations are sometimes conducted alongside an existing clinical trial or quasi-experimental study. In the case of the multi-site grant, the investigative team worked with existing clinical studies of adolescent SAT programs to add an economic evaluation module. Since choices regarding the study design and the length of follow-up period were already decided by the core research team who initially designed the study, economic evaluation plans had to adapt to these conditions.

Double-blinded randomized controlled trials (RCTs) have traditionally been regarded as the strongest design for estimating relative treatment effects in epidemiology, health services research, and medicine [30], [31], [32]. In Table 1, more than half of the economic evaluations were based on clinical studies that randomized adolescents to at least two different treatment conditions. Although adolescent SAT studies have used randomization to compare a new or enhanced treatment to the standard of care or usual services, other elements of the double-blinded RCTs are not entirely appropriate for adolescent SAT. RCTs in adolescent SAT have not used a no-treatment control group, since withholding care is ethically inappropriate and can result in serious consequences [1], [7], [33], [34], [35]. In addition, RCTs require significant funding and oversight, and it may not be possible to ensure all treatment options are available in a given location. Finally, masking patients, health care providers, or assessors may be more difficult when a nonpharmacologic intervention is being assigned, and inadequate blinding can lead to biased estimates [36].

Many adolescent SAT studies have relatively short follow-up periods because of the high cost of collecting long-term data, the increased rates of nonresponse by subjects, and the desire of stakeholders for quick answers. A few clinical studies in the literature have had follow-up periods for up to 4 years [1], [37], but economic evaluations have not been conducted beyond 18 months and most have had a follow-up period of 12 months or less (Table 1). Selecting an appropriate follow-up time for adolescents is critical because of the developmental changes during this period [27]. For example, marriage often coincides with decreases in substance use, while college enrollment may increase the risk of relapse. If the follow-up period is too short, estimation results may not capture the various trajectories adolescents can follow after treatment [37], [38], [39], failing to assess whether the immediate effects of treatment (e.g., decreased substance use) necessarily lead to other desirable longer-term outcomes (e.g., high school graduation). These benefits may increase over time and would be missed in an evaluation with a follow-up period of only 1 year. French et al [23] did not identify many statistically significant differences in substance use and delinquency outcomes across four adolescent treatment interventions, perhaps because of the short follow-up period of 7 months.

On the other hand, studies with follow-up periods of several years may make it difficult to determine whether positive outcomes are the result of the original intervention, subsequent treatment episodes, or unrelated developmental changes. Most with substance abuse receive three to four treatment episodes over many years before achieving sustained abstinence [40]. If so, an adolescent may be exposed to multiple interventions over subsequent years. In the 30 months after one intervention, 20% of the adolescent sample spent significant time in a controlled environment, such as juvenile detention or residential treatment [38]. Although adolescents in these settings may be abstinent, it is possible that their substance use behavior reflects their current environment, rather than the effect of treatment per se.

Small changes to the design of an adolescent addiction intervention can address certain concerns and preserve the strengths of a RCT. Although it might seem unrealistic to blind therapists or clients in RCTs of behavioral interventions, individuals performing the assessments can be blinded to reduce bias. For example, Liddle et al [34] used clinician raters who were blind to treatment condition and phase of assessment (whether they were at follow-up or intake) to rate drug use severity. Additionally, instead of randomizing individual subjects to treatment conditions, cluster randomization, where treatment centers are randomized, can be used [41]. If possible, the treatment group can be compared with a natural control group, such as patients on a wait list for an intervention [42], rather than a no-treatment control group.

Alternative study designs become necessary when randomization is not practical, ethical, or economically feasible. As seen in Table 1, several economic analyses were based on studies with an observational or quasi-experimental design [15], [16], [17], [20], [21], [22]. Evaluators of the Fort Bragg Demonstration project used a quasi-experimental design to compare mental health expenditures for patients aged 5–17 years at the Demonstration (continuum of care) with those at two comparison sites (where services were reimbursed by the Civilian Health and Medical Program of the Uniformed Services). In observational studies such as the Fort Bragg project, unobservable variables (e.g., health status) could jointly influence the decision to receive treatment and the outcome(s) of interest [43], [44]. Econometric methods such as propensity score matching, fixed-effects models, and instrumental variable techniques can be used to try to reduce this type of bias. In their partial evaluation, Balsa et al [20] used instrumental variables to analyze whether treatment initiation was associated with several behavioral and clinical outcomes, since simple multivariate models failed to account for selection into treatment based on differences in severity, motivation, or other unobserved factors. Results indicated that selection bias is an important concern when evaluating observational studies.

Sophisticated modeling can also be used to address concerns associated with the length of the follow-up period. If a follow-up period is shorter than desired, researchers can use modeling to extrapolate information beyond the clinical trial to later endpoints [45]. This approach has not been used in adolescent SAT, but epidemiologic data from long-term observational studies in other health care areas have been used to model survival or other outcomes beyond the follow-up period [46]. In order to apply this approach to adolescent SAT, research funds should be invested in collecting long-term observational data with standardized measures that will be useful in modeling the long-term impacts of adolescent SAT. Furthermore, modeling can address some of the concerns associated with very long follow-up periods by taking into account developmental transitions, time in controlled environments, and exposure to other treatment interventions. Besides standard outcome data, researchers should ensure that follow-up interviews collect information on all types of service delivery and living arrangements that could affect substance use.

Choosing the perspective for an economic evaluation of adolescent SAT 

As mentioned earlier, the analysis perspective can profoundly impact the costs and outcomes that are included in an economic evaluation. A narrow perspective might only consider the direct impact of a program on the funding agency. This approach would perhaps be appropriate to evaluate a therapy to treat Type I diabetes among adolescents, whereby an economic evaluation focused on the health care sector would capture the vast majority of program costs and benefits. When a primary outcome related to the patient's health status is easy to identify, the program's success can be assessed based on this outcome. This is not necessarily the case for adolescent SAT programs, however. Reductions in substance use can generate significant, long-term positive externalities for the adolescent's family and other sectors of society (e.g., employers and school, criminal justice, and health care system). With a narrow perspective, an economic evaluation of adolescent SAT may overlook important outcomes with significant economic benefits, especially if adolescents improve in certain areas but not others.

Two studies from Table 2 discussed broadening their perspective beyond the health care system. The original evaluation of the Fort Bragg Demonstration project was conducted in 1995 and based on the perspective of its sponsor, the U.S. Department of Defense. Results indicated that expenditures on mental health services for young children and adolescents at the Demonstration were substantially higher than at the comparison sites. Critics believed these results reflected the narrow perspective of the original analysis. Foster and Bickman [16] re-analyzed the data and included use of mental health services outside the main catchment areas, use of medical services by children and adolescents, and expenditures on mental health services by other family members. Although the differences in expenditures between those at the Demonstration and comparison sites were narrower after including this information, total family expenditures were still higher at the Demonstration site. Schoenwald et al had a similar experience in their economic evaluation comparing multi-systemic therapy (MST) to usual services [26]. The difference in the total costs per youth between MST and usual services was $1695 (costs for MST were higher). After accounting for the decrease in incarceration days, the total cost per youth for MST was still higher than for usual services, but the difference narrowed to $877.

In some instances, the sponsor of the economic evaluation may only be interested in a narrow perspective, but the analysis’ perspective should be made clear when reporting results. In evaluating adolescent SAT, a wider perspective is preferable because of the nature of adolescent SAT and the important externalities. The choice of perspective defines which costs and outcomes will be included in an economic analysis. The estimation of these costs and outcomes will be discussed next.

Estimating costs incurred by multiple parties 

The direct resources invested in adolescent SAT services include those pertaining to the client, caretaker and families, and program. Estimating the opportunity costs incurred by the client, caretaker, and family are especially difficult in evaluations of adolescent SAT. Many adolescents are mandated by the court, school, or their parents to attend treatment, and may not believe they have an addiction problem. The family is often key to the participation in, and success of, many contemporary adolescent addiction interventions, such as Multidimensional Family Therapy (MDFT). Because adolescent clients usually live at home and are dependent on parents/guardians financially, costs to the family may be significant. Assessing only the time and money spent by an adolescent to attend treatment may overlook important costs, such as parental absence from work to assist with and participate in treatment. Although studies from adult SAT offer some guidance for estimating costs, methods need to be adapted to address features specific to adolescents.

The Drug Abuse Treatment Cost Analysis Program (DATCAP) is a standardized and theory-driven instrument that can be used to collect data for personnel costs, supplies, buildings, equipment, contracted services, and other items from SAT programs. DATCAP results from 13 adolescent outpatient programs and 1 residential program indicated that adolescent interventions were more costly than adult programs [18]. On average, adolescents attended treatment for 13 weeks at a cost of $194 per week and $2678 per treatment episode. Other studies have estimated total annual economic costs, average economic cost of a treatment episode, and average weekly cost per client for adolescent SAT [19], [22], [23], [47] (Table 2).

Studies of adult SAT have estimated client costs incurred by those in treatment, such as the opportunity cost of time in treatment (e.g., forgone employment) and travel time [48], [49]. For adolescent SAT, costs to the family of adolescents in SAT may be more relevant than client (i.e., adolescent) costs. Evaluators should design data collection forms or utilize existing instruments to collect these data. For example, French et al [25] measured the days a parent missed work or school in addition to the time missed by adolescents in the CYT study. Another possibility would be to design an instrument for parents/guardians of adolescents in treatment similar to the Significant Other Survey, measuring the health, financial, legal, relationship, and emotional problems experienced by partners of adults with substance abuse problems [50]. In this regard, the Caregiver DATCAP is part of several on-going adolescent addiction studies (www.DATCAP.com).

Researchers in the adolescent addiction field could set the standard for other health care researchers in conducting a comprehensive assessment of treatment-related costs. The intervention studies currently being conducted could generate a rich data source concerning costs outside the health care sector, specifically those affecting the family, which plays a crucial role in the delivery and effectiveness of adolescent interventions.

Estimating benefits for multiple parties 

Although clinical evaluations of adolescent addiction interventions generally focus on measures of abstinence or reduced substance use as the primary outcome [4], more have begun to evaluate the effects of adolescent SAT on schooling, family problems, health care use, and psychologic adjustment [1], [2], [6], [51]. With adolescent SAT, certain sectors (e.g., health care) may bear the majority of costs, although most of the benefits accrue to other sectors (e.g., criminal justice), as seen in the study by Balsa et al [20]. Results of this study suggest that initiating treatment affects schooling and employment, domains beyond the sphere of the managed care organization that provided the treatment. Even if reductions in substance use are not sustained, there may be improvement in other areas [52]. Likewise, reduced substance use may not by itself generate a positive economic result if other outcomes do not improve. Cohen estimated that the typical heavy drug user generates $370,000-$970,000 in external costs to society [53]. Consequently, preventing high risk youth from lifetime drug use can result in substantial economic benefits. Because of the widespread consequences of adolescent substance abuse, economists must decide which treatment outcomes to measure and how to value them.

When applied to adolescent SAT, the economic evaluation methods discussed earlier each have strengths and weaknesses. CEAs are best suited for interventions that have one clear goal or several secondary outcomes that are strongly related to the primary outcome [13], [54]. Although several studies in Table 2 conducted a CEA, adolescent SAT is fundamentally a multi-outcome intervention. Thus, the economic merits of a program will depend on the outcome selected for the CEA [54]. Balsa et al examined the effects of treatment initiation on substance use, schooling, criminal activity, and employment [20]. No significant effects on criminal activity were found, but treatment initiation was associated with positive effects on the other outcomes. Had a CEA been conducted with criminal activity as the primary outcome, the authors would have concluded the intervention was not cost–effective. For this reason, several cost–effectiveness ratios should be estimated to assess multiple outcomes [23]. The decision maker can then rank the importance or value of each outcome and apply these rankings to the cost–effectiveness ratios. If the perspective of the evaluation leads to only one outcome, a CEA may be more appropriate and efficient. French et al discuss other methodologic challenges in conducting a CEA of adolescent SAT interventions [23].

The use of QALYs in a CUA of adolescent SAT overcomes some of the limitations of a CEA because QALYs are a generic measure of health gain [13]. To our knowledge, a CUA of adolescent SAT has yet to be conducted. Economists have begun to use QALYs to evaluate other adolescent programs, such as treatments for depression, but these studies have relied on utility weights for adults [55], [56]. Some question whether this approach can be applied to addiction interventions for adolescents. The formal consideration of preferences of adolescents with substance abuse and the subsequent calculation of QALY gains may be qualitatively different from valuing the QALY gains from other interventions. QALYs do not typically capture the benefits in other domains or the effects on individuals other than the client in treatment, although, in principle, the QALYs experienced by others (e.g., family members) could also be measured.

A BCA is best suited to measure the externalities associated with adolescent SAT because it includes a wider range of outcomes. Outcomes are first selected from an assessment instrument designed for SAT programs, such as the Global Appraisal of Individual Needs (GAIN) [19], [25]. One challenge with an instrument like the GAIN is that measures may be more applicable for clinical evaluations, making it difficult to convert them to economic benefits. In their BCA of the CYT study, French et al included the following SAT outcomes: health services use, subsequent SAT use, education and employment, and criminal activity [25]. For criminal activity, the GAIN only recorded the number of arrests, rather than the number of times adolescents committed individual crimes. Although revisions were made to the GAIN in this area, it is likely that the authors underestimated the true number of crimes and the benefits of treatment.

After selecting and measuring appropriate outcomes for SAT interventions, these outcomes must be converted to dollars or other currency using monetary conversion factors, which are not always readily available for adolescents. McCollister et al estimated the cost of criminal activity before and after treatment in a juvenile drug court [24]. The study relied on crime cost estimates based on adult criminal activity that incorporated victim costs, crime career costs, criminal justice system costs, and intangible costs related to the victim's pain and suffering. The authors explain that the juvenile justice system and crime career costs are probably different for adults and adolescents. Zavala et al [19] more extensively discuss how to value outcomes.

The solution to these valuation challenges is far from clear. In the short term, a reasonable approach would be to conduct a cost–consequences analysis. Rather than synthesize consequences and costs into one measure, various outcomes can be presented, making it easier for decision makers to select the outcomes that are most important from their perspective [57].

Allowing for the context-specific nature of adolescent substance abuse interventions 

The goal of an economic evaluation is to guide decision making about the appropriate allocation of scarce resources. If a state drug court program for adolescents produces cost savings in one state, other states might be interested in adopting a similar program. The wide range of organizations (e.g., school systems, local foundations, criminal justice system, religious organizations) involved in treating adolescents with substance abuse disorders makes it difficult to generalize results from an economic evaluation conducted in a specific setting. Differences in organizational structure, management, clinical practice patterns, and relative prices may lead to variations in costs and benefits across settings, and hence differences in cost–effectiveness and net benefit.

An example of this scenario and its impact on results from an economic evaluation can be seen in the CYT study, which evaluated five outpatient interventions delivered at four sites (Farmington, CT; St. Petersburg, FL; Madison County, IL; and Philadelphia, PA). Treatment costs varied significantly across sites because of cost of living differences, education and licensure of personnel, and management decisions [7], [25], [47]. For instance, the site in Philadelphia primarily hired therapists with doctorates and spent more on staff salaries than the other sites, which hired therapists with master's degrees. Certain sites also used more full-time employees to provide therapy, whereas others used part-time workers to better manage the workflow.

When estimating costs and benefits, economists should consider the actual setting and use monetary conversion factors specific to the geographic area [25]. The CYT study described above also reveals some potential solutions that can help address this challenge. CYT evaluators collected site-specific monetary conversion factors for health services use (e.g., emergency room visit), substance abuse treatment use (e.g., intensive outpatient program visit), education and employment (e.g., day missed of school or training), and criminal activity (e.g., day in prison). The value of an emergency room visit ranged from $189.24 in St. Petersburg, FL, to $210.39 in Farmington, CT.

Whenever feasible, data should be collected so that covariate adjustments pertaining to the site can be included in the analysis. A rigorous sensitivity analysis can account for uncertainty in the underlying assumptions and factors associated with location [58], [59]. These considerations may be more numerous and complex in evaluations of adolescent SAT, compared with other interventions, to account for regional variation related to organizations that interact with adolescents at a local level.