Advertisement

Validation of the Kriol and Belizean English Adaptation of the Revised Children's Anxiety and Depression Scale for Use With Adolescents in Belize

Open AccessPublished:November 15, 2022DOI:https://doi.org/10.1016/j.jadohealth.2022.10.002

      Abstract

      Purpose

      To validate a culturally-adapted Kriol and Belizean English version of the Revised Children's Anxiety and Depression Scale (RCADS) through comparison with clinical diagnoses made using the Kiddie Schedule of Affective Disorders and Schizophrenia.

      Methods

      Participants comprised of 256 adolescents aged 10-14 years and 15-19 years, who completed the adapted RCADS (10 depression items, 12 anxiety items) in one-on-one interviews, followed by a diagnostic assessment using Kiddie Schedule of Affective Disorders and Schizophrenia administered by trained clinicians. Sensitivity, specificity, positive predictive value, negative predictive value, diagnostic odds ratios, area under the curve (AUC), and Youden’s Index were calculated for RCADS cutoffs and scores on the total scale and anxiety and depression subscales.

      Results

      For adolescents aged 10-14 years (n = 161), the AUC was 0.72 for the full scale, 0.67 for anxiety subscale, and 0.76 for depression subscale. For adolescents aged 15-19 years (n = 95), the AUCs were 0.82, 0.77, and 0.83. Most depression items performed well in discriminating those with and without diagnoses. Separation anxiety items performed poorly. “Thoughts of death” were common even among adolescents not meeting diagnostic criteria. The RCADS depression subscale presented the strongest psychometric properties with adolescents aged 15-19 years (at cutoff of 13, sensitivity = 0.83, specificity = 0.77, positive predictive value = 0.47, negative predictive value = 0.95, odds ratio = 15.96).

      Conclusion

      The adapted RCADS-22 had acceptable categorization for adolescents aged 10-14 years and excellent categorization for adolescents aged 15-19 years; therefore, the tool is recommended for use among the latter age group. Based on sensitivity and specificity values at different cutoffs, guidance is provided to select different thresholds to suit clinical, public health, or other uses to detect and quantify adolescent depression and anxiety in Belize.

      Keywords

      Implications and Contribution
      A culturally-adapted version of the Revised Children's Anxiety and Depression Scale was compared with structured clinical interviews to determine how best to interpret results for clinical and public health purposes. Based on these findings, clinicians can select cut-offs that optimize sensitivity of the tool. For policy makers and public health practitioners, the psychometric properties can be used to produce more accurate estimates of prevalence balancing false negative and false positive rates at a population level.

      Background

      Mental health conditions constitute a major burden of disease for adolescents globally [
      • World Health Organization
      WHO
      Global Health Estimates GHE (2019).
      ]. In low-income and middle-income countries (LMICs), a lack of country-level data on adolescent mental health is one of the main bottlenecks for selecting and scaling up programs and services [
      • Carvajal-Velez L.
      • Requejo JH
      • Ahs JW.
      • et al.
      Increasing data and understanding of adolescent mental health worldwide: UNICEF’s measurement of mental health among adolescents at the population level initiative.
      ]. Cultural adaptation of data collection tools combined with appropriate clinical validation is crucial for improving the accuracy of estimated prevalence of mental illness with populations in LMICs, including among adolescents [
      • Kohrt B.A.
      • Jordans MJ
      • Tol WA
      • et al.
      Validation of cross-cultural child mental health and psychosocial research instruments: Adapting the depression self-Rating scale and child PTSD symptom scale in Nepal.
      ,
      • Van Ommeren M.
      • Sharma B.
      • Thapa S.
      • et al.
      Preparing instruments for transcultural research: Use of the translation monitoring form with Nepali-speaking Bhutanese.
      ,
      • Kaiser B.N.
      • Ticao C.
      • Anoje C.
      • et al.
      Adapting culturally appropriate mental health screening tools for use among conflict-affected and other vulnerable adolescents in Nigeria.
      ]. Instruments developed and validated with populations from high-income countries or other cultural settings cannot be directly translated with the expectation they will have the same psychometric properties in other cultural contexts [
      • Kaiser B.N.
      • Ticao C.
      • Anoje C.
      • et al.
      Adapting culturally appropriate mental health screening tools for use among conflict-affected and other vulnerable adolescents in Nigeria.
      ,
      • Hanlon C.
      • Medhin G,
      • Selamu M,
      • et al.
      Validity of brief screening questionnaires to detect depression in primary care in Ethiopia.
      ,
      • Kohrt B.A.
      • Luitel N.P.
      • Acharya P
      • Jordans MJD
      Detection of depression in low resource settings: Validation of the Patient health questionnaire (PHQ-9) and cultural concepts of distress in Nepal.
      ]. Cut-off scores established with specific populations are not necessarily comparable in other settings and may lead to misclassification and distortion of prevalence rates [
      • Kohrt B.A.
      • Kaiser B.N.
      Measuring mental health in humanitarian crises: A practitioner’s guide to validity.
      ].
      Validation studies are needed to establish psychometric properties of mental health assessment tools so that clinicians, policy makers, and researchers can select what cut-offs are relevant for their purposes and make informed decisions to advance mental health of young people. Moreover, cut-offs on screening tools that may be optimal for clinical settings may not be ideal for epidemiological studies and policy makers [
      • Kohrt B.A.
      • Kaiser B.N.
      Measuring mental health in humanitarian crises: A practitioner’s guide to validity.
      ]. For example, when assessment tools are used as screeners in clinical settings, it is important to identify those potentially in need of care and to prevent ‘false negatives’, that is, those youth who score below an established cut-off but are suffering from a mental health condition. Under clinical conditions, there is a trained health professional who can confirm whether or not a young person screening positive does have the condition requiring treatment. At a public health level when determining prevalence rates to accurately inform policy or when gathering epidemiological data to determine associations with risk factors, it is important to balance the risk of false negatives with the impact of ‘false positives’ (i.e., those youth score more than a screening cut-off but would not meet clinical standards for a condition and therefore not be indicated to initiate treatments such as medications and psychotherapy). This is relevant because commonly used mental health tools among adults provide estimates 2.5 times more than structured clinical interviews [
      • Levis B.
      • Benedetti A
      • Ioannidis JPA
      • et al.
      Patient health questionnaire-9 scores do not accurately estimate depression prevalence: Individual participant data meta-analysis.
      ]. A false positive in community prevalence studies, where clinical resources are often lacking to do confirmatory evaluations on all youth above cut-offs, risks financial burden on families to initiate care and potentially stigmatizes youth by being incorrectly labelled with a mental health condition. At a population level, if false positives substantially outweigh true positives, this can lead to misallocation of resources. For epidemiological studies, high numbers of false positives or false negatives may lead to erroneous associations with risk and protective factors and fail to identify important social and health-related contributors to or sequelae of mental health conditions. Although there are now tools for clinicians to estimate rates of false positives and negatives in their clinical populations (e.g., http://www.depressionscreening100.com/phq/), similar efforts have not been made for policy makers in ministries of health and international agencies that influence funding and mental health programs. Therefore, in addition to systematic transcultural translation and adaptation, clinical validation is needed to evaluate the performance of a tool in a particular population and setting, including providing a range of different cut-offs from which clinicians, policy makers, and researchers can choose the best for their purposes.
      United Nations Children’s Fund (UNICEF), in collaboration with the World Health Organization and other institutional, technical, and academic partners, has undertaken the Measurement of Mental Health among Adolescents at the Population Level (MMAP) initiative. This initiative involves developing and validating data collection tools to facilitate a country-level collection of robust data on adolescent mental health [
      • Carvajal-Velez L.
      • Requejo JH
      • Ahs JW.
      • et al.
      Increasing data and understanding of adolescent mental health worldwide: UNICEF’s measurement of mental health among adolescents at the population level initiative.
      ]. One of the tools being validated in this effort is the Revised Children's Anxiety and Depression Scale (RCADS), which has been used previously in many settings [
      • Chorpita B.F.
      • Yim L
      • Moffitt C
      • et al.
      Assessment of symptoms of DSM-IV anxiety and depression in children: A revised child anxiety and depression scale.
      ,
      • Piqueras J.A.
      • Martín-Vivar M
      • Sandin B
      • et al.
      The revised child anxiety and depression scale: A systematic review and reliability generalization meta-analysis.
      ]. Using MMAP protocols [
      • Carvajal-Velez L.
      • Ahs J.W.
      • Requejo J.H.
      • et al.
      Measurement of mental health among adolescents at the population level: A Multicountry protocol for adaptation and validation of mental health measures.
      ], the present study examines the concurrent criterion validity of a 22-item abbreviated version of the RCADS culturally adapted in Belize [
      • Carvajal-Velez L.
      • Ottman K.
      • Ahs J.W.
      • et al.
      Translation and adaptation of the revised Children’s anxiety and depression scale: A qualitative study in Belize.
      ], when compared with a structured clinical interview for depression and anxiety.

      Methods

      Settings

      Belize is a country in Central America with a population of approximately 398,000, of which about 20% are adolescents aged 10–19 years [
      Ministry of Health, B
      Adolescent health - Situational analysis.
      ]. It is culturally and ethnically diverse. English is the language of business and instruction and Kriol is spoken by most of the population. Mental illness is a growing cause of concern in the country as health facilities report increasing suicide attempts among the young [
      Ministry of Health, B
      Adolescent health - Situational analysis.
      ], and suicide is one of the top five causes of death for adolescent boys aged 15–19 years [
      Ministry of Health, B
      Adolescent health strategic plan.
      ].
      UNICEF, in collaboration with the Belize Ministry of Health, Ministry of Education, and other key stakeholders, conducted this study to ensure availability of a validated tool to measure adolescent mental health at the population level. The work involved conducting cultural translation and adaptation of the RCADS-22, of which results have been previously published [
      • Carvajal-Velez L.
      • Ottman K.
      • Ahs J.W.
      • et al.
      Translation and adaptation of the revised Children’s anxiety and depression scale: A qualitative study in Belize.
      ] and clinical validation following the MMAP protocol [
      • Carvajal-Velez L.
      • Ahs J.W.
      • Requejo J.H.
      • et al.
      Measurement of mental health among adolescents at the population level: A Multicountry protocol for adaptation and validation of mental health measures.
      ].

      Participants and recruitment

      A total of 256 adolescent girls and boys were recruited from eight schools in Belize City, seven in urban and peri-urban settings, and one school in a rural setting. Recruitment was divided between adolescents aged 10–14 years and 15–19 years, hereafter referred to as ‘younger’ and ‘older’ adolescents. The two age groups are consistent with World Health Organization's call for age standardization for data collection, reporting, and monitoring [
      • Diaz T.
      • Strong K.L.
      • Cao B.
      • et al.
      A call for standardised age-disaggregated health data.
      ].
      Given that secondary school enrolment in Belize is close to 90% [
      UNESCO, Institute for Statistics
      Database on education and Literacy.
      ] and there is interest by the government in expanding school-based mental health services [
      • Carvajal-Velez L.
      • Ottman K.
      • Ahs J.W.
      • et al.
      Translation and adaptation of the revised Children’s anxiety and depression scale: A qualitative study in Belize.
      ], it was deemed acceptable to focus the validation effort on school-going adolescents. Our study did not target participation of nonschool-going adolescents for the validation of the tool but they participated in the qualitative adaptation of the tool, ensuring a diversity of inputs in linguistic adaptation. Based on the experience from other settings, in which there are not different cut-off scores used for tools such as RCADS for school-going versus nonschool-going adolescents, [
      • Stevanovic D.
      • Bagheri Z.
      • Atilola O.
      • et al.
      Cross-cultural measurement invariance of the revised child anxiety and depression scale across 11 world-wide societies.
      ] we anticipate the psychometric properties and cut-off scores could generally be used among all adolescents regardless of school-going status.
      Sample sizes of around 300 are recommended for assessing validation psychometric properties of a measure [
      • Wolf E.J.
      • Harrington K.M.
      • Clark S.L.
      • Miller M.W.
      • et al.
      Sample size Requirements for structural Equation models: An evaluation of Power, Bias, and Solution Propriety.
      ]. In terms of sample distribution, our goal was to achieve an equally distributed sample among age groups and gender: 50% 15–19-year-olds, 50% 10–14-year-olds, and 50-50 split for girls and boys. The sampling was designed to be “enriched” with adolescents likely to have symptoms of depression or anxiety for a comparison with “controls” presumed to not have these conditions. The goal of the sample enrichment strategy was to achieve a 2:1 ratio among the selected participants for those likely to have depression and/or anxiety compared to participants less likely to have these conditions. This sampling strategy was used to ensure a large enough sample size for participants with these mental disorders to allow for conducting a full range of psychometric analyses for validation against clinical diagnoses [
      • Burkey M.D.
      • Adhikari R.P.
      • Ghimire L
      • et al.
      Validation of a cross-cultural instrument for child behavior problems: The Disruptive behavior international scale – Nepal version.
      ].
      We used the Community Case Detection Tool (CCDT) [
      • van den Broek M.
      • Ponniah P
      • Jeyakumar P.J.R.
      • et al.
      Proactive detection of people in need of mental healthcare: Accuracy of the community case detection tool among children, adolescents and families in Sri Lanka.
      ] (Supplementary Annex 1) to support the process for enriching the sample. The CCDT was used by CCDT-trained teachers to identify adolescents who may have a greater likelihood of depression or anxiety and students who did not appear likely to have these conditions. The CCDT is a vignette-based ‘proactive detection’ tool that describes symptoms of mental disorders like anxiety and depression [
      • van den Broek M.
      • Ponniah P
      • Jeyakumar P.J.R.
      • et al.
      Proactive detection of people in need of mental healthcare: Accuracy of the community case detection tool among children, adolescents and families in Sri Lanka.
      ]. It was adapted, for use among adolescents in Belize, from a version developed for proactive case detection among adults in Nepal, aiming to increase help seeking and mental healthcare utilization among individuals with mental disorders [
      • Jordans M.J.D.
      • Kohrt B.A.
      • Luitel N.P.
      • et al.
      Accuracy of proactive case finding for mental disorders by community informants in Nepal.
      ] and has been previously used with youth in Sri Lanka and the occupied Palestinian territories [
      • van den Broek M.
      • Ponniah P
      • Jeyakumar P.J.R.
      • et al.
      Proactive detection of people in need of mental healthcare: Accuracy of the community case detection tool among children, adolescents and families in Sri Lanka.
      ,
      • van den Broek M.
      • Hegazi L.
      • Ghazal N.
      • et al.
      Accuracy of a proactive case detection tool for internalizing and externalizing problems among children and adolescents.
      ]. The CCDT language and visuals were culturally and contextually adapted [
      • Subba P.
      • Luitel N.P.
      • Kohrt B.A.
      • Jordans M.J.D.
      Improving detection of mental health problems in community settings in Nepal: Development and pilot testing of the community informant detection tool.
      ] with inputs from mental health experts from the Ministry of Health in Belize. Teachers and school staff were trained for two days to use the adapted CCDT by mental health counselors to identify ‘at-risk’ and control adolescents. The training includes hallmark symptoms of depression and anxiety among youth, considerations for functional impairment associated with depression and anxiety, how to initiate conversations about mental health with youth, identifying if youth are interested in seeking services, and stigma and discrimination associated with mental health. The teachers are trained on the CCDT algorithm which includes considering whether youth distress matches prototypes for anxiety or depression, general appraisal of functional impairment, and willingness to seek help (the Belizean depression and anxiety CCDT tools are included in the supplemental files). CCDT positives (likely to have anxiety, depression, or both) and CCDT negatives (not likely to have these mental disorders) were referred by the teachers to participate in the study.

      Instruments

      Revised children's anxiety and depression scale

      The original RCADS comprises 47 items [
      • Chorpita B.F.
      • Yim L
      • Moffitt C
      • et al.
      Assessment of symptoms of DSM-IV anxiety and depression in children: A revised child anxiety and depression scale.
      ]. Items covering anxiety were adapted from the Spence Children's Anxiety Scale [
      • Spence S.H.
      Spence Children′s anxiety scale.
      ] supplemented with Diagnostic and Statistical Manual of Mental Disorders (DSM) IV symptoms [
      • American Psychiatric Association
      APA
      Diagnostic and statistical Manual of mental disorders DSM-4.
      ]. Items relevant to depression were designed to cover DSM-IV major depressive disorder. For purpose of this study, we started with the 25-item version of the RCADS [
      • Ebesutani C.
      • Reise S.P.
      • Chorpita B.F.
      • et al.
      The revised child anxiety and depression scale-short version: Scale reduction via exploratory bifactor modeling of the broad anxiety factor.
      ]. Prior to cultural adaptation of the tool, we removed three items pertaining to obsessive-compulsive disorder (OCD). This was done because OCD is no longer categorized as an anxiety disorder in the DSM-5 [
      • American Psychiatric Association
      A.P.A., Diagnostic and statistical manual of mental disorders (DSM-5®).
      ] and OCD symptoms tend to have low base-rate prevalence [
      • Muris P.
      • Meesters C.
      • Schouten E.
      A brief questionnaire of DSM-IV-defined anxiety and depression symptoms among children.
      ].
      Therefore, a 22-item RCADS version was selected for cultural adaptation and validation in Belize. It included 10 depression and 12 anxiety items, with the anxiety items covering panic disorder, separation anxiety disorder, social anxiety disorder, and generalized anxiety disorder. The cultural translation and adaptation process conducted in Belize following the MMAP protocol [
      • Carvajal-Velez L.
      • Ahs J.W.
      • Requejo J.H.
      • et al.
      Measurement of mental health among adolescents at the population level: A Multicountry protocol for adaptation and validation of mental health measures.
      ] is presented elsewhere [
      • Carvajal-Velez L.
      • Ottman K.
      • Ahs J.W.
      • et al.
      Translation and adaptation of the revised Children’s anxiety and depression scale: A qualitative study in Belize.
      ]. Total scores were calculated by summing all 22 items. Depression and anxiety subscale totals were also calculated.

      Kiddie Schedule for Affective Disorders and Schizophrenia

      The Kiddie Schedule for Affective Disorders and Schizophrenia (K-SADS) is a semi-structured interview designed for school children aged 6–18 years, updated for DSM-5 criteria [
      • Kaufman J.
      • Birmaher B.
      • Axelson D.
      • et al.
      The KSADS-PL DSM-5.
      ]. K-SADS was used as the ‘gold standard’ tool for the concurrent criterion validity analysis. For the purposes of this study, only the depression and anxiety sections of the K-SADS screening module and the depression and anxiety supplementary modules were used. Four clinicians (local school counselors and local psychotherapists) were recruited to conduct the diagnostics part of the assessment and were trained by a psychiatrist (B.K.) in using K-SADS. Training included calculating and reviewing inter-rater reliability to ensure participants were being scored by the clinicians in a consistent and comparable manner. Training includes discussion and identification of appropriate language and idioms to be used in the specific setting as an effort to address cultural adaptation needs. The Belizean clinical team, who had extensive experience working with adolescents, identified language in Belizean English and Kriol which they would consistently use for the probing questions while conducting the diagnostics assessment with K-SADS. Screener and probe symptoms are rated by the interviewer, as follows: (1) 'not present,’ (2) ‘subthreshold levels of symptomatology,’ and (3) ‘threshold levels of symptomatology met’/‘threshold criteria met’. The interviewer also judges the functional impairment associated with the potential syndrome. K-SADS provides criteria-based algorithms to assess the presence of current (ongoing) DSM-5 disorders based on the presence of recent ‘threshold criteria’ (clinically relevant symptoms), duration, and functional impairment. K-SADS–based diagnoses on major depressive disorder, separation anxiety, social anxiety, agoraphobia, and generalized anxiety were used to provide a clinical validation for anxiety and major depressive disorder on diagnoses.

      Data collection

      Adolescent participants were first interviewed by trained enumerators using the adapted RCADS-22, followed by additional mental health related questions (functional limitations, suicidality, care-seeking, and connectedness) and basic demographic information. We chose to use interviewer administration because this is standard for large scale global health and education surveys. All participants were subsequently assessed by trained clinicians using the K-SADS. Enumerators and counselors were blinded to the CCDT results and counselors conducting the diagnostics assessment using K-SADS were blinded to the RCADS-22 results. The entire process took up to 1 hour. K-SADS interviews that were not conducted immediately following the RCADS were scheduled within a two-day window. Interviews were conducted in Belizean English and/or Kriol, based on the adolescent's preference. All enumerators and counselors were trained local staff and fluent in both Belizean English and Kriol.
      Data from the two assessments were captured on digital tablets using Census and Survey Processing System software and data were reviewed by the data coordinator at the end of every day. Data collection took place during five consecutive weeks between February and March 2020. Data collection was suddenly interrupted due to COVID-19–related school closures in the country. This led to a reduction in the sample for this analysis from a total planned sample of 300 adolescents to a final sample of 256 participants.
      A written informed consent (and assent of minors) was obtained from all participants and parents or guardians and all participants were provided with age-appropriate information sheets. Counselors were available on-site and throughout study activities to respond in case of distress and to coordinate referrals as needed.

      Statistical analysis

      Statistical analysis was conducted using SPSS, version 26.0 for Windows statistical software (SPSS Inc., Chicago, Illinois), see Box 1.
      Psychometric analyses used in the validation analysis
      The present study examines the concurrent criterion validity of the 22-item RCADS tool, culturally adapted version for Belize [
      • Carvajal-Velez L.
      • Ottman K.
      • Ahs J.W.
      • et al.
      Translation and adaptation of the revised Children’s anxiety and depression scale: A qualitative study in Belize.
      ], against the K-SADS, a structured clinical interview, used as the ‘gold standard’ for validation purposes.
      The following psychometric measures were calculated: receiver operating characteristic (ROC), area under the curve (AUC), sensitivity, specificity, positive predictive value, negative predictive value, diagnostic odds ratio, and Youden's Index.
      ROC and AUC: Overall agreement of RCADS-22 scale scores with K-SADS for depression and a subset of K-SADS anxiety disorders was examined with ROC visualizations and accompanying AUC calculations, that is, the two-dimensional space below the ROC indicating the performance of classification models across all possible thresholds (Figure 1). The ROC can be used to illustrate sensitivity and specificity at different cut-off scores. To assess the precision of the estimates, we computed 95% confidence intervals for the AUC.
      Positive and negative predictive values (PPV and NPV) were calculated with caveat that predictive values are affected by prevalence rates, which are skewed by sampling through teacher referrals. We also computed the positive and negative likelihood ratios and diagnostic odds ratio.
      Following guidance of use of screening questionnaires for multiple purposes (e.g., clinical detection, school screening, public health prevalence rates), we calculated psychometric properties for a range of cut-off values. Using higher scale thresholds for determining the presence of a condition tends to increase false negatives but decrease false positives, which is important for accurate population level estimates. Using lower scale thresholds has the opposite effect of increasing false positives but decreasing false negatives, which is important for clinical uses of screening tools [
      • Kohrt B.A.
      • Kaiser B.N.
      Measuring mental health in humanitarian crises: A practitioner’s guide to validity.
      ].

      Ethical approval

      A research ethics approval was granted by Health Media Lab Institutional Review Board, an autonomous external ethics review committee in the United States authorized by the U.S. Office for Human Research Protections within the U.S. Department of Health and Human Services (HML Institutional Review Board Research Ethics Review ID#: 161EBLZ19). An ethical oversight of the study in Belize was provided by an Ethics committee arranged by the Ministry of Health of Belize who reviewed and approved the protocol prior to initiation of study activities.

      Results

      The sample consisted of 256 adolescents, all recruited using CCDT as criteria, of which 66% (170) were girls, 33% were boys (85), and one participant who self-identified as ‘other’. In terms of age, 63% of the sample was aged 10–14 years and 37% was aged 15–19 years (Table 1). Regarding recruitment based on the teacher nominations using the CCDT tool, 38% of participants were neither CCDT anxiety nor CCDT depression positive, 25% were CCDT anxiety positive, 28% were CCDT depression positive, and 8% were both CCDT anxiety and depression positive. Given COVID-19 containment strategies in country, which led to stopping recruitment ahead of time, it was not possible to achieve an intended age distribution (50% aged 10–14years and 50% aged 15–19years) and ratio of CCDT positive to CCDT negative participants (2:1).
      Table 1Demographics of participants in assessments using RCADS and K-SADS
      10–14 years old (n = 161)15–19 years old (n = 95)Total sample (n = 256
      All participants were recruited using the CCDT criteria.
      )
      Age
       Mean (SD)12.14 (1.40)16.12 (1.20)13.62 (1.34)
      Gender
       Male59 (36.6%)26 (27.4%)85 (33.2%)
       Female102 (63.4%)68 (71.6%)170 (66.4%)
       Other1 (1.1%)1 (0.4%)
      Highest level of education attended
       Primary100 (62.5%)(0%)100 (39.1%)
       Lower Secondary58 (36.2%)35 (36.5%)93 (36.3%)
       Upper Secondary1 (0.6%)60 (62.5%)61 (23.8%)
       Higher(0%)1 (1.0%)1 (0.4%)
       N/A1 (0.6%)01 (0.4%)
      Language of interview
       Belizean English only50 (31.25%)25 (26.0%)75 (29.3%)
       Belizean English and Kriol106 (65.6%)70 (73.9%)176 (68.8%)
       Kriol only5 (3.1%)(0%)5 (1.9%)
      Community Case Detection Tool (CCDT) for recruitment
       Negative (no condition)57 (35.4%)41 (32.2%)98 (38.3%)
       Anxiety CCDT positive49 (30.4%)16 (16.8%)65 (25.4%)
       Depression CCDT positive47 (29.2%)26 (27.4%)73 (28.5%)
       Both depression and anxiety CCDT positive8 (5.0%)12 (12.6%)20 (7.8%)
      Kiddie Schedule of Affective Disorders and Schizophrenia (K-SADS)
       No diagnosis124 (77.0%)59 (62.1%)183 (71.5%)
       Anxiety diagnosis28 (17.4%)27 (28.4%)55 (21.5%)
       Depression diagnosis17 (10.6%)23 (24.2%)40 (15.6%)
       Any diagnosis37 (23.0%)36 (37.9%)73 (28.5%)
      a All participants were recruited using the CCDT criteria.
      Regarding K-SADS results for clinical diagnoses, 71% had neither anxiety nor depression diagnoses on the K-SADS, 21% had anxiety diagnoses, and 16% had depression diagnoses. Of the entire sample, 29% of participants met K-SADS diagnostic criteria for depression and/or anxiety.

      Psychometric analysis

      The area under the curve for younger adolescents was 0.72 (95% confidence interval [CI], 0.64, 0.80) for the full scale, 0.67 (95% CI 0.57, 0.77) for the anxiety subscale, and 0.76 (95% CI 0.64, 0.89) for the depression subscale. For older adolescents, the areas under the curve were 0.82 (95% CI 0.73, 0.90) for the full scale, 0.77 (95% CI 0.66, 0.88) for the anxiety subscale, and 0.83 (95% CI 0.75, 0.92) for the depression subscale (Figure 1).
      Figure thumbnail gr1
      Figure 1Area under the curve for RCADS subscales by K-SADS depression or anxiety.
      Table 2 presents the psychometric properties of the RCADS when compared with the K-SADS for the full tool and anxiety and depression subscales, with results presented by age group. The tool performed best for discriminating depression among older adolescents (cut-off of ≥13, with the highest diagnostic odds ratio of 15.96, sensitivity = 0.83 (95% CI 0.66, 0.93), specificity = 0.77 (95% CI 0.69, 0.83), positive predictive value = 0.47 (95% CI 0.34, 0.60), and negative predictive value = 0.95 (95% CI 0.0.89, 0.98), with positive likelihood ratio = 3.57 (95% CI 2.55, 4.98) and negative likelihood ratio = 0.22 (95% CI 0.10, 0.46).
      Table 2Validation psychometrics of Revised Child Anxiety and Depression Scale (22-item Belize version) validation psychometrics of Revised Child Anxiety and Depression Scale (22-item Belize version) from comparison with the Kiddie Schedule of Affective Disorders and Schizophrenia
      Cut-off scoreSensitivitySpecificityPPVNPVPLRNLRDiagnostic ORYouden's index (J)TP (%)TN (%)FP (%)FN (%)Accurately classified (%)
      Total RCADS-22 score on anxiety and depression items
       10–14 years old≥190.950.370.310.961.510.1311.640.320.220.280.490.010.50
      ≥200.860.410.300.911.450.354.180.270.200.320.450.030.51
      ≥210.830.440.300.891.470.403.690.260.190.340.430.040.53
      ≥220.830.460.310.91.530.384.060.290.190.360.410.040.55
      ≥230.780.540.330.891.680.414.070.320.180.410.360.050.59
      ≥240.710.590.340.881.760.483.660.310.160.460.310.070.62
      ≥250.620.620.330.851.640.612.680.240.140.480.290.090.62
       15–19 years old≥210.910.480.450.921.750.199.390.390.290.330.360.030.62
      ≥220.860.550.470.891.890.267.200.400.270.370.310.050.64
      ≥230.860.560.480.891.960.257.700.420.270.380.300.050.66
      ≥240.860.610.510.92.210.239.450.470.270.420.270.050.69
      ≥250.800.640.510.882.260.307.420.450.250.440.240.060.69
      ≥260.750.730.560.862.750.348.000.480.240.500.190.080.73
      ≥270.750.750.580.873.030.339.100.500.240.510.170.080.75
      Anxiety RCADS-22 subscale of 12 items
       10–14 years old≥910.240.1911.310---0.240.150.200.650.000.35
      ≥100.880.270.180.931.210.432.800.160.130.230.610.020.37
      ≥110.880.340.190.941.340.353.870.220.130.290.560.020.43
      ≥120.880.390.210.951.450.304.740.270.130.330.520.020.47
      ≥130.830.450.210.941.520.374.100.280.130.380.470.030.51
      ≥140.790.550.240.931.740.394.470.340.120.470.380.030.59
      ≥150.790.600.260.941.970.365.520.390.120.510.340.030.63
       15–19 years old≥100.910.340.310.921.380.275.100.250.220.260.500.020.48
      ≥110.910.410.330.931.540.236.790.320.220.310.450.020.53
      ≥120.860.440.330.911.540.324.850.300.210.330.420.030.54
      ≥130.770.500.330.871.530.473.300.270.190.380.380.060.56
      ≥140.770.580.370.891.840.404.600.350.190.440.320.060.63
      ≥150.70.660.390.872.030.464.410.350.170.500.260.070.67
      ≥160.70.740.460.882.670.416.530.440.170.560.200.070.73
      Depression RCADS-22 subscale of 10 items
       10–14 years old≥610.240.1511.310---0.240.120.210.670.000.33
      ≥70.910.280.150.961.260.323.910.190.110.250.630.010.36
      ≥80.820.390.150.941.340.472.860.210.100.340.540.020.44
      ≥90.730.450.150.921.310.612.150.170.090.390.490.030.48
      ≥100.640.530.160.911.350.691.960.170.080.470.410.040.54
      ≥110.640.600.180.921.60.602.660.240.080.530.350.040.61
      ≥120.640.690.220.932.050.533.900.330.080.610.270.040.68
       15–19 years old≥910.380.2811.610---0.380.200.310.500.000.50
      ≥100.890.460.290.941.630.256.540.340.180.370.440.020.54
      ≥110.890.560.330.9520.219.720.440.180.450.360.020.62
      ≥120.830.680.390.942.610.2510.420.510.160.550.250.030.71
      ≥130.830.770.470.953.570.2215.960.600.160.620.190.030.78
      ≥140.710.80.460.923.50.369.740.510.140.640.160.060.78
      ≥150.660.820.470.913.590.428.550.470.130.660.150.070.79
      FN = False negative; FP = False positive; NLR = Negative likelihood ratio; NPV = Negative predictive value; OR = Odds ; PLR = Positive likelihood ratio; PPV = Positive predictive valueratio; RCADS = Revised Child Anxiety and Depression Scale; TN = True negative; TP = True positive.

      Item analyses

      Using the K-SADS as the ‘gold standard’ for diagnosis of anxiety and depression, item means were calculated separately for adolescents who were diagnosed as having an anxiety or depression disorder versus adolescents who did not receive any diagnosis. Table 3 presents discriminant ability of RCADS items for adolescents with and without diagnosis as per K-SADS results. Results are presented using means and standard deviations (SDs) and are grouped by age group and by K-SADS diagnostic status. For the adapted RCADS-22, in the total sample most of the depression items performed well by showing significant differences between respondent with and without diagnosis, with the exception of anhedonia (RCADS #4 “nothing is fun”) and fatigue (RCADS #21, “tired a lot”). The item about feelings of worthlessness (RCADS # 16 “Feel worthless”) did not discriminate in either age-specific subgroup, but it did in the total sample. In terms of anxiety diagnosis, in the total sample items RCADS #3 “afraid to be home alone,” RCADS#5 “worry bad things will happen to family,” RCADS #9 “scared to sleep alone,” and RCADS#25 “worry that something bad will happen to you” did not discriminate between diagnosed and nondiagnosed groups. As presented in the table, other items did not discriminate well across the separate age groups but performed well in the total sample. For the total sample RCADS-22 score, the mean among the adolescents with any anxiety and/or depression K-SADS diagnosis (M = 32.63, SD = 9.93, n = 73) was more than the mean among the adolescents without a K-SADS diagnosis (M = 22.48, SD = 9.59, n = 183, t = -5.75, p < .001).
      Table 3Discriminant ability of Revised Child Anxiety and Depression Scale (22-item version) items for adolescents with and without diagnoses on the Kiddie Schedule of Affective Disorders and Schizophrenia
      Item #Description10–14 years old (n = 161)15–19 years old (n = 95)Total sample (N = 256)
      Mean (SD)Mean (SD)Mean (SD)Mean (SD)Mean (SD)Mean (SD)
      AnxietyNo diagnosis (n = 124)Anxiety (n = 28)No diagnosis (n = 59)Anxiety (n = 27)No diagnosis (n = 183)Anxiety (n = 55)
      RCADS#2Worry when doing poorly1.56 (0.96)1.86 (0.97)1.69 (0.90)2.30 (0.78)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      1.61 (0.94)2.07 (0.90)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      RCADS #3Afraid to be home alone1.35 (1.27)1.39 (1.17)0.59 (0.93)0.74 (1.20)1.10 (1.22)1.07 (1.22)
      RCADS #5Worry bad things will happen to family1.50 (1.02)1.93 (0.94)1.64 (1.08)1.89 (1.01)1.55 (1.04)1.91 (0.97)
      RCADS #6Afraid to be in crowded places1.22 (1.10)1.68 (1.12)1.07 (1.08)1.78 (1.12)1.17 (1.09)1.73 (1.11)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      RCADS #7Worry what other people think1.29 (1.06)1.43 (1.07)0.81 (0.88)2.00 (1.14)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      1.14 (1.13)1.71 (1.13)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      RCADS #9Scared to sleep alone0.64 (0.92)0.93 (1.25)0.51 (0.90)1.00 (1.18)0.60 (0.91)0.96 (1.20)
      RCADS #11Suddenly dizzy for no reason0.78 (0.89)1.32 (0.90)0.59 (0.70)1.41 (1.08)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      0.72 (0.83)1.36 (0.99)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      RCADS #14Tremble or shake for no reason0.75 (0.97)1.14 (1.11)0.69 (0.86)1.37 (0.93)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      0.73 (0.93)1.25 (1.02)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      RCADS #18Think about death1.23 (1.09)1.71 (1.15)1.41 (1.00)2.00 (1.07)1.28 (1.06)1.85 (1.11)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      RCADS #20Worry about suddenly getting scared0.72 (0.77)1.07 (0.81)0.59 (0.67)1.37 (1.15)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      0.68 (0.74)1.22 (0.99)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      RCADS #22Afraid to look foolish in front of others1.15 (0.95)1.64 (0.99)1.00 (0.87)1.59 (1.05)1.10 (0.93)1.62 (1.01)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      RCADS #25Worry that something bad will happen to you1.24 (0.95)1.54 (1.10)1.29 (0.97)1.93 (1.07)1.26 (0.95)1.73 (1.10)
      Total RCADS anxiety subscale score (12 items)13.42 (5.98)17.64 (5.77)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      11.90 (5.69)19.37 (6.78)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      12.92 (5.91)18.49 (6.29)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      DepressionNo diagnosis (n = 124)Depression (n = 17)No diagnosis (n = 59)Depression (n = 23)No diagnosis (n = 183)Depression (n = 40)
      RCADS #1Sad1.09 (0.73)1.94 (0.90)1.02 (0.57)1.78 (0.60)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      1.07 (0.68)1.85 (0.74)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      RCADS #4Nothing is fun0.87 (0.92)1.18 (1.07)0.92 (0.95)1.48 (0.99)0.89 (0.93)1.35 (1.03)
      RCADS #8Trouble sleeping0.68 (0.94)1.24 (0.90)0.83 (0.93)1.70 (1.11)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      0.73 (0.94)1.50 (1.04)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      RCADS #10Problems with appetite0.94 (1.07)1.47 (1.18)0.90 (1.01)1.91 (1.00)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      0.93 (1.05)1.73 (1.09)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      RCADS #13No energy for things1.09 (0.85)1.59 (0.87)1.19 (0.86)1.78 (0.74)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      1.12 (0.85)1.70 (0.79)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      RCADS #15Cannot think clearly1.00 (0.79)1.53 (0.72)1.14 (0.84)1.78 (0.80)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      1.04 (0.80)1.68 (0.76)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      RCADS #16Feel worthless0.67 (0.85)1.29 (0.92)0.80 (0.96)1.48 (0.99)0.71 (0.89)1.40 (0.96)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      RCADS #19Do not want to move0.60 (0.84)1.00 (0.71)0.39 (0.56)1.26 (0.96)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      0.53 (0.76)1.15 (0.86)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      RCADS #21Tired a lot1.31 (0.94)1.18 (1.02)1.49 (0.84)1.96 (0.83)1.37 (0.91)1.63 (0.98)
      RCADS #24Feel restless1.12 (0.94)1.76 (0.83)1.29 (0.85)2.00 (1.00)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      1.17 (0.92)1.90 (0.93)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      Total RCADS depression subscale score (10 items)9.36 (4.75)14.18 (4.59)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      9.95 (4.66)17.13 (4.95)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      9.55 (4.72)15.88 (4.97)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      Any anxiety or depression diagnosisNo diagnosis (n = 124)Any diagnosis (n = 37)No diagnosis (n = 59)Any diagnosis (n = 36)No diagnosis (n = 183)Any diagnosis (n = 73)
      Total RCADS score (22 items)22.78 (9.63)30.81 (9.09)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      21.85 (9.56)34.50 (10.54)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      22.48 (9.59)32.63 (9.93)
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      The three items for obsessive-compulsive disorder in the RCADS-25 were not included in the study because of the reclassification in DSM-5 that no longer categorizes obsessive-compulsive disorder as an anxiety disorder; these are RCADS items # 12, 17, and 23.
      p < .05, t-tests for mean differences for diagnosed and nondiagnosed groups, with Bonferroni correction for multitesting (10 comparisons for depression items and 12 items for anxiety items).
      Figure 2 provides the means for the 22-RCADS items by K-SADS diagnostic status comparing no diagnosis with any diagnosis. Anxiety items such as “Worry when doing poorly” (RCADS#2) and “Worried that something bad will happen to your family” (RCADS #5) were the most frequently endorsed items in nondiagnosed adolescents. RCADS#5 and RCADS-21 (tired a lot), the second and third most frequently endorsed items, did not distinguish between groups. Of note, “thinking about death” (RCADS #18), which is an anxiety item in the RCADS, was the fourth most commonly endorsed symptom among nondiagnosed youth demonstrating a high frequency among participants. The item did distinguish between the diagnosed and nondiagnosed groups.
      Figure thumbnail gr2
      Figure 2RCADS item means for 22 items with 95% confidence intervals for adolescents with no K-SADS anxiety or depression diagnosis versus adolescents with an anxiety and/or depression K-SADS diagnosis. ∗p < .05, ∗∗p < .01 (t-tests) with Bonferroni corrections for 22 comparisons.
      Example application of validation results: adjustments for population prevalence
      The validation results can be used to aid policy makers to improve accuracy of population estimates of adolescent symptoms of anxiety and major depressive disorder. Using cut-offs that balance sensitivity and specificity, Figure 3 shows adjusted population estimates for anxiety and depression disorders based on detected prevalence rates in Belize using the RCADS-22. When adjusting for false positive and false negatives, the RCADS-22 identified prevalence rates for anxiety and depression that can be adjusted to approximate what the true prevalence may be in the population. For example, if a prevalence of 35% is identified for depression among older adolescents, the estimated true prevalence is likely closer to 20%. The degree of adjustment differs based on prevalence because of the contribution of false positive versus false negatives to the estimates made. The positive predictive value and negative predictive value for individual adolescents also vary by prevalence rates (also included in Figure 3). Tool psychometric properties are just one aspect of adjusting detected prevalence rates; other considerations should be the demographics of the population, potential confounding factors, and frequency of risk and protective factors in a particular context. Policy makers can be aided by algorithms or figures such as this and other tools, to make adjusted prevalence estimates when allocating resources and designing programs.
      Figure thumbnail gr3
      Figure 3Adjustments to estimate true prevalence based on RCADS-identified prevalence rates for anxiety and depression. The population true prevalence can be estimated based on the rates identified for RCADS subscales by age group. For example, an identified prevalence of 35% on the RCADS depression subscale for 15–19-year-olds is equivalent to an estimate of true population prevalence of 20%.

      Discussion

      This study provides psychometric properties of a 22-item version of the RCADS when compared with clinician diagnoses using the K-SADS. The ability of the tool to accurately categorize adolescents for the total scale and depression is acceptable for older adolescents. The ability to categorize adolescents with anxiety was poor for both age groups. An RCADS validation study conducted in Turkey among children and adolescents aged 8–17 years showed strong sensitivity (0.80) and specificity (0.78) for the depression subscale [
      • Gormez V.
      • Kılınçaslan A.
      • Orengul A.C.
      • et al.
      Psychometric properties of the Turkish version of the revised child anxiety and depression scale – child version in a clinical sample.
      ], performing comparably to the results obtained in this analysis for the sample of older adolescents in Belize.
      In our study, results showed that most individual items had good discriminant validity with the exception of the following items: anhedonia and fatigue for depression; separation anxiety items for anxiety; and the generalized anxiety disorder item ‘worry about something bad happening to one's family’. Poor performance of these particular items in distinguishing diagnosed from nondiagnosed may be partly explained by the culture and context. For example, in our qualitative research, many adolescents described ‘worry about their family’ (RCADS #5,) or ‘being home alone’ (RCADS #3) in relation to safety and crime concerns in the area. This raises broader questions of the cultural relevance of separation anxiety disorder as reflective of a mental health disorder in settings with more exposure to violence and other risk factors. Within U.S. populations, there is considerable cultural diversity in the interpretation of separation anxiety among European American, Indian American, and Puerto Rican mothers [
      • Sood E.D.
      • Mendez J.L.
      • Kendall P.C.
      Acculturation, Religiosity, and ethnicity Predict Mothers’ causal Beliefs about separation anxiety disorder and Preferences for help-seeking.
      ], which suggests that symptoms have different meaning and significance across cultures and context, even in the same country. Separation anxiety items also performed poorly in the validation conducted among Turkish children with sensitivity of 0.71 and specificity of 0.58 [
      • Gormez V.
      • Kılınçaslan A.
      • Orengul A.C.
      • et al.
      Psychometric properties of the Turkish version of the revised child anxiety and depression scale – child version in a clinical sample.
      ]. Based on our findings in Belize and results from other settings, we caution against interpreting the separation anxiety items as symptoms of a psychiatric disorder, unless there is a clinical evaluation exploring context of symptoms, level of impairment, and potential comorbidities.
      One striking finding was the prevalence of “think about death” (RCADS # 18 item). This was the fourth most commonly endorsed item by the nondiagnosed adolescents. This raises concerns about potentially high prevalence of death anxiety, catastrophic worry, or suicidality. Given that this work was conducted a few weeks before the declaration of the COVID-19 pandemic (February–March 2020) when COVID-19 was already in the news and prevention measures had already started to take place, this may have influenced responses in terms of thinking about death. In other LMICs, suicidality is often common even without a diagnosis of depression and this is especially true among young women, for example, in China, India, and Nepal [
      • Hagaman A.K.
      • Khadka S.
      • Lohani S.
      • Kohrt B.A.
      Suicide in Nepal: A modified psychological autopsy investigation from randomly selected police cases between 2013 and 2015.
      ,
      • Maselko J.
      • Patel V.
      Why women attempt suicide: The role of mental illness and social disadvantage in a community cohort study in India.
      ,
      • Nock M.K.
      • Borges G.
      • Bromet E.J.
      • et al.
      Cross-national prevalence and risk factors for suicidal ideation, plans and attempts.
      ]. More research is needed to determine how much the item “think about death” is associated with intents or plans to die or commit self-harm. For example, the item is not specific to the respondents thinking about their own death. Therefore, in a setting of high domestic and community violence where recurrent aggression or the potential loss of one's own life or of family members is elevated, “think about death” may be commonly endorsed.
      The outcomes of this study illustrate why local validation studies are so important. Without validating in the local context, there is no information on the burden of false negative and false positives for particular cut-points in the local context. False negative and false positive rates have major clinical and public health implications [
      • Kohrt B.A.
      • Kaiser B.N.
      Measuring mental health in humanitarian crises: A practitioner’s guide to validity.
      ]. Regarding the validation, this study does not endorse a particular cut-off score, and instead we recommend that the cut-off score be selected based on the intended purposes whether that be clinical, public health, research, or another initiative. As discussed above, lower cut-off scores provide greater sensitivity, however, that risks higher rates of false positives that may burden mental health programs or potentially lead to stigmatization of adolescents who perceive that they are being labelled with a mental illness. Higher cut-off scores have the advantage of fewer false positives but risk false negatives which could lead to not identifying all of the adolescents who would benefit from services. For example, among adolescents aged 15–19 years, a cut-off of ≥11 on the depression subscale would capture 89% of those with depression. If the clinicians were solely relying upon the screening tool, they would miss one of 10 older adolescents with depression. There would also be two false positives for every true positive. A clinician could distinguish among these to provide treatment appropriately. In a clinical setting, the trade-off of high sensitivity against low specificity would be appropriate to be sure youth were receiving the needed care. However, in a public health study to determine allocation of resources, using that same cut-off among a sample of 10,000 older adolescents could lead to double the actual prevalence with thousands of adolescents having false positive indications of depression or anxiety, and thus complicating resource allocation (see Box 2 and Figure 3). Either a higher cut-off threshold or a prevalence adjustment would help policy makers inform funding and planning for mental health services.

      Limitations

      This analysis is the first to include clinical validation with population-level measures of mental health among adolescents attending school in Belize. There were logistical challenges inherent to this type of population-based data collection work. Although the sample used in the analysis was larger than those in previous validation studies of similar nature [
      • Ganguly S.
      • Samanta M.
      • Roy P.
      • et al.
      Patient health questionnaire-9 as an effective tool for screening of depression among Indian adolescents.
      ,
      • Nandakumar A.L.
      • Vande Voort J.L.
      • Nakonezny P.A.
      • et al.
      Psychometric properties of the Patient health questionnaire-9 modified for major depressive disorder in adolescents.
      ], due to COVID-19–related restrictions, it was not possible to complete the planned sample size. Also, due to prematurely ending recruitment but also to low specificity of the CCDT, the expected balance of adolescents with clinical anxiety disorders was not achieved. Of note, the sample size is too small to make definitive statements about the validity for gender-by-age (e.g., comparing gender differences in the younger and older groups). Although there was a trend toward stronger psychometric properties among the girls, this is likely biased by the low numbers of boys who met criteria for depression and anxiety. Larger samples would be needed to evaluate the performance of the tool on specific anxiety disorders. Therefore, the current analysis is not suitable for definitive guidance as to which population subgroups are vulnerable to overestimation or underestimation of prevalence of the assessed disorders. Although the current results suggest that at a population level there is a greater risk for overestimation of depression among younger adolescents (aged 10–14 years) compared to older adolescents, additional research with larger younger samples is needed to definitively conclude these age differences.
      The validation addressed Belizean English and Kriol. Most adolescents preferred having questions asked in both Belizean English and Kriol; it was not possible to evaluate how each language performs separately. We do not anticipate that this will be an obstacle for clinicians and public health planners using this tool because of common mixing of Belizean English and Kriol in daily discourse. We were not able to validate the tool for other languages spoken in Belize such as Spanish, Maya, and Garifuna. This is important to address in future studies because the latter languages are spoken among indigenous groups who are often the most marginalized and have the least access to formal mental health services.
      Another limitation is that the validation is limited to school-going adolescents. Although we anticipate that the psychometric properties and cut-off scores could generally be used for all adolescents regardless of school-going status, future validation efforts should include nonschool adolescents as well. Another potential limitation is that using adolescents identified by teachers as potentially having mental health conditions may be impacted by potential biases related to race/ethnicity and mental illness. For example, in the United States, Black and Brown youth are at a greater risk of increased referrals for externalizing disorders [
      • Villodas M.T.
      • Pfiffner L.J.
      • Moses J.O.
      • et al.
      The roles of student gender, race, and psychopathology in teachers’ identification of students for services.
      ]. Similarly, qualitative research in Nepal supports that low-caste and ethnic minority youth may be more likely to be identified with externalizing disorders [
      • Burkey M.D.
      • Ghimire L
      • Adhikari RP
      • et al.
      The ecocultural context and child behavior problems: A qualitative analysis in rural Nepal.
      ]. However, this limitation may not have majorly impacted the results presented here because there is less precedent for major biases in internalizing disorders. Moreover, the CCDT approach tailors the design of vignettes and visual images to fit with the local cultural context to capture diverse youth groups and not be limited to one particular racial or ethnic group [
      • Subba P.
      • Luitel N.P.
      • Kohrt B.A.
      • Jordans M.J.D.
      Improving detection of mental health problems in community settings in Nepal: Development and pilot testing of the community informant detection tool.
      ]. Regarding the overall accuracy of the CCDT, this will be dealt with in a separate publication.
      Another possible concern is that the DSM-5 was used for the clinical evaluation, but the RCADS version is based on DSM-IV. A recent review of the RCADS as per DSM-5 symptoms identified the anxiety symptoms (31 items) that match the DSM-5 [
      • Radez J.
      • Waite P.
      • Chorpita B.F.
      Using the 11-item version of the RCADS to identify anxiety and depressive disorders in adolescents.
      ]. The subset of anxiety items in our study is within those that have been determined to match the DSM-5. Regarding depression, the only item that differs is suicidal ideation, which is in the DSM-5, but not in RCADS (the RCADS item refers to thoughts about death but not intentional self-harm). We included a separate suicidality set of items. However, those analyses will be presented separately. Thus, overall, the analysis is generally consistent with the DSM-5. Another concern is the cultural validity of the DSM-5 construct of adolescent depression. This is a topic that is currently being investigated qualitatively and quantitatively in diverse LMIC settings [
      • Manfro P.H.
      • Pereira R.B.
      • Rosa M.
      • et al.
      Adolescent depression beyond DSM definition: A network analysis.
      ]. Inclusion of local idioms of distress, somatic complaints, and more cross-culturally consistent symptoms for depression such as loneliness may have increased the validity of the tool [
      • Ottman K.
      • Wahid S.S.
      • Flynn R.
      • et al.
      Defining culturally compelling mental health interventions: A qualitative study of perspectives on adolescent depression in Lagos, Nigeria.
      ,
      • Viduani A.
      • Benetti S.
      • Martini T.
      • et al.
      Social isolation as a core feature of adolescent depression: A qualitative study in Porto Alegre, Brazil.
      ]. However, it is beyond the scope of the current analysis to provide a thorough cross-cultural critique of the DSM diagnoses for children and adolescents [
      • Canino G.
      • Alegría M.
      Psychiatric diagnosis – is it universal or relative to culture?.
      ].

      Conclusion

      This validation work contributes to improved measurement as it opens the door to measuring adolescent mental health conditions in Belize with a tool that has been culturally adapted and validated in the country. Using clinical validation helped to empirically evaluate how the transculturally translated and culturally adapted RCADS performed in Belize. We also provided information on a range of cut-off scores so that clinicians, policy makers, and researchers can use thresholds that best fit their intended objectives. Ultimately, more efforts in clinical validation are needed to improve the quality of data that will guide efforts to reduce the burden of depression and anxiety, particularly among younger adolescents around the world.

      Acknowledgments

      The authors wish to thank all dedicated study staff in Belize whose contributions were instrumental for carrying out the study activities. Special thanks go to Nurse Eleanor Bennet from Belize Ministry of Health for her support and engagement from the beginning of this project. The authors also thank Dr. Susan M. Kasedde (UNICEF representative at the time) for her leadership and commitment to children and adolescents' wellbeing, which made this work possible. We are also thankful to other UNICEF Belize personnel who supported with logistics and provided invaluable support to ensure continuity and completion of the field activities and proper engagement with key stakeholders in country: Michel Guinand, Angella Baitwabusa, Daedra Isaacs-Haylock, and Yuri Espiritu. We also thank the support from the UNICEF Regional Office for Latin America and the Caribbean, in particular Alejandra Trossero, who supported activities and processes in Belize. This work benefitted from the commitment of the mental health experts in Belize who contributed to field work by conducting clinical evaluations (Aimee Jex, Tina Cuellar, and Krystal Humes) as well and Language Experts. The team acknowledges the great work and support from the data collection team coordinated by Sherlene Neal and composed of Luciana Perez, Pamela Sedacy, Asha Brown, Laureen Roach, Deidra Cassanova, Monique Grant, and Kimmasha McField. The team also recognizes the staff from Project Heal and Youth Enhancement Services in Belize City for providing venues for conducting the interviews. The team also recognizes the great contribution from Mark Jordans for making the Community Case Detection Tool available for adaptation and use in Belize. The authors thank RCADS developers and copyright holders Prof. Bruce Chorpita and Prof. Susan Spence for making this relevant scale freely available for use in Belize at the time this work was performed. The authors are thankful to all adolescents, parents, and staff in Belize City who participated and contributed actively in the processes described in the article and whose engagement and contributions made this work possible.

      Supplementary Data

      Funding Sources

      This work was supported by the Bill & Melinda Gates Foundation [ INV-001395 ]. Under the grant conditions of the Foundation, a Creative Commons Attribution 4.0 Generic License has already been assigned to the Author Accepted Manuscript version that might arise from this submission.

      References

        • World Health Organization
        • WHO
        Global Health Estimates GHE (2019).
        WHO, Geneva, Switzerland2019
        • Carvajal-Velez L.
        • Requejo JH
        • Ahs JW.
        • et al.
        Increasing data and understanding of adolescent mental health worldwide: UNICEF’s measurement of mental health among adolescents at the population level initiative.
        J Adolesc Health. 2021; https://doi.org/10.1016/j.jadohealth.2021.03.019
        • Kohrt B.A.
        • Jordans MJ
        • Tol WA
        • et al.
        Validation of cross-cultural child mental health and psychosocial research instruments: Adapting the depression self-Rating scale and child PTSD symptom scale in Nepal.
        BMC Psychiatry. 2011; 11: 127
        • Van Ommeren M.
        • Sharma B.
        • Thapa S.
        • et al.
        Preparing instruments for transcultural research: Use of the translation monitoring form with Nepali-speaking Bhutanese.
        Transcultural Psychiatry. 1999; 36: 285-301
        • Kaiser B.N.
        • Ticao C.
        • Anoje C.
        • et al.
        Adapting culturally appropriate mental health screening tools for use among conflict-affected and other vulnerable adolescents in Nigeria.
        Glob Ment Health. 2019; 6
        • Hanlon C.
        • Medhin G,
        • Selamu M,
        • et al.
        Validity of brief screening questionnaires to detect depression in primary care in Ethiopia.
        J Affective Disord. 2015; 186: 32-39
        • Kohrt B.A.
        • Luitel N.P.
        • Acharya P
        • Jordans MJD
        Detection of depression in low resource settings: Validation of the Patient health questionnaire (PHQ-9) and cultural concepts of distress in Nepal.
        BMC Psychiatry. 2016; 16: e58
        • Kohrt B.A.
        • Kaiser B.N.
        Measuring mental health in humanitarian crises: A practitioner’s guide to validity.
        Conflict and Health. 2021; 15: 72
        • Levis B.
        • Benedetti A
        • Ioannidis JPA
        • et al.
        Patient health questionnaire-9 scores do not accurately estimate depression prevalence: Individual participant data meta-analysis.
        J Clin Epidemiol. 2020; 122: 115-128.e1
        • Chorpita B.F.
        • Yim L
        • Moffitt C
        • et al.
        Assessment of symptoms of DSM-IV anxiety and depression in children: A revised child anxiety and depression scale.
        Behav Res Ther. 2000; 38: 835-855
        • Piqueras J.A.
        • Martín-Vivar M
        • Sandin B
        • et al.
        The revised child anxiety and depression scale: A systematic review and reliability generalization meta-analysis.
        J Affective Disord 2017. 2017; 218: 153-169
        • Carvajal-Velez L.
        • Ahs J.W.
        • Requejo J.H.
        • et al.
        Measurement of mental health among adolescents at the population level: A Multicountry protocol for adaptation and validation of mental health measures.
        J Adolesc Health. 2022; https://doi.org/10.1016/j.jadohealth.2021.11.035
        • Carvajal-Velez L.
        • Ottman K.
        • Ahs J.W.
        • et al.
        Translation and adaptation of the revised Children’s anxiety and depression scale: A qualitative study in Belize.
        J Adolesc Health. 2022; 1-6https://doi.org/10.1016/j.jadohealth.2022.05.026
        • Ministry of Health, B
        Adolescent health - Situational analysis.
        MoH Belize, Belize City, Belize2018
        • Ministry of Health, B
        Adolescent health strategic plan.
        MoH Belize, Belize City, Belize2019
        • Diaz T.
        • Strong K.L.
        • Cao B.
        • et al.
        A call for standardised age-disaggregated health data.
        Lancet Healthy Longevity. 2021; 2: e436-e443
        • UNESCO, Institute for Statistics
        Database on education and Literacy.
        UNESCO, Montreal, Quebec, Canada2022
        • Stevanovic D.
        • Bagheri Z.
        • Atilola O.
        • et al.
        Cross-cultural measurement invariance of the revised child anxiety and depression scale across 11 world-wide societies.
        Epidemiol Psychiatr Sci. 2017; 26: 430-440
        • Wolf E.J.
        • Harrington K.M.
        • Clark S.L.
        • Miller M.W.
        • et al.
        Sample size Requirements for structural Equation models: An evaluation of Power, Bias, and Solution Propriety.
        Educ Psychol Meas. 2013; 73: 913-934
        • Burkey M.D.
        • Adhikari R.P.
        • Ghimire L
        • et al.
        Validation of a cross-cultural instrument for child behavior problems: The Disruptive behavior international scale – Nepal version.
        BMC Psychol. 2018; 6: 51
        • van den Broek M.
        • Ponniah P
        • Jeyakumar P.J.R.
        • et al.
        Proactive detection of people in need of mental healthcare: Accuracy of the community case detection tool among children, adolescents and families in Sri Lanka.
        Child Adolesc Psychiatry Ment Health. 2021; 15: 57
        • Jordans M.J.D.
        • Kohrt B.A.
        • Luitel N.P.
        • et al.
        Accuracy of proactive case finding for mental disorders by community informants in Nepal.
        Br J Psychiatry. 2018; 207: 501-506
        • van den Broek M.
        • Hegazi L.
        • Ghazal N.
        • et al.
        Accuracy of a proactive case detection tool for internalizing and externalizing problems among children and adolescents.
        J Adolesc Health. 2021; https://doi.org/10.1016/j.jadohealth.2021.03.011
        • Subba P.
        • Luitel N.P.
        • Kohrt B.A.
        • Jordans M.J.D.
        Improving detection of mental health problems in community settings in Nepal: Development and pilot testing of the community informant detection tool.
        Conflict and Health. 2017; 11: 28
        • Spence S.H.
        Spence Children′s anxiety scale.
        J Anxiety Disord. 1997;
        • American Psychiatric Association
        • APA
        Diagnostic and statistical Manual of mental disorders DSM-4.
        APA, Washington, D.C.2000
        • Ebesutani C.
        • Reise S.P.
        • Chorpita B.F.
        • et al.
        The revised child anxiety and depression scale-short version: Scale reduction via exploratory bifactor modeling of the broad anxiety factor.
        Psychol Assess. 2012; 24: 833-845
        • American Psychiatric Association
        A.P.A., Diagnostic and statistical manual of mental disorders (DSM-5®).
        APA, Washington, D.C.2013
        • Muris P.
        • Meesters C.
        • Schouten E.
        A brief questionnaire of DSM-IV-defined anxiety and depression symptoms among children.
        Clin Psychol Psychotherapy: An Int J Theor Pract. 2002; 9: 430-442
        • Kaufman J.
        • Birmaher B.
        • Axelson D.
        • et al.
        The KSADS-PL DSM-5.
        Kennedy Krieger Institute, Baltimore, MD2013
        • Gormez V.
        • Kılınçaslan A.
        • Orengul A.C.
        • et al.
        Psychometric properties of the Turkish version of the revised child anxiety and depression scale – child version in a clinical sample.
        Psychiatry Clin Psychopharmacol. 2017; 27: 84-92
        • Sood E.D.
        • Mendez J.L.
        • Kendall P.C.
        Acculturation, Religiosity, and ethnicity Predict Mothers’ causal Beliefs about separation anxiety disorder and Preferences for help-seeking.
        J Cross-Cultural Psychol. 2010; 43: 393-409
        • Hagaman A.K.
        • Khadka S.
        • Lohani S.
        • Kohrt B.A.
        Suicide in Nepal: A modified psychological autopsy investigation from randomly selected police cases between 2013 and 2015.
        Social Psychiatry Psychiatr Epidemiol. 2017; 52: 1483-1494
        • Maselko J.
        • Patel V.
        Why women attempt suicide: The role of mental illness and social disadvantage in a community cohort study in India.
        J Epidemiol Community Health. 2008; 62: 817
        • Nock M.K.
        • Borges G.
        • Bromet E.J.
        • et al.
        Cross-national prevalence and risk factors for suicidal ideation, plans and attempts.
        Br J Psychiatry. 2018; 192: 98-105
        • Ganguly S.
        • Samanta M.
        • Roy P.
        • et al.
        Patient health questionnaire-9 as an effective tool for screening of depression among Indian adolescents.
        J Adolesc Health. 2013; 52: 546-551
        • Nandakumar A.L.
        • Vande Voort J.L.
        • Nakonezny P.A.
        • et al.
        Psychometric properties of the Patient health questionnaire-9 modified for major depressive disorder in adolescents.
        J Child Adolesc Psychopharmacol. 2019; 29: 34-40
        • Villodas M.T.
        • Pfiffner L.J.
        • Moses J.O.
        • et al.
        The roles of student gender, race, and psychopathology in teachers’ identification of students for services.
        Child Youth Serv Rev. 2019; 107: 104468
        • Burkey M.D.
        • Ghimire L
        • Adhikari RP
        • et al.
        The ecocultural context and child behavior problems: A qualitative analysis in rural Nepal.
        Social Sci Med. 2016; 159: 73-82
        • Radez J.
        • Waite P.
        • Chorpita B.F.
        Using the 11-item version of the RCADS to identify anxiety and depressive disorders in adolescents.
        Res Child Adolesc Psychopathology. 2021; 49: 1241-1257
        • Manfro P.H.
        • Pereira R.B.
        • Rosa M.
        • et al.
        Adolescent depression beyond DSM definition: A network analysis.
        European Child & Adolescent Psychiatry, 2021
        • Ottman K.
        • Wahid S.S.
        • Flynn R.
        • et al.
        Defining culturally compelling mental health interventions: A qualitative study of perspectives on adolescent depression in Lagos, Nigeria.
        SSM - Ment Health. 2022; 2: 100093
        • Viduani A.
        • Benetti S.
        • Martini T.
        • et al.
        Social isolation as a core feature of adolescent depression: A qualitative study in Porto Alegre, Brazil.
        Int J Qual Stud Health Well-being. 2021; 16: 1978374
        • Canino G.
        • Alegría M.
        Psychiatric diagnosis – is it universal or relative to culture?.
        J Child Psychol Psychiatry. 2008; 49: 237-250