Evaluating Kupferstein’s claims of the relationship of behavioral intervention to PTSS for individuals with autism

Purpose – Kupferstein (2018) surveyed 460 respondents and found that 46 percent of respondents met the diagnostic threshold for posttraumatic stress disorder after exposure to applied-behavior-analysis-based intervention. The purpose of this paper is to provide an evaluation a critical analysis of Kupferstein (2018) including the experimental methods and discussion of the results. Design/methodology/approach – The authors evaluated the Kupferstein’s methodological rigor with respect to the use of hypothesis testing, use of indirect measures, selection of respondents, ambiguity in definitions, measurement system, and framing of the experimental question when conducting the correlational analysis in addition to Kupferstein’s analysis and discussion of the results. Findings – Based upon the analysis, Kupferstein’s results should be viewed with extreme caution due to several methodological and conceptual flaws including, but not limited to, leading questions used within a non-validated survey, failure to confirm diagnosis, and incomplete description of interventions. Originality/value – It is the authors’ hope that this analysis provides caregivers, clinicians, and service providers with a scientific lens which will useful in viewing the limitations and methodological flaws of Kupferstein.

Applied behavior analysis (ABA) constitutes one of three branches of the science of behavior analysis, the other two being the experimental analysis of behavior and behaviorism, or the philosophy of behavior (Cooper et al., 2007). As a science, ABA can be described as a systematic approach in understanding the behavior of social interest. As a practice, ABA has been referred to as the systematic application of behavior analytic principles to improve socially important behaviors. One of the most common applications of ABA has been the development of ABA-based interventions to improve the overall quality of the lives of individuals diagnosed with autism spectrum disorder (ASD) (Smith, 2012). Since the 1950s, there have been numerous empirical studies demonstrating the value of comprehensive ABA-based interventions for individuals diagnosed with ASD (e.g. Anderson et al., 1987;Ballaban-Gil et al., 1996;Cohen et al., 2006;Howard et al., 2005;Howard et al., 2014;Lovaas, 1987;Smith et al., 2000;Sallows and Graupner, 2005) and numerous studies demonstrating the effectiveness of specific ABAbased procedures (e.g. behavioral skills training; Miltenberger et al., 2009) to improve deficits associated with an ASD diagnosis (e.g. improved social behaviors; Leaf et al., 2017).
ABA-based interventions represent the largest category of established interventions for individuals diagnosed with ASD (see National Autism Center, 2015 for a complete review of the levels of evidence for interventions for individuals diagnosed with ASD). ABA-based interventions have been demonstrated to result in substantial positive changes in the quality of lives for individuals diagnosed with ASD (e.g. Howard et al., 2014;Linstead et al., 2017;Lovaas, 1987;McEachin et al., 1993), and have been endorsed by multiple organizations (e.g. Autism Speaks, The Association for Behavior Analysis International, The United States Surgeon General, National Institute of Mental Health, The American Psychological Association). Although there have been numerous demonstrations that have demonstrated the effectiveness of ABA-based interventions, there is a benefit in studies that demonstrate potential negative side effects. Since behavioral intervention is based upon the science of behavior, these studies are important for the continual evolution and growth of the field of ABA. When professionals, caregivers, or individuals diagnosed with ASD write critiques, conduct studies showing negative side effects, or conduct research comparing ABA-based interventions to other interventions, the results can truly be informative. These studies can allow behavior analysts, professionals, caregivers, and individuals diagnosed with ASD to understand some perceived and genuine pitfalls of ABA-based interventions. Data obtained from these studies are most informative when they are well designed, the results are clearly demonstrated, the conclusions are based upon objectively collected data, and they are not designed to support a preconceived notion or belief. When studies do not meet these criteria, they become less informative and can be potentially harmful to the field of behavior analysis and for individuals diagnosed with ASD and their families (Critchfield, 2015).
In a recent example of a critique of ABA-based interventions for individuals diagnosed with ASD,  evaluated the "evidence of increased PTSD symptoms in autistics exposed to applied behavior analysis." In this study, Kupferstein surveyed 460 respondents to evaluate a potential correlation between individuals receiving and/or who have received ABA-based intervention and the prevalence of posttraumatic stress symptoms (PTSS). Kupferstein evaluated this correlation through the use of a survey with a self-designed questionnaire. Kupferstein's results were concerning in that 46 percent of respondents met the diagnostic threshold for posttraumatic stress disorder (PTSD) after exposure to an ABA-based intervention. Additionally, this number was much higher than the participants who received interventions with limited to no empirical base (see Jacobson et al., 2005 Given the results of  and problems that might occur, it is necessary to evaluate the scientific nature of the methods and the discussion of the results. Doing so aligns with the purposes of the Autism Special Interest Group (Autism SIG). That is, one purpose of the Autism SIG is to protect individuals diagnosed with ASD and their families from ineffective or harmful treatments. As such, the Executive Council voted to provide an analysis of Kupferstein's study. We hope that this analysis will help inform practitioners, families, and individuals diagnosed with ASD how to interpret these findings. Unfortunately, the results of our analysis identified several methodological and conceptual flaws of Kupferstein's methods and discussion of the results. Each of these concerns outlined in our analysis are described in detail below.

Hypothesis testing bias
In the introduction, Kupferstein (2018) stated, "We hypothesized that exposure to ABA as compared to other autism interventions would be highly correlated with reported PTSS severity, and that lack of exposure to ABA would predict fewer reports in trauma symptoms" (p. 21). Research designed to prove or disprove a hypothesis can lead to advertently or inadvertently developing methods to support or negate that hypothesis. Given the authors documented premise that ABA-based interventions are harmful (e.g. Kupferstein, 2016), it appears as though various decisions with respect to the methods and discussion of the results may have been affected by a bias. Many of these decisions are expanded upon within this paper and included, but are not limited to, the selection of respondents for the survey, the development of a custom survey, and the correlation between survey responses and PTSD diagnostic criteria. Any research attempting to confirm or deny a bias requires careful evaluation. Bias can be ameliorated with the inclusion of safeguards such as independent evaluation, objective measurement, and blind evaluations. Unfortunately, Kupferstein failed to include most of these basic research safeguards which require the reader to take the results with caution.

Indirect measures
Kupferstein (2018) attempted to evaluate potential correlations between ABA-based intervention and PTSS for individuals diagnosed with ASD based upon data collected via survey. Although survey data can provide useful information (e.g. measures of social validity), there are also limitations when using a survey as the main source of data (Cooper et al., 2007;Kazdin, 2011). Surveys that require individuals to report performance and self-report data are notoriously inaccurate (Cooper et al., 2007;Kazdin, 2011). Sources of inaccuracy of the results obtained from self-report can stem from a lack of correspondence between reported and actual responding (i.e. a lack of say-do correspondence) and recency effects to name a few (Cooper et al., 2007;Kazdin, 2011). Further, survey questions can be constructed to produce outcomes benefiting the research question. Leading questions and broadly written questions can inflate or drive specific responding so that a greater number of responses in a category occur (described more later; Loftus, 1975). Finally, and most importantly, data acquired from surveys prevent the identification of causal relations (e.g. if-then relations; Kazdin, 2011). At best, survey data allow for the development of loose correlations, which should not be construed as causation. As such, Kupferstein's attempts to establish a relationship between ABA-based interventions and PTSS (e.g. "[…] we predict that nearly half of ABA-exposed autistic children will be expected to meet the PTSD criteria four weeks after commencing the intervention […]" p. 27) cannot be viewed as a causal relationship. Furthermore, to develop a possible correlational relationship would have required control of other variables (e.g. the use of a control group) that may have contributed to the findings (e.g. maturation, history, testing, selection interaction; Campbell and Stanley, 1963;Kazdin, 2011). Although survey data are used frequently in psychological or educational research, due to the aforementioned limitations, the results have to be taken with caution.

Respondent selection
Kupferstein (2018) included respondents who received a professional diagnosis of ASD, who were self-diagnosed with ASD, and caregivers of an individual diagnosed with ASD. Respondents were required to be at least 18 years of age. Respondents were recruited from a variety of sources including social media outlets, adult gatherings and the Autism Network Research database. A total of 460 respondents were recruited for participant in the survey. The reader should be cautioned about several aspects of the selection of respondents within Kupferstein's study.
First, there was no confirmation of diagnosis for any of the respondents, self-diagnosed or otherwise. In fact, Kupferstein (2018) specifically stated, "Diagnostic reports were not collected nor stored to protect confidentially of the participants, and [the] validity of self-report was presumed" (p. 21). Although it is important to protect confidentially in any study, it does not exceed the need to validate critical variables affecting the interpretation of the results (e.g. are those completing the survey an individual or a caregiver of an individual with an autism diagnosis?). Without validating diagnosis, it is difficult to know if any of the 460 respondents actually were an individual or a caregiver of an individual with an autism diagnosis.
Second, some respondents included individuals who had diagnosed themselves with ASD (i.e. selfdiagnosed). Currently, physicians and psychologists with diagnostic training conduct behavioral evaluations (e.g. Autism Diagnostic Observation Schedule; Lord et al., 2000, Autism Diagnostic Interview™;Lord et al., 1994, Gilliam Autism Rating Scale -Second Edition;Gilliam, 2006) to provide or confirm an autism diagnosis. Furthermore, diagnostic evaluation typically includes a team of professionals (e.g. doctor, speech and language pathologist, psychologist). Professionals who can provide or confirm diagnoses must meet the standards of the Diagnostic and Statistical Manual of Mental Disorders -Fifth Edition (DSM-5) or International Classifications of Diseasestenth edition (ICD-10). It may be entirely possible that these participants could be diagnosed with ASD, and research should continue to be conducted on the validity of self-diagnosis. However, self-diagnosis is not currently a means to obtain or confirm a legitimate ASD diagnosis. As such, these respondents would not meet the inclusion criteria for any studies evaluating the correlation between ABA-based intervention for individuals diagnosed with ASD and any other variable (e.g. PTSS).
Third, 47 percent of the respondents included in  were caregivers of individuals diagnosed with ASD. Caregiver reports of their and their child's experience with an ABA-based intervention can provide invaluable information (e.g. likes and dislikes, satisfaction with outcomes). However, gathering subjective information about ABA-based interventions was not the purpose of the study, and Kupferstein's purpose may leave the information gathered from caregivers invalidated, at best. Moreover, many of the questions sought to obtain information about the private events or covert behavior (e.g. thoughts, visions, nightmares) that occurred for an individual diagnosed with ASD. While these reports are likely not to be discounted by a caregiver (e.g. reports of nightmares are likely to result in consoling and care), it is improbable that a caregiver can provide accurate information for these questions that may be correlated with symptoms of PTSD. As a result, the data collected from nearly half of the participants (i.e. 47 percent) are questionable at best, and should most likely be excluded from the results.

Ambiguity in ABA-based interventions described
Kupferstein's (2018) purpose was to evaluate potential correlations between ABA-based interventions and PTSS for individuals diagnosed with ASD. Given the breadth of ABA-based interventions available (e.g. Lovaas Method, Applied Verbal Behavior, Pivotal Response Training; Romanczyk and McEachin, 2016), the differences between these interventions, and the potential outcomes of such interventions, it would be necessary to define and describe the various ABAbased interventions received by the respondents. Furthermore, without identifying the characteristics of the intervention provided it is unclear if the results are simply correlated with poorly conducted ABA-based interventions (i.e."[…] ABA done badly is, in fact, not really ABA […];" Critchfield, 2015, p. 125). Kupferstein's survey provided respondents an opportunity to list the types of early intervention from a pre-formatted list. However, these types appear to include all ABA-based interventions under one category, "ABA." Other early intervention options only included other commonly used, but not ABA-based, interventions (e.g. DIR/Floortime).
To examine potential correlations between ABA-based interventions and PTSS, the critical variables relating to various ABA-based interventions need to be specified. These variables include, but are not limited to the intensity (number of hours) and duration (number of months) of behavioral intervention; who provided the intervention (e.g. certified and/or qualified behavior analysts); where the intervention occurred; the intervention's focus (e.g. social skills, reduction of aberrant behavior, language development); behavior change technique used (e.g. punishment or reinforcement based); goals of the intervention; participant characteristics before and after intervention; were other interventions occurring simultaneously; caregiver satisfaction; and the cause for termination of intervention. All of these, and likely other, variables are important to determine potential correlations between PTSS and ABA-based interventions or if PTSS were correlated with other extraneous variables (e.g. an eclectic approach or caregiver dissatisfaction). Additionally, and similar to the difficulties with diagnosis, since there was no validation of the survey results (i.e. confirmation that an ABA-based intervention occurred), it is possible that respondents misidentified/mislabeled another therapy as an ABA-based intervention.
Given that the main purpose of  was to evaluate potential correlations between ABA-based interventions and PTSS, we hope this was done to help provide useful information to improve upon potentially harmful aspects of ABA-based interventions. To do so, it would have been critical to provide data on the characteristics of the ABA-based intervention. Providing this data would allow proponents of ABA-based interventions to improve upon the largest category of established interventions for individuals diagnosed with ASD (National Autism Center, 2015). Unfortunately, these data were not provided, which greatly limits the utility of the findings, making it impossible to evaluate a correlation between various ABA-based interventions and PTSS, and may mislead consumers that an intervention with vast amounts of empirical support is harmful rather than effective.

Measurement system
The main variable  attempted to measure was the prevalence of PTSS with individuals diagnosed with ASD. Again, this measure was obtained through responses to questions on a self-designed survey. Responses to these questions were then correlated with the criteria outlined by the DSM-5 for a PTSD diagnosis. Said differently, Kupferstein attempted to correlate subjectively reported measures of PTSS on Kupferstein's own developed evaluation with the PTSD diagnosis criteria. Subjectively classifying measures of PTSS to align them with a PTSD diagnosis creates several problems when interpreting the results.
First, similar to obtaining an ASD diagnosis, the only way for an individual to receive a PTSD diagnosis is from a doctor with experience in mental illnesses, such as a psychiatrist or a psychologist, who objectively use the criterion described in the DSM-V and ICD-10. However,  appears to make the leap from subjective measures of PTSS to a PTSD diagnosis when describing the results (e.g. Figure 3 providing a regression analysis of PTSD severity; Kupferstein, 2018, p. 25). While it is entirely possible that the respondents reported the presence of PTSS, the results should not be interpreted as a method for PTSD diagnosis or a correlation between receiving ABA-based interventions and receiving a PTSD diagnosis.
Second,  developed a survey that modeled questions from an existing survey (i.e. PCL-5; Weathers et al., 2013). It is important to note that the PCL-5 has three different versions: military, civilian, and specific. Also, the PCL-5 is not used to provide a formal diagnosis PTSD, rather, it is used as a monitoring system while receiving treatment, as a screening tool, and, at best, making a provisional PTSD diagnosis (Weathers et al., 2013). Kupferstein did not specify which of these three was used to develop the questions in the survey she used or if the questions she developed demonstrate the same internal consistency, test-retest consistency, and convergent and discriminant validity of the PCL-5 as demonstrated with veterans (e.g. Bovin et al., 2016). Therefore, one must question the validity of the survey used and, as a result, the results obtained from the survey.
Third, examples of the questions Kupferstein (2018) provided raise concerns. For example, one of the questions in the survey was, "When you were receiving therapy, were you embarrassed of the therapy you were receiving?" (p. 23). Questions such as this are referred to as a "leading" question, in which the wording of the question leads the respondent to provide a specific answer. The use of leading questions, such as the one mentioned above, within the survey may prevent accurate assessment of the results of the survey (Loftus, 1975). Also, questions such as "Do you believe you have met the therapist's goals yet?" (Kupferstein, 2018, p. 22) imply the therapist in ABA-based interventions is the sole determiner of goals. This is a misrepresentation of ABA-based interventions, which also suggest a potential bias, that use a myriad of methods to determine goals of programming including input from individual receiving the intervention (Bannerman et al., 1900;Van Houten et al., 1988).
Taken together, the use of subjective measures of PTSS, development of a survey which has not been validated as the main source of data, and the misleading nature of the questions all create major methodological flaws that need to be considered when interpreting  results.

Introductory framework
A final concern stems from Kupferstein's (2018) introductory framework, which attempts to erroneously align ABA-based interventions as harmful for individuals diagnosed with ASD. Kupferstein includes a discussion of the effects of shock on the behavioral functioning of rats. It is unclear how the discussion of shock with rats applies to the authors' purpose for the study. It is possible that Kupferstein was attempting to align ABA-based intervention with shock, and, in doing so, misleads readers to assume that ABA-based interventions either include the use of shock, which does not align with the present literature base, or that shock and ABA-based interventions yield the same effects on humans and rats, which does not align with the present literature base. This attempted linkage, along with others, misrepresents ABA-based interventions that many individuals diagnosed with ASD receive. Additionally, the authors transition immediately from an inaccurate description of ABA-based interventions to a discussion of potentially traumatic events (PTEs). This appears to be an attempt to align ABA-based interventions with PTEs without evidence or cause. Also, given the complexities of ABA-based interventions, it remains unclear what components of ABA-based interventions lead to or functions as a PTE. For instance, could the implementation of a functional behavioral assessment, which are designed in keep an individual free from harm, function as a PTE?  concluded that "individuals exposed to ABA had a 46 percent likelihood of indicating PTSS" (p. 27). Furthermore, Kupferstein (2018) "predict[ed] that nearly half of ABAexposed autistic children will be expected to meet the PTSD criteria four weeks after commencing the intervention; if ABA intervention persists, there will tend to be an increase in parent satisfaction despite no decrease in PTSS severity" (p. 27). These are distressing conclusions, and it is likely that many will contact this study and Kupferstein's conclusions. As part of the Autism SIG's mission, we deemed it necessary to take a closer look at Kupferstein's methods and conclusions. Based upon our analysis, the results and claims based upon those results should, at the least, be viewed with caution and could be potentially harmful for individuals diagnosed with ASD and their families. Kupferstein's study included major methodological and conceptual flaws paired with what appears to be a biased analysis which led to striking claims with little to no evidence to support these claims.

Conclusions
Numerous studies have documented the positive outcomes of ABA-based interventions for individuals diagnosed with ASD (e.g. Anderson et al., 1987;Ballaban-Gil et al., 1996;Cohen et al., 2006;Howard et al., 2005;Howard et al., 2014;Leaf et al., 2011;Lovaas, 1987;Smith et al., 2000;Sallows and Graupner, 2005). Nonetheless, the mere presence of studies such as Kupferstein (2018) highlights a major predicament for those within the field of ABA and ASD intervention. Even with the tremendous success of ABA-based interventions for individuals diagnosed with ASD, it is important to frequently assess the acceptability of our interventions. Those within the field should take every accusation that ABA-based interventions could be potentially traumatic for the individuals receiving those interventions seriously. As such, we encourage research to assess consumers' opinions of ABA-based interventions as well as research which accurately measures potential positive and negative collateral effects of ABA-based interventions without bias or prejudice. This kind of research would help applied behavior analysts to continue to refine methods and improve ABA-based interventions for individuals diagnosed with ASD. It is unfortunate that methodological and other flaws prevent Kupferstein's results from providing this information.
Perhaps the most concerning possibility resulting from  is the potential for families to avoid seeking out and receiving what has been documented as the largest category of established interventions for individuals diagnosed with ASD (see National Autism Center, 2015 for a complete review of the levels of evidence of interventions for individuals diagnosed with ASD). The dramatic and startling claims made within Kupferstein's study could create the premise to deny families effective intervention and/or turn families away from obtaining ABA-based interventions. We hope that this analysis of Kupferstein' study highlights the limitations and methodological flaws that bring the results and claims based on those results into question. Based upon this analysis, it our contention that service providers, behavior analysts, funding agencies, and parents should carefully and objectively evaluate this study prior to avoiding making recommendations for ABA-based interventions for individuals diagnosed with ASD based upon the results.