2017 YRBS Mischievous Responders

Resource Type
  • Cimpian, Joseph (New York University)
Publication Date
Funding Reference
  • United States Department of Education. Institute of Education Sciences
    • Award Number: R305B170009
  • Abstract

    The data for this project correspond to Cimpian and Timmer's AERA Open paper on mischievous responders in estimates of LGBQ–heterosexual disparities using the 2017 YRBS data. Their abstract is:
    Although numerous survey-based studies have found that students who identify as lesbian, gay, bisexual, or questioning (LGBQ) have elevated risk for many negative academic, disciplinary, psychological, and health outcomes, the validity of the types of data on which these results rest have come under increased scrutiny. Over the past several years, a variety of data-validity screening techniques have been used in attempts to scrub datasets of “mischievous responders,” youth who systematically provide extreme and untrue responses to outcome items and who tend to falsely report being LGBQ. We conducted a pre-registered replication of Cimpian et al. (2018) with the 2017 Youth Risk Behavior Survey (YRBS) to (1) estimate new LGBQ–heterosexual disparities on 20 outcomes; (2) test a broader, mechanistic theory relating mischievousness effects with a feature of items (i.e., item response-option extremity); and (3) compare four techniques used to address mischievous responders. Our results are consistent with Cimpian et al.’s findings that potentially mischievous responders inflate LGBQ–heterosexual disparities, do so more among boys than girls, and affect outcomes differentially. For example, we find that removing students suspected of being mischievous responders can cut male LGBQ–heterosexual disparities in half overall, and can completely or mostly eliminate disparities in outcomes including fighting at school, driving drunk, and using cocaine, heroin, and ecstasy. Methodologically, we find that some methods are better than others at addressing the issue of data integrity, with boosted regressions coupled with data removal leading to potentially very large decreases in the estimated of LGBQ–heterosexual disparities, but regression adjustment having almost no effect. While the empirical focus of this paper is on LGBQ youth, the issues discussed are relevant to research on other minority groups and youth generally, and speak to survey development, methodology, and the robustness and transparency of research.
  • Cimpian, Joseph, and Jennifer Timmer. “Large-Scale Estimates of LGBQ–Heterosexual Disparities in the Presence of Potentially Mischievous Responders: A Pre-Registered Replication and Comparison of Methods.” AERA Open, n.d.

Update Metadata: 2019-10-23 | Issue Number: 1 | Registration Date: 2019-10-23