Saturday, June 15, 2024

Volunteer Bias in Interlaboratory Studies

The National Institute for Standards and Technology (NIST) is soliciting laboratories to join an "interlaboratory study" of CG-MS (gas chromatography mass spectrometry) for seized drug analysis. The announcement that I received, in abbreviated form, reads:

Forensic Science Quality Assurance Program
Seized Drugs General Method for GC-MS Reporting Limits Study

. . .
Study Design, Purpose, and Rationale
The goals of the study are 1) to capture the range of methods, instrumentation, and analytical approaches used in the community, 2) investigate mass spectral variability across methods, and 3) investigate how different reporting practices effect the limit of seized drug reporting.
Timeline and Commitment
Registration is currently open and will close on July 5, 2024. To participate in this study, laboratories must be accredited forensic laboratories based in the United States and have a valid Schedule I & II DEA license, a validated seized drug screening method using GC-MS, and a documented reporting practice. To be considered for the study, participants will be required to complete a pre-study questionnaire pertaining to the method that will be used for sample analysis. After acceptance into the study, participants will be provided a kit of 10 solutions containing mixtures of controlled substances and asked to analyze the solutions and report whether the analytes are present above their established reporting thresholds. Participants will also be required to report chromatographic peak height/area and retention time of each peak and provide the raw datafile from each run. Standards used for comparison will also be reported. [D]ue to a limited number of available kits, completion of the pre-study questionnaire does not guarantee acceptance into the study. . . .
Publication of Results
Upon closure of data entry, laboratories will receive a preliminary report containing a summary of reported data, consensus results, and a summary of analytes present in each mixture. [A] final report . . . will be made publicly available by Spring 2025. . . . NIST will not knowingly reveal laboratory identities associated with study results.
For questions, contact andrea.yarberry@nist.gov
To signup, go to: https://forms.gle/gPDU9aENHguPkw1D7

The effort is laudable, but one might ask why NIST is not beginning with a sampling frame of laboratories created to represent "the community" and then drawing a probability sample from this list. Will the laboratories that notice the announcement and ask to participate present the full "range of methods, instrumentation, and analytical approaches used in the community"? Will the volunteer sample be skewed toward higher quality labs? Will it include all the "different reporting practices [that] effect [sic] the limit of seized drug reporting [whatever this "limit" denotes in the population of laboratories]"?

Rigor in sampling may not be required to answer certain questions, but it seems relevant to determining what the sign-up form refers to as "the current landscape of GC-MS methods and associated reporting practices and how those factors effect [sic] the concentration of drug that is/is not ultimately reported." Certainly, it should be a consideration for the legal community if and when the results of the study are presented as an indication of "the known or potential rate of error" for GC-MS analysis as practiced in forensic-science laboratories (Daubert v. Merrell Dow Pharm., 509 U.S. 579, 594 (1993)).