In United States v. Johnson, (S5) 16 Cr. 281 (PGG), 2019 WL 1130258 (S.D.N.Y. Mar. 11, 2019), the Government brought firearms-related charges against Latique Johnson, the alleged leader of the New York and Pennsylvania street gang “Blood Hound Brims.” One indictment alleged that he shot at rival gang members in a restaurant in the Bronx. New York City Police Department Detective Jonathan Fox concluded that “toolmark identification” analysis showed that bullets collected from the restaurant were fired from an AK 47 semi-automatic assault rifle that an undercover officer purchased from a former gang member a year-and-a-half later (and that was rumored to be the one Johnson used in the restaurant).
Johnson moved before trial to prevent Detective Fox from testifying, or at least to bar an opinion that the ammunition was fired from the AK 47. During the trial, the court conducted a hearing. Detective Fox testified that the premise for “what we [microscopists] do” is “that the tools that are used to manufacture ... firearms leave marks on the inside of the firearms that are unique to that particular tool.”
The court did not question the premise that characteristics are “individual characteristics” that must be “unique” when viewed under a microscope. Why insist on empirical proof of that theoretical proposition? Surely, the marks made by guns on ammunition are unique at the molecular level. But that intuition proves too much. It means that every shell or cartridge case fired from the same gun will be unique on that scale.
So the claim of uniqueness has to pertain to features as they actually are measured or perceived, and that claim requires empirical verification. Nevertheless, the belief in universal uniqueness is not critical to the work of firearms examiners. They could be incredibly accurate in associating expended ammunition with specific guns even if the high level of similarity between ammunition components from test fires and components recovered from a crime scene is not quite unique with respect to every gun that has ever existed.
In Johnson, the court found that Detective Fox followed the Association of Firearm and Tool Mark Examiners’ “Theory of Identification as it Relates to Tool Marks.” This theory of how to interpret marks requires “sufficient agreement” for a positive association. “Sufficient agreement” comes from the examiner’s sense that the level of agreement is greater than that for different guns and is within the range for ammunition from the same gun. It “means that the agreement of individual characteristics is of a quantity and quality that the likelihood another tool could have made the mark is so remote as to be considered a practical impossibility.” 1/
In challenging the AFTE theory and its application, Johnson relied “primarily on the 2008 National Research Council Report, Ballistic Imaging ... ; the 2009 National Research Council Report, Strengthening Forensic Science in the United States: A Path Forward ... ; and the 2016 report of the President’s Council of Advisors on Science and Technology, Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods (2016) ... .” The court recognized that these reports “challenge, to varying degrees, the scientific basis for toolmark identification analysis.”
After summarizing the statements in the three reports on the limited scientific knowledge of the accuracy of firearms-mark identification 2/ and reviewing the federal case law, 3/ the court “conclude[d] that toolmark identification analysis — at least as performed by Detective Fox — is sufficiently reliable to be presented to the jury.” In reaching this result, the court considered the canonical “Daubert factors.”
"Testability"
First, it found sufficient testing of the ability of examiners to reach the correct results in experiments and proficiency tests. According to the PCAST report, the published experiments were not properly designed and hence could not establish validity. As for proficiency tests, the court later recognized that they may not be representative of casework and they were not blind. So although the AFTE theory surely is testable, the extent to which it has been adequately tested remains open to debate despite the court's conclusion.
Oddly, the court also proposed that “photographic documentation and independent review” in case work tests the theory or practice. Taking pictures is useful, but it does not validate the process, and even blind verification only tests consistency between two examiners. Such reproducibility is not the same as accuracy, and validation requires comparing the examiners’ judgments to the true state of affairs — what forensic scientists sometimes call “ground truth.”
Peer Review and Publication
Second, the court determined that there was adequate “peer review and publication” — “mostly” in a journal of the Association of Firearm and Tool Mark Examiners. This part of the opinion is perfunctory. The court accepted at face value the assurance in a 2003 AFTE Journal article that the journal uses “experts within the scientific community” for “technical review.”
Academic and other critics have complained that the publication is basically a trade journal without peer review by qualified scientists. For example, one forensic scientist wrote that
The AFTE Journal ... is not freely available and requires written testimony [sic] from existing AFTE members. It has extremely limited dissemination beyond the members of AFTE — it is found only in 18 libraries of the 72,000 libraries listed in World CAT, the largest catalog of library materials — and completely lacks integration with any of the voluminous networks for the production and exchange of scientific research information. The journal engages in peer review that is neither blind nor draws on an extensive network of researchers. This is not an attitude in keeping with the openness that is a part of any true scientific research culture. 4/That was 2014. Today, the journal is in only nine more libraries. Only sixteen university libraries make it available.
Courts evaluating the extent of publication and peer review may wish to consider the views of the National Commission on Forensic Science. This Commission, formed by the Department of Justice and the National Institute of Standards and Technology, issued guidance on what constitutes “scientific literature” for supporting forensic science and practice. The Commision’s criteria include publication “in a journal that utilizes rigorous peer review with independent external reviewers to validate the accuracy in its publications and their overall consistency with scientific norms of practice.”
AFTE emphasizes that it has long used a formal peer review system. A position paper published in its journal in 2015 commented as follows:
While it is unclear what the NCFS considers “rigorous” and “independent”, all submissions to the AFTE Journal undergo a thorough two-stage review process conducted by subject matter experts who are members of the AFTE Editorial Committee. Occasionally, a reviewer may need to enlist the knowledge of an expert outside of the field to complete a review; however, it would be extremely difficult to use only external reviewers because most qualified potential reviewers also tend to be AFTE members. Research manuscripts submitted to the AFTE Journal are independently reviewed by at least two reviewers from the Editorial Committee ... .
Controlling Standards
Third, the court expressed more hesitation in finding “controlling standards.” It wrote that
[B]oth courts and the scientific community have voiced serious concerns about the “sufficient agreement” standard, characterizing it as “tautological,” “wholly subjective,” “circular,” “leav[ing] much to be desired,” and “not scientific.” The Court shares some of these concerns.Nonetheless, Judge Gardephe deemed “photographic documentation and verification requirements” and “extensive AFTE training and proficiency testing” to be controlling standards.
This logic is hard to follow. Record-keeping, verification, training, and proficiency testing all are critical components of a quality assurance system. But they are not standards that control how a subjective judgment is made. Detective Fox was clear about the degree of subjectivity. He "stated that ... he employs a holistic approach incorporating his 'training as a whole' and his experience 'based on all the cartridge casings and ballistics that [he] ha[s] identified and compared.'" Still, the court managed to extract "certain principles that ground his conclusions":
For example, the CMS standard — six consecutive matching striations or two groups of three matching striations — represents a “bottom standard” or a floor for declaring a match. Detective Fox will not declare that “sufficient agreement” exists unless microscopic examination reveals a toolmark impression with one area containing six consecutive matching individual characteristics, or two areas with three consecutive matching individual characteristics. ... Detective Fox’s analysis does not end at that point, however. Instead, Detective Fox goes on to examine every impression on the ballistics evidence. “All these lines should match,” as well, and if they do not, Detective Fox will not find “sufficient agreement.”“These criteria.” the court concluded, “provide standards for Detective Fox’s findings as to “sufficient agreement.”
"Rate of Error"
Fourth, the court regarded proficiency test data as suggestive of a modest error rate and added that “even accepting the PCAST Report’s assertion that the error rate could be as high as 1 in 46, or close to 2.2%, such an error rate is not impermissibly high.” The court did not maintain that the one unpublished and unreplicated study of cartridge shell casings that PCAST used to generate this figure (for a Ruger pistol) provided a precise estimate of a false-positive error rate. Somewhat limply, it concluded “that the absence of a definite error rate for toolmark identification does not require that such evidence be precluded.” The judge did not discuss PCAST's recommendation that the upper-bound error rate be presented to the jury.
General Acceptance
Finally, the court spent little energy on the issue of general acceptance. Assuming that the relevant scientific community was limited to “forensic scientists, and firearms examiners in particular,” it perceived “no dispute.”
* * *
In denying the motion to exclude or limit the testimony, the court ironically saw the fact that the problems with firearms identifications are apparent as favoring — or at least not inhibiting — admission. It described “the weaknesses in the methodology of toolmark identification analysis” as “readily apparent,” “as discussed at length in the scientific literature,” and as “not particularly complicated or difficult to grasp.” Thus, they “are likely to be understood by jurors if addressed on cross-examination.”
At the same time, Judge Gardephe did not approve of all types of firearms-marks testimony. He recognized that overclaiming can occur; however, he saw no risk of it happening in this case:
Having heard Detective Fox’s testimony at the Daubert hearing, it is clear that he does not intend to assert — and the Government does not intend to elicit — any particular degree of certainty as to his opinions regarding the ballistics match. ... Indeed, Detective Fox’s repeated concession at the Daubert hearing that his conclusions are “based on [his] subjective opinion” stands in stark contrast to the “tendency of [other] ballistics experts ... to make assertions that their matches are certain beyond all doubt.” ... Detective Fox also testified that he “would never” state his conclusion that ballistics evidence matches to a particular firearm “to the exclusion of all other firearms ... in a court proceeding[,] ... because I haven’t looked at all other firearms.”Although the last concession is refreshing, it creates a puzzle -- Are "individual characteristics" truly individual? If the “individual characteristics” obviously matched — as apparently they did 5/ — why shouldn't the examiner testify that they exclude every other gun? Isn’t it “practically impossible,” to use the AFTE phrase, that another AK 47 fired the bullets?
NOTES
- The current version of the official theory is at https://afte.org/about-us/what-is-afte/afte-theory-of-identification.
- The opinion also quoted the trivially true statements in the reports that in describing the scientific status of the pattern-identification methods, the groups were not themselves taking a stand on the legal question of their admissibility.
- The district court wrote that
In assessing reliability, ‘the district court must focus on the principles and methodology employed by the expert, without regard to the conclusions the expert has reached or the district court’s belief as to the correctness of those conclusions.’ Amorgianos v. Nat’l R.R. Passenger Corp., 303 F.3d 256, 266 (2d Cir. 2002).
In General Electric Co. v. Joiner, 522 U.S. 136 (1997), however, the Supreme Court wrote that “conclusions and methodology are not entirely distinct from one another,” and Rule 702(d) specifies that “the expert has reliably applied the principles and methods to the facts of the case.” - David Klatzow, Justice Denied: Role of Forensic Science in the Miscarriage of Justice (2014).
- The court was impressed that “[t]he ‘matching’ ... is stark, even to an untrained observer.”
- David H. Kaye, Firearm-Mark Evidence: Looking Back and Looking Ahead, 68 Case W. Res. L. Rev. 723-45 (2018)
No comments:
Post a Comment