Maryland Supreme Court Resists "Unqualified" Firearms-toolmark Testimony

This week, the Maryland Supreme Court became the first state supreme court to hold, unequivocally, that a firearms-toolmark examiner may not testify that a bullet was fired from a particular gun without a disclaimer indicating that source attribution is not a scientific or practical certainty. \1/ The opinion followed two trials, \2/ two evidentiary hearings (one on general scientific acceptance \3/ and one on the scientific validity of firearms-toolmark identifications \4/) and affidavits from experts in research methods or statistics. The Maryland court did not discuss the content of the required disclaimer. It merely demanded that the qualified expert's opinion not be "unqualified." In addition, the opinions are limited to source attributions via the traditional procedure of judging presumed "individual" microscopic features with no standardized rules for concluding that the markings match.

The state contended that Kobina Ebo Abruquah murdered a roommate by shooting him five times, including once in the back of the head. A significant part of the state's case came from a firearms examiner for the Prince George’s County Police Department. The examiner "opined that four bullets and one bullet fragment ... 'at some point had been fired from [a Taurus .38 Special revolver belonging to Mr. Abruquah].'" A bare majority of four justices agreed that admission of the opinion was an abuse of the trial court's discretion. Three justices strongly disputed this conclusion. Two of the three opinions in the case included tables displaying counts or percentages from experiments in which analysts compared sets of either bullets or cartridge casings fired from a few types of handguns to ascertain how frequently their source attributions and exclusions were correct and how often they were wrong.

There is a lot one might say about these opinions, but here I attend only to the statistical parts. \5/ As noted below (endnotes 3 an 4), neither party produced any statisticians or research scientists with training or extensive experience in applying statistical methods. The court did not refer to the recent, burgeoning literature on "error rates" in examiner-performance studies. Instead, the opinions drew on (or questioned) the analysis in the 2016 report of the President's Council of Advisors on Science and Technology (PCAST). The report essentially dismissed the vast majority of the research on which one expert for the state (James Hamby, a towering figure in the firearms-toolmark examiner community) relied. These studies, PCAST explained, usually asked examiners to match a set of questioned bullets to a set of guns that fired them. 

A dissenting opinion of Justice Steven Gould argued—with the aid of a couple of probability calculations—that the extremely small number of false matches in these "closed set" studies demonstrated that examiners were able to perform much better than would be expected if they were just guessing when they lined up the bullets with the guns. \6/

That is a fair point. "Closed set" studies can show that examiners are extracting some discriminating information. But they do not lend themselves to good estimates of the probability of false identifications and false exclusions. For answering the "error rate" question in Daubert, they indicate lower bounds—the conditional error probabilities for examiners under the test conditions could be close to zero, but they could be considerably larger.

More useful experiments simulate what examiners do in cases like Abruquah—where they decide whether a specific gun fired the bullet that could have come from guns beyond those in an enumerated set of known guns. To accomplish this, the experiment can pair test bullets fired from a known gun with a "questioned" bullet and have the examiner report whether the questioned bullet did or did not travel through the barrel of the tested gun.

The majority opinion, written by Chief Justice Matthew Fader, discussed two such experiments known as "Ames I" and "Ames II" (because they were done at the Ames National Laboratory, "a government-owned, contractor-operated national laboratory of the U.S. Department of Energy, operated by and located on the campus of Iowa State University in Ames, Iowa"). The first experiment, funded by the Department of Defense and completed in 2014, "was designed to provide a better understanding of the error rates associated with the forensic comparison of fired cartridge cases." The experiment did not investigate performance with regard to toolmarks on the projectiles (the bullets themselves) propelled from the cases, through the barrel of a gun, and beyond. Apparently referring to the closed-set kind of studies, the researchers observed that "[five] previous studies have been carried out to examine this and related issues of individualization and durability of marks ... , but the design of these previous studies, whether intended to measure error rates or not, did not include truly independent sample sets that would allow the unbiased determination of false-positive or false-negative error rates from the data in those studies." \7/

However, their self-published technical report does not present the results in the kind of classification table that statisticians would expect. Part of such a table is on this blog: \8/ 

The researchers enrolled 284 volunteer examiners in the study, and 218 submitted answers (raising an issue of selection bias). The 218 subjects (who obviously knew they were being tested) “made ... l5 comparisons of 3 knowns to 1 questioned cartridge case. For all participants, 5 of the sets were from known same-source firearms [known to the researchers but not the firearms examiners], and 10 of the sets were from known different-source firearms.” 3/ Ignoring “inconclusive” comparisons, the performance of the examiners is shown in Table 1.

Table 1. Outcomes of comparisons (derived from pp. 15-16 of Baldwin et al.)
	~S	S
–E	1421	4	1425
+E	22	1075	1097
	1443	1079
–E is a negative finding (the examiner decided there was no association). +E is a positive finding (the examiner decided there was an association). S indicates that the cartridges came from bullets fired by the same gun. ~S indicates that the cartridges came from bullets fired by a different gun.

False negatives. Of the 4 + 1075 = 1079 judgments in which the gun was the same, 4 were negative. This false negative rate is Prop(–E |S) = 4/1079 = 0.37%. ("Prop" is short for "proportion," and "|" can be read as "given" or "out of all.") Treating the examiners tested as random samples of all examiners of interest, and viewing the performance in the experiment as representative of the examiners' behavior in casework with materials comparable to those in the experiment, we can estimate the portion of false negatives for all examiners. The point estimate is 0.37%. A 95% confidence interval is 0.10% to 0.95%. These numbers provide an estimate of how frequently all examiners would declare a negative association in all similar cases in which the association actually is positive.Instead of false negatives, we also can describe true negatives, or specificity. The observed specificity is Prop(E|~S) = 99.63%. The 95% confidence interval around this estimate is 99.05% to 99.90%.

False positives. The observed false-positive rate is Prop(+E |~S) = 22/1443 = 1.52%, and the 95% confidence interval is 0.96% to 2.30%. The observed true-positive rate, or sensitivity, is 98.48%, and its 95% confidence interval is 97.7% to 99.04%.

Taken at face value, these results seem rather encouraging. On average, examiners displayed high levels of accuracy, both for cartridge cases from the same gun (better than 99% specificity) and from different guns (better than 98% sensitivity).

I did not comment on the implications of the fact that analysts often opted out of the binary classifications by declaring that an examination was inconclusive. This reporting category has generated a small explosion of literature and argumentation. There are two extreme views. Some organizations and individuals maintain that just because specimens did or not come from the same source, the failure to discern which of these two states of nature applies is an error. A more apt term would be "missed signal"; \9/ it is hardly obvious that Daubert's fleeting reference to "error rates" was meant to encompass not only false positives and negatives but also test results that are neither positive nor negative. At the other pole are claims that all inconclusive outcomes should be counted as correct in computing the false-positive and false-negative error proportions seen in an experiment. 

Incredibly, the latter is the only way in which the Ames laboratory computed the error proportions. I would like to think that had the report been subject to editorial review at a respected journal, a correction would have been made. Unchastened, the Ames laboratory again only counted inconclusive responses as if they were correct when it wrote up the results of its second study.

This lopsided treatment of inconclusives was an issue in Abruquah. The majority opinion described the two studies as follows (citations and footnotes omitted):

Of the 1,090 comparisons where the “known” and “unknown” cartridge cases were fired from the same source firearm, the examiners [in the Ames I study] incorrectly excluded only four cartridge cases, yielding a false-negative rate of 0.367%. Of the 2,180 comparisons where the “known” and “unknown” cartridge cases were fired from different firearms, the examiners incorrectly matched 22 cartridge cases, yielding a false-positive rate of 1.01%. However, of the non-matching comparison sets, 735, or 33.7%, were classified as inconclusive, id., a significantly higher percentage than in any closed-set study.

The Ames Laboratory later conducted a second open-set, black-box study that was completed in 2020 ... The Ames II Study ... enrolled 173 examiners for a three-phase study to test for ... foundational validity: accuracy (in Phase I), repeatability (in Phase II), and reproducibility (in Phase III). In each of three phases, each participating examiner received 15 comparison sets of known and unknown cartridge cases and 15 comparison sets of known and unknown bullets. The firearms used for the bullet comparisons were either Beretta or Ruger handguns and the firearms used for the cartridge case comparisons were either Beretta or Jimenez handguns. ... As with the Ames I Study, although there was a “ground truth” correct answer for each sample set, examiners were permitted to pick from among the full array of the AFTE Range of Conclusions—identification, elimination, or one of the three levels of “inconclusive.”

The first phase of testing was designed to assess accuracy of identification, “defined as the ability of an examiner to correctly identify a known match or eliminate a known nonmatch.” In the second phase, each examiner was given the same test set examined in phase one, without being told it was the same, to test repeatability, “defined as the ability of an examiner, when confronted with the exact same comparison once again, to reach the same conclusion as when first examined.” In the third phase, each examiner was given a test set that had previously been examined by one of the other examiners, to test reproducibility, “defined as the ability of a second examiner to evaluate a comparison set previously viewed by a different examiner and reach the same conclusion.”

In the first phase, ... [t]reating inconclusive results as appropriate answers, the authors identified a false negative rate for bullets and cartridge cases of 2.92% and 1.76%, respectively, and a false positive rate for each of 0.7% and 0.92%, respectively. Examiners selected one of the three categories of inconclusive for 20.5% of matching bullet sets and 65.3% of nonmatching bullet sets. [T]he results overall varied based on the type of handgun that produced the bullet/cartridge, with examiners’ results reflecting much greater certainty and correctness in classifying bullets and cartridge cases fired from the Beretta handguns than from the Ruger (for bullets) and Jimenez (for cartridge cases) handguns.

The opinion continues with a description of some statistics for the level of intra- and inter-examiner reliability observed in the Ames II study, but I won't pursue those here. The question of accuracy is enough for today. \10/ To some extent, the majority's confidence in the reported low error proportions (all under 3%) was shaken by the presence of inconclusives: "if at least some inconclusives should be treated as incorrect responses, then the rates of error in open-set studies performed to date are unreliable. Notably, if just the 'Inconclusive-A' responses—those for which the examiner thought there was almost enough agreement to identify a match—for non-matching bullets in the Ames II Study were counted as incorrect matches, the 'false positive' rate would balloon from 0.7% to 10.13%."

But should any of the inconclusives "be treated as incorrect," and if so, how many? Doesn't it depend on the purpose of the studies and the computation? If the purpose is to probe what the PCAST Report neoterically called "foundational validity"—whether a procedure is at least capable of giving accurate source conclusions when properly employed by a skilled examiner—then inconclusives are not such a problem. They represent lost opportunities to extract useful information from the specimens, but they do not change the finding that, within the experiment itself, in those instances in which the examiner is willing to come down on one side or the other, the conclusion is usually correct.

One justice stressed this fact. Justice Gould insisted that

[T]he focus of our inquiry should not be the reliability of the AFTE Theory in general, but rather the reliability of conclusive determinations produced when the AFTE Theory is applied. Of course, an examiner applying the AFTE Theory might be unable to declare a match (“identification”) or a non-match (“elimination”), resulting in an inconclusive determination. But that's not our concern. Rather, our concern is this: when the examiner does declare an identification or elimination, we want to know how reliable that determination is.

He was unimpressed with the extreme view that every failure to recognize "ground truth" is an "error" for the purpose of evaluating an identification method under Daubert. \11/ He argued for error proportions like those used by PCAST: \12/

This brings us to a different way of looking at error rates, one that received no consideration by the Majority ... I am referring to calculating error by excluding inconclusives from both the numerator and the denominator. .... [C]ontrary to Mr. Faigman's unsupported criticism, excluding inconclusives from the numerator and denominator accords with both common sense and accepted statistical methodologies. ... PCAST ... contended that ... false positive rates should be based only on conclusive examinations “because evidence used against a defendant will typically be based on conclusive, rather than inconclusive, determinations.” ... So, far from being "crazy" ... , excluding inconclusives from error rate calculations when assessing the reliability of a positive identification is not only an acceptable approach, but the preferred one, at least according to PCAST. Moreover, from a mathematical standpoint, excluding inconclusives from the denominator actually penalizes the examiner because errors accounted for in the numerator are measured against a smaller denominator, i.e., a smaller sample size.

So what happens when the error proportions for the subset of positive and negative conclusions are computed with the Ames data? The report's denominator is too large, but the resulting bias is not so great in this particular case. For Ames I, Justice Gould's opinion tracks Table 1 above:

With respect to matching [cartridge] sets, the number of inconclusives was so low that whether inconclusives are included in the denominator makes little difference to error rates. Of the 1,090 matching sets, only 11, or 1.01 percent, were inconclusives. Of the conclusive determinations, 1,075 were correctly identified as a match (“identifications”) and four were incorrectly eliminated (“eliminations”). ... Measured against the total number of matching sets (1,090), the false elimination rate was 0.36 percent. Against only the conclusive determinations (1,079), the false elimination rate was 0.37 percent. ...

Of 2,178 non-matching sets, examiners reported 735 inconclusives for an inconclusive rate of 33.7 percent, 1,421 sets as correct eliminations, and 22 sets as incorrect identifications (false positives). ... As a percentage of the total 2,178 non-matching sets, the false positive rate was 1.01 percent. As a percentage of the 1,443 conclusive determinations, however, the false positive rate was 1.52 percent. Either way, the results show that the risk of a false positive is very low

For Ames II,

There were 41 false eliminations. As a percentage of the 1,405 recorded results, the false elimination rate was 2.9 percent. As a percentage of only the conclusive results, the false elimination rate increased to 3.7 percent ... .

... There were 20 false positives. Measured against the total number of recorded results (2,842), the false positive rate was 0.7 percent. Measured against only the conclusive determinations, however, the false positive rate increases to 2.04 percent.

In sum, on the issue of whether a substantial number of firearms-toolmark examiners can generally avoid erroneous source attributions and exclusions when tested as in Ames I and Ames II, the answer seems to be that, yes, they can. Perhaps this helps explain the Chief Justice's concession that "[t]he relatively low rate of 'false positive' responses in studies conducted to date is by far the most persuasive piece of evidence in favor of admissibility of firearms identification evidence." But the court was quick to add that "[o]n balance, however, the record does not demonstrate that that rate is reliable, especially when it comes to actual casework."

Extrapolating from the error proportions in experiments to those in casework is difficult indeed. Is the largely self-selected sample of examiners who enroll in and complete the study representative of the general population of examiners doing casework? Does the fact that the enrolled examiners know they are being tested make them more (or less) careful or cautious? Do examiners have expectations about the prevalence of true sources in the experiment that differ from those they have in casework? Are the specimens in the experiment comparable to those in casework? \13/ Do error probabilities for comparing marks on cartridge cases apply to the marks on the bullets they house? Does it matter if the type of gun used in the experiment is different from the type in the case?

Most of the questions are matters of external validity. Some of them are the subject of explicit discussion in the opinions in Abruquah. For example, Justice Gould rejects, as a conjecture unsupported by the record, the concern that examiners might be more prone to avoid a classification by announcing an "inconclusive" outcome in an experiment than in practice.

To different degrees, the generalizability questions interact with the legal question being posed. As I have indicated, whether the scientific literature reveals that a method practiced by skilled analysts can produce conclusions that are generally correct for evidence like that in a given case is one important issue under Daubert. Whether the same studies permit accurate estimates of error probabilities in general casework is a distinct, albeit related, scientific question. How to count or adjust for inconclusives in experiments is but a subpart of the latter question.

And, how to present source attributions in the absence of reasonable error-probability estimates for casework is a question that Abruquah barely begins to answer. No opinion embraced the defendant's argument that only a limp statement like "unable to exclude as a possible source" is allowed. But neither does the case follow other courts that allow statements such as the awkward and probably ineffectual "reasonable degree of ballistic certainty" for expressing the difficult-to-quantify uncertainty in toolmark source attributions. After Abruquah, if an expert makes a source attribution in Maryland, some kind of qualification or caveat is necessary. \14/ But what will that be?

Toolmark examiners are trained to believe that their job is to provide source conclusions for investigators and courts to use, but neither law nor science compels this job description. Perhaps it would be better to replace conclusion-centered testimony about the (probable) truth of sources conclusions with evidence-centered statements about the degree to which the evidence supports a source conclusion. The Abruquah court wrote that

The reports, studies, and testimony presented to the circuit court demonstrate that the firearms identification methodology employed in this case can support reliable conclusions that patterns and markings on bullets are consistent or inconsistent with those on bullets fired from a particular firearm. Those reports, studies, and testimony do not, however, demonstrate that that methodology can reliably support an unqualified conclusion that such bullets were fired from a particular firearm.

The expert witness's methodology provides "support" for a conclusion, and the witness could simply testify about the direction and magnitude of the support without opining on the truth of the conclusion itself. \15/ "Consistent with" testimony is a statement about the evidence, but it is a minimal, if not opaque, description of the data. Is it all that the record in Abruquah—not to mention the record in the next case—should allow? Only one thing is clear—fights over the legally permissible modes for presenting the outcomes of toolmark examinations will continue.

Notes

1. In Commonwealth v. Pytou Heang, 942 N.E.2d 927 (Mass. 2011), the Massachusetts Supreme Judicial Court upheld source-attribution testimony "to a reasonable degree of scientific certainty" but added "that [the examiner] could not exclude the possibility that the projectiles were fired by another nine millimeter firearm." The Court proposed "guidelines" to allow source attribution to no more than "a reasonable degree of ballistic certainty."
2. The defendant was convicted at a first trial in 2013, then retried in 2018. In 2020, the Maryland Supreme Court changed its standard for admitting scientific evidence from a requirement of general acceptance of the method in the relevant scientific communities (denominated the "Frye-Reed standard" in Maryland) to a more direct showing of scientific validity described in Daubert v. Merrell Dow Pharmaceuticals, 509 U.S. 579 (1993), and an advisory committee note accompanying amendments in 2000 to Federal Rule of Evidence 702 (called the "Daubert-Rochkind standard" in Abruquah).
3. The experts at the "Frye-Reed hearing" were William Tobin (a "Principal of Forensic Engineering International," https://forensicengineersintl.com/about/william-tobin/, and "former head of forensic metallurgy operations for the FBI Laboratory" (Unsurpassed Experience, https://forensicengineersintl.com/); James Hamby ("a laboratory director who has specialized in firearm and tool mark identification for the past 49 years" and who is "a past president of AFTE [the Association of Firearm and Tool Mark Examiners] ... and has trained firearms examiners from over 15 countries worldwide," Speakers, International Symposium on Forensic Science, Lahore, Pakistan, Mar. 17-19, 2020, https://isfs2020.pfsa.punjab.gov.pk/james-edward); Torin Suber ("a forensic scientist manager with the Maryland State Police"); and Scott McVeigh (the firearms examiner in the case).
4. The experts at the supplemental "Daubert-Rochkind hearing" were James Hamby (a repeat performance), and David Faigman, "Chancellor & Dean, William B. Lockhart Professor of Law and the John F. Digardi Distinguished Professor of Law" at the University of California College of the Law, San Francisco.
5. Remarks on the legal analysis will appear in the 2024 cumulative supplement to The New Wigmore, A Treatise on Evidence: Expert Evidence.
6. The opinion gives a simplified example:

   The test administrator fires two bullets from each of 10 consecutively manufactured handguns. The administrator then gives you two sets of 10 bullets each. One set consists of 10 “unknown” bullets—where the source of the bullet is unknown to the examiner—and the other set consists of 10 “known” bullets—where the source of the bullet is known. You are given unfettered access to a sophisticated crime lab, with the tools, supplies, and equipment necessary to conduct a forensic examination. And, like the vocabulary tests from grade school requiring you to match words with pictures, you must match each of the 10 unknown bullets to the 10 known bullets.
   
   Even though you know that each of the unknowns can be matched with exactly one of the knowns, you probably wouldn't know where to begin. If you had to resort to guessing, your odds of correctly matching the 10 unknown bullets to the 10 knowns would be one out of 3,628,800. [An accompanying note 11 explains that: "[w]ith 10 unknown bullets and 10 known bullets, the odds of guessing the first pair correctly are one out of 10. And if you get the first right, the odds of getting the second right are one out of nine. If you get the first two right, the odds of getting the third right are one out of eight, and so on. Thus, the odds of matching each unknown bullet to the correct known is represented by the following calculation: (1/10) x (1/9) x (1/8) x (1/7) x (1/6) x (1/5) x (1/4) x (1/3) x (1/2) x (1/1)."] Even if you correctly matched five unknown bullets to five known bullets and guessed on the remaining five unknowns, your odds of matching the remaining unknowns correctly would be one out of 120. [Note 12: "(1/5) x (1/4) x (1/3) x (1/2) x (1/1)."] Not very promising.
   
   The closed-set and semi-closed-set studies before the trial court—the studies which PCAST discounted—show that if you were to properly apply the AFTE Theory, you would be very likely to match correctly each of the 10 unknowns to the corresponding knowns. See Validation Study; Worldwide Study; Bullet Validation Study. ... Your odds would thus improve from virtually zero (one in 3,628,800) to 100 percent. Yet according to PCAST, those studies provide no support for the scientific validity of the AFTE Theory. ...

7. David P. Baldwin, Stanley J. Bajic, Max Morris & Daniel Zamzow, A Study of False-positive and False-negative Error Rates in Cartridge Case Comparisons, Ames Laboratory, USDOE, Tech. Rep. #IS-5207 (2014), at https://afte.org/uploads/documents/swggun-false-postive-false-negative-usdoe.pdf [https://perma.cc/4VWZ-CPHK].
8. David H. Kaye, PCAST and the Ames Bullet Cartridge Study: Will the Real Error Rates Please Stand Up?, Forensic Sci., Stat. & L., Nov. 1, 2016, http://for-sci-law.blogspot.com/2016/11/pcast-and-ames-study-will-real-error.html
9. David H. Kaye et al., Toolmark-comparison Testimony: A Report to the Texas Forensic Science Commission, May 2, 2022, available at http://ssrn.com/abstract=4108012.
10. I will note, however, that the report apparently strains to make the attained levels for reliability seem high. Alan H. Dorfman & Richard Valliant, A Re-Analysis of Repeatability and Reproducibility in the Ames-USDOE-FBI Study, 9 Stat. & Pub. Pol'y 175 (2022).
11. The opinion attributes this view to

    Mr. Abruquah's expert, Professor David Faigman, [who declared] that "in the annals of scientific research or of proficiency testing, it would be difficult to find a more risible manner of measuring error." To Mr. Faigman, the issue was simple: in Ames I and II, the ground truth was known, thus "there are really only two answers to the test, like a true or false exam[ple]." Mr. Faigman explained that "the common sense of it is if you know the answer is either A or B and the person says I don't know, in any testing that I've ever seen that's a wrong answer." He argued, therefore, that inconclusives should be counted as errors.

12. See also NIST Expert Working Group on Human Factors in Latent Print Analysis, Latent Print Examination and Human Factors: Improving the Practice Through a Systems Approach (David H. Kaye ed. 2012), available at ssrn.com/abstract=2050067 (arguing against counting inconclusives in error proportions that are supposed to indicate the probative value of actual conclusions).
13. Testing examiner performance in the actual flow of cases would help address the last three questions. A somewhat confusing analysis of results in such an experiment is described in a posting last year. David H. Kaye, Preliminary Results from a Blind Quality Control Program, Forensic Sci., Stat. & L., July 9, 2022, http://for-sci-law.blogspot.com/2022/07/preliminary-results-from-blind-quality.html.
14. The court wrote that:

    It is also possible that experts who are asked the right questions or have the benefit of additional studies and data may be able to offer opinions that drill down further on the level of consistency exhibited by samples or the likelihood that two bullets or cartridges fired from different firearms might exhibit such consistency. However, based on the record here, and particularly the lack of evidence that study results are reflective of actual casework, firearms identification has not been shown to reach reliable results linking a particular unknown bullet to a particular known firearm.

15. See, e.g., David H. Kaye, The Nikumaroro Bones: How Can Forensic Scientists Assist Factfinders?, 6 Va. J. Crim. L. 101 (2018).

LAST UPDATED 29 June 2023

Tibbs. Shipp, and Harris on "Meaningul" Peer Review of Studies on Firearms-toolmark Matching

The Supreme Court's celebrated (but ambiguous) opinion in Daubert v. Merell Dow Pharmaceuticals, \1/ was a direct response to a seemingly simple rule--results that are not published in the peer-reviewed scientific literature are inadmissible to prove that a scientific theory or method is generally accepted in the scientific community. The Court unanimously rejected this strict rule--and more broadly, the very requirement of general acceptance--in favor of a multifaceted examination guided by four or five criteria that have come to be known as "the Daubert factors."

But "peer review and publication" lives on--not as a formal requirement, but as one of these factors. Thus, courts routinely ask whether the peer-reviewed scientific literature supports the reasoning or data that an expert is prepared to present at trial. All too often, however, the examination of the literature is cursory or superficial. The temptation, especially for overburdened judges not skilled in sorting through biomedical and other journals, is to check that there are articles on point, and if  the theory has been discussed (critically or otherwise) in the literature, to write that the "peer review and publication" factor supports admission of the testimony.

One area in which this dynamic is apparent is traditional testimony of firearms examiners matching marks from guns to bullets or shell casings. \2/ Defendants have strenuously objected that traditional associations of particular guns to ammunition components is an inscrutable judgment call that does not pass muster under Daubert. Perhaps the most meticulous analysis of this issue comes from an unpublished opinion of Judge Todd Edelman in United States v. Tibbs. \3/ Judge Edelman's discussion of peer review and publication is unusually thorough and may have been penned as an antidote to the strategy in which the government gives the court a laundry list of articles that have discussed the procedure and the court checks off the "peer review and publication" box.

Being an opinion for a trial court (the District of Columbia Superior Court), Tibbs is not binding precedent for that court or any other, but it has not gone unnoticed. Two federal district courts recently reached mutually opposing conclusions about Judge Edelman's analysis of one large segment of the literature cited in support of admitting match determinations--namely, the extensive research reported in the AFTE Journal. ("AFTE" stands for the Association of Firearms and Toolmark Examiners. The organization was formed in 1969 in "recognition of the need for the interchange of information, methods, development of standards, and the furtherance of research, [by] a group of skilled and ethical firearm and/or toolmark examiners" who "stand prepared to give voice to this otherwise mute evidence." \4/)

Tibbs' Analysis of the AFTE Journal

Because of the AFTE Journal's orientation and editorial process, Tibbs did not give "the sheer number of studies conducted and published" there much weight. \5/ Judge Edelman made essentially four points about the journal:

• Contrary to the testimony of the government’s experts, post-publication comments or later articles are not normally considered to be “peer review”;
• The AFTE pre-publication peer review process is “open,” meaning that “both the author and reviewer know the other's identity and may contact each other during the review process”;
• The reviewers who form the editorial board are all “members of AFTE” who may well “be trained and experienced in the field of firearms and toolmark examination, but do not necessarily have any ... training in research design and methodology” and who “have a vested, career-based interest in publishing studies that validate their own field and methodologies”; and
• “AFTE does not make this publication generally available to the public or to ... reviewers and commentators outside of the organization's membership [and] unlike other scientific journals, the AFTE Journal ... cannot even be obtained in university libraries.” \6/

The court contrasted these aspects of the journal’s peer review to a "double-blind" process and observed that the AFTE “open” process was “highly unusual for the publication of empirical scientific research.” \7/ The full opinion, which develops these ideas more completely can be found online.

Shipp

Senior Judge Nicholas Garaufis of the Eastern District of New York was impressed with "this thorough opinion." \8/ His opinion in United States v. Shipp referred to the "several pages analyzing the AFTE Journal's peer review process [that] highlight[] several reasons for assigning less weight to articles published in the AFTE Journal than in other publications" and added that

The court shares these concerns about the AFTE Journal's peer review process. In particular, the court is concerned that the reviewers, who are all members of the AFTE, have a vested, career-based interest in publishing studies that validate their own field and methodologies. Also concerning is the possibility that the reviewers may be trained and experienced in the field of firearms and toolmark identification, but [may] not necessarily have any specialized or even relevant training in research design and methodology. \9/

Harris

In contrast, Judge Rudolph Contreras of the U.S. District Court for the District of Columbia, writing in United States v. Harris, \10/ had nothing complimentary to say about Tibbs. This court defended the AFTE Journal research articles said to demonstrate the validity of firearms-toolmark identification with two rejoinders to Tibbs. First, Judge Contreras maintained that “there is far from consensus in the scientific community that double-blind peer review is the only meaningful kind of peer review.” \11/ This is true enough, but the issue raised by the criticism of “open” review is not whether double-blind review is better than single-blind review (in which the author does not know the identity of the referees) or some other system. It is whether “open” review conducted exclusively by AFTE members is the kind of peer review envisioned as a strong indicator of scientific soundness in Daubert. The factors enumerated in Tibbs make that a serious question.

Second, Judge Contreras observed that the Journal of Forensic Sciences, which uses double-blind review, republished one AFTE study. This solitary event, the Harris opinion suggests, is a “compelling” rebuttal of “the allegation by Judge Edelman in Tibbs that the AFTE Journal does not provide 'meaningful' review." \12/ But Judge Edelman never proposed that every article in the AFTE journal was without scientific merit. Rather, his point was far less extreme. It was merely that courts should not “accept at face value the assertions regarding the adequacy of the journal's peer review process.” \13/ That one article—or even dozens—published in the AFTE Journal could have been published in other journals reveals very little about the level and quality of AFTE review. After all, even a completely fraudulent review process that accepted articles for publication by flipping a coin would result in the publication of some excellent articles—but not because the review process was meaningful or trustworthy. In addition, one might ask whether the very fact that an article had to be republished in a more widely read journal fortifies the fourth point in Tibbs, that the journal’s circulation is too restricted to make its publications part of the mainstream scientific literature. The discussion of peer review and publication in Harris ignores this concern.

Beyond the AFTE Journal

The significant concerns exposed in Tibbs do not prove that the peer-reviewed scientific literature, taken as a whole, undermines firearms identification as commonly practiced. They simply mean that the list of publications over the years in the AFTE Journal may not be entitled to great weight in evaluating whether the scientific literature supports the claim of firearms and toolmark examiners to be able to supply generally accurate and reliable "opinions relative to evidence which otherwise stands mute before the bar of justice." \14/

Fortunately, newer peer-reviewed studies exist, and not all the older research appears in the AFTE Journal. \15/ Thus, the Harris court asserted that

[E]ven if the Court were to discount the numerous peer-reviewed studies published in the AFTE Journal, Mr. Weller's affidavit also cites to forty-seven other scientific studies in the field of firearm and toolmark identification that have been published in eleven other peer-reviewed scientific journals. This alone would fulfill the required publication and peer review requirement. \16/

The last sentence could be misunderstood. As a statement that the 47 studies could be the basis of an scientifically informed judgment about the validity of firearms-toolmark matching, the conclusion is correct. As a statement that checking the "peer review and publication" box on the basis of a large number of studies published in the right places "alone" is a reason to admit the challenged testimony, it would be more problematic. The "required ... requirement" (to the extent Daubert imposes one) is for a substantial body of peer-reviewed papers that form a solid foundation for a scientific assessment of a method. Unless this research literature is actually supportive of the method, however, satisfying the "the required publication and peer review requirement" is not a reason to admit the evidence. 

Do the 47 studies (old and new) in widely accessible, quality journals all show that examiners' opinions derived from comparing toolmarks are consistently correct and stable for the kinds of comparisons made in practice? If so, then it is high time to stop the arguments over scientific validity. If not, if the 47 studies are of varying quality, scope, and relevance to ascertaining how repeatable, reproducible, and accurate the opinions rendered by firearms-toolmark examiners are, then there is room for further analysis of whether and how these experts can provide valuable information for the legal factfinders.

NOTES

1. 509 U.S. 579 (1993),
2. No. 2016 CF1 19431, 2019 D.C. Super. LEXIS 9, 2019 WL 4359486 (D.C. Super. Ct., Sept. 5, 2019).
3. "[T]he process that most firearms examiners use when analyzing evidence" is desctibed in graphic detail in "[t]he Firearms Process Map, which captures the ‘as-is’ state of firearms examination, provides details about the procedures, methods and decision points most frequently encountered in firearms examination." NIST, OSAC's Firearms & Toolmarks Subcommittee Develops Firearms Process Map Jan. 19, 2021, https://www.nist.gov/news-events/news/2021/01/osacs-firearms-toolmarks-subcommittee-develops-firearms-process-map.
4. AFTE Bylaws, Preamble, https://afte.org/about-us/bylaws
5. 2019 D.C. Super. LEXIS 9, at *35. For a decade or so, both legal academics and forensic scientists had pointed to the AFTE Journal as an example of a practitioner-oriented outlet for publications that did not follow the peer review and publication practices of other scientific journals. See, e.g., David H. Kaye, Firearm-Mark Evidence: Looking Back and Looking Ahead, 68 Case W. Res. L. Rev. 723 (2018); Jennifer L. Mnook--n et al., The Need for a Research Culture in the Forensic Sciences, 58 UCLA L. Rev. 725 (2011).
6. 2019 D.C. Super. LEXIS 9, at *32-*33.
7. Id. at *33.
8. United States v. Shipp, 422 F.Supp.3d 762, 776 (E.D.N.Y. 2019).
9. Id. (citations and internal quotation marks omitted). Nevertheless, the court found "sufficient peer review." It wrote that "even assigning limited weight to the substantial fraction of the literature that is published in the AFTE Journal, this factor still weighs in favor of admissibility. Daubert found the existence of peer-reviewed literature important because “submission to the scrutiny of the scientific community ... increases the likelihood that substantive flaws in the methodology will be detected.” Daubert, 509 U.S. at 593. Despite AFTE Journal’s open peer-review process, the AFTE Theory has still been subjected to significant scrutiny. ... Therefore, the court finds that the AFTE Theory has been sufficiently subjected to 'peer review and publication' [outside of the AFTE Journal].” Daubert, 509 U.S. at 594."
10. 502 F.Supp.3d 28 (D.D.C. 2020).
11. Id. at 40.
12. Id.
13. Tibbs, 2019 D.C. Super. LEXIS 9, at *29.
14. AFTE Bylaws, Preamble, https://afte.org/about-us/bylaws.
15. AFTE has sought to remedy at least one complained-of feature of its peer review process. In 2020, it instituted the double-blind peer review that the Harris court found unnecessary. AFTE Peer Review Process – January 2020, https://afte.org/afte-journal/afte-journal-peer-review-process. Whether the qualifications and backgrounds of the journal's referrees have been changed is not apparent from the AFTE website.
16. Harris, 502 F.Supp.3d at 40.

OSAC-approved Testimony

According to the charter of the Department of Commerce's Organization of Scientific Area Committees for Forensic Science, better known as OSAC (but not to be confused with the Department of State's Overseas Security Advisory Council, also known as OSAC), "[t]he aims of the OSAC are to:

• populate the OSAC Registry of Standards
• promote the use of OSAC-endorsed standards by the forensic community, accreditation and certification bodies, and by the legal system
• provide insight on each forensic science discipline’s research and measurement standard needs
• enlist stakeholder involvement from a broad community; and to [sic]
• establish and maintain working relationships with other similar organizations. 1/

OSAC is not a regulatory agency, and it does not review the work of forensic science laboratories or practitioners. Yet one court thought otherwise. In United States v. Lang, 2/ the defendant moved to exclude testimony from "the Government's firearms and toolmark expert—M.L. Cooper." The government wanted "Mr. Cooper ... to testify about what latent prints and DNA are and the factors that influence whether a latent print or DNA are left on a surface." Evidently, the point of the testimony was to explain the absence of fingerprint and DNA evidence in the case because of "the difficulty in obtaining latent prints and DNA from ammunition and the low rate at which such types of trace evidence are recovered."

Senior Judge Raymond L. Finch of the District Court of the Virgin Islands (formerly the court's Chief Judge) denied the motion. His unreported opinion on his pretrial ruling states that

On direct examination, Mr. Cooper testified that he reviewed a Crime Scene Evidence Report prepared by VIPD Officer Don Peter, photographs of the five buckshot shotgun shells recovered from Defendant's apartment, and photographs of the black garbage bag that contained the shells. He also testified that he confirmed his resulting opinion with OSAC (emphasis added).

This was not all that Mr. Cooper, and hence the court, had to say about OSAC. Judge Finch wrote that "Mr. Cooper is currently a member of the Association of Firearms and Toolmark Examiners—an organization that according to Mr. Cooper deals with the recovery of trace evidence—and maintains contact with the Organization of Scientific Area Committees ('OSAC')" (emphasis added). The nature of Mr. Cooper's "contact" with OSAC--or is it AFTE's "contact"--was left unstated, but the court thought it was important enough to add the footnote that "OSAC coordinates the 'development of standards and guidelines for the forensic science community to improve quality and consistency of work in the forensic science community.'"

Relying on an OSAC-approved standard for an opinion is one thing. But there are no standards on how to ascertain from a "review of the Crime Scene Evidence Report and photographs" that police would not be expected to recover a latent fingerprint or an adequate quantity of DNA to analyze, and being in "contact with" OSAC does not tell the court anything about the expertise of a witness or an organization.

Other parts of the opinion are more comprehensible (but not necessarily correct). The court wrote that the testimony "satisfies the reliability test under Daubert" just because it "will be based upon his more than thirty years of experience working in the field of forensic science." Experience can be a good thing, but it is not the scientific validity discussed in Daubert. More plausibly, the court moved away from its equation of experience to science. It stated that Mr. Cooper's methodology did not have to satisfy Daubert after all. "Rather than [applying] a methodology that satisfies the requirements of Daubert," Mr. Cooper would do little more than recount his personal experience with "the recovery of trace evidence" That much, the court insisted, "is entirely appropriate under Rule 702. See United States v. McNeil, 2010 U.S. Dist. LEXIS 290, at *8, 2010 WL 834667 (M.D. Pa. Jan. 5, 2010) ('To the extent that the expert has knowledge of the frequency of firearms without latent prints, the expert [can] testify to that knowledge.')."

NOTES

1. OSAC Charter and Bylaws, Sept. 26, 2019, Version 1.6.
2. Crim. Action No. 2015-0013, 2016 WL 1734087 (D. V. I., Apr. 28, 2016).

State v. Simmer: The First High Court Opinion on "Probabilistic Genotyping"

Last week, the Nebrasks Supreme Court affirmed the admission of testimony about what has been dubbed "probabilistic genotyping." Twelve years ago, in 2007, Joy Blanchard was discovered on the floor of her home. Two knives protruded from her neck. A spindle broken from the banister lay nearby. The cause of death was blunt force trauma and stab wounds to the head and neck. Investigators swabbed the spindle, the handles on both knives, and the interior front door knob. Nine years passed before DNA testing indicated the presence of DNA from Joy's nephew, Charles Simmer, on one of the knife handles and the door knob.

Charged with first-degree murder Simmer filed a pretrial motion to preclude “any and all testimony concerning DNA testing and the results of said testing,” including “identification” and “comparison.” After an evidentiary hearing, the district court denied the motion. At the jury trial, the state presented both "probabilistic genotyping" from Dr. Mark Perlin, the inventor of a complex software program named TrueAllele, that "the match between the knife handle and Simmer was 3.71 thousand times more probable than a coincidental match to an unrelated person, while the match between the doorknob and Simmer was 4.22 quintillion times more probable than a coincidental match to an unrelated person." Simmer was convicted and sentenced to life imprisonment.

What made the "genotyping" "probabilistic" is not that genes were typed. The STRs used in the identification are not genes. They are repeated, noncoding sequences of DNA base pairs (such as TAGATAGATAGATAGATAGATAGATAGA) that are sprinkled within and between genes. Neither Is the typing "probabilistic" because probabilities were quoted. Ordinary DNA typing results are accompanied by "random match probabilities."

Rather, unlike ordinary DNA profiling of a single-source samples, these probabilities were not derived from any one individual's features as inferred by a human examiner. The samples were mixtures of DNA from (at least) two men, and no one could say which two individual profiles were in the mixture. The computer program, however, could estimate the probability of the mixed-up collection of DNA features displaying the measured "peak heights" when coming from DNA from several individuals including the defendant as opposed to the probability of the same data coming from DNA from individuals not including the defendant.

Dr. Perlin testified to the ratio of those probabilities (more technically, of those probability densities) using the confusing phrase "match probability" when in fact there was no simple match. He framed the question as "How much more does the suspect match the evidence at this location than a random person?" But any random profile either would match -- in the sense of having the same alleles as those giving rise tom the observed peaks -- or it would not. Because of the ambiguity as to the true profiles in the mixture, the common-sense notion of a match -- of inclusions to and exclusions from -- the class of possible contributors to the mixture is not part of the logic of "probabilistic geneotyping."

Be that as it may, the issue on appeal in State v. Simmer, 304 Neb. 369 (2019), was not how to present the output of the software. It was whether that output could be the subject of any testimony at all. The Nebraska Supreme Court ran through the usual "Daubert factors." Despite arguments about the type of validation studies, the court was impressed with the sheer number of studies and the length of time that the system has been used.

The defendant's expert was Nathaniel Adams, who "has a bachelor’s degree in computer science and is employed by a company that 'consult[s] with lawyers.'" Mr. Adams testified (correctly) that one cannot directly validate the accuracy of particular likelihood ratios and that the software had not been developed and tested from the standpoint of software engineering, which, he suggested, requires independent study of source code over long periods of time. The Supreme Court was not impressed. It held that validation "from a software engineering perspective" was not essential and that the publication of research on the performance of the software (as best as that could be measured for known mixtures and via simulations) was sufficient.

The defense also maintained that "the validation studies must be discounted because Perlin is a coauthor of some of the publications and, as the owner of the company that owns. TrueAllele, has a financial interest in seeing it found reliable" To this, the Supreme Court responded that there were plenty of studies in which Perlin was not an author, but it also suggested that the source of validation studies is irrelevant because that fact pertains only to credibility:

In other words, Simmer is contending that there are reasons to question Perlin’s credibility. Even assuming that is true, an attack on the credibility of an expert witness is not aimed at the expert’s reasoning or methodology and is thus not the proper basis for a Daubert/Schafersman challenge. See Smith v. Colorado Organ Recovery Sys., 269 Neb. 578, 694 N.W.2d 610 (2005) (stating that challenge to expert witnesses' credibility is not Daubert/Schafersman claim).

This cannot be correct. Although the credibility of a particular witness certainly is fair game at trial, Independent replication of experiments is an important factor in scientific practice. The absence of validation from independent researchers should count against admissibility, although it would not necessarily be dispositive under Daubert.

Another questionable feature of the opinion lies in its discussion of the PCAST Report. In 2016 and 2017, the President's Council of Advisors on Science and Technology evinced dissatisfaction with the extent to which probabilisitc genotyping as applied in some situations had been validated. The software developers disputed the report -- but later responded with additional studies. The peculiarity in Simmer is not that the court determined that the output of TrueAllele was admissible despite PCAST's reservations. It is that the Nebraska Supreme Court refused to read the PCAST Report because it "is not in [the] record [on appeal]." Instead, it looked only at an obscure and incomplete "Washington trial court order containing a description of the report." On that basis, it wrote that "with no scientific evidence in the record indicating that TrueAllele could not generate reliable results under the circumstances here, we cannot say that a conclusion that TrueAllele could reliably be applied to the facts of this case would amount to an abuse of discretion."

But why was appellant was "unable to cite directly to the report"? It is a duly published report of a prestigious advisory body. Appellate lawyers and courts frequently cite books, articles, and reports about scientific research and policy. National Academy reports and Office of Technology Assessment reports have long been cited and discussed in opinions on the admissibility of DNA evidence. They address "legislative facts." Legislative facts apply across cases." In contrast, "adjudicative facts" are not general truth; they are the how, what, where, and when of specific cases.

Under the Nebraska Rule of Evidence (see, e.g., Neb. R. Evid. 201(a)), only adjudicative facts need to be proved at trial. Both appellate and trial courts have the power to consult published scientific opinions on the limitations of a scientific method. Here, the alleged limitation is that the software was not shown to work well with mixtures that contain very small fractions of a suspect's DNA. Courts should be more open to receiving information in the scientific and related literature on such issues.

Distorting Daubert and Parting Ways with PCAST in Romero-Lobato

United States v. Romero-Lobato 1/ is another opinion applying the criteria for admissibility of scientific evidence articulated in Daubert v. Merrell Dow Pharmaceuticals 2/ to uphold the admissibility of a firearms examiner's conclusion that the microscopic marks on recovered bullets prove that they came from a particular gun. To do so, the U.S. District Court for the District of Nevada rejects the conclusions of the President's Council of Advisors on Science and Technology (PCAST) on validating a scientific procedure.

This is not to say that the result in the case is wrong. There is a principled argument for admitting suitably confined testimony about matching bullet or ammunition marks. But the opinion from U.S. District Court Judge Larry R. Hicks does not contain such an argument. The court does not reach the difficult question of how far a toolmark expert may go in forging a link between ammunition and a particular gun. It did not have to. In what seems to be a poorly developed challenge to firearms-toolmark expertise, the defense sought to exclude all testimony about such an association.

This posting describes the facts of the case, the court's description of the law on the admissibility of source attributions by firearms-toolmark examiners, and its review of the practice under the criteria for admitting scientific evidence set forth by the Supreme Court in Daubert.

I. THE FIREARMS EVIDENCE IN THE CASE

A grand jury indicted Eric Romero-Lobato for seven felonies. On March 4, 2018, he allegedly tried to rob the Aguitas Bar and Grill and discharged a firearm (a Taurus PT111 G2) into the ceiling. On May 14, he allegedly stole a woman's car at gunpoint while she was cleaning it at a carwash. Later that night, he crashed the car in a high-speed chase On the front passenger's seat, was a Taurus PT111 G2 handgun.

Steven Johnson, a supervising criminalist in the Forensic Science Division of the Washoe County Sheriff's Office," 3/ was prepared to testify that the handgun had fired a round into the ceiling of the bar. Romero-Lobato moved "to preclude the testimony." The district court held a pretrial hearing at which Johnson testified to his background, training, and experience. He explained that he matched the bullet to the gun using the "AFTE method" advocated by the Association of Firearm and Tool Mark Examiners.

Defendant's challenge rested "on the critical NAS and PCAST Reports as evidence that 'firearms analysis' is not scientifically valid and fails to meet the requisite threshold for admission under Daubert and Federal Rule of Evidence 702." Apparently, the only expert at the hearing was the Sheriff Department's criminalist. Judge Hicks denied the motion to exclude Johnson's expert opinion testimony and issued a relatively detailed opinion.

II. SETTING THE STAGE FOR A DAUBERT ANALYSIS

Skipping over the early judicial resistance to "this is the gun" testimony, 4/ the court noted that despite misgivings about such testimony on the part of several federal district courts, only one reported case has barred all source opinion testimony 5/ and the trend among the more critical courts is to search for ways to admit the conclusion with qualifications on its certainty.

Judge Hicks did not pursue the possibility of admitting but constraining the testimony, apparently because the defendant did ask for that. Instead, the court reasoned that to overcome the inertia of the current caselaw, a defendant must have extraordinarily strong evidence (although it also recognized that the burden is on the government to prove scientific validity under Daubert) .The judge wrote:

[T]he defense has not cited to a single case where a federal court has completely prohibited firearms identification testimony on the basis that it fails the Daubert reliability analysis. The lack of such authority indicates to the Court that defendant's request to exclude Johnson's testimony wholesale is unprecedented, and when such a request is made, a defendant must make a remarkable argument supported by remarkable evidence. Defendant has not done so here.

Defendant's less-than-remarkable evidence was primarily two consensus reports of scientific and other experts who reviewed the literature on firearms-mark comparisons. 6/  Both are remarkable. The first document was the highly publicized National Academy of Sciences committee report on improving forensic science. The committee expressed concerns about the largely subjective comparison process and the absence of studies to adequately measure the uncertainty in the evaluations. The court deemed these to be satisfied by a single research report submitted to Department of Justice, which funded the study:

The NAS Report, released in 2009, concluded that “[s]ufficient studies have not been done to understand the reliability and repeatability” of firearm and toolmark examination methods. ... The Report's main issue with the AFTE method was that it did not provide a specific protocol for determining a match between a shell casing or bullet and a specific firearm. ... Instead, examiners were to rely on their training and experience to determine if there was a “sufficient agreement” (i.e. match) between the mark patterns on the casing or bullet and the firearm's barrel. ... During the Daubert hearing, Johnson testified about his field's response to the NAS Report, pointing to a 2013 study from Miami-Dade County (“Miami-Dade Study”). The Miami-Dade Study was conducted in direct response to the NAS Report and was designed as a blind study to test the potential error rate for matching fired bullets to specific guns. It examined ten consecutively manufactured barrels from the same manufacturer (Glock) and bullets fired from them to determine if firearm examiners (165 in total) could accurately match the bullets to the barrel. 150 blind test examination kits were sent to forensics laboratories across the United States. The Miami-Dade Study found a potential error rate of less than 1.2% and an error rate by the participants of approximately 0.007%. The Study concluded that “a trained firearm and tool mark examiner with two years of training, regardless of experience, will correctly identify same gun evidence.”

A more complete (and accurate) reading of the Miami Dade Police Department's study, shows that it was not designed to measure error rates as they are defined in the NAS report and that the "error rate" was much closer to 1%. That's still small, and, with truly independent verification of an examiners' conclusions, the error rate should be smaller than that for examiners whose findings are not duplicated. Nonetheless, as an earlier posting shows, the data are not as easily interpreted and applied to case work as the report from the crime laboratory suggests.The research study, which has yet to appear in any scientific journal. has severe limitations.

The second report, released late in 2016 by the President's Council of Advisors on Science and Technology (PCAST) flatly maintained that microscopic firearms-marks comparisons had not been scientifically validated. Essentially dismissing the Miami Dade Police and earlier research as not properly designed to measure the ability of examiners to infer whether the same gun fired test bullets and ones recovered from a crime scene, PCAST reasoned that (1) AFTE-type identification had yet to be shown to be "reliable" within the meaning of Rule 702 (as PCAST interpreted the rule); (2) if courts disagreed with PCAST's legal analysis of the rule's requirements, they should at least require examiners associating ammunition with a particular firearm to give an upper bound, as ascertained from controlled experiments, on false-positive associations. (These matters are discussed in previous postings.)

The court did not address the second conclusion and gave little or no weight to the first one. It wrote that the 2016 report

concluded that there was only one study done that “was appropriately designed to test foundational validity and estimate reliability,” the Ames Laboratory Study (“Ames Study”). The Ames Study ... reported a false-positive rate of 1.52%. ... The PCAST Report did not reach a conclusion as to whether the AFTE method was reliable or not because there was only one study available that met its criteria.

All true. PCAST certainly did not write that there is a large body of high quality research that proves toolmark examiners cannot associate expended ammunition with specific guns. PCAST's position is that a single study is not a body of evidence that establishes a scientific theory--replication is crucial. If the court believed that there is such a body of literature, it should have explained the basis for its disagreement with the Council's assessment of the literature. If it agreed with PCAST that the research base is thin, then it should have explained why forensic scientists should be able to testify--as scientists--that they know which gun fired which bullet. This opinion does neither. (I'll come to court's discussion of Daubert below.)

Instead, the court repeats the old news that

the PCAST Report was criticized by a number of entities, including the DOJ, FBI, ATF, and AFTE. Some of their issues with the Report were its lack of transparency and consistency in determining which studies met its strict criteria and which did not and its failure to consult with any experts in the firearm and tool mark examination field.

Again, all true. And all so superficial. That prosecutors and criminal investigators did not like the presidential science advisors' criticism of their evidence is no surprise. But exactly what was unclear about PCAST's criteria for replicated, controlled, experimental proof? In fact, the DOJ later criticized PCAST for being too clear--for having a "nine-part" "litmus test" rather than more obscure "trade-offs" with which to judge what research is acceptable. 7/

And what was the inconsistency in PCAST's assessment of firearms-marks comparisons? Judge Hicks maintained that

The PCAST Report refused to consider any study that did not meet its strict criteria; to be considered, a study must be a “black box” study, meaning that it must be completely blind for the participants. The committee behind the report rejected studies that it did not consider to be blind, such as where the examiners knew that a bullet or spent casing matched one of the barrels included with the test kit. This is in contrast to studies where it is not possible for an examiner to correctly match a bullet to a barrel through process of elimination.

This explanation enucleates no inconsistency. The complaint seems to be that PCAST's criteria for a validating a predominantly subjective feature-comparison procedure are too demanding or restrictive, not that these criteria were applied inconsistently. Indeed, no inconsistency in applying the "litmus test" for an acceptable research design to firearms-mark examinations is apparent.

Moreover, the court's definition of "a 'black box' study" is wrong. All that PCAST meant by "black box" is that the researchers are not trying to unpack the process that examiners use and inspect its components. Instead, they say to the examiner, "Go ahead, do your thing. Just tell us your answer, and we'll see if you are right." The term is used by software engineers who test complex programs to verify that the outputs are what they should be for the inputs. The Turing test for the proposition that "machines can think" is a kind of black box test.

Nonetheless, this correction is academic. The court is right about the fact that PCAST gave no credence to "closed tests" like those in which an examiner sorts bullets into pairs knowing in advance that every bullet has a mate. Such black-box experiments are not worthless. They show a nonzero level of skill, but they are easier than "open tests" in which an examiner is presented with a single pair of bullets to decide whether they have a common source, then another pair, and another, and so on. In Romero-Lobato, the examiner had one bullet from the ceiling to compare to a test bullet he fired from one suspect gun. There is no "trade-off" that would make the closed-test design appropriate for establishing the examiner's skill at the task he performed.

All that remains of the court's initial efforts to avoid the PCAST report is the tired complaint about a "failure to consult with any experts in the firearm and tool mark examination field." But what consultation does the judge think was missing? The scientists and technologists who constitute the Council asked the forensic science community for statements and literature to support their practices. It shared a draft of its report with the Department of Justice before finalizing it. After releasing the report, it asked for more responses and issued an addendum. Forensic-services providers may complain that the Council did not use the correct criteria, that its members were closed-minded or biased, or that the repeated opportunities to affect the outcome were insufficient or even a sham. But a court needs more than a throw-away sentence about "failure to consult" to justify treating the PCAST report as suspect.

 III. DISTORTING DAUBERT

Having cited a single, partly probative police laboratory study as if it were a satisfactory response to the National Academy's concerns and having colored the President's Council report as controversial without addressing the limited substance of the prosecutors' and investigators' complaints, the court offered a "Daubert analysis." It marched through the five indicia that the Supreme Court enumerated as factors that courts might consider in assessing scientific validity and reliability.

A. It Has Been Tested

The Romero-Lobato opinion made much of the fact that "[t]he AFTE methodology has been repeatedly tested" 8/ through "numerous journals [sic] articles and studies exploring the AFTE method" 9/ and via Johnson's perfect record on proficiency tests as proved by his (hearsay and character evidence) testimony. Einstein once expressed impatience "with scientists who take a board of wood, look for its thinnest part and drill a great number of holes where drilling is easy." 10/ Going through the drill of proficiency testing does not prove much if the tests are simple and unrealistic. A score of trivial or poorly designed experiments should not engender great confidence. The relevant question under Daubert is not simply "how many tests so far?" It is how many challenging tests have been passed. The opinion makes no effort to answer that question. It evinces no awareness of the "10 percent error rate in ballistic evidence" noted in the NAS Report, that prompted corrective action in the Detroit Police crime laboratory.

Instead of responding to PCAST's criticisms of the design of the AFTE Journal studies, the court wrote that "[a]lthough both the NAS and PCAST Reports were critical of the AFTE method because of its inherent subjectivity, their criticisms do not affect whether the technique they criticize has been repeatedly tested. The fact that numerous studies have been conducted testing the validity and accuracy of the AFTE method weighs in favor of admitting Johnson's testimony."

But surely the question under Daubert is not whether there have been "numerous studies." It is what these studies have shown about the accuracy of trained examiners to match a single unknown bullet with control bullets from a single gun. The court may have been correct in concluding that the testing prong of Daubert favors admissibility here, but its opinion fails to demonstrate that "[t]here is little doubt that the AFTE method of identifying firearms satisfies this Daubert element."

B. Publication and Peer Review

Daubert recognizes that, to facilitate the dissemination, criticism, and modification of theories, modern science relies on publication in refereed journals that members of the scientific community read. Romero-Lobato deems this factor to favor admission for two reasons. First, the AFTE Journal in which virtually all the studies dismissed by PCAST appear, uses referees. That it is not generally regarded as a significant scientific journal -- it is not available through most academic libraries, for example -- went unnoticed.

Second, the court contended that "of course, the NAS and PCAST Reports themselves constitute peer review despite the unfavorable view the two reports have of the AFTE method. The peer review and publication factor therefore weighs in favor of admissibility." The idea that the rejection in consensus reports of a series of studies as truly validating a theory "weighs in favor of admissibility" is difficult to fathom. Some readers might find it preposterous.

C. Error Rates

Just as the court was content to rely on the absolute number of studies as establishing that the AFTE method has been adequately tested, it takes the error rates reported in the questioned studies at face value. Finding the numbers to be "very low," and implying (without explanation) that PCAST's criteria are too "strict," it concludes that Daubert's "error rate" factor too "weighs in favor of admissibility."

A more plausible conclusion is that a large body of studies that fail to measure the error rates (false positive and negative associations) appropriately but do not indicate very high error rates is no more than weakly favorable to admission. (For further discussion, see the previous postings on the court's discussion of the Miami Dade and Ames Laboratory technical reports.)

D. Controlling Standards

The court cited no controlling standards for the judgment of  "'sufficient agreement' between the 'unique surface contours' of two toolmarks." After reciting the AFTE's definition of "sufficient agreement," Judge Hicks decided that "matching two tool marks essentially comes down to the examiner's subjective judgment based on his training, experience, and knowledge of firearms. This factor weighs against admissibility."

However, the opinion adds that "the consecutive matching striae ('CMS') method," which Johnson used after finding "sufficient agreement," is "an objective standard under Daubert." It is "objective" because an examiner cannot conclude that there is a match unless he "observes two or more sets of three or more consecutive matching markings on a bullet or shell casing." The opinion did not consider the possibility that this numerical rule does little to confine discretion if no standard guides the decision of  whether a marking matches. Instead, the opinion debated whether the CMS method should be considered objective and confused that question with how widely the method is used.

The relevant inquiry is not whether a method is subjective or objective. For a predominantly subjective method, the question is whether standards for making subjective judgments will produce more accurate and more reliable (repeatable and reproducible) decisions and how much more accurate and reliable they will be.

E. General Acceptance

Finally, the court found "widespread acceptance in the scientific community." But the basis for this conclusion was flimsy. It consisted of statements from other courts like "the AFTE method ... is 'widely accepted among examiners as reliable'" and "[t]his Daubert factor is designed to prohibit techniques that have 'only minimal support' within the relevant community." Apparently, the court regarded the relevant community as confined to examiners. Judge Hicks wrote that

it is unclear if the PCAST Report would even constitute criticism from the “relevant community” because the committee behind the report did not include any members of the forensic ballistics community ... . The acceptance factor therefore weighs in favor of admitting Johnson's testimony.

If courts insulate forensic-science service providers from the critical scrutiny of outside scientists, how can they legitimately use the general-acceptance criterion to help ascertain whether examiners are presenting "scientific knowledge" à la Daubert?

NOTES

1. No. 3:18-cr-00049-LRH-CBC, 2019 WL 2150938 (D. Nev. May 16, 2019).
2. 509 U.S. 579 (1993).
3. For a discussion of a case involving inaccurate testimony from the same laboratory that caught the attention of the Supreme Court, see David H. Kaye, The Interpretation of DNA Evidence: A Case Study in Probabilities, National Academies of Science, Engineering and Medicine, Science Policy Decision-making Educational Modules, 2016, available at https://ssrn.com/abstract=2810744; McDaniel v. Brown: Prosecutorial and Expert Misstatements of Probabilities Do Not Justify Postconviction Relief — At Least Not Here and Not Now, Forensic Sci., Stat. & L., July 7, 2014, http://for-sci-law.blogspot.com/2014/07/mcdaniel-v-brown-prosecutorial-and.html.
4. See David H. Kaye, Firearm-Mark Evidence: Looking Back and Looking Ahead, 68 Case W. Res. L. Rev. 723, 724-25 (2018), available at https://papers.ssrn.com/abstract_id=3117674. The court relied on the article's explication of more modern case law.
5. The U.S. District Court for the District of Colorado  excluded toolmark conclusions in the prosecutions for the bombing of the federal office building in Oklahoma City. The toolmarks there came from a screwdriver. David H. Kaye et al., The New Wigmore, A Treatise on Evidence: Expert Evidence 686-87 (2d ed. 2011).
6. The court was aware of an earlier report from a third national panel of experts raising doubts about the AFTE method, but it did not cite or discuss that report's remarks. Although the 2008 National Academies report on the feasibility of establishing a ballistic imaging database only considered the forensic toolmark analysis of firearms in passing, it gave the practice no compliments. Kaye, supra note 2, a 729-32.
7. Ted Robert Hunt, Scientific Validity and Error Rates: A Short Response to the PCAST Report, 86 Fordham L. Rev. Online Art. 14 (2017), https://ir.lawnet.fordham.edu/cgi/viewcontent.cgi?article=1013&context=flro.
8. Quoting United States v. Ashburn, 88 F.Supp.3d 239, 245 (E.D.N.Y. 2015).
9. Citing United States v. Otero, 849 F.Supp.2d 425, 432–33 (D.N.J. 2012), for "numerous journals [sic] articles and studies exploring the AFTE method."
10. Philipp Frank, Einstein's Philosophy of Science, Reviews of Modern Physics (1949).

FURTHER READING

• David H. Kaye, Firearm-Mark Evidence: Looking Back and Looking Ahead, 68 Case W. Res. L. Rev. 723-45 (2018)
• David H. Kaye et al., The New Wigmore, A Treatise on Evidence: Expert Evidence  (2d ed. 2011, with annual updates)

MODIFIED: 7 July 2019 9:10 EST

Index to comments and cases on the PCAST report

Frye-Daubert Flip Flops in Florida

For years, the Florida Supreme Court rebuffed suggestions that it adopt the standard for scientific evidence that the U.S. Supreme Court articulated for the federal judiciary in Daubert v. Merrell Dow Pharmaceuticals, 1/ Instead, it "repeatedly reaffirmed [its] adherence to the Frye standard for admissibility of evidence." 2/

In 2013, the Florida legislature passed a statute to replace Frye with the "reliability" wording of Federal Rule of Evidence 702 -- wording intended to codify Daubert and its progeny. Some Florida courts concluded that this brought Florida into the ranks of jurisdictions that use the Daubert standard of "evidentiary reliability" based on "scientific validity." 3/ However, the Florida Supreme Court has held that only it can implement "procedural" changes to the Florida Rules of Evidence (FLRE). 4/ The legislature has the power to promulgate "substantive" rules of evidence, but it may not force "procedural" ones down the judiciary's throat.

So the Florida Bar's Code and Rules of Evidence Committee reviewed the law. By a narrow margin, it recommended leaving Frye in place. The Florida Supreme Court agreed. It declined to adopt the Daubert amendment "due to the constitutional concerns raised" by certain Committee members and commenters. 5/ "Those concerns," the court explained, "include undermining the right to a jury trial and denying access to the courts." 6/ The next year, in DeLisle v. Crane Co., 7/ the court confirmed that the legislative switch to Daubert was purely procedural. Because the court did not bless it, the law was constitutionally ineffective.

Then last month, the court flip-flopped. It adopted the "Daubert amendments" under its "exclusive rule-making authority." 8/ Although the amendment to Rule 702 did not percolate through the rules committee a second time, the court decided that its earlier reservations about switching to Daubert "appear unfounded." 9/

Indeed, the arguments that the court considered "grave" two years ago are anything but. The two standards -- general scientific acceptance (Frye) and evidentiary reliability encompassing scientific validity (Daubert) -- each seek to screen out expert evidence that is insufficiently validated to warrant its use in court in light of the danger that it will be given too much weight. One standard (Daubert) asks judges to assess directly the validity of scientific theories. The other (Frye) has them do so indirectly, by looking only for a consensus in the scientific community. This difference in the mode of analysis does not make one approach constitutional and the other unconstitutional. Daubert does not create an inherently more demanding test than Frye. 10/ It describes more criteria for answering the same underlying question -- is the proposed evidence probative enough to come in as "science" (or some other form of expertise).

Certainly, there is room to debate the relative merits of the two approaches -- and room for different jurisdictions to go their own ways -- but the choice between Daubert and Frye (or other reasonable standards) does not pose a serious constitutional question.

NOTES

1. 509 U.S. 579 (1993).
2. Marsh v. Valyou, 977 So.2d 543, 547 (Fla. 2007) (holding that the "general acceptance" standard fashioned in Frye v. United States, 293 F. 1013 (D.C.Cir.1923), and expressly adopted in Florida in Bundy v. State, 471 So.2d 9, 18 (Fla.1985), and Stokes v. State, 548 So.2d 188, 195 (Fla.1989), does not even apply to "pure opinion" testimony "causally linking trauma to fibromyalgia ... based on the experts' experience and training").
3. Perez v. Bell So. Telecommunications, Inc., 138 So.3d 492, 497 (Fla. Dist. Ct. App. 2014). The phrases "evidentiary reliability" and "scientific validity" appear in the Daubert opinion.
4. DeLisle v. Crane Co., 258 So.3d 1219 (Fla. 2018).
5. In re Amendments to Florida Evidence Code, 210 So.3d 1231, 1239 (Fla. 2017).
6. Id.
7. 258 So.3d 1219 (Fla. 2018).
8. In re Amendments to the Florida Evidence Code, No. SC19-107, 2019 WL 2219714 (Fla. May 23, 2019). Thanks are due to Ed Imwinkelried for calling the case to my attention.
9. Id.
10. The Florida Supreme Court had previously written that Frye imposed a "higher standard of reliability" than the "more lenient standard" in Daubert. Brim v. State, 695 So.2d 268, 271–72 (Fla. 1997). It is tempting to ask how Daubert's "more lenient" reliability requirement could be unconstitutional when Frye's more exacting standard is constitutionally sound. I suppose one could argue that because Frye (as construed in Florida)  does not bar "pure opinion" testimony that has not been shown to be scientifically reliable, it has less of an impact on "access to the courts." However, as discussed in The New Wigmore on Evidence: Expert Evidence (2d ed. 2011),  the "pure opinion" exception to either Frye or Daubert is untenable.

Shoeprints in Indiana: Confronting a "Skilled Witness" with the PCAST Report

Last week, in Hughes v. State, 1/ the Indiana Court of Appeals wrote an opinion on the admissibility of shoeprint evidence and a defense attempt to present part of the 2016 PCAST report on feature-matching evidence. Mark Adrian Hughes was convicted for breaking into two newly constructed homes and stealing the appliances in them. "Sean Matusko, a forensic scientist with the ISP laboratory's latent-print unit" 2/ testified "that shoeprints found at both crime scenes were made by Hughes's shoes." The trial court overruled defendant's objection to this testimony and barred him from introducing into evidence a part of the PCAST report and from cross-examining Matusko about the content of the report. It reasoned "that Matusko was a "skilled witness" but not an expert one (preventing cross-examination), and that the report was hearsay (preventing its use as evidence).

The unpublished court of appeals opinion, penned by Judge Robert R. Altice, Jr., reversed defendant's convictions, but not because of these rulings. The appellate court determined that the prosecutor improperly introduced evidence of earlier, similar crimes. In remanding the case for a new trial, the court of appeals also discussed the shoeprint rulings. Its analysis is puzzling. The court wrote that

Hughes challenges the trial court's treatment of the State's shoeprint examiner, Matusko, as a skilled witness. Here, Matusko did not simply testify based on his personal experience ... . Rather, ... Matusko identified himself as a forensic scientist assigned to the latent print identification unit of the Indiana State Police, set out his academic background, detailed his training with regard to shoeprint identification, and explained in detail the process he used to identify shoeprints at both crime scenes as being made by Hughes's shoes. [O]ur Supreme Court has indicated that it is not inclined to consider all testimony relating to shoeprint identification to be opinion testimony governed by Evid. R. 702. In light of such precedent and our standard of review, we cannot say that the trial court abused its discretion in admitting Matusko's testimony under Evid. R. 701.

It is inconceivable that a witness who represents himself as a scientist applying a process with which lay jurors are unfamiliar and thereby deducing that a specific pair of shoes left the impressions is not testifying as an expert under Rule 702. He was not there, and he did not see what happened. If he knows anything about the source of the shoprints, it is because of he possesses special knowledge and skill beyond the ken or ordinary witnesses. Indiana Rule of Evidence 702(a) governs all witnesses with "specialized knowledge" who rely on their unusual "knowledge, skill, experience, training, or education [to] help the trier of fact to understand the evidence or to determine a fact in issue." Rule 701, on the other hand, governs opinions from "lay witnesses." It limits them to inferences that would be difficult or tedious to present as more primitive statements of the details the witness perceived. 3/ The division these rules create reflects an ancient distinction in the common law between ordinary fact witnesses -- the Rule 701 category -- and expert witnesses -- the Rule 702 group.

To be sure, a witness sometimes can testify in both capacities. A physician can be an ordinary fact witness in part -- "I saw that the patient was having trouble breathing" -- and a skilled witness in part -- "My diagnosis was pneumonia." But the latter opinion must satisfy Rule 702 to be admissible, and the doctor is subject to cross-examination to suggest that his or her diagnosis is unfounded. If a medical journal states that the diagnosis is not warranted without additional symptoms, the doctor can be asked about that as long as "the publication is established as a reliable authority" under Rule 803(18)(c), for "learned treatises."

Similarly, Matusko could testify -- as an ordinary fact witness under Rule 701 -- that the shoes he was asked to compare to the shoeprints were "Nike Air Jordan athletic shoes with a Jumpman logo molded into the soles." But Rule 701 would not let him speak as a "skilled witness" giving an opinion as to origin of the footprint based on his special skill as a shoeprint examiner. That task always has been reserved for expert witnesses. 4/

Now there might be a reason to present an expert (who does not appear as a "scientist") as merely a "skilled" witness. Indiana Rule of Evidence 702(b) codifies the rule of heightened scrutiny for scientific expert testimony articulated by the U.S. Supreme Court for the federal courts in Merrell Daubert v. Merrell Dow Pharmaceuticals. 5/ Like Daubert, Indiana Rule 702 specifies that "[e]xpert scientific testimony is admissible only if the court is satisfied that the expert testimony rests upon reliable scientific principles."

Whether Matsuko's source attribution can pass that bar is doubtful. At least, the scientists and engineers on the President's Council of Advisors doubted it. They concluded that identifying a particular shoe as the source of a print has yet to be scientifically validated. Because "[t]he entire process—from choice of features to include (and ignore) and the determination of rarity—relies entirely on an examiner’s subjective judgment," PCAST wanted to see studies that tested the performance of criminalists with impressions from the same shoes and from different shoes. Because no such studies exist, PCAST reported that

there are no appropriate empirical studies to support the foundational validity of footwear analysis to associate shoeprints with particular shoes based on specific identifying marks (sometimes called “randomly acquired characteristics"). Such conclusions are unsupported by any meaningful evidence or estimates of their accuracy and thus are not scientifically valid.

The last sentence, if correct, means that the shoeprint testimony cannot be introduced against Hughes on retrial -- not if the witness claims to be acting as a scientist applying a scientific procedure. Furthermore, even if the police criminalist distances himself from scientific titles and airs, the court must decide whether footwear identification is reliable enough to qualify as a different form of expert testimony. The absence of scientific validation will not be determinative, but it is a relevant factor.

Let's assume, though, that the trial court decides that a highly "de-scientized" version of Mr. Matusko's opinion is admissible under Rule 702. Can he be impeached with a scientific report? The Indiana Court of Appeals thought so. A footnote states that

As set out above, Hughes sought but was denied the opportunity to cross-examine Matusko with findings set out in the PCAST publication concerning reliability of shoeprint identification. Although we did not reach the merits of the admissibility of the PCAST publication, cross-examination regarding the findings therein was permissible regardless of whether Matusko was an expert or a skilled witness.

If correct, this conclusion -- that a criminalist can be impeached by confronting him with the PCAST report -- would be a boon to defendants across the nation. Without the trouble and expense of calling an expert witness, defense counsel can wave the devastating critique in front of the witness (and the jury). But the rule on introducing "learned treatises" requires proof that the material is authoritative before it can be used for impeachment. 6/ This limitation makes sense because, unlike impeachment by self-contradiction, the out-of-court statement -- what the President's advisors had to say -- has value only to the extent that it is true. Thus, the document is hearsay.

Nevertheless, even without an expert to establish that PCAST is a reliable authority, the report could be admissible over a hearsay objection. It is, after all, a government report. The public records exception to the rule against hearsay extends to "factual findings from a legally authorized investigation," 7/ as long as "neither the source of information nor other circumstances indicate a lack of trustworthiness." 8/ Law enforcement groups have loudly proclaimed that this particular report is not trustworthy, but much of the criticism is more reflexive than reasoned. Very little of it focuses on footwear analysis. 9/

NOTES

1. No. 18A-CR-1007, 2019 WL 2094045 (Ind. Ct. App. May 14, 2019) (unreported, available at https://www.in.gov/judiciary/opinions/pdf/05141901rra.pdf).
2. The witness is featured in an educational state police YouTube video.
3. Indiana Rule of Evidence 701 applies to "Opinion Testimony by Lay Witnesses." It provides that "If a witness is not testifying as an expert, testimony in the form of an opinion is limited to one that is: (a) rationally based on the witness's perception; and (b) helpful to a clear understanding of the witness's testimony or to a determination of a fact in issue."
4. Indeed, in Buchman v. State, 59 Ind. 1, 26 Am.Rep. 75 (Ind. 1877), the Indiana Supreme Court held that a physician could not be compelled to testify to a professional opinion without special compensation. Its opinion used the words "skilled witness" and "expert" as synonyms.
5. 509 U.S. 579 (1993).
6. David H. Kaye, David Bernstein & Jennifer L. Mnookin, The New Wigmore on Evidence: Expert Evidence ch. 5 (2d ed. 2011).
7. Ind. R. Evid. 803(8)(A)(i)(c).
8. Ind. R. Evid. 803(8)(A)(ii).
9. The Department of Justice's more thoughtful disagreements with the report's general approach to ascertaining scientific validity are presented in Ted Robert Hunt, Scientific Validity and Error Rates: A Short Response to the PCAST Report, 86 Fordham L. Rev. Online 24 (2018). After the initial frosty reception of its report from prosecutors, police, and forensic practitioners, PCAST requested input from the forensic-science community for a second time. It then issued an addendum to its report. With regard to shoeprints, this document stated that

       In its report, PCAST considered feature-comparison methods for associating a shoeprint with a specific shoe based on randomly acquired characteristics (as opposed to with a class of shoes based on class characteristics). PCAST found no empirical studies whatsoever that establish the scientific validity or reliability of the method.
       The President of the International Association for Identification (IAI), Harold Ruslander, responded to PCAST’s request for further input. He kindly organized a very helpful telephonic meeting with IAI member Lesley Hammer. (Hammer has conducted some of the leading research in the field—including a 2013 paper, cited by PCAST, that studied whether footwear examiners reach similar conclusions when they are presented with evidence in which the identifying features have already been identified.)
       Hammer confirmed that no empirical studies have been published to date that test the ability of examiners to reach correct conclusions about the source of shoeprints based on randomly acquired characteristics. Encouragingly, however, she noted that the first such empirical study is currently being undertaken at the West Virginia University. When completed and published, this study should provide the first actual empirical evidence concerning the validity of footwear examination. The types of samples and comparisons used in the study will define the bounds within which the method can be considered reliable.

   An Addendum to the PCAST Report on Forensic Science in Criminal Courts, Jan. 2017, at 5-6.

Index to comments and cases on the PCAST report