Showing posts with label fingerprint. Show all posts
Showing posts with label fingerprint. Show all posts

Friday, January 12, 2024

What's Uniqueness Got to Do with It?

Columbia University has announced that "AI Discovers That Not Every Fingerprint Is Unique"! The subtitle of the press release of January 10, 2024, boldly claims that

Columbia engineers have built a new AI that shatters a long-held belief in forensics–that fingerprints from different fingers of the same person are unique. It turns out they are similar, only we’ve been comparing fingerprints the wrong way!

Forensic Magazine immediately and uncritically rebroadcast (quoting verbatim without acknowledgment from the press release) the confused statements about uniqueness. According to the Columbia release and Forensic Magazine, "It’s a well-accepted fact in the forensics community that fingerprints of different fingers of the same person—or intra-person fingerprints—are unique and therefore unmatchable." Forensics Magazine adds that "Now, a new study shows an AI-based system has learned to correlate a person’s unique fingerprints with a high degree of accuracy."

Does this mean that the "well-accepted fact" and "long-held belief" in uniqueness been shattered or not? Clearly, not. The study is about similarity, not uniqueness. In fact, uniqueness has essentially nothing to do with it. I can classify equilateral triangles drawn on a flat surface as triangles rather than as other regular polygons whether or not the triangles are each different enough from one another (uniqueness within the set of triangles) that I notice these differences. To say that objects "are unique and therefore unmatchable" is a nonsequitur. A human genome is probably unique to that individual, but forensic geneticists know that six-locus STR profiles are "matchable" to those of other individuals in the population. A cold hit to a person who could not have been the source of the six-locus profile in the U.K. database occurred long ago (as was to be expected for the random-match probabilities of the genotypes).

Perhaps the myth that the study shatters is that it is impossible to distinguish fingerprints left by different fingers of the same individual X from fingerprints left by fingers of different individuals (not-X). But there is no obvious reason why this would be impossible even if every print is distinguishable from every other print (uniqueness).

The Columbia press release describes the study design this way:

[U]ndergraduate senior Gabe Guo ... who had no prior knowledge of forensics, found a public U.S. government database of some 60,000 fingerprints and fed them in pairs into an artificial intelligence-based system known as a deep contrastive network. Sometimes the pairs belonged to the same person (but different fingers), and sometimes they belonged to different people.

Over time, the AI system, which the team designed by modifying a state-of-the-art framework, got better at telling when seemingly unique fingerprints belonged to the same person and when they didn’t. The accuracy for a single pair reached 77%. When multiple pairs were presented, the accuracy shot significantly higher, potentially increasing current forensic efficiency by more than tenfold.

The press release reported the following odd facts about the authors' attempts to publish their study in a scientific journal:

Once the team verified their results, they quickly sent the findings to a well-established forensics journal, only to receive a rejection a few months later. The anonymous expert reviewer and editor concluded that “It is well known that every fingerprint is unique,” and therefore it would not be possible to detect similarities even if the fingerprints came from the same person.

The team ... fed their AI system even more data, and the system kept improving. Aware of the forensics community's skepticism, the team opted to submit their manuscript to a more general audience. The paper was rejected again, but [Professor Hod] Lipson ... appealed. “I don’t normally argue editorial decisions, but this finding was too important to ignore,” he said. “If this information tips the balance, then I imagine that cold cases could be revived, and even that innocent people could be acquitted.” ...

After more back and forth, the paper was finally accepted for publication by Science Advances. ... One of the sticking points was the following question: What alternative information was the AI actually using that has evaded decades of forensic analysis? ... “The AI was not using ... the patterns used in traditional fingerprint comparison,” said Guo ... . “Instead, it was using something else, related to the angles and curvatures of the swirls and loops in the center of the fingerprint.”

Proprietary fingerprint matching algorithms also do not arrive at matches the way human examiners do. They "see" different features in the patterns and tend to rank the top candidates for true matches in a database trawl differently than the human experts. Again, however, these facts about automated systems neither prove nor disprove claims of uniqueness. And, theoretical uniqueness has little or nothing to do with the actual probative value of assertions of matches by humans, automated systems, or both.

Although not directly applicable, the day after the publicity on the Guo et al. paper, I came across the following report on "Limitations of AI-based predictive models" in a weekly survey of papers in Science:

A central promise of artificial intelligence (AI) in health care is that large datasets can be mined to predict and identify the best course of care for future patients. Unfortunately, we do not know how these models would perform on new patients because they are rarely tested prospectively on truly independent patient samples. Chekroud et al. showed that machine learning models routinely achieve perfect performance in one dataset even when that dataset is a large international multisite clinical trial (see the Perspective by Petzschner). However, when that exact model was tested in truly independent clinical trials, performance fell to chance levels. Even when building what should be a more robust model by aggregating across a group of similar multisite trials, subsequent predictive performance remained poor. -- Science p. 164, 10.1126/science.adg8538; see also p. 149, 10.1126/science.adm9218

Note: This posting was last modified on 1/12/24 2:45 PM

Friday, July 7, 2023

No "Daubert Hearing" on Latent Fingerprint Matching in US v. Ware

Last month, in United States v. Ware, 69 F.4th 830 (11th Cir. 2023), the U.S. Court of Appeals for the Eleventh Circuit  "carefully review[ed]" the convictions of Dravion Sanchez Ware arising out of a month-long crime spree near Atlanta, in 2017. He was found to have participated "in robbing ... three spas, four massage parlors, a nail salon, and a restaurant." The opinion  recounts the nine brutal robberies in luxuriant detail. It also discusses Mr. Ware's argument that the district court erred "by not holding a formal Daubert hearing before admitting expert fingerprint evidence."

In a word, the Eleventh Circuit rejected the argument as "unpersuasive." No surprise there. More surprising is the opinion's incoherent discussion of the 2009 NRC report on forensic science and the 2016 PCAST follow-up report. \1/ On the one hand, we are told that "[t]he science could not possibly have been so unreliable as to be inadmissible." On the other hand, "[t]he District Court here could have held a Daubert hearing to assess the relatively new reports Ware presented." So which is it? If a type of evidence cannot possibly be excluded as scientifically invalid under Daubert, how can it be proper to hold a pretrial testimonial hearing on admissibility under Daubert? And, was the court of appeals correct in concluding that the two reports do not impeach, to the point of requiring a hearing, the traditional practice of admitting latent fingerprint comparisons?

During Ware's trial, an unnamed "crime lab scientist with the Georgia Bureau of Investigation Division of Forensic Sciences" "outlined the science behind fingerprints themselves, including their uniqueness" and explained the four-step process the lab follows ... : 'Analysis, Comparison, Evaluation, and Verification,' or ACEV.” The last step "involves another examiner completing the whole process a second time." The opinion does not indicate whether the verifying analyst is blinded to the knowledge of the main examiner's finding. Interestingly as well (think Confrontation Clause), the opinion implies that the testifying expert in Ware was not the main examiner. "[S]he was the verifying examiner," and "she testified that the lab concluded the latent print ... led to an identification conclusion matched to Ware's left middle finger." After that,

Defense counsel specifically asked about the PCAST report [and] vigorously cross-examined ... discussing the possibility of a latent fingerprint not being usable ... , the subjectiveness of every step ... , and the bias that may creep into the verification process ... . The expert and defense counsel discussed ... the potential for false positives and negatives. On cross, the defense also attacked the expert's claim that she did not know of the Georgia Bureau of Investigation ever misidentifying someone with a fingerprint comparison, and that she did not know the rate at which a verifier disagrees with the original assessment.

To preclude such testimony about his unique fingerprint on an item stolen in one of the robberies, Ware had moved before the trial for an order excluding fingerprint-comparison evidence. Of course, such a ruling would have been extraordinary, but the defense contended that the 2009 and the 2016 reports required nothing less \2/ and asked for a full-fledged pretrial hearing on the matter. In response, "[t]he District Court conditionally denied the motion ... unless Ware's counsel could produce before trial a case from this Court or a district court in this Circuit that favors excluding fingerprint expert evidence under Daubert." \3/

The court of appeals correctly observed that "[f]ingerprint comparison has long been accepted as a field worthy of expert opinions in this Circuit, as well as in almost every one of our sister circuits." The only problem is that all the opinions cited to show this solid wall of precedent predate the NRC or the PCAST reports. A more complete analysis has to establish that the scientists' reviews of friction-ridge pattern matching do not raise enough of a doubt to expect that a hearing would let the defense breach the wall. 

Along these lines, the court of appeals wrote that

The [District] Court considered the reports and arguments presented and found that fingerprint evidence was reliable enough as a general matter to be presented to the jury. Many of the critiques of fingerprint evidence found in the PCAST report go to the weight that ought to be given fingerprint analysis, not to the legitimacy of the practice as a whole. Appellant Br. at 25 (“The studies collectively demonstrate that many examiners can, under some circumstances, produce correct answers at some level of accuracy.” (emphasis in original)).

This quotation from the PCAST report is faint praise. Although the court of appeals was sure that "Ware's contrary authority even says that fingerprint evidence can be reliable," the depth of its knowledge about the PCAST (and the earlier NRC committee) reports is open to question. The circuit court had trouble keeping track of the names of the groups. It transformed the National Research Council (the operating arm of the National Academies of Science, Engineering, and Medicine) into a "United States National Resource Council" (69 F.4th at 840) and then imagined an "NCAST report[]" (id. at 848). \4/ 

Deeper inspection of "Ware's contrary authority" is in order. The 2009 NRC committee report quoted with approval the searing conclusion of Haber & Haber that “[w]e have reviewed available scientific evidence of the validity of the ACE-V method and found none.” It reiterated the Habers' extreme claim that because "the standards upon which the method’s conclusions rest have not been specified quantitatively ... the validity of the ACE-V method cannot be tested." To be sure, the committee agreed that fingerprint examiners had something going for them. It wrote that "more research is needed regarding the discriminating value of the various ridge formations [to] provide examiners with a solid basis for the intuitive knowledge they have gained through experience." But does "intuitive knowledge" qualify as "scientific knowledge" under Daubert? Is a suggestion that friction-ridge comparisons need a more solid basis equal to a statement that the comparisons are "reliable" within the meaning of that opinion? The response to "NCAST" was underwhelming.

But research has progressed since  2009. The second "contrary authority," the PCAST report, reviewed this research. At first glance, this report supports the court's conclusion that no hearing was necessary. It assures courts that "latent fingerprint analysis is a foundationally valid subjective methodology." In doing so, it rejects the NRC committee's notion that the absence of quantitative match rules precludes testing whether examiners can reach valid conclusions. It discusses two so-called black-box studies of the work of examiners operating in the "intuitive" mode. Yet, the Ware court does not cite or quote the boxed and highlighted finding (Number 5).

Perhaps the omission reflects the fact that the PCAST finding is so guarded. PCAST added that "additional black-box studies are needed to clarify the reliability of the method," undercutting the initial assurance, which was "[b]ased largely on two ... studies." Furthermore, according to PCAST, to be "scientifically valid," latent-print identifications must be accompanied by admissions that "false positive rates" could be very high (greater than 1 in 18). \5/

The Ware court transforms all of this into a blanket and bland assertion that the report establishes reliability even though it "may cast doubt on the error rate of fingerprint analysis and comparison." The latter concern, it says, goes not to admissibility, but only to "weight" or "credibility." 

Can it really be this simple? Are not "error rates" an explicit factor affecting admissibility (as well as weight) under Daubert? Certainly, the Eleventh Cicuit's view that the problems with fingerprint comparisons articulated in the two scientific reports are not profound enough to force a wave of pretrial hearings is defensible, but the court's explanation of its position in Ware is sketchy.

At bottom, the problem with the fingerprint evidence introduced against Ware (as best as one can tell from the opinion) is not that it is speculative or valueless. The difficulty is that the judgments are presented as if they were scientific truths. The Ware court is satisfied because "Defense counsel put the Government's expert through his paces during cross-examination, and counsel specifically asked the expert about the findings in the PCAST report." But would it be better to moderate the presentations to avoid overclaiming in the first place? 

The impending amendment to Rule 702 of the Federal Rules of Evidence is supposed to encourage this kind of "gatekeeping." Defense counsel might be more successful in constraining overreaching experts than in excluding them altogether. That too should be part of the "considerable leeway" granted to district courts seeking to reconcile expert testimony.with modern scientific knowledge.

Notes

  1. President's Council of Advisors on Sci. & Tech., Exec. Office of the President, Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods (2016),  [https://perma.cc/R76Y-7VU]
  2. In Ware
    The pretrial motion to exclude the fingerprint identification "relied on the 2009 United States National Resource Counsel (“NRC”) report and subsequent 2016 President's Counsel of Advisors on Science and Technology (“PCAST”) report, which supposedly revealed a dearth of "proper scientific studies of fingerprint comparison evidence" and claimed that "there is no scientific basis for concluding a fingerprint was left by a specific person," positing that "because fingerprint analysis involves individual human judgement, the resulting [fingerprint comparison] conclusion can be influenced by cognitive bias."
  3. Why insist on a pre-existing determination in one particular geographic region that scientific validity is lacking in order to grant a hearing on whether scientific validity is present? Is the "science" underlying fingerprint comparisons different in Georgia and the other southeastern states comprising the 11th Circuit different from that in the rest of the country?
  4. OK, these peccadillos are not substantive, but one would have thought that three circuit court judges, after "carefully reviewing the record," could have gotten the names and acronyms straight. Senior Judge Gerald Tjoflat wrote the panel opinion. At one point, he was a serious contender for the Supreme Court seat filled by Justice Anthony Kennedy. After Judge Tjoflat announced that he would retire to senior status on the bench in 2019, President Donald Trump nomined Robert J. Luck to the court. In addition to Judge Luck, Judge Kevin C. Newsom, a 2017 appointee of President Trump was on the panel. Judicial politics being what it is, over 30 senators voted against the confirmation of Judges Newsom and Luck.
  5. PCAST suggested that if a court agreed that what it called "foundational validity" were present, then to achieve "validity as applied" some very specific statements about "error rates" would be required:
    Overall, it would be appropriate to inform jurors that (1) only two properly designed studies of the accuracy of latent fingerprint analysis have been conducted and (2) these studies found false positive rates that could be as high as 1 in 306 in one study and 1 in 18 in the other study. This would appropriately inform jurors that errors occur at detectable frequencies, allowing them to weigh the probative value of the evidence.
    The studies actually found conditional false-positive proportions of 6/3628 (0.17%) and 42/995 (4.2%, or 7/960 = 1.4% if one discards "clerical errors.") (P. 98, tbl. 1). Earlier postings discuss these FBI-Noblis and Miami Dade police department numbers.

Monday, November 11, 2019

More on Rule 704 and Source Attribution in Forensic Science

On July 17, I observed that
At a webinar on probabilistic genotyping organized by the FBI, the Department of Justice’s Senior Advisor on Forensic Science, Ted Hunt, summarized the rules of evidence that are most pertinent to scientific and expert testimony. In the course of a masterful survey, he suggested that Federal Rule of Evidence 704 somehow conflicts with the evidence-centric approach to evaluating laboratory results recommended by a subcommittee of the National Commission on Forensic Science, by the American Statistical Association, and by European forensic-science service providers.
I proceeded to analyze Rule 704 and its application to the conclusion-centric testimony used in US courts. In a comment posted on July 21 (and reproduced below), Mr. Hunt generously took the time to respond. Taken together, his webinar presentation, my posting on it, and his comment on this blog discuss three issues:
(1) What does Rule 704 reveal about the proper or permissible role of an expert witness?
(2) Are certain publications on the desirability of the evidence-centric approach deficient or misleading for failing to mention Rule 704?
(3) Does Rule 704 pertain to every source-attribution opinion?
As explained below, the short answers are "not much," "no," and "no," respectively. In addition, an essay on the first and third issues will appear in the Winter 2020 issue of Jurimetrics Journal.

I

Mr. Hunt commented that “I don’t disagree that Rule 704 ‘does not counsel in favor’ of admitting expert opinion on an ultimate issue. I never said or implied anything or the sort.” At the same time, he wrote that “the Federal Rules (and most state rules of evidence) affirmatively allow an expert to offer an opinion about a proposition.”

Listening to the talk, I thought I was hearing the germ of a legal argument that Rule 704 not only allows, but also affirmatively supports the admission of source-attribution opinions such as “I am sure that the defendant is the source of the latent print.” The recording of the webinar is on the Forensic Technology Center of Excellence website, I transcribed the two small parts that pertain to Rule 704. I think I have captured all the words on Rule 704 and source opinions; they are in Appendix A.

The talk begins with a reference to “lots of statements” on the role of the expert witness “that have been pushed out in the literature” and therefore merit “a deeper dive and look at the American judicial system and the role of the expert in it.” After reviewing the most important rules affecting expert and scientific testimony generally, Mr. Hunt quoted several statements from organizations to the effect that experts ought to explain the degree to which the scientific data support the relevant hypotheses (“propositions” about the source of the data) rather than opine on the truth of the hypotheses themselves. Various forensic scientists, psychologists, law professors, and others have been advocating this approach for decades.

Mr. Hunt cautioned the audience as follows:
However, in the US system there is Rule 704, which is a rule that you don't hear much about, and I think there are some people who would like to pretend that it doesn't exist, but it actually goes against that school of thought and allows an expert to opine on an ultimate contested proposition before the court, not only on the strength of the evidence for or against the proposition. Specifically an opinion is not objectionable just because it embraces an ultimate issue, and you can add proposition in there as well.
He concluded his exposition of Rule 704 with the observation "So in the DNA context this would allow a source-attribution statement" and then described a few cases that accepted DNA source conclusions.

To my ear, the suggestion that people are pretending that Rule 704 does not exist because "it actually goes against that school of thought" sounded like an argument that Rule 704 is an affirmative reason for a court to allow traditional source-attribution testimony. My criticism of this argument (whether or not it was Mr. Hunt's) had two components. First, sometimes source opinions are not opinions on "ultimate" issues or facts—things like whether the killer was insane or whether the deceased committed suicide. In those cases, the rule does not even come into play.

Second, to say that Rule 704(a) "allows" source attributions is merely to say that a previous and ill-defined common-law rule on opinion testimony is no longer a reason to exclude the testimony. Rule 704(a) just cancels out a categorical rule of exclusion, leaving the court with a big zero. The rule does not militate either for or against the idea that forensic scientists should avoid proffering source opinions. It merely means that the objecting party had better have some other reason for the court to exclude the opinion.

This much is true whether the issue or fact is called ultimate or intermediate. Parties opposing source-attribution opinions could argue for exclusion under the helpfulness requirement of Rule 702 or the balancing mandated by Rule 403. The documents cited by Mr. Hunt do not invoke these rules, just as they do not discuss Rule 704, because they are not making a legal argument.

In emphasizing that Rule 704 is merely a rule of non-exclusion, I hasten to add that I am not suggesting that Mr. Hunt misrepresented the rule as mandating admission of ultimate-opinion testimony. He did nothing of the kind. The undisputed legal principle is that Rule 704 is agnostic toward ultimate-opinions. As Mr. Hunt agreed, it does not counsel in favor of their admission. Once that much is understood, readers can decide for themselves whether the rule “actually goes against that school of thought” that asks criminalists to avoid expressing opinions on the truth of hypotheses.

II

To emphasize that he was not suggesting "that Federal Rule of Evidence 704 somehow conflicts with the evidence-centric approach to evaluating laboratory results," as I originally put it, Mr. Hunt explained that he only was trying to correct a possible misunderstanding of what he called the "role assignment" documents:
My comments about FRE 704 were directed to the claim of ‘role assignment’ by the authors of the cited sources. The documents I cited (ENFSI, Evett, NIFS, AFSP, NCFS, ASA) make statements such as ‘the role of the expert is to comment on the probability of their findings, given these propositions and not on the propositions themselves.’ (NIFS).

I simply noted that this purported assignment of ‘roles’ is more limited than that which is permitted by the Federal Rules of Evidence. None of these documents concede (or even acknowledge) that the Federal Rules (and most state rules of evidence) affirmatively allow an expert to offer an opinion about a proposition, rather than limiting that opinion to the relative strength of the evidence in support of competing propositions.

Omitting this information may mislead some to believe that the ‘role’ assigned by these documents to an expert is instead assigned by legal rules, rather than a ‘best practices’ world view.
***
In conclusion, to the extent there is any ‘tension’ or ‘inconsistency’ between Rule 704 and the noted references, it was created by the referenced documents’ omission of a discipline-specific basis for their claim that ‘the role of the expert is to comment on the probability of their findings, given these propositions and not on the propositions themselves.’ Each of the cited sources fail to make it clear that this limited ‘role’ is self-imposed, not legal in nature. That was the point of my observation.
Informing people that the law allows well founded source-attribution opinions is entirely reasonable. Only a few courts depart from that generalization. But the position papers and guides to evaluative reporting with which I am familiar surely are not deficient, deceptive, or misleading about this.

The document from which the quoted words, "role of the expert," come is the Introductory Guide to Evaluative Reporting prepared by a special committee created by the National Institute of Forensic Science. NIFS is a directorate within the Australian and New Zealand Policing Advisory Agency (ANZPAA). The "role assignment" language occurs within a box contrasting "traditional reporting" and "evaluation using a logical framework." I don't care for the phrase "logical framework" (to the extent it intimates that nothing else is logical), but the words "role of the expert" are appropriately used only to refer to the role of the expert within the stated alternative to the traditional approach. Let’s put the words back into the full sentence in which they occur (with the language that Mr. Hunt questioned in bold):
The fundamental principles of evaluative reporting or interpretation are that (i) the crime must be considered to have occurred within a framework of circumstances, (ii) that the findings must be considered in light of at least two competing propositions that will be guided by the case circumstances and (iii) that the role of the expert is to comment on the probability of their findings, given these propositions and not on the propositions themselves. (P. 6).
The first page of the NIFS Guide explains that the purpose is to introduce readers to an alternative to traditional and continuing practice and to “provide[] managers with advice on implementing evaluative reporting where they consider it appropriate.” (P. 1, emphasis added). A "discipline-specific basis for their claim" that reporting likelihoods has cross-disciplinary advantages over elimination-versus-identification opinions would make little sense in a document designed to enable "many traditional comparative forensic disciplines" to produce reports and testimony in "ways that can be standardised between analysts within, and between, organisations and forensic disciplines." (P. 6).

Little would be gained by reviewing the details of the other documents Mr. Hunt feels should have "conceded" or "acknowledged" the existence of Rule 704. I cannot see how many readers of these documents could construe them as describing the current rules of U.S. evidence law (or that of other countries), 1/ but it is always good to to distinguish between legal and statistical or scientific principles.

Adhering to this distinction, there is a legal argument for the role that the documents recommend for the expert. Proponents of this “likelihood-ratio” or “Bayes-factor” mode of presentation believe that it nicely captures the role of a scientist supplying the legal factfinder with information based strictly on the physical or chemical features of the specimens that are compared. DNA experts do not normally testify that the defendant's DNA is present in the crime-scene DNA. Instead, they testify to probability of the measured quantities of identifying DNA fragments (or to the probability of the alleles inferred from these measurements) if the defendant's DNA is not present in the sample. They also may testify to the probability if it is, or to the ratio of these probabilities.

To reach the further conclusion that the defendant's DNA actually is in the questioned sample would require information outside of the fragment lengths measured in the laboratory. In Bayesian terms, any conclusion about the true source "necessarily rests, in part, on an implicit assessment or assumption about the prior odds that the suspect is the source," and “[m]aking those assessments or assumptions takes forensic scientists beyond their scientific expertise in ways that arguably usurp the role of legal fact-finders." 2/ The core of the argument is that experts exceed their legally assigned role of assessing the physical evidence when they venture beyond the scientific data to which the likelihood ratio pertains. "Experts are rarely in a good position to evaluate the prior odds that the items they are comparing have a common source, and arguably have no business doing so." 3/

That criminalists dealing with other types of evidence were implicitly opining on prior probabilities beyond their expertise was not obvious in the days when it was thought that every specimen—I'll use fingerprints as an example—is unique. Uniqueness of prints makes it sound as if the criminalist does not have to think about the population from which a latent print came in order to conclude that the print had to be the defendant's. But uniqueness of friction ridge skin is not a sufficient basis for a source attribution. It overlooks the fact that latent print examiners do not deal with three-dimensional skin. They compare impressions of a complete and clear set of prints to incomplete and often low quality impressions. The two-dimensional representations vary from one impression to the next. The only question the analyst can answer strictly on the basis of the features (and knowledge of their joint distribution in a relevant population) is how the probability of a latent print being as similar as it is to the exemplar print—when it is, in fact, from the finger that produced the exemplar print—compares to the probability of the latent print being so similar when it is from someone else's finger.

This "role-assignment" argument is not conclusive. What if experts legitimately can say that in every relevant population of plausible size, the probability is very high that only one finger could have produced the observed features? Or that the likelihood ratio is so huge that it swamps the prior probability for an individual randomly drawn from that population? Then the expert who indulges in source attribution (without specifying a prior probability for the judge or jury to consider) arguably would not exceed the role of the expert defined in Federal Rule of Evidence 702.

III

Moving back to the content of Rule 704, it seems that we have a sharp disagreement about the reach of the "ultimate issue" rule that Rule 704 (building on earlier case law and academic criticism) decisively rejected. The rule applies to any opinion from any witness—expert or lay—that "embraces an ultimate issue." But what does that phrase mean? I maintained that Rule 704 does not apply to all source attributions because "the identity of the source of a trace is not necessarily an ultimate issue." To make this point, I wrote as follows:
To use the example of latent-print identification given in the webinar, the traditional opinion that a named individual is the source of a print is not an opinion on an ultimate issue. Courts have long allowed examiners to testify that the print lifted from a gun comes from a specific finger. But this conclusion is not an opinion on whether the murder defendant is the one who pulled the trigger. The examiner’s source attribution bears on the ultimate issue of causing the death of a human being, but the examiner who reports that the prints were defendant's is not opining that the defendant not only touched the gun (or had prints planted on it) but also pulled the trigger. Indeed, the latent print examiner would have no scientific basis for such an opinion on an element of the crime of murder.
Mr. Hunt's comment insists that every source attribution is an ultimate-issue opinion to which Rule 704 applies. Here is his argument:
[Y]our blog states that ‘the identity of the source of a trace is not necessarily an ultimate issue.’ As evidence for that assertion, you used my latent print example, claiming that ‘the traditional opinion that a named individual is the source of a print is not an opinion on an ultimate issue.’

Your claim is directly contrary to the contents of the judicial opinion I cited in the questioned slide, which states in full: ‘[Defendant] . . . contends that the district court erred in allowing the expert to specifically testify that he had no doubt that the prints on the parking stub were [Defendant’s] because the expert was testifying to the "ultimate issue’ of whether it was Sherwood's print. This argument fails in light of Fed. R. Evid. 704, which permits expert testimony on the ultimate issue in a case.” United States v. Sherwood, 98 F.3d 402, 408 (9th Cir. 1996).

In Sherwood, the ultimate issue was not described as a connection between the latent print and activity-level behavior; rather, it was the expert’s testimony about the source of that print. Your description of an ultimate issue as limited to one in which (using your example), ‘the murder defendant is the one who pulled the trigger’ is a narrower description of the types of factual assertions entailed by Rule 704 than those described throughout case law. As such, your explanation of the Rule is not entirely correct.

For other examples of courts describing identification testimony as ‘ultimate’ in nature, see U.S. v. Rose, 731 F.2d 1337, 1346-47 (print match to shoe) United States v. Bice-Bey, 701 F.2d 1086 (4th Cir. 1883) (voice identification); United States v. Sellers, 566 F.2d 884 (4th Cir. 1977) (identification of suspect in a picture); United States v. Couser, 15 Fed. R. Evid. Serv. (CBC) 837 (4th Cir. Apr. 5, 1984) (comparison of suspect’s features to photograph). These cases clearly demonstrate that source identification testimony constitutes an opinion that embraces an ‘ultimate’ fact.
It is true that the Ninth Circuit in Sherwood referred to the source attribution as an "ultimate" opinion. I assume every other case he lists did the same. However, this use of authority is an instance of looking for law in all the wrong places. Every time a party objects, "Your honor, my identity as the source of the trace is an ultimate issue!", the court can summarily reject the objection by citing Rule 704's text. Why bother to try to trace the boundaries of the "the so-called 'ultimate issue' rule" (Fed. R. Evid. 704 Advisory Committee's Note) when the murky rule "is specifically abolished" (id.)? Regardless of the reach of Rule 704, the outcome is the same. If the expert opinion is outside the old rule, then it is "not objectionable" just because it embraces an ultimate issue. If it is covered by the old rule, it is "not objectionable" because the old rule is no longer a barrier.

So the fact that modern courts often assume—without analysis—that a source opinion is ultimate-opinion testimony tells us next to nothing about how the opinion would have fared under the defunct common-law exclusionary rule. To answer that arcane question, we must look to the old cases applying that rule to exclude opinion testimony.

Where should we begin? McCormick on Evidence § 12, at 80 (7th ed. 2013), documents the "general doctrine that witnesses may never give their opinions or conclusions on an ultimate fact in issue" with two cases: United States v. Spaulding, 293 U.S. 498 (1935), and State v. Carr, 196 N.C. 129, 144 S.E. 698 (1928). In Spaulding the "ultimate issue" was whether the plaintiff was totally disabled while an insurance policy was in force. He produced "medical opinions" that he "became totally and permanently disabled before his policy lapsed." The Supreme Court wrote "that question is not to be resolved by opinion evidence. It was the ultimate issue ... . The experts ought not to have been asked or allowed to state their conclusions on the whole case." 293 U.S. at 506.

Carr is a homicide case. The defense was that a man who died in bed of a gunshot wound had committed suicide. The state produced a medical doctor who testified that "I don't think it is possible for the deceased to have fired the gun and made the wound that I saw." 144 S.E. at 700. The North Carolina Supreme Court held that the trial court erred in admitting the opinion. It explained that
[T]he opinion or inference of the witness must not be an answer to the exact issue which the jury is to determine. When the witness testified that he did not think it possible for the deceased to have fired the gun and to have made the wound he necessarily testified in effect that in his opinion the deceased did not kill himself. True, the "exact issue" was whether the defendants are guilty, but if the deceased killed himself the conclusion that the defendants did not kill him would necessarily follow.
Id. These cases reveal that the essence of the ultimate-issue rule is that the witness may not opine on a fact that determines whether an element of the offense or complaint, or of the defense, is satisfied.

Other facts are intermediate. They are links in a chain of reasoning that terminates in the answer to one of these ultimate issues. In an action for a declaration of paternity, for example, an expert opinion that (based on DNA tests) the defendant is the father, falls under the former ultimate-issue rule. But suppose the government introduces an expert opinion that a murder defendant is the father of a fetus whose remains were found along with the mother's merely to show that he had a motive to kill the mother, who refused to have an abortion as he had demanded. Now the source conclusion is not an opinion on an ultimate issue. The jury can accept the source opinion but conclude that defendant, despite the motive, was not the killer. In contrast to Spaulding, the expert opinion does not decide "the whole case." Unlike Carr, the ultimate conclusion does not "necessarily follow."

On which side of the ultimate-intermediate line the examiner's opinion in Sherwood falls I cannot say. The opinion is too sparse. But the question is academic. One purpose of Rule 704 is to spare the courts from having to pin these labels on opinion testimony. (See Advisory Committee's Note, observing that the old rule was "difficult of application"). With the rule in place, they can just say "so what—the argument fails in light of Rule 704." But that does not mean that a source opinion is always an opinion on "an ultimate issue." That depends on how the opinion fits into the case.When the source opinion is near enough to the "ultimate" endpoint, Rule 704 does not say, "OK, let it in." In conjunction with Rule 702, it says "Well, yes, but there is no mechanical rule excluding it on that basis alone—just address how helpful it is for the jury to hear it."

When courts address the question of helpfulness, they just might conclude that instead of old-style source attributions, statements about the probabilities of the findings under different hypotheses would fulfill the role of helping the factfinder find the facts better that traditional expressions of an examiner's belief that the defendant is the source of the trace. Or they might not. Either way, no one is trying to deny that Rule 704 exists. Why would they when it sheds so little light on the choice between traditional conclusion-centric opinons (source attributions that, to a Bayesian, reflect an expert's personal prior probabilities and utilities) and evidence-centric ones (such as Bayes factors that enable experts to express the probative value of the findings without venturing into that nonscientific territory)?


NOTES
  1. Australia's uniform Evidence Acts § 80(a), adopted by many of its jurisdictions, abolished the common-law rule against ultimate opinions ("[e]vidence of an opinion is not inadmissible only because it is about—a fact in issue or an ultimate issue."). New Zealand did the same in its 2006 Evidence Act ("(2) An opinion by an expert is not inadmissible simply because it is about—(a) an ultimate issue to be determined in a proceeding; or (b) a matter of common knowledge."). 
  2. William C. Thompson, How Should Forensic Scientists Present Source Conclusions?, 48 Seton Hall L. Rev. 773, 791 (2018).
  3. Id.; see also David H. Kaye et al., The New Wigmore, A Treatise on Evidence: Expert Evidence § 2.2.2, at 44 (2d ed. 2011) (notes omitted):
    But this form of opinion leaves the jury with an inappropriate choice. If the jury is to accept the opinion on a matter within the witness’s expertise, it must also accept the expert’s (presumably unarticulated) assessment of evidence that has nothing to do with the expertise and that falls within the jury’s ability to asses. Or, it must accept an unvalidated and implausible theory of individualization.

APPENDIX A

Ted R. Hunt, The Expert’s Role in the American Judicial System (Probabilistic Genotyping of Evidentiary DNA Typing Results – An Online Workshop Series, Module 8: Probabilistic Genotyping Summation and Special Topics, July 17th, 2019, available at https://forensiccoe.org/ (excerpt of all remarks on Rule 704 and source conclusion testimony)

*** My talk is entitled "the role of the expert" because there aren't special rules that just apply to DNA analysts and no one else, and the reason that I want to talk about it today is, in the forensic literature in the last number of years, lots of statements have been made about the role of the courts and the role of the experts, and those statements aren't specific to any judicial system. They're pretty much general statements that have been pushed out in the literature so I thought we'd take a deeper dive and look at the American judicial system and the role of the expert in it.
***

Now I mentioned before at the beginning of this talk there's a lot of literature on the role of the courts and the role of the expert in the forensic literature. I have some examples here. The European Network has its guidelines for evaluative reporting in forensic science. Number 3 here in this excerpt talks about the role of the forensic practitioner to consider the probability of the findings considering the propositions and not the probability of those propositions. Similarly, other documents in the community from Australia and New Zealand, from Dr. Evett, from the Association of Forensic Science Service Providers. All basically say the same thing, talking about the role of the expert to comment on the probability of the findings given those propositions and not the prepositions themselves.

This has gotten into the US forensic community to a certain extent. On the left is the National Commission [on Forensic Science] Views Document. I just want to be clear this document did not get voted to approval through the Commission, but nevertheless there is a final stand-alone version to that document that basically says the same thing: Forensic practitioners should not state that a specific individual or object is the source of forensic evidence and refers to the formal method of evaluative reporting as well. And basically the same people who wrote that document wrote the American Statistical Association document which, for the most part, says the same thing, discouraging the statements to the effect that a specific person or object is the source of evidence.

However, in the US system there is Rule 704, which is a rule that you don't hear much about, and I think there are some people who would like to pretend that it doesn't exist, but it actually goes against that school of thought and allows an expert to opine on an ultimate contested proposition before the court, not only on the strength of the evidence for or against the proposition. Specifically an opinion is not objectionable just because it embraces an ultimate issue, and you can add proposition in there as well.

I need to add that 704 does not lower the bar to admit otherwise inadmissible evidence. It will work in conjunction with the other rules of evidence. The only exception in the American system or the US federal system is that the expert may not opine on the defendant's mental state —an ultimate legal proposition before the court—or witness credibility. However, they can opine on an ultimate issue.

Just as an example, this is a latent fingerprint case from the 9th circuit in 1996 [United States v. Sherwood, 98 F.3d 402 (9th Cir. 1996)] and a defendant objected that an expert was specifically testifying that he had no doubt that the prints on the piece of evidence were the defendant’s. The defendant objected because the expert was testifying to an ultimate issue, and the court basically said, so what—the argument fails in light of 704, which permits an expert to opine on ultimate issues.

Like I said before, I’m trying to be very descriptive. I’m not being normative, and I’m not weighing in on the logic or the advisability of this. I'm simply trying to correctly state what I understand the law to be.

So in the DNA context this would allow a source attribution statement. ***


APPENDIX B

Comment posted to this blog, July 21, 2019

Professor Kaye,
     My comments about FRE 704 were directed to the claim of ‘role assignment’ by the authors of the cited sources. The documents I cited (ENFSI, Evett, NIFS, AFSP, NCFS, ASA) make statements such as ‘the role of the expert is to comment on the probability of their findings, given these propositions and not on the propositions themselves.’ (NIFS).
     I simply noted that this purported assignment of ‘roles’ is more limited than that which is permitted by the Federal Rules of Evidence. None of these documents concede (or even acknowledge) that the Federal Rules (and most state rules of evidence) affirmatively allow an expert to offer an opinion about a proposition, rather than limiting that opinion to the relative strength of the evidence in support of competing propositions.
     Omitting this information may mislead some to believe that the ‘role’ assigned by these documents to an expert is instead assigned by legal rules, rather than a ‘best practices’ world view. I never said or implied that Rule 704 ‘stands in the way’ of an expert’s provision of a more limited opinion about the strength of the evidence relative to competing propositions.
     In addition, your blog states that ‘the identity of the source of a trace is not necessarily an ultimate issue.’ As evidence for that assertion, you used my latent print example, claiming that ‘the traditional opinion that a named individual is the source of a print is not an opinion on an ultimate issue.’
     Your claim is directly contrary to the contents of the judicial opinion I cited in the questioned slide, which states in full: ‘[Defendant] . . . contends that the district court erred in allowing the expert to specifically testify that he had no doubt that the prints on the parking stub were [Defendant’s] because the expert was testifying to the "ultimate issue’ of whether it was Sherwood's print. This argument fails in light of Fed. R. Evid. 704, which permits expert testimony on the ultimate issue in a case.” United States v. Sherwood, 98 F.3d 402, 408 (9th Cir. 1996).
      In Sherwood, the ultimate issue was not described as a connection between the latent print and activity-level behavior; rather, it was the expert’s testimony about the source of that print. Your description of an ultimate issue as limited to one in which (using your example), ‘the murder defendant is the one who pulled the trigger’ is a narrower description of the types of factual assertions entailed by Rule 704 than those described throughout case law. As such, your explanation of the Rule is not entirely correct.
     For other examples of courts describing identification testimony as ‘ultimate’ in nature, see U.S. v. Rose, 731 F.2d 1337, 1346-47 (print match to shoe) United States v. Bice-Bey, 701 F.2d 1086 (4th Cir. 1883) (voice identification); United States v. Sellers, 566 F.2d 884 (4th Cir. 1977) (identification of suspect in a picture); United States v. Couser, 15 Fed. R. Evid. Serv. (CBC) 837 (4th Cir. Apr. 5, 1984) (comparison of suspect’s features to photograph). These cases clearly demonstrate that source identification testimony constitutes an opinion that embraces an ‘ultimate’ fact.
     I don’t disagree that Rule 704 ‘does not counsel in favor’ of admitting expert opinion on an ultimate issue. I never said or implied anything or the sort.
    In conclusion, to the extent there is any ‘tension’ or ‘inconsistency’ between Rule 704 and the noted references, it was created by the referenced documents’ omission of a discipline-specific basis for their claim that ‘the role of the expert is to comment on the probability of their findings, given these propositions and not on the propositions themselves.’ Each of the cited sources fail to make it clear that this limited ‘role’ is self-imposed, not legal in nature. That was the point of my observation.
     Thanks for your participation in the webinar series and for your interest in my talk.

Ted R. Hunt
Senior Advisor on Forensic Science
U.S. Department of Justice

Thursday, March 21, 2019

Admitting Palm Print Evidence in United States v. Cantoni

A second U.S. District Court judge in the Eastern District of New York has ruled that latent fingerprint analysts may testify to categorical source attributions despite concerns voiced in prominent reviews of the field. 1/ Given the research demonstrating the ability of latent fingerprint examiners to associate prints with their sources, that is no surprise. More problematically, the case suggests that defense experts should not be permitted to testify about error rates in a forensic method unless they "conclude that [the analyst’s] conclusions were erroneous or that the process used was fundamentally unreliable."

In United States v. Cantoni, No. 18-cr-562 (ENV), 2019 WL 1259630 (E.D.N.Y. Mar. 19, 2019), the government was prepared to call three examiners from the New York City Police Department Latent Print Section to testify that a palm print found on a bank robber's note demanding money is Cantoni’s. Cantoni moved before trial for an order excluding this source conclusion or, alternatively, directing the examiners to include cautions about the risk of a false identification. In what the court called "throwing incense in Caesar’s face," Cantoni apparently relied on a 2017 report from a committee established by the AAAS, 2/ a 2016 report of the President’s Council of Scientific Advisors (PCAST), 3/ a 2012 report of a committee formed by the National Institute of Standards and Technology (NIST), 4/ and a 2006 report of the Inspector General of the Department of Justice. 5/ These did not get him very far.

Conclusions Admissible

In response to the demand for outright exclusion, district judge Eric N. Vitaliano pointed out that the PCAST report acknowledged the validity of latent print analysis. The studies of the ability of examiners to draw categorical conclusions that impressed PCAST involved fingerprints, but presumably those findings can be extrapolated to palm prints.

Cantoni tried to sidestep this evaluation by arguing that the NYPD examiners did not follow all the recommendations in the PCAST report. These were said to include testing examiners for minimal proficiency, disclosing the order in which the prints were examined and any extraneous facts that could have influenced the conclusions, documenting the comparison, and verifying that the latent print is comparable in quality to the prints used in validation studies.

The court noted that several of these matters, such as the fact that the examiners underwent proficiency testing, were not at issue in the case and that any remaining deficiencies did not make the process “so fundamentally unreliable as to preclude the testimony of the experts.” The court wrote that considering “Daubert’s liberal standard for the admission of expert testimony,” concerns about cognitive bias that underlay the PCAST procedures “are fodder for cross-examination rather than grounds to exclude the latent print evidence entirely.”

Limitations on the Testimony

Although the court did not exclude the examiners’ testimony in toto, it agreed to ensure that the testimony not include assertions “that their conclusion is certain, that latent print analysis has a zero error rate, or that their analysis could exclude all other persons who might have left the print.” The government had no problem with this constraint. It acknowledged that “[t]he language and claims that are of concern to defense counsel are disfavored in the latent print discipline.” One might think that analysts would not testify that way anymore, 6/ but some latent print examiners continue to offer conclusions in the traditional manner. 7/

Defendant’s remaining efforts to shape the testimony on direct examination were less successful. He wanted “the government [to] acknowledge, through the examiners or by stipulation, that studies have found the [false positive] error rate to be as high as 1 in 18 or 1 in 306.”

These numbers do not fairly or fully represent the findings of the validation studies from which they are drawn. These studies found smaller rates of false-positive errors. The quoted figures are inflated by taking the upper limit of a one-sided 95% confidence interval above the observed false-positive proportion. That a large error rate is not incompatible with a small study (one that lacks statistical power) cannot reasonably be interpreted as a finding that the error rate is large.

Perhaps the government pointed out this misuse of statistics, but the opinion describes a different rejoinder: “The government suggests that the studies Cantoni cites are inapposite because they involved the Federal Bureau of Investigation and the Miami-Dade Police Department.”

That response is disappointing. The most compelling study is the FBI-Noblis test of the accuracy of latent fingerprint examiners generally. It consisted of an experiment in which the researchers — but not the analysts — knew which pairs of prints were from the same finger and which were from fingers of different individuals. The test subjects were hardly limited to FBI examiners. To the contrary,
In order to get a broad cross-section of the latent print examiner community, participation was open to practicing latent print examiners from across the fingerprint community. A total of 169
latent print examiners participated; most were volunteers, while the others were encouraged or required to participate by their employers. Participants were diverse with respect to organization,
training history, and other factors. 8/
The government may have meant to argue that whereas the results of studies can be used to demonstrate that error rates are small among representative examiners, such an average figure is not a precise measure of the probability of error in a specific case. That much is true, but it does not mean that the error rates in the experiments are irrelevant or useless to a proper understanding of the risk of error in New York City.

The court did not reach such nuances. Judge Vitaliano wrote that “[c]ross-examination is the appropriate means to elicit weaknesses in direct testimony” and that “Cantoni may explore the error rates generated by the studies on cross-examination.” Yet, the court refused to let the defense inform the jury of error rates and other limitations on latent-print examinations through an expert of its choice — the social scientist, Simon Cole, who has researched the history of fingerprinting and written extensively about it.

Keeping Cole Away

The court provided two reasons for excluding Cole’s testimony. First, “this is a matter that may be explored on cross-examination and does not require an expert to offer an opinion.” However, the opportunity to cross-examine one expert normally does not preclude a party from calling its own expert. In a toxic tort case in which a manufacturer denies that its product is toxic, for example, the manufacturer ordinarily would be able not only to cross-examine plaintiff’s physicians, toxicologists or epidemiologists, but also to present its own comparable experts.

The difference, one might argue, is that “Dr. Cole’s opinions appear to be directed at NYPD’s methods for latent print analysis in general rather than specific issues in this case. ... [H]e does not thereby conclude that NYPD’s conclusions were erroneous or that the process used was fundamentally unreliable.” However, no rule of law requires an expert to give a final opinion. Simply providing background information — such as the accuracy of a medical diagnostic test — may assist the jury in a tort case in which plaintiff’s testifying physician relied on the diagnostic test in forming his or her opinion.

Second, the court suggests that the rule against hearsay makes the educational expert’s testimony inadmissible. Judge Vitaliano wrote that “to the extent that Dr. Cole merely plans to convey the contents of studies he did not conduct, he is poised to act as a conduit for hearsay, which is a prohibited role for an expert.” In the toxic tort case, however, the defendant’s epidemiologist could report the findings of other researchers. Federal Rule of Evidence 703 often permits experts to discuss inadmissible hearsay as the basis for their own opinion. In our toxic tort case, the defense need not call the authors of the studies as witnesses to lay a foundation for the epidemiologist to describe all the studies on point to explain why the substance has not been scientifically established to be toxic.

Again, one might argue that there is a difference. Epidemiologists study the types and quality of proof of toxicity; as a result of their specialized training in methodology, they can give an expert opinion on the state of the science. To inform that opinion, the epidemiologist can — indeed, must — review the published work of other researchers. But is Dr. Cole, as a sociologist of science, qualified to give the opinion that “there is now consensus in the scientific and governmental community that categorical conclusions of identification — such as the one made in this case — are scientifically indefensible”? The opinion in Cantoni does not squarely confront this Rule 702 question.

Whatever the answer to the question of qualifications, 9/ the hearsay rule should not preclude a statistician or other methodological expert from giving an opinion that well-designed research suggests that latent fingerprint examiners who have been the subject of experiments reach incorrect source conclusions under certain conditions or at certain rates. Analogous expert testimony about the accuracy of eyewitness identifications is now plainly admissible (in the court's discretion) in most jurisdictions. Cross-examination (this time by the government) can explore the extent to which these findings are applicable to the case at bar. If the known error rates are clearly inapposite, they should be excluded under Rule 403, but the hearsay objection seems misplaced.

NOTES
  1. The previous cases are noted in Ignoring PCAST’s Explication of Rule 702(d): The Opinions on Fingerprint Evidence in Pitts and Lundi, Forensic Sci., Stat. & L., July 16, 2018, and More on Pitts and Lundi: Why Bother with Opposing Experts?, Forensic Sci., Stat. & L., July 17, 2018.
  2. William Thompson, John Black, Anil Jain & Joseph Kadane, Forensic Science Assessments: A Quality and Gap Analysis, Latent Fingerprint Examination (2017).
  3. Executive Office of the President, President’s Council of Advisors on Sci. & Tech., Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods (2016).
  4. NIST Expert Working Group on Human Factors in Latent Print Analysis, Latent Print Examination and Human Factors: Improving the Practice Through a Systems Approach (David H. Kaye, ed. 2012).
  5. U.S. Dep't of Justice, Office of the Inspector General, A Review of the FBI’s Handling of the Brandon Mayfield Case (2006).
  6. Cf. Another US District Court Finds Firearms-mark Testimony Admissible in the Post-PCAST World, Forensic Sci., Stat. & L., Mar. 15, 2019.
  7. Nicole Westman, Bad Police Fingerprint Work Undermines Chicago Property Crime Cases, Chi. Reporter, Mar. 21, 2019.
  8. Bradford T. Ulery et al., Accuracy and Reliability of Forensic Latent Fingerprint Decisions, 108 Proc. Nat’l Acad. Sci.  7733, 7734 (2011). For more discussion of the characteristics of the volunteers, see Part II of Fingerprinting Under the Microscope: Examiners Studied and Methods, Forensic Sci., Stat. & L., Apr. 27, 2011.
  9. For discussion of the qualifications of non-forensic scientists to opine on the state of forensic science methods, see David H. Kaye et al., The New Wigmore on Evidence: Expert Evidence, Ch. 2 (2d ed. 2011) (2019 cumulative update).

Tuesday, July 17, 2018

More on Pitts and Lundi: Why Bother with Opposing Experts?

In the post-PCAST cases of United States v. Pitts 1/ and United States v. Lundi 2/, the government prevented a scholar of the development and culture of fingerprinting from testifying for the defense. The proposed witness was Simon Cole, Professor of Criminology, Law and Society at the Department of Social Ecology of the University of California (Irvine). Pitts “contend[ed] that Dr. Cole’s testimony [was] necessary ‘contrary evidence’ that calls into question the reliability of fingerprint analysis.” Lundi wanted Cole to testify about “forensic print analysis, in particular in the areas of accuracy and validation,” including "best practices." The federal district court would not allow it.

Qualifications of an Expert Witness

In Pitts, the government first denied that Cole had the qualifications to say anything useful about fingerprinting. It maintained (in the court's words) that “Dr. Cole (1) is ‘not a trained fingerprint examiner’; (2) ‘has not published peer-reviewed scientific articles on the topic of latent fingerprint evidence’; and (3) ‘has not conducted any validation research in the field.’” The court neither accepted nor clearly rejected this argument, for it decided to keep Cole away from the jury on a different ground.

Exclusion for lack of qualifications would have done violence to Federal Rule of Evidence 702. First, Cole was not going to offer an opinion as a criminalist (which he is not) nor as an interdisciplinary scholar (which he is) on whether the examiner in the case had accurately perceived, compared, and evaluated the images. More likely, he would have opined on the extent to which scientific studies have shown that fingerprint examiners can distinguish between same-source and different-source prints. Training and experience in conducting fingerprint identifications is largely irrelevant to this task.

Second, Rule 702 does not require someone to publish peer-reviewed articles on a topic to be qualified to give an opinion as to the state of the scientific literature. Epidemiologists and toxicologists, for example, can opine about the toxicity of a compound without first publishing their own research on the compound's toxicity. Finally, there is no support in logic or law for the notion that someone has to conduct his or her own validity study to have helpful information on the studies that others have done and what these studies prove.

The Panacea of Cross-examination

The government’s other argument was “that Dr. Cole’s testimony will not assist the trier of fact.” But this argument was garbled:
Specifically, the government points out that Dr. Cole’s only disclosed opinion is that the government’s expert’s testimony “exaggerates the probative value of the evidence because such testimony improperly purports to eliminate the probability that someone else might be the source of the latent print.” “Professor Cole fails to provide any analysis of why latent fingerprint evidence [in general] is so unreliable that it should not be submitted to the jury or, if such evidence can be reliable in some circumstances, what precisely the NYPD examiners did incorrectly in this case.” Dr. Cole is not expected to testify that the identification made by the government’s expert in this case is unreliable or that the examiners made a misidentification. Therefore, the government argues Dr. Cole’s opinion goes to the weight of the government’s evidence, not its admissibility. (Citations and internal quotation marks omitted.)
Chief Judge Dora L. Irizarry had already ruled that a source attribution made with an acknowledgment of at least a theoretical possibility that the match could be a false positive was admissible. If expert evidence is admitted, the opposing party is normally permitted to counter it with expert testimony that it deserves little weight. To argue that just because evidence goes to weight, rebuttal evidence about its weight is inadmissible makes no sense.  Once the evidence is admitted, it's weight is the only game in town.

The real issue is what validity or possibility-of-error testimony would add to the jury's knowledge. In this regard, Judge Irizarry wrote that
The Court is not convinced that Dr. Cole’s testimony would be helpful to the trier of fact. The only opinion Defendant seeks to introduce is that fingerprint examiners “exaggerate” their results to the exclusion of others. However, the government has indicated that its experts will not testify to absolutely certain identification nor that the identification was to the exclusion of all others. Thus, Defendant seeks admit Dr. Cole’s testimony for the sole purpose of rebutting testimony the government does not seek to elicit. Accordingly, Dr. Cole’s testimony will not assist the trier of fact to understand the evidence or determine a fact in issue. (Citations omitted.)
At first blush, this seems reasonable. If the only thing Cole was prepared to say was that fingerprinting does not permit “absolutely certain identification,” and if the fingerprint examiners will have said this anyway, why have him repeat it?

But surely Cole (or another witness— say, a statistician) could have testified to something more than that. An expert with statistical knowledge could inform the jury that although there is very little direct evidence on how frequently fingerprinting experts err in making source attributions in real casework, experiments have tested their accuracy, and the researchers detected errors at various rates. This information could “assist the trier of fact to understand the evidence or determine a fact in issue.” So why keep Cole from giving this “science framework” testimony?

The court’s answer boils down to this:
Moreover, the substance of Dr. Cole’s opinion largely appears in the reports and attachments cited in Defendant’s motion to suppress .... For example, Dr. Cole’s article More Than Zero contains a lengthy discussion about error rates in fingerprint analysis and the rhetoric in conveying those error rates ... , and the PCAST Report notes that jurors assume that error rates are much lower than studies reveal them to be (PCAST Report at 9-10 (noting that error rates can be as high as one in eighteen)). Defendant identifies no additional information or expertise that Dr. Cole’s testimony provides beyond what is in these articles and does not explain why cross-examination of the government’s experts using these reports would be insufficient. 3/
Now, I think the 1 in 18 figure is mildly ridiculous, 4/ but there is no general rule that because published findings could be introduced via cross-examination, a party cannot call on an opposing expert to present or summarize the findings. First, the expert being cross-examined might not concede that the findings are from authoritative sources. This occurred repeatedly when a number of prosecution DNA experts flatly refused to acknowledge the 1992 NAS report on forensic DNA technology as authoritative. That created a hearsay problem for defendants. After all, the authors of the report were not testifying and hence were not subject to cross-examination. The rule against hearsay applies to such statements because the jury would have to evaluate the truthfulness of the statements without hearing from the individuals who wrote them.

Therefore, counsel could not quote or paraphrase the report’s statements over a hearsay objection unless the report fell under some exception to the rule against hearsay. The obvious exception—for “learned treatises”—does not apply unless the report first is “established as a reliable authority by the testimony or admission of the witness or by other expert testimony or by judicial notice.” 5/

In Pitts, however, it appears that the government’s experts were willing to concede that the NAS and PCAST reports were authoritative (even though a common complaint from vocal parts of the forensic-science community about both reports was that they were not credible because they lacked representation from enough practicing forensic scientists). Moreover, a court might well have to admit the PCAST report under the hearsay exception for government reports.

Nonetheless, a second problem with treating cross-examination as the equivalent of testimony from an opposing expert is that it is not equivalent. By way of comparison, would  judges in a products liability case against the manufacturer of an alleged teratogen reason that the defense cannot call an expert to present and summarize the results of studies that address the strength of the  association between exposure and birth defects but rather can only ask the plaintiff’s experts about the studies?

In criminal cases, even if the defense expert eschews opinions on whether the defendant is the source of the latent prints as beyond his (or anyone’s?) expertise, the jury might consider this expert to be more credible and more knowledgeable about the underlying scientific literature than the latent print examiners. Examiners understandably can have great confidence in their careful judgments and in the foundations of the important work that they do. It would not be surprising for their message on cross-examination (or re-direct examination) to be, yes, errors are possible and they have occurred in artificial experiments and a few extreme cases, but, really,the process is highly valid and reliable. An outside observer may have a less sanguine perspective to offer even when discussing the same underlying literature.

Cross-examination is all well and good, but cross-examination of experts is delicate, difficult, and dangerous. Confining the defense to posing questions about specific studies in lieu of its own expert testimony about these studies is not normal. Court instructions about error probabilities (analogous to instructions about the factors that degrade eyewitness identifications) might be a device to avoid unduly time-consuming defense witnesses, but those do not yet exist. The opportunity to cross-examine the other party's witnesses rarely warrants depriving a party of the right to present testimony from its experts.

NOTES
  1. United States v. Pitts, 16-CR-550 (DLI), 2018 WL 1169139 (E.D.N.Y. Mar. 2, 2018).
  2. United States v. Lundi, 17-CR-388 (DLI), 2018 WL 3369665 (E.D.N.Y. July 10, 2018).
  3. The opinion in Lundi is similar:
    The government seeks to preclude Defendant’s proposed expert, Dr. Cole, from testifying, and points to this Court’s decision in Pitts, ... . The government argues that, as was the case in Pitts, Dr. Cole’s anticipated testimony would serve to rebut testimony from the government’s experts that the government does not expect to elicit. ... The government argues further that Dr. Cole’s additional proposed testimony, which would address the reliability of fingerprint examinations and the “best practices” to be followed when conducting such examinations, is not distinguishable from the information contained in the reports Defendant attached to his motion, and with which he can cross examine the government’s experts. ...

    Defendant claims that Dr. Cole’s testimony is necessary in this case because the reports could not be introduced through the government’s experts. .... However, the government has given every indication that its experts would recognize these reports, such that Defendant can use them on cross-examination. See Opp’n at 18 (“[t]o the extent the defendant wants to cross examine the [fingerprint] examiners on the basis of the empirical studies in which the error rates cited in the defendant’s motion were found, the defendant is free to do so....”). The Court finds that Dr. Cole’s testimony would not assist the trier of fact. See Pitts, 2018 WL 1169139, at *3. Accordingly, the testimony is precluded.
  4. See David H. Kaye, On a “Ridiculous” Estimate of an “Error Rate for Fingerprint Comparisons”, Forensic Sci., Stat. & L., Dec. 10, 2016, http://for-sci-law.blogspot.com/2016/12/on-ridiculous-estimate-of-error-rate.html.
  5. Federal Rule of Evidence 803(18); see generally David H. Kaye, David A. Bernstein, & Jennifer L. Mnookin, The New Wigmore: A Treatise on Evidence: Expert Evidence § 5.4 (2d ed. 2011).

Monday, July 16, 2018

Ignoring PCAST’s Explication of Rule 702(d): The Opinions on Fingerprint Evidence in Pitts and Lundi

With the release of an opinion in February and another in July 2018, the District Court for the Eastern District of New York became at least the second federal district court to find that the 2016 report of the President’s Council of Advisors on Science and Technology (PCAST) [1] did not militate in favor of excluding testimony that a defendant is the source of a latent fingerprint. Chief Judge Dora L. Irizarry wrote both opinions.

United States v. Pitts [2]

The first ruling came in United States v. Pitts. The government alleged that Lee Andrew Pitts “entered a branch of Chase Bank ... and handed [the manager at a teller window] a withdrawal slip that had written on it: “‘HAND OVER ALL 100, 50, 20 I HAVE A GUN I WILL SHOOT.’” After the manager repeatedly said that she had no money, the would-be robber fled on foot ... leaving behind the withdrawal slip” with latent fingerprints. A trawl of a fingerprint database — the court does not say which one or how it was conducted — led New York police to arrest Pitts two weeks later.

Facing trial on charges of entering the bank with the intent to rob it, Pitts moved “to preclude the government from introducing expert opinion testimony as to latent fingerprint and handwriting analysis.” The opinion does not specify the exact nature of the expert's fingerprint testimony. Presumably, it would have been an opinion that Pitts is the source of the print on the withdrawal slip. The court merely noted that the government "claims that its fingerprint experts do not intend to testify that fingerprint analysis has a zero or near zero error rate."

Judge Irizarry made short work of Pitts’s contention that such testimony would contravene Federal Rule of Evidence 702 and Daubert v. Merrell Dow Pharmaceuticals, Inc., 509 U.S. 579 (1993). Pitts relied “chiefly on the findings of the PCAST Report, the [2009] NAS Report [3], and several out-of-circuit court decisions that question the reliability of latent fingerprint analysis.” The judge was “not persuaded.” She acknowledged that “[t]he PCAST and NAS Reports [indicate that] error rates are much higher than jurors anticipate” and that “the NAS Report [stated] that “[w]e have reviewed available scientific evidence of the validity of the ACE-V method and found none.” But she was “dismayed that Defendant’s opening brief failed to address an addendum to the PCAST Report.” According to the court,
[The 2017 Addendum] applaud[ed] the work of the friction-ridge discipline” for steps it had taken to confirm the validity and reliability of its methods. ... The PCAST Addendum further concluded that “there was clear empirical evidence” that “latent fingerprint analysis [...] method[ology] met the threshold requirements of ‘scientific validity’ and ‘reliability’ under the Federal Rules of Evidence.”
Actually, the Addendum [4] adds little to the 2016 report. It responds to criticisms from the forensic-science establishment. The assessment of the scientific showing for the admissibility of latent fingerprint identification under Rule 702 is unchanged. The original report stated that “latent fingerprint analysis is a foundationally valid subjective methodology—albeit with a false positive rate that is substantial and is likely to be higher than expected by many jurors based on longstanding claims about the infallibility of fingerprint analysis.” It added that “[i]n reporting results of latent-fingerprint examination, it is important to state the false-positive rates based on properly designed validation studies.” The Addendum does not retreat from or modify these conclusions in any way.

Both the Report and the Addendum reinforce the conclusion that, despite the lack of detailed, objective standards for evaluating the degree of similarity between pairs of prints, experiments have shown that analysts can reach the conclusion that two prints have a common source with good accuracy. But the Report also lists five more conditions that bear on whether a particular analyst has reached the correct conclusion in a given case. It creates the neoteric phrase “validity as applied” for the showing that a procedure has been properly applied in a the case at bar:
Scientific validity as applied, then, requires that an expert: (1) has undergone relevant proficiency testing to test his or her accuracy and reports the results of the proficiency testing; (2) discloses whether he or she documented the features in the latent print in writing before comparing it to the known print; (3) provides a written analysis explaining the selection and comparison of the features; (4) discloses whether, when performing the examination, he or she was aware of any other facts of the case that might influence the conclusion; and (5) verifies that the latent print in the case at hand is similar in quality to the range of latent prints considered in the foundational studies.
The opinion does not discuss whether the court accepts or rejects this five-part test for admitting the proposed testimony. It jumps to the unedifying conclusion that defendant’s “critiques [do not] go to the admissibility of fingerprint analysis, rather than its weight.”

United States v. Lundi [5]

Chief Judge Irizarry returned to the question of admissibility of source attributions form latent prints in United States v. Lundi.  In the middle of thre afternoon of February 20, 2017, three men entered a check cashing and hair salon on Flatbush Avenue in Brooklyn. They forced an employee in a locked glass booth to let them in by pointing a gun at the head of a customer. They made off with approximately $13,000, but one of them had put his hands on top of the glass booth. Police ran an image of the latent prints from the booth through a New York City automated fingerprint identification system (AFIS) database. They decided that those prints came from Steve Lundi. Federal charges followed.

In advance of trial, Lundi moved to exclude the identification. He avoided the Pitts pitfall of arguing that there was no adequate scientific basis for expert latent print source attributions (although the more recent report of an American Association for the Advancement of Science (AAAS) working group would have lent some credence to such a claim [6]). Instead, Lundi “challeng[ed] the application of that [validated] science to the specific examinations conducted in the instant case.” It is impossible to tell from the opinion whether the court was made aware of the PCAST five-part test admissibility under Rule 702(d). Again, citing the unpublished opinion of a federal court in Illinois, Judge Irizarri apparently leapt over this part of the Report to the conclusion that
This Court is not persuaded that Defendant’s challenges go to the admissibility of the government’s fingerprint evidence, rather than to the weight accorded to it. Moreover, as this Court noted in Pitts, fingerprint analysis has long been admitted at trial without a Daubert hearing. ... The Court sees no reason to preclude such evidence here. Accordingly, Defendant’s motion to preclude fingerprint evidence is denied.
Again, it is impossible to tell from the court's cursory and conclusory analysis whether the theory is that an uncontroverted assurance that an expert undertook an “analysis,” a “comparison,” and an “evaluation” and that another expert did a “verification” ipso facto satisfies Rule 702(d).  The judge noted that “the government points to concrete indicators of how the ACE-V method actually was followed by Detective Skelly,” but it would be hard to find a modern fingerprint identification in which there were no indications that the examiner (1) analyzed the latent print (decided that it was of adequate quality to continue), (2) picked out features to compare and compared them, and then (3) evaluated what was seen. If this is all it takes to satisfy the Rule 702(d) requirement that “the expert has reliably applied the principles and methods to the facts of the case,” then the normal burden on the advocate of expert evidence to show that it meets all the rule’s requirements has evaporated into thin air.

Yet, this could be all that the court required. It suggested that all expert evidence is admissible as long as it is reliable in some general sense, writing that “our adversary system provides the necessary tools for challenging reliable, albeit debatable, expert testimony” and “[v]igorous cross-examination, presentation of contrary evidence, and careful instruction on the burden of proof are the traditional and appropriate means of attacking shaky but admissible evidence” (citing Daubert, 509 U.S. at 596).

The suggestion assumes what is to be proved—that the evidence—“shaky” or unshakeable—is “admissible.” The PCAST Report tried to give meaning to the case-specific reliability prong of Rule 702 (which simply codifies post-Daubert jurisprudence) by spelling out, for highly subjective procedures like ACE-V, what is necessary to demonstrate the legally reliable application in a specific case. Perhaps the “concrete indicators” showed that PCAST’s conditions were satisfied. Perhaps they did not go that far. Perhaps the PCAST conditions are too demanding. Perhaps they are too flaccid. Judge Irizarri does not tell us what she thinks.

After Lundi and Pitts, courts should strive to fill the gap in the analysis of the application of a highly subjective procedure. They should reveal what they think of PCAST’s effort to clarify (or, more candidly, to prescribe) what is required for long-standing methods in forensic science to be admissible under Rule 702(d).

REFERENCES
  1. Executive Office of the President, President’s Council of Advisors on Science and Technology, Report to the President: Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods, Sept. 2016.
  2. United States v. Pitts, 16-CR-550 (DLI), 2018 WL 1116550 (E.D.N.Y. Feb. 26, 2018).
  3. Comm. on Identifying the Needs of the Forensic Sci. Cmty., Nat'l Research Council, Strengthening Forensic Science in the United States: A Path Forward (2009).
  4. PCAST, An Addendum to the PCAST Report on Forensic Science in Criminal Courts, Jan. 6, 2017.
  5. United States v. Lundi, 17-CR-388 (DLI), 2018 WL 3369665 (E.D.N.Y. July 10, 2018).
  6. William Thompson, John Black, Anil Jain & Joseph Kadane, Forensic Science Assessments: A Quality and Gap Analysis, Latent Fingerprint Examination (2017).

Saturday, May 26, 2018

Against Method: ACE-V, Reproducibility, and Now Preproducibility

Forensic-science practitioners like to describe their activities as scientific. Indeed, if the work they did were not scientific, how could one say that they were practicing forensic science?

Thus, one finds textbooks with impressive titles like "Forensic Comparative Science" devoted to “[t]he comparative science disciplines of finger prints, firearm/tool marks, shoe prints/tire prints, documents and handwriting,” and more. 1/ The practitioners of these "science disciplines" describe their work as "analogous to scientific method of critically observing details in images, determining similarities or differences in the data, performing comparative measurements to experiment whether the details in the images actually agree or disagree," and so on. 2/ They insist that they follow a multistep process that "is a scientific methodology" for "hypothesis testing"3/ — even if the process lacks any defined threshold for deciding when perceived (or even objectively measured) features are sufficiently similar or different to reach a conclusion.

An entire article in the Journal of Forensic Identification — “a scientific journal that provides more than 100 pages of articles related to forensics ... written by forensic authorities from around the world who are practitioners or academics in forensic science fields” 4/ — is devoted to demonstrating that “[a]nalysis, comparison, evaluation, and verification (ACE-V) is a scientific methodology that is part of the scientific method.” 5/ The abstract observes that
Several publications have attempted to explain ACE-V as a scientific method or its role within the scientific method, but these attempts are either not comprehensive or not explicit. This article ... outlines the scientific method as a seven-step process. The scientific method is discussed using the premises of uniqueness, persistence, and classifiability. Each step of the scientific method is addressed specifically as it applies to friction ridge impression examination in casework. It is important for examiners to understand and apply the scientific method, including ACE-V, and be able to articulate this method. 6/
The Scientific Working Group on Friction Ridge Analysis, Study, and Technology (SWGFAST) agreed, urging examiners to write in their reports that ACE-V is nothing less than “[t]he acronym for a scientific method: Analysis, Comparison, Evaluation, and Verification.” 7/

It is revealing to contrast such assertions with comments on the meaning of reproducibility in science that appear in an essay published this week in Nature. There, Philip Stark, the Associate Dean of the Division of Mathematical and Physical Sciences and Professor of Statistics at the University of California (Berkeley), noted that reproducibility means different things in different fields, but pointed to “preproducibility” as a  prerequisite to reproducibility:
An experiment or analysis is preproducible if it has been described in adequate detail for others to undertake it. ... The distinction between a preproducible scientific report and current common practice is like the difference between a partial list of ingredients and a recipe. To bake a good loaf of bread, it isn’t enough to know that it contains flour. It isn’t even enough to know that it contains flour, water, salt and yeast. The brand of flour might be omitted from the recipe with advantage, as might the day of the week on which the loaf was baked. But the ratio of ingredients, the operations, their timing and the temperature of the oven cannot.

Given preproducibility — a ‘scientific recipe’ — we can attempt to make a similar loaf of scientific bread. If we follow the recipe but do not get the same result, either the result is sensitive to small details that cannot be controlled, the result is incorrect or the recipe was not precise enough ... . 8/
Descriptions of procedures for subjective pattern-matching in traditional forensic science are much like the list of ingredients. There are no quantitative instructions for how many and which of the potentially distinguishing features to use and how long to process or cook these ingredients at each step of the process. Indeed, it is even worse than that. Even though trained examiners know what ingredients to choose from, they can pick any subset of them that they think could be effective for the case at hand. Thus, although ACE-V can be described as a series of steps within "a broadly stated framework," 9/ that does not make it a “scientific recipe.” Imprecision at every step deprives it of  "preproducibility." It might be called a "process" rather than a "method," 10/ but in the end, “ACE-V is an acronym, not a methodology." 11/

It does not follow, however, that the comparisons and conclusions are of no epistemic value or that they cannot be studied scientifically. Quite the contrary. Unlike the natural sciences, the absence of preproducibility does not preclude reproducibilty. Another laboratory expert can start with the same materials to be compared, and we can see if the outcome is the same. Moreover, we can even conduct blind tests of examiner performance to determine how accurately criminalists are able to classify traces originating from the same source and traces coming from different sources.

Such validation studies show that latent print examiners, for example, have real expertise, but these findings do not mean that expert examiners are following a particularly “scientific method” in making their judgments. Psychologists report that some individuals are phenomenally accurate in recognizing faces, 12/ but that does not mean that the “super recognizers” are using a well-defined, or indeed, any kind of scientific procedure to accomplish these feats. The training and experience that criminalists receive may include instruction in facts and principles of biology and physics, and their performance may be generally accurate and reliable, but that does not mean that they are applying a scientific method. A flow chart is not a scientific test.

Until criminalists can articulate and follow a preproducible procedure, they should not present their work as deeply scientific. In court, they can explain that scientists have studied the nature of the patterns they analyze. They can refer to any well-designed studies proving that subjective pattern-matching by trained analysts can be valid and reliable. They can vouch for the fact that criminalists have been making side-by-side comparisons for a long time. If courts are persuaded that the resulting individual opinions are helpful, then skilled witnesses can give those opinions. But such opinions should not be gussied up as a scientific method of hypothesis testing. 13/ As one critic of such rhetoric explained, "forensic science could show that it does have validation, certification, accreditation, oversight, and  basic  research  without  showing  that  it  uses  the  'scientific method.'" 14/

NOTES
  1. John R. Vanderkolk, Forensic Comparative Science xii (2009).
  2. Id. at 90.
  3. M. Reznicek, R.M. Ruth & D.M. Schilens, ACE-V and the Scientific Method, 60 J. Forensic Identification 87, 87 (2010). See also Michele Triplett & Lauren Cooney, The Etiology of ACE-V and its Proper Use: An Exploration of the Relationship Between ACE-V and the Scientific Method of Hypothesis Testing, 56 J. Forensic Identification 345, 353 (2006), http://www.fprints.nwlean.net/JFI.pdf (“ACE-V is synonymous with hypothesis testing. A more in-depth understanding of scientific methodology can be found by reading the works of well-known scientists and philosophers such as Aristotle, Isaac Newton, Francis Bacon, Galileo Galilei, and Karl Popper, to name just a few.”).
  4. Abstract of Journal of Forensic Identification (JFI), https://www.theiai.org/publications/jfi.php, accessed May 25, 2018.
  5. Reznicek et al., supra note 3, at 87.
  6. Id. The Department of Justice has retreated from this phrasing, preferring to call “an examiner’s belief” an “inductive inference . . . made in a logical and scientifically defensible manner.” Department of Justice, Approved Uniform Language for Testimony and Reports for the Forensic Latent Print Discipline, Feb. 22, 2018, at 2 & 2 n.2, https://www.justice.gov/file/1037171/download.
  7. Scientific Working Group on Friction Ridge Analysis, Study, and Technology, Standard for Reporting Friction Ridge Examinations (Latent/Tenprint), Appendix, at 4 n.2, 2012, https://www.nist.gov/sites/default/files/documents/2016/10/26/swgfast_standard_reporting_2.0_121124.pdf (emphasis added).
  8. Philip B. Stark, Before Reproducibility must Come Preproducibility, 557 Nature 613 (2018), doi: 10.1038/d41586-018-05256-0, https://www.nature.com/articles/d41586-018-05256-0.
  9. Comm. on Identifying the Needs of the Forensic Sci. Cmty., Nat'l Research Council, Strengthening Forensic Science in the United States: A Path Forward 142 (2009).
  10. Michele Triplett, Is ACE-V a Process or a Method?, IDentification News, June/July 2012, at 5–6, http://www.fprints.nwlean.net/ProcessOrMethod.pdf.
  11. Sandy L. Zabell, Fingerprint Evidence, 13 J. L. & Pol'y 143, 178 (2005).
  12. Richard Russell, Brad Duchaine, and Ken Nakayama, Super-recognizers: People with Extraordinary Face Recognition Ability, 16 Psychonomic Bull. and Rev. 252 (2009), doi: 10.3758/PBR.16.2.252.
  13. David H. Kaye, How Daubert and Its Progeny Have Failed Criminalistics Evidence, and a Few Things the Judiciary Could Do About It, 86 Fordham L. Rev. 1639 (2018), https://ssrn.com/abstract=3084881.
  14. Simon A. Cole, Acculturating Forensic Science: What Is ‘Scientific Culture’, and How Can Forensic Science Adopt It?, 38 Fordham Urb. L.J. 436, 451 (2010).