ISO Standards on Forensic Science: Pay to Play?

"ISO, the International Organization for Standardization, brings global experts together to agree on the best way of doing things – for anything from making a product to managing a process." ^1/ For the last few years, it has been devising the following overarching set of standards for all of forensic science:

• Forensic sciences (TC 272) ISO/DIS 21043-1, Forensic sciences - Part 1: Terms and definitions - 5/27/2024, $58.00
• ISO/DIS 21043-3, Forensic Sciences - Part 3: Analysis - 5/26/2024, $62.00
• ISO/DIS 21043-4, Forensic Sciences - Part 4: Interpretation - 5/26/2024, $67.00
• ISO/DIS 21043-5, Forensic Sciences - Part 5: Reporting - 5/26/2024, $53.00 ^2/

These are

part of a series which, when completed, will include the different components of the forensic process from scene to courtroom ... . The series describes primarily “what” is standardized, not the “how” or “who”. Best practice manuals and standard operating procedures should describe “how” the requirements of this document would be met. ^3/

It sounds like the standards in progress will not specify "the best way of doing things." Will they merely list the things that are in need of "standardization"? Will they be too open-ended to constitute what the U.S. Supreme Court refers to as "standards controlling the technique's operation"^4/?

I cannot answer these questions because I have not seen the drafts that were open for public comment. Members of the public cannot read the drafts without paying IS0 the $240 listed above. If anyone who has paid to play has thoughts on these documents that they would like to share beyond the TC (Technical Committee) that drafted the standards, I'll post them--at no charge.

Notes

1. Int'l Org. for Standardization, About ISO.
2. ANSI Standards Action, Mar. 15, 2024, at 48.
3. ISO 21043-1:2018(en) Forensic sciences — Part 1: Terms and definitions.
4. Daubert v. Merrell Dow Pharm., 509 U.S. 579, 594 (1993).

SWGDE's Best Practices for Remote Collection of Digital Evidence from a Networked Computing Environment

SWGDE 22-F-003-1.0, Best Practices for Remote Collection of Digital Evidence from a Networked Computing Environment, is a forensic-science standard proposed for inclusion on the Organization of Scientific Area Committees for Forensic Science (OSAC) Registry—"a repository of selected published and proposed standards … to promote valid, reliable, and reproducible forensic results.”

The best practices “may not be applicable in all circumstances.” In fact, “[w]hen warranted, an examiner may deviate from these best practices and still obtain reliable, defensible results.” I guess that is why they are called best practices rather than required practices. But what circumstances would justify using anything but the best practices? On this question, the standard is silent. It merely says that “[i]f examiners encounter situations warranting deviation from best practices, they should thoroughly document the specifics of the situation and actions taken.” 

Likewise, the best practices for “preparation” seem rather rudimentary. “Examiners should ascertain the appropriate means of acquiring data from identified networked sources.” No doubt, but how could they ever prepare to collect digital information without ascertaining how to acquire data? What makes a means “appropriate”? All that a digital evidence expert can glean from this document is that he or she “should be aware of the limitations of each acquisition method and consider actions to mitigate these limitations if appropriate” and should consider “methods and limitation variables as they relate to various operating systems.” How does such advice regularize or improve anything?

Same thing with a recommendation that “[p]rior to the acquisition process, examiners should prepare their destination media”? What steps for preparing the destination media are best? Well, [s]terilization of destination media [whatever the process of “sterilization” is in this context] is not generally required.” But it is required “when needed to satisfy administrative or organizational requirements or when a specific analysis process makes it a prudent practice.” When would sterilization be prudent? The drafters do not seem to be very sure. “[E]xaminers may need to sanitize destination media provided to an external recipient to ensure extraneous data is not disclosed.” Or maybe they don’t? “Examiners may also be required to destroy copies of existing data to comply with legal or regulatory requirements.” Few people would dispute that the best practice is to follow the law, but examiners hardly need best practices documents from standards developing organizations to know that.

The standard is indeterminate when it comes to what it calls “triage”—“preview[ing] the contents of potential data sources prior to acquisition.” We learn that “[e]xaminers may need to preview the contents of potential data sources prior to acquisition” to “reduce the amount of data acquired, avoid acquiring irrelevant information, or comply with restrictions on search authority.” What amount of data makes "triage" a best practice? How does the examiner know that irrelevant information may be present? Why can "triage" sometimes be skipped? When it is desirable and how should it be done? The standard merely observes that “[t]here may be multiple iterations of triage … .” When are multiple iterations advisable? Well, it “depend[s] on the complexity of the investigation.” Equally vague is the truism that “[e]xaminers should use forensically sound processes to conduct triage to the extent possible.” 

Finally, designating steps like “perform acquisition” and “validate collected data” as “best practices” does little to inform examiners of how to collect digital evidence from a network. To be fair, a few parts of the standard are more concrete, and, possibly, other SWGDE standards fill in the blanks. But, on its face, much of this remote acquisition standard simply gestures toward possible best practices. It does not expound them. In this respect, it resembles other forensic-science standards that emerge from forensic-science standards developing organizations only to be criticized as vague at critical points.

Bones of Contention: A Standard for Analyzing Skeletal Trauma in Forensic Anthropology

The Academy Standards Board (ASB) of the American Association of Forensic Sciences (AAFS) posted the second proposed draft of a "Standard for Analyzing Skeletal Trauma in Forensic Anthropology" for public comment. The standard does not go far toward standardizing procedures or showing that the procedures to which it applies have been scientifically tested. Of course, it could well be that ample, well designed studies have demonstrated that forensic anthropologists can consistently and accurately classify skeletal defects in human remains according to the categories the standard mentions. But the standard contains no bibliography and no citations to show that this is the case. 

It contains some negative injunctions and a few positive suggestions about reporting -- for example:

• Forensic anthropologists shall not determine cause or manner of death.
• Practitioners shall not estimate the temperature or duration of heat exposure based on thermal defects to bone.
• Practitioners may report the minimum number of traumatic events (e.g., blunt impacts, projectile entry defects, or sharp defects) observed skeletally, but shall not report a definitive maximum number of impacts, as skeletal trauma evidence may not reflect all impacts to the body.
• When a suspect tool is submitted for analysis, similarities between the tool and defect may be reported; conclusions shall be reported in terms of an exclusion or failure to exclude.

As such, ASB 147-21 is not without any redeeming legal value. Nevertheless, it does not articulate any analytical process by which the classifications it calls for should be made (cf. "vacuous standards"); it requires no reporting of the uncertainty in this process; it does not contemplate the possibility of evidence-based rather than conclusion-based statements of the implications of the data; and it refers to an all-inclusive list of methods as "acceptable." If I may elaborate:

Is "Interpretation" Limited to an Opinion on the Inference (Conclusion) from the Data?

The revision defines "trauma interpretation" as "Opinion regarding the mechanism of, timing, direction of impact(s) or minimum number of impacts associated with skeletal defect(s) based on quantitative and/or qualitative observations." The phrase "based on ... observations" indicates that the opinion expresses a belief in the truth, falsity, or probability of an inference being drawn from the data. Interpretation should include the possibility of describing the strength of the evidence in favor of the inference rather than opining on the truth, falsity, or probability of the conclusion itself. In addition, if the opinion-statement is an assertion that the hypothesis about what happened is true or false (either categorically or to some probability), it is not just based on the data but on a prior probability for the hypothesis as well.

Despite these definitions, the standard sanctions "interpretation" in the form of rudimentary statements about the extent to which the data prove the hypothesis in question. Section 6 notes that "Trauma interpretation shall be clearly identified in the report using terms such as ‘indicative of’ and ‘consistent with’ or by using a subheading titled ‘Interpretation.’"These phrases have their problems, but they are one manner of referring to the probability of the evidence given the truth of certain probabilities rather than vice versa.

Is Interpretation Based on Non-scientific Evidence and Inference?

The revision introduces the following (non)criteria for deciding that blasts or explosions caused skeletal trauma: "Blasts/explosive events often cause blunt (including concussive) and projectile trauma to the body. When the trauma pattern and circumstantial information support a blast event, the trauma mechanism should be classified as 'blast trauma'”. The undefined notion of "support" is too vague to give any guidance. Is "consistent with" considered "support"? Let's hope not -- patterns can be "consistent with" one hypotheses (it could occur when the hypothesis is true) but much more probable under the opposite hypothesis.

And then there is the green light this recommendation gives to presenting a conclusion based on nonscientific "circumstantial evidence" as if it were based on expertise involving the skeletal evidence. Knowing that a blast occurred can drive the conclusion that the damage to the skeleton is "blast trauma." Should there also be a report on the skeletal evidence from an analyst blinded to the other information uncovered in the investigation?

Is Everything Acceptable?

ASB 147-21 states that "Skeletal trauma shall be examined. Acceptable methods to examine trauma include gross, microscopic, radiographic, and other analytical methods." This formulation deems every conceivable analytical method as "acceptable" no matter how poorly conceived it may be. Labeling everything as "acceptable" is troublesome in a standard that does not include criteria and procedures for performing the analysis and that does not lead the reader to any evidence of the reliability and validity of the undefined "analytical procedures."

Of course, forensic anthropologists know that some procedures do not work well, and only an outlier would use them. The drafters of ASB 147-21 undoubtedly appreciate the need for suitable methods (and hence prohibit certain conclusions that cannot be drawn with any existing method). Well motivated and informed forensic anthropologists will not be led astray if they consult the standard. But outliers do appear in court. Remember Louise Robbins. Unless the dubious method yields one of the explicitly prohibited statements in this standard, the outlier witnesses could maintain that they have proceeded exactly as the standard requires. Standards with this potential for abuse should be reformed.They should strive to standardize the methods they govern, and they should state what is known about the accuracy and reliability of these methods.

"The Judgment of an Experienced Examiner"

The quotation from a textbook on forensic science in the left-hand panel invites the question of what the author was thinking. That an examiner's judgment is more important in comparisons of "class" features than "individual" ones? That the features that are present are less important than the examiner's judgment of them? Neither interpretation makes much sense. The examiner's judgment does not dictate anything about the features. It is the other way around. In applying a valid method, a proficient examiner generally will make correct judgments of significance as dictated by the features that are present.

As with most class evidence, the significance of a fiber comparison is dictated by the circumstances of the case, by the location, number, and nature of the fibers examined, and, most important, by the judgment of an experienced examiner.

---

Richard Saferstein, Criminalistics: An Introduction to Forensic Science 272 (12rth ed. 2018, Pearson Education Inc.) (emphasis added)

Over the years, scientific and legal scholars have called for the implementation of algorithms (e.g., statistical methods) in forensic science to provide an empirical foundation to experts’ subjective conclusions. ... Reactions have ranged from passive skepticism to outright opposition, often in favor of traditional experience and expertise as a sufficient basis for conclusions. In this paper, we explore why practitioners are generally in opposition to algorithmic interventions and how their concerns might be overcome. We accomplish this by considering issues concerning human-algorithm interactions in both real world domains and laboratory studies as well as issues concerning the litigation of algorithms in the American legal system. [W]e propose a strategy for approaching the implementation of algorithms ... .

---

Henry Swofford & Christophe Champod, Implementation of Algorithms in Pattern & Impression Evidence: A Responsible and Practical Roadmap, 3 Forensic Sci. Int'l: Synergy 100142 (2021) (abstract)

Another example of sloppy phrasing about expert judgment is the boilerplate disclaimers or admonitions in ASTM standards for forensic-science methods. For example, the 2019 Standard Guide for Forensic Analysis of Fibers by Infrared Spectroscopy (E2224−19) insists (in italics no less) that

This standard cannot replace knowledge, skills, or abilities acquired through education, training, and experience and is to be used in conjunction with professional judgment by individuals with such discipline-specific knowledge, skills, and abilities.

Does the first independent clause mean that fiber analysts are free to depart from the standard on the basis of their general "knowledge, skills, or abilities acquired through education, training, and experience"? Ever since Congress funded the Organization of Scientific Area Committees for Forensic Science (OSAC) to write new and better standards, lawyers in the organization have objected to the ASTM wording (without doubting that expert methods should be applied by responsible experts).

Now rumor has it that ASTM will be changing its stock sentence to the less ambiguous observation that

This standard is intended for use by competent forensic science practitioners with the requisite formal education, discipline-specific training (see Practice E2917), and demonstrated proficiency to perform forensic casework.

That's innocuous. Indeed, you might think it goes without saying.

Distorting Daubert and Parting Ways with PCAST in Romero-Lobato

United States v. Romero-Lobato 1/ is another opinion applying the criteria for admissibility of scientific evidence articulated in Daubert v. Merrell Dow Pharmaceuticals 2/ to uphold the admissibility of a firearms examiner's conclusion that the microscopic marks on recovered bullets prove that they came from a particular gun. To do so, the U.S. District Court for the District of Nevada rejects the conclusions of the President's Council of Advisors on Science and Technology (PCAST) on validating a scientific procedure.

This is not to say that the result in the case is wrong. There is a principled argument for admitting suitably confined testimony about matching bullet or ammunition marks. But the opinion from U.S. District Court Judge Larry R. Hicks does not contain such an argument. The court does not reach the difficult question of how far a toolmark expert may go in forging a link between ammunition and a particular gun. It did not have to. In what seems to be a poorly developed challenge to firearms-toolmark expertise, the defense sought to exclude all testimony about such an association.

This posting describes the facts of the case, the court's description of the law on the admissibility of source attributions by firearms-toolmark examiners, and its review of the practice under the criteria for admitting scientific evidence set forth by the Supreme Court in Daubert.

I. THE FIREARMS EVIDENCE IN THE CASE

A grand jury indicted Eric Romero-Lobato for seven felonies. On March 4, 2018, he allegedly tried to rob the Aguitas Bar and Grill and discharged a firearm (a Taurus PT111 G2) into the ceiling. On May 14, he allegedly stole a woman's car at gunpoint while she was cleaning it at a carwash. Later that night, he crashed the car in a high-speed chase On the front passenger's seat, was a Taurus PT111 G2 handgun.

Steven Johnson, a supervising criminalist in the Forensic Science Division of the Washoe County Sheriff's Office," 3/ was prepared to testify that the handgun had fired a round into the ceiling of the bar. Romero-Lobato moved "to preclude the testimony." The district court held a pretrial hearing at which Johnson testified to his background, training, and experience. He explained that he matched the bullet to the gun using the "AFTE method" advocated by the Association of Firearm and Tool Mark Examiners.

Defendant's challenge rested "on the critical NAS and PCAST Reports as evidence that 'firearms analysis' is not scientifically valid and fails to meet the requisite threshold for admission under Daubert and Federal Rule of Evidence 702." Apparently, the only expert at the hearing was the Sheriff Department's criminalist. Judge Hicks denied the motion to exclude Johnson's expert opinion testimony and issued a relatively detailed opinion.

II. SETTING THE STAGE FOR A DAUBERT ANALYSIS

Skipping over the early judicial resistance to "this is the gun" testimony, 4/ the court noted that despite misgivings about such testimony on the part of several federal district courts, only one reported case has barred all source opinion testimony 5/ and the trend among the more critical courts is to search for ways to admit the conclusion with qualifications on its certainty.

Judge Hicks did not pursue the possibility of admitting but constraining the testimony, apparently because the defendant did ask for that. Instead, the court reasoned that to overcome the inertia of the current caselaw, a defendant must have extraordinarily strong evidence (although it also recognized that the burden is on the government to prove scientific validity under Daubert) .The judge wrote:

[T]he defense has not cited to a single case where a federal court has completely prohibited firearms identification testimony on the basis that it fails the Daubert reliability analysis. The lack of such authority indicates to the Court that defendant's request to exclude Johnson's testimony wholesale is unprecedented, and when such a request is made, a defendant must make a remarkable argument supported by remarkable evidence. Defendant has not done so here.

Defendant's less-than-remarkable evidence was primarily two consensus reports of scientific and other experts who reviewed the literature on firearms-mark comparisons. 6/  Both are remarkable. The first document was the highly publicized National Academy of Sciences committee report on improving forensic science. The committee expressed concerns about the largely subjective comparison process and the absence of studies to adequately measure the uncertainty in the evaluations. The court deemed these to be satisfied by a single research report submitted to Department of Justice, which funded the study:

The NAS Report, released in 2009, concluded that “[s]ufficient studies have not been done to understand the reliability and repeatability” of firearm and toolmark examination methods. ... The Report's main issue with the AFTE method was that it did not provide a specific protocol for determining a match between a shell casing or bullet and a specific firearm. ... Instead, examiners were to rely on their training and experience to determine if there was a “sufficient agreement” (i.e. match) between the mark patterns on the casing or bullet and the firearm's barrel. ... During the Daubert hearing, Johnson testified about his field's response to the NAS Report, pointing to a 2013 study from Miami-Dade County (“Miami-Dade Study”). The Miami-Dade Study was conducted in direct response to the NAS Report and was designed as a blind study to test the potential error rate for matching fired bullets to specific guns. It examined ten consecutively manufactured barrels from the same manufacturer (Glock) and bullets fired from them to determine if firearm examiners (165 in total) could accurately match the bullets to the barrel. 150 blind test examination kits were sent to forensics laboratories across the United States. The Miami-Dade Study found a potential error rate of less than 1.2% and an error rate by the participants of approximately 0.007%. The Study concluded that “a trained firearm and tool mark examiner with two years of training, regardless of experience, will correctly identify same gun evidence.”

A more complete (and accurate) reading of the Miami Dade Police Department's study, shows that it was not designed to measure error rates as they are defined in the NAS report and that the "error rate" was much closer to 1%. That's still small, and, with truly independent verification of an examiners' conclusions, the error rate should be smaller than that for examiners whose findings are not duplicated. Nonetheless, as an earlier posting shows, the data are not as easily interpreted and applied to case work as the report from the crime laboratory suggests.The research study, which has yet to appear in any scientific journal. has severe limitations.

The second report, released late in 2016 by the President's Council of Advisors on Science and Technology (PCAST) flatly maintained that microscopic firearms-marks comparisons had not been scientifically validated. Essentially dismissing the Miami Dade Police and earlier research as not properly designed to measure the ability of examiners to infer whether the same gun fired test bullets and ones recovered from a crime scene, PCAST reasoned that (1) AFTE-type identification had yet to be shown to be "reliable" within the meaning of Rule 702 (as PCAST interpreted the rule); (2) if courts disagreed with PCAST's legal analysis of the rule's requirements, they should at least require examiners associating ammunition with a particular firearm to give an upper bound, as ascertained from controlled experiments, on false-positive associations. (These matters are discussed in previous postings.)

The court did not address the second conclusion and gave little or no weight to the first one. It wrote that the 2016 report

concluded that there was only one study done that “was appropriately designed to test foundational validity and estimate reliability,” the Ames Laboratory Study (“Ames Study”). The Ames Study ... reported a false-positive rate of 1.52%. ... The PCAST Report did not reach a conclusion as to whether the AFTE method was reliable or not because there was only one study available that met its criteria.

All true. PCAST certainly did not write that there is a large body of high quality research that proves toolmark examiners cannot associate expended ammunition with specific guns. PCAST's position is that a single study is not a body of evidence that establishes a scientific theory--replication is crucial. If the court believed that there is such a body of literature, it should have explained the basis for its disagreement with the Council's assessment of the literature. If it agreed with PCAST that the research base is thin, then it should have explained why forensic scientists should be able to testify--as scientists--that they know which gun fired which bullet. This opinion does neither. (I'll come to court's discussion of Daubert below.)

Instead, the court repeats the old news that

the PCAST Report was criticized by a number of entities, including the DOJ, FBI, ATF, and AFTE. Some of their issues with the Report were its lack of transparency and consistency in determining which studies met its strict criteria and which did not and its failure to consult with any experts in the firearm and tool mark examination field.

Again, all true. And all so superficial. That prosecutors and criminal investigators did not like the presidential science advisors' criticism of their evidence is no surprise. But exactly what was unclear about PCAST's criteria for replicated, controlled, experimental proof? In fact, the DOJ later criticized PCAST for being too clear--for having a "nine-part" "litmus test" rather than more obscure "trade-offs" with which to judge what research is acceptable. 7/

And what was the inconsistency in PCAST's assessment of firearms-marks comparisons? Judge Hicks maintained that

The PCAST Report refused to consider any study that did not meet its strict criteria; to be considered, a study must be a “black box” study, meaning that it must be completely blind for the participants. The committee behind the report rejected studies that it did not consider to be blind, such as where the examiners knew that a bullet or spent casing matched one of the barrels included with the test kit. This is in contrast to studies where it is not possible for an examiner to correctly match a bullet to a barrel through process of elimination.

This explanation enucleates no inconsistency. The complaint seems to be that PCAST's criteria for a validating a predominantly subjective feature-comparison procedure are too demanding or restrictive, not that these criteria were applied inconsistently. Indeed, no inconsistency in applying the "litmus test" for an acceptable research design to firearms-mark examinations is apparent.

Moreover, the court's definition of "a 'black box' study" is wrong. All that PCAST meant by "black box" is that the researchers are not trying to unpack the process that examiners use and inspect its components. Instead, they say to the examiner, "Go ahead, do your thing. Just tell us your answer, and we'll see if you are right." The term is used by software engineers who test complex programs to verify that the outputs are what they should be for the inputs. The Turing test for the proposition that "machines can think" is a kind of black box test.

Nonetheless, this correction is academic. The court is right about the fact that PCAST gave no credence to "closed tests" like those in which an examiner sorts bullets into pairs knowing in advance that every bullet has a mate. Such black-box experiments are not worthless. They show a nonzero level of skill, but they are easier than "open tests" in which an examiner is presented with a single pair of bullets to decide whether they have a common source, then another pair, and another, and so on. In Romero-Lobato, the examiner had one bullet from the ceiling to compare to a test bullet he fired from one suspect gun. There is no "trade-off" that would make the closed-test design appropriate for establishing the examiner's skill at the task he performed.

All that remains of the court's initial efforts to avoid the PCAST report is the tired complaint about a "failure to consult with any experts in the firearm and tool mark examination field." But what consultation does the judge think was missing? The scientists and technologists who constitute the Council asked the forensic science community for statements and literature to support their practices. It shared a draft of its report with the Department of Justice before finalizing it. After releasing the report, it asked for more responses and issued an addendum. Forensic-services providers may complain that the Council did not use the correct criteria, that its members were closed-minded or biased, or that the repeated opportunities to affect the outcome were insufficient or even a sham. But a court needs more than a throw-away sentence about "failure to consult" to justify treating the PCAST report as suspect.

 III. DISTORTING DAUBERT

Having cited a single, partly probative police laboratory study as if it were a satisfactory response to the National Academy's concerns and having colored the President's Council report as controversial without addressing the limited substance of the prosecutors' and investigators' complaints, the court offered a "Daubert analysis." It marched through the five indicia that the Supreme Court enumerated as factors that courts might consider in assessing scientific validity and reliability.

A. It Has Been Tested

The Romero-Lobato opinion made much of the fact that "[t]he AFTE methodology has been repeatedly tested" 8/ through "numerous journals [sic] articles and studies exploring the AFTE method" 9/ and via Johnson's perfect record on proficiency tests as proved by his (hearsay and character evidence) testimony. Einstein once expressed impatience "with scientists who take a board of wood, look for its thinnest part and drill a great number of holes where drilling is easy." 10/ Going through the drill of proficiency testing does not prove much if the tests are simple and unrealistic. A score of trivial or poorly designed experiments should not engender great confidence. The relevant question under Daubert is not simply "how many tests so far?" It is how many challenging tests have been passed. The opinion makes no effort to answer that question. It evinces no awareness of the "10 percent error rate in ballistic evidence" noted in the NAS Report, that prompted corrective action in the Detroit Police crime laboratory.

Instead of responding to PCAST's criticisms of the design of the AFTE Journal studies, the court wrote that "[a]lthough both the NAS and PCAST Reports were critical of the AFTE method because of its inherent subjectivity, their criticisms do not affect whether the technique they criticize has been repeatedly tested. The fact that numerous studies have been conducted testing the validity and accuracy of the AFTE method weighs in favor of admitting Johnson's testimony."

But surely the question under Daubert is not whether there have been "numerous studies." It is what these studies have shown about the accuracy of trained examiners to match a single unknown bullet with control bullets from a single gun. The court may have been correct in concluding that the testing prong of Daubert favors admissibility here, but its opinion fails to demonstrate that "[t]here is little doubt that the AFTE method of identifying firearms satisfies this Daubert element."

B. Publication and Peer Review

Daubert recognizes that, to facilitate the dissemination, criticism, and modification of theories, modern science relies on publication in refereed journals that members of the scientific community read. Romero-Lobato deems this factor to favor admission for two reasons. First, the AFTE Journal in which virtually all the studies dismissed by PCAST appear, uses referees. That it is not generally regarded as a significant scientific journal -- it is not available through most academic libraries, for example -- went unnoticed.

Second, the court contended that "of course, the NAS and PCAST Reports themselves constitute peer review despite the unfavorable view the two reports have of the AFTE method. The peer review and publication factor therefore weighs in favor of admissibility." The idea that the rejection in consensus reports of a series of studies as truly validating a theory "weighs in favor of admissibility" is difficult to fathom. Some readers might find it preposterous.

C. Error Rates

Just as the court was content to rely on the absolute number of studies as establishing that the AFTE method has been adequately tested, it takes the error rates reported in the questioned studies at face value. Finding the numbers to be "very low," and implying (without explanation) that PCAST's criteria are too "strict," it concludes that Daubert's "error rate" factor too "weighs in favor of admissibility."

A more plausible conclusion is that a large body of studies that fail to measure the error rates (false positive and negative associations) appropriately but do not indicate very high error rates is no more than weakly favorable to admission. (For further discussion, see the previous postings on the court's discussion of the Miami Dade and Ames Laboratory technical reports.)

D. Controlling Standards

The court cited no controlling standards for the judgment of  "'sufficient agreement' between the 'unique surface contours' of two toolmarks." After reciting the AFTE's definition of "sufficient agreement," Judge Hicks decided that "matching two tool marks essentially comes down to the examiner's subjective judgment based on his training, experience, and knowledge of firearms. This factor weighs against admissibility."

However, the opinion adds that "the consecutive matching striae ('CMS') method," which Johnson used after finding "sufficient agreement," is "an objective standard under Daubert." It is "objective" because an examiner cannot conclude that there is a match unless he "observes two or more sets of three or more consecutive matching markings on a bullet or shell casing." The opinion did not consider the possibility that this numerical rule does little to confine discretion if no standard guides the decision of  whether a marking matches. Instead, the opinion debated whether the CMS method should be considered objective and confused that question with how widely the method is used.

The relevant inquiry is not whether a method is subjective or objective. For a predominantly subjective method, the question is whether standards for making subjective judgments will produce more accurate and more reliable (repeatable and reproducible) decisions and how much more accurate and reliable they will be.

E. General Acceptance

Finally, the court found "widespread acceptance in the scientific community." But the basis for this conclusion was flimsy. It consisted of statements from other courts like "the AFTE method ... is 'widely accepted among examiners as reliable'" and "[t]his Daubert factor is designed to prohibit techniques that have 'only minimal support' within the relevant community." Apparently, the court regarded the relevant community as confined to examiners. Judge Hicks wrote that

it is unclear if the PCAST Report would even constitute criticism from the “relevant community” because the committee behind the report did not include any members of the forensic ballistics community ... . The acceptance factor therefore weighs in favor of admitting Johnson's testimony.

If courts insulate forensic-science service providers from the critical scrutiny of outside scientists, how can they legitimately use the general-acceptance criterion to help ascertain whether examiners are presenting "scientific knowledge" à la Daubert?

NOTES

1. No. 3:18-cr-00049-LRH-CBC, 2019 WL 2150938 (D. Nev. May 16, 2019).
2. 509 U.S. 579 (1993).
3. For a discussion of a case involving inaccurate testimony from the same laboratory that caught the attention of the Supreme Court, see David H. Kaye, The Interpretation of DNA Evidence: A Case Study in Probabilities, National Academies of Science, Engineering and Medicine, Science Policy Decision-making Educational Modules, 2016, available at https://ssrn.com/abstract=2810744; McDaniel v. Brown: Prosecutorial and Expert Misstatements of Probabilities Do Not Justify Postconviction Relief — At Least Not Here and Not Now, Forensic Sci., Stat. & L., July 7, 2014, http://for-sci-law.blogspot.com/2014/07/mcdaniel-v-brown-prosecutorial-and.html.
4. See David H. Kaye, Firearm-Mark Evidence: Looking Back and Looking Ahead, 68 Case W. Res. L. Rev. 723, 724-25 (2018), available at https://papers.ssrn.com/abstract_id=3117674. The court relied on the article's explication of more modern case law.
5. The U.S. District Court for the District of Colorado  excluded toolmark conclusions in the prosecutions for the bombing of the federal office building in Oklahoma City. The toolmarks there came from a screwdriver. David H. Kaye et al., The New Wigmore, A Treatise on Evidence: Expert Evidence 686-87 (2d ed. 2011).
6. The court was aware of an earlier report from a third national panel of experts raising doubts about the AFTE method, but it did not cite or discuss that report's remarks. Although the 2008 National Academies report on the feasibility of establishing a ballistic imaging database only considered the forensic toolmark analysis of firearms in passing, it gave the practice no compliments. Kaye, supra note 2, a 729-32.
7. Ted Robert Hunt, Scientific Validity and Error Rates: A Short Response to the PCAST Report, 86 Fordham L. Rev. Online Art. 14 (2017), https://ir.lawnet.fordham.edu/cgi/viewcontent.cgi?article=1013&context=flro.
8. Quoting United States v. Ashburn, 88 F.Supp.3d 239, 245 (E.D.N.Y. 2015).
9. Citing United States v. Otero, 849 F.Supp.2d 425, 432–33 (D.N.J. 2012), for "numerous journals [sic] articles and studies exploring the AFTE method."
10. Philipp Frank, Einstein's Philosophy of Science, Reviews of Modern Physics (1949).

FURTHER READING

• David H. Kaye, Firearm-Mark Evidence: Looking Back and Looking Ahead, 68 Case W. Res. L. Rev. 723-45 (2018)
• David H. Kaye et al., The New Wigmore, A Treatise on Evidence: Expert Evidence  (2d ed. 2011, with annual updates)

MODIFIED: 7 July 2019 9:10 EST

Index to comments and cases on the PCAST report

Another US District Court Finds Firearms-mark Testimony Admissible in the Post-PCAST World

A new opinion on firearms-toolmark identification from the U.S. District Court for the Southern District of New York continues the clear trend of admitting such identifications notwithstanding the misgivings of three scientific study groups. The opinion of Paul G. Gardephe is more careful than most, but it has some loose ends.

In United States v. Johnson, (S5) 16 Cr. 281 (PGG), 2019 WL 1130258 (S.D.N.Y. Mar. 11, 2019), the Government brought firearms-related charges against Latique Johnson, the alleged leader of the New York and Pennsylvania street gang “Blood Hound Brims.” One indictment alleged that he shot at rival gang members in a restaurant in the Bronx. New York City Police Department Detective Jonathan Fox concluded that “toolmark identification” analysis showed that bullets collected from the restaurant were fired from an AK 47 semi-automatic assault rifle that an undercover officer purchased from a former gang member a year-and-a-half later (and that was rumored to be the one Johnson used in the restaurant).

Johnson moved before trial to prevent Detective Fox from testifying, or at least to bar an opinion that the ammunition was fired from the AK 47. During the trial, the court conducted a hearing. Detective Fox testified that the premise for “what we [microscopists] do” is “that the tools that are used to manufacture ... firearms leave marks on the inside of the firearms that are unique to that particular tool.”

The court did not question the premise that characteristics are “individual characteristics” that must be “unique” when viewed under a microscope. Why insist on empirical proof of that theoretical proposition? Surely, the marks made by guns on ammunition are unique at the molecular level. But that intuition proves too much. It means that every shell or cartridge case fired from the same gun will be unique on that scale.

So the claim of uniqueness has to pertain to features as they actually are measured or perceived, and that claim requires empirical verification. Nevertheless, the belief in universal uniqueness is not critical to the work of firearms examiners. They could be incredibly accurate in associating expended ammunition with specific guns even if the high level of similarity between ammunition components from test fires and components recovered from a crime scene is not quite unique with respect to every gun that has ever existed.

In Johnson, the court found that Detective Fox followed the Association of Firearm and Tool Mark Examiners’ “Theory of Identification as it Relates to Tool Marks.” This theory of how to interpret marks requires “sufficient agreement” for a positive association. “Sufficient agreement” comes from the examiner’s sense that the level of agreement is greater than that for different guns and is within the range for ammunition from the same gun. It “means that the agreement of individual characteristics is of a quantity and quality that the likelihood another tool could have made the mark is so remote as to be considered a practical impossibility.” 1/

In challenging the AFTE theory and its application, Johnson relied “primarily on the 2008 National Research Council Report, Ballistic Imaging ... ;  the 2009 National Research Council Report, Strengthening Forensic Science in the United States: A Path Forward ... ; and the 2016 report of the President’s Council of Advisors on Science and Technology, Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods (2016) ... .” The court recognized that these reports “challenge, to varying degrees, the scientific basis for toolmark identification analysis.”

After summarizing the statements in the three reports on the limited scientific knowledge of the accuracy of firearms-mark identification 2/ and reviewing the federal case law, 3/ the court “conclude[d] that toolmark identification analysis — at least as performed by Detective Fox — is sufficiently reliable to be presented to the jury.” In reaching this result, the court considered the canonical “Daubert factors.”

"Testability"

First, it found sufficient testing of the ability of examiners to reach the correct results in experiments and proficiency tests. According to the PCAST report, the published experiments were not properly designed and hence could not establish validity. As for proficiency tests, the court later recognized that they may not be representative of casework and they were not blind. So although the AFTE theory surely is testable, the extent to which it has been adequately tested remains open to debate despite the court's conclusion.

Oddly, the court also proposed that “photographic documentation and independent review” in case work tests the theory or practice. Taking pictures is useful, but it does not validate the process, and even blind verification only tests consistency between two examiners. Such reproducibility is not the same as accuracy, and validation requires comparing the examiners’ judgments to the true state of affairs — what forensic scientists sometimes call “ground truth.”

Peer Review and Publication

Second, the court determined that there was adequate “peer review and publication” — “mostly” in a journal of the Association of Firearm and Tool Mark Examiners. This part of the opinion is perfunctory. The court accepted at face value the assurance in a 2003 AFTE Journal article that the journal uses “experts within the scientific community” for “technical review.”

Academic and other critics have complained that the publication is basically a trade journal without peer review by qualified scientists. For example, one forensic scientist wrote that

The AFTE Journal ... is not freely available and requires written testimony [sic] from existing AFTE members. It has extremely limited dissemination beyond the members of AFTE — it is found only in 18 libraries of the 72,000 libraries listed in World CAT, the largest catalog of library materials — and completely lacks integration with any of the voluminous networks for the production and exchange of scientific research information. The journal engages in peer review that is neither blind nor draws on an extensive network of researchers. This is not an attitude in keeping with the openness that is a part of any true scientific research culture. 4/

That was 2014. Today, the journal is in only nine more libraries. Only sixteen university libraries make it available.

Courts evaluating the extent of publication and peer review may wish to consider the views of the National Commission on Forensic Science. This Commission, formed by the Department of Justice and the National Institute of Standards and Technology, issued guidance on what constitutes “scientific literature” for supporting forensic science and practice. The Commision’s criteria include publication “in a journal that utilizes rigorous peer review with independent external reviewers to validate the accuracy in its publications and their overall consistency with scientific norms of practice.”

AFTE emphasizes that it has long used a formal peer review system. A position paper published in its journal in 2015 commented as follows:

While it is unclear what the NCFS considers “rigorous” and “independent”, all submissions to the AFTE Journal undergo a thorough two-stage review process conducted by subject matter experts who are members of the AFTE Editorial Committee. Occasionally, a reviewer may need to enlist the knowledge of an expert outside of the field to complete a review; however, it would be extremely difficult to use only external reviewers because most qualified potential reviewers also tend to be AFTE members. Research manuscripts submitted to the AFTE Journal are independently reviewed by at least two reviewers from the Editorial Committee ... .

Controlling Standards

Third, the court expressed more hesitation in finding “controlling standards.” It wrote that

[B]oth courts and the scientific community have voiced serious concerns about the “sufficient agreement” standard, characterizing it as “tautological,” “wholly subjective,” “circular,” “leav[ing] much to be desired,” and “not scientific.” The Court shares some of these concerns.

Nonetheless, Judge Gardephe deemed “photographic documentation and verification requirements” and “extensive AFTE training and proficiency testing” to be controlling standards.

This logic is hard to follow. Record-keeping, verification, training, and proficiency testing all are critical components of a quality assurance system. But they are not standards that control how a subjective judgment is made. Detective Fox was clear about the degree of subjectivity. He "stated that ... he employs a holistic approach incorporating his 'training as a whole' and his experience 'based on all the cartridge casings and ballistics that [he] ha[s] identified and compared.'" Still, the court managed to extract "certain principles that ground his conclusions":

For example, the CMS standard — six consecutive matching striations or two groups of three matching striations — represents a “bottom standard” or a floor for declaring a match. Detective Fox will not declare that “sufficient agreement” exists unless microscopic examination reveals a toolmark impression with one area containing six consecutive matching individual characteristics, or two areas with three consecutive matching individual characteristics. ... Detective Fox’s analysis does not end at that point, however. Instead, Detective Fox goes on to examine every impression on the ballistics evidence. “All these lines should match,” as well, and if they do not, Detective Fox will not find “sufficient agreement.”

“These criteria.” the court concluded, “provide standards for Detective Fox’s findings as to “sufficient agreement.”

"Rate of Error"

Fourth, the court regarded proficiency test data as suggestive of a modest error rate and added that “even accepting the PCAST Report’s assertion that the error rate could be as high as 1 in 46, or close to 2.2%, such an error rate is not impermissibly high.” The court did not maintain that the one unpublished and unreplicated study of cartridge shell casings that PCAST used to generate this figure (for a Ruger pistol) provided a precise estimate of a false-positive error rate. Somewhat limply, it concluded “that the absence of a definite error rate for toolmark identification does not require that such evidence be precluded.” The judge did not discuss PCAST's recommendation that the upper-bound error rate be presented to the jury.

General Acceptance

Finally, the court spent little energy on the issue of general acceptance. Assuming that the relevant scientific community was limited to “forensic scientists, and firearms examiners in particular,” it perceived “no dispute.”

* * *

In denying the motion to exclude or limit the testimony, the court ironically saw the fact that the problems with firearms identifications are apparent as favoring — or at least not inhibiting — admission. It described “the weaknesses in the methodology of toolmark identification analysis” as “readily apparent,” “as discussed at length in the scientific literature,” and as “not particularly complicated or difficult to grasp.” Thus, they “are likely to be understood by jurors if addressed on cross-examination.”

At the same time, Judge Gardephe did not approve of all types of firearms-marks testimony. He recognized that overclaiming can occur; however, he saw no risk of it happening in this case:

Having heard Detective Fox’s testimony at the Daubert hearing, it is clear that he does not intend to assert — and the Government does not intend to elicit — any particular degree of certainty as to his opinions regarding the ballistics match. ... Indeed, Detective Fox’s repeated concession at the Daubert hearing that his conclusions are “based on [his] subjective opinion” stands in stark contrast to the “tendency of [other] ballistics experts ... to make assertions that their matches are certain beyond all doubt.” ... Detective Fox also testified that he “would never” state his conclusion that ballistics evidence matches to a particular firearm “to the exclusion of all other firearms ... in a court proceeding[,] ... because I haven’t looked at all other firearms.”

Although the last concession is refreshing, it creates a puzzle -- Are "individual characteristics" truly individual? If the “individual characteristics” obviously matched — as apparently they did 5/ — why shouldn't the examiner testify that they exclude every other gun? Isn’t it “practically impossible,” to use the AFTE phrase, that another AK 47 fired the bullets?

NOTES

1. The current version of the official theory is at https://afte.org/about-us/what-is-afte/afte-theory-of-identification.
2. The opinion also quoted the trivially true statements in the reports that in describing the scientific status of the pattern-identification methods, the groups were not themselves taking a stand on the legal question of their admissibility.
3. The district court wrote that

   In assessing reliability, ‘the district court must focus on the principles and methodology employed by the expert, without regard to the conclusions the expert has reached or the district court’s belief as to the correctness of those conclusions.’ Amorgianos v. Nat’l R.R. Passenger Corp., 303 F.3d 256, 266 (2d Cir. 2002).

   In General Electric Co. v. Joiner, 522 U.S. 136 (1997), however, the Supreme Court wrote that “conclusions and methodology are not entirely distinct from one another,” and Rule 702(d) specifies that “the expert has reliably applied the principles and methods to the facts of the case.”
4. David Klatzow, Justice Denied: Role of Forensic Science in the Miscarriage of Justice (2014).
5. The court was impressed that “[t]he ‘matching’ ... is stark, even to an untrained observer.”

FURTHER READING

• David H. Kaye, Firearm-Mark Evidence: Looking Back and Looking Ahead, 68 Case W. Res. L. Rev. 723-45 (2018)

Index to comments and cases on the PCAST report

Reflections on Glass Standards: Statistical Tests and Legal Hypotheses

Statistical Applicata (Italian Journal of Applied Statistics) recently published several issues (volume 27, nos. 2 & 3) devoted to statistics in forensic science and law. They include an invited article I prepared in 2016 on the statistical logic of declaring pieces of glass "indistinguishable" in their physical properties. 1/ The article contains some of the views expressed in postings on this blog (e.g., Broken Glass: What Do the Data Show?). However, the issue is much broader than glass evidence. The article notes the potential for confusion in reporting that any kind of trace-evidence samples match (or cannot be distinguished) without also describing data on the frequency of such matches in a relevant population. I am informed that NIST's Organization of Scientific Area Committees on Forensic Science (OSAC) is preparing guidelines or standards for explaining the probative value of results obtained from ASTM-approved test methods.

Abstract

---

The past 50 years have seen an abundance of statistical thinking on interpreting measurements of chemical and physical properties of glass fragments that might be associated with crime scenes. Yet, the most prominent standards for evaluating the degree of association between specimens of glass recovered from suspects and crime scenes have not benefitted from much of this work. Being confined to a binary match/no-match framework, they do not acknowledge the possibility of expressing the degree to which the data support competing hypotheses. And even within the limited match/no-match framework, they focus on the single step of deciding whether samples can be distinguished from one another and say little about the second stage of the matching paradigm–characterizing the probative value of a match. This article urges the extension of forensic-science standards to at least offer guidance for criminalists on the second stage of frequentist thinking. Toward that end, it clarifies some possible sources of confusion over statistical terminology such as “Type I” and “Type II” error in this area, and it argues that the legal requirement of proof beyond a reasonable doubt does not inform the significance level for tests of whether pairs of glass fragments have identical chemical or physical properties.

Note

1. The article is David H. Kaye, Reflections on Glass Standards: Statistical Tests and Legal Hypotheses, 27 Statistica Applicata -- Italian J. Applied Stat. 173 (2015). Despite the publication date assigned to the issue, the article, as stated above, was not written until 2016.

NCFS to Ask NIST to Institute an Additional "Technical Merit" Review Mechanism for Some Types of OSAC Forensic Science Standards?

A subcommittee of the National Commission on Forensic Science (NCFS) has approved a draft Recommendation to the Attorney General on Technical Merit Evaluation of Forensic Science Methods and Practices. It is available for public comment until August 15, 2016.

It asks the Attorney General (to whom NCFS reports) to ask the National Institute of Standards and Technology (NIST) to establish a committee of "measurement scientists and statisticians" who would review the scientific literature and do whatever else is needed to establish the technical merit of test methods and practices used in forensic science disciplines. The document also asks OSAC (the Organization of Scientific Area Committees for Forensic Science that NIST established a few years ago) not to place standards for forensic science test methods and practices on its Registry of Approved Standards before their technical merit has been reviewed by such an "independent scientific body."

In full, the draft recommendations are as follows:

1. NIST should establish an in-house entity with the capacity to conduct independent scientific evaluations of the technical merit of test methods and practices used in forensic science disciplines.

2. The results of the evaluations will be issued by NIST as publicly available resource documents. NIST’s evaluation may include but is not limited to: a) research performed by other agencies and laboratories, b) its own intramural research program, or c) research studies documented in already published scientific literature. NIST should initially begin its work by piloting three resource documents to establish their design and requirements. The release of these documents should be broadly disseminated in the scientific and criminal justice communities and accompanied by judicial trainings.

3. The Organization of Scientific Area Committees for Forensic Science (OSAC) leadership, the Forensic Science Standards Board (FSSB), should commit to placing consensus documentary standards on the OSAC Registry of Approved Standards for only those forensic science test methods and practices where technical merit has been established by NIST, or in the interim, established by an independent scientific body. An example of an interim independent scientific body could be an OSAC created Technical Merit Resource Committee composed of measurement scientists and statisticians appointed by NIST and tasked with the evaluation of technical merit.

The surrounding explanation can be found at https://www.justice.gov/ncfs/work-products . Presumably, the Commission will discuss the recommendation at its September 12-13 meeting at, appropriately enough, NIST's campus in Gaithersburg, Maryland.

If the Glass Fits, Declare It: The Justice Department's ULTR for Glass that "Physically Fits Together"

The Department of Justice’s “Proposed Uniform Language for Testimony and Reports for the Forensic Glass Discipline” allows a broad range of possible conclusions and statements of probability. The strongest conclusion of association would be that “the glass fragments were once part of the same broken object.” The draft ULTR permits a criminalist to “state or imply” this source conclusion—but only “when two or more pieces of broken glass physically fit together.”

The notion that fragments that fit together must have come from the same larger piece of  glass sounds like common sense. Almost all of us have broken glass or ceramic materials at one time or another and seen that the fragments (at least the major ones) can be reassembled. Apparently, criminalists have had the same experience. The supporting documentation for the ULTR notes that “[i]t has long been reported by forensic glass examiners that two glass objects that physically fit together were once part of the same broken object.” 1/

But what type of scientific inquiry would establish the truth of the common sense idea? Let’s do a series of thought experiments. I have ten drinking glasses in my cupboard. (Actually, six are in the dishwasher, but I’ll pretend all are clean and back in the cupboard.) They sure look alike to me, being the same size, shape, and color.

• Experiment 1. I take one glass out and strike it with a hammer. Then I pick up the pieces and discover that all pairs of fragments that were adjacent in the unbroken glass fit together.


• Experiment 2. I take out a second glass and strike it with a hammer. Then I put all the pieces in a bag with those from the first glass and mix them up. Will I find any pairs of fragments that seem to fit together that are Glass1-Glass2 (heteroginous, if I can coin a word that means “of different origin”), or will all the pairs that fit together be Glass1-Glass1 or Glass2-Glass2 (homoginous, a neologism for "of the same origin")? Let’s suppose that the subset of all the possible pairs that I find to match are indeed homoginous.


• Experiment 3. You guessed it. I take a third glass, strike it with the hammer, and examine all possible pairs from all the fragments from the three glasses. This is getting very time-consuming, for the number of pairs to consider is growing exponentially, but I am persistent. Can we conclude, on the basis of common sense, that each and every physically matching pair will be homoginous?

You can see where I am going with this. Common sense plus some cogitation suggests that homoginality — a given pair of matching fragments coming from the same original piece of glass — depends on how many fragments are in the relevant population and the complexity of the match (that is, the extent of the irregularities in the edges of the fragments being fitted together). It is not so obvious that every physically matching pairs of glass fragments that can possibly exist are homoginous. That glass examiners have said that all matching pairs must be homoginous does not make it so. Neither does labeling mechanical fit as an individual as opposed to a class characteristic (to use the popular terminology in forensic science). The labeling exercise begs the question.

Nonetheless, it is quite reasonable to conjecture that only a tiny fraction (which could include 0) of apparently matching fragments that have complex edge patterns are heteroginous, and thus that finding a physical fit is powerful evidence of homoginality. 2/ Of course, the reasonableness of the conjecture depends on the criteria for a fit and the method for declaring one—neither of which are mentioned in the DOJ materials—but let’s assume the existence and maintenance of rigorous criteria and a reliable method. Can criminalists currently estimate how often the conclusion of homoginality is correct when the fragments (unbeknownst to the analyst) actually come from different pieces of glass?

Even if the answer to this question is in the negative, there are ways to present the physical match without overclaiming. Why must examiners “state or imply that the glass fragments were once part of the same broken object”? Examiners who believe that there is a physical fit can contribute a great deal simply by documenting and exhibiting the fit itself. If, additionally, they have some special expertise in moving from the fit to the conclusion of homoginality, then doing also could assist the fact finder. But if their inference is simply that of common sense, the expert aspect of the probative value of that part of the testimony is questionable. In these circumstances, the criminalist might be better advised to show how well the fragments fit together and leave it to the judge or jury to assess the weight of this demonstrable fact.

Another alternative to testimony “to the exclusion of all other sources” (see note 1) is a statement that the observed match is more probable when fragments really come from the same broken object than when they come from different broken objects. I do not know whether specific studies have been performed to justify this likelihood-ratio-type statement, but it should not be outside the capacity of forensic science to generate data that would support this rather modest statement. Indeed, the claim probably is too weak. My intuition is that the likelihood ratio is much greater than 1—it is a heck of a lot more probable to generate a matching pair of fragments by breaking the same object than by breaking different ones. A lawyer can appeal to give the evidence substantial weight on the basis of similar beliefs. If criminalists have experimental or observational studies to quantify how much more probable the physical match is for homoginous fragments, then they can provide that likelihood ratio.

Based on the material the DoJ has supplied, however, the more ambitious criminalists who want to “render an opinion of positive identity” and asservate “to the exclusion of all other sources” (see note 1), as the proposed ULTR apparently allows, should be required to express the limitation that no matter how positive they are in their opinion, the core of that opinion is a common sense judgment of what it means for pieces of glass to fit together rather than the product of elaborate scientific experimentation.

Note

1. An accompanying footnote cites four sources:
   • Kirk, P., Density and Refractive Index: Their Application in Criminal Identification, Charles C. Thomas, Springfield, IL, 1951, p. 4-5.
   • “Don’t Overlook Evidentiary Value of Glass Fragments”, Law Enforcement Bulletin, Vol. 33, No. 10, October 1964, p. 19.
   • Bottrell, M.C., “Forensic Glass Comparison: Background Information Used in Data Interpretation”, Forensic Science Communications [Online], (April, 2009). Available: http://www.fbi.gov/about-us/lab/forensic-science-communications/fsc/april2009/review/2009_04_review01.htm. accessed on August 15, 2013.
   • Koons, R. D., Buscaglia, J., Bottrell, M., and Miller, E. T. Forensic glass comparisons. In: Forensic Science Handbook. vol. I, 2nd ed. Richard Saferstein, Ed., Prentice Hall, Upper Saddle River, New Jersey, 2002, p. 161–213.
   
   I do not have Paul Kirk’s 1951 book at hand. The Law Enforcement Bulletin is not generally regarded as a substantial scientific publication. The 2009 publication is a generally informative and comprehensive review article from the FBI. However, it contains no reports of examiners' experiences or experiments with physically fitting fragments of glass back together. In fact, only two sentences even refer to physically fitting fragments—and neither of them explains the basis for the conclusion that the ULTR endorses. (The first sentence is this: “Only physically matching two or more broken glass fragments allows for their association with each other to the exclusion of all other sources (Scientific Working Group for Materials Analysis [SWGMAT] 2005c).” The second is equally devoid of information on the foundation for the conclusion. It reads: “The ... Trace Evidence Unit Quality Assurance Manual (2006) states [that a] glass association is defined as two or more glass samples that can be fracture fitted together, or that exhibit indistinguishable observable properties and/or range overlap in all measured properties.”)
   
   Finally, the FBI chapter in Richard Saferstein’s handbook discusses “types of fractures” (pp. 173-77) and later, “the mechanical fit.” The authors write that

   The mechanical fit is one of the most desirable of forensic glass examinations because the examiner can render an opinion of positive identity—that two or more pieces of glass were once a portion of, and were broken from, the same pane or object. Glass is particularly suitable for this type of examination. It is amorphous and brittle, is neither stretched nor distorted by breakage, and can be reassembled to its original configuration. Because it is amorphous, no two glass objects will break in exactly the same way. (P. 179).

   As has been discussed extensively for other types of matching evidence, the claim of uniqueness at some possibly unattainable level of precision of measurement does not imply the truth of an unqualifiedly positive opinion of identity.


2. See generally David H. Kaye, Beyond Uniqueness: The Birthday Paradox, Source Attribution, and Individualization in Forensic Science Testimony, 12 Law, Probability & Risk 3 (2013); David H. Kaye, Probability, Individualization, and Uniqueness in Forensic Science Evidence: Listening to the Academies, 75 Brooklyn L. Rev. 1163 (2010); David H. Kaye, Identification, Individuality, and Uniqueness: What's the Difference?, 8 Law, Probability & Risk 85 (2009).

Proposed Uniform Language for Forensic Serology

The second document listed in the announcement of "Uniform Language" (UL) documents for forensic science is the “Proposed Uniform Language for Testimony and Reports For the Forensic Examination of Serology.” It pertains to the detection of sera, not the examination of serology, but you get the idea. The task is one of classification: Is this a blood stain? Is it a semen stain?

The approved statements (with emphasis added) are

• blood or semen may be present on an item of evidence [using "the appropriate presumptive testing procedure(s)"]
• blood or semen was identified on an item of evidence [using "the appropriate confirmatory procedure"].

In terms of accuracy, the difference between "presumptive" and "confirmatory" is that presumptive tests are less specific — more substances will be test positive with a presumptive test. A supporting document explains that confirmatory testing can include either a highly specific test — or just a second, presumptive test, if based on “different chemical principles” and “not subject to the same limitations.” (Or maybe a second and a third screening test are required; it is hard to tell from the document.)

Although the examiner is not allowed to say that a positive result is "infallible," that is only because an examiner might goof in following a protocol that is perfectly discriminating. The confirmatory procedure “identif[ies] blood or semen to the exclusion of all other substances.” (P. 5.)

The documents display little sympathy for probabilistic thinking. Apparently, examiners may not give an error probability for an "identification," for "the analytical processes and procedures ... do not have a calculable error rate due to the unpredictability of human error." This observation motivates the proscription of statements of a "zero error rate," but by this logic, the probability of commercial pilot error, of a mishap in a nuclear power plant, or of medication errors in ICUs could not be calculated.

Aside from this allegedly unquantifiable risk of strictly "human error," identifications are subject to no uncertainty:

• when crystals are formed under the proper testing conditions, there is no doubt of the nature of the stain tested ... no false positives.
• [if] the Takayama test ... is positive, blood is conclusively present.

Despite this certitude in the supporting document, the UL document itself cautions that

An examiner may not state or imply that a level of numerical certainty is calculated to support the identification of blood or semen.

The claims of "no doubt" and "conclusively" positive results can be reconciled with the prohibition on what can be said or reported; after all, that the number corresponding to certainty is not "calculated" is literally true. But if the examiner can report that (1) the testing conditions were proper, (2) a positive result means that "blood is conclusively present," then why cannot the examiner state that (3) "my tests demonstrate that the stain surely contains blood — the probability is 1 — unless I have done something wrong"? An argument might be that expressing conditional certainty as the number 1 is not necessary and might unduly impress readers of reports or legal factfinders.

Closely related postings

• Proposed Uniform Language for Forensic Toxicology, June 6, 2016, http://for-sci-law.blogspot.com/2016/06/proposed-uniform-language-for-forensic.html
• The Department of Justice's "Proposed Uniform Language for Testimony and Reports," June 4, 2016, http://for-sci-law.blogspot.com/2016/06/the-department-of-justices-proposed.html

Proposed Uniform Language for Forensic Toxicology

The first document listed in the announcement of "Uniform Language" (UL) documents for forensic science is the Department of Justice Proposed Uniform Language for Testimony and Reports for the Forensic Toxicology Discipline.  At first glance, it seems rather bland, but if the document truly demarcates the range of the "acceptable range of opinions" that DoJ toxicologists may provide, it is impoverished.

For example, the Supporting Documentation observes that "[a]n examiner may be asked if certain symptoms are consistent with those caused by a particular drug," but for decades, forensic scientists, statisticians, lawyers, and psychologists have complained that "consistent with" testimony is vacuous. Evidence usually is consistent with more than one hypothesis. Drowsiness is consistent with ingestion of barbituates, but it also is consistent with lack of sleep. For the symptoms to be probative of some hypothesis, alternative hypotheses must be considered. Diagnosticity depends on the probability of the symptoms under all the hypotheses.

The proposed uniform language obscures this fundamental concept. For example, the UL document provides that an "examiner may ... state that a reported blood concentration is within the therapeutic range, toxic range, or consistent with reported fatal concentrations" and "that a drug or poison found in a hair specimen is consistent with exposure (either ingestion or environmental) to the drug or poison." If a measured concentration falls into the range within which a poison has killed someone in the past, the "blood concentration" (more precisely, the concentration of the poison) would be "consistent with reported fatal concentrations." But it may be that this "fatal concentration" is rarely fatal and is "consistent with" an alternative cause of death. Surely, a toxicologist should be permitted to say as much "based on data published in peer reviewed literature or other authoritative sources."

At the other pole, suppose the concentration is even higher than any fatal dose ever reported. The toxicologist should not be limited to the uniform language. In this situation (and in some less extreme ones), an expert can give a more powerful, useful, and scientifically accurate opinion than "consistent with."

The UL also (inelegantly) cautions that "[a]n examiner may not report or state an opinion that suggests his/her interpretation of the effects of a drug or poison can be specified to the individual whose sample was tested." Does this mean that the toxicologist can say "the defendant had a blood alcohol level [BAC] that indicates intoxication" but cannot say that "the defendant had a blood alcohol level that indicates she was intoxicated"? Both opinions express (or should express) the same idea — that the measured BAC is more probable when the defendant is intoxicated than when the defendant is not intoxicated. (If this statement about probabilities does not hold, then the BAC does not support the hypothesis of intoxication and hence does not indicate intoxication.)

The UL document first suggests that the toxicologist can only speak of "the average human." Then it redefines the average human to be someone exactly like the defendant in our example:

An examiner may report and/or state his/her opinion as to the effects of drugs or poisons on the average human. This opinion should be based on the facts of the case, medical information about the individual that the specimens were collected from (e.g., weight, height, disease state, age), current published studies, and/or the examiner’s training in the fields of pharmacology, physiology, pathology, clinical chemistry, and/or toxicology.

So the examiner must use all "facts of the case" and all the anatomical and physiological variables specific to the defendant but must not say that the opinion pertains to the defendant? Even if the toxicologist hews to this line, what happens when defense counsel asks the expert whether focusing on the defendant by using all the information about the case exceeds the admonition to confine the testimony and reporting to "the average human"? The phrasing in this paragraph needs refinement if not rethinking.

Closely related postings

• The Department of Justice's "Proposed Uniform Language for Testimony and Reports," June 4, 2016, http://for-sci-law.blogspot.com/2016/06/the-department-of-justices-proposed.html

The Department of Justice's "Proposed Uniform Language for Testimony and Reports"

The US Department of Justice (DoJ) has "Proposed Language Regarding Expert Testimony and Lab Reports in Forensic Science." The public comment period on www.regulations.gov closes July 8. Because words are the subject of the "Uniform Language" documents, as they are styled, it may be appropriate (if somewhat tedious) to jot down a few notes in the manner of English composition teachers. Readers who want remarks on the substantive merits of the documents will have to wait. 1/ What follows is merely about clarity of expression and ugliness (words like and "his/her" and "and/or").

To begin with, an initial disclaimer, boxed and bolded in the original, reads as follows:

This document provides examples of the scientifically-supported conclusions and opinions that may be contained in Department of Justice reports and testimony. These examples are not intended to be all inclusive and may be dependent upon the precedent set by the judge or locality in which a testimony is provided. Further, these examples are not intended to serve as precedent for other forensic laboratories and do not imply that statements by other forensic laboratories are incorrect, indefensible, or erroneous. This document is not intended to, does not, and may not be relied upon to create any rights, substantive or procedural, enforceable by law by any party in any matter, civil or criminal, nor does it place any limitation on otherwise lawful investigative and litigative prerogatives of the Department.

What would a careful copy editor have to say about this?

• First sentence. The documents do not give examples of scientifically-supported statements. They give (1) examples of conclusions and (2) sometimes specify the kind of scientific support that would be needed for some of them. The toxicology "Uniform Language" document, for example, refers to "analytical findings" without regard to the scientific basis of the analyses. Surely not every conceivable analytical method is scientifically adequate. (As I said at the outset, this is just a problem with the wording of the introductory disclaimer. I am not suggesting that the analytical methods currently employed by DoJ toxicologists are invalid. In fact, separate "Supporting Documentation" asserts that every one has been validated. However, the introductory statement refers to "this document," and the Uniform Language document does not constrain the range of conclusions about "analytical findings" to scientifically supported ones. In comparison, it states that "[t]he examiner may report and/or state the pharmacokinetic and pharmacodynamic effects of drugs and poisons based on data published in peer reviewed literature or other authoritative sources.")
• Second sentence. "The examples are not intended to be all inclusive" of what? Of "the scientifically-supported conclusions and opinions that may be contained in Department of Justice reports and testimony"? Presumably, this clause is meant to say that "The list of permissible statements is not exhaustive. Other statements might be acceptable."
• Second sentence (con't). "These examples ... may be dependent upon the precedent set by the judge or locality in which a testimony is provided." Stilted language ("may be dependent on") aside, the examples are examples — regardless of local law. The scientifically supportable ones in "this document" are examples of scientifically acceptable statements for the field without regard to their admissibility. The unacceptable ones should be scientifically unacceptable regardless of what a trial judge (whose rulings are not technically "precedent" for any other case) or other judges, lawyers, or legislatures do.
• Third sentence. "[T]hese examples are not intended to serve as precedent for other forensic laboratories and do not imply that statements by other forensic laboratories are incorrect, indefensible, or erroneous." DoJ has no power to make legal precedent. That it considers specific statements unfit for use by its laboratories surely has implications. Presumably, the sentence is meant to state the obvious — that DoJ's adoption for its laboratories of this list of acceptable and unacceptable examples is not intended to require other laboratories to do anything.
• Final sentence. "This document is not intended to, does not, and may not be relied upon to create any rights, substantive or procedural, enforceable by law by any party in any matter, civil or criminal ... ." But any judge is free to rely on this document to create an enforceable right. Presumably, the intended meaning is that the document, in itself, does not create such a right.
• Final sentence (cont'd). "nor does it place any limitation on otherwise lawful investigative and litigative prerogatives of the Department." What is the function of "otherwise lawful"? Naturally, the draft document places no limitation on anything DoJ wants to do. Does the full sentence mean that after adopting these guidelines, DoJ lawyers and laboratory staff will be free to depart from them? I think it means that the document is for internal guidance only and is not intended to be enforceable by any litigant.

Of course, the box of disclaimers at the beginning of each documents is just boilerplate. But skimming a couple of the Uniform Language documents also shows room for improvements in the mechanics of the writing. Here are some examples:

• Extensive use of the barbarism "and/or" 2/ and an occasional "his/her." 
• The phrase "[a]n examiner may state or imply" is common. Inasmuch as any examiner who legitimately may state that a proposition is true also may imply that it is true, "may state" the phrase "or imply" is surplusage. 3/
• Another common locution is "This document provides the acceptable range of opinions expressed in both laboratory reports and during expert witness testimony ... ." I think this is intended to state that "This document specifies the range of opinions that examiners may provide in laboratory reports or expert testimony." (It is not the "range" that is acceptable — it is the opinions.)
• The last statement is always qualified by the clause "while acknowledging that this document cannot address every variable in every examination." The qualification seems pointless or obscure. What does "address" mean here? Who would have imagined that the Uniform Language documents are supposed to address "every variable" in every forensic test, method, or analysis? If attention should have been given to an important "variable" but was not, how can we know that the range of opinions allowed is appropriate?

NOTES

1. Update: Remarks on a few of the discipline-specific documents are at

• Proposed Uniform Language for Forensic Toxicology, June 6, 2016, http://for-sci-law.blogspot.com/2016/06/proposed-uniform-language-for-forensic.html;

2. Bryan A. Garner, A Dictionary of Modern Legal Usage 56 (2d ed. 1995) (observing that "and/or has been vilified for most of its life—and rightly so. ... 'the only safe rule to follow is not to use the expression in any legal writing, document or proceeding, under any circumstances.'") (quoting Dwight G. McCarty, That Hybrid "and/or," 39 Mich. State B.J. 9, 17 (1960)); Leslie O’Flahavan, Is Using "and/or" in a Sentence Just Plain Dumb?, Jan. 25, 2010, http://ewriteonline.com/writing-matters/2010/01/using-andor-in-a-sentence-is-just-plain-dumb/ (surveying advice from journal editors and others and concluding that "the verdict is in: and/or is JPD (just plain dumb)").

3. The situation with "may not state or imply" is different.

SWGFAST on "a decision to reach a conclusion" of individualization for latent fingerprints

Before SWGFAST (the Scientific Working Group on Friction Ridge Analysis, Study, and Technology) dissolved to make way for OSAC (the Organization of Scientific Area Committees in Forensic Science), it produced a working draft of a Guideline for the Articulation of the Decision-Making Process for the Individualization in Friction Ridge Examination (Latent/Tenprint) (dated 3/13/13). The Guideline tells examiners how to explain the thinking that leads to what SWGFAST called “a decision to reach a conclusion of individualization” (§ 10.1) and how to report this decision (§ 11). Some interesting parts follow:

11.2.2. The SWGFAST Individualization/Identification Position Statement (Document #103) states that: “The ability of a latent print examiner to individualize a single latent impression, with the implication that they have definitely excluded all other humans in the world, is not supported by research and was removed from SWGFAST’s definition of individualization.”

This is a major and salutary departure from historical practice, but what remains of the definition? SWGFAST’s last definition of “individualization” was “[t]he determination by an examiner that there is sufficient quality and quantity of detail in agreement to conclude that two friction ridge impressions originated from the same source.” SWGFAST, Standard Terminology of Friction Ridge Examination (Latent/Tenprint), Mar. 13, 2013, at 5. “If two friction ridge impressions originated from the same source,” however, they did not originate from any other source, that is, from any “other human[] in the world.”

The only interpretation of the Guideline that seems to make sense is that an “individualization” is a tentative rather than a “definitive” conclusion that all other humans in the world have been excluded. Indeed, the Guideline continues:

11.2.3. Specific words and phrases conveying absolute certainty are inappropriate or misleading and should not be used to express conclusions. Specific problematic phrases include: [e]xclusion of all others, 100% certainty (as an absolute fact), [and z]ero error rate / infallible method.

But it is easier to say what should not be said than to prescribe what should be stated. The Guideline offers the following positive-plus-negative advice:

11.2.4. These concepts should rather be expressed as the conclusion of the examiner, based upon data observed and interpreted through the examiner’s training and experience. The examiner has a level of personal confidence associated with the accuracy and reliability of this conclusion; however, the accuracy and precision of this personal level of confidence cannot currently be measured and reported. For this reason, certainty should not be reported numerically or in absolute terms.

Evidently, the examiner should say something like "My decision is that X is the source of the latent print. I base this decision on my training and experience. I have [enormous? high? medium? low?] confidence in my personal decision." Of course, if a categorical conclusion is given, what the judge or jury would like to hear is the probability that this conclusion is correct, not that an examiner has some personal belief. But how a latent print examiner could provide such a posterior probability without considering a case-specific prior probability is an unsolved problem. A more measurable quantity is a conditional error probability—here, the probability that an examiner would identify X when X is not the source. Although some experiments have examined the rate of false positive (and negative) errors by latent print examiners, the Guideline does not propose using these figures.

Instead, it implies in the future it will be possible "to measure a personal level of confidence." I suppose an examiner could be forced to pick the probability at which he or she is indifferent to a gamble in a game of chance that has a well-defined probability for winning and losing. Would that make the personal confidence measurable? If so, why can't it be done currently?

In any event, the lack of a workable mechanism to measure personal confidence is not why claims of absolute certainty should be avoided. The reason is that nothing empirical is absolutely certain. Tomorrow, the laws of physics could change and heavy objects could float upward.

Neither does the difficulty of devising a suitable psychometric instrument for personal certainty explain why an examiner cannot report any numbers. People quantify personal judgments that cannot be measured objectively when their doctors ask them to report the severity of pain on a scale of 1 to ten. Why cannot an examiner report a defensible level of confidence numerically if the subjective precision is provided? E.g., “I think X is the source, and the strength of my belief is 9 on a scale of 1-10, but obviously this is not an exact number. I could have said 8, or even 7 as well, but I would not want to go below that.”

I am not arguing that examiners should grade their personal belief in a source attribution on a scale of 1-10 or express it as a numerical probability. My point merely is that SWGFAST's argument about reporting a number for relative certainty is superficial and underdeveloped.