Is OSAC Painting Itself Out of the Picture? Time to Comment on ASTM E1640-14

The Organization of Scientific Area Committees (OSAC) started on the wrong foot with the first standard it chose to place on its blue-ribbon Registry of Approved Standards. Two more standards from the Chemistry and Instrumental Analysis Committee are currently up for public comment. Here, I discuss the first of the two, known in the field as ASTM E1610-14.

ASTM E1610-14 is a "Standard Guide for Forensic Paint Analysis and Comparison." It seeks "to assist individuals who conduct forensic paint analyses in their evaluation, selection, and application of tests that may be of value to their investigations." To a large extent, it accomplishes this goal.

At the same time, however, this Standard Guide fails to provide much useful guidance on a matter of critical concern to the legal system — reporting results in a way that fairly conveys their probative value and the inevitable uncertainty of any scientifically validated test. In fact, the Guide fails to reflect or acknowledge modern thinking about the interpretation of forensic-science test results.

The premise of this document seems to be that the main task of the criminalist is to make "physical matches between known and questioned samples" (sec. 7.1) based on "significance assessments" (sec. 8.3) in which a "significant difference" is "a difference between two samples that indicates that the two samples do not have a common origin." (Sec. 3.2.10). Although this definition of “significant” offers little or no guidance, these matches are expected to be "conclusive." (Sec. 8.6.1). This categorical approach does not represent the modern view of the evaluative statements that criminalists should make. Most literature on forensic inference now maintains that analysts should present statements about the weight of the evidence rather than categorical conclusions. 1/

If analysts studying traces of paint are to make "conclusive" statements, however, and if the NIST-supported OSAC organization is to follow through on the National Academy of Sciences' recommendation to incorporate estimates of uncertainty into forensic science, the conditional error probabilities for these conclusions must be provided.

The concept of validity in measurement is closely connected to estimating uncertainty. Section 1.2 announces that “[t]he need for validated methods and quality assurance guidelines is also addressed,” but this sentence is the only place in the document where the words "validity," "validated," or "validation" appear, and classifications based on a process with unknown sensitivity and specificity cannot be considered validated. Indeed, this ASTM Standard Guideline indirectly endorses a minority notion (from the legal perspective) on what it takes to establish validity. It approvingly refers (in sec. 4.1) to quality assurance guidelines from a 1999 SWGMAT document (actually published in 2000). These guidelines include the observation that "[t]echniques and procedures ... currently accepted by the scientific community should be considered valid." But it is widely appreciated that the general-acceptance criterion for scientific validity is not necessarily sufficient under Federal Rule of Evidence 702 and the rules of many states.

ASTM 1610-14 also contemplates testimony about a "physical match" (sec. 8.6) based on postulated "individualizing characteristics" rather than more appropriate probabilistic assessments. Section 8.6.1 advises that "[t]he most conclusive type of examination that can be performed on paint samples is physical matching. ... The corresponding features must possess individualizing characteristics."

The requirement of "individualizing characteristics" is too strict. Although intuition indicates that a combination of enough characteristics can constitute very strong evidence of a common source, it is not clear that any single characteristic is "individualizing." And, even if the existence of strictly individualizing characteristics has been demonstrated in the scientific literature, a criminalist should be able to use other characteristics in forming an expert opinion. After all, a combination of other characteristics that are known to be less than perfectly discriminating when considered one at a time can be highly discriminating when evaluated in toto.

Thus, the approach to "physical matches" in section 8.6.1 seems inconsistent with the logic of section 5.2 of the same ASTM document. This section explains that

Searching for differences between questioned and known samples is the basic thrust of forensic paint analysis and comparison. However, differences in appearance, layer sequence, size, shape, thickness, or some other physical or chemical feature can exist even in samples that are known to be from the same source. A forensic paint examiner’s goal is to assess the significance of any observed differences. The absence of significant differences at the conclusion of an analysis suggests that the paint samples could have a common origin. The strength of such an interpretation is a function of the type or number of corresponding features, or both.

This language is important for legal purposes because it affects the presentation of negative findings (“absence of significant differences”) that would incriminate a suspect or defendant. Surprisingly, there is nothing in the standard to guide or inform the analyst about how to report these results. Is the analyst expected to report only that “I could find no significant differences”? That seems insufficient. That “I am suggesting that the paint samples could have a common origin”? That even though “differences can exist even in samples that are known to be from the same source,” in this case there is “an absence of significant differences”? That may connote more than is appropriate. That, given “the type or number of corresponding features,” the interpretation of “common source” is very strong? What studies establish that a criminalist accurately can quantify (even verbally) the strength of the common-source interpretation? What is the uncertainty associated with such judgments? To be sure, the Standard Guide lists 69 references, but it is not clear which, if any, of them answer these questions.

Perhaps I am being too critical. Readers are invited to examine the ASTM Standard Guide for themselves and tell OSAC what they think.

Note

1. E.g., John S. Buckleton et al., An Extended Likelihood Ratio Framework for Interpreting Evidence, 46 Sci. & Just. 69, 70 (2006) ("The idea of assessing the weight of evidence using a relative measure (known as the likelihood ratio) ... dominates the literature as the method of choice for interpreting forensic evidence across evidence types."); Angel Carrecedo, Forensic Genetics: History, in Forensic Biology 19, 22 (Max M. Houck ed. 2015) (“the single most important advance in forensic genetic thinking is the realization that the scientist should address the probability of the evidence.”).