Reflections on Glass Standards: Statistical Tests and Legal Hypotheses

Statistical Applicata (Italian Journal of Applied Statistics) recently published several issues (volume 27, nos. 2 & 3) devoted to statistics in forensic science and law. They include an invited article I prepared in 2016 on the statistical logic of declaring pieces of glass "indistinguishable" in their physical properties. 1/ The article contains some of the views expressed in postings on this blog (e.g., Broken Glass: What Do the Data Show?). However, the issue is much broader than glass evidence. The article notes the potential for confusion in reporting that any kind of trace-evidence samples match (or cannot be distinguished) without also describing data on the frequency of such matches in a relevant population. I am informed that NIST's Organization of Scientific Area Committees on Forensic Science (OSAC) is preparing guidelines or standards for explaining the probative value of results obtained from ASTM-approved test methods.

Abstract

---

The past 50 years have seen an abundance of statistical thinking on interpreting measurements of chemical and physical properties of glass fragments that might be associated with crime scenes. Yet, the most prominent standards for evaluating the degree of association between specimens of glass recovered from suspects and crime scenes have not benefitted from much of this work. Being confined to a binary match/no-match framework, they do not acknowledge the possibility of expressing the degree to which the data support competing hypotheses. And even within the limited match/no-match framework, they focus on the single step of deciding whether samples can be distinguished from one another and say little about the second stage of the matching paradigm–characterizing the probative value of a match. This article urges the extension of forensic-science standards to at least offer guidance for criminalists on the second stage of frequentist thinking. Toward that end, it clarifies some possible sources of confusion over statistical terminology such as “Type I” and “Type II” error in this area, and it argues that the legal requirement of proof beyond a reasonable doubt does not inform the significance level for tests of whether pairs of glass fragments have identical chemical or physical properties.

Note

1. The article is David H. Kaye, Reflections on Glass Standards: Statistical Tests and Legal Hypotheses, 27 Statistica Applicata -- Italian J. Applied Stat. 173 (2015). Despite the publication date assigned to the issue, the article, as stated above, was not written until 2016.

If the Glass Fits, Declare It: The Justice Department's ULTR for Glass that "Physically Fits Together"

The Department of Justice’s “Proposed Uniform Language for Testimony and Reports for the Forensic Glass Discipline” allows a broad range of possible conclusions and statements of probability. The strongest conclusion of association would be that “the glass fragments were once part of the same broken object.” The draft ULTR permits a criminalist to “state or imply” this source conclusion—but only “when two or more pieces of broken glass physically fit together.”

The notion that fragments that fit together must have come from the same larger piece of  glass sounds like common sense. Almost all of us have broken glass or ceramic materials at one time or another and seen that the fragments (at least the major ones) can be reassembled. Apparently, criminalists have had the same experience. The supporting documentation for the ULTR notes that “[i]t has long been reported by forensic glass examiners that two glass objects that physically fit together were once part of the same broken object.” 1/

But what type of scientific inquiry would establish the truth of the common sense idea? Let’s do a series of thought experiments. I have ten drinking glasses in my cupboard. (Actually, six are in the dishwasher, but I’ll pretend all are clean and back in the cupboard.) They sure look alike to me, being the same size, shape, and color.

• Experiment 1. I take one glass out and strike it with a hammer. Then I pick up the pieces and discover that all pairs of fragments that were adjacent in the unbroken glass fit together.


• Experiment 2. I take out a second glass and strike it with a hammer. Then I put all the pieces in a bag with those from the first glass and mix them up. Will I find any pairs of fragments that seem to fit together that are Glass1-Glass2 (heteroginous, if I can coin a word that means “of different origin”), or will all the pairs that fit together be Glass1-Glass1 or Glass2-Glass2 (homoginous, a neologism for "of the same origin")? Let’s suppose that the subset of all the possible pairs that I find to match are indeed homoginous.


• Experiment 3. You guessed it. I take a third glass, strike it with the hammer, and examine all possible pairs from all the fragments from the three glasses. This is getting very time-consuming, for the number of pairs to consider is growing exponentially, but I am persistent. Can we conclude, on the basis of common sense, that each and every physically matching pair will be homoginous?

You can see where I am going with this. Common sense plus some cogitation suggests that homoginality — a given pair of matching fragments coming from the same original piece of glass — depends on how many fragments are in the relevant population and the complexity of the match (that is, the extent of the irregularities in the edges of the fragments being fitted together). It is not so obvious that every physically matching pairs of glass fragments that can possibly exist are homoginous. That glass examiners have said that all matching pairs must be homoginous does not make it so. Neither does labeling mechanical fit as an individual as opposed to a class characteristic (to use the popular terminology in forensic science). The labeling exercise begs the question.

Nonetheless, it is quite reasonable to conjecture that only a tiny fraction (which could include 0) of apparently matching fragments that have complex edge patterns are heteroginous, and thus that finding a physical fit is powerful evidence of homoginality. 2/ Of course, the reasonableness of the conjecture depends on the criteria for a fit and the method for declaring one—neither of which are mentioned in the DOJ materials—but let’s assume the existence and maintenance of rigorous criteria and a reliable method. Can criminalists currently estimate how often the conclusion of homoginality is correct when the fragments (unbeknownst to the analyst) actually come from different pieces of glass?

Even if the answer to this question is in the negative, there are ways to present the physical match without overclaiming. Why must examiners “state or imply that the glass fragments were once part of the same broken object”? Examiners who believe that there is a physical fit can contribute a great deal simply by documenting and exhibiting the fit itself. If, additionally, they have some special expertise in moving from the fit to the conclusion of homoginality, then doing also could assist the fact finder. But if their inference is simply that of common sense, the expert aspect of the probative value of that part of the testimony is questionable. In these circumstances, the criminalist might be better advised to show how well the fragments fit together and leave it to the judge or jury to assess the weight of this demonstrable fact.

Another alternative to testimony “to the exclusion of all other sources” (see note 1) is a statement that the observed match is more probable when fragments really come from the same broken object than when they come from different broken objects. I do not know whether specific studies have been performed to justify this likelihood-ratio-type statement, but it should not be outside the capacity of forensic science to generate data that would support this rather modest statement. Indeed, the claim probably is too weak. My intuition is that the likelihood ratio is much greater than 1—it is a heck of a lot more probable to generate a matching pair of fragments by breaking the same object than by breaking different ones. A lawyer can appeal to give the evidence substantial weight on the basis of similar beliefs. If criminalists have experimental or observational studies to quantify how much more probable the physical match is for homoginous fragments, then they can provide that likelihood ratio.

Based on the material the DoJ has supplied, however, the more ambitious criminalists who want to “render an opinion of positive identity” and asservate “to the exclusion of all other sources” (see note 1), as the proposed ULTR apparently allows, should be required to express the limitation that no matter how positive they are in their opinion, the core of that opinion is a common sense judgment of what it means for pieces of glass to fit together rather than the product of elaborate scientific experimentation.

Note

1. An accompanying footnote cites four sources:
   • Kirk, P., Density and Refractive Index: Their Application in Criminal Identification, Charles C. Thomas, Springfield, IL, 1951, p. 4-5.
   • “Don’t Overlook Evidentiary Value of Glass Fragments”, Law Enforcement Bulletin, Vol. 33, No. 10, October 1964, p. 19.
   • Bottrell, M.C., “Forensic Glass Comparison: Background Information Used in Data Interpretation”, Forensic Science Communications [Online], (April, 2009). Available: http://www.fbi.gov/about-us/lab/forensic-science-communications/fsc/april2009/review/2009_04_review01.htm. accessed on August 15, 2013.
   • Koons, R. D., Buscaglia, J., Bottrell, M., and Miller, E. T. Forensic glass comparisons. In: Forensic Science Handbook. vol. I, 2nd ed. Richard Saferstein, Ed., Prentice Hall, Upper Saddle River, New Jersey, 2002, p. 161–213.
   
   I do not have Paul Kirk’s 1951 book at hand. The Law Enforcement Bulletin is not generally regarded as a substantial scientific publication. The 2009 publication is a generally informative and comprehensive review article from the FBI. However, it contains no reports of examiners' experiences or experiments with physically fitting fragments of glass back together. In fact, only two sentences even refer to physically fitting fragments—and neither of them explains the basis for the conclusion that the ULTR endorses. (The first sentence is this: “Only physically matching two or more broken glass fragments allows for their association with each other to the exclusion of all other sources (Scientific Working Group for Materials Analysis [SWGMAT] 2005c).” The second is equally devoid of information on the foundation for the conclusion. It reads: “The ... Trace Evidence Unit Quality Assurance Manual (2006) states [that a] glass association is defined as two or more glass samples that can be fracture fitted together, or that exhibit indistinguishable observable properties and/or range overlap in all measured properties.”)
   
   Finally, the FBI chapter in Richard Saferstein’s handbook discusses “types of fractures” (pp. 173-77) and later, “the mechanical fit.” The authors write that

   The mechanical fit is one of the most desirable of forensic glass examinations because the examiner can render an opinion of positive identity—that two or more pieces of glass were once a portion of, and were broken from, the same pane or object. Glass is particularly suitable for this type of examination. It is amorphous and brittle, is neither stretched nor distorted by breakage, and can be reassembled to its original configuration. Because it is amorphous, no two glass objects will break in exactly the same way. (P. 179).

   As has been discussed extensively for other types of matching evidence, the claim of uniqueness at some possibly unattainable level of precision of measurement does not imply the truth of an unqualifiedly positive opinion of identity.


2. See generally David H. Kaye, Beyond Uniqueness: The Birthday Paradox, Source Attribution, and Individualization in Forensic Science Testimony, 12 Law, Probability & Risk 3 (2013); David H. Kaye, Probability, Individualization, and Uniqueness in Forensic Science Evidence: Listening to the Academies, 75 Brooklyn L. Rev. 1163 (2010); David H. Kaye, Identification, Individuality, and Uniqueness: What's the Difference?, 8 Law, Probability & Risk 85 (2009).

What Is a "Conservative" Method in Forensic Statistics?

Statistical hypothesis testing involves a "null hypothesis" against an "alternative hypothesis." If data are not well outside the range of what would be expected if the null hypothesis is true, then that hypothesis cannot be rejected in favor of the specified alternative. It is usually thought that the more demanding the statistical test, the more "conservative" it is. For example, if a researcher claims to have discovered a new treatment that cures cancer, the null hypothesis is that the new therapy does not work. Sticking with this belief retains the status quo (of not using the novel treatment). In this example, the "conservative" thing to do is to insist on a small p-value (results that have a small probability of arising if the treatment is ineffective) before accepting the alternative.

Does this carry over to forensic science? Is it conservative to retain the null hypothesis unless there is strong evidence against it? Consider the following excerpt from an FBI publication on forensic glass comparisons 1/:

A conservative threshold will differentiate all samples from different sources but may also indicate that a difference exists in specimens that are actually from the same source. A high threshold for differentiation may not be able to differentiate all specimens from sources that are genuinely different but will not differentiate specimens that are actually from the same source.

The "conservative" scientific stance therefore tends to support or preserve the prosecution's case. The state can produce a witness who can testify "conservatively" to finding that the broken window at the crime scene is chemically indistinguishable from the bit of glass removed from the defendant's sweatshirt.

On the other hand, a committee of the National Academic of Sciences that studied forensic DNA testing defined "conservative" in terms of impact on a defendant's claim of innocence 2/:

Conservative—favoring the defendant. A conservative estimate is deliberately chosen to be more favorable to the defendant than the best (unbiased) estimate would be.

Plainly, the FBI document's use of "conservative" is difficult to square with the NAS committee's definition of the word. The FBI document treats the hypothesis that favors the hypothesis supporting the prosecution's case as the status quo that should be retained unless there is strong evidence to the contrary.

This use of the prosecution's hypothesis that the broken window is the source of the incriminating fragment as the null hypothesis is not necessarily wrong, but it engenders confusion. The confusion can be dispelled if the presentation of the findings includes a statement of how rare or common "indistinguishable" windows are in a relevant population. Evaluating the data about the glass thus would have two steps. In step 1, the data are classified as"indistinguishable" (or not). If the samples are indistinguishable, then a random match probability is provided to indicate its probative value with respect to the hypothesis that the glass originated from the broken window.

Of course, if one could articulate the probability of the data given the hypothesis that the source is broken window versus the probability that the glass associated with the defendant had a different origin, this two-step process would not be needed. The expert could present these probabilities.

References

1. Maureen C. Bottrell, Forensic Glass Comparison: Background Information Used in Data Interpretation, 11 Forensic Sci. Communications No. 2 (2009)
2. National Research Council, Committee on The Evaluation of Forensic DNA Evidence: An Update, The Evaluation of Forensic DNA Evidence 215 (1996)

Broken Glass: What Do the Data Show?

In Broken Glass, Mangled Statistics, I noted "a plethora of statistical issues" in ASTM E2926-13, a Standard Test Method for Forensic Comparison of Glass Using Micro X-ray Fluorescence (μ-XRF) Spectrometry, that is working its way through the process for inclusion on the OSAC Registry of Approved Standards. The questions I raised about the Standard's procedures and criteria for declaring matches between glass specimens were based on elementary statistical theory but not data. Even if the ASTM's hypothesis testing procedures are idiosyncratic or conceptually flawed, they could have desirable properties.

There are some collections of glass that have been used to test the performance of the matching rules for some of the variables used in forensic testing. An FBI publication from 2009 offers the following summary:

Databases of refractive indices and/or chemical compositions of glass received in casework have been established by a number of crime laboratories (Koons et al. 1991). Although these glass databases are undeniably valuable, it should be noted that they may not be representative of the actual population of glass, and the distribution of glass properties may not be normal. Although these are not direct indicators of the rarity in any specific case, they can be used to show that the probability of a coincidental match is rare.

Koons and Buscaglia (1999) used the data from a chemical composition database and refractive index database to calculate the probability of a coincidental match. They estimated that ... the chance of finding a coincidental match in forensic glass casework using refractive index and chemical composition alone is 1 in 100,000 to 1 in 10 trillion, which strongly supports the supposition that glass fragments recovered from an item of evidence and a broken object with indistinguishable [refractive index] and chemical composition are unlikely to be from another source and can be used reliably to assist in reconstructing the events of a crime.

Range overlap on glass analytical data that include chemical composition data is considered a conservative standard. In one study, on a data set consisting of three replicate measurements each for 209 specimens, the range-overlap test discriminated all specimens, and all other statistical analysis-based tests performed worse (Koons and Buscaglia 2002).

Range-overlap tests, however, may achieve their high discrimination by indicating that two specimens from the same source are differentiable. Another study showed that when using a range-overlap test, the number of specimens differentiated that were actually from the same source may have been as high as seven percent (Bottrell et al. 2007).

The range-overlap approach, however, seems prudent given that other tests with higher thresholds for differentiation, such as t-tests with Welch modification (Curran et al. 2000) or Bayesian analysis (Walsh 1996), lower the number of specimens differentiated that were actually from the same source by worsening the ability to differentiate specimens that are genuinely different, a result that is unacceptable.

If I understand the argument, the author contends that high sensitivity is more important than high specificity. That makes sense for a screening test that will be followed by a more specific test, but in general, is it better to avoid falsely associating a defendant with crime-scene glass or to avoid falsely associating the defendant with the known glass? Any decision rule as to what is "indistinguishable" will generate a mix of false positives and false negatives. Should not the ASTM standards provide estimates from data (that might be representative of some relevant population) of these risks for each decision rule that the standards endorse or mandate?

References

• Maureen C. Bottrell, Forensic Glass Comparison: Background Information Used in Data Interpretation, 11 Forensic Sci. Communications No. 2 (2009), https://www.fbi.gov/about-us/lab/forensic-science-communications/fsc/april2009/review/2009_04_review01.htm/
• Broken Glass, Mangled Statistics, Forensic Science, Statistics, and the Law, Feb. 3, 2016

Broken Glass, Mangled Statistics

The motto of ASTM International is “Helping Our World Work Better.” This internationally recognized standards development organization contributes to the world of forensic science by promulgating standards of various kinds for performing and interpreting chemical and other tests.

By mid-August 2015, five ASTM Standards were up for public comment to the Organization of Scientific Area Committees. OSAC “is part of an initiative by NIST and the Department of Justice to strengthen forensic science in the United States.” [1] Operating as “[a] collaborative body of more than 500 forensic science practitioners and other experts,” [1] OSAC is reviewing and developing documents for possible inclusion on a Registry of Approved Standards and a Registry of Approved Guidelines. 1/ NIST promises that “[a] standard or guideline that is posted on either Registry demonstrates that the methods it contains have been assessed to be valid by forensic practitioners, academic researchers, measurement scientists, and statisticians ... .” [2]

Last month, OSAC approved its first Registry entry (notwithstanding some puzzling language), ASTM E2329-14, a Standard Practice for Identification of Seized Drugs. Another standard on the list for OSAC’s quasi-governmental seal of approval is ASTM E2926-13, a Standard Test Method for Forensic Comparison of Glass Using Micro X-ray Fluorescence (μ-XRF) Spectrometry (available for a fee). It will be interesting to see whether this standard survives the scrutiny of measurement scientists and statisticians, for it raises a plethora of statistical issues.

What It is All About

Suppose that someone stole some bottles of beer and money from a bar, breaking a window to gain entry. A suspect’s clothing is found to contain four small glass fragments. Various tests are available to help determine whether the four fragments (the “questioned” specimens) came from the broken window (the “known”). The hypothesis that they did can be denoted H₁, and the “null hypothesis” that they did not can be designated H₀.

Micro X-ray Fluorescence (μ-XRF) Spectrometry involves bombarding a specimen with X-rays. The material then emits other X-rays at frequencies that are characteristic of the elements that compose it. In the words of the ASTM Standard, “[t]he characteristic X-rays emitted by the specimen are detected using an energy dispersive X-ray detector and displayed as a spectrum of energy versus intensity. Spectral and elemental ratio comparisons of the glass specimens are conducted for source discrimination or association.” Such “source discrimination” would be a conclusion that H₀ is true; “association” would be a conclusion that H₁ is true. The former finding would mean that the suspect's glass fragments did not come from the crime scene; the latter would mean either that (one way or another) they came from the broken window at the bar (or from another piece of glass somewhere that has a similar elemental composition).

Unspecified "Sampling Techniques"
for Assessing Variability Within the Pane of Window Glass

One statistical issue arises from the fact that the known glass is not perfectly homogeneous. Even if measurements of the ratios of the concentrations of different elements in a specimen are perfectly precise (the error of measurement is zero), a fragment from one location could have a different ratio than a fragment from another place in the known specimen. This natural variability must be accounted for in deciding between the two hypotheses. The Standard wisely cautions that “[a]ppropriate sampling techniques should be used to account for natural heterogeneity of the material, varying surface geometries, and potential critical depth effects.” But it gives no guidance at all as to what sampling techniques can accomplish this and how measurements that indicate spatial variation should be treated.

The Statistics of "Peak Identification"

The section of ASTM E-2926 on “Calculation and Interpretation of Results” advises analysts to “[c]ompare the spectra using peak identification, spectral comparisons, and peak intensity ratio comparisons.” First, “peak identification” means comparing “detected elements of the questioned and known glass spectra.” The Standard indicates that when “[r]eproducible differences” in the elements detected in the specimens are found, the analysis can cease and the null hypothesis H₀ can be presented as the outcome of the test. No further analysis is required. The criterion for when an element “may be” detected is that “the area of a characteristic energy of an element has a signal-to-noise ratio of three or more.” Where did this statistical criterion come from? What is the sensitivity and specificity of a test for the presence of an element based on this criterion?

The Statistics of Spectral Comparisons

Second, “spectral comparisons should be conducted,” but apparently, only “[w]hen peak identification does not discriminate between the specimens.” This procedure amounts to eyeballing (or otherwise comparing?) “the spectral shapes and relative peak heights of the questioned and known glass specimen spectra.” But what is known about the performance of criminalists who undertake this pattern-matching task? Has their sensitivity and specificity been determined in controlled experiments, or are judgments accepted on the basis of self-described but incompletely validated “knowledge, skill, ability, experience, education, or training ... used in conjunction with professional judgment,” to use a stock phrase found in many an ASTM Standard?

The Statistics of Peak Intensity Ratios

Third, only “[w]hen evaluation of spectral shapes and relative peak heights do not discriminate between the specimens” does the Standard recommend that “peak intensity ratios should be calculated.” These “peak intensity ratio comparisons” for elements such as “Ca/Mg, Ca/Ti, Ca/Fe, Sr/Zr, Fe/Zr, and Ca/K” “may be used” “[w]hen the area of a characteristic energy peak of an element has a signal-to-noise ratio of ten or more.” To choose between “association” and “discrimination of the samples based on elemental ratios,” the Standard recommends, “when practical,” analyzing “a minimum of three replicates on each questioned specimen examined and nine replicates on known glass sources.” Inasmuch as the Standard emphasizes that “μ-XRF is a nondestructive elemental analysis technique” and “fragments usually do not require sample preparation,” it is not clear just when the analyst should be content with fewer than three replicate measurements—or why three and nine measurements provide a sufficient sampling to assess measurement variability in two sets of specimens, respectively.

Nevertheless, let’s assume that we have three measurements on each of the four questioned specimens and nine on the known specimen. What should be done with these two sets of numbers? The Standard first proposes a “range overlap” test. I’ll quote it in full:

For each elemental ratio, compare the range of the questioned specimen replicates to the range for the known specimen replicates. Because standard deviations are not calculated, this statistical measure does not directly address the confidence level of an association. If the ranges of one or more elements in the questioned and known specimens do not overlap, it may be concluded that the specimens are not from the same source.

Two problems are glaringly apparent. First, statisticians appreciate that the range is not a robust statistic. It is heavily influenced by any outliers. Second, if the  properties of the "ratio ranges" are unknown, how can one know what to conclude—and what to tell a judge, jury, or investigator about the strength of the conclusion? Would a careful criminalist who finds no range overlap have to quote or paraphrase the introduction to the Standard, and report that "the specimens are indistinguishable in all of these observed and measured properties," so that "the possibility that they originated from the same source of glass cannot be eliminated"? Would the criminalist have to add that there is no scientific basis for stating what the statistical significance of this inability to tell them apart is? Or could an expert rely on the Standard to say that by not eliminating the same-source possibility, the tests "conducted for source discrimination or association" came out in favor of association?

The Standard offers a cryptic alternative to the simplistic range method (without favoring one over the other and without mentioning any other statistical procedures):

±3s—For each elemental ratio, compare the average ratio for the questioned specimen to the average ratio for the known specimens ±3s. This range corresponds to 99.7 % of a normally distributed population. If, for one or more elements, the average ratio in the questioned specimen does not fall within the average ratio for the known specimens ±3s, it may be concluded that the samples are not from the same source.

The problems with this poorly written formulation of a frequentist hypothesis test are legion:

1. What "population" is "normally distributed"? Apparently, it is the measurements of the elemental ratios in the questioned specimen. What supports the assumption of normality?

2. What is "s"? The standard deviation of what variable? It appears to be the sample standard deviation of the nine measurements on the known specimen.

3. The Standard seems to contemplate a 99.7% confidence interval (CI) for the mean μ of the ratios in the known specimen. If the measurement error is normally distributed about μ, then the CI for μ is approximately the known specimen's sample mean ±4.3s. This margin of error is larger than ±3s because the population standard deviation σ is unknown and the sample mean therefore follows a t-distribution with eight degrees of freedom. The desired 99.7% is the coverage probability for a ±3σ CI. Using ±3 with the estimator s rather than the true value σ results in a confidence coefficient below 99%. One would have to use a number greater than ±4 rather than ±3 to achieve 99.7% confidence.

4. The use of any confidence interval for the sample mean of the measurements in the known specimen is misguided. Why ignore the variance in the measured ratios in the questioned specimens? That is, the recommendation tells the analyst to ask whether, for each ratio in each questioned specimen, the miscomputed 99.7% CI covers “the average ratio in the questioned specimen.” But this “average ratio” is not the true ratio. The usual procedure (assuming normality) would be a two-sample t-test of the difference between the mean ratio for the questioned sample and the mean for the known specimen.

5. Even with the correct test statistic and distribution, the many separate tests (one for each ratio Ca/Mg, Ca/Ti, Fe/Zr, etc.) cloud the interpretation of the significance of the difference in a pair of sample means. Moreover, with multiple unknown specimens, the probability of finding a significant difference in at least one ratio for at least one unknown fragment is greater than the significance probability in a single comparison. The risk of a false exclusion for, say, ten independent comparisons could be ten times the nominal value of 0.003.

6. Why ±3 as opposed to, say, ±4? I mention ±4 not because it is clearly better, but because it is the standard for making associations using a different test method (ASTM E2330). What explains the same standards development organization promulgating facially inconsistent statistical standards?

7. Why strive for a 0.003 false-rejection probability as opposed to, say, 0.01, 0.03, or anything else? This type of question can be asked about any sharp cutoff. Why is a difference of 2.99σ dismissed as not useful when 3σ is definitive? Within the classical hypothesis-testing framework, an acceptable answer would be that the line has to be drawn somewhere, and the 0.003 significance level is needed to protect against the risk of a false rejection of the null hypothesis in situations in which a false rejection would be very troublesome. Some statistics textbooks even motivate the choice of the less demanding but more conventional significance level of 0.05 by analogizing to a trial in which a false conviction is much more serious than a false acquittal.

Here, however, that logic cuts in the opposite direction. The null hypothesis H₀ that should not be falsely rejected is that the two sets of measurements come from fragments that do not have a common source. But 0.003 is computed for the hypothesis H₁ that the fragments all come from the same, known source. The significance test in ASTM E2926-13 addresses (in its own way) the difference in the means when sampling from the known specimen. Using a very demanding standard for rejecting H₁ in favor of the suspect’s claim H₀ privileges the prosecution claim that the fragments come from different sources. 2/ And it does so without mentioning the power of the test: What is probability of reporting that fragments are indistinguishable — that there is an association — when the fragments do come from different sources? Twenty years ago, when a National Academy of Sciences panel examined and approved the FBI's categorical rule of "match windows" for DNA testing, it discussed both operating characteristics of the procedure—the ability to declare a match for DNA samples from the same source (sensitivity) and the ability to declare a nonmatch for DNA samples from different sources. [3] By looking only to sensitivity, ASTM E2926-13 takes a huge step backwards.

8. Whatever significance level is desired, to be fair and balanced in its interpretation of the data, a laboratory that undertakes hypothesis tests should report the probability of differences in the test statistic as large or larger than those observed under the two hypotheses: (1) when the sets of measurements come from the same broken window (H₁); and (2) when the sets of measurements come from different sources of glass in the area in which the suspect lives and travels (H₀). The ASTM Standard completely ignores H₀. Data on the distribution of the elemental composition of glass in the geographic area would be required to address it, and the Standard should at least gesture to how such data should be used. If such data are missing, the best the analyst can do is to report candidly that the questioned fragment might have come from the known glass or from any other glass with a similar set of elemental concentrations and, for completeness, to add that how often other glass like this is present is unknown.

9. Would a likelihood ratio be a better way to express the probative value of the data? Certainly, there is an argument to that effect in the legal and forensic science literature. [4-8] Quantifying and aggregating the spectral data that the ASTM Standard now divides into three, lexically ordered procedures and combining them with other tests on glass would be a challenge, but it merits thought. Should not the Standard explicitly acknowledge that reporting on the strength of the evidence rather than making categorical judgments is a respectable approach?

* * *

In sum, even within the framework of frequentist hypothesis testing, ASTM E2926 is plagued with problems — from the wrong test statistic and procedure for the specified level of “confidence,” to the reversal of the null and alternative hypotheses, to the failure to consider the power of the test. Can such a Standard be considered “valid by forensic practitioners, academic researchers, measurement scientists, and statisticians”?

Notes

1. The difference between the two is not pellucid, since OSAC-approved standards can be a list of “shoulds” and guidelines can include “shalls.”
2. The best defense I can think of for it is a quasi-Bayesian argument that by the time H₁ gets to this hypothesis test, it has survived the qualitative "peak identification" and "spectral comparison" tests. Given this prior knowledge, it should require unusually surprising evidence from the peak intensity ratios to reject H₁ in favor of the defense claim H₀.

References

1. OSAC Registry of Approved Standards and OSAC Registry of Approved Guidelines http://www.nist.gov/forensics/osac/osac-registries.cfm, last visited Feb. 2, 2016
2. NIST, Organization of Scientific Area Committees, http://www.nist.gov/forensics/osac/index.cfm, last visited Feb. 2, 2016
3. National Research Council Committee on Forensic DNA Science: An Update, The Evaluation of Forensic DNA Evidence (1996)
4. Colin Aitken & Franco Taroni, Statistics and the Evaluation of Evidence for Forensic Science (2d ed. 2004)
5. James M. Curran et al., Forensic Interpretation of Glass Evidence (2000)
6. ENFSI Guideline for Evaluative Reporting in Forensic Science (2015)
7. David H. Kaye et al., The New Wigmore: Expert Evidence (2d ed. 2011)
8. Royal Statistical Soc'y Working Group on Statistics and the Law, Fundamentals of Probability and Statistical Evidence in Criminal Proceedings: Guidance for Judges, Lawyers, Forensic Scientists and Expert Witnesses (2010)

Postscript: See Broken Glass: What Do the Data Show?, Forensic Sci., Stat. & L., Feb. 13, 2016,

Disclosure and disclaimer: Although I am a member of the Legal Resource Committee of OSAC, the views expressed here are mine alone. They are not those of any organization. They are not necessarily shared by anyone inside (or outside) of NIST, OSAC, any SAC, any OSAC Task Force, or any OSAC Resource Committee.

First NIST OSAC Forensic Science Standards Up for Public Comment

One of the responses to the 2009 NRC report on forensic science was the creation last year of an Organization of Scientific Area Committees (OSAC) for forensic science organized by the National Institute of Standards and Technology (NIST). This organization is developing new standards for forensic disciplines to follow.

Last week, NIST opened a 30-day public comment period for five standards from the Chemistry Scientific Area Committee. They are continuations or updates of existing ASTM (American Society of Testing and Materials) standards. The NIST OSAC News Release on the public comment period is at http://www.nist.gov/forensics/osac/osac-opens-public-comment.cfm. The five standards under consideration for inclusion on the OSAC Registry of Approved Standards are as follows:

• ASTM E2329-14 Standard Practice for Identification of Seized Drugs
• ASTM E2330-12 Standard Test Method for Determination of Concentrations of Elements in Glass Samples Using Inductively Coupled Plasma Mass Spectrometry (ICP-MS) for Forensic Comparisons
• ASTM E2548-11e1 Standard Guide for Sampling Seized Drugs for Qualitative and Quantitative Analysis
• ASTM E2881-13e1 Standard Test Method for Extraction and Derivatization of Vegetable Oils and Fats from Fire Debris and Liquid Samples with Analysis by Gas Chromatography-Mass Spectrometry
• ASTM E2926-13 Standard Test Method for Forensic Comparison of Glass Using Micro X-ray Fluorescence (µ-XRF) Spectrometry

Although they may seem technical and have forbidding names, some of these proposed standards should be of interest to lawyers as well as forensic scientists and statisticians who might want them to address how findings should be presented in court or in reports. For example, one standard involving glass fragments requires a difference of 3 standard deviations before the analyst can reject the hypothesis that the fragments on the suspect came from the crime scene. But 2.9 usually would be pretty good evidence that the suspect's fragments are from some other glass. What should the standard require or allow an expert to report in cases like this, where the suspect's fragments lie within the broad window for measurement error? Should there be an adjustment to the rejection range if more than one fragment has been tested? Should there even be a fixed window, or should the analyst simply report the probability of differences in the measurements as or more extreme as those observed if the fragments on the suspect came from the crime-scene glass? Better still, can a likelihood ratio be provided?

It appears that this 30-day period also offers an opportunity to view related ASTM standards on forensic science tests. Normally, ASTM, as the copyright holder, does not make its standards freely available.

Directions for subscribing to the OSAC newsletter and receiving announcements of comment periods, new standards, etc., are at the above URL and at http://www.nist.gov/forensics/osac/osac-launches-monthly-newsletter.cfm.

Disclosure and disclaimer: Although I am a member of the Legal Resource Committee of OSAC, the views expressed here (to the extent I have expressed any) are mine alone. They are not those of any organization. They are not necessarily shared by anyone inside (or outside) of NIST, OSAC, any SAC, any OSAC Task Force, or anyone else in the Legal Resource Committee.