PCAST’s Sampling Errors (Part I)

The President’s Council of Advisors on Science and Technology (PCAST) has reported to the President that important steps must be taken to improve forensic science. Among other valuable recommendations, its report on Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods insists that the uncertainty associated with forensic-science findings of a positive association between a criminal defendant and some form of incriminating trace, impression, or pattern evidence should be estimated with reasonable accuracy, and that this estimate must be presented in court. These broad propositions are ones to which everyone concerned with doing justice should subscribe.

Yet, the PCAST report also argues in favor of presenting, through a statistical device known as a one-sided confidence interval, only part of the picture concerning the statistical error in studies of the performance of forensic examiners. In expressing this more detailed position, the PCAST report misstates the meaning of confidence intervals and offers a dubious justification.

The report states that “[b]y convention, a confidence level of 95 percent is most widely used—meaning that there is a 5 percent chance the true value exceeds the bound.” (P. 153). This explanation reiterates a similar statement in a much earlier report of a committee of the National Research Council of the National Academies. In its 1992 report on DNA Technology in Forensic Science, the NRC committee discussed “the traditional 95% confidence limit, whose use implies that the true value has only a 5% chance of exceeding the upper bound.” (P. 76).

This loose language did not escape the attention of statisticians. Bruce Weir, for example, promptly responded that “[a] court may be excused for such nonstatistical language, but a report issued by the NRC ... must lose some credibility with statisticians.”  B. S. Weir, Population Genetics in the Forensic DNA Debate, 89 Proc. Nat’l Acad. Sci. USA 11654, 11654 (1992). As most statistics textbooks recognize, “a confidence level of 95%” does not mean “that there is a 5 percent chance the true value” lies outside the particular interval. In the theory that motivates confidence intervals, the true value is an unknown constant, not a variable that has a probability associated with it. The 5% figure is the expected proportion of instances in which many intervals (most of them somewhat different in their central location and width) will cover the unknown, true interval. Only a Bayesian analysis can supply a probability that the true value falls outside a given interval.

This criticism may sound rather technical, and it is, but considering the scientific horsepower on PCAST, one would have expected its description of statistical concepts to be unobjectionable. More disturbing is the report’s dismissal of the presentation of standard, two-sided confidence intervals as “obfuscation”:

Because one should be primarily concerned about overestimating SEN [sensitivity] or underestimating FPR [false positive rate], it is appropriate to use a one-sided confidence bound. By convention, a confidence level of 95 percent is most widely used—meaning that there is a 5 percent chance the true value exceeds the bound. Upper 95 percent one-sided confidence bounds should thus be used for assessing the error rates and the associated quantities that characterize forensic feature matching methods. (The use of lower values may rightly be viewed with suspicion as an attempt at obfuscation.)

P. 153. Without unearthing the technical details of one-sided and two-sided confidence intervals, and in full agreement with the notion that good science requires acknowledging the possibility that false-positive error probabilities could occur more often than seen in a single study, I have to say that this paragraph seems to contradict the ideal of a forensic scientist who does not take sides.

Certainly, the law (not science) treats false convictions are more serious than false acquittals. But does this asymmetry imply that expert witnesses should not discuss the fact that sampling error (which is the only thing that confidence intervals address) can work in both directions? The legal and social policy judgment that it is better to risk a false acquittal than a false conviction requires the state to prove its case decisively — by a body of evidence that leaves no reasonable doubt about the defendant’s guilt. At the same time, it presumes that evidence should be presented and assessed for what is worth — neither more nor less. Consequently, the uncertainties in scientific evidence should be made clear — whichever way they cut. If possible, they should be expressed fairly, without favoring the prosecution's theory or the defense's.

As such, we should amend PCAST’s talk of “obfuscation.” It is fair to say that the exclusive use of lower values, or even point estimates — instead of both upper and lower values — may rightly be viewed with suspicion as an attempt at obfuscation. It is equally fair to say that the exclusive use of upper values also may rightly be viewed with suspicion as an attempt at obfuscation. Finally, presentation of the full range of uncertainty can be viewed with approbation as an attempt at transparency. In sum it is far from clear that the one-sided 95% confidence interval best achieves the objectives of either law or science.

Technical postscript of 12/8/16 (for people who want to use PCAST's estimates of sampling error)

A forensic scientist wrote me that he was unable to replicate the numbers PCAST provided for one-sided 95% confidence intervals. The report intimidatingly states that

For technical reasons, there is no single, universally agreed method for calculating these confidence intervals (a problem known as the “binomial proportion confidence interval”). However, the several widely used methods give very similar results, and should all be considered acceptable: the Clopper-Pearson/Exact Binomial method, the Wilson Score interval, the Agresti-Coull (adjusted Wald) interval, and the Jeffreys interval. 396/ Web-based calculators are available for all of these methods. 397/ For example, if a study finds zero false positives in 100 tries, the four methods mentioned give, respectively, the values 0.030, 0.026, 0.032, and 0.019 for the upper 95 percent confidence bound. From a scientific standpoint, any of these might appropriately be reported to a jury in the context “the false positive rate might be as high as.” (In this report, we used the Clopper-Pearson/Exact Binomial method.)

P. 153. Relying on the PCAST approach to testify to a range for error rates could be dangerous. The article that the PCAST report cites does not conclude that the four methods perform equally well. Neither does it recommend the Clopper-Pearson (CP) method. The article is Lawrence D. Brown, T. Toni Cai & Anirban DasGupta, Interval Estimation for a Binomial Proportion, 16 Stat. Sci. 101 (2001). The abstract "recommend[s] the Wilson interval or the equal-tailed Jeffreys prior interval for small n and the interval suggested in Agresti and Coull for larger n." The authors explain that "for small n (40 or less), we recommend that either the Wilson or the Jeffreys prior interval should be used. They are very similar, and either may be used depending on taste." Id. at 102. "For larger n (n > 40), the Wilson, the Jeffreys and the Agresti–Coull interval are all very similar, and so for such n, due to its simplest form, we come to the conclusion that the Agresti–Coull interval should be recommended." Id. at 103. Brown et al. were especially critical of the procedure that PCAST used. They described the CP interval as "rather inaccurate" and concluded that

The Clopper–Pearson interval is wastefully conservative and is not a good choice for practical use, unless strict adherence to the prescription C (p, n ) ≥ 1−α is demanded. Even then, better exact methods are available ... .

The statistical calculator (EpiTools) that PCAST recommended likewise warns that "the Clopper-Pearson Exact method is very conservative and tends to produce wider intervals than necessary."

How much of a difference does this really make? I have not done the necessary computations, and I will be surprised if they produce big swings in the estimated upper bounds. After I get the chance to grind out the numbers, I will supply them in a later posting.

Index to comments and cases on the PCAST report

PCAST on "Foundational Validity," Evidentiary Reliability, and the Admissibility of "Firearms Analysis"

Finding 6 of President's Council of Scientific Advisors (PCAST), on "firearms analysis," is blunt:

PCAST finds that firearms analysis currently falls short of the criteria for foundational validity, because there is only a single appropriately designed study to measure validity and estimate reliability. The scientific criteria for foundational validity require more than one such study, to demonstrate reproducibility.

(P. 112). In the next breath, the report adds, " Whether firearms analysis should be deemed admissible based on current evidence is a decision that belongs to the courts." Id.

The juxtaposition of these observations is puzzling, for the report also seems to equate "foundational validity" to the legal standard for admissibility in Federal Rule of Evidence 702(c). This part of Rule 702 requires expert testimony to be "the product of reliable principles and methods." Seeking "complete clarity about our intent," PCAST (p. 43) states that

we have adopted specific terms to refer to the scientific standards for two key types of scientific validity, which we mean to correspond, as scientific standards, to the legal standards in Rule 702 (c,d)):
(1) by “foundational validity,” we mean the scientific standard corresponding to the legal standard of evidence being based on “reliable principles and methods,” and
(2) by “validity as applied,” we mean the scientific standard corresponding to the legal standard of an expert having “reliably applied the principles and methods.”

In other words, the PCAST report apparently advances these claims:

1. Evidence E ("firearms analysis") lacks quality Q ("foundational validity," which is a "key type of scientific validity").
2. Q is a sine qua non for the admissibility of all expert evidence under Rule 702.
3. Courts can admit E if they want to.

Can the third claim be reconciled with the first two? I can think of three possibilities:

1. A court might not be bound by Rule 702(c) because the jurisdiction does not follow this rule. (The rule was amended to include the words "reliable principles and methods" to codify the Daubert v. Merrell Dow Pharmaceuticals, 509 U.S. 579 (1993), line of cases (particularly General Electric Co. v. Joiner, 522 U.S. 136 (1997)), but a substantial minority of states apply a different version of the rule.)
2. A court can reject PCAST's claim that "foundational validity" is "the scientific standard corresponding to the legal standard" (at least in the manner that PCAST defines "foundational validity").
3. A court can reject PCAST's claim that the scientific literature does not establish "foundational validity" as the term is defined and applied in the report to "firearms analysis."

By introducing definitions of the terms "validity" and "reliability" that depart from the established meanings in statistics and the social sciences (p. 47 n.107), the PCAST scientists may not have achieved "complete clarity." On the one hand, we are told that established scientific criteria for "validity" correspond to evidentiary "reliability" under Rule 702. On the other, we are assured that the absence of "validity" does not dictate a legal conclusion. For me, this nomenclature falls short of complete clarity.

Index to comments and cases on the PCAST report

The First Opinion To Discuss the PCAST Report

In United States v. Chester, No. 13 CR 00774 (N.D. Ill. Oct. 7, 2016), a federal district court denied “defendants’ second joint renewed motion to exclude expert testimony regarding firearm toolmark analysis.” Normally, yet another federal district court opinion admitting testimony of a positive association in toolmarks would not be newsworthy. But this trial court decision came in response to a recent report from the President’s Council of Advisors on Science and Technology (PCAST). This report, Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods, argues that source-attribution testimony for bullet markings has not been demonstrated to possess scientific validity. 1/ Thus, Forensic Magazine ran this headline: “Firearms Evidence Allowed in Chicago Hobos Gang Trial—Despite PCAST Argument.” 2/ So what is PCAST's analysis, and how did the court overcome it?

PCAST expressed concern that the

“Theory of Identification as it Relates to Toolmarks”—which defines the criteria for making an identification—is circular. The “theory” states that an examiner may conclude that two items have a common origin if their marks are in sufficient agreement,” where “sufficient agreement” is defined as the examiner being convinced that the items are extremely unlikely to have a different origin. In addition, the “theory” explicitly states that conclusions are subjective (p. 104).

The court did not dispute the absence of a standardized procedure with well-defined judgmental criteria for source attribution. Rather, it maintained that the very scientific literature cited by PCAST does not contradict a previous ruling in the case that the judgments of firearms examiners amount to the kind of “scientific knowledge” necessary to admit scientific evidence under the Supreme Court's opinion in Daubert v. Merrell Dow Pharmaceuticals, 509 U.S. 579 (1993). As discussed at length in many publications (and occasionally in depth), Justice Blackmun's opinion in Daubert articulates a loose, multifactor standard for ascertaining the scientific soundness of proposed testimony, 3/ and the district court had examined the “Daubert factors.” in its first ruling. This time, it limited its analysis to “error rates.” Judge John J. Tharp, Jr., wrote that:

As such, the report does not dispute the accuracy or acceptance of firearm toolmark analysis within the courts. Rather, the report laments the lack of scientifically rigorous “blackbox” studies needed to demonstrate the reproducibility of results, which is critical to cementing the accuracy of the method. Id. at 11. The report gives detailed explanations of how such studies should be conducted in the future, and the Court hopes researchers will in fact conduct such studies. See id. at 106. However, PCAST did find one scientific study that met its requirements (in addition to a number of other studies with less predictive power as a result of their designs). That study, the “Ames Laboratory study,” found that toolmark analysis has a false positive rate between 1 in 66 and 1 in 46. Id. at 110. The next most reliable study, the “Miami-Dade Study” found a false positive rate between 1 in 49 and 1 in 21. Thus, the defendants’ submission places the error rate at roughly 2%. ³ The Court finds that this is a sufficiently low error rate to weigh in favor of allowing expert testimony. See Daubert v. Merrell Dow Pharms., 509 U.S. 579, 594 (1993) (“the court ordinarily should consider the known or potential rate of error”); United States v. Ashburn, 88 F. Supp. 3d 239, 246 (E.D.N.Y. 2015) (finding error rates between 0.9 and 1.5% to favor admission of expert testimony); United States v. Otero, 849 F. Supp. 2d 425, 434 (D.N.J. 2012) (error rate that “hovered around 1 to 2%” was “low” and supported admitting expert testimony). The other factors remain unchanged from this Court’s earlier ruling on toolmark analysis. See ECF No. 781.

3. Because the experts will testify as to the likelihood that rounds were fired from the same firearm, the relevant error rate in this case is the false positive rate (that is, the likelihood that an expert’s testimony that two bullets were fired by the same source is in fact incorrect).

But is it true that “the report does not dispute the accuracy ... of firearm toolmark analysis within the courts”? The report claims that the accuracy of the statements that appear in court is not known with adequate precision. According to the authors, no convincing range for the risks of errors can be derived from the existing scientific literature. PCAST insists that “[b]ecause firearms analysis is at present a subjective feature-comparison method, its foundational validity can only be established through multiple independent black box studies ... .” 4/ PCAST is emphatic (some would say dogmatic) in insisting that “the sole way to establish foundational validity is through multiple independent ‘black-box’ studies that measure how often examiners reach accurate conclusions across many feature-comparison problems involving samples representative of the intended use. In the absence of such studies, a feature-comparison method cannot be considered scientifically valid” (pp. 66, 68).

Under this criterion for scientific validity, it is hard to see how the PCAST report can be characterized as not disputing accuracy. The Miami-Dade study that the opinion relies on barely counts for PCAST. The report lists it under the heading of “Non-black-box studies of firearms analysis” (p. 106). That leaves a single study, and a single study cannot satisfy the report’s demand for multiple studies. For better or worse, PCAST's bottom line is clear

At present, there is only a single study that was appropriately designed to test foundational validity and estimate reliability (Ames Laboratory study). Importantly, the study was conducted by an independent group, unaffiliated with a crime laboratory. Although the report is available on the web, it has not yet been subjected to peer review and publication.

The scientific criteria for foundational validity require appropriately designed studies by more than one group to ensure reproducibility. Because there has been only a single appropriately designed study, the current evidence falls short of the scientific criteria for foundational validity. There is thus a need for additional, appropriately designed black-box studies to provide estimates of reliability.

The district court in Chester read this passage as meaning that the science is there, but that it would be nice to have a few more studies to show that other researchers can replicate the small error rates in the unpublished study. But is not PCAST really saying that there is a paucity of acceptable experiments from which to ascertain applicable error probabilities? In this regard, much more than "reliability" (in the statistical sense) and "reproducibility" of a number is at issue. One should ask not just whether a a second research group has replicated a given study, but more broadly, whether a solid body of studies with varied designs and different samples of examiners establish that the findings as a whole are robust and generalizable.

In contrast, the Chester court is satisfied with two studies that it understands to reveal false positive error probabilities in the neighborhood of 2%. Where PCAST is unable to perceive evidence of “validity” in the sense of reasonably well known error probabilities, the court finds the probability of error to be small enough to allow testimony as to the origin of the bullets.

But which probability does the court conclude is comfortingly small? The PCAST report defines a false-positive error probability one way. The Chester court expresses a different understanding of the meaning of this probability. Stay tuned for later discussion of this "false-positive error fallacy."

Notes

1. See Eric Lander, William Press, S. James Gates, Jr., Susan L. Graham, J. Michael McQuade, and Daniel Schrag, PCAST Releases Report on Forensic Science in Criminal Courts, Sept. 20, 2016, https://www.whitehouse.gov/blog/2016/09/20/pcast-releases-report-forensic-science-criminal-courts 
2. Seth Augerstein, Firearms Evidence Allowed in Chicago Hobos Gang Trial—Despite PCAST Argument, Forensic Mag., Oct. 13, 2016. http://www.forensicmag.com/news/2016/10/firearms-evidence-allowed-chicago-hobos-gang-trial-despite-pcast-argument 
3. See, e.g., David H. Kaye, David E. Bernstein & Jennifer L. Mnookin, The New Wigmore: A Treatise on Evidence: Expert Evidence (2d ed. 2011) (updated annually).
4. P. 106. The definition of "black-box study" at page 48, Box 2, is seriously incomplete. There, the report explains that "[b]y a 'black-box study,' we mean an empirical study that assesses a subjective method by having examiners analyze samples and render opinions about the origin or similarity of samples." But an "empirical" studies come in a multitude of designs. The PCAST authors have specific ideas about the design of empirical studies that they call "black-box studies."

More on the PCAST Report

• Index to this blog's comments and judicial cases discussing the PCAST report

Does log-LR separate Hillary Clinton from Donald Trump?

Many forensic statisticians regard likelihood ratios (or their logarithms) as a providing an ideal measure of the probative value of comparisons of patterns and other test results. Moving into another context, in medium.com "Data Scientist" Maixent Chenebaux describes differences between the acceptance speeches of Hillary Clinton and Donald Trump at their party's nominating convention:

we would like to find which are the candidates’ favorite words. To get the “Trumpian” and the “Clintonian” vocabularies, we have to find the words that occur the most in one candidate’s talk and, at the same time, the least in the opponent’s. For example, the word “really” is found 15 times in Trump’s speech but only once in Clinton’s. One way to determine this is to calculate the odds ratio for each word. The odds ratio (here named OR) was, for each word, computed using the following formula:

OR(word_i) = log { p_trump(word_i) / p_clinton(word_i) }
The first term of the ratio is the probability of a word being in Trump’s vocabulary, and the other one is the probability of the same word being in Clinton’s. The log function allows us to efficiently sort each word in one category or the other: when the probabilities are equal, the log function is null. In any other cases, it is either negative (a word is Clintonian) or positive (a word is Trumpian).

There is a simpler way to say this. If a word is used proportionally more often in Trump's speech than in Clinton's, it is Trumpian; if it occurs proportionally more in Clinton's speech, it is Clintonian.

The "odds ratio" in the box is nit really an odds ratio. It is the logarithm of a Bayes factor (the ratio of posterior to prior odds) or the log of the likelihood ratio. But seems like extra baggage. We are not trying to deduce the author of the speech from the frequencies of the distinctive words. We are only trying to pick out the words that are distinctive. Their usefulness in classifying additional speech samples by author would require further research. If we had a transcript of a speech that we knew was either Clinton's or Trump's--and had to guess the speaker without knowing anything about politics and semantics--then the discriminators obtained from this study might be useful. We could claim scientific validity in using them to discriminate between the two possible authors if these terms had been shown to be sensitive and specific (and, hence, to have high likelihood ratios) when cross-validated on other speeches.

In any event, the words that emerged as discriminators for the nomination speeches and that were "almost exclusively" by that candidate (p_{other candidate}(word) ≈ 0 ?) were

Clinton	Trump
America working hard together Donald millions campaign enough wants	great got China really Mexico nice problem(s) Iran disaster

One proposed interpretation is that Trump uses shorter words, but that is not obvious from this list. Another is

Fine observers will note that “Trump” does not appear in the Clintonian wordset above, the reason being that Trump himself made numerous mentions of his last name in his speech (10 times), bringing the ratio way down. As a point of comparison, Clinton’s name is only used twice: once in Hillary’s speech (about her husband Bill Clinton), and once in Trump’s. Moreover, the Clintonian word “Wants” that shows up in the list is mostly used to criticize her opponent (“He wants to divide us […]”, “He wants us to fear the future and fear each other.”). It clearly shows that Clinton talked about Trump, and Trump talked about … himself!

A word whose LR was about 1 is "thanks." It does not work well as a classifier. But here semantics, not statistics, indicates a difference between the two candidates:

They both use “thank(s)” numerous times, but in a different manner: while Clinton specifically thanked a group of people or an individual [e.g. "I want to thank Bernie Sanders"] Trump’s “thanks” were mostly employed when the crowd was applauding him [e.g., "That's really nice, thanks"].

Returning to statistics alone,

Most of Trump’s sentences are short: more than 21% of Trump’s speech is made of sentences that contain 5 or 6 words. Clinton’s sentence lengths are more evenly distributed, 12-word sentences being the most frequent. ... Obama, during his first nomination speech, employed an average of 25.7 words per sentence, which is almost equal to Clinton and Trump combined. Obama also repeated himself 24% less than Clinton, and 42% less than Trump.

That's it for now! I would not want to repeat myself.

Reference

Maixent Chenebaux, Semantics — What Does Data Science Reveal about Clinton and Trump?, Oct. 5, 2016, https://medium.com/reputation-squad/semantics-what-does-data-science-reveal-about-clinton-nd-trump-afdf427e833b#.ctkkoyu2o

The PCAST Report and Argumentum Ad Hominem

I suppose it was inevitable. If you don't like the message, just dismiss the messengers as ill-informed or biased. Toss in a few simplistic generalities, and some praise for the people who agree with you, and, voila, you can avoid trying to analyze of the substantive basis for the message. This dynamic is at work in the reactions to the report of the President's Council of Advisors on Science and Technology (PCAST) on forensic science. I'll give two examples — one from a pro-report commentator, and another from an anti-report group.

On the pro-report side, we have Randy Balko writing on the Washington Post website that the PCAST report should be seen as scientifically correct because it is written by "people such as Eric Lander, a geneticist, biologist and mathematician at the Massachusetts Institute of Technology; Sylvester James Gates Jr., who studies superstring theory and particle physics at the University of Maryland; Susan Graham, acclaimed computer scientist at the University of California at Berkeley; and William H. Press, an astrophysicist, theoretical physicist, computer scientist and computational biologist at the University of Texas." 1/ In contrast, the prosecutors who attacked PCAST's assessments of the scientific literature in a press release a few weeks ago are "district attorneys — people who work in an adversarial profession; who have achieved great success in a field where rhetoric and persuasion are just as if not more important than facts (remember, most DAs are elected) [and their] ability to win convictions stands to get much more difficult should these disciplines be barred from the courtroom."

Now this is not a fallacious argumentum ad hominem. Everything else being equal, it is rational to believe scientists' views about science over lawyers'. But the acclaim that individuals have achieved in other fields of science is not a sure guide to the validity of their judgments of the scientific foundations of various techniques and theories. William Shockley's outstanding work on electrons and holes in semiconductors did not lend substantive weight to his arguments on intelligence, heredity, and race. The contribution of biochemist Kary Mullis — another Nobel Laureate — to PCR amplification reveals little about the truth of his views on the scientific foundations of climate change, ozone depletion, and the cause of AIDS. 2/

This is not to say that the reputation of a scientist has nothing to do with the reception of his or her ideas. Neither is it a comparison of any one scientist to another or a claim that the PCAST report is mistaken. It is simply a statement that the authority proposing an idea or making an argument matters more to lawyers than it does (or, more precisely, than it should) to scientists. In science, "arguments from authority are worthless," or so it has been said. 3/ Given this Galilean ideal, 4/ one would hope that the responses to the PCAST report from forensic scientists either would concur with PCAST's observations or reject them in the light of a more thorough or more perceptive  study of the foundational literature.

So far, I have not seen such responses — although it is way too early to expect to find them. Instead, one group of forensic scientists have spoken out against the report by taking the unscientific tack of ad hominem argumentation. At least, this is a prominent feature of the Position Statement of the American Congress of Forensic Science Laboratories. 5/ These forensic scientists had this to say about the report:

Unfortunately, it was born of an imbalanced and inexperienced working group whose make-up included no forensic practitioners nor any other professionals with demonstrated experience in the practice of forensic science. The Chair of the aforementioned working group, Eric Lander, sits on the Board of Directors of the Innocence Project, a legal-activism group that has itself been publicly criticized on numerous occasions (including within peer reviewed literature) for the unfairness of its public statements and the conflicts of interest that have long called into question its motives. In addition, the working group’s writer, Tania Simoncelli,4 has publicly authored previous opinions that DNA database collections violate civil liberties.5

The argument in the first sentence that a competent literature review cannot be written by a group of scientists that does not include "forensic practitioners" or "professionals [who] practice ... forensic science" is the mark of an insular guild that resists scientific norms. Anyone skilled in empirical research methods and statistics should be able to survey and comment on the state of the science. "Just as an epidemiologist who does not treat patients may be qualified to testify to the state of the scientific proof of a chemical’s toxicity, a scientist with expertise in research methods and statistics need not be an expert in the details of toolmark examinations to determine whether the scientific literature supports a practitioner’s professed abilities." 6/

The remainder of the polemic "is not [there] to disparage any individuals or their motives," but it is hard to see how else it is relevant. I have disagreed with some of the legal, scientific, and policy positions espoused by Tania Simoncelli, 7/ but a person's views on controversial issues such as the constitutionality of DNA sampling on arrest — a procedure that four U.S. Supreme Court Justices insisted was unconstitutional — hardly makes someone unsuitable as a technical writer. Finally, the idea that a co-chair of the PCAST working group is tainted by an association with the Innocence Project — an advocacy group that itself relies on forensic science — is about as far from a substantive critique as one can get.

Forensic scientists who want to show that parts of the PCAST report are flawed will have to resist the temptation to dismiss the report as "a political phenomenon" born of "ignoran[ce] of the ugly realities associated with solving crimes like murder and rape as quickly and accurately as possible." 8/ They will have to respond in a scientifically defensible manner.

Notes

1. Randy Balko, White House Science Council: Bite-mark Matching Is Junk Science, The Watch, Sept. 7, 2016, https://www.washingtonpost.com/news/the-watch/wp/2016/09/07/white-house-science-council-bite-mark-matching-is-junk-science/?utm_term=.dbfc1848fa3e
2. Even within the fields in which a given scientist has made important contributions, examples of erroneous judgments from that scientist are commonplace. Arguably, Eric Lander's conclusions about "spectacular deviations from Hardy-Weinberg, indicating the presence of genetically distinct subgroups" in People v. Castro, an early DNA-evidence case noted in the PCAST report, falls into this category. See David H. Kaye, The Double Helix and the Law of Evidence 115-20 (2010). Even Albert Einstein made his share of mistakes in physics and mathematics. Hans C. Ohanian, Einstein's Mistakes: The Human Failings of Genius (2008).
3. Carl Sagan, Cosmos 277 (1985).
4. "In questions of science, the authority of a thousand is not worth the humble reasoning of a single individual." Galileo Galilei, as quoted in François Arago, Biographies of Distinguished Scientific Men 365 (1859).
5. American Congress of Forensic Science Laboratories, Position Statement, Sept. 21, 2016, http://www.fslabs.org/resources/Documents_OPC/2016_0921_PS_PCAST.pdf
6. David H. Kaye et al., The New Wigmore on Evidence: Expert Evidence § 373 (Cum. Suppl. 2016). The better reasoned judicial opinions on the qualifications of expert witnesses to testify to their reviews adopt this approach. E.g., State v. Romero, 365 P.3d 358, 362 (Ariz. 2016).
7. See, e.g., Hot Off the Presses: Chimeric Criminals, Forensic Sci., Stat. & L., Mar. 23, 2013, http://for-sci-law.blogspot.com/2013/03/hot-off-presses-chimeric-criminals.html; "Genetic Justice": The Potential and the Real, Forensic Sci., Stat. & L., June 6, 2011, http://for-sci-law.blogspot.com/2011/06/genetic-justice-potential-and-real.html.
8. Position Statement, supra note 5.

Index to comments and cases on the PCAST report