State v. Ghigliotti, Computer-assisted Bullet Matching, and the ASB Standards

In State v. Ghigliotti, 232 A.3d 468, 471 (N.J. App. Div. 2020), a firearms examiner concluded that a particular gun did not fire the bullet (or, more precisely, a bullet jacket) removed from the body of a man found shot to death by the side of a road in Union County, New Jersey. That was 2005, and the case went nowhere.

Ten years later, a detective prevailed on a second firearms examiner to see what he thought of the toolmark evidence. After considerable effort, this examiner reported that the microscopic comparisons with many test bullets from the gun in question were inconclusive.

However, at a training seminar in New Orleans he learned of two tools developed and marketed by Ultra Electronics Forensic Technology, the creator of the Integrated Ballistics Identification System (IBIS) that "can find the 'needle in the haystack', suggesting possible matches between pairs of spent bullets and cartridge cases, at speeds well beyond human capacity. The Bullettrax system “digitally captures the surface of a bullet in 2D and 3D, providing a topographic model of the marks around its circumference.” As “[t]he world’s most advanced bullet acquisition station” it uses “intelligent surface tracking that automatically adapts to deformations of damaged and fragmented bullets.”

The complementary Matchpoint is an “analysis station” with “[p]owerful visualization tools [that] go beyond conventional comparison microscopes to ease the recognition of high-confidence matches. Indeed, Matchpoint increases identification success rates while reducing efforts required for ultimate confirmations.” It features multiple side-by-side view of images from the Bullettrax data and score analysis. The court explained that “the Matchpoint software ... included tools for flattening and manipulating the images, adjusting the brightness, zooming in, and ‘different overlays of ... color scaling.’”

But the examiner did not make the comparisons based on the digitally generated and enhanced images, and he did not rely on any similarity-score analysis. Rather, he “looked at the images side-by-side on a computer screen using Matchpoint [only] ‘to try and target areas of interest to determine ... if (he) was going to go back and continue with further [conventional] microscopic comparisons or not.’” He found four such areas of agreement. Conducting a new microscopic analysis of these and other areas a few weeks later, he “‘came to an opinion of an identification or a positive identification’ ... grounded in his ‘training and experience and education as a practitioner in firearms identification’ and his handling of over 2300 cases.” 232 A.3d at 478–49.

The trial court “determined that a Frye hearing was necessary to demonstrate the reliability of the computer images of the bullets produced by BULLETTRAX before the expert would be permitted to testify at trial.” Id. at 471, The state filed an interlocutory appeal, arguing that the positive identification did not depend on Ultra’s products. The Appellate Division affirmed, holding that the hearing should proceed.

I do not know where the case stands, but its facts provide the basis for a thought experiment. At about the same time as the Ghigliotto court affirmed the order for a hearing, the American Academy of Forensic Sciences Standards Board (ASB) published a package of standards on toolmark comparisons. Created in 2015, ASB describes itself as “an ANSI [American National Standards Institute]-accredited Standards Developing Organization with the purpose of providing accessible, high quality science-based consensus forensic standards.” Academy Standards Board, Who We Are, 2022. Two of its standards concern three-dimensional (3D) data and inferences in toolmark comparisons, while the third is specific to software for comparing 2D or 3D data.

We can put the third to the side, for it is limited to software that "seeks to assess both the level of geometric similarity (similarity of toolmarks) and the degree of certainty that the observed similarity results from a common origin." ANSI/ASB Standard 062, Standard for Topography Comparison Software for Toolmark Analysis § 3.1 (2021). The data collection and visualization software here does neither, and the scoring feature of Matchpoint was not used.

ANSI/ASB Standard 061, Firearms and Toolmarks 3D Measurement Systems and Measurement Quality Control (2021), is more apposite although it is only intended “to ensure the instrument’s accuracy, to conduct instrument calibration, and to estimate measurement uncertainty for each axis (X, Y, and Z).” It promises “procedures for validation of 3D system hardware” but not software. It “does not apply to legacy 2D type systems,” leaving one to wonder whether there are any standards for validating them.

Even for "3D system hardware," the procedure for “developmental validity” (§ 4.1) is nonexistent. There are no criteria in this standard for recognizing when a measurement system is valid and no steps that a researcher must follow to study validity. Instead, the section on “Developmental Validation (Mandatory)” states that an “organization with appropriate knowledge and/or [sic] expertise” shall complete “a developmental validation”; that this validation “typically” (but not necessarily) consists of library research (“identifying and citing previously published scientific literature”); and that “ample”—but entirely uncited— literature exists “to establish the underlying imaging technology” for seven enumerated technologies. In full, the three sentences on “developmental validation” are

As per ANSI/ASB Standard 063, Implementation of 3D Technologies in Forensic Firearm and Toolmark Comparison Laboratories, a developmental validation shall be completed by at least one organization with appropriate knowledge and/or expertise. The developmental validation of imaging hardware typically consists of identifying and citing previously published scientific literature establishing the underlying imaging technology. The methods defined above of coherence scanning interferometry, confocal microscopy, confocal chromatic microscopy, focus variation microscopy, phase-shifting interferometric microscopy, photometric stereo, and structured light projection all have ample published scientific literature which can be cited to establish an underlying imaging technology.

Perhaps the section is merely there to point the reader to the different standard, ASB 063, on implementation of 3D technologies. \1/ But that standard seems to conceive of “developmental validation” as a process that occurs in a forensic laboratory or other organization by a predefined process with a “technical reviewer” to sign off on the resulting document that becomes the object of further review through “[p]eer-reviewed publication (or other means of dissemination to the scientific community, such as a peer-reviewed presentation at a scientific meeting).” § 4.1.3.4. The data and the statistics needed to assess measurement validity are left to the readers' imaginations (or statistical acumen). \2/

ASB 061 devotes more attention to what it calls “deployment validation” on the part of every laboratory that chooses to use a 3D measuring instrument. This part of the standard describes some procedures for checking whether X, Y, and Z “scales” that should reveal whether measurements of the coordinates of points on the surface of the material are close to what they should be. For example, § 4.2.5.4.1 specifies that

Using calibrated geometric standards (e.g., sine wave, pitch, step heights), measurements shall be conducted to check the X and Y lateral scales as well as the vertical Z scale. Ten measurements shall be performed consecutively ... . The measurement uncertainty of the repeatability measurements shall overlap with the certified value and uncertainty of the geometric standard used.

The phrasing is confusing (to me, at least). I assume that a “geometric standard” is the equivalent of a ruler of known length (a “certified value” of, say, 1 ± 0.01 microns). But what does the edict that “[t]he measurement uncertainty of the repeatability measurements shall overlap with the certified value and uncertainty of the geometric standard used” mean operationally?

The best answer I can think of is that the standard contemplates comparing two intervals. One is the scale value (along, say, the X-axis). Imagine that the “geometric standard” that is taken to be the truth is certified as having a length of 1± 0.01 microns. Let’s call this the “certified standard interval.”

Now the laboratory makes ten measurements for its “deployment validation” to produce what we can call a “sample interval” from the ten measurements. The ASB standard does not contain any directions on how this is to be done. One approach would be to compute a confidence interval on the assumption that the sample measurements are normally distributed. Suppose the observed sample mean for them is 0.80, and the standard error computed from the ten sample measurements is s = 0.10 microns. The confidence interval is then 0.80 ± k(0.10), where is k is some constant. If the confidence interval includes any part of the certified interval, this part of the deployment-validation requirement is met.

What values of k would be suitable for the instrument to be regarded as “deploymentally valid”? The standard is devoid of any insight into this critical value and its relationship to confidence. It does not explain what the interval-overlap requirement is supposed to accomplish, but if the confidence interval is part of it, it is an ad hoc form of hypothesis testing with an unstated significance level.

Is the question of whether the hypothesis that there is no difference between the standard reference value of 1 and the true sample mean can be rejected at some preset significance level all that important here? Should not the question be how much the disparities between the sample of ten measured values and the geometric-standard value would affect the efficacy of the measurements? An observed sample mean that is 20% too low does not lead to the rejection of the hypothesis that the instrument’s measurements are, in the long run, exactly correct, but with only ten measurements in the sample, that may tell us more about the lack of the statistical power of the test than about the ability of the instrumentation to measure what it seeks to measure with suitable accuracy for the applications to which it is put.

In sum, the standard’s section on “Developmental Validation (Mandatory)” mandates nothing that is not trivially obvious—the court already knows that it should look for support for the 3D scanning and image-manipulation methods in the scientific literature, and the standard does not reveal what the substance of this validation should be. “Deployment Validation (Mandatory)” is supposed to ensure that the laboratory is properly prepared to use a previously validated system for casework. It is of little use in a hearing on the general acceptance of the scanning system and the theories behind it. (One could argue that scientists would accept a system that a laboratory has been rigorously pretested and shown to be perform accurately, even with no other validation, but it is not clear that the standard describes an appropriate, rigorous pretesting procedure.)

Moreover, the standard explicitly excludes software from its reach, making it inapplicable to the Matchpoint image-manipulation tools the helped the examiner in Ghigliotti zero in on the regions that altered his opinion. The companion standard on software does not fill this gap, for it deals only with software that produces similarity scores or random-match probabilities. Finally, ASB 063's substantive requirements for "deployment validation" prior to laboratory implementation might well prohibit an examiner from going to the developer of hardware and software not yet adopted by his or her employer for help with locating features for further visual analysis, as occurred in Ghigliotti. But that is not responsive to the legal question of whether the developer's system is generally accepted as valid in the scientific community.

NOTES

1. ANSI/ASB 063 is even more devoid of references. The entire bibliography consists of a webpage entitled “control chart.” There, attorneys, courts, or experts seeking to use the standard will discover that a “control chart is a graph used to study how a process changes over time.” That is great for quality control of instrumentation, but it is irrelevant to validation.
2. Under § 4.1.2.4, "The plan for developmental validation study shall include the following:
   "a) the limitations of the procedure;
   "b) the conditions under which reliable results can be obtained;
   "c) critical aspects of the procedure that shall be controlled and monitored;
   "d) the ability of the resulting procedure to meet the needs of the given application."

Last updated: 12 June 2022

The New York Court of Appeals Returns to Probabilistic Genotyping Software (Part II—General Acceptance)

The New York Court of Appeals returned to the contentious issue of “probabilistic genotyping software” (PGS) in People v. Wakefield, 2022 N.Y. Slip Op. 02771, 2022 WL 1217463 (N.Y. Apr. 26, 2022). As previously discussed, in People v. Williams, 147 N.E.3d 1131 (N.Y. 2020), a slim majority of the court held that the output of a computer program should not have been admitted without a full evidentiary hearing on its general acceptance within the scientific community. The majority opinion described a confluence of considerations:

1. The program had only been tested in the laboratory that developed it (“an invitation to bias,” id. at 1141);
2. The only evidentiary hearing ever conducted on the program had only shown “internal validation” and formal approval by a subcommittee of a state forensic science commission that was a “narrow class of reviewers, some of whom were employed by the very agency that developed the technology,” id. at 1142;
3. Given “the ‘black box’ nature of that program,” the developer's “secretive approach ... was inconsistent with quality assurance standards” id.; and
4. Submissions for hearings in other cases “suggested that the accuracy calculations of that program may be flawed,” id.

But which of these four factors were dispositive? Was it the combination of all four, or something in between, that rendered the evidence inadmissible? If the developer were to change its “secretive approach” so as to allow defense experts to study the program’s source code, would that, plus the “internal validation,” be enough to establish general scientific acceptance? Would it be sufficient for the state to refute the suggestions of flawed “accuracy calculations of the program” through testimony from its experts? Just what did the court mean when it summarized its analysis with the statement that “[i]n short, the [PGS] should be supported by those with no professional interest in its acceptance. Frye demands an objective, unbiased review”?

The opinion did not reveal how the majority might answer these questions. Of course, in holding that a hearing was necessary, the Williams majority implied that some information outside of the normal scientific literature could fill the gap created by the absence of replicated developmental validation studies from external (“objective, unbiased”) researchers. But what might that information be?

The court’s encounter with PGS last month did not answer this open question, for the court in Wakefield found that there were replicated studies from the developer of a more sophisticated computer program and other researchers. In addition, it pointed to other evaluations or uses of the program. The totality of the evidence, it reasoned, was stronger than the developer-only record in Williams and demonstrated the requisite general acceptance. But the opinion provoked one member of the court to complain of a "jarring turnabout" from "the same view unsuccessfully advocated by a minority in Williams two years ago."

This posting describes the case, the DNA evidence, and aspects of the discussions of general acceptance that struck me as interesting or puzzling.

The Crime, the Samples, and Some Misunderstood Probabilities of Exclusion

In 2010, John Wakefield strangled the occupant of an apartment with a guitar amplifier cord and made off with various items. The New York State Police laboratory analyzed samples from four areas: the front part of the collar of the victim's shirt, the rear part of the collar; the victim's forearm; and the amplifier cord. The laboratory concluded that the DNA on the collar was “consistent with at least two donors, one of which was the victim, and defendant could not be excluded as the other contributor”; that the DNA from the forearm “was consistent with DNA from the victim, as the major contributor, mixed with at least two additional donors” and the DNA on the cord was “a mixture of at least two donors, from which the victim could not be excluded as a possible contributor.” 2022 WL 1217463, at *1.

At this point, the court’s description of the State Police laboratory’s work becomes had to follow. The court wrote that:

[T]he analyst did not call any alleles based on peaks on the electropherogram below [the pre-established stochastic] threshold. As a result, there was insufficient data to allow the Lab to calculate probabilities for the unknown contributors to the DNA mixtures found on the amplifier cord and the front of the shirt collar.

No alleles at all? It takes only one allele to compute a probability of exclusion, although with such a limited profile, the exclusion probability might be close to zero, meaning that the data are uninformative. In any event, for the other two samples, “[t]he Lab was able to call ... 4 ... STR loci” that enabled “the analyst, using the combined probability of inclusion method, [to opine that] the probability an unrelated individual contributed DNA to the outside rear shirt collar was 1 in 1,088" and “that the probability an unrelated individual contributed DNA ... was 1 in 422" for “the profile obtained from the victim's forearm.”

Or so the court said. As explained in Box 1, these numbers are not “the probability an unrelated individual contributed DNA.” They are estimates of the probability that a randomly selected, unrelated individual could not be excluded as a possible source. Given a large number of unrelated individuals in the region, there easily could be more than a hundred people with STR profiles compatible with the mixtures.

BOX 1: TRANSPOSITION

The probability of inclusion is not the probability that an included individual is the contributor. It is the probability of not excluding an individual as a possible contributor. That probability is not necessarily equal to the probability that an included individual actually contributed to the sample from which he or she could not be excluded. If C stands for contributor and I for included, the probability of inclusion for any randomly selected individual can be written P(I given C). The source probability for the individual is different. It is P(C given I). Equating the two is known as the transposition fallacy (or the “prosecutor’s fallacy,” though it could be called the “judges” fallacy as well).

We do not need any symbols to see that the two conditional probabilities are not necessarily equal. The population of Schnectady county, where the crime occurred, was about 155,000 in 2010. Let’s round down to 150,000. That ought to remove all of Wakefield’s relatives. Excluding all but 1 in 1,088 individuals would leave 138 people as possible perpetrators. Of course, some would be far more plausible suspects than others, but based on the DNA evidence alone, how can the court claim that “the probability an unrelated individual contributed DNA to the outside rear shirt collar was 1 in 1,088”? That probability cannot be determined from the DNA evidence alone. It can be computed only if we are willing to assign a “prior probability” of being the murderer to each of the unrelated individuals in Schnectady (or anywhere else).

Suppose we assume that, ab initio, everyone in the county has an equal probability of being a source of the DNA on the collar. At that point, Wakefield’s probability is quite small. It is 1/150,000. Since the DNA testing would have excluded all but some 138 people, and because Wakefield is one of them, the probability attached to him is larger. Now the probability is 1/138. But that still leaves the vast bulk of the probability with the 137 unrelated individuals. Instead of transposing, we should say that “the probability an unrelated individual contributed DNA to the outside rear shirt collar was 137 out of in 138” rather than the court’s “1 in 1,088.” Of course, our assumption of equal probabilities for every unrelated individual is unrealistic, but that does not impeach the broader point that the mathematics does not make the probability of an unrelated individual the number that the court supplied.

Cybergenetics to the Rescue

To secure a better and more complete analysis, “the electronic data from the DNA testing of the four samples at issue was then sent to Cybergenetics [for] calculating a likelihood ratio—using all of the information generated on the electropherogram, including peaks that fall below a laboratory's stochastic threshold.” Cybergenetics is a private company whose “flagship TrueAllele® technology resolves complex forensic evidence, providing accurate and objective DNA match statistics.” TrueAllele's calculations of the likelihood ratios, using the hypothesis that the four samples contained DNA from an unrelated black individual as the alternative to the hypothesis that Wakefield’s DNA was present were 5.88 billion for the cord, 170 quintillion for the outside rear shirt collar, 303 billion for the outside front shirt collar, and 56.1 million for the forearm.

Wakefield moved to exclude these findings, The Schnectady County Supreme Court held a pretrial evidentiary hearing “over numerous days.” People v. Wakefield, 47 Misc.3d 850, 851, 9 N.Y.S.3d 540 (2015). (New York calls its trial courts supreme courts.) Finding “that Cybergenetics TrueAllele Casework is not novel but instead is ‘generally accepted’ under the Frye standard,” \1/ Justice Michael V. Coccoma (New York calls its trial judges justices) denied the motion. 47 Misc.3d at 859. A jury convicted Wakefield of first degree murder and robbery. The Appellate Division affirmed, and seven years after the trial, so did the Court of Appeals (New York calls its most supreme court the Court of Appeals).

Changes in New York’s Highest Court

Back in Williams, the Court of Appeals judges had split 4-3 on whether New York City's home-grown PGS had attained general acceptance. The three judges led by Chief Judge Janet M. DiFiore * objected to the majority’s negative comments about PGS and propounded a narrower rationale for requiring a Frye hearing. But even if one could have confidently applied the majority reasoning in Williams to the scientific status of TrueAllele in Wakefield, the exercise in legal logic might have been futile. In the two short years since Williams, the composition of the court had changed. One concurring judge died, and the majority-opinion bloc lost half its members, including the opinion’s author, to retirements. The reconstituted court gave Chief Judge DiFiore the opportunity to write a more laudatory opinion for a new and larger majority.

Only one judge stood apart from this new majority. Having been in the majority in Williams, Judge Jenny Rivera now found herself in the Chief Judge’s situation in Williams, composing a dissenting opinion with respect to the reasoning on general acceptance but concurring in the result. Drawing on Williams, Judge Rivera maintained that “the court erred in admitting the TrueAllele results but the error ... was harmless” in view of the other evidence of guilt.

The Court’s Understanding of TrueAllele

The opinions are vague about the inner workings of TrueAllele. The majority opinion suggests that what is distinctive about PGS is that it cranks out a likelihood ratio. \2/ But “likelihood ratio,” for present purposes, simply denotes the probability of data given one hypothesis divided by the probability of the same data given a (simple) alternative hypothesis. It has nothing to do with the probabilistic part of TrueAllelle. Indeed, TrueAllele only computes a likelihood ratio after the probability analysis is completed. It does this by dividing (i) the final posterior odds that favor one source hypothesis as compared to another by (ii) the initial prior odds. This division gives a “Bayes' factor” that states how much the data have changed the odds.

Let me try saying this another way. In effect, TrueAllele starts with prior odds based solely on the frequencies of various DNA alleles (and hence genotypes) in some population, performs successive approximations to converge on a better estimate of the odds, and divides the adjusted odds by the prior odds to yield what Cybergenetics calls “the match statistic.” If all goes well, this quotient (call it a likelihood ratio, a Bayes' factor, a match statistic, or whatever you want) reveals how powerful the DNA evidence is (which is not necessarily the same as the odds that any hypothesis is true). At least, that is what I think goes on. The court contents itself with warm and fuzzy statements such as “a probability model to assess the values of a genotype objectively” “based on mathematical computations from all the data in the electropherograms.” and “separates the genotypes using the mathematical probability principle of the Markov chain Monte Carlo (MCMC) search to calculate the probability for what the different genotypes could be.” (This last clause may not be so warm and fuzzy; it begins to unpack what I simplistically called successive approximations.)

The Timing for General Acceptance

Wakefield is a backwards-looking case. The main question before the Court of Appeals was whether, in 2015, TrueAllele reasonably could have been deemed to have been generally accepted in the scientific community. That is what New York law requires. \3/ The Chief Judge’s analysis of the general acceptance of TrueAllele starts with the observation that “[t]he well-known Frye test applied to the admissibility of novel scientific evidence (Frye v. United States, 293 F. 1013 [D.C. Cir.1923]) is 'whether the accepted techniques, when properly performed, generate results accepted as reliable within the scientific community generally' (People v. Wesley, 83 N.Y.2d 417, 422, 611 N.Y.S.2d 97, 633 N.E.2d 451 [1994]).”

Wesley is an interesting case to cite here. One would not know from the citation or the analysis in Wakefield that in Wesley there was no opinion for a majority of the seven judges on the court. There was one opinion for three judges and another opinion for two judges concurring only in the result. The remaining two judges did not participate. The concurring opinion was written by the late Chief Judge Judith S. Kaye, the longest-serving chief judge in New York history.

Chief Judge Kaye’s concurrence is memorable for its skepticism about finding general acceptance on the basis of studies from the developer of a method. Current Chief Judge Janet DiFiore briefly summarized that discussion (as did the majority in Williams). A more complete exposition is in Box 2. Chief Judge DiFiore then suggests that the Wesley concurrence was satisfied because “[n]otwithstanding these concerns, Chief Judge Kaye ultimately agreed that, at the time the appeal was decided, "RFLP-based forensic analysis [was] generally accepted as reliable" and those testing procedures were accepted as the standard methodology used in the scientific community until the advent of the PCR STR method used today.”

This presentation places an odd spin on the Wesley concurrence. The sole basis for the concurrence was that “it can fairly be said that use of DNA evidence was harmless beyond a reasonable doubt” because the DNA evidence “added nothing to the People's case.” 83 N.Y.2d at 444–45. The observations that five years after the hearing in Wesley, it had become clear that “in principle” RFLP-VNTR testing was “fundamentally sound” and was generally accepted were clearly dicta. Chief Judge Kaye was not suggesting that because a method had become generally accepted later, its earlier admission was vindicated. The dicta on later general acceptance was intended to inform trial courts that while they were at liberty to admit RFLP-VNTR evidence without pretrial hearings on general acceptance, they still needed to probe “the adequacy of the methods used to acquire and analyze samples ... case by case.” Id. at 445.

In contrast to Wesley, which emphasized the state of the science “at the time of the Frye hearing in 1988,” 83 N.Y.2d at 425 (plurality opinion), and whether “in 1988, ... there was consensus,” id. at 439 (concurring opinion), Chief Judge DiFiore’s opinion is less precise on when general acceptance came into existence:

BOX 2. PEOPLE v. WESLEY 83 N.Y.2d 417, 439–41, 611 N.Y.S.2d 97, 633 N.E.2d 451 (N.Y. 1994) (Chief Judge Kaye, concurring) (citations and footnote omitted)

The inquiry into forensic analysis of DNA in this case also demonstrates the "pitfalls of self-validation by a small group" Before bringing novel evidence to court, proponents of new techniques must subject their methods to the scrutiny of fellow scientists, unimpeded by commercial concerns.

A Frye court should be particularly cautious when — as here — "the supporting research is conducted by someone with a professional or commercial interest in the technique" DNA forensic analysis was developed in commercial laboratories under conditions of secrecy, preventing emergence of independent views. No independent academic or governmental laboratories were publishing studies concerning forensic use of DNA profiling. The Federal Bureau of Investigation did not consider use of the technique until 1989. Because no other facilities were apparently conducting research in the field, the commercial laboratory's unchallenged endorsement of the reliability of its own techniques was accepted by the hearing court as sufficient to represent acceptance of the technique by scientists generally. The sole forensic witness at the hearing in this case was Dr. Michael Baird, Director of Forensics at Lifecodes laboratory, where the samples were to be analyzed. While he assured the court of the reliability of the forensic application of DNA, virtually the sole publications on forensic use of DNA were his own or those of Dr. Jeffreys, the founder of Cellmark, one of Lifecodes' competitors. Nor had the forensic procedure been subjected to thorough peer review. ***

The opinions of two scientists, both with commercial interests in the work under consideration and both the primary developers and proponents of the technique, were insufficient to establish "general acceptance" in the scientific field. The People's effort to gain a consensus by having their own witnesses "peer review" the relevant studies in time to return to court with supporting testimony was hardly an appropriate substitute for the thoughtful exchange of ideas in an unbiased scientific community envisioned by Frye. Our colleagues' characterization of a dearth of publications on this novel technique as the equivalent of unanimous endorsement of its reliability ignores the plain reality that this technique was not yet being discussed and tested in the scientific community.

"Although the continuous probabilistic approach was not used in the majority of forensic crime laboratories at the time of the hearing, the methodology has been generally accepted in the relevant scientific community based on the empirical evidence of its validity, as demonstrated by multiple validation studies, including collaborative studies, peer-reviewed publications in scientific journals and its use in other jurisdictions. The empirical studies demonstrated TrueAllele's reliability, by deriving reproducible and accurate results from the interpretation of known DNA samples."

Presumably, and notwithstanding citations to materials appearing after 2015, \4/ she meant to write that the methodology had been generally accepted in 2015 because the indications listed were present then. (Whether the decisive time for general acceptance should be that of the trial rather than the appeal is not completely obvious. If a technique becomes generally accepted later, why should the defendant be entitled to a new trial in which the evidence that should have been excluded has become admissible anyway? The defendant's interest in the time-of-trial rule is the interest in not being convicted with the help of scientifically sound evidence (as per the general-acceptance standard based on the best current knowledge). A counter-argument is that a large pool of potential defense experts to question the application of the general accepted method in the particular case did not exist at the time of trial because the evidence was too novel.)

Quantifying the Accuracy of PGS

Turning to the question of the state of acceptance as of 2015, the majority opinion maintains that

]T]he methodology has been generally accepted in the relevant scientific community based on the empirical evidence of its validity, as demonstrated by multiple validation studies, including collaborative studies, peer-reviewed publications in scientific journals and its use in other jurisdictions. The empirical studies demonstrated TrueAllele's reliability, by deriving reproducible and accurate results from the interpretation of known DNA samples.

Both the fact that the software was written to implement uncontroversial mathematical ideas and the published empirical evidence are important. If the software were designed to implement a mathematically invalid procedure, the game would be over before it began. But techniques such as Bayes’ rule and sampling methods for getting a representative picture of the posterior distribution only work when they are developed appropriately for a particular application. Acknowledging that these tools have been used to solve problems in many fields of science is a bit like saying that the mathematics of probability theory is undisputed. The validity of the mathematical ideas are a necessary but hardly a sufficient condition for a finding that software intended to apply the ideas functions as intended. Using a particular mathematical formula or method to describe or predict real-world phenomena is an endeavor that is subject to and in need of empirical confirmation. Because PGS models the variability in the empirical data that emerge from chemical reactions and electronic detectors, “empirical evidence ... of its accuracy” is indispensable to establishing its accuracy.

Unfortunately, Wakefield is short on details from the “multiple validation studies” and “peer-reviewed publications.” What do the studies and publications reveal about the accuracy of output such as “5.88 billion times more probable” and “170 quintillion times more probable”? The Supreme Court opinion is devoid of any quantitative statement of how well the deconvoluted individual profiles and their Bayes’ factors reported by TrueAllele correspond to the presence or absence of those profiles in samples constructed with or otherwise known to contain DNA from given individuals. So is the Appellate Division opinion. So too with the Court of Appeals’ opinions. The court is persuaded that “[t]he empirical studies demonstrated TrueAllele's reliability, by deriving reproducible and accurate results from the interpretation of known DNA samples.” But how well did True Allele perform in the “many published and peer reviewed” validity studies?

A separate posting summarizes parts of the six studies circa 2015 that are both published and peer reviewed. The numbers in these studies suggest that within certain ranges (with regard to the quantity of DNA, the number of contributors, and the fractions from the multiple contributors), TrueAllele’s likelihood ratios discriminate quite well between samples paired with true contributors and the same samples paired with noncontributors. For example, in one experiment, LR was never greater than 1 for 600,000 simulations of false contributors to 10 two-person mixtures containing 1 nanogram of DNA—no observed false positives! Conversely, LR was never less than 1 for every true contributor to the same ten mixtures—no observed false negatives in 20 comparisons. Moreover, the program’s output behaves qualitatively as it should, generally producing smaller likelihood ratios for electrophoretic data that are more complex or more bedeviled by stochastic effects on peak heights and locations.

Such results suggest that TrueAllele’s LRs are in the ballpark. Yet, it is hard to gauge the size of the ballpark. Is a computed LR of 5.88 billion truly a probability ratio of 5.88 billion? Could the ratio be a lot less or a lot more? The validity studies do not give quantitative answers to these questions about “accuracy.” \5/

The Developer’s Involvement

On appeal, Wakefield had to convince the court that the unchallenged studies and other indicia of general acceptance were too weak to permit a finding of general acceptance. To do so, he pointed to “the dearth of independent validation as a result of Dr. Perlin's involvement in the large majority of studies produced at the hearing.” (Indeed, Dr. Perlin is the lead author of every one of the five published validity studies and a co-author of a sixth published study that also helps show validity.)

The majority acknowledged “legitimate concern” but decided that it was overcome “by the import of the empirical evidence of reliability demonstrated here and the acceptance of the methodology by the relevant scientific community.” However, the discussion of “the import of the empirical evidence” seems somewhat garbled.

1

First, the court notes that “the FBI Quality Assurance Standards requires ‘a developmental validation for a particular technology’ be published.” That the FBI might be satisfied with a single publication from the developer of a method does not speak to what the broader scientific community regards as essential to the validation. Along with the QAS, the court cites "NIST, DNA Mixture Interpretation: A NIST Scientific Foundation Review, at 64 (June 2021 Draft report)." The page merely reports that the NIST staff were able to examine “[p]ublicly available data on DNA mixture interpretation performance ... from five sources [including] published PGS studies” and that “conducting mixture studies may be viewed as a necessity to meet published guidelines or QAS requirements ... .” That scientists and other NIST personnel who choose to review a technology will read the scientific reports of the developers of the technology does not tell us much about defendant’s claim that Cybergenetics’ involvement in the published validation studies gravely diminishes “the import of the empirical evidence.”

2

Second, the Court of Appeals maintained that “the interest of the developer was addressed at the Frye hearing in this case.” As the court described the hearing, the response to this concern was that “[a]lthough Dr. Perlin was involved in and coauthored most of the validation studies, his interest in TrueAllele was disclosed as required by the journals who published the studies and the empirical evidence of the reliability of TrueAllele was not disputed.”

These responses seem rather flaccid. Some of the articles contain conflict-of-interest statements; most do not. \7/ But the presence or absence of obvious disclaimers does not come to grips with the complaint. Defendant’s argument is not that there are hidden funding sources or financial relationships. It is that interests in the outcomes of the studies somehow may affect the results. The claim is not that validation data were fabricated or that the data analysis was faulty. As with the movement for replication and “open science,” it is a response to more subtle threats.

3

Third, the opinion asserts that “the scientific method” is “entirely consistent with” proof of validity coming from the inventors, discoverers, or commercializers (citing President's Council of Advisors on Sci. and Tech., Exec. Office of the President, Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods, at 46 (2016)). Again, however, the argument is not that only disinterested parties do and should participate in scientific dialog. It is that "[w]hile it is completely appropriate for method developers to evaluate their own methods, establishing scientific validity also requires scientific evaluation by other scientific groups that did not develop the method.” Id. at 80 (https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/PCAST/pcast_forensic_science_report_final.pdf [https://perma.cc/R76Y-7VU]).

4

That precept leads to the court’s last and most telling response to the “legitimate concern” over “the dearth of independent validation.” The Chief Judge finally wrote that “there were [not only] developer [but also] independent validation studies and laboratory internal validation studies, many published and peer-reviewed.”

But is this a fair characterization of the scientific literature as of 2015? From what I can tell, no more than five or six studies appear in peer-reviewed journals, and none are completely “independent validation studies.” The NIST report cited in Wakefield lists but a single “internal validation” study, from Virginia in 2013, apparently released in response to a Freedom of Information Act request, Although the NIST reviewers limited themselves to laboratory studies or data posted to the Internet, they concluded that “[c]urrently, there is not enough publicly available data to enable an external and independent assessment of the degree of reliability of DNA mixture interpretation practices, including the use of probabilistic genotyping software (PGS) systems.” 

Of course, this “Key Takeaway #4.3” is merely part of a draft report and is not a judgment as to what conclusions on validity should be reached on the basis of the published studies and the internal ones. Nevertheless, the court overlooks this prominent “takeaway” (and others). Instead, the Chief Judge asserts that “[t]he technology was approved for use by NIST”—even though NIST is not a regulatory agency that approves technologies—and that “NIST's use of the TrueAllele system for its standard reference materials likewise demonstrates confidence within the relevant community that the system generates accurate results.”

~~~

This is not to say that the scientific literature was patently insufficient to support the court’s assessment of the general scientific acceptance of TrueAllele for interpreting the DNA data in the case. But it does raise the question of whether the court’s assertions about the large number of “independent validity studies” and internal ones that have been “published and peer-reviewed” are exaggerated.

Source Code and General Acceptance

The defense also contended that the state’s testimony and exhibits from “the Frye hearing [were] insufficient because, absent disclosure of the TrueAllele source code for examination by the scientific community, its ‘proprietary black box technology’ cannot be generally accepted as a matter of law.” This argument bears two possible interpretations. On the one hand, it could be a claim that scientists demand open-source programs—those with every line of code deposited somewhere for everyone to see—before they will consider a program suitable for data analysis or other purposes. We can call this position the open-source theory.

On the other hand, the claim might be “that disclosure of the TrueAllele source code [to the defense, perhaps with an order to protect against more widespread dissemination of trade secrets] was required to properly conduct the Frye hearing” and that without at least that much discovery of the code, scientists would not regard TrueAllele as valid. We can call this position the discovery-based theory. It implies that, in establishing general scientific acceptance in a Frye hearing, pretrial discovery of secret code is an adequate substitute for exposing the code to the possible scrutiny of the entire scientific community. \8/

The Wakefield opinions are not entirely clear on about which theory they embrace or reject. Judge Rivera’s concurrence may have endorsed both theories. In addition to accentuating “the need to provide defendant with access to the source code,” she decried the absence of “objective, expert third-party access,” writing that

The court's decision was an abuse of discretion as a matter of law because it relied on validation studies by interested parties and evaluations founded on incomplete information about TrueAllele's computer-based methodology. Without defense counsel and objective, expert third-party access to and evaluation of the underlying algorithms and source code, the court could not conclude that TrueAllele's brand of probabilistic genotyping was generally accepted within the forensic science community.

The “evaluations founded on incomplete information” were from a standards developing organization, a state forensic science commission, and NIST. They were incomplete because, according to Judge Rivera, “without the source code, the agencies could not adequately evaluate the use of TrueAllele for this type of DNA mixture analysis ... .”

Focusing on the discovery-based theory, the rest of the court determined that “[d]isclosure ... was not needed in order to establish at the Frye hearing the acceptance of the methodology by the relevant scientific community. The Chief Judge gave two, somewhat confusingly stated, reasons. The first was that Wakefield sought the source code under a rule for discovery that did not apply and then “made no further attempt to demonstrate a particularized need for the source code by motion to the court.” But it is not clear how the failure “to demonstrate a particularized need” overcomes (or even responds to) the argument that the scientific community will not accept software as validly implementing algorithms unless the source code is either open source or given only to the defendant.

The Chief Judge continued:

Moreover, defendant's arguments as to why the source code had to be disclosed pay no heed to the empirical evidence in the validation studies of the reliability of the instrument or to the general acceptance of the methodology in the scientific community—the issue for the Frye hearing—and are directed more toward the foundational concern of whether the source code performed accurately and as intended (see Wesley, 83 N.Y.2d at 429, 611 N.Y.S.2d 97, 633 N.E.2d 451).

The meaning of the sentence may not be immediately apparent. The defense argument is that giving a defendant (or perhaps the scientific community generally) access to source code is a prerequisite to general acceptance of the proposition that the software correctly implements theoretically sound algorithms. If this broad proposition is false dogma, the court should simply say so. It should announce that source code need not be disclosed because there is an alternative, reasonably effective means for establishing that the software performs as it should. The first part of the first sentence starts out that way, but the sentence then states that “whether the source code performed accurately and as intended” is not a matter of general acceptance at all. It is only “foundational” in the sense identified by Chief Judge Kaye in Wesley, who, as we saw (Box 2), wrote that even though RFLP-VNTR testing was generally accepted, the complete “foundation” for admitting DNA evidence entails proof that the generally accepted procedure was performed properly in the case at bar.

But regarding the argument about source code as falling outside of the Frye inquiry misapprehends the defense argument. Neither the open-source nor the discovery-based theories pertain to the execution of valid software. They question the premise that validity can be generally accepted without disclosure of the program’s source code. Yet, the majority elaborates on its non-Frye "foundational" classification for the source-code argument as follows:

To the extent the testimony at the hearing reflected that the TrueAllele Casework System may generate less reliable results when analyzing more complex mixtures (see also President's Council of Advisors on Sci. and Tech., Exec. Office of the President, Forensic Science in Criminal Courts: Ensuring Scientific Validity of Feature-Comparison Methods, at 80 [2016] https://obamawhitehouse.archives.gov/sites/default/files/microsites/ostp/PCAST/pcast_forensic_science_report_final.pdf [published after the Frye hearing was held]), defendant did not refine his challenge to address the general acceptance of TrueAllele on such complex mixtures or how that hypothesis would have been applicable to the particular facts of this case. As a result, it is unclear that any such objection would have been relevant to defendant's case, where the samples consisted largely of simple (two-contributor) mixtures with the victim as a known contributor (see also NIST, DNA Mixture Interpretation: A NIST Scientific Foundation Review, at 3 [June 2021 Draft report] https://nvlpubs.nist.gov/nistpubs/ir/2021/NIST.IR.8351-draft.pdf).

These citations to the PCAST and NIST reports actually undercut any suggestion that source-code secrecy does not implicate Frye. The NIST draft repeatedly states that

Forensic scientists interpret DNA mixtures with the assistance of statistical models and expert judgment. Interpretation becomes more complicated when contributors to the mixture alleles. Complications can also arise when random variations, also known as stochastic effects, make it more difficult to confidently interpret the resulting DNA profile.

Not all DNA mixtures present these types of challenges. We agree with the President’s Council of Advisors on Science and Technology (PCAST) that “DNA analysis of single-source samples or simple mixtures of two individuals, such as from many rape kits, is an objective method that has been established to be foundationally valid” (PCAST 2016).

NIST, DNA Mixture Interpretation: A NIST Scientific Foundation Review, at 2-3 & 11-12 (June 2021 draft) (citations omitted). To demand that “defendant ... refine his challenge to address the general acceptance of TrueAllele on ... complex mixtures or ... the particular facts of this case” is to hold that TrueAllele is generally accepted for use with “single-source samples or simple mixtures of two individuals”—even though the source code is hidden. But if science does not demand the disclosure of source code for general acceptance inside the "single" or simple "zone," then why would it demand disclosure for general acceptance outside that zone?

The court's remarks make more sense as a response to Wakefield’s different discovery argument about the need for source code for trial purposes. This argument does not claim that disclosure of source is essential to general acceptance to exist. It looks to the trial rather than the pretrial Frye hearing. The thought may be that if the accuracy of the program for the “simple” cases is assured, then the need for discovery of the code to prepare for trial testimony is less compelling. The court appears to be responding that because “the samples consisted largely of simple (two-contributor) mixtures with the victim as a known contributor,” there was little need for discovery of the code in this case.

Although this rejoinder departs from the topic of what Wakefield teaches us about general acceptance, I would note that it is difficult to reconcile this characterization of the case with Chief Judge DeFiore’s own description of the samples. The court mentioned four samples. Its initial description of them indicates that the New York laboratory deemed the sample on the amplifying cord to be “at least” a three-person mixture and stated that “because of the complexity of the mixture,” the laboratory could not even compare “results generated from the amplifier cord ... to defendant's DNA profile.” 2022 WL 1217463, at *1. Because of the “stochastic threshold,” the laboratory could discern peaks at only 4 out of 15 loci for “the outside rear shirt collar” and “for the profile obtained from the victim's forearm.” Id. Presumably, the “insufficient data” on “the unknown contributors to the DNA mixtures found on the amplifier cord and the front of the shirt collar” is what led the state to call Cybergenetics for help. These samples are not instances of what PCAST called “DNA analysis of single-source samples or simple mixtures of two individuals, such as from many rape kits” or what the NIST group called “two-person mixtures involving significant quantities of DNA.” They are “more complicated” situations that arise “when contributors to the mixture share common alleles [and] when random variations, also known as stochastic effects” are present.

In sum, the deeper one looks into the Wakefield opinions, the more there is to wonder about. But whatever quirks and quiddities reside in the writing, the nearly unanimous opinion of the Court of Appeals signals that a trial court can choose to dispense with the general-acceptance inquiry for at least one PGS program—TrueAllele—for nonchallenging single samples or two-person mixtures and for samples of somewhat greater complexity as well.

NOTES

* UPDATE: On July 12, 2022, Chief Judge DiFiore announced that she will resign on August 31. See, e.g., Jimmy Vielkind & Corinne Ramey, New York’s Top Judge Resigns Amid Misconduct Proceeding: Attorney for Court of Appeals Judge Janet DiFiore Said Her Resignation Wasn’t Related to a Claim that She Improperly Attempted to Influence a Disciplinary Hearing, Wall St. J., July 12, 2022 8:31 am ET, https://www.wsj.com/articles/new-yorks-top-judge-resigns-amid-misconduct-proceeding-11657629111.

1. This formulation conflates the issue of novelty with the issue of general acceptance, which can change over time. See Williams, 35 N.Y.3d at 43, 147 N.E.3d at 1143.
2. The description begins with the remark that “The likelihood ratio in its modern form was developed by Alan Turing during World War II as a code-breaking method.” That is a possibly defective bit of intellectual history, inasmuch as Turing did not develop the likelihood ratio. To decipher messages, Turing relied on a logarithmic scale for the Bayes’ factor in two ways—as indicating the strength of evidence, and as a tool for sequential analysis. Sir Harold Jeffreys had done the former in his 1939 Theory of Probability book. The sequential analysis problem is not clearly connected to PGS. It arises when the sample size is not fixed in advance and the data are evaluated continuously as they are collected. PGS processes all the data at once.
3. As the court wrote in People v. Williams, 35 N.Y.3d 24, 147 N.E.3d 1131, 1139–40, 124 N.Y.S.3d 593 (N.Y. 2020), “[r]eview of a Frye determination must be based on the state of scientific knowledge and opinion at the time of the ruling (see Cornell, 22 N.Y.3d at 784-785, 986 N.Y.S.2d 389, 9 N.E.3d 884 (‘a Frye ruling on lack of general causation hinges on the scientific literature in the record before the trial court in the particular case’”).
4. E.g., 2022 WL 1217463 at *7 n.10 (“TrueAllele is not an outlier in the use of the continuous probabilistic genotyping method. Other types of probabilistic genotyping software, such as STRMix, have likewise been found to be generally accepted (see e.g. United States v. Gissantaner, 990 F.3d 457, 467 (6th Cir.2021)).”
5. Cf. David H. Kaye, Theona M. Vyvial & Dennis L. Young, Validating the Probability of Paternity, 31 Transfusion 823 (1991) (comparing the empirical LR distribution for parentage using presumably true and false mother-child-father trios derived from a set of civil paternity cases to the “paternity index” (PI), a likelihood ratio computed with software applying simple genetic principles to the inheritance of HLA types, and reporting that the theoretical PI diverged from the empirical LR for PI > 80 or so).
6. At trial, “Gary Skuse, Ph.D., a professor of biological sciences at the Rochester Institute of Technology, testified at trial as a defense witness [and] agreed ... that defendant's DNA was present in the mixtures found on the shirt collar and amplifier cord and that it was ‘most likely’ present on the victim's forearm.”
7. The articles in the Journal of Forensic Sciences and Science and Justice have no such statements. The “Competing Interests” paragraph in a PloS One article advises that “I have read the journal’s policy and have the following conflicts. Mark Perlin is a shareholder, officer and employee of Cybergenetics in Pittsburgh, PA, a company that develops genetic technology for computer interpretation of DNA evidence. Cybergenetics manufactures the patented TrueAllele Casework system, and provides expert testimony about DNA case results. Kiersten Dormer and Jennifer Hornyak are current or former employees of Cybergenetics. Lisa Schiermeier-Wood and Dr. Susan Greenspoon are current employees of the Virginia Department of Forensic Science, a government laboratory that provides expert DNA testimony in criminal cases and is adopting the TrueAllele Casework system. This does not alter our adherence to all the PLOS ONE policies on sharing data and materials.”
8. The defense advanced another different discovery theory in arguing that it could not adequately cross-examine and confront Dr. Perlin at trial unless it could access the source code. The court rejected this theory too.

The New York Court of Appeals Returns to Probabilistic Genotyping Software (Part I—Williams)

For the past ten years or so, motions to exclude testimony of “probabilistic genotyping” results have been commonly lodged—and almost always denied. With rare exceptions, appellate courts have held these rulings to be proper (or at least within the trial judge’s discretion). Then came People v. Williams, 35 N.Y.3d 24, 147 N.E.3d 1131, 124 N.Y.S.3d 593 (N.Y. 2020).

In this murder case, New York’s highest court held that the output of one form of probabilistic genotyping software (PGS) was being admitted prematurely, before the scientific community had an adequate chance to evaluate it. But that was two years ago. Two weeks ago, in People v. Wakefield, 2022 N.Y. Slip Op. 02771, 2022 WL 1217463 (N.Y. Apr. 26, 2022), the court returned to the issue of PGS evidence. As in Williams, PGS produced "likelihood ratios" associating the defendant with a murder weapon, but this time the court held that the PGS in question had achieved the general scientific acceptance required to admit scientific evidence in New York.

This posting discusses Williams. A separate posting will consider how Wakefield differs from Williams, and why.

Cadman Williams was accused of a fatal shooting in the Bronx in 2008. The DNA in the case came from a gun hidden in William’s former girlfriend’s apartment. At trial, an expert from the New York City Office of the Chief Medical Examiner (OCME) testified “that it was millions of times more likely that the DNA mixture found on the gun contained contributions from defendant and one unknown, unrelated person, rather than from two unknown, unrelated people.” 35 N.Y.3d at 31. At least, that is how the court understood the testimony. It is not, however, an accurate statement of a likelihood ratio involving the identity of the individual (or individuals) whose DNA is in the sample. Such a likelihood ratio involves only the probability of the DNA data conditional on source hypotheses, not the other war around. (With large enough likelihood ratios, however, the distinction can be academic.) A further issue is the choice of hypotheses. Why not the former girlfriend rather than a random person's DNA?

Nonetheless, the Williams opinion did not need to consider the proper interpretation of the likelihood ratio for a pair of hypotheses or the selection of those hypotheses. The appeal only concerned the general scientific acceptance of the method that the OCME had devised for generating likelihood ratios for minute quantities of DNA. The Court of Appeals held that the trial court erred “in admitting expert testimony with respect to LCN and FST results in the absence of a Frye hearing.” 35 N.Y.3d at 42. LCN stands for “low copy number” and refers to the fact that the OCME laboratory tweaked the usual method for producing data on the personally identifying features of these fragments to obtain some results. FST stands for “Forensic Statistical Tool,” a computer program developed within the OCME to calculate likelihood ratios. Frye refers to Frye v. United States, 293 F. 1013 (D.C. Cir. 1923), a famous case from the District of Columbia that announced the rule that a scientific method had to achieve acceptance within the scientific community before its results can be received as evidence in court.

Although the Court of Appeals referred to one New York trial court opinion finding general acceptance of the OCME method as “questionable,” the Williams court did not hold that the computer output from the FST was necessarily inadmissible. The precise legal error lay in the trial court’s allowing the testimony without first conducting an evidentiary hearing on whether OCME’s methods were generally accepted within the scientific community. The majority opinion, written by Judge Eugene M. Fahey  distinguished between the LCN and FST parts of the OCME method and determined that neither could be said to be generally accepted based on the information presented to the trial judge (namely, prior opinions, mostly without evidentiary hearings; scientific publications; internal studies from the OCME laboratory; and a review conducted by a subcommittee of New York’s forensic science commission).

“With respect to the FST issue,” the prosecution had “maintained that such evidence should be admitted without a Frye hearing because ‘numerous articles published in peer-reviewed scientific journals’ supported the point that ‘the analytical software employs well-established principles such as Bayesian statistics and likelihood ratios which are used in many areas of science including forensics, medicine and social sciences.’” 35 N.Y.3d at 35 (note omitted). The prosecution added that “given both the thorough review of the FST by DNA Subcommittee of the New York State Forensic Science Committee [sic] and the exhaustive validation of that tool by OCME, the relevant scientific community had accepted the FST as reliable.” 35 N.Y.3d at 31.

The Court of Appeals was unmoved. It wrote that:

If the analysis was as simple as determining whether FST is comprised of existing mathematical formulas that are individually accepted as generally reliable within the relevant scientific community, then FST evidence probably would be admissible even in the absence of a Frye hearing. [¶] The point remains, however, that FST is a proprietary program exclusively developed and controlled by OCME. The sole developer and the sole user are the same. That is not “an appropriate substitute for the thoughtful exchange of ideas ... envisioned by Frye” (Wesley, 83 N.Y.2d at 441, 611 N.Y.S.2d 97, 633 N.E.2d 451 [Kaye, Ch. J., concurring] ). It is an invitation to bias. .... [That the] tool has ... been vetted and approved by “the distinguished scientists making up the DNA Subcommittee of the New York State Forensic Science Committee” is certainly relevant [but] that insular endorsement is no substitute for the scrutiny of the relevant scientific community. [¶] Indeed, here, defendant was hamstrung in demonstrating the existence of conflicting scientific opinions in order to show the need for Frye review of the FST based on the “black box” nature of that program, but his papers adequately showed that OCME's secretive approach to the FST was inconsistent with quality assurance standards within the relevant scientific community. Those papers also showed that facts adduced in challenges to the FST made in Frye applications in other proceedings suggested that the accuracy calculations of that program may be flawed. .... In short, the FST should be supported by those with no professional interest in its acceptance. Frye demands an objective, unbiased review.

147 N.E.3d at 1141–42.

This language was too much for three of the seven judges. Their concurring opinion written by Chief Judge Janet M. DiFiore balked at the “pejorative view of the ... OCME's ... LCN DNA typing technique and its ... probabilistic genotyping software program ... .” 147 N.E.3d at 1147. To be sure, the concurrence agreed that “the issues ... in this 2014 motion [were] ripe for a Frye hearing”—but only because it appeared that the internal studies did not appear to encompass the small quantity of DNA that was analyzed in this case. According to the concurrence,

The LCN DNA profiles drive the FST analysis, and FST results are only as reliable as the predicate assumptions integrated into the FST software program. The People did not meet their burden of establishing the validity of the empirical data used to fuel the calculations performed by this statistical model, including the manner of accounting for the occurrence of the stochastic effect and allelic dropout in a multiple contributor sample of less than 25 picograms, in a manner sufficient to bypass a Frye hearing. Fundamentally, the combined use of that statistical tool with DNA typing on samples that fell beneath validated thresholds may have impacted the reliability of the results, raising a valid challenge to the admissibility of that evidence in a criminal prosecution.

35 N.Y.3d at 52–53. In other words, the concurring judges seemed to buy the state’s broad argument that there was no need for a Frye hearing to find general acceptance of the OCME system in many cases. But not in this case, where the laboratory pushed beyond what it had validated to its own satisfaction. These judges evinced no concern with the limited outside testing of the FST and expressed no doubts about the conclusions of general acceptance reached in the clear preponderance of trial court rulings on motions to exclude FST likelihood ratios. And the entire court agreed that admission in the case was harmless error because the other evidence against Williams was overwhelming. 

In sum, Williams was a case with broad reasoning (by the majority) on a narrow topic—the OCME’s home-grown Forensic Statistics Tool as applied to data from an especially challenging DNA sample (as stressed by the concurrence). What about other FST likelihood ratios with data from larger DNA samples? Other brands of PGS?

One thing was clear. Williams would not be the last word on probabilistic genotyping. Another case, involving an older and more established computer program known as TrueAllele, was on its way to the Court of Appeals. Stay tuned for thoughts on this case.

Commonwealth v. Ross: Is Bitemark Identification Evidence Becoming Toothless?

In a case that has received national attention, a Pennsylvania Superior Court (the state's intermediate appellate court) finally decided that the trial judge had to at least hold a hearing on the general scientific acceptance of bitemark identification evidence. The trial court had ruled that no such "Frye hearing" was required before admitting proposed testimony from two forensic odontologists that the defendant cannot be excluded as the source of human bite marks on the rape-murder victim whereas several other potential suspects can be.

Mysteriously, the appellate court chose to keep its lengthy opinion unreported and of "no precedential value." (It is reproduced at the end of this posting and available as a pdf file.) Many appellate courts do not publish opinions that simply apply settled law and would be of interest only to the parties, but that is not the situation here. The trial judge's refusal to grant the defense's request for such a "Frye hearing" has received national condemnation, 1/ and the new opinion is one of a very few to take seriously widespread misgivings about bitemark evidence.

As described in an NBC report complaining that "the justice system ignores science," 2/

A little after noon on June 24, 2004, a fisherman on Canoe Creek Lake in Blair County, Pennsylvania, found the bound, beaten and sexually abused body of Tina Miller near a boat launch. Suspicion focused on [Paul Aaron] Ross, who’d been with Miller nearby the night before.

There were no witnesses to the killing, leaving authorities to build a case on circumstantial evidence: a man who said he had dropped Miller and Ross off near the boat launch, former girlfriends who said he’d abused them, a mark found on one of Miller’s breasts, and items collected from the scene ─ including a shoe print and a beer bottle. None of it definitively linked Ross to the murder, and Ross denied killing Miller, asserting that another man had picked her up in a truck. But prosecutors argued that, taken collectively, the evidence proved his guilt.

He went to trial in 2005, and a bite-mark examiner helped seal his conviction.

The examiner, Dr. Dennis Asen, was a dentist with 25 years of experience identifying bodies by their teeth and examining bite marks. On the stand, he recalled viewing Miller’s body at the morgue and concluding that her breast bore patterns of a bite mark. He took a photograph of it and made an “overlay tracing” that he uploaded to a computer and compared with molds of Ross’ teeth and the teeth of four other suspects, including three who’d been in contact with Miller in the hours before her death. Asen said he found the mark “very highly consistent” only with Ross’ teeth. Asen acknowledged, however, that he couldn’t say with 100 percent certainty that Ross had made the mark. The defense hired its own bite-mark examiner, who said he was unable to determine whether Ross left the mark.

Ross was convicted of first-degree murder and sentenced to life in prison. But the conviction was overturned by an appeals court that ruled the trial court had “abused its discretion” in refusing to give Ross’ lawyer more time to prepare, and allowing the former girlfriends’ testimony.

Prosecutors plan to try Ross again, but the process has stalled amid a battle over the bite-mark evidence.

Defense lawyers, including members of the New York-based Innocence Project, have cited government reports and academic studies, including from the National Academy of Sciences, the President's Council of Advisors on Science and Technology and the Texas Forensic Science Commission.

The reports have shown that no one has sufficiently proven that human bite marks are unique, or that skin is a reliable material to record them ─ or that forensic dentists can reliably identify a human bite mark in the first place.

What was the Commonwealth's rejoinder? The Blair County district attorney is "The Honorable Richard A. Consiglio." (That is his title on the county's website, which emphasizes his success in dealth penalty cases along with his receipt of "an award ... for standing up to the Blair County Courts on behalf of the unborn." "Rich," as the website also refers to him, told now retired Blair County Judge Jolene Grubb Kopriva in 2016 that

the authors of scientific reports lack “real world” experience. He ... pointed out that no court in any state has ruled against admitting it. “This is evidence, your honor, that has been accepted for 40 or 50 years, not quite as old as I am, but 40 or 50 years, maybe longer, accepted by courts everywhere.” 3/

Acknowledging that "the use of bite-mark evidence is beginning to face challenges," Judge Kopriva ruled the evidence admissible on the basis of the lawyers' arguments. She wrote in Match 2017 that "it would be premature for this court to order that the methodology is no longer generally accepted in the relevant scientific community.” 4/

In a blog post (since deleted), "the Innocence Project described the case as an example of "'willful scientific illiteracy among the judiciary.' ... Judge Kopriva imposed a gag order preventing either side from talking publicly about the case outside of court." 5/

But she also certified an interlocutory appeal of her order denying the motion for a Frye hearing. (Ordinarily, such evidentiary rulings are only reviewed as part of an appeal from the verdict at the end of the trial.) The Superior Court refused to accept the interlocutory appeal. But then the Pennsylvania Supreme Court vacated the Superior Court's order and directed it to decide whether there had to be an evidentiary pretrial hearing on the admissibility of bitemark evidence

This week, on November 21, the Superior Court released its opinion. The court held that the trial court abused its discretion in agreeing to admit bitemark opinion testimony without first conducting a Frye hearing. The opinion, written by Judge Mary Murray, discusses Frye v. United States, 293 F. 1013 (D.C. Cir. 1923), the Superior Court's understanding of the general-acceptance requirement as it has been construed in Pennsylvania, and how this requirement applies to a request for an evidentiary hearing on bitemark evidence.

According to the Ross opinion, "Frye contemplated a judicial inquiry, informed by experts, into the general acceptance of the scientific methods used." Yet, Frye contains no mention of "a judicial inquiry" in the form of a pretrial evidentiary hearing on the status of a method among scientists. Without any such hearing to draw on, the Court of Appeals in Frye determined that "the systolic blood pressure deception test has not yet gained such standing and scientific recognition among physiological and psychological authorities as would justify the courts in admitting expert testimony deduced from the discovery, development, and experiments thus far made." 6/ The court simply announced its conclusion in an enigmatic eight-paragraph opinion. The task of devising suitable substitutes for judicial fiat fell to later generations.

Today, the use of hearings with expert testimony on scientific acceptance is well developed. In Pennsylvania, however, the touchstone for deciding when such hearings are essential often is said to be "novelty." Thus, Judge Kopriva relied on the lack of novelty in denying the defense's requested hearing, and the Superior Court focuses on the supposed requirement as well.

But Frye itself imposed no "novelty" requirement. All that Frye said in this regard that was the blood pressure test had yet to cross "the line between the experimental and demonstrable stages." 7/ The opinion did not consider whether a procedure could cross the line, but then fall back once more into the darker side of "this twilight zone." 8/ Bitemark evidence once was thought to have been valid (by its practitioners at least), but, as the defense in Ross argued, today the larger scientific community (and a growing number of dental experts) do not recognize its scientific validity.

Fortunately, the Superior Court, after flailing about in an effort to decide what needed to be novel, also addressed the more appropriate question of whether the relevant scientific community still believes (if it ever did) that the kind of testimony that Judge Kopriva was going to admit is generally accepted. It rejected District Attorney Consiglio's idea that acceptance within the forensic odontology community is sufficient, and it recognized that recent literature indicated significant dissension even within that community and outright rejection by a broader scientific community. Consequently, it held that the trial court could not admit the prosecution's evidence without a meaningful hearing on general acceptance in the broader community.

Of course, this holding does not dictate the outcome of the hearing. Indeed, the trial court could avoid an evidentiary hearing entirely, by reviewing the readily available scientific literature (with briefing by the parties) and concluding that it does not support the general acceptance of the comparative analysis that the prosecution proposes to present. After all, the proponent of the scientific evidence bears the burden of demonstrating its general acceptance, and that will not easy for the state to do in this case.

Nonetheless, that is too great an about-face to expect from the trial court. More likely, the court will proceed with the full-blown hearing. That hearing should focus on the specific testimony that the two dentists slated to testify might give. Is there sufficient "scientific recognition" 9/ that the forensic odontologists can validly determine that the bite marks on the breast of the victim's partially submerged body are human (if that is what they propose to say)? That (whether or not the marks are from a human source) they do not come from the four other potential suspects? That they were "very highly consistent" with Ross's dentition (as Dr. Asen testified at the original trial)? If the testimony were kept to a bare minimum, such as "unable to exclude as a possible source of the marks," with no explanation of the implications of such an inclusion (because scientific knowledge is too thin), 10/ would it be sufficiently helpful to be admissible? 11/ Or would it be inadmissible as being inscrutable or too easily overvalued?

The time when these issues could be avoided by arguments about "novelty" and "real world experience" has passed.

NOTES

1. John Shuppe, 'We Are Going Backward': How the Justice System Ignores Science in the Pursuit of Convictions, NBS News, Jan. 23, 2019, https://www.nbcnews.com/news/us-news/we-are-going-backward-how-justice-system-ignores-science-pursuit-n961256.
2. Id.
3. Id. There are exceptions to the District Attorney's bald assertion that "courts everywhere" deem bitemark testimony admissible. See infra note 11. Mr. Consiglio's timeline also was incorrect. The earliest use of bitemarks as identification in a U.S. criminal case may have been some 150 years ago, in 1870. See Barry E. Lipton et al., History of Odontology, in Manual of Forensic Odontology 1, 27 (David R. Senn & Richard A. Weems eds., 2013). The first reported case was a 1954 murder trial in Texas. Id. at 28. But see Michael J. Saks et al., Forensic Bitemark Identification: Weak Foundations, Exaggerated Claims, 3 J. L. & Biosci. 538 (2016), available at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5570687/ (asserting that before 1974 a some years thereafter, it was "the rule among forensic dentists that crime scene bite marks could not be trusted to yield accurate source identifications.).
4. Id.
5. Id.
6. Frye v. United States, 293 F. 1013, 1014 ( D.C.. Cir 1923).
7. Id.
8. Id. 
9. Id.
10. This seems to be the only type of inclusionary testimony the American Board of Forensic Odontology accepts from its members. See ABFO, Standards and Guidelines for Evaluating Bitemarks § 3 (2018) ("bitemark linkage conclusions should  only a) exclude  or b) not exclude  (include) a  dentition.The  specific  terms  found  in 2.c.are: a)  for  exclusion, Excluded  as Having  Made  the  Bitemark,  and  b)  for  inclusion, Not  Excluded  as  Having Made  the  Bitemark.Stronger  terms of  attribution are  not condoned ... .").
11. Cf. Ege v. Yukins, 485 F.3d 364 (6th Cir. 2007) (the more useful testimony of exclusions and "highly consistent" matching followed by "my expert opinion" that "nobody else" among the "three and a half million people" in the Detroit area "would match up" was without foundation and violated due process). The American Board of Forensic Odontology no longer condones such testimony. See supra note 10..

---

Commonwealth v. Ross

No. 1738 WDA 2018 (Pa. Super. Ct. Nov. 21, 2019)
(not reported, pdf file available at
www.pacourts.us/assets/opinions/Superior/out/J-S55007-19m%20-%2010423159687925902.pdf

BEFORE: MURRAY, J., McLAUGHLIN, J., and COLINS, J.
MEMORANDUM BY MURRAY, J.:

Paul Aaron Ross (Appellant) appeals from the order denying his request for a Frye hearing. 1/ After careful consideration, we vacate the trial court’s order denying Appellant’s request for a Frye hearing and remand to the trial court for proceedings consistent with this decision.

This appeal arises from the June 27, 2004 murder of Tina Miller at Canoe Creek Lake in Canoe Creek State Park. Appellant was arrested and charged with Ms. Miller’s murder. On November 23, 2005, a jury found Appellant guilty of first-degree murder, aggravated assault, involuntary deviate sexual intercourse, unlawful restraint, simple assault, false imprisonment, and indecent assault. 2/
A prior panel of this Court summarized the post-trial procedural history:

At the sentencing phase of the trial, the jury rejected imposition of the death penalty. The trial court then proceeded to sentence [Appellant] to life in prison plus 24 to 48 years. [Appellant] filed post-trial motions, which the trial court denied on January 30, 2006. On February 10, 2006, [Appellant] filed a timely notice of appeal, but [Appellant’s] counsel failed to file an appellate brief and the appeal was consequently dismissed. On September 26, 2008, [Appellant] filed a PCRA petition seeking the reinstatement nunc pro tunc of his direct appeal rights, which the trial court granted on August 14, 2009.

Commonwealth v. Ross, 57 A.3d 85, 90 (Pa. Super. 2012) (en banc).

On October 12, 2012, an en banc panel of this Court vacated Appellant’s judgment of sentence and remanded for a new trial. Id. at 105. On November 17, 2015, the Pennsylvania Supreme Court denied the Commonwealth’s petition for allowance of appeal.

On March 1, 2016, Appellant filed pre-trial motions in which he sought, inter alia, the exclusion of any expert evidence relating to bite mark identification and a Frye hearing. At trial, the Commonwealth intends to introduce the testimony of Dr. Dennis Asen (Dr. Asen) and Dr. Lawrence Dobrin (Dr. Dobrin). Dr. Asen and Dr. Dobrin are both dentists and practice in the field of forensic odontology (the study of the structure of teeth). Dr. Asen and Dr. Dobrin intend to testify that the mark on Ms. Miller’s left breast was caused by a human bite, and when they compared five sets of teeth molds, including one from Appellant, Dr. Asen and Dr. Dobrin could exclude four of the molds from having made the bite mark, but not Appellant’s.

On December 2, 2016, following the filing of several supplemental motions by Appellant and objections by the Commonwealth, the trial court heard oral argument on Appellant’s request for a Frye hearing. On March 8, 2017, after the parties submitted additional briefs on Appellant’s request for a Frye hearing, the trial court entered an order concluding that bite mark identification evidence is not novel and therefore a Frye hearing was not warranted. The court further provided that the Commonwealth’s experts were to adhere to the guidelines set forth by the American Board of Forensic Odontologists (ABFO).

On April 5, 2017, Appellant filed a motion to amend the March 8, 2017 order to include language relating to Pennsylvania Rule of Evidence 702(c), so that the trial court could address whether the expert methodology is generally accepted in the relevant field. Appellant also requested that the court certify for immediate appeal its decision not to hold a Frye hearing on the bite mark identification evidence.

On November 6, 2017, the trial court entered an amended order once again denying Appellant’s request for a Frye hearing. The trial court also included in the order language addressing Rule 702(c) and granting Appellant’s request for certification of immediate appeal. On December 5, 2017, Appellant filed a petition for permission to file an interlocutory appeal with this Court, which we denied by per curiam order on May 7, 2018. On June 1, 2018, Appellant filed a petition for allowance of appeal to the Pennsylvania Supreme Court. On November 20, 2018, our Supreme Court granted Appellant’s petition for allowance of appeal, vacated this Court’s order denying Appellant’s petition for permission to file an interlocutory appeal, and remanded the case to this Court for disposition.

On appeal, Appellant presents the following issues for review:

1) WHETHER THE SUBSTANTIAL EVIDENCE [APPELLANT] PROPOSED TO PRESENT AT A MOVED-FOR FRYE HEARING DISCREDITING BITE MARK ANALYSIS COMES FROM AND IS PART OF THE RELEVANT SCIENTIFIC COMMUNITY FOR FRYE PURPOSES THUS ENTITLING HIM TO A FRYE HEARING; OR WHETHER PENNSYLVANIA COURTS MUST LIMIT THEIR RELIABILITY INQUIRY TO THE VIEWS OF CURRENT PRACTICTIONERS OF THE PARTICULAR TECHNIQUE AT ISSUE?
2) WHETHER THE SUBSTANTIAL EVIDENCE PROFFERED BY [APPELLANT] AT THE MOVED-FOR FRYE HEARING PRESENTED A LEGITIMATE DISPUTE REGARDING THE RELIABILITY OF BITE MARK ANALYSIS THUS ENTITLING HIM TO A FRYE HEARING; AND SUCH THAT THE TRIAL COURT’S ADMISSION OF THE SAME WITHOUT A FRYE HEARING WAS IN ERROR?

Appellant’s Brief at 5.

Both of Appellant’s issues related. Therefore, we address them together. We begin with our standard of review:

As a general rule, this Court’s standard of review of a trial court’s evidentiary ruling, including a ruling whether expert scientific evidence is admissible against a Frye challenge, is limited to determining whether the trial court abused its discretion. Grady v. Frito–Lay, Inc., 839 A.2d 1038, 1046 (Pa. 2003); Zieber v. Bogert, 773 A.2d 758, 760 n.3 (Pa. 2001) (citing Commonwealth v. Minerd, 753 A.2d 225 (Pa. 2000)). “An abuse of discretion may not be found merely because an appellate court might have reached a different conclusion, but requires a result of manifest unreasonableness, or partiality, prejudice, bias, or ill-will, or such lack of support so as to be clearly erroneous.” Grady, 839 A.2d at 1046 (citing Paden v. Baker Concrete Constr., Inc., 658 A.2d 341, 343 (Pa. 1995)).

Commonwealth v. Dengler, 890 A.2d 372, 379 (Pa. 2005) (citations modified).

Appellant argues that the trial court abused its discretion in denying his request for a Frye hearing on the admissibility of the Commonwealth’s bite mark identification evidence. This Court recently articulated the legal standards implicated by a Frye analysis:

The Frye standard originally was intended to prevent the situation in which a party would seek to introduce scientific evidence that was so new that it would be impossible to “produce rebuttal experts, equally conversant with the mechanics and methods of a particular technique.” [U.S.] v. Addison, 498 F.2d 741, 744 (D.C. App. 1974). Frye contemplated a judicial inquiry, informed by experts, into the general acceptance of the scientific methods used. The standard required that “the thing from which the [expert’s] deduction is made must be sufficiently established to have gained general acceptance in the particular field in which it belongs.” Frye, supra at 1014. At issue in Frye was admissibility of the systolic blood pressure deception test, commonly known as the lie detector test. The trial court excluded the evidence, and the court affirmed that ruling on appeal, explaining:

Just when a scientific principle or discovery crosses the line between the experimental and demonstrable stages is difficult to define. Somewhere in this twilight zone the evidential force of the principle must be recognized, and while courts will go a long way in admitting expert testimony deduced from a well-recognized scientific principle or discovery, the thing from which the deduction is made must be sufficiently established to have gained general acceptance in the particular field in which it belongs.

Frye, supra at 1014. Pennsylvania adopted the Frye standard in Commonwealth v. Topa, 369 A.2d 1277 (Pa. 1977), a case involving the propriety of the trial court’s admission of voice print identification evidence through an expert, Lieutenant Nash, of the Michigan State Police. Our High Court, applying Frye, reasoned that

[t]he requirement of general acceptance in the scientific community assures that those most qualified to assess the general validity of a scientific method will have the determinative voice. Additionally, the Frye test protects prosecution and defense alike by assuring that a minimal reserve of experts exists who can critically examine the validity of a scientific determination in a particular case. Since scientific proof may in some instances assume a posture of mystic infallibility in the eyes of a jury of laymen, the ability to produce rebuttal experts, equally conversant with the mechanics and methods of a particular technique, may prove to be essential.

Topa, supra at 1282 (quoting Addison, supra at 744). The Topa Court went on to conclude that the testimony of one expert could not satisfy this standard, citing commentaries questioning the reliability of sound spectrographs and voiceprints and demonstrating that it was not generally accepted within the field of acoustical science.
Thus, the Frye standard originally was intended to prevent a party from introducing scientific evidence that was so new that it would be impossible to “produce rebuttal experts, equally conversant with the mechanics and methods of a particular technique.” Addison, supra at 744. Frye contemplated a judicial inquiry, informed by experts, into the general acceptance of the scientific methods used.

In the years since the adoption of the Frye standard, this Court has clarified that “Frye only applies to determine if the relevant scientific community has generally accepted the principles and methodology the scientist employs, not the conclusions the scientist reaches.” Trach v. Fellin, 817 A.2d 1102, 1112 (Pa.Super. 2003) (en banc).

Walsh v. BASF Corp., 191 A.3d 838, 842-43 (Pa. Super. 2018), appeal granted, 203 A.3d 976 (Pa. 2019). 3/

The above principles have been incorporated into Pennsylvania Rule of Evidence 702, which states:

A witness who is qualified as an expert by knowledge, skill, experience, training, or education may testify in the form of an opinion or otherwise if: (a) The expert’s scientific, technical, or other specialized knowledge is beyond that possessed by the average layperson; (b) The expert’s scientific, technical, or other specialized knowledge will help the trier of fact to understand the evidence or to determine a fact in issue; (c) The expert’s methodology is generally accepted in the relevant field. Pa.R.E. 702 (emphasis added).

Appellant argues that the trial court erred in determining a Frye hearing was unnecessary in this case. Appellant asserts that the court wrongly concluded that the methodology employed by the Commonwealth’s experts in conducting their bite mark identification analysis was generally accepted in the scientific community of forensic odontology. Additionally, Appellant contends that the trial court should have examined whether the expert’s methodology for bite mark identification analysis was generally accepted in the broader scientific community, as opposed to limiting its examination to the field of forensic odontology. Appellant maintains that bite mark identification analysis has applications in several different scientific communities, and several of these communities have determined that bite mark identification analysis is not reliable and has little or no evidentiary value.

In support of his argument, Appellant relies on our Supreme Court’s decision in Betz v. Pneumo Abex, LLC, 44 A.3d 27 (Pa. 2012). At issue in Betz was the admissibility of expert opinion evidence relating to the “any-exposure” theory of legal causation, i.e., “that each and every exposure to asbestos – no matter how small – contributes substantially to the development of asbestos-related diseases.” Id. at 30. As part of its analysis, the Supreme Court had to first address, as a threshold issue, whether the trial court was correct in concluding that a Frye hearing was necessary to determine the admissibility of the “any-exposure” evidence. Id. at 52-55.

In concluding that the trial court’s decision to conduct a Frye hearing was correct, the Supreme Court explained:

There is inherent tension among the various measures for admissibility of expert testimony. The threshold common law test requires merely some reasonable pretension to specialized knowledge. See, e.g., Miller v. Brass Rail Tavern, Inc., 664 A.2d 525, 528 (Pa. 1995). Our evidentiary rules, on the other hand, suggest trial courts may take a greater role in assessing whether the testimony will assist the trier of fact to understand the evidence or determine a fact in issue, see Pa.R.E. 702, and in screening evidence to avoid unfair prejudice, confusion of the issues, or misleading of the jury, see Pa.R.E. 403. For better or for worse, however, in the context of the more conventional realms of science, the Pennsylvania decisions tend to downplay the courts’ screening function. See, e.g., Commonwealth v. Nazarovitch, 436 A.2d 170, 172 (Pa. 1981) (“[C]ourts will go a long way in admitting expert testimony deduced from a well-recognized scientific principle or discovery[.]” (quoting Frye, 293 F. at 1014)). A manifestation of this trend is that challenges generally are vetted through the Frye litmus, which winnows the field of the attacks by application of the threshold requirement of novelty. See Grady, 839 A.2d at 1043-44.

Various reasons underlie the preference to limit the courts’ involvement in determining the admissibility of scientific evidence. There is the concern that liberality in allowing challenges would substantially increase the number of challenges (and cases in which lengthy pre-trial proceedings would ensue). The competency of trial judges to accept or reject scientific theories remains a legitimate subject of controversy. Additionally, a claim or defense in many cases may rise or fall based upon expert testimony and, therefore, there is some reluctance on the part of courts to deprive litigants of their day in court.

On the other hand, this Court has recognized the influential nature of expert testimony on complex subjects, and the potential that distortions have to mislead laypersons. See id. at 1045; Topa, 369 A.2d at 1281-82. It would be naïve, in this regard, to assume that the possibility for distortion is limited to the very newest realms of science. Cf. Grady, 839 A.2d at 1045 (explaining that Frye applies not only to novel science, but also where scientific methods are utilized in a novel way).

We therefore agree with Appellants that a reasonably broad meaning should be ascribed to the term “novel.” Furthermore, we conclude that a Frye hearing is warranted when a trial judge has articulable grounds to believe that an expert witness has not applied accepted scientific methodology in a conventional fashion in reaching his or her conclusions. Accord id. We believe a narrower approach would unduly constrain trial courts in the appropriate exercise of their discretion in determining the admissibility of evidence. See id. at 1046.

Id. at 52-53 (emphasis added, footnote omitted, citations modified).

Additionally, in Betz, the Supreme Court rejected the notion that the defendants could not challenge the methodology of the plaintiff’s expert pathologist with the testimony of risk assessors, toxicologists, and epidemiologists. Id. at 54. The Court reasoned that the expert pathologist’s “any-exposure” opinion “was not couched in terms of a methodology or standard peculiar to the field of pathology[,]” and “was plainly grounded on risk assessment.” Id. at 54-55. Thus, the Supreme Court determined that the subjects at issue “are not within the particular expertise of a pathologist, but, rather, are interdisciplinary in character.” Id. at 55.

In this case, the trial court reached two conclusions in denying Appellant’s request for a Frye hearing. First, the trial court determined that “[b]ite mark evidence is currently generally accepted in the relevant scientific community of forensic odontologists.” Trial Court Opinion, 11/6/17, at 5. The court explained:

The American Board of Forensic Odonotologists (ABFO) has promulgated guidelines regarding the approved methodology concerning bite mark evidence. This [c]ourt notes that the aforementioned methodology rejects the notion that human dentitions are unique to the individual. Any evidence proffered by the Commonwealth to suggest that an individual can be identified by a bite mark, rather than merely excluded or not excluded as having made the mark, would be inadmissible.

Id.

Second, the trial court rejected Appellant’s reliance on Betz, finding that case distinguishable:

It is true that the Pennsylvania Supreme Court allowed the Betz defendants to address the methodology of a pathologist through the testimony of risk assessors, toxicologists, and epidemiologists. However, Betz can be distinguished from the instant case. The Betz Court found that the pathologist’s opinion was not “couched in terms of a methodology or standard peculiar to the field of pathology.” [Betz, 44 A.3d at 54]. Rather, the pathologist’s opinion was “plainly grounded on risk assessment” and was “interdisciplinary in character.” Id. at 55. The Betz Court based its decision not upon the role of pathologists generally, but upon the specific methodology employed by the individual pathologist in question. Betz does not make a broad assertion that Pennsylvania law “mandates a broader definition of the relevant scientific community” where the methodology of a pathologist is at issue. Here, the Commonwealth does not seek to offer the kind of broad-scale scientific testimony that was at issue in Betz.

Id. at 6.

After careful consideration, we find support for Appellant’s position that a Frye hearing was warranted in this case. First, with respect to the trial court’s conclusion that bite mark identification analysis has general acceptance in the field of forensic odontology, Appellant offered evidence indicating that there is a lack of consensus among forensic odontologists on whether bite mark identification analysis is reliable and valid. While there is no dispute that the ABFO has established standards and a methodology for conducting bite mark identification analysis, see Commonwealth’s Third Brief in Opposition to Appellant’s Request for a Frye Hearing, 2/21/17, Exhibit B, Appellant presented numerous reports to the trial court indicating that practitioners within the community of forensic odontology question whether this methodology reliably enables forensic odontologists to identify an injury as a human bite mark.

For example, Appellant presented evidence from a presentation by Dr. David Senn, DDS, Vice-President of the American Board of Forensic Odontology, to the National Academies: Committee on Identifying the Needs of the Scientific Community. See Defendant’s Post-Argument Supplement to “Defendant’s Motion in Limine: Frye Test – Bite Mark Evidence” (hereinafter Defendant’s Post-Argument Supplement), 1/17/17, Exhibit 4 (Presentation to the National Academies: Committee on Identifying the Needs of the Forensic Science Community – Forensic Odontology Bite Marks (hereinafter Senn Presentation), 4/23/07, at 31-34). While Dr. Senn opined that bite mark identification analysis was important to the investigation and adjudication of certain crimes, id. at 45, Dr. Senn identified several “major problems” with bite mark identification analyses, including: “[t]he uniqueness of the human dentition has not been scientifically established”; “[t]he ability of the dentition, if unique, to transfer a unique pattern to human skin and maintain that uniqueness has not been scientifically established”; “[a] clear statement of the type, quality, and number of class and individual characteristics or other features required to indicate that a bite mark has reached a threshold of evidentiary value has not been established”; and “Forensic Odontology certifying organizations have not created or administered bite mark analysis proficiency tests for their board certified members.” Id. at 31-34.

Likewise, Appellant presented a report by the President’s Council of Advisors on Science and Technology, which revealed the following:

Empirical research suggest that forensic odontologists do not consistently agree even on whether an injury is a human bitemark at all. A study of the American Board of Forensic Odontology (ABFO) involved showing photos of 100 patterned injuries to ABFO board-certified bitemark analysts, and asking them to answer three basic questions concerning (1) whether there was sufficient evidence to render an opinion as to whether the patterned injury is a human bitemark; (2) whether the mark is a human bitemark, suggestive of a human bitemark, or not a human bitemark; and (3) whether distinct features (arches and toothmarks) were identifiable. Among the 38 examiners who completed the study, it was reported that there was unanimous agreement on the first question in only 4 of the 100 cases and agreement of at least 90 percent in only 20 of the 100 cases. Across all three questions, there was agreement of at least 90 percent in only 8 of the 100 cases.

Defendant’s Post-Argument Supplement, 1/17/17, Exhibit 5 (PRESIDENT’S COUNCIL OF ADVISORS ON SCIENCE AND TECHNOLOGY, REPORT TO THE PRESIDENT: FORENSIC SCIENCE IN CRIMINAL COURTS: ENSURING SCIENTIFIC VALIDITY OF FEATURE-COMPARISON METHODS (hereinafter PCAST Report) 84-85 (2016)) (emphasis added).

The studies proffered by Appellant challenge the trial court’s conclusion that the ABFO’s methodology for using bite marks to eliminate persons as suspects is generally accepted in the field of forensic odontology. These reports reflect that individuals within the forensic odontology community question not only whether the ABFO’s methodology can reliably aid experts in using bite marks to validly identify or exclude individuals as criminal actors, but also whether the methodology enables experts to identify a wound as a human bite mark. Therefore, Appellant provided the trial court with articulable grounds that the Commonwealth’s expert witnesses on bite mark identification analysis have not applied accepted scientific methodology in reaching their conclusions. See Betz, 44 A.3d at 53. Accordingly, we conclude that the trial court abused its discretion in denying Appellant’s request for a Frye hearing.

We are likewise persuaded by Appellant’s argument that the trial court should not have limited consideration of the general acceptance of the experts’ methodology in this case to the field of forensic odontology. As several of the reports Appellant cites reflect, bite mark identification analysis implicates numerous scientific fields. See generally Defendant’s Post-Argument Supplement, 1/17/17, Exhibit 2 (NATIONAL ACADEMY OF SCIENCES, COMMITTEE ON IDENTIFYING THE NEEDS OF THE FORENSIC SCIENCES COMMUNITY, STRENGTHENING FORENSIC SCIENCE IN THE UNITED STATES: A PATH FORWARD (2009), Exhibit 5 (PCAST Report), Exhibit 11 (Michael J. Saks, et al., Forensic bitemark identification: weak foundations, exaggerated claims, 3(3) J. LAW BIOSCI. 1 (2016)). As these studies indicate, bite mark identification analysis not only involves concepts relating to forensic science generally, but also pathology, biology, statistics, and metrology. See Defendant’s Post-Argument Supplement, 1/17/17, Exhibit 11 (Michael J. Saks, et al., Forensic bitemark identification: weak foundations, exaggerated claims, 3(3) J. LAW BIOSCI. 1 (2016)) (noting that the forensic identification process is “fundamentally probabilistic” and that it involves the interplay of different scientific disciplines including blood (pathology), skin (biology and dermatology), and measurements (metrology)). Because the act of biting a human involves not only the biter’s teeth, but also the skin, muscle, tissue, and blood with which the teeth make contact, the notion that bite mark identification analysis involves scientific disciplines beyond forensic odontology is reasonable.

Finally, we emphasize that our decision in no way represents a determination as to the general acceptance of the methodology underlying bite mark identification analysis utilized by the Commonwealth’s experts in this case. We make no judgment as to the admissibility of the bite mark identification evidence at issue. Rather, we simply conclude that Appellant provided the trial court with articulable grounds to believe that the Commonwealth’s expert witnesses on bite mark identification analysis may not have applied generally accepted scientific methodology in reaching their conclusions, and consequently, the trial court erred in concluding that a Frye hearing was not necessary. While Appellant’s evidence expresses negative opinions on bite mark identification analysis, we cite it only to support our conclusion that a Frye hearing is proper for the resolution of these discrepancies, and to afford both parties the opportunity to present evidence in support of their positions. Therefore, we vacate the order denying Appellant’s request for a Frye hearing and remand this matter to the trial court for a hearing in accordance with Frye.

Order vacated. Case remanded. Jurisdiction relinquished. Judgment Entered.

Footnotes
* Retired Senior Judge assigned to the Superior Court.
1. A Frye hearing, named after the decision in Frye v. United States, 293 F. 1013 (D.C. Cir. 1923), “is a hearing held for the trial court to determine whether the general scientific community has reached a general acceptance of the principles and methodology used by the expert witness.” Commonwealth v. Walker, 92 A.3d 766, 769 n.1 (Pa. 2014).
2. 18 Pa.C.S.A. §§ 2501(a), 2702(a)(1), 3123(a)(1), 2902(a)(1), 2701(a)(1), 2903(a), 3126(a)(2).
3. This case was argued before our Supreme Court on October 15, 2019, and is awaiting disposition. See Walsh v. BASF Corp., 203 A.3d 976 (Pa. 2019). The issue before our Supreme Court is whether, in conducting a Frye analysis, “trial courts are not permitted to act as ‘gatekeepers’ to ensure the relevance and reliability of scientific studies offered by experts to support their opinions by scrutinizing whether those studies actually support their opinions.” Id. at 978. While the Supreme Court may overturn our Walsh decision, their disposition will have no bearing on the outcome of this case or the general standards triggering the necessity of a Frye hearing that were thoughtfully set forth in the Walsh decision.