Showing posts with label source code. Show all posts
Showing posts with label source code. Show all posts

Saturday, November 9, 2019

State v. Simmer: The First High Court Opinion on "Probabilistic Genotyping"

Last week, the Nebrasks Supreme Court affirmed the admission of testimony about what has been dubbed "probabilistic genotyping." Twelve years ago, in 2007, Joy Blanchard was discovered on the floor of her home. Two knives protruded from her neck. A spindle broken from the banister lay nearby. The cause of death was blunt force trauma and stab wounds to the head and neck. Investigators swabbed the spindle, the handles on both knives, and the interior front door knob. Nine years passed before DNA testing indicated the presence of DNA from Joy's nephew, Charles Simmer, on one of the knife handles and the door knob.

Charged with first-degree murder Simmer filed a pretrial motion to preclude “any and all testimony concerning DNA testing and the results of said testing,” including “identification” and “comparison.” After an evidentiary hearing, the district court denied the motion. At the jury trial, the state presented both "probabilistic genotyping" from Dr. Mark Perlin, the inventor of a complex software program named TrueAllele, that "the match between the knife handle and Simmer was 3.71 thousand times more probable than a coincidental match to an unrelated person, while the match between the doorknob and Simmer was 4.22 quintillion times more probable than a coincidental match to an unrelated person." Simmer was convicted and sentenced to life imprisonment.

What made the "genotyping" "probabilistic" is not that genes were typed. The STRs used in the identification are not genes. They are repeated, noncoding sequences of DNA base pairs (such as TAGATAGATAGATAGATAGATAGATAGA) that are sprinkled within and between genes. Neither Is the typing "probabilistic" because probabilities were quoted. Ordinary DNA typing results are accompanied by "random match probabilities."

Rather, unlike ordinary DNA profiling of a single-source samples, these probabilities were not derived from any one individual's features as inferred by a human examiner. The samples were mixtures of DNA from (at least) two men, and no one could say which two individual profiles were in the mixture. The computer program, however, could estimate the probability of the mixed-up collection of DNA features displaying the measured "peak heights" when coming from DNA from several individuals including the defendant as opposed to the probability of the same data coming from DNA from individuals not including the defendant.

Dr. Perlin testified to the ratio of those probabilities (more technically, of those probability densities) using the confusing phrase "match probability" when in fact there was no simple match. He framed the question as "How much more does the suspect match the evidence at this location than a random person?" But any random profile either would match -- in the sense of having the same alleles as those giving rise tom the observed peaks -- or it would not. Because of the ambiguity as to the true profiles in the mixture, the common-sense notion of a match -- of inclusions to and exclusions from -- the class of possible contributors to the mixture is not part of the logic of "probabilistic geneotyping."

Be that as it may, the issue on appeal in State v. Simmer, 304 Neb. 369 (2019), was not how to present the output of the software. It was whether that output could be the subject of any testimony at all. The Nebraska Supreme Court ran through the usual "Daubert factors." Despite arguments about the type of validation studies, the court was impressed with the sheer number of studies and the length of time that the system has been used.

The defendant's expert was Nathaniel Adams, who "has a bachelor’s degree in computer science and is employed by a company that 'consult[s] with lawyers.'" Mr. Adams testified (correctly) that one cannot directly validate the accuracy of particular likelihood ratios and that the software had not been developed and tested from the standpoint of software engineering, which, he suggested, requires independent study of source code over long periods of time. The Supreme Court was not impressed. It held that validation "from a software engineering perspective" was not essential and that the publication of research on the performance of the software (as best as that could be measured for known mixtures and via simulations) was sufficient.

The defense also maintained that "the validation studies must be discounted because Perlin is a coauthor of some of the publications and, as the owner of the company that owns. TrueAllele, has a financial interest in seeing it found reliable" To this, the Supreme Court responded that there were plenty of studies in which Perlin was not an author, but it also suggested that the source of validation studies is irrelevant because that fact pertains only to credibility:
In other words, Simmer is contending that there are reasons to question Perlin’s credibility. Even assuming that is true, an attack on the credibility of an expert witness is not aimed at the expert’s reasoning or methodology and is thus not the proper basis for a Daubert/Schafersman challenge. See Smith v. Colorado Organ Recovery Sys., 269 Neb. 578, 694 N.W.2d 610 (2005) (stating that challenge to expert witnesses' credibility is not Daubert/Schafersman claim).
This cannot be correct. Although the credibility of a particular witness certainly is fair game at trial, Independent replication of experiments is an important factor in scientific practice. The absence of validation from independent researchers should count against admissibility, although it would not necessarily be dispositive under Daubert.

Another questionable feature of the opinion lies in its discussion of the PCAST Report. In 2016 and 2017, the President's Council of Advisors on Science and Technology evinced dissatisfaction with the extent to which probabilisitc genotyping as applied in some situations had been validated. The software developers disputed the report -- but later responded with additional studies. The peculiarity in Simmer is not that the court determined that the output of TrueAllele was admissible despite PCAST's reservations. It is that the Nebraska Supreme Court refused to read the PCAST Report because it "is not in [the] record [on appeal]." Instead, it looked only at an obscure and incomplete "Washington trial court order containing a description of the report." On that basis, it wrote that "with no scientific evidence in the record indicating that TrueAllele could not generate reliable results under the circumstances here, we cannot say that a conclusion that TrueAllele could reliably be applied to the facts of this case would amount to an abuse of discretion."

But why was appellant was "unable to cite directly to the report"? It is a duly published report of a prestigious advisory body. Appellate lawyers and courts frequently cite books, articles, and reports about scientific research and policy. National Academy reports and Office of Technology Assessment reports have long been cited and discussed in opinions on the admissibility of DNA evidence. They address "legislative facts." Legislative facts apply across cases." In contrast, "adjudicative facts" are not general truth; they are the how, what, where, and when of specific cases.

Under the Nebraska Rule of Evidence (see, e.g., Neb. R. Evid. 201(a)), only adjudicative facts need to be proved at trial. Both appellate and trial courts have the power to consult published scientific opinions on the limitations of a scientific method. Here, the alleged limitation is that the software was not shown to work well with mixtures that contain very small fractions of a suspect's DNA. Courts should be more open to receiving information in the scientific and related literature on such issues.

Saturday, March 10, 2018

Exposing Source Code in Law and Science

With the increasing use of "probabilistic genotyping software" has come a push for the making the software public. 1/ Indeed, one federal judge in the Southern District of New York went so far as to compel New York City's Office of the Chief Medical Examiner to make public the source code of the program it created to interpret the pattern of peaks and valleys in the graphs used to determine the nature of the DNA giving rise to them. 2/

Whether this will produce a more thorough analysis of how well the software performs (compared to other ways of testing the program, including releasing the code for inspection by defense experts subject to protective orders) remains to be seen. I have heard more than one expert involved in developing such software observe that releasing the source code is not going to do much to help anyone understand how well the program works. Very lengthy and complex programs may not consist entirely of modules that can be tested separately, and it can be difficult for testers who are not developing the software to develop the insight to conduct effective white-box testing. As far as I know, the FAA and the FDA do not insist on disclosure of all source code to approve the use of avionics and medical-device software, respectively.

Thus, a new editorial policy announced by a preeminent scientific journal, Nature, is noteworthy. 3/ It expresses a strong preference — but not an absolute requirement — for making the source code available to peer reviewers and then to the scientific community. Excerpts follow:
... From this week, Nature journal editors handling papers in which code is central to the main claims or is the main novelty of the work will, on a case-by-case basis, ask reviewers to check how well the code works .... Computational science — like other disciplines — is grappling with reproducibility problems, partly because researchers find it difficult to reproduce results based on custom-built algorithms or software. ...

Some journals have for years ... ensured that the new code or software is checked by peer reviewers and published along with the paper. When relevant, Nature Methods, Nature Biotechnology and, most recently, journals including Nature and Nature Neuroscience encourage authors to provide the source code, installation guide and a sample data set, and to make this code available to reviewers for checking. ...

According to the guidelines, authors must disclose any restrictions on a program’s accessibility when they submit a paper. [I]n some cases — such as commercial applications — authors may not be able to make all details fully available. Together, editors and reviewers will decide how the code or mathematical algorithm must be presented and released to allow the paper to be published.

... We also recognize that preparing code in a form that is useful to others, or sharing it, is still not common in some areas of science.

Nevertheless, we expect that most authors and reviewers will see value in the practice. Last year, Nature Methods and Nature Biotechnology between them published 47 articles that hinged on new code or software. Of these, approximately 85% included the source code for review.

[A]lthough many researchers already embrace the idea of releasing their code on publication, we hope this initiative will encourage more to do so.
Notes
  1. See Jason Tashea, Code of Silence: Defense Lawyers Want to Peek Behind the Curtain of Probabilistic Genotyping, ABA J., Dec, 2017, at 18. The article asserts that "companies developing these tools 'black-box.' This means there is limited or no capacity to review the math;therefore, it cannot be independently challenged." Id.at 19. In computer scientists and engineering, however, "black box" is not a pejorative term, and it certainly does not mean that no testing is possible. Quite the contrary, it refers to a type of testing (also known as "behavioral testing") that "independently challenges" the ;program. It does so by checking whether the program's output is correct for different inputs. This type of testing has advantages and disadvantages compared to "white box" testing. Obviously, a combination of both types of testing is more complete than either one by itself.
  2. E.g., Jason Tashea, Federal Judge Releases DNA Software Source Code That Was Used by New York City's Crime Lab, ABA J., Oct. 20, 2017, http://www.abajournal.com/news/article/federal_judge_releases_dna_software_source_code. The article asserts that "[p]robabilistic genotyping does not define a DNA sample itself; rather, it is an interpretive software that runs multiple scenarios, like risk analysis tools used in finance, to analyze the sample." The explanation could be clearer. The New York City OCME's program that was the subject of the judge's order models the stochastic process that produces the "peaks" that indicate the presence of certain features of the DNA in the sample. See David H. Kaye, SWGDAM Guidelines on "Probabilistic Genotyping Systems" (Part 2), Forensic Sci., Stat. & L., Oct. 25, 2015, http://for-sci-law.blogspot.com/2015/10/guidelines-on-probabilistic-genotyping.html. 
  3. Editorial, Does Your Code Stand Up to Scrutiny?, 555 Nature 142 (2018), https://www.nature.com/articles/d41586-018-02741-4?WT.ec_id=NATURE-20180309.