A Draft Standard on "Terminology for a Suspected Pattern of Dental Origin"

The Academy Standards Board (ASB) of the American Academy of Forensic Sciences is seeking comments by September 9 on a Technical Report 194, First Edition, 2024, titled "Terminology for a Suspected Pattern of Dental Origin." Although only a "template" for organizing comments is listed on the ASB website, the draft standard can be found at https://www.aafs.org/sites/default/files/media/documents/194_TR_Ballot01.pdf.

The OSAC page on “Standards Open for Comment” (which you won't see by perusing the OSAC website unless you click on “How To Work With Us” in the navigation pane) says “NOTE: This is OSAC 2021-N-0030, Terminology for a Suspected Pattern of Dental Origin, currently on the OSAC Registry.” Does that mean the ASB committee believed that nothing in the OSAC product, which emerged with no review from an advisory scientific and technical panel of experts, needed improvement? It is good to go public as an SDO-approved standard for terminology unless someone objects and proposes something better? 

Alas, it is not that good. Although a discussion of the choice of various terms to define and the definitions themselves could occupy pages, it is too tedious an undertaking for me to write or for many readers to plough through. Suffice it to say that some of the standard has a stream-of-consciousness feeling to it. Like "spurious observation anomaly not intrinsically present feature not related to the source."

The larger question is why try to promulgate a free-floating standard terminology rather than articulate standard procedures with appropriate terms? What might these standard practices be? Presumably, the proposed terminology is a precursor to performing "suspected pattern of dental origin analysis," which is defined as "forensic examination, analysis, and determination of the pattern for potential links to dental origins." This "potential links" study seems to be subdivided into (1) "bitemark assessment analysis," (2) "bitemark analysis," (3) "bitemark comparison analysis," and (4) "bitemark individualization analysis." Can any of these analyses produce results of "evidentiary value" (defined as "information of sufficient usefulness to serve as the basis for making an empirically significant scientific determination")?

The dentists are not prepared to say so. But neither are they willing to list in their bibliography any of the well-known articles and reports concluding that demonstrations of the scientific validity of these analyses are little more than wishful thinking. Rather than claim that these terms refer to procedures that have "evidentiary value" or opine that they lack such value, they merely note that their definitions are "not an endorsement of [the] scientific validity" of the processes they are supposed to describe. Apparently, the drafters from OSAC are agnostics rather than atheists. Or maybe they have their doubts about "bitemark individualization analysis." That phrase is marked "deprecated." Yet, the process of "visual comparison" is not deprecated as unvalidated or invalid, and a note suggests that other "individualization method[s]" for bitemarks could be just dandy.

So the best comment might be a recommendation to jettison this standard. Researchers can use their own clearly defined terms in devising and validating procedures that can be used in criminal investigations that involve what might be toothmarks, bitemarks, or wounds from other sources and mechanisms. When demonstrably valid procedures become available, the time will be ripe for a standard with uniform terminology.

Meanwhile, promulgating these terms and definitions, even with the agnostic disclaimers, risks encouraging the acceptance of dubious forensic "science." The very existence of an expert standard with these terms and definitions might suggest that the words describe something meaningful and encourage testimony that the field has a standardized system of some kind. I can imagine testimony that

The terms I am using in the analysis of what I have determined to be bitemarks are generally accepted in forensic odontology, medicine, and forensic science. I am following the ASB technical report on the subject. The report was produced with funding from the National Institute of Standards and Technology and is included and recommended for adoption by the government-supported Organization of Scientific Area Committees for Forensic Science.

Is this what forensic science and the law needs?

“Predictive or Profiling Evidence” and Diaz v. United States

Today the Association of American Law School’s (AALS) Section on Evidence distributed the following announcement to its members:

In a divided decision, the Supreme Court recently concluded that expert testimony about the likely mental state of individuals arrested with drugs in their position is admissible in criminal trials under the Federal Rules of Evidence in Diaz v. United States, 602 U. S. ____ (2024). This decision was sharply contested in its own right, but also drew attention to an area of broader controversy in the law of evidence: the increasing use of "predictive" or "profiling" evidence, by which expert witnesses present testimony suggesting that an individual is more or less likely to have had a particular mental state or behaved in a particular way based on their personal circumstances or characteristics. Scholars who have written about this phenomenon have expressed disquiet about the use of such evidence (and some courts have limited the use of such evidence in edge cases involving particular prejudice or overbroad characterizations), but no consensus has emerged as to the reasons for objecting to predictive evidence or as to how to systematically distinguish between such evidence and other forms of indirect and circumstantial evidence that is routinely admitted. This panel brings together scholars to discuss predictive and profiling evidence from a variety of perspectives. Was the Supreme Court right to find such evidence consistent with the federal rules governing the admissibility of expert evidence? Is such evidence generally consistent with due process and equal protection concerns? Is it generally empirically sound? Do we need new federal rules or common law doctrines to limit the admissibility of some forms of predictive or profiling evidence?

Who can object to a scholarly program on the subject, even if it is hardly new, having been the subject of a multitude of opinions, a number of statutes, and even a previous AALS program decades ago? Hopefully, the courts and the professoriate have made some progress in understanding what has come to be called "framework evidence." Neither would it be fair to criticize a necessarily brief announcement for not defining "predictive or profiling evidence." Presumably, the evidence teachers do not need a definition.

But I was surprised to read my fellow law professors’ sweeping characterization of Diaz v. United States, 144 S.Ct. 1727 (2024), as establishing that "such evidence [is] consistent with the federal rules governing the admissibility of expert evidence" and "that expert testimony about the likely mental state of individuals arrested with drugs in their position [sic?] is admissible in criminal trials under the Federal Rules of Evidence." Although I doubt that the announcement will cause its narrow range of readers to believe that the case stands for more than it does, I do worry that the same kind of language will crop up in undiscerning judicial opinions and commentary on Diaz. Therefore, it may be be worth listing some problematic aspects of the statements.

First, the case plainly held that certain "expert testimony about the likely mental state of individuals arrested with drugs in their position [possession?] is" not "admissible in criminal trials under the Federal Rules of Evidence." Every Justice agreed that no expert can testify that a defendant charged with importing proscribed drugs knew that they were transporting drugs. That would be explicit ultimate-opinion testimony on a criminal defendant's state of mind. (There are cases limiting the Rule 704(b) ban to mental health professionals, but the Court did not consider that possible way to interpret the rule. It stuck with a more literal reading of the text.)

Second, no “predictive or profiling evidence” was introduced in the case, and only one Justice thought it worth discussing. Certainly, Diaz is not a case of a criminal profiler predicting (“inferring” would be more precise) the characteristics of a criminal from the type or manner of the crimes under investigation (or any other such “predictive or profiling evidence”). As the excerpts from the trial transcript reproduced below show, the witness did not claim such expertise; furthermore, the trial judge barred him from stating a belief about the defendant’s knowledge (although one could well think that his testimony made it plain enough what his belief was). 

Third, the issue before the Court was not admissibility under the rules of evidence writ large. It was the scope of a single part of a solitary rule. Federal Rule 704(b), which has no counterpart in the rules of most states, declares that no expert witness may "state an opinion about whether the defendant did or did not have a mental state or condition that constitutes an element of the crime charged or of a defense" because "[t]hose matters are for the trier of fact alone." All that the Diaz Court held was that when such an opinion is not stated and is an inference that does not necessarily follow (as a matter of deductive logic) from the witness’s statements, then Rule 704(b) does not preclude its admission. It did not—and could not—have held that the Rule makes it admissible. Cf. David H. Kaye, The Ultimate Opinion Rule and Forensic Science Identification, 60 Jurimetrics J. 75 (2020).

Thus, Diaz should not be read as supporting—or opposing—the use of “predictive or profiling evidence” generally, or even in the subcategory of testimony offered to prove a defendant’s state-of-mind. 

More thoughts on the three opinions in the case and how the result fits into the range of possible interpretations of Rule 704(b) will appear in the upcoming supplement to The New Wigmore on Evidence: Expert Evidence § 2.2.3(b) (available online in VitalLaw) and, time permitting, in a further positing about the case here.

---

Excerpts from Trial Transcript
Mar. 18, 2021

The most pertinent portion of the testimony at issue in Diaz is as follows (with italics added):

BY MR. OLAH [Assistant US Attorney]:
Q. Where do you work?
A. I’m a special agent with Homeland Security Investigations.
Q. And how long have you been with HSI?
A. I’ve been a special agent since 1996. So going on 20 — I believe 28 years.
Q. Were you in law enforcement before joining HSI?
A. Prior to becoming a special agent, I was a U.S. Border Patrol agent. And prior to that, I was a sheriff’s corrections deputy.
***
Q. Have you been involved in drug trafficking investigations as a special agent with HSI?
A. Yes, I have.
Q. Approximately how many such investigations?
A. I’ve been involved in over 500 investigations dealing with distribution of drugs and also the – which would include the importation of drugs.
Q. And can you summarize for the jury the various investigation techniques you’ve used?
A. The techniques I’ve used, I’ve utilized wiretaps, where you actually listen to a drug trafficker talk on the telephone and how they conduct business. I’ve done controlled purchases where I utilized an undercover agent or a cooperating source. And we actually go out on the street and buy the drugs. I’ve spoken with cooperating defendants that have been arrested for drug trafficking related offenses. I’ve talked to cooperating sources that have information related to the distribution of drugs and drug trafficking organizations. I have spoken with other agents that work drug trafficking organizations and have worked on task forces with other agencies such as the Federal Bureau of Investigation, the Drug Enforcement Administration, and local police departments dealing with drug trafficking related crimes.
***
Q. Agent Flood, why are drugs imported into the United States?
MS. IREDALE: Objection, 401.
THE COURT: Overruled.
THE WITNESS: Based upon drugs that – some drugs are manufactured in Mexico and outside the United States. Therefore, they’re brought across the border, into the United States to be sold.
BY MR. OLAH:
***
Q. With respect to vehicles, can you describe the general process of movement from Mexico to wherever it goes?
A. From Mexico, they are packaged. They are put into *** a vehicle. *** I have seen drugs hidden in every area of a vehicle. *** And then they are transported from point A to point B across the border.
Q. And based on your training and experience, are the transporters compensated for their efforts?
A. Yes. It’s a job. It’s to take it from point A to point B.
***
Q. Agent Flood, based on your training and experience, are large quantities of drugs entrusted to drivers that are unaware of those drugs?
MS. IREDALE: Objection. 401, 403.
THE COURT: Overruled.
THE WITNESS: No. In extreme circumstances – actually, in most circumstances, the driver knows they are hired. It’s a business. They are hired to take the drugs from point A to point B.
BY MR. OLAH:
Q. And why aren’t – why don’t they use unknowing couriers, generally?
MS. IREDALE: Objection. 401, 403.
THE COURT: Overruled. You may answer.
THE WITNESS: Generally, it’s a risk of your – your cargo not making it to the new market; not knowing where it’s going; not being able to retrieve it at the ending point, at your point B. So there’s a risk of not delivering your product and, therefore, you’re not going to make any money.
***
Cross Examination
***
Q. So you said that unknowing couriers are very rare.
A. Yes.
***
Q. You work for HSI. Right?
A. Correct.
Q. And you’re aware that your own agency has identified many schemes where drug trafficking organizations use unknowing couriers. Right?
A. I – I know of three schemes that were primarily identified as being possible for an unknowing courier. It doesn’t necessarily mean that they are unknowing couriers. ***

Volunteer Bias in Interlaboratory Studies

The National Institute for Standards and Technology (NIST) is soliciting laboratories to join an "interlaboratory study" of CG-MS (gas chromatography mass spectrometry) for seized drug analysis. The announcement that I received, in abbreviated form, reads:

Forensic Science Quality Assurance Program
Seized Drugs General Method for GC-MS Reporting Limits Study

. . .
Study Design, Purpose, and Rationale
The goals of the study are 1) to capture the range of methods, instrumentation, and analytical approaches used in the community, 2) investigate mass spectral variability across methods, and 3) investigate how different reporting practices effect the limit of seized drug reporting.
Timeline and Commitment
Registration is currently open and will close on July 5, 2024. To participate in this study, laboratories must be accredited forensic laboratories based in the United States and have a valid Schedule I & II DEA license, a validated seized drug screening method using GC-MS, and a documented reporting practice. To be considered for the study, participants will be required to complete a pre-study questionnaire pertaining to the method that will be used for sample analysis. After acceptance into the study, participants will be provided a kit of 10 solutions containing mixtures of controlled substances and asked to analyze the solutions and report whether the analytes are present above their established reporting thresholds. Participants will also be required to report chromatographic peak height/area and retention time of each peak and provide the raw datafile from each run. Standards used for comparison will also be reported. [D]ue to a limited number of available kits, completion of the pre-study questionnaire does not guarantee acceptance into the study. . . .
Publication of Results
Upon closure of data entry, laboratories will receive a preliminary report containing a summary of reported data, consensus results, and a summary of analytes present in each mixture. [A] final report . . . will be made publicly available by Spring 2025. . . . NIST will not knowingly reveal laboratory identities associated with study results.
For questions, contact andrea.yarberry@nist.gov
To signup, go to: https://forms.gle/gPDU9aENHguPkw1D7

The effort is laudable, but one might ask why NIST is not beginning with a sampling frame of laboratories created to represent "the community" and then drawing a probability sample from this list. Will the laboratories that notice the announcement and ask to participate present the full "range of methods, instrumentation, and analytical approaches used in the community"? Will the volunteer sample be skewed toward higher quality labs? Will it include all the "different reporting practices [that] effect [sic] the limit of seized drug reporting [whatever this "limit" denotes in the population of laboratories]"?

Rigor in sampling may not be required to answer certain questions, but it seems relevant to determining what the sign-up form refers to as "the current landscape of GC-MS methods and associated reporting practices and how those factors effect [sic] the concentration of drug that is/is not ultimately reported." Certainly, it should be a consideration for the legal community if and when the results of the study are presented as an indication of "the known or potential rate of error" for GC-MS analysis as practiced in forensic-science laboratories (Daubert v. Merrell Dow Pharm., 509 U.S. 579, 594 (1993)).

ISO Standards on Forensic Science: Pay to Play?

"ISO, the International Organization for Standardization, brings global experts together to agree on the best way of doing things – for anything from making a product to managing a process." ^1/ For the last few years, it has been devising the following overarching set of standards for all of forensic science:

• Forensic sciences (TC 272) ISO/DIS 21043-1, Forensic sciences - Part 1: Terms and definitions - 5/27/2024, $58.00
• ISO/DIS 21043-3, Forensic Sciences - Part 3: Analysis - 5/26/2024, $62.00
• ISO/DIS 21043-4, Forensic Sciences - Part 4: Interpretation - 5/26/2024, $67.00
• ISO/DIS 21043-5, Forensic Sciences - Part 5: Reporting - 5/26/2024, $53.00 ^2/

These are

part of a series which, when completed, will include the different components of the forensic process from scene to courtroom ... . The series describes primarily “what” is standardized, not the “how” or “who”. Best practice manuals and standard operating procedures should describe “how” the requirements of this document would be met. ^3/

It sounds like the standards in progress will not specify "the best way of doing things." Will they merely list the things that are in need of "standardization"? Will they be too open-ended to constitute what the U.S. Supreme Court refers to as "standards controlling the technique's operation"^4/?

I cannot answer these questions because I have not seen the drafts that were open for public comment. Members of the public cannot read the drafts without paying IS0 the $240 listed above. If anyone who has paid to play has thoughts on these documents that they would like to share beyond the TC (Technical Committee) that drafted the standards, I'll post them--at no charge.

Notes

1. Int'l Org. for Standardization, About ISO.
2. ANSI Standards Action, Mar. 15, 2024, at 48.
3. ISO 21043-1:2018(en) Forensic sciences — Part 1: Terms and definitions.
4. Daubert v. Merrell Dow Pharm., 509 U.S. 579, 594 (1993).

What's Uniqueness Got to Do with It?

Columbia University has announced that "AI Discovers That Not Every Fingerprint Is Unique"! The subtitle of the press release of January 10, 2024, boldly claims that

Columbia engineers have built a new AI that shatters a long-held belief in forensics–that fingerprints from different fingers of the same person are unique. It turns out they are similar, only we’ve been comparing fingerprints the wrong way!

Forensic Magazine immediately and uncritically rebroadcast (quoting verbatim without acknowledgment from the press release) the confused statements about uniqueness. According to the Columbia release and Forensic Magazine, "It’s a well-accepted fact in the forensics community that fingerprints of different fingers of the same person—or intra-person fingerprints—are unique and therefore unmatchable." Forensics Magazine adds that "Now, a new study shows an AI-based system has learned to correlate a person’s unique fingerprints with a high degree of accuracy."

Does this mean that the "well-accepted fact" and "long-held belief" in uniqueness been shattered or not? Clearly, not. The study is about similarity, not uniqueness. In fact, uniqueness has essentially nothing to do with it. I can classify equilateral triangles drawn on a flat surface as triangles rather than as other regular polygons whether or not the triangles are each different enough from one another (uniqueness within the set of triangles) that I notice these differences. To say that objects "are unique and therefore unmatchable" is a nonsequitur. A human genome is probably unique to that individual, but forensic geneticists know that six-locus STR profiles are "matchable" to those of other individuals in the population. A cold hit to a person who could not have been the source of the six-locus profile in the U.K. database occurred long ago (as was to be expected for the random-match probabilities of the genotypes).

Perhaps the myth that the study shatters is that it is impossible to distinguish fingerprints left by different fingers of the same individual X from fingerprints left by fingers of different individuals (not-X). But there is no obvious reason why this would be impossible even if every print is distinguishable from every other print (uniqueness).

The Columbia press release describes the study design this way:

[U]ndergraduate senior Gabe Guo ... who had no prior knowledge of forensics, found a public U.S. government database of some 60,000 fingerprints and fed them in pairs into an artificial intelligence-based system known as a deep contrastive network. Sometimes the pairs belonged to the same person (but different fingers), and sometimes they belonged to different people.

Over time, the AI system, which the team designed by modifying a state-of-the-art framework, got better at telling when seemingly unique fingerprints belonged to the same person and when they didn’t. The accuracy for a single pair reached 77%. When multiple pairs were presented, the accuracy shot significantly higher, potentially increasing current forensic efficiency by more than tenfold.

The press release reported the following odd facts about the authors' attempts to publish their study in a scientific journal:

Once the team verified their results, they quickly sent the findings to a well-established forensics journal, only to receive a rejection a few months later. The anonymous expert reviewer and editor concluded that “It is well known that every fingerprint is unique,” and therefore it would not be possible to detect similarities even if the fingerprints came from the same person.

The team ... fed their AI system even more data, and the system kept improving. Aware of the forensics community's skepticism, the team opted to submit their manuscript to a more general audience. The paper was rejected again, but [Professor Hod] Lipson ... appealed. “I don’t normally argue editorial decisions, but this finding was too important to ignore,” he said. “If this information tips the balance, then I imagine that cold cases could be revived, and even that innocent people could be acquitted.” ...

After more back and forth, the paper was finally accepted for publication by Science Advances. ... One of the sticking points was the following question: What alternative information was the AI actually using that has evaded decades of forensic analysis? ... “The AI was not using ... the patterns used in traditional fingerprint comparison,” said Guo ... . “Instead, it was using something else, related to the angles and curvatures of the swirls and loops in the center of the fingerprint.”

Proprietary fingerprint matching algorithms also do not arrive at matches the way human examiners do. They "see" different features in the patterns and tend to rank the top candidates for true matches in a database trawl differently than the human experts. Again, however, these facts about automated systems neither prove nor disprove claims of uniqueness. And, theoretical uniqueness has little or nothing to do with the actual probative value of assertions of matches by humans, automated systems, or both.

Although not directly applicable, the day after the publicity on the Guo et al. paper, I came across the following report on "Limitations of AI-based predictive models" in a weekly survey of papers in Science:

A central promise of artificial intelligence (AI) in health care is that large datasets can be mined to predict and identify the best course of care for future patients. Unfortunately, we do not know how these models would perform on new patients because they are rarely tested prospectively on truly independent patient samples. Chekroud et al. showed that machine learning models routinely achieve perfect performance in one dataset even when that dataset is a large international multisite clinical trial (see the Perspective by Petzschner). However, when that exact model was tested in truly independent clinical trials, performance fell to chance levels. Even when building what should be a more robust model by aggregating across a group of similar multisite trials, subsequent predictive performance remained poor. -- Science p. 164, 10.1126/science.adg8538; see also p. 149, 10.1126/science.adm9218

Note: This posting was last modified on 1/12/24 2:45 PM