Blinding Forensic Analysts to Task-irrelevant Information: A National Commission (NCFS) Speaks Out

This week, the National Commission on Forensic Science (NCFS) approved a “views document” entitled Ensuring That Forensic Analysis Is Based Upon Task-Relevant Information. 1/ If these views are translated into practice, it will be a major step forward in making sure forensic science findings are based on scientific data and not on extraneous information. The document is thus cause for celebration.

Here, I describe how the document defines task-relevance. I identify an arguable inconsistency in the Commission’s terminology and elaborate on the use of what is known in probability theory as conditional independence.

The NCFS’ views are these:

1. FSSPs [Forensic Science Service Providers] should rely solely on task-relevant information when performing forensic analyses.
2. The standards and guidelines for forensic practice being developed by the Organization of Scientific Area Committees (OSAC) should specify what types of information are task-relevant and task-irrelevant for common forensic tasks.
3. Forensic laboratories should take appropriate steps to avoid exposing analysts to task-irrelevant information through the use of context management procedures detailed in written policies and protocols.

The analysis and explication that follows this enumeration tries to define task-relevance both in words and in symbols involving conditional probabilities. The NCFS definition is in two parts:

(1) [I]nformation is task-relevant for analytic tasks if it is necessary for drawing conclusions: (i.) about the propositions in question, (ii.) from the physical evidence that has been designated for examination, (iii.) through the correct application of an accepted analytic method by a competent analyst.
(2) Information is task-irrelevant if it is not necessary for drawing conclusions about the propositions in question, if it assists only in drawing conclusions from something other than the physical evidence designated for examination, or if it assists only in drawing conclusions by some means other than an appropriate analytic method.

Taken literally, this formulation seems to dismiss as task-irrelevant information that could help the analyst assess the strength of the evidence yet is not necessary for drawing conclusions about the propositions.  For example, suppose the proposition P in question is whether a trace sample that has both clear and ambiguous features came from a suspect. Viewing a tape of someone who looks like (and thus might be) the suspect leaving the mark is task-irrelevant under (2). No analyst needs to view the tape to compare the questioned mark to a known exemplar. The analyst can reach a conclusion of some sort without the video.

Nevertheless, viewing the tape could help the analyst doing a side-by-side comparison resolve the ambiguities in the features in the mark and thereby “assess the strength of the inferential connection between the physical evidence being examined and the propositions the analyst is evaluating.” For example, if the mark is a distorted fingerprint, observing how it was deposited might help the analyst. It seems as if the tape should be declared task-relevant, but (1) requires that it be necessary to the analysis. Strictly speaking, it is not.

Indeed, a few sentences later, the document states that information “is task-relevant if it helps the analyst assess the strength of the inferential connection between the physical evidence being examined and the propositions the analyst is evaluating.” Not everything that is helpful is necessary.

That the Commission did not really mean to require necessity also can be gleaned from the “more formal definition of task-relevance and task-irrelevance ... in the technical appendix.” For “two mutually exclusive propositions P and NP that a forensic science service provider (FSSP) is asked to evaluate,” and for E defined as “the features or characteristics of the physical evidence that has been designated for examination,”

(1) information is task-relevant if it has the potential to assist the examiner in evaluating either the conditional probability of E under P—which can be written p(E|P)—or the conditional probability of E under NP—which can be written p(E|NP);
(2) information is task-irrelevant if it has no bearing on the conditional probabilities p(E|P) or p(E|NP).

Again, necessity is not crucial: The phrase “has the potential to assist” has been substituted for “is necessary,” and “has no bearing” has replaced “is not necessary.”

Technical definitions (1) and (2) also depart from (or refine) the main definitions (1) and (2) in that the only “conclusions” that can be considered in judging task-relevance are conditional probabilities for “features” given certain propositions. These conditional probabilities often are called “likelihoods” to distinguish them from the posterior probabilities of the propositions given the features. Traditionally, analysts testified about posterior probabilities expressed qualitatively or categorically. For example, the statement P that “Jane Doe’s thumb is the source of the latent print” is a categorical conclusion meaning that the posterior probability Pr(P|E) is close to 1.

Using likelihoods could be valuable in clarifying task-relevance, but the concepts of “bearing” and “potential to assist” remain undefined. It would seem that the NCFS intends to equate task-irrelevance with conditional independence. Let I denote the information that might be task-irrelevant. E and I are conditionally independent given some proposition R if and only if (iff) Pr(E&I|R) = P(E|R) P(I|R). An equivalent definition looks to whether Pr(E|I&R) = Pr(E|R). The idea is that once R is known, knowing I brings no additional information about E.

If we take conditional independence to be the NCFS’ technical definition of task-irrelevance, and we use R to stand for either P or NP, then we can rewrite the NCFS definitions as

(3) I is task-relevant iff Pr(E|I&R) ≠ Pr(E|R);
(4) I is task-irrelevant iff Pr(E|I&R) = Pr(E|R).

This more precise definition is easier to write than to apply. The “physical evidence” itself — bits of soil, specimens of handwriting, latent and rolled fingerprints, and so on — is not “E” in the probability function. Instead, E is “the features or characteristics of the physical evidence.” But are these the actual features or the declared features?

I think the formal definition works better when E refers to the true features (although the judge or jury only knows what the expert thinks they are). First, let’s look at an easy case. Suppose that E refers to the DNA alleles present at each locus in the suspect’s DNA (A0) and the profile in the crime-scene DNA (A1). Thus, E = A0 & A1. P means that the suspect is the source of both DNA samples; let Q mean that someone else is. I is a credible report that the suspect was near the crime scene just after the crime occurred. Finally, suppose that A0 and A1 are the same — the DNA in both samples have the same true features.

If P is true, then the samples must have the same features, so Pr(E|P) = Pr(E|I&P) =1. If Q is true, then whether the samples have the same features also does not depend on I — if someone else left the DNA, the suspect’s propinquity does not affect the alleles that the true contributor possesses and left at the crime-scene. Consequently, under (4), I is task-irrelevant, just as it should be.

Now, let’s make it more complicated. The laboratory is asked to assess whether a suspect’s “touch” DNA is present on a gun used in a killing. Several small peaks in the electropherogram are at the positions one would expect if this were the case, but they are at the limit of detectability. Some analysts would treat them as real (true peaks), but others would see them as spurious. The question is whether the analyst should be able to know the profile reported for the suspect — let’s call it r[A0] — before ascertaining the profile A1 in the crime-scene sample. Is I = r[A0] task-relevant to the determination of A1?

Some analysts might argue that I is task-relevant because knowing what is in the suspect’s DNA helps them understand what really is in the crime-scene DNA. They could say that the fact that the small peaks in the crime-science sample are located at just the same places as their larger homologs in the suspect’s sample helps them resolve the ambiguity arising from the small peak heights. Of course, if they are thoughtful, they also will recognize that the related information I could bias them, and they might well agree that they should not be exposed to it because it does not contribute enough to the accuracy of their determinations. But are they wrong in their claim that I is task-relevant (applying the NCFS definition)?

The views document does not give an explicit answer. It concludes with the observation that 

[Task-irrelevant information] might help the analyst draw conclusions about the propositions, but it does not help the analyst draw conclusions from the physical evidence that has been designated for examination through correct application of an accepted analytic method. Any inferences analysts might draw from the task-irrelevant information involve matters beyond their scientific expertise that are more appropriately considered by others in the justice system, such as police, prosecutors, and jurors.

This relative-expertise criterion, however, does not quite define task-irrelevance. The inference that a small peak in an electropherogram is the result of chemiluminescence from alleles as opposed to an artifact or background noise may be difficult to make correctly, but it is not clear that it is lies more squarely within the expertise of police, prosecutors, and jurors than of DNA analysts. 2/

The formal definition can help us out here. If P is true, then regardless of what the peaks look like and irrespective of the suspect’s reported profile r[A0], the true profile of the crime-scene sample is A1 = A0. Thus, Pr(E=A1|P) = Pr(A1|P&I) = Pr(A1|P&r[A0]) = 1. Likewise, if someone else’s DNA is on the gun instead of the suspect’s, then the probability that the profiles match also is unrelated to a report of what is in the suspect’s DNA sample. Once again, the conditional-independence definition of task-irrelevance seems to work.  Sometimes probability notation is purely window dressing, but the approach begun in the technical appendix might do some useful work in spotting task-irrelevant information. 3/

Notes

1. The document should appear on the Commission’s web page http://www.justice.gov/ncfs/work-products-adopted-commission in the near future. The principal drafter of the document was Bill Thompson, who is the chair of the Human Factors Resource Committee of the NIST Organization of Scientific Area Committees (OSACs) that is developing standards for forensic science.
2. One can argue that looking at the suspect's profile or peaks before resolving ambiguities in the crime-scene profile is not a "correct application of an accepted analytic method." However, given that the task is to ascertain and compare the two DNA samples, it seems odd to call this information about the profiles "irrelevant" as opposed to improper or not acceptable. And, if there were no standard in place rejecting this practice (as was true for a period of time), this criterion would not render the information task-irrelevant.
3. This is so even though the likelihoods involved are not necessarily the ones that determine the probative value of the forensic analysis with respect to the two competing hypotheses P and Q. Those likelihoods are Pr(E*|P) and Pr(E*|Q). The asterisk is attached because we do not know the true features E in the samples. We have data E* on them (fallible measurements or observations of them). The probative value of E* (or, if you like, of the analysis that generates these data) is the likelihood ratio Pr(E*|P) / Pr(E*|Q). For example, even though we can speak of the likelihood ratio for the true DNA profiles (A1 & A0) for present purposes, the court’s evidence is the reported profiles: E* = r[A1 & A0].