More on Semrau: The Other Daubert Factors

In United States v. Semrau, the U.S. Court of Appeals for the Sixth Circuit upheld the exclusion of a defendant’s “unilateral” fMRI testing for conscious deception. Previously, I focused on the court’s discussion of error rates. Known error rates implicate admissibility under both Federal Rule of Evidence 702 and Federal Rule 703. (Rule 702 is the locus of the scientific validity standard adopted in Daubert v. Merrell Dow Pharmaceutics, and Rule 703 states the common law, ad hoc balancing test for virtually all evidence.) I do not think the opinion is as clear as it could have been on which error rate pertained to what. Nevertheless, it is encouraging that the court recognized that two parameters are necessary to describe the accuracy of a procedure that classifies items or people into two categories (liar or truth teller).

But Daubert's list of factors extends beyond error rates, and the Semrau court’s handling of the other Daubert subissues also merits a mixed review. First, the court suggested that fMRI lie detection satisfied Daubert’s criteria for testing and peer review. It referred to “several factors in Dr. Semrau's favor,” namely:

“[T]he underlying theories behind fMRI-based lie detection are capable of being tested, and at least in the laboratory setting, have been subjected to some level of testing. It also appears that the theories have been subjected to some peer review and publication.” Semrau, 2010 WL 6845092, at *10. The Government does not appear to challenge these findings, although it does point out that the bulk of the research supporting fMRI research has come from Dr. Laken himself.

The suggestion that these factors favor the defendant treats Daubert’s references to testing, peer review, and publication rather superficially. That a scientific theory is “capable of being tested” tells us almost nothing about the validity of the theory. The theory that in the year 2075, the moon will turn into a blob of green cheese is capable of being tested, but that does not help validate it today. Likewise, the mere existence of peer reviewed publications means nothing without examining the content of the publications and the reactions to them in the scientific literature.

The court came closer to addressing the true Daubert issue of whether peer reviewed publications,  considered as a whole, validate a technique or theory when it responded to defendant’s argument that the district court was overly concerned with the realism of validity studies.  In that context, the court of appeals quoted the caveat in one fMRI study that:

This study has several factors that must be considered for adequate interpretation of the results. Although this study attempted to approximate a scenario that was closer to a real-world situation than prior fMRI detection studies, it still did not equal the level of jeopardy that exists in real-world testing. The reality of a research setting involves balancing ethical concerns, the need to know accurately the participant's truth and deception, and producing realistic scenarios that have adequate jeopardy.... Future studies will need to be performed involving these populations.

But even this mention of the content of one study does not explain why the experiments are inadequate to demonstrate validity. Why would it be harder to detect a lie that has grave consequences to the subject of the laboratory experiment or field study than one that has more trivial consequences?

Second, the court of appeals wrote that the “controlling standards factor” had not been satisfied because “[w]hile it is unclear from the testimony what the error rates are or how valid they may be in the laboratory setting, there are no known error rates for fMRI-based lie detection outside the laboratory setting, i.e., in the ‘real-world’ or ‘real-life’ setting.” But what does the realism of laboratory experiments have to do with the existence of a clear protocol for gathering and interpreting data? Naturally, if a test is not standardized, it is hard to ascertain its error rate—a point that has been prominent in debates over fingerprinting. And, if the tester departs slightly from the standard test protocol, the probative value of the test should be questioned under Rule 403. But the issue of external validity should not be confused with the issue of whether standards are in place for administering a test.

Finally, the court implied that without realistic field testing, there could be no general scientific acceptance of a method of lie detection in the forensic setting. This may be true, but all empirical studies pertain to particular times, places, and subjects. Deciding what generalizations are reasonable or generally accepted depends on understanding the phenomena in question. Can laboratory experiments alone show that certain factors tend to affect the accuracy of eyewitness identifications? For years, many experimental psychologists seemed willing to accept forensic applications of laboratory results that lacked complete realism. The ability of fingerprint examiners to match true pairs of prints and exclude false pairs of prints can be demonstrated in laboratory studies with artificially created pairs. Applying the error rates from such laboratory experiments to actual forensic setting could well be hazardous, but the experiments still prove that there is information that analysts can use to make valid judgments. In that situation, it is doubtful that raising the stakes of a judgment will render the technique invalid.

The Semrau court does not pinpoint the source of its discomfort with pure laboratory experiments. As we have just seen, a court should not assume that laboratory experiments never can establish validity of a technique as applied to casework. However, in the case of fingerprint identification, it seems clear enough that the prints do not change depending on whether they are deposited in the course of a crime or produced at another location. The fMRI data might well be different when generated under fully realistic circumstances. As a result, proving that there is detectable brain activity specific to conscious deception under low stakes conditions might not establish that the same pattern arises under high stakes conditions. Without a generally accepted theory of underlying mechanisms to justify extrapolations to the usual conditions of casework, low stakes laboratory findings may not suffice show general acceptance of validity under those conditions.

In sum, Semrau should not be read as establishing that the existence of testability carries any significant weight in favor of admission, that publication in a peer reviewed journal necessarily demonstrates validity, that a lack of complete realism in laboratory studies proves that there  are no “controlling standards” in practice, or that only field studies can establish general acceptance.

These concerns about the wording of the opinion notwithstanding, the problem of generalizing from the laboratory studies to the conditions of the Semrau case are substantial, and the court’s conclusion is difficult to dispute.