Physical scientists are trained to make rough approximations of quantities — both large an small — and they should not be pilloried for trying to use a simple probability model to get a sense of how rare an event might be. But experts should not testify to rough calculations of extremely persuasive probabilities without any error bounds and without trying to verify the assumptions of the modeling effort by collecting and analyzing a reasonable amount of data.
The prosecution's enthusiasm for the largely theoretical probability model in McKreith extended not merely to the shirt mentioned in the ProPublica article, but also to stripes on a handbag. Dr. Vorder Bruegge's testimony on direct examination was relatively mild. He testified, without computing any probabilities, that a photograph and a handbag were "indistinguishable" with respect to "class" and "individual" features that the jurors could see for themselves:
Q. I believe we were at the Mary Kay bag, Government Exhibit 14. [W]ere you able to identify this exhibit as something that was presented to you for ... photographic comparison purposes?
A. Yes, this is the bag that was submitted to me at the FBI laboratory for comparison in this case.
Q. [W]hat observations were you able to make in terms of the class characteristics of the bag?
A. Basically, this is a handbag that has ... two dark straps. It's got a pocket on [one] side ... . It's made primarily of ... multiple panels — two side panels, two end panels, a bottom and a secondary panel that is overlapping this side panel. It's a striped bag that has these bright snaps on the end. ... [B]asically, it's ... dark silver and black stripes on the bag. ... The manufacturer of the bag is indicated by the name Mary Kay on the side.
Q. [C]an you identify ... what distinguishing features there are about the stripes?
A. Well, the stripes are evenly sized stripes. Each one is about a quarter of an inch wide, and they are alternating black and silver stripes.
Q. At the spacing, the same consistent —
A. The spacing is consistent throughout, across the bag, yes.
Q. And you indicated that it has hand straps?
A. Yes. ...
Q. [S]howing you what is marked as Government’s Exhibit 7-EE, regarding the bank robbery at SouthTrust, is this a chart that you prepared for comparison analysis? ...
A. Yes. Government exhibit V7B-EE is a chart that I prepared. ...
Q. ... [W]ere you able to identify, from looking at the bag itself, any individual characteristics, identifying characteristics, which would make this bag unique from all the other Mary Kay bags that may have come off the assembly line at sometime, using the same fabric, being the same sized bag?
A. [T]here are some small white markings on the bag ... . It's not clear to me whether it would be marker or some kind of staining on the bag that occurs at various places around the bag. [O]n the back here [are] little white marks that could be used to differentiate this bag from all of the bags.
As far as the manufacturing characteristics, we've got another repeating pattern, much like the one in the shirt. in which we've got dark bright stripes. [I]f we look at this end panel, for example, you'll see that the very top stripe on this end of the bag is a ... totally black stripe. And if we look at this end ... where the top is, ... there's actually a little silver there. Likewise, if you look at the edges ... [at] the very top of this [side panel] is about half of one of those silver lines. On the other side, it's not quite a half of one of those silver lines. [Also, it’s] basically silver on the top of the sides — silver on the top of this side panel, end panel, but black [with] maybe a little bit of silver there, [but] when it's folded over, it's black on the top. Also, if you look at where the end panel meets the side panel, you've basically got ... the silver effectively lining up with the black, going from the end panel to the side panel. At the other end, it's a slight offset, slightly different, where it's kind of half and half. It doesn't match up exactly.
Q. ... Are the results of the randomly identifying features in terms of the way the bag was manufactured when these parts were sewn together?
A. Now, I have never been to a bag manufacturing plant, but assuming that the same sewing practices were used —
MR. HOWES: Judge, I'm going to object.
COURT: Sustained.
BY MR, STEFIN:
Q. [S]o you don't know the manufacturing process with respect to that particular bag?
A. That's correct.
Q. Okay. Were you able to do a comparison in any regard to determine whether or not there were points of identification which are similar to the government's Exhibit 14 with the bag depicted in the robbery photos from the SouthTrust bank robbery?
A. Yes I was. ...
Q. And what ... did your comparison yield?
A. Basically, I found a similarity in class characteristics between the bag, the Mary Kay bag — Government’s Exhibit 14 — and the bag carried by the Robert in the SouthTrust bank robbery as depicted on the left hand side of Government’s Exhibit VB7-EE. ...
Q. Are you able to offer an opinion as to whether Government Exhibit 14 is indistinguishable from the bag that's depicted in the bank surveillance photographs of the SouthTrust Bank robbery?
A. Yes.
Q. And what is your opinion?
A. This Government’s Exhibit 14 is indistinguishable from the bag in Government Exhibit 7-EE.
A. Yes, this is the bag that was submitted to me at the FBI laboratory for comparison in this case.
Q. [W]hat observations were you able to make in terms of the class characteristics of the bag?
A. Basically, this is a handbag that has ... two dark straps. It's got a pocket on [one] side ... . It's made primarily of ... multiple panels — two side panels, two end panels, a bottom and a secondary panel that is overlapping this side panel. It's a striped bag that has these bright snaps on the end. ... [B]asically, it's ... dark silver and black stripes on the bag. ... The manufacturer of the bag is indicated by the name Mary Kay on the side.
Q. [C]an you identify ... what distinguishing features there are about the stripes?
A. Well, the stripes are evenly sized stripes. Each one is about a quarter of an inch wide, and they are alternating black and silver stripes.
Q. At the spacing, the same consistent —
A. The spacing is consistent throughout, across the bag, yes.
Q. And you indicated that it has hand straps?
A. Yes. ...
Q. [S]howing you what is marked as Government’s Exhibit 7-EE, regarding the bank robbery at SouthTrust, is this a chart that you prepared for comparison analysis? ...
A. Yes. Government exhibit V7B-EE is a chart that I prepared. ...
Q. ... [W]ere you able to identify, from looking at the bag itself, any individual characteristics, identifying characteristics, which would make this bag unique from all the other Mary Kay bags that may have come off the assembly line at sometime, using the same fabric, being the same sized bag?
A. [T]here are some small white markings on the bag ... . It's not clear to me whether it would be marker or some kind of staining on the bag that occurs at various places around the bag. [O]n the back here [are] little white marks that could be used to differentiate this bag from all of the bags.
As far as the manufacturing characteristics, we've got another repeating pattern, much like the one in the shirt. in which we've got dark bright stripes. [I]f we look at this end panel, for example, you'll see that the very top stripe on this end of the bag is a ... totally black stripe. And if we look at this end ... where the top is, ... there's actually a little silver there. Likewise, if you look at the edges ... [at] the very top of this [side panel] is about half of one of those silver lines. On the other side, it's not quite a half of one of those silver lines. [Also, it’s] basically silver on the top of the sides — silver on the top of this side panel, end panel, but black [with] maybe a little bit of silver there, [but] when it's folded over, it's black on the top. Also, if you look at where the end panel meets the side panel, you've basically got ... the silver effectively lining up with the black, going from the end panel to the side panel. At the other end, it's a slight offset, slightly different, where it's kind of half and half. It doesn't match up exactly.
Q. ... Are the results of the randomly identifying features in terms of the way the bag was manufactured when these parts were sewn together?
A. Now, I have never been to a bag manufacturing plant, but assuming that the same sewing practices were used —
MR. HOWES: Judge, I'm going to object.
COURT: Sustained.
BY MR, STEFIN:
Q. [S]o you don't know the manufacturing process with respect to that particular bag?
A. That's correct.
Q. Okay. Were you able to do a comparison in any regard to determine whether or not there were points of identification which are similar to the government's Exhibit 14 with the bag depicted in the robbery photos from the SouthTrust bank robbery?
A. Yes I was. ...
Q. And what ... did your comparison yield?
A. Basically, I found a similarity in class characteristics between the bag, the Mary Kay bag — Government’s Exhibit 14 — and the bag carried by the Robert in the SouthTrust bank robbery as depicted on the left hand side of Government’s Exhibit VB7-EE. ...
Q. Are you able to offer an opinion as to whether Government Exhibit 14 is indistinguishable from the bag that's depicted in the bank surveillance photographs of the SouthTrust Bank robbery?
A. Yes.
Q. And what is your opinion?
A. This Government’s Exhibit 14 is indistinguishable from the bag in Government Exhibit 7-EE.
The court sustained the objection to testimony about "randomly identifying features in terms of the way the bag was manufactured when these parts were sewn together" because Dr. Vorde Bruegge forthrightly acknowledged that he was assuming certain facts not in evidence (and outside his expertise as an image analyst). Evidently, the court wanted more than a mere assumption "that the same sewing practices were used." But the next day, on re-direct examination, the prosecutor had Dr. Vorde Bruegge present the same probability model for the placement of the bag's stripes that he had used for the shirt:
Q. ... And with respect to the Mary Kay bag, Government Exhibit 14, didn't you identify individual characteristics of that bag which makes it
different than other Mary Kay bags that may have come off the same assembly line?
A. Yes, I did.
Q. And in fact, how many different characteristics were you able to identify looking at that exhibit, in comparison with the bank surveillance photographs of a bag being carried by the robber?
A. There were four specific characteristics that I noted.
Q. And would you remind us of what those four individual characteristics were?
A. The first one was the alignment of the black and silver stripes from the back side of the bag with the end of the bag, the fact that the silver lines on the inside line up with the black lines on the back side. The second characteristic was the location of the snaps at the top on a silver line. The third characteristic was the very small silver line at the top of the back piece. And the last characteristic was the silver line at the very top of the back piece.
Q. And did you come up with any ... odds or probabilities that these items would appear exactly as they are on that bag in a random fashion?
A. Yes, I did. ...
Q. [H]ow were you able to arrive at a probability as far as the individual characteristic that would exist?
A. Basically I'm dealing with a black or white situation. In this case, black or silver. Either you're going to get the black line in one place or you're going to get the silver line in that place. I'm not breaking down by 50% of the black line or 50% of the silver line. I'm just saying it's either a black line or a silver line, which is a 50/50. You got like one chance in two of a specific feature being black or silver. In particular, these silver snaps on the end can either be on a silver line or a black line. They're on a silver line. That eliminates all of the other bags that would have the snaps on a black line.
Likewise at the top, there is either a silver line at the top or there's a black line at the top. One chance in two, 50/50. So with this, the snaps and the top of the side of the back, it’s one in four. With the addition of the back of the bag silver at the top, it's 1 in 8 — 2 times 2 times 2. And then with the sides here having silver aligning with black, the silver’s either going to align with black, or the silver’s going to align with silver. That's another one in two chance. So 2 times 2 times 2 is 1 in 16.
Q. 2 times 2 times 2 times 2?
A. Yes, correct. 2 to the 4th power.
Q. 2 to the 4th power. So it is possible then to eliminate 15 out of 16 silver bags that would be coming the manufacturing process from whatever company made those bags?
A. That would be the hypothesis, correct.
Q. And did you, fact, find those four same characteristics in the photographs depicting the robber carrying the same bag?
A. Yes. Yes, I did.
A. Yes, I did.
Q. And in fact, how many different characteristics were you able to identify looking at that exhibit, in comparison with the bank surveillance photographs of a bag being carried by the robber?
A. There were four specific characteristics that I noted.
Q. And would you remind us of what those four individual characteristics were?
A. The first one was the alignment of the black and silver stripes from the back side of the bag with the end of the bag, the fact that the silver lines on the inside line up with the black lines on the back side. The second characteristic was the location of the snaps at the top on a silver line. The third characteristic was the very small silver line at the top of the back piece. And the last characteristic was the silver line at the very top of the back piece.
Q. And did you come up with any ... odds or probabilities that these items would appear exactly as they are on that bag in a random fashion?
A. Yes, I did. ...
Q. [H]ow were you able to arrive at a probability as far as the individual characteristic that would exist?
A. Basically I'm dealing with a black or white situation. In this case, black or silver. Either you're going to get the black line in one place or you're going to get the silver line in that place. I'm not breaking down by 50% of the black line or 50% of the silver line. I'm just saying it's either a black line or a silver line, which is a 50/50. You got like one chance in two of a specific feature being black or silver. In particular, these silver snaps on the end can either be on a silver line or a black line. They're on a silver line. That eliminates all of the other bags that would have the snaps on a black line.
Likewise at the top, there is either a silver line at the top or there's a black line at the top. One chance in two, 50/50. So with this, the snaps and the top of the side of the back, it’s one in four. With the addition of the back of the bag silver at the top, it's 1 in 8 — 2 times 2 times 2. And then with the sides here having silver aligning with black, the silver’s either going to align with black, or the silver’s going to align with silver. That's another one in two chance. So 2 times 2 times 2 is 1 in 16.
Q. 2 times 2 times 2 times 2?
A. Yes, correct. 2 to the 4th power.
Q. 2 to the 4th power. So it is possible then to eliminate 15 out of 16 silver bags that would be coming the manufacturing process from whatever company made those bags?
A. That would be the hypothesis, correct.
Q. And did you, fact, find those four same characteristics in the photographs depicting the robber carrying the same bag?
A. Yes. Yes, I did.
This computation of 1/16 for the probability of a four-feature match sounds suspiciously like an application of the principle of insufficient reason discussed yesterday. Every feature is "black or white," present or absent. Not knowing which is more probable, we can presume that each state (present or absent) is equally probable. Four independent features then create 16 equally likely states of nature.
Over a century ago, the New York Court of Appeals soundly rejected this Laplacean reasoning. In People v. Risley, 108 N.E. 200 (N.Y. 1915), a mathematician "was permitted to testify that, by the application of the law of mathematical probabilities, the chance of such defects [in letters typed on an allegedly altered affidavit] being produced by another typewriting machine was so small as to be practically a negative quantity." The mathematics professor "defined the law of probabilities as 'a proper fraction expressing the ratio of the number of ways an event may happen, divided by the total number of ways in which it can happen.'" The court wrote that the extended multiplication of one-half for each peculiarity "was not based upon actual observed data, but was simply speculative ... ."
If the choice of 1/2 for the probability of each binary feature on the handbag was based on nothing more than the fact that that there are two possibilities -- present or absent -- then it too is "simply speculative." If it was based on the more plausible assumption that the bag is sewn together in a way that would be expected to produce a uniform distribution, then the objection -- that the expert had not even visited a Mary Kay plant to learn how the bags were made -- applies. But McKreith's lawyer did not renew the objection. Neither did he argue that the applicability of the model was not verified by data on a sample of Mary Kay bags. If anything, the 1/16 figure for the four-feature handbag match is more speculative than the one in 650 billion probability of the eight-seam shirt match.
Risley and McKreith are not the only cases in which experts have multiplied a lot of small fractions together to get a smaller number. In the 1968 California case of People v. Collins, a prosecutor had a local mathematics professor testify to the product (one in 12 million) of a series of probabilities for characteristics supplied by eyewitnesses who desctibed an interracial couple that drove a yellow automobile away from the scene of a robbery. The probabilities were data-free estimates that the prosecutor fed to the mathematician. The California Supreme Court reversed the conviction, famously remarking that "[m]athematics, a veritable sorcerer in our computerized society, while assisting the trier of fact in the search for truth, must not cast a spell over him," and sparking much distrust in the legal community of probability calculations.
However, the probability model in McKreith, for handbags as well as shirts, is more plausible than the one in Collins. For one thing, the assumption of uncorrelated features is more plausible, and there is at least a modicum of knowledge of the process generating the features. Nevertheless, as I observed with regard to another case in which appellate lawyers for a defendant put an explicitly probabilistic assessment of evidence into their brief, "[t]he attempt to use probability theory ... was heroic. Like many acts of heroism, it also was hasty. Although there were some measurements and estimates of quantities bearing on guilt or innocence, the empirical data were so sketchy that the computations inevitably were more creative than convincing." 1/
NOTES
- D. H. Kaye, Book Review, Statistics for Lawyers and Law for Statistics, 89 Mich. L. Rev. 1520, 1543 (1991).
No comments:
Post a Comment