Tuesday, June 7, 2016

More NCFS Draft Documents Released for Public Comment

The National Commission on Forensic Science (NCFS) has released nine new initial draft documents for public comment. A list of these subcommittee documents and some excerpts from each are reproduced below.The comment period will close on July 5, 2016.

★ Proficiency and Other Testing of Examiners

Subcommittee on Accreditation and Proficiency Testing, Initial Draft Recommendation to the Attorney General on Proficiency Testing, May 19, 2016, DOJ-LA-2016-0011-0004
The National Commission on Forensic Science recommends that the Attorney General:
• Recommendation #1: Require all DOJ FSSP’s to participate in a proficiency testing program applicable to the area(s) in which they conduct forensic analysis within three years of acceptance.
• Recommendation #2: Encourage other federal, state & local FSSP’s, to participate in proficiency testing programs by all means possible, with any available enforcement mechanism.
• Recommendation #3: Require all DOJ FSSP’s to seek proficiency testing programs that provide sufficiently rigorous samples that are representative of the challenges of forensic casework.
• Recommendation #4: Encourage external vendors that provide proficiency tests to DOJ to share their aggregate data with entities doing research and analysis.
Subcommittee on Human Factors, Initial Draft Views of the Commission on Optimizing Human Performance in Crime Laboratories through Testing and Feedback, May 16, 2016, DOJ-LA-2016-0011-0010
Overview: How might testing be done to assess and improve the performance of forensic science service providers (FSSPs) on routine analytic tasks? What kind of testing programs will be most helpful for achieving goals such as: (1) establishing the conditions under which analytic methods can (and cannot) be expected to achieve accurate results; (2) providing feedback to help examiners improve their skills; (3) estimating the rate of error for routine analytic tasks and better understanding variables that affect the rate of error? What steps should be taken to facilitate the development of effective performance testing in forensic laboratories in order better to achieve such goals?
★ Certification of Examiners

Subcommittee on Accreditation and Proficiency Testing, Initial Draft Views of the Commission on Certification of Forensic Science Practitioners, May 13, 2016, DOJ-LA-2016-0011-0003
It is the view of the Commission that forensic science agencies should
• Determine applicability of certification depending on the job responsibilities of the specific individual and the availability of certification programs, including but not limited to, managers, analysts and technical support.
• Develop a process that ensures all practitioners apply for certification, if a program is available and provide time and resources to achieve and maintain certification.
• Include certification requirements in position descriptions, where possible.
• Provide similar support to practitioners, if there is not a certification program available in a specific category of testing, to maintain knowledge and skills.
It is the view of the Commission that all forensic science practitioners should:
• Become certified in all categories of testing in which examinations are performed as soon as the requirements of the certification body are met, provided a certification examination is available.
• Obtain certification within five years from implementation of this document. Personnel entering the field after implementation shall obtain certification within one year of eligibility or within the required time limit of the certifying body.
Subcommittee on Accreditation and Proficiency Testing, Initial Draft Views of the Commission on Accreditation and Recognition of Forensic Science Certification Bodies, May 13, 2016,  DOJ-LA-2016-0011-0002
It is the view of the Commission that certification bodies should:
• Seek compliance to ISO/IEC 17024 Conformity Assessment – General Requirements for Bodies Operating Certification of Persons. This process should be accomplished within ten years of implementation.
• Be accredited in accordance with the requirements of ISO/IEC 17024 by an accreditation body, operating in accordance with ISO/IEC 17011 – General Requirements for Accreditation bodies Accrediting Conformity Assessment bodies and signatory to the International Accreditation Forum (IAF) Multilateral Recognition Arrangement (MLA)..
• Collaborate with other certification bodies to develop uniform certification requirements.
• Ensure that certification examinations are continually reviewed to incorporate new technologies and remove obsolete information.
★ Accreditation

Subcommittee on Accreditation & Proficiency Testing, Initial Draft Views of the Commission Regarding Accreditation Program Requirements, May 16, 2016,  DOJ-LA-2016-0011-0005
It is the view of the National Commission on Forensic Science that the overall accreditation of forensic science service providers could be strengthened by, at a minimum, full compliance with ISO/IEC 17011 for all accrediting bodies offering services to forensic science service providers. Further improvements could be achieved with the consistent adoption/incorporation of the following requirements by accrediting bodies:
• Shorten the time period between on-site assessments.
• Require surveillance visits every year a facility does not have a full assessment.
• Expand the requirements of ISO 17011 definition of surveillance visits.
• Ensure appropriate technical assessors for categories of testing conduct assessments. (Assessors should have recent work as an analyst in the discipline.)
• Require accrediting bodies to provide continuous training and feedback to assessors.
• Incorporate unannounced visits/assessments.
• Include targeted and random sampling as well as case observations/witnessing when reviewing casework.
• Develop standards for sampling plans that account for casework volume and number of analysts.
• Increase the number of annual case observations/witnessing.
• Increase the percentage of technical reviews.
• Require a specific percentage of blind re-examinations.
• Increase requirements for proficiency testing plans (e.g. number of external per year/cycle based on a percentage of analysts.)
• Increase the robustness of the testimony monitoring program including in-person monitoring and transcript review with a mandated frequency.
• Require method validation to include both external sources/studies of the overall performance and reliability and internal studies of appropriateness and performance.
• DOJ should seek aggregate data issues/non-conformities from all assessments of forensic science service providers from each accrediting body. Anonymity and confidentiality should be maintained as the purpose of the report is to review systemic issues in forensic science that should be addressed.
• DOJ should encourage continuous improvement of the accreditation process and provide support through funding, research and other initiatives.
★ Case Reporting and Records

Subcommittee on Reporting and Testimony, Initial Draft Recommendation to the Attorney General on Documentation, Case Record and Report Contents, DOJ-LA-2016-0011-0007
The National Commission on Forensic Science recommends that the Attorney General have Department of Justice Forensic Science Service Providers (FSSPs) develop and maintain written policies for documenting the examination, testing, and interpretation of evidence and for reporting results, interpretations, and conclusions that are consistent with the following requirements:
1. Records should be created during the examination of evidence and the technical review that allow another analyst or scientist, with proper training and experience, to understand and evaluate all the work performed, and independently analyze and interpret the data and draw conclusions.
2. Providing all of the documentation described above in a single report in every case is impractical. Instead, if not in the report, the documentation described herein must be maintained in a case record.1 Generic documentation such as Standard Operating Procedures and definitions must either be a part of the case record or be easily accessible (e.g., posted on a website, available on request).
3. Reports should clearly state: the purpose of the examination or testing; the method and materials used; a description or summary of the data or results; any conclusions derived from those data or results; any discordant results or conclusions; the estimated uncertainty or variability; and possible sources of error and limitations in the method, data, or conclusions.
4. Every report should state that the report does not contain all of the documentation associated with the work performed and that to understand and evaluate all the work performed, and independently analyze and interpret the data and draw conclusions requires a review of the case record.
5. The case record should be organized and made available in a manner consistent with the discovery recommendations of the National Commission on Forensic Science.
Subcommittee on Reporting and Testimony, Initial Draft View of the Commission on Report and Case Record Contents, April 23, 2016, DOJ-LA-2016-0011-0006
It is the view of the National Commission on Forensic Science that a report and a case record describing the results of forensic testing should, at a minimum, contain the information identified in Appendix A.
★ Medicolegal Death Investigation

Subcommittee on Medicolegal Death Investigation, Initial Draft Recommendation to the Attorney General for the Formation of a National Office for Medicolegal Death Investigation, May 18, 2016, DOJ-LA-2016-0011-000
The National Commission on Forensic Science recommends that the Attorney General take the following action(s):
Recommendation #1: The Attorney General should work with the Whitehouse Office of Science and Technology Policy¡|s Medicolegal Death Investigation Working Group and other federal agencies and professional organizations to develop a permanent National Office of Medicolegal Death Investigation which would coordinate ongoing support of the nation¡|s medicolegal death investigation systems to improve quality, consistency and the meeting of criminal justice and public health needs. Such support would not only be for daily and routine operations, but for development and implementation of new technologies, equipment, personnel, and the fostering of research with federal funding and agency support.
Recommendation #2: The Attorney General should, through the National Office in conjunction with the National Institute of Justice, recommend ongoing funding and support to improve the recruitment and retention of forensic pathologists, modernization of facilities and creation of facilities in underserve areas, accreditation of medicolegal offices and certification of its personnel, and the establishment of a national information network for the nation’s medical examiner and coroner offices.
Subcommittee on Medicolegal Death Investigation, Initial Draft Views of the Commission Regarding Communication with Next of Kin and Other Family Members, May 16, 2016, DOJ-LA-2016-0011-0009
Currently, many medicolegal death investigation offices lack policies and procedures relating to the communication and interactions they have with Next of Kin (NOK) and other family members during death investigations. (1) Additionally, accreditation standards do not directly address this issue and therefore inconsistencies exist that greatly impacts the individual family, the investigation and our society overall.

Monday, June 6, 2016

Proposed Uniform Language for Forensic Toxicology

The first document listed in the announcement of "Uniform Language" (UL) documents for forensic science is the Department of Justice Proposed Uniform Language for Testimony and Reports for the Forensic Toxicology Discipline.  At first glance, it seems rather bland, but if the document truly demarcates the range of the "acceptable range of opinions" that DoJ toxicologists may provide, it is impoverished.

For example, the Supporting Documentation observes that "[a]n examiner may be asked if certain symptoms are consistent with those caused by a particular drug," but for decades, forensic scientists, statisticians, lawyers, and psychologists have complained that "consistent with" testimony is vacuous. Evidence usually is consistent with more than one hypothesis. Drowsiness is consistent with ingestion of barbituates, but it also is consistent with lack of sleep. For the symptoms to be probative of some hypothesis, alternative hypotheses must be considered. Diagnosticity depends on the probability of the symptoms under all the hypotheses.

The proposed uniform language obscures this fundamental concept. For example, the UL document provides that an "examiner may ... state that a reported blood concentration is within the therapeutic range, toxic range, or consistent with reported fatal concentrations" and "that a drug or poison found in a hair specimen is consistent with exposure (either ingestion or environmental) to the drug or poison." If a measured concentration falls into the range within which a poison has killed someone in the past, the "blood concentration" (more precisely, the concentration of the poison) would be "consistent with reported fatal concentrations." But it may be that this "fatal concentration" is rarely fatal and is "consistent with" an alternative cause of death. Surely, a toxicologist should be permitted to say as much "based on data published in peer reviewed literature or other authoritative sources."

At the other pole, suppose the concentration is even higher than any fatal dose ever reported. The toxicologist should not be limited to the uniform language. In this situation (and in some less extreme ones), an expert can give a more powerful, useful, and scientifically accurate opinion than "consistent with."

The UL also (inelegantly) cautions that "[a]n examiner may not report or state an opinion that suggests his/her interpretation of the effects of a drug or poison can be specified to the individual whose sample was tested." Does this mean that the toxicologist can say "the defendant had a blood alcohol level [BAC] that indicates intoxication" but cannot say that "the defendant had a blood alcohol level that indicates she was intoxicated"? Both opinions express (or should express) the same idea — that the measured BAC is more probable when the defendant is intoxicated than when the defendant is not intoxicated. (If this statement about probabilities does not hold, then the BAC does not support the hypothesis of intoxication and hence does not indicate intoxication.)

The UL document first suggests that the toxicologist can only speak of "the average human." Then it redefines the average human to be someone exactly like the defendant in our example:
An examiner may report and/or state his/her opinion as to the effects of drugs or poisons on the average human. This opinion should be based on the facts of the case, medical information about the individual that the specimens were collected from (e.g., weight, height, disease state, age), current published studies, and/or the examiner’s training in the fields of pharmacology, physiology, pathology, clinical chemistry, and/or toxicology.
So the examiner must use all "facts of the case" and all the anatomical and physiological variables specific to the defendant but must not say that the opinion pertains to the defendant? Even if the toxicologist hews to this line, what happens when defense counsel asks the expert whether focusing on the defendant by using all the information about the case exceeds the admonition to confine the testimony and reporting to "the average human"? The phrasing in this paragraph needs refinement if not rethinking.

Closely related postings



Saturday, June 4, 2016

The Department of Justice's "Proposed Uniform Language for Testimony and Reports"

The US Department of Justice (DoJ) has "Proposed Language Regarding Expert Testimony and Lab Reports in Forensic Science." The public comment period on www.regulations.gov closes July 8. Because words are the subject of the "Uniform Language" documents, as they are styled, it may be appropriate (if somewhat tedious) to jot down a few notes in the manner of English composition teachers. Readers who want remarks on the substantive merits of the documents will have to wait. 1/ What follows is merely about clarity of expression and ugliness (words like and "his/her" and "and/or").

To begin with, an initial disclaimer, boxed and bolded in the original, reads as follows:
This document provides examples of the scientifically-supported conclusions and opinions that may be contained in Department of Justice reports and testimony. These examples are not intended to be all inclusive and may be dependent upon the precedent set by the judge or locality in which a testimony is provided. Further, these examples are not intended to serve as precedent for other forensic laboratories and do not imply that statements by other forensic laboratories are incorrect, indefensible, or erroneous. This document is not intended to, does not, and may not be relied upon to create any rights, substantive or procedural, enforceable by law by any party in any matter, civil or criminal, nor does it place any limitation on otherwise lawful investigative and litigative prerogatives of the Department.
What would a careful copy editor have to say about this?
  • First sentence. The documents do not give examples of scientifically-supported statements. They give (1) examples of conclusions and (2) sometimes specify the kind of scientific support that would be needed for some of them. The toxicology "Uniform Language" document, for example, refers to "analytical findings" without regard to the scientific basis of the analyses. Surely not every conceivable analytical method is scientifically adequate. (As I said at the outset, this is just a problem with the wording of the introductory disclaimer. I am not suggesting that the analytical methods currently employed by DoJ toxicologists are invalid. In fact, separate "Supporting Documentation" asserts that every one has been validated. However, the introductory statement refers to "this document," and the Uniform Language document does not constrain the range of conclusions about "analytical findings" to scientifically supported ones. In comparison, it states that "[t]he examiner may report and/or state the pharmacokinetic and pharmacodynamic effects of drugs and poisons based on data published in peer reviewed literature or other authoritative sources.")
  • Second sentence. "The examples are not intended to be all inclusive" of what? Of "the scientifically-supported conclusions and opinions that may be contained in Department of Justice reports and testimony"? Presumably, this clause is meant to say that "The list of permissible statements is not exhaustive. Other statements might be acceptable."
  • Second sentence (con't). "These examples ... may be dependent upon the precedent set by the judge or locality in which a testimony is provided." Stilted language ("may be dependent on") aside, the examples are examples — regardless of local law. The scientifically supportable ones in "this document" are examples of scientifically acceptable statements for the field without regard to their admissibility. The unacceptable ones should be scientifically unacceptable regardless of what a trial judge (whose rulings are not technically "precedent" for any other case) or other judges, lawyers, or legislatures do.
  • Third sentence. "[T]hese examples are not intended to serve as precedent for other forensic laboratories and do not imply that statements by other forensic laboratories are incorrect, indefensible, or erroneous." DoJ has no power to make legal precedent. That it considers specific statements unfit for use by its laboratories surely has implications. Presumably, the sentence is meant to state the obvious — that DoJ's adoption for its laboratories of this list of acceptable and unacceptable examples is not intended to require other laboratories to do anything.
  • Final sentence. "This document is not intended to, does not, and may not be relied upon to create any rights, substantive or procedural, enforceable by law by any party in any matter, civil or criminal ... ." But any judge is free to rely on this document to create an enforceable right. Presumably, the intended meaning is that the document, in itself, does not create such a right.
  • Final sentence (cont'd). "nor does it place any limitation on otherwise lawful investigative and litigative prerogatives of the Department." What is the function of "otherwise lawful"? Naturally, the draft document places no limitation on anything DoJ wants to do. Does the full sentence mean that after adopting these guidelines, DoJ lawyers and laboratory staff will be free to depart from them? I think it means that the document is for internal guidance only and is not intended to be enforceable by any litigant.
Of course, the box of disclaimers at the beginning of each documents is just boilerplate. But skimming a couple of the Uniform Language documents also shows room for improvements in the mechanics of the writing. Here are some examples:
  • Extensive use of the barbarism "and/or" 2/ and an occasional "his/her." 
  • The phrase "[a]n examiner may state or imply" is common. Inasmuch as any examiner who legitimately may state that a proposition is true also may imply that it is true, "may state" the phrase "or imply" is surplusage. 3/
  • Another common locution is "This document provides the acceptable range of opinions expressed in both laboratory reports and during expert witness testimony ... ." I think this is intended to state that "This document specifies the range of opinions that examiners may provide in laboratory reports or expert testimony." (It is not the "range" that is acceptable — it is the opinions.)
  • The last statement is always qualified by the clause "while acknowledging that this document cannot address every variable in every examination." The qualification seems pointless or obscure. What does "address" mean here? Who would have imagined that the Uniform Language documents are supposed to address "every variable" in every forensic test, method, or analysis? If attention should have been given to an important "variable" but was not, how can we know that the range of opinions allowed is appropriate?

NOTES

1. Update: Remarks on a few of the discipline-specific documents are at
2. Bryan A. Garner, A Dictionary of Modern Legal Usage 56 (2d ed. 1995) (observing that "and/or has been vilified for most of its life—and rightly so. ... 'the only safe rule to follow is not to use the expression in any legal writing, document or proceeding, under any circumstances.'") (quoting Dwight G. McCarty, That Hybrid "and/or," 39 Mich. State B.J. 9, 17 (1960)); Leslie O’Flahavan, Is Using "and/or" in a Sentence Just Plain Dumb?, Jan. 25, 2010, http://ewriteonline.com/writing-matters/2010/01/using-andor-in-a-sentence-is-just-plain-dumb/ (surveying advice from journal editors and others and concluding that "the verdict is in: and/or is JPD (just plain dumb)").

3. The situation with "may not state or imply" is different.

Saturday, May 28, 2016

SWGFAST on "a decision to reach a conclusion" of individualization for latent fingerprints

Before SWGFAST (the Scientific Working Group on Friction Ridge Analysis, Study, and Technology) dissolved to make way for OSAC (the Organization of Scientific Area Committees in Forensic Science), it produced a working draft of a Guideline for the Articulation of the Decision-Making Process for the Individualization in Friction Ridge Examination (Latent/Tenprint) (dated 3/13/13). The Guideline tells examiners how to explain the thinking that leads to what SWGFAST called “a decision to reach a conclusion of individualization” (§ 10.1) and how to report this decision (§ 11). Some interesting parts follow:
11.2.2. The SWGFAST Individualization/Identification Position Statement (Document #103) states that: “The ability of a latent print examiner to individualize a single latent impression, with the implication that they have definitely excluded all other humans in the world, is not supported by research and was removed from SWGFAST’s definition of individualization.”
This is a major and salutary departure from historical practice, but what remains of the definition? SWGFAST’s last definition of “individualization” was “[t]he determination by an examiner that there is sufficient quality and quantity of detail in agreement to conclude that two friction ridge impressions originated from the same source.” SWGFAST, Standard Terminology of Friction Ridge Examination (Latent/Tenprint), Mar. 13, 2013, at 5. “If two friction ridge impressions originated from the same source,” however, they did not originate from any other source, that is, from any “other human[] in the world.”

The only interpretation of the Guideline that seems to make sense is that an “individualization” is a tentative rather than a “definitive” conclusion that all other humans in the world have been excluded. Indeed, the Guideline continues:
11.2.3. Specific words and phrases conveying absolute certainty are inappropriate or misleading and should not be used to express conclusions. Specific problematic phrases include: [e]xclusion of all others, 100% certainty (as an absolute fact), [and z]ero error rate / infallible method.
But it is easier to say what should not be said than to prescribe what should be stated. The Guideline offers the following positive-plus-negative advice:
11.2.4. These concepts should rather be expressed as the conclusion of the examiner, based upon data observed and interpreted through the examiner’s training and experience. The examiner has a level of personal confidence associated with the accuracy and reliability of this conclusion; however, the accuracy and precision of this personal level of confidence cannot currently be measured and reported. For this reason, certainty should not be reported numerically or in absolute terms.
Evidently, the examiner should say something like "My decision is that X is the source of the latent print. I base this decision on my training and experience. I have [enormous? high? medium? low?] confidence in my personal decision." Of course, if a categorical conclusion is given, what the judge or jury would like to hear is the probability that this conclusion is correct, not that an examiner has some personal belief. But how a latent print examiner could provide such a posterior probability without considering a case-specific prior probability is an unsolved problem. A more measurable quantity is a conditional error probability—here, the probability that an examiner would identify X when X is not the source. Although some experiments have examined the rate of false positive (and negative) errors by latent print examiners, the Guideline does not propose using these figures.

Instead, it implies in the future it will be possible "to measure a personal level of confidence." I suppose an examiner could be forced to pick the probability at which he or she is indifferent to a gamble in a game of chance that has a well-defined probability for winning and losing. Would that make the personal confidence measurable? If so, why can't it be done currently?

In any event, the lack of a workable mechanism to measure personal confidence is not why claims of absolute certainty should be avoided. The reason is that nothing empirical is absolutely certain. Tomorrow, the laws of physics could change and heavy objects could float upward.

Neither does the difficulty of devising a suitable psychometric instrument for personal certainty explain why an examiner cannot report any numbers. People quantify personal judgments that cannot be measured objectively when their doctors ask them to report the severity of pain on a scale of 1 to ten. Why cannot an examiner report a defensible level of confidence numerically if the subjective precision is provided? E.g., “I think X is the source, and the strength of my belief is 9 on a scale of 1-10, but obviously this is not an exact number. I could have said 8, or even 7 as well, but I would not want to go below that.”

I am not arguing that examiners should grade their personal belief in a source attribution on a scale of 1-10 or express it as a numerical probability. My point merely is that SWGFAST's argument about reporting a number for relative certainty is superficial and underdeveloped.

Sunday, May 1, 2016

BIT Global Group's Next Flaky Forensics Conference

BIT Global Group is notable for the scope and intensity of its spam promoting what appear to be parodies of academic or professional conferences. The latest email "on behalf of the organizing committee" cordially invited me "to attend BIT’s 3rd International Congress of Forensics & Police Tech Expo 2016, which will be held during October 27-29, 2016 at International Conference Center, Dalian, China."

The organizing committee (or, as the conference's website designates it, the program committee) consists of one individual, Dr. Xiaodan Mei, President of BIT Group Global Ltd., China. But the "Advisory Board Members" are "coming soon," and already, "the Plenary Session will invite a distinguished international panel including Dr. Henry Lee and many others to talk about the impact of recent forensic scientific and technological breakthroughs around the world."

Indeed, if the website is accurate, the "Renowned Speakers of WCF-2016" who have signed on are
  • "Dr. Henry C. Lee, Chief Emeritus for Scientific Services, University of New Haven, USA" and
  • "Dr. John Zheng Wang, Professor and Director of Certificate Programs, California State University [Long Beach], USA,"
Postscript

On May 16, 24, and 29, 2016, Ms. Cherry Dong, Organizing Committee of the Forensics & Police Tech Expo 2016, advised me that
I’m writing to follow up my last invitation letter several weeks ago, it is regretful we couldn’t receive your reply. Now we would like to extend our cordially invitation to you again to join the BIT’s 3rd International Congress of Forensics & Police Tech Expo 2016 ... . Forensic science has attracted enormous attentions ... . Therefore, we establish this focused event ... So please kindly inform us if you are available to give us a presentation in the conference, it will be highly appreciated you can give us a prompt reply with a title and brief (3-5 sentences) summary on your recent work in the area.
As of June 13, 22, and July 3, 2016, the wording changed to
How are you doing? I am Cherry, the Program Coordinator ... . I am writing the letter to make sure if you have received my letter previously, it is about the conference: WCF-2016 which will be held during October 27-29, 2016 ... . Maybe there some problems with my mailbox ... . Now we have invited some distinguished experts including Dr. Henry Lee and Dr. John Zheng Wang and many others ... . In the past two congresses, we have invited more than 300 worldwide leading experts, famous police officers, and law practitioners ...
In June, the renowned speakers list had grown with the addition of
  • Mathias Gaertner, Publicly Accredited and Sworn in Expert Witness for Information Technology, Germany
  • Eric Kreuter, a Partner in the Financial Advisory Services Group at Marks Paneth LLP
  • Erik Laykin, Managing Director, Duff & Phelps LLC
  • Frank Prieels, Professor, University of Duesseldorf, Belgium
  • Andre Stuart, CEO, 21st Century Forensic Animation
  • Linda Xiao, Professional Officer (Technical), University of Technology, Australia.
and by early July, another keynote speech, from Raymond Hsieh, Professor and Director of Justice Studies, Department of Criminal Justice, California University of Pennsylvania, was added to the schedule.

For more information on BIT's "congresses" and practices, see Wikipedia and Flaky Academic Conferences.