Showing posts with label iq. Show all posts
Showing posts with label iq. Show all posts

Friday, November 23, 2018

Muddling Through the Measurement of IQ

IQ scores are a critical component in the diagnosis of intellectual disability. That measurements of IQ are subject to various sources of measurement error is widely appreciated, but by and large, lawyers and psychologists have supplied rather imprecise -- and sometimes incorrect -- explanations of the statistics involved. A recent example is Intellectual Disability and the Death Penalty: Current Issues and Controversies, a book intended as "a valuable resource for mental health experts, attorneys, investigators, mitigation specialists, and other members of legal teams, as well as judges." 1/ The authors explain the "standard scores" that put the mean IQ score in the population at 100 as follows:
A person's standard score on a test is calculated by transforming the individual's obtained raw score on Test A (e.g., the sum of the number of correct responses on a test) using the population's known mean and standard deviation on Test A, which transforms the individual's test performance onto a common metric allowing us to compare his or her score to anyone else tested with Test A. Standard scores are possible only for tests where, if administered to the entire population, the distribution of all test scores on said test would be normally distributed ... . A percentile score is one form of a standard score that permits the interpretation of a person's performance in relation to a reference group. Although not a requirement, in the case of many psychological tests the scale for standard scores is set to have a mean or average score of 100 and a standard deviation of 15. Thus, a test performance that results in a standard score of 70 is said to be "significantly" below average or approximately two standard deviations below the population mean. A standard deviation is a unit of measure that indicates the distance from the average. During the standardization phase of the development of a standardized test, the test and its items are administered to a large anc representative sample of the reference group of interest or population. This is generally referred to as the standardization sample or norming group. From this norming group, the test developers compute the population's mean score and standard deviation on the test. The mean score and standard deviation are essential to transforming subsequently obtained raw scores (i.e., the sum of the number of correct items) on said test to a standard scale score (e.g., intelligence quotient, or IQ). 2/
Percentiles and Standard Scores

Standard scores have some value in "compar[ing one individual's] score to anyone else tested with Test A." Unlike raw scores, they incorporate the variance in the scores across different test-takers into the reported score. They are perhaps more useful for comparing scores from different tests (or different forms of the same test, or from tests administered to populations that are changing over time)

But whatever the motivation for a standardized reporting scale, it is strange to describe percentiles as standard scores. A standard score is just a particular linear transformation of a raw score that specifies "the number of standard deviations above (+) or below (-) the mean you are." 3/ As an example, suppose that the raw-score population mean for "Test A" is 60; that the population standard deviation is 12; and that a test taker has a raw score of 50. The standard score is 5/6s of a standard deviation below the mean: z = (50 - 60)/12 = -5/6 = -0.83.

To translate the raw score (or the corresponding z-score of -0.83) into a percentile, we need to know how the raw scores are distributed. For example, if raw scores were uniformly distributed from about 39 to 81, then some 26% of them would be 50 or less. If the raw scores were normally distributed (with the same mean and standard deviation), then 20% of the population would have a raw score of 50 (or less). Other distributions would produce other percentiles. Consequently, the percentile is not "one form of a standard score." At best, the percentile can be deduced from the standard score and other information.

A Standardized Scale Does Not Require Normality

Why are "[s]tandard scores ... possible only for tests where, if administered to the entire population, the distribution of all test scores on said test would be normally distributed"? Standard scores can be constructed for any distribution of test scores with a defined mean and standard deviation. Normality may be convenient or common, but it is not essential to a standardized score scale.

So What?

Not much turns on these corrections to the explanation in Intellectual Disability and the Death Penalty. IQ scores are more or less normally distributed, and the use of IQ scores of 70 and below (z ≤ -2) as the range in which an individual can be diagnosed as intellectually disabled limits the diagnosis to no more than roughly 2.3% of the general population.

But why should "a standard score of 70 [be] said to be 'significantly' below average"? Why is not an IQ score of 71 -- or even 80 -- significantly below the mean of 100? There is no statistical reason to focus on 70 as a cut off. In Hall v. Florida, 572 U.S. 5 (2014), a  majority of the Supreme Court was content with categorically excluding from the zone of intellectual disability (for the purpose of deciding potential eligibility for capital punishment) all defendants with true IQ scores above 70. Yet, no one could explain the basis for this fundamental choice. It is a convention currently in vogue among experts who want to have some such threshold. 4/

Quantifying Measurement Error

At the same time that the Court limited eligibility for the constitutional exemption from capital punishment because of intellectual disability to a small fraction of the population by approving of the z ≤ -2 range for true scores, it held that a slightly higher cutoff for observed scores was constitutionally necessary to ensure that random error in measuring IQ does not preclude too many defendants with true scores of 70 or less from consideration. Intellectual Disability and the Death Penalty explained this refinement as follows:
The Supreme Court of the United States in Hall v. Florida ruled that states must consider the test's standard error of measurement when interpreting obtained IQ scores in cases where the defendant is making an intellectual decision claim. ...
The standard error of measurement (SEM) is a direct measure of the test's reliability and is computed by administering the test to a large and representative sample of the population to be assessed on the test and computing the test's reliability coefficient, which can then be translated into an average error of measurement for the population ... . Generally, the SEM is computed and then used to create confidence intervals around the obtained standard scores (e.g., 95% certainty). A confidence interval of 95% represents a statistical certainty that, based on the knowledge of this test's reliahility coefficient, there is a 95% chance that the person's true score falls within a confidence interval that is +/-2 times the test's SEM. Thus, a professional reporting on an assessed individual's "obtained" full-scale IQ score of 70 on IQ Test A and knowing that Test A has a SEM of 2.5 around its full-scale IQ score, he would report that there is a 95% certainty that the assessed person's "true" full-scale IQ score falls within the range of 65-75 (i.e., 2x2.5= +/-5 points). 5/
This passage is garbled in two ways. To begin with, SEM is not "a direct measure of the test's reliability." It is a statistic derived from "the test's reliability coefficient." There are many ways to estimate reliability, and the logic behind the move from reliability to SEM is subtle. A better statistic for estimating the uncertainty in the observed score would be the standard error of estimate (SEE). The SEM is an average across all scores. The SEE takes into account the fact that uncertainty increases as one moves away from the population mean (IQ = 100). A description of the SEE can be found elsewhere. 6/

Second, the 95% in a 95% confidence interval is neither a "statistical certainty" nor "a 95% chance that the person's true score falls within [the computed] confidence interval." This interpretation of "confidence" is ubiquitous -- and widely known (to statisticians) to be wrong. The misinterpretation was apparent in the dissenting opinion written for four Justices by Justice Alito. It probably was implicit in the majority opinion penned by Justice Kennedy. Although we would expect (in the long run) 95% of all 95% confidence intervals  to contain the true value, the probability that a particular interval covers the true score cannot be computed with the machinery of confidence intervals. 7/ Interval estimates that can be said to provide such probabilities require Bayes theorem. Again, discussion and examples for IQ scores are available elsewhere. 8/

Clinical psychologists, lawyers, and judges are not statisticians. They do not have to compute means, standard deviations, standard errors, confidence intervals, or Bayesian credible regions. Nevertheless, to become more astute users of such statistics, they need a better understanding of the reasoning behind standard scores and expressions for measurement error.

NOTES
  1. Marc L. Tassé & John H. Blume, Intellectual Disability and the Death Penalty: Current Issues and Controversies vii (Prager 2018).
  2. Id. at 87.
  3. Penn State University Eberly College of Science, STAT 100: Statistical Concepts and Reasoning § 5.2 (2018), https://onlinecourses.science.psu.edu/stat100/node/13/
  4. David H. Kaye, Deadly Statistics: Quantifying an "Unacceptable Risk" in Capital Punishment, 16 Law, Probability & Risk 7-34 (2017), http://ssrn.com/abstract=2788377.
  5. Tassé & Blume, supra note 1, at 90.
  6. Kaye, supra note 4.
  7. For an elaboration in legal settings, see David H. Kaye, Apples and Oranges: Confidence Coefficients and the Burden of Persuasion, 73 Cornell L. Rev. 54 (1987).
  8. Kaye, supra note 4.
POSTINGS ON IQ SCORES AND CAPITAL PUNISHMENT
Posted by DH Kaye at 9:04 AM 0 comments
Labels: , , , ,

Wednesday, March 29, 2017

After Moore v. Texas Is a Single IQ Score Really Determinative?

Bobby J. Moore has been on death row for the last 37 years. On Monday, the Supreme Court ruled that the Texas Court of Criminal Appeals (the state’s highest court for criminal cases) erred in finding that Moore is not intellectually disabled. Justice Ginsburg wrote for the five-member majority. The Chief Justice wrote a strong dissent for the other three justices. Neither opinion (on my quick reading at least) comes to grips with an obvious statistical principle—that combining information reduces uncertainty.

Moore v. Texas is the third case to try to clarify the rule in Atkins v. Virginia, 536 U.S. 304 (2002). There, the Supreme Court held that the Eighth Amendment’s Cruel and Unusual Punishment Clause prevents a state from executing an intellectually disabled offender, but it left the states with latitude in defining the disability. In Moore, the Court held that the Texas tribunal applied a medically outdated—and (hence?) constitutionally impermissible—standard in rejecting Moore’s claim of disability. Most of the majority opinion concerns “adaptive functioning,” which must be substantially impaired for a diagnosis of intellectual disability to be made.

However, the Court in Hall v. Florida, 572 U.S. 5 (2014), allowed a state to refuse to inquire into adaptive functioning if an offender’s true IQ score is at least 70. Hall explicitly stated that the following statutory definition of intellectual disability was constitutionally acceptable:
“significantly subaverage general intellectual functioning existing concurrently with deficits in adaptive behavior and manifested during the period from conception to age 18,” where “significantly subaverage general intellectual functioning” is “performance that is two or more standard deviations from the mean score on a standardized intelligence test.”
Because IQ scores for the whole population are roughly normally distributed with a mean of approximately 100 and a standard deviation of about 15, Hall allows the state to execute offenders whose "true scores" are above 70.

In deciding whether a true score is above 70, Hall demanded that the state attend to the error of measurement. As the Moore Court, quoting from Hall, explained, "'[f]or purposes of most IQ tests,' [the] imprecision in the testing instrument 'means that an individual’s score is best understood as a range of scores on either side of the recorded score . . . within which one may say an individual’s true IQ score lies.'" For a single test with a standard error of 2.5 IQ points, it follows (for normally distributed errors) that the measured score must be greater than or equal to 75 (= 70 + two standard errors) to avoid "an unacceptable risk that persons with intellectual disability will be executed." 1/

But what about multiple scores? In that common situation, the Hall Court seemed conflicted. Justice Kennedy opaquely opined that “[e]ven when a person has taken multiple tests, each separate score must be assessed using the SEM, and the analysis of multiple IQ scores jointly is a complicated endeavor.” Does this mean that no matter how many IQ tests have been administered and no matter how many of them lie above 70, a single score of 75 or less makes a conclusive case for “significantly subaverage general intellectual functioning”?

From a statistical perspective, a lowest-single-score seems very strange indeed. If I want to know whether I have a fever and I take ten measurements of my temperature (with ten thermometers), I would not say that I have a fever just because one thermometer gives a high reading. I would use an average, and the mean temperature would have greater precision (smaller standard error) than the single highest reading of the ten.

Justice Ginsburg’s opinion in Moore seems to fly in the face of this common-sense statistical point. The Texas court focused on two test scores — "a 78 in 1973 and 74 in 1989." It pointed to factors that might have biased the latter score toward the low end, leaving the higher one as entitled to more weight. Specifically, it wrote that there was expert testimony that Moore might not have been putting much effort into answering the questions in the lower-scoring test, which was given to him in prison, and that he "also took the WAIS–R under adverse circumstances; he was on death row and facing the prospect of execution, and he had exhibited withdrawn and depressive behavior." Ex Parte Moore, 470 S.W.3d 481, 519 (Tex. Ct. Crim. App. 2015). Thus, the court concluded,
These considerations might tend to place his actual IQ in a somewhat higher portion of that 69 to 79 range. ... Considering these factors together, we find no reason to doubt that applicant's [higher] WAIS–R score accurately and fairly represented his intellectual functioning as being above the intellectually disabled range.
The Supreme Court assumed that it was necessary to consider each test in isolation and without making a clinical adjustment to the statistically determined plus-or-minus-five-point margin of error. Justice Ginsburg called the statistical range of error "clinically established." She described and condemned the Texas court's evaluation of the clinical testimony as follows:
Based on the two scores, but not on the lower portion of their ranges, the court concluded that Moore’s scores ranked “above the intellectually disabled range” (i.e., above 70). ... But the presence of other sources of imprecision in administering the test to a particular individual, cannot narrow the test-specific standard-error range. [W]e require that courts continue the inquiry and consider other evidence of intellectual disability where an individual’s IQ score, adjusted for the test’s standard error, falls within the clinically established range for intellectual-functioning deficits.
Thus, she insisted that just because "Moore’s score of 74, adjusted for the standard error of measurement, yields a range of 69 to 79" so that "the lower end ... falls at or below 70, the [Court of Criminal Appeals] had to move on to consider Moore’s adaptive functioning." (Emphasis added.)

In sum, Moore seems to say that a clinician cannot tinker with the statistical margin of error (two standard errors as constitutionalized in Hall). The dissent vigorously disagreed with this rule and maintained that the constitution permits states to make adjustments for individual circumstances that experts agree affect performance. A statistical argument for the dissent's position would be this: Computationally, the standard error reflects the variation in performance of a population of test-takers. This population-based figure is then applied to all individuals regardless of how strongly the sources of error apply to them. IQ tests administered in prison to inmates exhibiting signs of depression may not be part of that population. Those scores might have a larger or a smaller standard error, and they are generally lower than the true score for a person taking the test in normal circumstances. In other words, the clinician is not modifying the margin of error as much as adjusting the entire estimate upward.

This analysis does not necessarily render Moore's rule legally faulty. It might be undesirable to give clinicians this latitude to adjust scores. Under the majority's approach to the Eighth Amendment, the issue becomes whether the clinical guidelines for diagnosing disability allow individualized modifications of the statistical rule. The guidelines discussed in Hall are not completely clear. The dissent reads them as requiring an expert or a court to take the usual standard error seriously in interpreting an IQ score, but permitting reasoned and reasonable departures from them.

Even if Moore forbids individual adjustments to the statistical rule of plus-or-minus two standards errors (for the general population), why allow the confidence interval for a single test score to be dispositive when multiple tests scores are present? The clinical guidelines do not mandate this rule, and it is not so obvious that Moore does. Texas apparently made no effort to combine the two scores into a single point estimate with a margin of error applicable to the combined statistic. Hall claimed that combining scores from different IQ test forms was "complicated," although the literature it cited gave a simple procedure for doing so. So neither Hall nor Moore can be said to firmly establish that an appropriately averaged score is impermissible. After all, neither case presented the Court with an interval estimate for the true IQ score derived from multiple scores by an accepted statistical procedure, and the many-thermometer example given above illustrates the statistical deficiency in a rule that looks to every measured IQ score in isolation.

The single-score-too-low rule bends over backward to avoid misclassifying a disabled offender as normal. The rule might be defended on exactly that ground. But that is not the logic of Moore, which only asks what clinical guidelines for interpreting IQ scores allow. Moreover, if the real objective is make determinations of intellectual disability as fully informed as possible, it would seem more direct just to demand the inquiry into adaptive functioning along with IQ scores in all cases. On the other hand, if true IQ scores matter as a threshold to a richer inquiry into both intellectual and adaptive functioning, then statistically sound procedures for integrating all the IQ test results ought to be followed.

Further reading: David H. Kaye, Deadly Statistics: Quantifying an "Unacceptable Risk" in Capital Punishment, 16 Law, Probability & Risk 7 (2017).

NOTE
  1. If the standard error were substantially less than 2.5, then the measured score would not have to be all five points above 70. The use of two standard errors also is on the high side; 1.96 standard errors provides 95% coverage. Justice Kennedy's opinion in Hall was not as clear as it should have been on these points, but this is the only interpretation consistent with the concept of confidence intervals and standard errors used in the opinion. In Bromfield v. Cain, 135 S.Ct. 2269 (2015), however, the Court wrote that after "[a]ccounting for this margin of error, Brumfield's reported IQ test result of 75 was squarely in the range of potential intellectual disability." Id. at 2278. The Court did not disclose the standard error of measurement for the test.
POSTINGS ON IQ SCORES AND CAPITAL PUNISHMENT
Posted by DH Kaye at 10:43 PM 0 comments
Labels: , ,

Wednesday, June 4, 2014

Quarreling and Quibbling over Psychometrics in Hall v. Florida (part 3)

After explaining that Florida’s statutory cutoff of –2σx corresponds to an IQ score of 70 (because IQ tests are normed to have a mean of 100 and a standard deviation of 15), Justice Kennedy observes that:
Florida's rule disregards established medical practice in two interrelated ways. It takes an IQ score as final and conclusive evidence of a defendant's intellectual capacity, when experts in the field would consider other evidence. It also relies on a purportedly scientific measurement of the defendant's abilities, his IQ score, while refusing to recognize that the score is, on its own terms, imprecise.
Here, I show that these two limitations on IQ scores are less “interrelated” than Justice Kennedy suggests.

The first issue: validity

The first limitation involves what social scientists call “validity”—the extent to which something measures the real quantity of interest. For example, measuring the volume of a box by attending only to one dimension, such as height, is invalid because it ignores the two other determinative variables of width and depth.

The second limitation concerns the precision or “reliabilility” of the measurement—regardless of validity. Being able to measure the height of a box to the nearest millimeter time after time achieves precision and reliability, but it still lacks validity (with respect to the variable of volume). Moreover, measuring width or depth—even somewhat imprecisely—will add to validity but will do nothing to enhance the precision of the measurement of height.

Likewise, the “other evidence” to which the Court referred does not make IQ scores any more precise. Rather, it relates to what is known in the trade as “adaptive functioning.” The opinion defines adaptive functioning as “the inability to learn basic skills and adjust behavior to changing circumstances.” The Court disparages the “mandatory cutoff” of –2σx because this cut-score means that
sentencing courts cannot consider even substantial and weighty evidence of intellectual disability as measured and made manifest by the defendant's failure or inability to adapt to his social and cultural environment, including medical histories, behavioral records, school tests and reports, and testimony regarding past behavior and family circumstances. This is so even though the medical community accepts that all of this evidence can be probative of intellectual disability, including for individuals who have an IQ test score above 70.
If one were to follow this we-need-another-variable theory of “intellectual disability” to its logical limit, no IQ score could preclude the more comprehensive assay of all forms of “substantial and weighty evidence of intellectual disability.” An individual with above-average IQ scores also might “manifest [a] failure or inability to adapt to his social and cultural environment [as shown by] medical histories, behavioral records, school tests and reports, and testimony regarding past behavior and family circumstances.”

But surely the Court cannot claim that the execution of intellectually gifted but maladapted criminals is cruel and unusual while the execution of intellectually gifted and socially well adjusted criminals is not. To avoid such anomalies, the Court follows the contemporary (and prior) mental health practice of limiting “intellectual disability” to “concurrent deficits in intellectual and adaptive functioning” (emphasis added), which requires “significantly subaverage intellectual functioning” in addition to mere “deficits in adaptive functioning.” If it is clear that an individual is able to function intellectually within a broad but “normal” range, then a state need not entertain a claim of “intellectual disability” based solely on problems in adaptive functioning. Therefore, an IQ within the normal range should suffice to displace the offender from the potentially death-disqualified group.

So the possibility of weighty evidence of deficits in adaptive functioning, although relevant to clinicians, turns out to be no explanation for why Florida cannot draw the line at –2σx. If evidence of an adaptive-function deficit does not bar the state from executing criminals with IQs in the broad range of normalcy, why cannot the state define all IQs above 70 (–2σx) as lying within that range? That “experts in the field would consider other evidence” than IQ scores is not an answer. The answer has to be that (1) there is a range above which IQ, in of of itself, is a valid measure of the absence of “intellectual disability,” and (2) this range does not extend all the way down to 70. When these conditions hold, experts would not (or need not) consider “other evidence.”

Ironically, the Court’s opinion contradicts the second proposition. It clearly implies that the state could use a perfectly precise IQ measurement just above –2σx as conclusive evidence of intellectual disability. But if that is so, then the problem is not the failure to allow evidence of adaptive functioning. It is solely the existence of nonzero measurement error of IQ alone.

The second issue: precision (reliability)

Apparently (and dubiously) reserving the term “scientific” for precise measurements, Justice Kennedy stated that the “purportedly scientific measurement of the defendant's abilities, his IQ score, ... is, on its own terms, imprecise.” The problem is that although “there is evidence that Florida's Legislature intended to include the measurement error in the calculation ... the Florida Supreme Court ... has held that a person whose test score is above 70, including a score within the margin for measurement error, does not have an intellectual disability ... .”

In other words, a legislature that wants to preclude the more elaborate evaluations of all offenders with IQ scores below 70 could do so if only it had a way to measure IQs with perfect accuracy. Because of the “measurement error” of IQ tests, this legislature must adopt a higher cutoff. The Court, relying on the diagnostic literature, repeatedly refers to a cutoff of 75 as assuring an adequate safety margin.

The dissent had harsh words for the choice of 75, and I will get to those later, after examining where the figure of 75 comes from. At this point, no excursion into statistical theory is required to recognize that there is something weird about saying that IQ scores are problematic because they are an incomplete measure of “intellectual disability,” but then using them—and only them—within a band that accounts only for the error in measuring IQ. By definition, this band does not attend to the other factors that should be part of the full analysis. To put it another way, if the problem lies with using IQ alone, the solution lies in defining the range of IQ scores in which the other factors realistically could produce a different diagnosis. However, the error in IQ measurements has no clear connection to the range in which the failure to look beyond IQ makes a difference.

The majority’s response is essentially that if the mental health profession generally agrees that incompleteness is only a significant concern within the logically unrelated range of IQ-score error, then that is all that the Cruel and Unusual Punishment Clause demands. To which the dissent replies that abdicating the line drawing to the professionals makes no constitutional sense and “will also lead to serious practical problems.”

The dissent’s peculiar proof of “instability”

The first such problem is “instability.” According to Justice Alito:
This danger is dramatically illustrated by the most recent publication of the APA, on which the Court relies. This publication fundamentally alters the first prong of the longstanding, two-pronged definition of intellectual disability that was embraced by Atkins and has been adopted by most States. In this new publication, the APA discards “significantly subaverage intellectual functioning” as an element of the intellectual-disability test. Elevating the APA's current views to constitutional significance therefore throws into question the basic approach that Atkins approved and that most of the States have followed. 1/
The American Psychiatric Association’s latest version of its venerable Diagnostic and Statistical Manual of Mental Disorders—the DSM-5—“was published in May 2013 amid a storm of controversy and bitter criticism.” 2/ In general, critics maintain that “D.S.M.’s diagnostic categories lacked validity, that they were not ‘based on any objective measures,’ and that, ‘unlike our definitions of ischemic heart disease, lymphoma or AIDS,’ which are grounded in biology, they were nothing more than constructs put together by committees of experts.” 3/ Neither opinion even hints at such turmoil. The majority genuflects to clinical expertise and guidelines. The dissent raises no questions about validity and subjectivity, but objects to substituting “the standards of professional associations, which at best represent the views of a small professional elite” for “the standards of the American people.”

As for “instability,” the DSM-5 has brought a profusion of new or redefined disorders, but it does not radically change the definition of “intellectual disability” or dispense with the criterion of “significantly subaverage intellectual functioning.” It simply substitutes the word “intellectual ... deficit” for “significantly subaverage.” The diagnostic criteria have remained remarkably similar over the 19 years between the DSM-4 and the DSM-5.

The DSM-5 specifies that “[t]he first diagnostic criterion that “must be met” is “A. Deficits in intellectual functions ... confirmed by ... both clinical assessment and standardized intelligence testing.” If the intelligence testing does not demonstrate subaverage performance, it is hard to see how it could confirm the existence of a meaningful deficit. Moreover, the DSM-5 elaborates, making it plain that significantly subaverage IQ remains a sine qua non for the diagnosis:
The essential features ... are deficits in general mental abilities (Criterion A) and impairment in everyday adaptive functioning ... (Criterion B) [with o]nset is during the developmental period (Criterion C). The diagnosis of ... is based on both clinical assessment and standardized testing ... . Intellectual functioning is typically measured with ... tests of intelligence. Individuals with intellectual disability have scores of approximately two standard deviations or more below the population mean, including a margin for measurement error (generally +5 points). On tests with a standard deviation of 15 and a mean of 100, this involves a score of 65–75 (70 ± 5).
Compare this to the DSM-4 (or the DSM-4-TR cited by Justice Alito, which uses the same words):
The essential feature of Mental Retardation is significantly subaverage general intellectual functioning (Criterion A) that is accompanied by significant limitations in adaptive functioning ... (Criterion B) [with] onset ... before age 18 years (Criterion C). ... General intellectual functioning is defined by the intelligence quotient ... obtained by assessment with ... intelligence tests ... . Significantly subaverage intellectual functioning is defined as an IQ of about 70 or below (approximately 2 standard deviations below the mean). It should be noted that there is a measurement error of approximately 5 points in assessing IQ, although this may vary from instrument to instrument ... . Thus, it is possible to diagnose Mental Retardation in individuals with IQs between 70 and 75 who exhibit significant deficits in adaptive behavior. Conversely, Mental Retardation would not be diagnosed in an individual with an IQ lower than 70 if there are no significant deficits or impairments in adaptive functioning.
Thus, there are wording changes over the 19 years from 1994 to 2013, but Criterion A remains Criterion A, IQ tests remain critical to the diagnosis, and the range of test scores that lend themselves to the diagnosis is the same. The APA has changed the emphasis somewhat, and it has spelled out the constructs a little more (in words not quoted here). Nevertheless, to claim that the shift “dramatically illustrate[s a] fundamental[] alter[ation in] ... the longstanding ... definition of intellectual disability” seems, well, melodramatic.

State laws that rely on –2σx plus a margin of safety for measurement error, are compatible with Atkins, Hall, DSM-4, and DSM-5. Of course, whether this is a logically or functionally appropriate manner of defining “intellectual disability” for purposes of capital punishment is open to debate. Resolving this debate requires a more detailed and accurate understanding of the concept of measurement error than the Hall opinions provide.

Footnotes
  1. The second problem is that “changes adopted by professional associations are sometimes rescinded.” This problem is just a form of instability. The third problem is hypothetical (thus far) as it relates to intellectual disability determinations: “what if professional organizations disagree? The Court provides no guidance for deciding which organizations' views should govern.” The fourth and final “practical problem” is actually conceptual—and quite important. “[D]efinitions of intellectual disability ... are promulgated for use in making a variety of decisions that are quite different from the decision whether the imposition of a death sentence in a particular case would serve a valid penological end. ... [I]n determining eligibility for social services, adaptive functioning may be much more important.”
  2. Nat’l Health Service Choices, Controversy over DSM-5: New Mental Health Guide, Aug. 15, 2013.
  3. Gary Greenberg, The Rats of N.I.M.H., New Yorker, May 16, 2013 (quoting Thomas Insel, the director of the National Institute of Mental Health). 

POSTINGS ON IQ SCORES AND CAPITAL PUNISHMENT
Posted by DH Kaye at 2:47 PM 0 comments
Labels: , ,

Monday, June 2, 2014

Quarreling and Quibbling over Psychometrics in Hall v. Florida (part 2)

Justice Kennedy described the Florida law that prompted the trial court to reject Hall’s claim of intellectual disability as follows:
Florida's statute defines intellectual disability for purposes of an Atkins proceeding as “significantly subaverage general intellectual functioning existing concurrently with deficits in adaptive behavior and manifested during the period from conception to age 18.” Fla. Stat. § 921.137(1) (2013). The statute further defines “significantly subaverage general intellectual functioning” as “performance that is two or more standard deviations from the mean score on a standardized intelligence test.” Ibid. The mean IQ test score is 100. The concept of standard deviation describes how scores are dispersed in a population. Standard deviation is distinct from standard error of measurement, a concept which describes the reliability of a test and is discussed further below. The standard deviation on an IQ test is approximately 15 points, and so two standard deviations is approximately 30 points. Thus a test taker who performs “two or more standard deviations from the mean” will score approximately 30 points below the mean on an IQ test, i.e., a score of approximately 70 points.
Because standard deviations are fundamental to the Florida law, to the Court’s conclusions about it, and to its dicta regarding the lowest mandatory IQ cut-off that a state can use, I am going to be persnickety in unpacking this paragraph.

Although the Court is only discussing the standard deviation of IQ scores, “standard deviation” (SD) has a much broader meaning. When one considers the general meaning of the term, it becomes clear that the SD of the scores does not quite describe “how scores are dispersed in a population.” It merely indicates how much they are dispersed in either a population or a sample.

For example, the trial court heard testimony about at least four IQ test scores for Hall—71, 72, 73, and 80. (It declined to consider a score of 69 on another test because the psychologist who administered and scored the test was dead and Hall’s counsel had violated an order “to provide the State with the [underlying] testing materials and raw data.” Indeed, Hall had taken as many as nine IQ tests over a 40-year period.)  The standard deviation of the scores considered by the trial court is the square root of the average squared deviation from the mean—namely,

SD = {[(71–74)2 + (72–74)2 + (73–74)2 + (80–74)2]/4}1/2 = 3.53.

Tossing in the excluded score of 69 increases the SD to 3.74. The SD increases because the additional score is below the range of the other four, thus creating more variability in the sample (and lowering the mean from 74 to 73).

Of course, the Court’s number of 15 for the SD of IQ scores does not come from Hall’s scores. At this point, I use his scores only to elaborate on the Court’s observation that a standard deviation is a statistic that indicates how much the numbers in some set of numbers fluctuate around their mean. The standard deviation of 15 IQ points is an estimate of how much the scores of everyone in the general population—a large batch of numbers indeed—would vary if everyone took the test. The average score would be approximately 100, and there would be a lot of scatter around this mean. (In fact, the raw scores on the test are transformed in light of their mean and SD to force them to have a desired mean and SD near 100 and 15, respectively.) And, yes, 100 – (2×15) = 70, so Florida’s choice of 2 SDs to demarcate “significantly subaverage general intellectual functioning” translates into a score of 70 on a test with this mean and SD.

But the fact that every batch of numbers has a SD does not tell us “how [these numbers] are dispersed.” The numbers could be highly concentrated around a single value, with outliers on the flanks. Their distribution could be flat, with an equal fraction of the numbers spread out everywhere. The distribution might show clustering at several locations, and so on.

IQ scores, however, are dispersed approximately according to a “normal” or “Gaussian” curve. This distribution is the bell-shaped one prominent in elementary statistics courses. There are other bell-shaped curves, and all kinds of other interesting and important families of curves, but IQ scores, like many physical variables (such as weight and height), tend to be normally distributed across the members of a population (and hence in representative samples of that population).

The exact shape of all such normal distributions can be determined from two numbers—the mean and the standard deviation. The mean states where the bell sits, and the standard deviation determines how steeply its sides flow down from the top.You can see for yourself by entering your favorite means and standard deviations into the demonstration program in the OnlineStatBook.

Using the variable X to denote IQ scores and the symbol σx to designate their standard deviation, the particular normal distribution used in the Court’s calculation is such that, 2.28% of the scores lie below 70 (which, as the Court calculated it, corresponds to –2σx), and 4.75% fall below 75 (which, for the mean of 100 and standard deviation σx of 15, corresponds to –1.67σx). The latter IQ score, x = 75, is significant because Hall conceded (and the Court seemed to agree) that Florida could have chosen this score as its cut-off. For example, the Court expressed dissatisfaction that, in light of its calculations, the effect of Florida’s cut-off of –2σx was to preclude legally effective “professional[] diagnose[s of] intellectual disability [in a case like Hall’s, for which] the individual's IQ score is 75 or below.”

The dissent insisted that states should have more discretion to set cut-off scores. Unless –1.67σx (or 75) corresponds to the level of impairment that justifies a categorical rule, the majority has no satisfying reason to select one cut-off over the other. Why is the Court’s choice of 1.67 standard deviations below the mean the highest that the Constitution permits? Why is Florida’s two-standard-deviation rule insufficient?

The Court’s answer leans heavily on the standard error of measurement — another technical term that appears in the paragraph quoted above: “Standard deviation is distinct from standard error of measurement, a concept which describes the reliability of a test and is discussed further below.” But the standard error of measurement is also a standard deviation, one that is estimated, almost magically, from test reliability statistics. Thus, a more precise sentence would have been: “The standard deviation of all test scores is distinct from another standard deviation known as the standard error of measurement, which depends on the reliability of the test. We discuss the standard error of measurement below.”

POSTINGS ON IQ SCORES AND CAPITAL PUNISHMENT
Posted by DH Kaye at 1:42 PM 0 comments
Labels: , ,

Saturday, May 31, 2014

Quarreling and Quibbling over Psychometrics in Hall v. Florida (part 1)

In Hall v. Florida, 134 S.Ct. 1986 (2014), the Supreme Court struck down Florida’s practice of using an IQ score of 70 points or less as a dispositive measure of the level of the “intellectual disability” that precludes capital punishment. Justice Kennedy’s opinion (joined by Justices Breyer, Ginsburg, Sotomayor, and Kagan) summarizes the case pithily:
This Court has held that the Eighth and Fourteenth Amendments to the Constitution forbid the execution of persons with intellectual disability. Atkins v. Virginia, 536 U.S. 304, 321 (2002). Florida law defines intellectual disability to require an IQ test score of 70 or less. If, from test scores, a prisoner is deemed to have an IQ above 70, all further exploration of intellectual disability is foreclosed. This rigid rule, the Court now holds, creates an unacceptable risk that persons with intellectual disability will be executed, and thus is unconstitutional.
Id. at 1990. Led by Justice Alito, Chief Justice Roberts and Justices Scalia, Kennedy, and Thomas dissented. The Court, they maintained, was overruling Atkins and adopting “a uniform national rule that is both conceptually unsound and likely to result in confusion.” Id. at 2002 (Alito, J., dissenting). Among other things, the dissent warns that the Court “misunderstands” the statistical concepts of standard error and confidence intervals, id. at 2009, and that it therefore “makes factual mistakes that will surely confuse States attempting to comply with its opinion.” Id. at 2010.

The dissent has a point. Parts of the majority opinion are elliptical and potentially confusing. Nonetheless, some of the harsh critique is overdrawn. Moreover, Justice Alito's presentation of psychometric concepts also is hardly error-free, inviting a rejoinder of "tu quoque."

Therefore, in a series of postings yet to come, I will, in Justice Alito's words, "wade[] into technical matters that must be understood in order to see where the Court goes wrong." But I'll do the same for the dissenting opinion's presentation of these matters.

POSTINGS ON IQ SCORES AND CAPITAL PUNISHMENT
Posted by DH Kaye at 10:00 PM 0 comments
Labels: , ,