Loomis v. Wisconsin
COMPAS risk scores on trial
Country: United States
Forum: Supreme Court of Wisconsin
Within the criminal justice system, judges are responsible for imposing an appropriate punishment on those who have been convicted of a crime. The process of imposing a sentence often includes consideration of the risk that a particular offender will commit a criminal offence again (this is referred to as “recidivism”).
With the rise of Big Data, recidivism risk assessments have become increasingly common in criminal courts in the US in assisting judges in reaching a decision on sentencing. These assessments try to calculate the likelihood of an individual committing another crime based on an evaluation of statistical data. The assessment used in this case to sentence an individual (Eric Loomis) was the COMPAS system, a risk scoring model originally designed for assessing offender needs in relation to treatment and supervision, not sentencing.
Eric Loomis sought to challenge his sentence on the basis that the sentencing court took into account the risk assessment outcomes produced by this hidden statistical model that was unable to make individualised decisions (it was restricted to identifying high risk groups, rather than individuals).
He argued that, by considering his COMPAS risk assessment when reaching a decision on his sentence, the court had violated his “due process rights.” These rights serve the purpose of ensuring fair procedures are followed when depriving an individual of life, liberty or property.
Eric’s case went to the Wisconsin Supreme Court, which found that, if used properly, a court’s consideration of a COMPAS risk assessment at sentencing would not amount to a violation of an offender’s right to due process.
In Eric’s case, because the court explained that its decision was supported by other independent factors than the risk assessment, the use of COMPAS was not a determinative factor in his sentencing decision. Therefore, his due process rights were not violated.
Nonetheless, the Supreme Court did recognise that there were only limited circumstances when a COMPAS system could be used by sentencing courts, and that sentencing courts must be made aware of the limitations and cautions associated with the system to avoid due process violations.
In the US, risk assessment tools have been adopted across the criminal justice system. Initially they were primarily used to inform decisions on supervision and treatment of offenders. However, these assessment tools started to expand into decisions on sentencing.
This case concerned Eric Loomis, who was charged in 2013 with five offences related to a drive-by shooting. He entered a guilty plea for some of these charges, and others were dismissed.
At this stage, a Wisconsin Department of Corrections officer prepared a Presentencing Investigation Report. This report usually includes background information on the offender and, on this occasion, it also included a COMPAS risk assessment.
The COMPAS risk assessment tool estimates the risk of recidivism based on an interview with the offender and information from the offender’s criminal history. Risk scores are displayed as a bar chart, which consists of three bars showing (1) pretrial recidivism risk, (2) general recidivism risk, and (3) violent recidivism risk. These risk scores fall on a scale of one to ten.
It is only the estimates of recidivism risk that are reported to the court, the model used for reaching these estimates is protected as a “trade secret.” Northpointe Inc, the developer of COMPAS, considers COMPAS a proprietary instrument and a “trade secret.” Accordingly, it does not disclose how risk scores are determine or how factors are weighed.
According to Eric’s COMPAS risk score, he was indicated as having a high risk of recidivism in all three bars in the bar chart. The risk scores were accompanied by a warning to the court against misuse of the scores. It also warned that the risk assessment scores “are not intended” to determine the severity of a sentence or whether an offender should be incarcerated.
The court sentenced Eric to six years of imprisonment and five years of extended supervision. They based this decision, in part, on Eric’s COMPAS risk scores. In sentencing, the court also made reference to the seriousness of the offences he pled guilty to, Eric’s criminal history, his history under supervision and the other charges that had been brought against him in relation to the drive-by shooting.
Eric challenged this decision, before the same court, claiming that the court’s consideration of the COMPAS risk assessment at sentencing was a violation of his due process rights. Eric had an expert witness give testimony to the court on the fact that COMPAS was not designed for incarceration decisions and ran the risk of overestimating individual risk in such a context. The court disagreed with Eric, stating that it simply used the COMPAS risk assessment to corroborate its finding and that it would have imposed the same sentence regardless of the risk scores. This finding was appealed to the Wisconsin Supreme Court.
Eric argues that his right to due process had been violated for three reasons: (1) his right to be sentenced based upon accurate information was violated, in part, because he was unable to assess the accuracy of COMPAS due to its proprietary nature, (2) his right to an individualised sentence was violated, and (3) COMPAS assessments take gender into account.
In the US, due process is a core constitutional right provided through the Fifth and Fourteenth Amendments.
Fifth Amendment of the US Constitution: No person shall be… deprived of life, liberty, or property, without due process of law.
Fourteenth Amendment of the US Constitution: [No State may] deprive any person of life, liberty, or property without due process of law; nor deny to any person within its jurisdiction the equal protection of laws.
The Wisconsin Supreme Court noted the need for evidence-based sentencing and warned against decisions being based on a judge’s intuition. At the same time, the Court highlighted the differences between a sentencing decision and a corrections decision. It noted that the COMPAS tool had been designed for the latter and were being transitioned into the sentencing context where due process guarantees apply.
They also noted concerns raised around the risk of such tools being misused where decisionmakers, like judges, are not made aware of their limitations and concerns around efficacy. The Court concluded that risk scores may not be considered as the determinative factor in deciding whether an offender can be supervised safely and effectively in the community.
The Supreme Court went on to consider whether Eric’s due process rights had been violated in his specific case, before going on to raise some cautions that sentencing courts should take into account when using the COMPAS tool.
Right to be sentenced on accurate information
The Supreme Court considered whether Eric’s due process right to be sentenced on the basis of accurate information had been violated, taking into account the fact that he was unable to challenge the scientific validity of the risk assessment score because the model used for reaching the risk score could not be disclosed to him.
The Supreme Court acknowledged that this right included the right to review and verify information contained in the Presentencing Investigation Report upon which the court bases its sentencing decision. If details in this report are incorrect or incomplete, the offender is the person best placed to refute, explain or supplement this information.
The Supreme Court disagreed that Eric needed access to how factors are weight and how risk scores are determined to challenge the accuracy of the Presentencing Investigation Report. He could have reviewed and challenged the resulting risk scores set forth in the report.
Furthermore, Eric had access to the questions and answers that were treated as “static factors” in the risk assessment. The Supreme Court reasoned that he could have verified the accuracy of these.
The Supreme Court then turned to whether the risk assessment tool was itself accurate. It noted that some validation studies had been carried out on the COMPAS system that indicated its reliability. However, it also noted the concerns that risk assessment tools may disproportionately classify minority offenders as higher risk, often due to factors that were outside their control.
It acknowledged that studies had indicated that COMPAS was more predictive of recidivism among white offenders than black offenders. The Supreme Court also highlighted the need to closely monitor and maintain such tools for accuracy. These concerns were later reflected in the limitations and cautions the Supreme Court believed should be provided to sentencing courts when considering COMPAS scores (see below).
Right to individualised sentencing
The Supreme Court then looked at Eric’s argument that he did not receive an individualised sentence because consideration of COMPAS risk assessments amounts to sentencing based on group data.
The Supreme Court agreed that COMPAS is designed to assess group data. COMPAS is intended to identify groups of high-risk offenders rather than a particular high-risk individual.
The Supreme Court acknowledged that where a COMPAS risk score is a determinative factor in sentencing, which it found not to be the case with Eric’s sentencing, this would raise “due process challenges regarding whether a defendant received an individualised sentence.” However, the court did not find fault with risk scores being used with other supporting factors to reach an individualised sentence.
Right not to be sentenced on basis of gender
The Supreme Court turned to Eric’s argument that, because COMPAS risk scores appear to take gender into account, this violated his due process right not to be sentenced on the basis of gender.
As there was statistical evidence that men (on average) have higher reoffending rates compared to women, the Supreme Court concluded that there was a factual basis underlying COMPAS’ use of gender in calculating risk scores. It reasoned that “any risk assessment tool which fails to differentiate between men and woman [sic] will misclassify both genders.” Since inclusion of gender promoted accuracy, the court concluded that it served the interests of justice rather than a discriminatory purpose.
The Supreme Court also reasoned that there was no evidence to show that the sentencing court actually relied on gender as a factor in sentencing.
The Supreme Court came to the conclusion that, although it could not be determinative, a sentencing court could use COMPAS risk scores as a relevant factor for:
Diverting low-risk prison-bound offenders to a non-prison alternative;
Assessing whether an offender can be supervised safely and effectively in the community; and
Imposing terms and conditions of probation or supervision (e.g. reporting requirements, drug testing, electronic monitoring, community service, and treatment strategies).
It went on to note that COMPAS was an ill fit for determining the length and severity of a sentence. That is because the system was not aimed at addressing all the goals of a sentence. Its aim was to address treatment needs of an individual and identify risk of recidivism. Sentencing serves other purposes, such as deterrence, rehabilitation, and retribution.
As a result, a sentencing court should consider a COMPAS risk assessment at sentencing subject to limitations. It should not be used to determine whether an offender is incarcerated or to determine the severity of a sentence. Furthermore, it should not be used as a determinative factor in deciding whether an offender be supervised safely and effectively in the community.
A sentencing court must, therefore, explain the factors in addition to COMPAS that independently supports the sentence imposed.
The Supreme Court also stated that any Presentence Investigation Report containing a COMPAS risk assessment filed with the court must contain a “written advisement” listing the limitations of the system. It should include the following cautions:
The proprietary nature of COMPAS has been invoked to prevent disclosure of information relating to how factors are weighed or how risk scores are determined;
Because COMPAS risk assessment scores are based on group data, they are able to identify groups of high-risk offenders – not a particular high-risk individual;
Some studies of COMPAS risk assessment scores have raised questions about whether they disproportionately classify minority offenders as having a higher risk of recidivism;
A COMPAS risk assessment compares defendants to a national sample, but no cross-validation study for a Wisconsin population has yet been completed. Risk assessment tools must be constantly monitored and re-normed for accuracy due to changing populations and sub-populations;
COMPAS was not developed for use at sentencing, but was intended for use by the Department of Corrections in making determinations regarding treatment, supervision, and parole.
It went on to say that such cautions should be regularly updated as additional data becomes available.
Was Eric’s case a permissible use?
The Supreme Court finally applied their position on permissible uses of COMPAS to the facts of Eric’s case. It concluded that the sentencing court was aware of the limitations of COMPAS, two of which were mentioned in the Presentencing Investigation Report, and had given the scores “little or no weight” in sentencing Eric Loomis.
Instead, it was used to corroborate the court’s findings and the court would have imposed the same sentence if it did not receive the COMPAS risk assessment. The Supreme Court noted that the sentencing court, in reaching its decision, considered the seriousness of the crime, Eric’s criminal history, his character and rehabilitative needs, and the need to protect the public.
Accordingly, the Supreme Court found that the COMPAS risk assessment was not determinative in deciding whether Eric should be incarcerated, the severity of the sentence, or whether he could be supervised safely and effectively in the community. In short, Eric’s due process rights had not been violated through the sentencing court’s consideration of COMPAS.
The Wisconsin Supreme Court ultimately upheld the constitutionality of Eric’s sentence. Nonetheless, it clarified the limitations on the use of COMPAS risk assessments in criminal sentencing.