A secret algorithm used to fire teachers
Country: United States of America
Forum: United States District Court, Texas, Houston Division
Between 2011 and 2015, the Houston Independent School District started using a secret statistical model to evaluate and fire teachers. This model was developed by a private company, that treated its source code and other information about the statistical modelling as a trade secret.
The model, referred to as the “Educational Value-Added Assessment System,” sought to measure teacher effectiveness by tracking student growth. It tried to do this by linking student test scores to their teachers and measuring changes in students’ standardised test scores over time. This data was then aggregated to give a teacher a value-added estimate. This estimate was then compared with similar teachers to give a teacher an effectiveness rating.
In 2012, the Houston Independent School District introduced an aggressive policy to terminate the contracts of teachers rated as ineffective. A number of teachers lost their jobs without gaining access to the statistical modelling used to evaluate their effectiveness.
The Houston Federation of Teachers, alongside six teachers, sued the Houston Independent School District arguing that the use of EVAAS scores in the termination of teacher contracts violated due process rights.
The District Court agreed with one of the teachers’ arguments, that teachers lacked sufficient information to meaningfully challenge terminations based on low EVAAS scores. After this decision, the Houston Independent School District agreed to stop using EVAAS scores in personnel decisions.
In 2010, the Houston Independent School District started relying on a “data driven” teacher appraisal system. When it was rolled out, this system evaluated teachers based on three factors: (i) instructional practice, (ii) professional expectations, and (iii) student performance. The third factor, student performance, involved the use of a value-added model for evaluating teacher effectiveness.
This model tried to evaluate a teacher’s impact by measuring growth in student scores in standardised tests. Under the Houston Independent School District’s policy, student growth would be measured by a value-added statistical model called the “Educational Value-Added Assessment System” (EVAAS) that had been developed by the private software company called SAS and licensed for use by the school district.
In general terms, the system gave a teacher a score based on a comparison between the average test score growth of students taught by the teacher and the statewide average for students in that grade or course. This score was generated by an algorithm that was treated as a trade secret by SAS. The score would then be used to sort teachers into one of five “value-added” effectiveness ratings: (i) well above, (ii) above, (iii) no detectable difference, (iv) below, (v) well below.
The system had been criticised by teachers for its inherent unfairness. For example, some teachers complained that they were being evaluated via EVAAS using tests that did not match the subject areas or curricula they taught. Some teachers also alleged that EVAAS failed to control for biasing factors that impact how well teachers perform but that were outside of teachers’ control, such as race, socioeconomic, and language factors.
In 2012, shortly after adopting EVAAS to rate teacher effectiveness, the Houston Independent School District added “insufficient student academic growth as reflected by value-added data” as a reason for non-renewal in their teacher non-renewal policy. They also announced a district-wide goal of ensuring that “no more than 15% of teachers with ratings of ineffective are retained district-wide.”
Between 2012 and 2014, dozens of teachers were proposed for termination or non-renewal of their teaching contracts due to “performance and a significant lack of student progress attributable to the educator.”
Some limited information was provided by the school district to teachers about EVAAS, such as an overview of the value-added growth as a measure of student learning, a general description of the EVAAS test methods and how they are applied in the district, and how to read the EVAAS teacher value added report. The source codes and other information underlying the EVAAS statistical methodology were not made available by SAS to teachers, or even the school district itself.
The Houston Federation of Teachers and six teachers sought to challenge the constitutionality of this system. They argued that the use of EVAAS violated due process rights. These rights serve the purpose of ensuring fair procedures are followed when depriving an individual of life, liberty or property. They argued that the Houston Independent School District’s use of EVAAS violated due process rights in the following ways:
There was a lack of sufficient information to meaningfully challenge terminations based on low EVAAS scores;
There was no rational relationship between EVAAS scores and the school district’s goal of employing effective teachers; and
The system was too vague to provide notice to teachers of how to achieve higher ratings and avoid adverse employment consequences.
The Houston Independent School District sought a summary judgment, which is where a court considers whether arguments made by a party are viable for a full trial.
In the US, due process is a core constitutional right provided through the Fifth and Fourteenth Amendments.
Fifth Amendment of the US Constitution: No person shall be… deprived of life, liberty, or property, without due process of law.
Fourteenth Amendment of the US Constitution: [No State may] deprive any person of life, liberty, or property without due process of law; nor deny to any person within its jurisdiction the equal protection of laws.
The Court began by reiterating that the right to due process prohibits a state from depriving any person of property without following due process of law. Therefore, in order to determine whether the right was engaged in this case, the Court had to ascertain whether the case concerned a constitutionally protected property interest.
In this regard, the Court concluded that the Houston Independent School District’s value-added appraisal system posed a realistic threat to deprive teachers of their constitutionally protected property interests in employment.
The Court then went on to consider the argument that the use of EVAAS was in violation of due process because it did not provide sufficient information for a teacher to meaningfully challenge terminations based on their EVAAS score.
The Court noted that minimum standards of due process, in the context of public school teacher terminations, included the right to be “advised of the cause for their termination in sufficient detail so as to enable them to show any error that may exist.”
On this point, the Court took into account a number of factors. They noted that the school district itself did not calculate the EVAAS score for its teachers, instead this was carried out by the third-party vendor SAS. The scores were generated by the application of complex, secret algorithms that were not divulged to either the school district or the teachers themselves. The school district did not carry out verification or auditing on the scores received from SAS. The Court also recognised that it was impossible for teachers to replicate their scores with the limited information that was made available to them.
The Court was also concerned about the fragility of the system for calculating these scores. It observed that scores can be erroneously calculated for a number of reasons, ranging from data-entry mistakes to computer glitches. Furthermore, because of the interconnectivity of the system, an inaccurate score for one single teacher can alter the scores of every other teacher in the district. In other words, the accuracy of one score hinges on the accuracy of all. This was described as “house-of-cards fragility” by the Court. It also meant that any teacher facing termination for a low value-added score would necessarily be unable to verify that their own score is error-free without access to data supporting all teacher scores.
The Court rejected an argument advanced by the school district that the scores were merely one factor that might be considered by relevant decisionmakers. It observed that the school district’s aggressive goal of “exiting” 85% of teachers with “ineffective” EVAAS ratings made it near impossible for a decisionmaker to freely disregard an EVAAS score.
The Court concluded that the information made available to teachers was inadequate. It found that “without access to SAS’s proprietary information – the value-added equations, computer source codes, decision rules, and assumptions – EVAAS scores will remain a mysterious ‘black box,’ impervious to challenge.” This meant that teachers had no meaningful way of ensuring correct calculation of their scores and were, therefore, unfairly subject to mistaken deprivation of constitutionally protecting property interests in their jobs.
The Court did not agree with the other arguments put forward by the parties taking the case. They found that, even if the EVAAS could be considered a “blunt tool” with only marginal results, there was a sufficiently rational relationship between EVAAS scores and the school district’s goal of employing effective teachers. Finally, the Court found that the use of EVAAS was not too vague to provide notice to teachers of how to achieve higher ratings and avoid adverse employment consequences. It reasoned that, although teachers could not verify the accuracy of their scores, they were given sufficient information to evaluate their behaviour’s conformity to the dismissal standard provided by the policies implementing EVAAS.
The District Court upheld the claim that the use of EVAAS scores by the Houston Independent School District violated procedural due process rights. Because this was a summary judgment, it signalled that the claim was worthy of being considered at a full trial. However, before the case could go to full trial, the Houston Independent School District settled the case agreeing to pay the other parties’ legal fees and that they would stop using EVAAS to make personnel decisions.