Spotting There Is A Problem


In order to take a human rights case to court, you have to first identify the human rights problem that you are asking the court to (re)solve.


In human rights cases, this problem will usually be in the form of a harm or threat of harm to one or more of the legally binding rights set out under human rights law.


The rights that an individual can legally enforce will depend on the law that is applicable to a specific context (which is explained in more detail here).


However, in order to aid the process of identifying a human rights problem, this article will examine the international human rights principles that are most commonly engaged by machine learning and similar algorithmic technologies.


It is important to note that the assessment of whether a person has a strong human rights case is a two-step process. Broadly speaking, it requires an examination of the following questions:

  • Has a human right been engaged? This involves a determination of whether a human right has been restricted, denied, threatened or otherwise interfered with.

  • Can this be justified by the other party? The law permits or allows certain restrictions on human rights. This means an assessment has to be made of whether the party responsible for engaging the right(s) was lawfully justified in doing so.

Assessing the strength of a human rights claim in a particular scenario can be a complex question, so it is always good to consult a lawyer to help do this.



Has a human right been engaged?


There are a range of human rights protected by international human rights law. It may be that one or more are engaged through the process of developing, designing, training, testing, using or maintaining a system that relies on machine learning or algorithmic processes.


To identify whether a right has been engaged involves looking at all the acts, decisions, behaviours or omissions that take place around these systems.


For instance, human rights can be engaged through the collection or use of data to build or run a system.


Alternatively, human rights might be engaged by an action, omission or measure taken (sometimes solely) on the basis of the output of a system. This might, for example, be a decision to sentence someone to prison or to determine what social security payments they are entitled to.


They can also be engaged by the more general impact their deployment or use has on society. For instance, human rights law has recognised (mostly in relation to free speech) a concept known as the “chilling effect.” This is where certain circumstances have the result of hindering or deterring people from exercising their rights. For example, having machine learning technologies around the home might lead to the feeling or fear of constantly being watched, and could prevent people from engaging in certain activities or behaviours protected by human rights.


This article will briefly summarise some of the human rights that are particularly likely to be engaged through the adoption of machine learning technologies. They are:


The Right to Equality and Non-Discrimination


It is a central tenet of human rights law that all people should be able to enjoy their human rights and freedoms on an equal basis. It is a cross-cutting human rights obligation.


This means that countries are not permitted to act in a discriminatory manner or pass discriminatory laws when it comes to individual rights. It also means that countries must put in place laws that protect individuals from discrimination.


There are a number of ways in which unlawful discrimination can manifest itself:

  • Direct Discrimination: this is the act of treating people in analogous or similar situations differently on the basis of an identifiable characteristic or status (e.g age, disability, gender, race, religion, sex, sexual orientation). This difference in treatment might be in the form of distinguishing, excluding, restricting or preferring a person on the basis of this characteristic or status. For example, using an automated system against one group and not another can amount to direct discrimination. Or where an automated system uses a protected characteristic or status to treat someone differently from others in similar circumstances will amount to direct discrimination. In this case study, for example, a computer system directly discriminated by refusing or deliberately omitting applications to medical school on the basis of race and gender. Direct discrimination can also occur where a person is treated differently because they are perceived, by an algorithm or otherwise, to have a certain characteristic or status.

  • Indirect Discrimination: this is where a general measure which, although seemingly neutral, has a particularly prejudicial and discriminatory effect or impact on a particular group. For example, some “neutral” factors weighed in an algorithm may in practice serve as a proxy for a protected characteristic or status. In this case study, for example, a gig economy algorithm that was seemingly neutral had a disproportionate and discriminatory impact on trade union workers’ ability to access work.

  • Discrimination by Association: this is where a person is discriminated against on the basis of the characteristics or status of a different individual somehow connected to that person. For example, an algorithm that flags individuals for extra scrutiny by police because of the religion of their parents will amount to discrimination by association. It has been suggested that association could include having been assumed (or profiled) to have a certain characteristic or status.

  • Proactive Action: this is where there has been a failure to take action where different treatment is necessary in order to correct for inequality. For example, a government might need to ensure that their system of welfare payments sufficiently takes into account disability in light of the inequalities of need brought about by this protected status.

In recent years, there have been ground-breaking studies that have exposed the bias of machine learning and automated systems by the likes of Timnit Gebru, Joy Buolamwini, Rediet Abebe, Inioluwa Deborah Raji, Safiya Noble, Ruha Benjamin, Simone Browne, Meredith Broussard, Caroline Criado Perez, and Cathy O’Neill.


These researchers have shown how machine learning and automated systems can increase and perpetuate inequality in ways that violate the right to non-discrimination.


In their work, they have highlighted that discrimination can happen at both a substantive and structural levels:

  • Substantive: the systems themselves, through the decisions they make and the inferences they draw, can discriminate. They can treat individuals in similar situations differently, including on the basis of race, gender, political opinion, nationality, religion and other protected characteristics. They can also (without necessarily being programmed to) have a disproportionate and discriminatory impact or effect on protected groups and communities. Moreover, machine learning technologies have consistently been found to struggle with understanding the full context around them, hindering their potential to take relevant differences into account to correct for inequality in a particular context. This spans the whole spectrum of unlawful discrimination outlined above.

  • Structural: the systems are also being used within processes and structures that are already discriminatory. This means they are contributing to, supporting, and further optimising or entrenching discriminatory practices. When systems are trained to “learn” patterns and rules by analysing existing data, they will replicate the discriminatory decision-making or practices represented by that data, thus further perpetuating existing discriminatory structures of oppression.

When machine learning and other technologies are developed and used, it is crucial that their human rights implications be assessed not only as to whether they engage the discrete rights set out below, but whether their engagement with these rights has occured in a discriminatory way. Emerging technologies have been shown to be capable of engaging a variety of rights in ways that sustain racial and ethnic exclusion in systemic or structural terms.



The Right to a Remedy


Another cross-cutting principle of international human rights law is the right to an effective remedy.


This right means that governments must ensure that individuals are actually able to bring claims to courts and other official bodies that are capable of resolving or redressing violations to their rights.


In other words, there must be real (rather than illusionary) accountability for violations.


Where machine learning technologies are implicated in an action, omission or behaviour that may have violated human rights, such as the right to privacy or non-discrimination, their “black box” nature can also engage the right to a remedy.


This is because their opacity and inscrutability can preclude individuals from challenging such measures before a court or other independent body. In other words, if people are unable to understand the factors, weighting, and reasons behind why something that engaged their rights was done, they will be unable to effectively challenge it.


Sometimes it is not just the difficulties around understanding how a system operates that can engage this right, but also the fact that such systems are often protected as “trade secrets” or by confidentiality laws preventing those impacted by (or even using) the systems from scrutinising them.


It has been recognised that a precondition to this right is “ensuring that individuals know that they have been subject to an algorithmic decision (including one that is suggested by an artificial intelligence system and approved by a human interlocutor) and are equipped with information about the logic behind that decision.”


Therefore, where a decision engaging human rights has been taken using an algorithmic system, and the individual affected either (i) does not know they have been subjected to such a system or (ii) does not know how it went about reaching that decision, then this right will likely be engaged.



The Right to Privacy


It is a recognised human right that everyone has the right to privacy. This means freedom from intrusion upon your private or family life. This aspect of the right will be engaged where, for example, a decision is made (whether with the aid of machine learning or not) to break up the family unit.


The right also includes the right to information self-determination, which means that people must be able to decide what information about themselves is used by others and under what circumstances.


This right to self-determination applies to situations where their personal data is collected, stored or used. Personal data is any data relating to a person who could be identified either from the data itself or other information.


This means the very act of collecting or receiving data can engage this right, as will the use and storage of such data. As set out in our Illustrated Guide to the Machine Learning Process, the process of working with machine learning involves working with data. Furthermore, when machine learning tools are run they can produce more data. So wherever and however these technologies are being used, they run the real risk of engaging this right.


The right includes a number of core principles aimed at ensuring data is handled in a fair, lawful and transparent way, including:

  • Purpose limitation: the specific purpose for which data is going to be used needs to be made clear at the outset to the person whose data it is, and any further use of such data needs to conform to this purpose. This means that re-purposing of data can engage the right to privacy. For example, data that has been collected in a medical context to treat a patient should not be repurposed in other contexts, e.g. to serve purposes of employers, businesses or social security services.

  • Collection limitation or data minimisation: data cannot be inadequate, irrelevant or more than what is necessary for the purpose it is used for. For example, machine learning technologies that are designed and used in the medical context to diagnose illness should not collect or require demographic data where that is not necessary for diagnosis.

  • Consent: central to the right is the principle that people should retain a level of control over their data. Therefore, in many contexts, data cannot be used without the free, specific, informed consent of the person whose data it is. This means that if the training, testing or use of machine learning technologies involves the collection or use of the data of people who were unaware of, did not agree to, changed their mind about or otherwise objected to such use, the right to privacy will likely be engaged. Furthermore, it has become a recognised aspect of the right to privacy that people should not be subject to a decision affecting their lives that has been taken by a machine without first having their views taken into account.

  • Data quality: steps need to be taken to make sure that data that is collected or stored is accurate and up to date. People should also be able to have their data rectified or erased. This aspect of the right to privacy is particularly crucial in machine learning contexts where the reliability of such systems depends on accurate data.

  • Design: privacy should be considered at the very beginning of any project that involves working with data, this means before a machine learning system is developed, and before any data is collected or used in the machine learning process. Privacy should lie at the heart of machine learning, and not just performance and functionality. This means that assessments should be carried out on machine learning technologies on their likely human rights impacts, to ensure an understanding of their risks and steps that can be taken to safeguard against such risk.

  • Transparency: data should be used in a transparent manner. This means people need to know how data is gathered and processed, by whom, for what purpose, for how long and with whom it is shared. When it comes to machine learning, people must also be able to know the logic and assumptions of such systems, as well as the significance of their envisioned consequences.

  • Sensitive data: some data is particularly sensitive, such as genetic, biometric and health data, as well as data on a person’s sexual orientation or sex-life. Machine learning tools might even generate sensitive data about people from non-sensitive data, for example by categorising someone as “gay” on the basis of their social media “likes.” This kind of data can only be used under strictly limited circumstances that are explicitly set out in a country’s laws.

  • Storage and security: data should not be kept longer than is needed, and it should be handled and used in a secure environment. This means protecting the data against loss, destruction or damage. Also, privacy should be the default, so if anonymous or pseudonymous data can be used, they should.

Machine learning techniques pose risks to many of these principles. Data is the fuel of such systems. So, where they are being developed, trained, tested or deployed, there is likely to be an engagement of the right to privacy.



Right to Liberty and a Fair Trial


The right to liberty and security of the person protects people from having their physical freedom restrained (i.e. the act of being detained) unlawfully or arbitrarily.


It is also recognised that all people are equal before the courts and that they should have a fair and public hearing. Furthermore, they have the right to be presumed innocent.


These principles are capable of being undermined by machine learning and similar technologies.


The use of risk scoring and other probabilistic models are increasingly being adopted in the criminal justice system. This includes contexts where decisions are made that result in the detention (or deprivation of liberty) of the individual (e.g. sentencing and bail decisions). These decisions are capable of engaging the right to liberty and a fair trial, and can jeopardise the fairness of a hearing (e.g. if the individual is unable to assess the accuracy and completeness of the evidence upon which the decision was based).


The presumption of innocence is also undermined by the “allure of objectivity” presented by machine learning and other algorithmic systems. When an individual is flagged as a “risk,” it can be difficult for a decisionmaker to freely disregard this output even when it might be based on inaccurate, discriminatory or flawed factors.


Although not likely to happen soon, if machine learning technologies completely took over the role of judges in court, this could engage the right in other ways. Judges must be impartial. Not only does this mean they cannot allow their judgment to be influenced by personal bias or prejudice, they must also “appear to a reasonable observer to be impartial.” It may be