• Jonathan McCully

Spotting There Is A Problem


In order to take a human rights case to court, you have to first identify the human rights problem that you are asking the court to (re)solve.


In human rights cases, this problem will usually be in the form of a harm or threat of harm to one or more of the legally binding rights set out under human rights law.


The rights that an individual can legally enforce will depend on the law that is applicable to a specific context (which is explained in more detail here).


However, in order to aid the process of identifying a human rights problem, this article will examine the international human rights principles that are most commonly engaged by machine learning and similar algorithmic technologies.


It is important to note that the assessment of whether a person has a strong human rights case is a two-step process. Broadly speaking, it requires an examination of the following questions:

  • Has a human right been engaged? This involves a determination of whether a human right has been restricted, denied, threatened or otherwise interfered with.

  • Can this be justified by the other party? The law permits or allows certain restrictions on human rights. This means an assessment has to be made of whether the party responsible for engaging the right(s) was lawfully justified in doing so.

Assessing the strength of a human rights claim in a particular scenario can be a complex question, so it is always good to consult a lawyer to help do this.



Has a human right been engaged?


There are a range of human rights protected by international human rights law. It may be that one or more are engaged through the process of developing, designing, training, testing, using or maintaining a system that relies on machine learning or algorithmic processes.


To identify whether a right has been engaged involves looking at all the acts, decisions, behaviours or omissions that take place around these systems.


For instance, human rights can be engaged through the collection or use of data to build or run a system.


Alternatively, human rights might be engaged by an action, omission or measure taken (sometimes solely) on the basis of the output of a system. This might, for example, be a decision to sentence someone to prison or to determine what social security payments they are entitled to.


They can also be engaged by the more general impact their deployment or use has on society. For instance, human rights law has recognised (mostly in relation to free speech) a concept known as the “chilling effect.” This is where certain circumstances have the result of hindering or deterring people from exercising their rights. For example, having machine learning technologies around the home might lead to the feeling or fear of constantly being watched, and could prevent people from engaging in certain activities or behaviours protected by human rights.


This article will briefly summarise some of the human rights that are particularly likely to be engaged through the adoption of machine learning technologies. They are:


The Right to Equality and Non-Discrimination


It is a central tenet of human rights law that all people should be able to enjoy their human rights and freedoms on an equal basis. It is a cross-cutting human rights obligation.


This means that countries are not permitted to act in a discriminatory manner or pass discriminatory laws when it comes to individual rights. It also means that countries must put in place laws that protect individuals from discrimination.


There are a number of ways in which unlawful discrimination can manifest itself:

  • Direct Discrimination: this is the act of treating people in analogous or similar situations differently on the basis of an identifiable characteristic or status (e.g age, disability, gender, race, religion, sex, sexual orientation). This difference in treatment might be in the form of distinguishing, excluding, restricting or preferring a person on the basis of this characteristic or status. For example, using an automated system against one group and not another can amount to direct discrimination. Or where an automated system uses a protected characteristic or status to treat someone differently from others in similar circumstances will amount to direct discrimination. In this case study, for example, a computer system directly discriminated by refusing or deliberately omitting applications to medical school on the basis of race and gender. Direct discrimination can also occur where a person is treated differently because they are perceived, by an algorithm or otherwise, to have a certain characteristic or status.

  • Indirect Discrimination: this is where a general measure which, although seemingly neutral, has a particularly prejudicial and discriminatory effect or impact on a particular group. For example, some “neutral” factors weighed in an algorithm may in practice serve as a proxy for a protected characteristic or status. In this case study, for example, a gig economy algorithm that was seemingly neutral had a disproportionate and discriminatory impact on trade union workers’ ability to access work.

  • Discrimination by Association: this is where a person is discriminated against on the basis of the characteristics or status of a different individual somehow connected to that person. For example, an algorithm that flags individuals for extra scrutiny by police because of the religion of their parents will amount to discrimination by association. It has been suggested that association could include having been assumed (or profiled) to have a certain characteristic or status.

  • Proactive Action: this is where there has been a failure to take action where different treatment is necessary in order to correct for inequality. For example, a government might need to ensure that their system of welfare payments sufficiently takes into account disability in light of the inequalities of need brought about by this protected status.

In recent years, there have been ground-breaking studies that have exposed the bias of machine learning and automated systems by the likes of Timnit Gebru, Joy Buolamwini, Rediet Abebe, Inioluwa Deborah Raji, Safiya Noble, Ruha Benjamin, Simone Browne, Meredith Broussard, Caroline Criado Perez, and Cathy O’Neill.


These researchers have shown how machine learning and automated systems can increase and perpetuate inequality in ways that violate the right to non-discrimination.


In their work, they have highlighted that discrimination can happen at both a substantive and structural levels:

  • Substantive: the systems themselves, through the decisions they make and the inferences they draw, can discriminate. They can treat individuals in similar situations differently, including on the basis of race, gender, political opinion, nationality, religion and other protected characteristics. They can also (without necessarily being programmed to) have a disproportionate and discriminatory impact or effect on protected groups and communities. Moreover, machine learning technologies have consistently been found to struggle with understanding the full context around them, hindering their potential to take relevant differences into account to correct for inequality in a particular context. This spans the whole spectrum of unlawful discrimination outlined above.

  • Structural: the systems are also being used within processes and structures that are already discriminatory. This means they are contributing to, supporting, and further optimising or entrenching discriminatory practices. When systems are trained to “learn” patterns and rules by analysing existing data, they will replicate the discriminatory decision-making or practices represented by that data, thus further perpetuating existing discriminatory structures of oppression.

When machine learning and other technologies are developed and used, it is crucial that their human rights implications be assessed not only as to whether they engage the discrete rights set out below, but whether their engagement with these rights has occured in a discriminatory way. Emerging technologies have been shown to be capable of engaging a variety of rights in ways that sustain racial and ethnic exclusion in systemic or structural terms.



The Right to a Remedy


Another cross-cutting principle of international human rights law is the right to an effective remedy.


This right means that governments must ensure that individuals are actually able to bring claims to courts and other official bodies that are capable of resolving or redressing violations to their rights.


In other words, there must be real (rather than illusionary) accountability for violations.


Where machine learning technologies are implicated in an action, omission or behaviour that may have violated human rights, such as the right to privacy or non-discrimination, their “black box” nature can also engage the right to a remedy.


This is because their opacity and inscrutability can preclude individuals from challenging such measures before a court or other independent body. In other words, if people are unable to understand the factors, weighting, and reasons behind why something that engaged their rights was done, they will be unable to effectively challenge it.


Sometimes it is not just the difficulties around understanding how a system operates that can engage this right, but also the fact that such systems are often protected as “trade secrets” or by confidentiality laws preventing those impacted by (or even using) the systems from scrutinising them.


It has been recognised that a precondition to this right is “ensuring that individuals know that they have been subject to an algorithmic decision (including one that is suggested by an artificial intelligence system and approved by a human interlocutor) and are equipped with information about the logic behind that decision.”


Therefore, where a decision engaging human rights has been taken using an algorithmic system, and the individual affected either (i) does not know they have been subjected to such a system or (ii) does not know how it went about reaching that decision, then this right will likely be engaged.



The Right to Privacy


It is a recognised human right that everyone has the right to privacy. This means freedom from intrusion upon your private or family life. This aspect of the right will be engaged where, for example, a decision is made (whether with the aid of machine learning or not) to break up the family unit.


The right also includes the right to information self-determination, which means that people must be able to decide what information about themselves is used by others and under what circumstances.


This right to self-determination applies to situations where their personal data is collected, stored or used. Personal data is any data relating to a person who could be identified either from the data itself or other information.


This means the very act of collecting or receiving data can engage this right, as will the use and storage of such data. As set out in our Illustrated Guide to the Machine Learning Process, the process of working with machine learning involves working with data. Furthermore, when machine learning tools are run they can produce more data. So wherever and however these technologies are being used, they run the real risk of engaging this right.


The right includes a number of core principles aimed at ensuring data is handled in a fair, lawful and transparent way, including:

  • Purpose limitation: the specific purpose for which data is going to be used needs to be made clear at the outset to the person whose data it is, and any further use of such data needs to conform to this purpose. This means that re-purposing of data can engage the right to privacy. For example, data that has been collected in a medical context to treat a patient should not be repurposed in other contexts, e.g. to serve purposes of employers, businesses or social security services.

  • Collection limitation or data minimisation: data cannot be inadequate, irrelevant or more than what is necessary for the purpose it is used for. For example, machine learning technologies that are designed and used in the medical context to diagnose illness should not collect or require demographic data where that is not necessary for diagnosis.

  • Consent: central to the right is the principle that people should retain a level of control over their data. Therefore, in many contexts, data cannot be used without the free, specific, informed consent of the person whose data it is. This means that if the training, testing or use of machine learning technologies involves the collection or use of the data of people who were unaware of, did not agree to, changed their mind about or otherwise objected to such use, the right to privacy will likely be engaged. Furthermore, it has become a recognised aspect of the right to privacy that people should not be subject to a decision affecting their lives that has been taken by a machine without first having their views taken into account.

  • Data quality: steps need to be taken to make sure that data that is collected or stored is accurate and up to date. People should also be able to have their data rectified or erased. This aspect of the right to privacy is particularly crucial in machine learning contexts where the reliability of such systems depends on accurate data.

  • Design: privacy should be considered at the very beginning of any project that involves working with data, this means before a machine learning system is developed, and before any data is collected or used in the machine learning process. Privacy should lie at the heart of machine learning, and not just performance and functionality. This means that assessments should be carried out on machine learning technologies on their likely human rights impacts, to ensure an understanding of their risks and steps that can be taken to safeguard against such risk.

  • Transparency: data should be used in a transparent manner. This means people need to know how data is gathered and processed, by whom, for what purpose, for how long and with whom it is shared. When it comes to machine learning, people must also be able to know the logic and assumptions of such systems, as well as the significance of their envisioned consequences.

  • Sensitive data: some data is particularly sensitive, such as genetic, biometric and health data, as well as data on a person’s sexual orientation or sex-life. Machine learning tools might even generate sensitive data about people from non-sensitive data, for example by categorising someone as “gay” on the basis of their social media “likes.” This kind of data can only be used under strictly limited circumstances that are explicitly set out in a country’s laws.

  • Storage and security: data should not be kept longer than is needed, and it should be handled and used in a secure environment. This means protecting the data against loss, destruction or damage. Also, privacy should be the default, so if anonymous or pseudonymous data can be used, they should.

Machine learning techniques pose risks to many of these principles. Data is the fuel of such systems. So, where they are being developed, trained, tested or deployed, there is likely to be an engagement of the right to privacy.



Right to Liberty and a Fair Trial


The right to liberty and security of the person protects people from having their physical freedom restrained (i.e. the act of being detained) unlawfully or arbitrarily.


It is also recognised that all people are equal before the courts and that they should have a fair and public hearing. Furthermore, they have the right to be presumed innocent.


These principles are capable of being undermined by machine learning and similar technologies.


The use of risk scoring and other probabilistic models are increasingly being adopted in the criminal justice system. This includes contexts where decisions are made that result in the detention (or deprivation of liberty) of the individual (e.g. sentencing and bail decisions). These decisions are capable of engaging the right to liberty and a fair trial, and can jeopardise the fairness of a hearing (e.g. if the individual is unable to assess the accuracy and completeness of the evidence upon which the decision was based).


The presumption of innocence is also undermined by the “allure of objectivity” presented by machine learning and other algorithmic systems. When an individual is flagged as a “risk,” it can be difficult for a decisionmaker to freely disregard this output even when it might be based on inaccurate, discriminatory or flawed factors.


Although not likely to happen soon, if machine learning technologies completely took over the role of judges in court, this could engage the right in other ways. Judges must be impartial. Not only does this mean they cannot allow their judgment to be influenced by personal bias or prejudice, they must also “appear to a reasonable observer to be impartial.” It may be difficult for machine learning technologies to be trusted in this way.


Other threats to the right to liberty and security come from the use of machine learning technologies as an alternative to physical or chemical restraint, setting boundaries around where people can roam freely with the use of geolocation technologies. Concerns around such technology have been raised in the writing of law professor Chaz Arnett, who talks about the move from mass incarceration to mass e-carceration in the US, through the use of ankle bracelets, and the impact this has on rights such as the right to liberty.



The Right to Life

International human rights law recognises that every human being has the right to life. This means that nobody is permitted to end the life of another person.


Governments are also expected to introduce laws that protect life and, in some circumstances, take steps to protect against risk to life.


This means that where machine learning systems factor into life and death decisions, this right can be engaged.


For example, technological developments in machine learning capabilities have introduced new threats to this right, with the possibility for weapons to select and engage targets without intervention by a human operator. There is an international campaign to bring about an international ban on such weapons because of their threat to this and other rights.



Freedom of Opinion, Expression, Religion and Protest


International human rights law recognises a number of basic freedoms. It recognises the right of all people to form their own opinions, speak freely, look for and receive information, form peaceful protests, and have and practice religion. These rights and freedoms can be engaged by the context in which machine learning and similar technologies are used.


The right to freedom of expression recognises the right of all people to seek, receive and impart information and ideas of all kinds. Machine learning and other technologies dominate the online environment, including by removing, blocking, filtering, ranking, and personalising content. They are also used to aid the decisions of human content moderators. These practices raise their potential to engage the right to freedom of expression by undermining the rights of individuals to disseminate and access information without restriction or censorship.


The right to form an opinion and to develop this by way of reasoning might also be engaged by this use of algorithmically driven tools. It is possible that content curation and “micro-targeting,” through the use of machine learning and other technologies, can affect a person’s capacity to form an opinion.


Another aspect of the right to freedom of expression is the right to access information held by public bodies. This right is exercised most frequently through the filing of freedom of information requests with such bodies, which we discuss in more detail here. Such a right might be engaged if, for example, a machine learning or algorithmic system blocks the processing of such a request or rejects the request for illegitimate reasons. Where public authorities are using machine learning and similar technologies, it could be engaged by their refusal to disclose details of those technologies (such as the source code or algorithms).


The rights to speak freely, form peaceful protests, and have and practice religion can also be engaged by the profiling that can take place through machine learning and similar technologies. If an individual is flagged or treated differently in some way because they are perceived to have said something, attended a protest or practiced a certain religion, this might create a climate of fear or inhibition. Even by flagging or treating differently those who are perceived to have relationships with people exercising these rights can add to this climate. In this way, machine learning and other technologies are capable of having a “chilling effect” on the rights to freedom of expression, peaceful protest and religion.


Although there may be occasions when machine learning algorithms are limiting speech, there are also occasions where their failure to do so can raise human rights issues. It is an established principle of international human rights law that propaganda for war and advocacy of national, racial or religious hatred that incites discrimination, hostility or violence should be prohibited. Machine learning systems can be programmed or, through learning patterns in datasets, trained to produce this type of prohibited speech. If a government fails to sanction or take appropriate measures to prevent such technologies being used in this way they will be failing this obligation.



Rights of Persons with Disabilities and Older Persons


Under international human rights law, countries must protect and promote the rights of persons with disabilities and older persons by eliminating discrimination, enabling them to live independently in the community, and ensuring they are protected from all forms of exploitation, violence and abuse.


The emergence of technologies that rely on machine learning and other capabilities has been heralded as potentially promoting the security, inclusion and independence of people with disabilities and older persons.


However, the use of such capabilities in tools like mobility aids, as well as assistive technologies and devices, also gives rise to real threats to a number of rights:

  • Autonomy: an interference with human rights will occur where the use of the technology interferes with a person’s ability to make decisions for themselves and their ability to maintain choice and control of their living arrangements, such as their daily schedule, routine and way of life. In care contexts, automated technologies should not be used to override or substitute the decisions of a person receiving support. Furthermore, individuals must be able to give free and informed consent to assistive technologies being used or withdrawn from use. They must also be able to change their mind on the use of such technologies. For consent to be free and informed, an individual must be provided with clear and accurate information in order to assess the implications of the technology before giving consent.

  • Dignity: the overreliance of machine learning and similar capabilities, like “care robots," poses a particular threat to the dignity of those most in need of care or support. It can lead to segregation, neglect and abandonment by dehumanising care practices and encouraging collective disengagement in providing human care and support. Therefore, international human rights law is clear. These technologies cannot be a substitute for human care. There is also a risk that, by poor training or malfunction, technologies could contribute towards abuse, neglect or maltreatment.

  • Equality of access: access to assistive technologies or support cannot be to the exclusion of others who have an equal right and comparable need to access support. This means that countries should be prioritising the research and development of affordable technologies so a greater proportion of society can benefit from access.

  • Indirect impact (“chilling effect”): it also has been noted by a human rights expert that technology that involves increased monitoring of caregivers in care contexts might lead to their withdrawal of care (or it might change the way they provide care) to the detriment of the needs of people with disabilities and older people.

Where automated systems are used in a way that engage any of these threats, there may be potential to raise a human rights claim.



Right to an Adequate Standard of Living


The right to an adequate standard of living requires that everyone shall enjoy the rights to adequate food and nutrition, clothing, housing, social security and other basic needs.


This right will be interfered with where access to basic needs is impeded or denied, including through the application of automated decision-making systems.


In the social security context, for example, such systems have been used to verify identity, assess eligibility for welfare, determine need, calculate payments, detect and prevent fraud, and communicate with welfare beneficiaries (i.e. through chatbots). The increased and thoughtless adoption of these systems has been referred to as “stumbling zombie-like into a digital welfare dystopia.” Similar systems can also be found in the provision of other basic needs, such as housing and care, and they have been criticised for profiling, policing and punishing the poor.


Machine learning and other technologies can malfunction, rely on inaccurate data and aid or make discriminatory decisions in a way that can effectively deny people access to basic sustenance and resources they need to survive. As a case in India has shown, the technical malfunctioning of a system used in food provision has led to starvation and even death.


Where machine learning technologies are being used in a way that denies or otherwise inhibits a person’s ability to attain an adequate standard of living, there will be an interference with that person’s human rights.



Right to Work


International human rights law recognises the right of everyone to the opportunity to gain their living by work which they freely choose or accept. It also recognises the right to just and favourable work conditions. Also, under this right, is the right to form trade unions and to strike.


Automated systems are transforming the world of work in ways that can interfere with these rights. For example;