A risk model for detecting welfare fraud violates privacy
Country: The Netherlands
Forum: District Court of the Hague
This case concerned a legal challenge to a piece of Dutch law that set out the process for government use of a secret risk model to identify individuals at risk of having committed welfare and other types of fraud. A coalition of civil society organisations, and two public figures, brought the case.
The tool that was used in this process was called the Systeem Risicoindicatie (System Risk Indication), or SyRI for short. Although the algorithms powering the system were kept a secret, the Dutch government claimed that the tool would link and compare personal data from various government databases and these comparisons would be analysed by a risk model for “potential hits.” These “potential hits” were further assessed for inclusion in a risk report that would be shared with relevant government bodies, the Public Prosecution Service or the police.
The system would process a large quantity of data about an individual. This data was pulled from a variety of different sources, and included data about their tax obligations, their place of residence, their pension entitlements, the property they owned, their criminal history, and whether they had health insurance.
Individuals whose data was processed in SyRI were only informed about the data processing if they were the subject of a risk report, and only upon their request.
At the time the case was brought, the SyRI system was targeted against “problem neighbourhoods.” These were areas with high numbers of residents on welfare. There was no evidence that there was a higher prevalence of welfare fraud in these neighbourhoods.
The case brought before the District Court of the Hague, in the Netherlands, challenged the law underpinning the use of SyRI on the basis that it violated a number of human rights, particularly the right to privacy.
In its decision, the Court held that the SyRI law violated the right to privacy. Although the Court was not against the use of technologies to prevent and combat welfare and other types of fraud, it was critical of the lack of safeguards in the SyRI law for ensuring that privacy was sufficiently protected when it was being used.
The Court found that the law did not strike a fair balance between the social interest in combatting fraud, and the violation of private life to which the legislation gave rise to. In particular, the Court was unimpressed by the lack of transparency and verifiability around the system. It concluded that, under the SyRI law, individuals would not have a reasonable expectation that their private life was sufficiently respected when the system was applied.
It ruled that the law should have no binding effect.
The Systeem Risicoindicatie (otherwise known as SyRI) was an instrument that was used by the Dutch government for the purpose of preventing and combatting fraud in social security, social benefits, taxes, and employment. According to the Dutch government, SyRI was a technical infrastructure that facilitated the linking and analysis of data “anonymously” for the purpose of generating risk reports. These risk reports determined whether a person or company was deemed worthy of investigation for fraud and related crimes.
The instrument was used across a number of government bodies, including municipal authorities, tax authorities, immigration services, and social affairs and employment agencies. The data files which these bodies had available were processed through the system, which linked the data together in order to identify “discrepancies.” The technical infrastructure that facilitated this pre-existed a specific law on the system.
In 2015, legislation was finally passed that provided the legal basis for SyRI and defined the process for its application. It established that a number of designated administrative bodies or entities could jointly make a request to the Minister of Social Affairs and Employment (the Minister) for the purposes of obtaining a risk report produced through the application of SyRI. It was a requirement that the bodies be collaborating before they could make a request. Even though the Public Prosecution Service and the police were not able to make joint requests for the application of SyRI, they could still receive risk reports from the Minister at their request for the performance of their duties.
The legislation stipulated that bodies requesting the application of SyRI had to be collaborating, and requesting a risk report, for the purpose of “preventing or combatting the unlawful use of government funds and government schemes in the area of social security and income-dependent schemes, preventing and combating taxes and social security fraud and non-compliance with labour laws.” When such a request was made, and accepted, the bodies would share data to be processed by SyRI with the Minister.
The kind of data that could be processed by SyRI included:
identifying data (name, address, city, postal address, date of birth, gender and administrative characteristics);
social benefit allowances and subsidy data (data from which the financial support of an individual or company could be established);
work data (data that could establish work performed by a person);
data on permits and exemptions obtained for activities;
tax data (data that disclosed tax obligations);
housing data (data on actual or other place of residence or place of business);
education data (data from which the financial support for the funding of education could be established);
pension data (data that disclosed pension entitlements);
health care insurance data (data that could establish whether a person had health insurance);
debt burden data (data about any debts of a person);
data on movable and immovable property (data establishing possession and use of certain property);
compliance data (data providing insight on the history of compliance with laws);
data on administrative measures and sanctions (data proving that a fine or other such measure had been imposed against a person);
data on grounds for exclusion from social assistance benefit or other benefits; and
civic integration/reintegration data (data with which it could be established if an obligation to participate in a civic integration programme or reintegration obligations had been imposed on a person and whether or not they had been or were being met).
The legislation set out the detailed steps to be taken in the SyRI process. When a request was made by bodies collaborating together, such a request had to meet a number of conditions. For example, the risk indicators and the risk model to be applied had to be identified clearly in the request in order to avoid a “fishing expedition.” At the time of the case, only one risk model was being used by the relevant government authority. It was named the “neighbourhood-orientated approach.”
The request also had to explain how the potential harm to persons whose data was to be processed by the system would not be disproportionate to the purpose of applying the SyRI system. In other words, the request had to show that the aim of preventing, investigating or detecting the alleged fraud outweighed the impact on those whose data was to be processed by SyRI.
Where the Minister was satisfied that the relevant conditions had been met, data records were to be shared by the requesting bodies/entities. Two stages of data processing were then to be carried out by two government bodies, the Benefits Intelligence Agency (BIA) and the Social Affairs and Employment Inspectorate.
First, the BIA collated the data files and pseudonymised them. Personal names and company names, unique personal ID numbers, and addresses were among the data that were to be replaced with a code (a pseudonym). The pseudonymised data was then run through the SyRI risk model for a “potential hit” that would indicate an increased risk of fraud.
The SyRI legislation did not provide information on how this risk model operated, and the risk model itself was a secret. However, during the case, the Dutch government alleged that the risk model consisted of i) links, ii) risk indicators and iii) a cut-off point threshold. Existing and new data files were linked automatically and compared to each other with a view to finding hits that were indicative of an increased risk of abuse or fraud (risk indicators). These indicators were given a score that reflected the probability of the risk indicator occurring. The more improbable it was that the specific risk indicator would occur, the higher the score. The cut-off point was then applied to determine whether there was a “potential hit.” Where a score fell below the cut-off point it would not result in a “potential hit.”
When, based on the risk model, certain persons or addresses were flagged as a “potential hit,” they were to be decrypted by the BIA. All data related to these “potential hits” were then to be forwarded to the Minister for the second phase of risk analysis and the BIA was obliged to destroy relevant files within four weeks.
For the second phase, the decrypted data was analysed more closely by the Social Affairs and Employment Inspectorate to assess their worthiness for investigation. This resulted in a definitive risk selection, on the basis of which the Minister produced a risk report for the requesting bodies and, in some circumstances, the Public Prosecution Service or the police.
Under the law, the risk report was described as “the provision of individualised information from [SyRI] containing a finding of an increased risk of unlawful use of government funds or government schemes in the area of social security and income-dependent schemes, taxes and social security fraud or non-compliance with labour laws by a natural person or legal person, and of which the risk analysis, consisting of coherently presented data from the [SyRI], forms part.”
When a risk report was produced to a body, the Public Prosecution Service or the police, it was to be included in a risk report register. It was then possible that an investigation or similar action could follow submission of the risk report. Individuals whose data was processed in SyRI were only informed about the data processing if they were the subject of a risk report, and only upon their request.
The SyRI law also included a process of feedback, whereby use of the risk reports by the requesting bodies had to be communicated to the Minister. The Minister was then obliged to evaluate the risk model based on this feedback and, if necessary, the Social Affairs and Employment Inspectorate could adjust the risk model. The law also set retention periods for the data processed in the context of SyRI, up to a maximum of two years. It also applied a duty of confidentiality to the data processed.
In 2018, a coalition of organisations and individuals filed a case challenging the SyRI legislation. The parties taking the case included the Dutch Section of the International Commission of Jurists (Nederlands Juristen Comité voor de Mensenrechten), Platform for the Protection of Civil Rights (Stichting Platform Bescherming Burgerrechte), Privacy First (Stichting Privacy First), the Organisation of Psychotherapists and Psychiatrists Against the Digital Diagnostic Register (Stichting Koepel van DBC-vrije Praktijken van Psychotherapeuten en Psychiaters), the National Clients’ Council (Landelijke Cliëntenraad), the Federation of Dutch Trade Unions (Federatie Nederlandse Vakbeweging) and two well-known authors (Maxim Februari and Tommy David Wiering).
These parties argued that the SyRI legislation violated a range of human rights, including the right to privacy, data protection, a fair trial, and an effective remedy. Their arguments centred particularly around violations of the rights to privacy and data protection. The parties alleged that the legislation permitted the processing of huge quantities of data concerning large sections of the population and, therefore, did not involve the processing of minimal amounts of data for a specifically prescribed purpose. The parties also maintained that the legislation did not contain sufficient safeguards to protect privacy.
The UN Special Rapporteur on extreme poverty and human rights filed a third-party intervention in the case. In this intervention, he supported a further argument made by the parties that the use of SyRI had a discriminatory and stigmatising effect. The system was being targeted at neighbourhoods with higher numbers of residents on welfare, despite the lack of evidence suggesting higher rates of benefit fraud in those areas, which gave rise to a serious risk that the system was being used in a discriminatory manner.
Article 8 of the European Convention on Human Rights
1. Everyone has the right to respect for his private and family life, his home and his correspondence.
2. There shall be no interference by a public authority with the exercise of this right except such as is in accordance with the law and is necessary in a democratic society in the interests of national security, public safety or the economic well-being of the country, for the prevention of disorder or crime, for the protection of health or morals, or for the protection of the rights and freedoms of others.
The Court began by recognising the importance of the social security system to Dutch society, and the crucial role that combatting fraud plays in maintaining public support for the system. In this context, the Court agreed that new technological possibilities should be utilised in preventing and combatting fraud. However, it also observed that the development of new technologies meant that protection of privacy and data protection gained in significance. It observed that protecting privacy inspired trust in government as much as combatting fraud did, and that failure to respect privacy would result in citizens being less willing to provide data to governments.
The Court had little difficulty in finding that the right to privacy had been engaged, or interfered with, by the SyRI legislation. The key question for the Court was whether this interference was justified according to the European Convention on Human Rights. If it was, then there would be no violation of the right to privacy. This assessment required consideration of whether the interference was (i) provided by law, (ii) in pursuit of a legitimate aim, and (iii) necessary and proportionate in a democratic society. When looking at these factors, the Court also took into account the general principles of data protection found in the General Data Protection Regulation.
The Court confirmed that the interference pursued a legitimate aim. It reasoned that the law was directed at the legitimate purpose of preventing or combatting fraud. It also found that the SyRI legislation itself pursued a “pressing social need” since it sought to pursue a “sufficiently compelling purpose” to justify an interference with private life. On this point, the Court made reference to the negative financial impact social security and welfare fraud had on the country. It reasoned that “the direct and indirect damage of fraud in this area justifies the conclusion of the legislator that there is a pressing social need to take measures provided for by the SyRI legislation in the interest of the economic wellbeing of the Netherlands.”
The Court did not deal with the question of whether the interference was “provided by law,” instead the crux of the case rested on the question of whether SyRI, and its associated procedures and safeguards, was necessary and proportionate. To determine this, the Court had to look at whether the SyRI legislation struck a sufficient balance between the benefits of the technology in detecting and combatting fraud and the interference with privacy rights. The Court concluded that no such balance had been struck.
Extent & seriousness of SyRI
In reaching this conclusion, the Court first examined the extent and seriousness of SyRI. The two sides in the case were at odds on the nature of the SyRI system. The parties taking the case argued that the system involved Big Data, deep learning and machine learning algorithms. The State disagreed, saying that SyRI simply compared data sets in order to identify discrepancies. According to the State, this comparison took place through the implementation of a “simple decision tree.” The State also said that it was not a tool to predict whether or not an individual could commit an offence.
The Court was, however, unable to confirm the State’s position because the Dutch government had not disclosed any verifiable information detailing the nature of the SyRI system, including the risk model and the risk indicators of which the risk model was composed or may have been composed.
The Court noted that the SyRI legislation provided for large quantities of data to be processed in the application of SyRI. Furthermore, when files were compared with each other with a view to producing “potential hits,” those hits were indicative of an increased risk of fraud. This was recognised as a form of profiling. The legislation also left it open for predictive analyses, deep learning and data mining in the context of SyRI, even if it was to be assumed such systems were not being used at the time of the case.
The Court also took account of the fact that the SyRI legislation did not provide for a duty of disclosure to those whose data was being processed in SyRI so that these individuals could know that their data was being processed in this way. The legislation also did not oblige individuals to be notified if a risk report was produced about them. There was only an obligation to announce the start of a SyRI project to the public and to provide access to the register of risk reports upon request from an individual whose data was processed for a risk report. Apart from this, individuals whose data was processed would only be informed about data processing if there was a an investigation in response to a risk report, which did not happen as a matter of course.
The Court went on to note that there was still a significant impact on the private life of an individual whose data had been processed by the system in circumstances where there was no investigation following the risk report. This significant effect, coupled with the secrecy around the system, were important factors in determining whether the SyRI legislation sufficiently protected the right to privacy under the European Convention on Human Rights.
Necessary and proportionate in a democratic society
After examining the extent and seriousness of SyRI, the Court considered whether SyRI met the requirements of necessity and proportionality under the European Convention on Human Rights. It reasoned that there had to be a fair balance between the purposes of the SyRI legislation and the invasion of private life the legislation caused.
The Court noted that, with the rise of government use of algorithms and new technologies to supervise individuals, data protection and privacy had become particularly important. This was because the collection and analysis of data carried out by these technologies could interfere extensively with the private lives of those to whom the data pertained. In light of this, the Court acknowledged that the State had a “special responsibility” when using a system like SyRI and had to ensure sufficient safeguards were in place against abuse or arbitrariness.
The Court concluded that the safeguards in the SyRI legislation were insufficient for the protection of private life of those whose data could be processed by the system. In particular, the Court noted that the system was insufficiently transparent and verifiable to conclude that it amounted to a necessary and proportionate restriction on the right to privacy. In reaching this conclusion, the Court found the following factors particularly concerning:
Lack of Transparency on Risk Indicators and Risk Model: The Court noted that the legislation did not provide information on the factual data (i.e. risk indicators) that were being used to justify the conclusion that there was an increased risk of fraud. The Court also noted that the legislation did not provide information on the functioning of the risk model, such as the type of algorithms used in the model. It also did not detail the risk analysis that was to be carried out by the Social Affairs and Employment Inspectorate. Furthermore, the SyRI legislation did not afford insight into the validation of the risk model and the verification of the risk indicators. This meant the Court could not inform itself of the objective criteria underlying the validity of the risk indicators and risk models used in the application SyRI.
Inability to Track or Challenge Data: Due to this lack of transparency, the Court recognised the seeming impossibility of an individual being able to defend themselves against a risk report being shared about them under SyRI. Even where a risk report was not produced, an individual whose data had been processed under the system would not be aware of whether their data had been processed on correct grounds due to the lack of transparency. The Court observed that the “right to respect for private life also means that a data subject must reasonably be able to track their personal data.”
Risk of Discriminatory Effects: The Court also noted that the risk model and risk analysis had the potential of having discriminatory effects. The Court observed that SyRI had, at the time of the decision, only been applied to so-called “problem neighbourhoods.” The Court went on to say that, taking into account the large amounts of data that qualified for processing in SyRI and the circumstances in which risk profiles were used, there was a risk that SyRI was inadvertently biased, for example, on the basis of socio-economic or immigration status. Again, due to the lack of transparency, it was not possible for the Court to verify whether such a risk had been neutralised. According to the Court, the fact that there was human review in the process, aimed at identifying and removing false positives and false negatives, was insufficient for neutralising the risk of discriminatory effects.
Insufficient Attention to Purpose Limitation and Data Minimisation: The Court also held that the SyRI legislation paid insufficient attention to the principles of purpose limitation and data minimisation (i.e. data could only be collected that was strictly necessary to achieve a specific purpose). The Court noted that the purposes pursued by the legislation were legitimate, but these purposes had to be considered in conjunction with the fact that the system processed a large quantity of data from a large variety of sources. The Court observed that it was “hard to imagine any type of personal data that [was] not eligible for processing in SyRI.”
Insufficient Independent Oversight: The Court also noted that the assessment of whether the application of SyRI was necessary in a particular case was to be carried out by different stakeholders in the process. There was no comprehensive and independent assessment prior to the Minister approving the application of SyRI as to whether or not the resulting interference with private life would be necessary and proportionate. Furthermore, there was no independent assessment of the risk model and the risk indicators to be used. The Court reasoned that this meant an individual whose data was to be processed by SyRI would not be sufficiently certain that their privacy would be safeguarded when SyRI was used.
In conclusion, although the State was not prevented from using risk profiles or even new technologies to combat this type of fraud, the SyRI legislation contained insufficient safeguards to protect the right to privacy. Therefore, the right to privacy under the European Convention on Human Rights had not been complied with. The Court was also critical of the fact that a Data Protection Impact Assessment had not been carried out for each SyRI project.
The Court ruled that the relevant provisions of the SyRI legislation had no binding effect as they were contrary to the right to respect for private life under Article 8 of the European Convention on Human Rights. In short, the SyRI legislation was overturned.