Introduction

Several cognitive and social psychology studies suggest that judges are susceptible to various implicit biases which, unlike overt prejudice, they tend to be unaware of. These can influence their decisions in ways that are problematic considering the duty of impartiality and the right to a fair trial. The desire to increase objectivity, accuracy, and consistency in judicial decision-making has prompted the adoption of artificial intelligence (AI) to assist with different decision points throughout proceedings. At the same time, recognizing judges’ susceptibility to biases raises the issue of how to mitigate them, and it is worth questioning whether AI might have a role to play in it. Hence, the goal of this article is twofold: First, it will describe how AI has been adopted for decision-support in judicial systems, and the challenges arising from such use; and second, it will evaluate the possibility of using AI for helping to detect and counteract judges’ implicit biases, recommending which extralegal factors should be considered when doing so. It is a theoretical and bibliographic research, drawing on direct and indirect sources for a comprehensive analysis of the theme. The article will proceed as follows: I will first provide an overview of how implicit biases affect judicial decision-making, consequently giving rise to arguments favoring the adoption of algorithms to promote more accurate and objective decisions. Subsequently, I will explore their current use in judicial settings, focusing on decision-support tools that are adopted to promote risk assessments. The implications of using automated procedures in the judicial process will be evaluated, with the desirability of their adoption being called into question. I will then offer a different possibility of AI use in the judiciary, namely, to help identify and counteract judicial bias, therefore increasing fairness and legal certainty, before drawing some conclusions.

Biases in adjudication

There is ample scientific evidence demonstrating how judges – like jurors and laypeople – are prone to both cognitive and social biases. While the former entails some broadly erroneous form of reasoning, the latter entails reasoning based on stereotypes (Zenker 2021). Biases have the potential to reduce the accuracy of a judgment and, throughout different stages of proceedings, can influence judicial decisions. To give some examples of relevant findings:

• Judges’ sentencing decisions and compensation awards were found not only to be anchored by the initial demand made by the prosecutor, but also by random and unrelated factors to the decision at hand (Bystranowski et al. 2021).
• In a criminal investigation scenario, irrelevant contextual information affected judges’ conviction rate, and confirmation bias led them to prefer incriminating investigations (Rassin 2020). Similarly, the pretrial detention of defendants later influenced judges’ assessments of their guilt in criminal cases (Lidén et al. 2019).
• Judges’ decisions were biased by the gender of the parties in studies involving hypothetical cases about child custody and relocation, employment discrimination, and criminal sentencing (Miller 2019; Rachlinski and Wistrich 2021).
• Data analysis of judges’ bail decisions revealed racial bias against black defendants, even after controlling for variables such as criminal history and past pretrial misconduct (Arnold et al. 2020). In a virtual reality courtroom, minority defendants were treated more harshly by evaluators – including judges – during conviction (Bielen et al. 2021).

The idea that judicial decision-making can be influenced by extralegal factors is problematic considering judges’ duty of impartiality and the right to a fair trial. Article 6 of the European Convention on Human Rights (ECHR) establishes the right to a fair trial by an independent and impartial tribunal. Impartiality requires that judicial decisions are based on the objective circumstances of the case, in accordance with the law, and free from external influences. Moreover, it excludes the existence of a prior disposition of the judge’s mind that could lead them to favor or harm either party. The European Court of Human Rights (ECtHR) has distinguished between an objective aspect of this requirement, linked with the appearance of impartiality, and a subjective one, linked to “the personal conviction and behavior of a particular judge, that is, whether the judge held any personal prejudice or bias in a given case”[1]. The existence of a subjective approach may lead one to believe that there is an effective remedy to fight against judges’ implicit biases, but such remedy is truly limited. The ECtHR has recognized the hardship of establishing a breach of Article 6 on account of subjective partiality, given the difficulty to procure evidence with which to rebut the presumption of impartiality, and has thus justified its common recourse to the objective analysis.[2]

Given the high stakes involved in judicial decision-making, the issue of how to mitigate judicial bias is important.

One way to do so is through the implementation of debiasing techniques, which seek to address biases’ negative effects by improving either the decision-making process or some relevant characteristics of the decision-maker (Zenker 2021). Another possibility relates to the adoption of artificial intelligence in judicial systems as decision-support or decision-making tools. ‘Artificial intelligence’ is used as an umbrella term to describe various human-designed technologies that exhibit intelligent behavior, analyzing their environment, and taking actions – with a certain level of autonomy – to achieve specific goals. The use of AI brings with it the promise of more accuracy, objectivity, and consistency, with governments increasingly adopting the technology “to attain greater accuracy when making predictions, replace biased human decisions with ‘objective’ automated ones, and promote more consistent decision-making” (Green 2022, p. 3). However, and at least for the time being, not only do these systems also have several limitations that can further deepen the problem of bias in adjudication, but there is also a risk-magnifying potential associated with AI that is not present with human decision-making (Dietterich 2019). In the following session, I will examine how AI has been adopted in judicial systems, specifically as decision-aid tools for assessing risk, and the challenges posed by this use.

AI in judicial systems

To assist adjudication, several countries are experimenting with and integrating digital technologies, particularly AI, in their judicial systems. Applications like advanced case-law search engines, online dispute resolution, or document categorization and screening can potentially lower the cost of dispute resolution and help courts address their backlog of cases, many of which are low-volume, low-value, and low-complexity matters (Steponenaite and Valcke 2020). Furthermore, some evidence suggests that algorithms are better at making policy-relevant predictions than public servants (Kleinberg et al. 2018). This makes the prospects of adopting digital technologies in judicial systems, particularly supportive and advisory AI-based tools, quite promising. On the other side, the use of algorithms to make consequential decisions about the application of public policy to individuals in street-level bureaucracies like courts, police departments, and welfare agencies has been highly controversial (Angwin et al. 2016; Heaven 2020; Allhutter et al. 2020).

Considering this scenario, institutions such as the European Union (EU) and the Council of Europe (CoE) are working towards ensuring that the development, implementation, and use of AI is done in an ethical and lawful way, especially in contexts where there is a high impact on individuals’ fundamental rights. While the EU is in the final stages of approving a regulation creating standardized rules for AI (European Commission 2021), the CoE’s Commission for the Efficiency of Justice adopted the first European Ethical Charter on the use of AI in judicial systems (CEPEJ 2018). In it, the Commission, which is responsible for evaluating European judicial systems and defining concrete ways to improve their performance, provides an ethical framework to guide private and public stakeholders throughout the development and implementation of AI in the judiciary. Continuing this work, in April 2023 CEPEJ also launched a Resource Centre on Cyberjustice and AI, which aims to serve as a publicly accessible focal point for reliable information on AI systems applied in the transformation of judicial systems (CEPEJ 2023). One of its first endeavors was to obtain an overview of these systems, providing a starting point for further examination of their risks and benefits for professionals and end-users. A total of 58 systems were identified in CoE member states, and then classified according to their main field of application. Categories include, e.g., ‘anonymization tools’, which are used for removing identifying information of court users, and ‘natural language processing tools’, used for speech recognition and the automatic transcription of court procedures. My analysis here will focus, however, on the category of ‘decision-support and decision-making’, which encompasses tools meant to facilitate or fully automate decision-making processes in justice systems, considering that some think that highly accurate AI systems could improve the performance of judges, or even come to replace them (Chatziathanasiou 2022).

First, it is worth highlighting that the use of the tools mapped by CEPEJ by judges themselves is still quite limited, and the initiative for their development remains primarily within the private sector, focusing on insurance companies, lawyers, and legal services wishing to reduce the uncertainty and unpredictability of judicial decisions. The French application Predictice, for example, is a predictive justice tool developed to calculate the chances of success of a legal action according to different variables, using jurisprudence analysis algorithms. More recently, it has launched a generative AI tool called Assistant, developed to answer legal professionals’ questions by citing reliable and up-to-date sources (Larret-Chahine 2023). Even though most applications of this kind have their use currently restricted to private agents, “public decision-makers are beginning to be increasingly solicited by a private sector wishing to see these tools […] integrated into public policies” (CEPEJ 2018, p. 14).

Second, not all the applications listed at the Resource Centre can technically be categorized as AI, as is the case for many of the risk assessment tools, mainly used for assessing the risk of recidivism. These make up for the majority of ‘decision-support and decision-making’ tools that have been listed by CEPEJ as being currently used in the public sector, namely by judges. Examples include OASys, the Offender Assessment System used by the prison and probation services in England and Wales (Justice Data Lab 2016), RITA, the Finish Risk and Needs Assessment Form (Salo et al. 2019), or RISC, the Recidivism Assessment Scale adopted in the Netherlands (van Essen et al. n.d.). Risk assessments are perceived and often marketed as an objective means of overcoming human bias in decision-making and have been adopted to assist with several decision points throughout the criminal justice system, from pretrial release to post-conviction sentencing, probation, and parole. These tools do not use new statistical methods commonly associated with AI, such as machine learning (ML), but are rather overwhelmingly based on regression models (Barabas et al. 2018). The main goal of regression is to identify a set of variables (e.g., prior arrest) that are predictive of a given outcome variable (e.g., risk of reoffending). This process can be automatized and improved using ML methods (Ghasemi et al. 2021), but still constitute an incremental innovation in the way risk assessments have historically worked, instead of being truly transformational.

One example of a risk assessment tool that incorporates a machine learning approach is the Correctional Offender Management Profiling for Alternative Sanctions, or COMPAS, used by some United States’ courts to assess the likelihood of recidivism (van Dijck 2022). Ever since an exposé by the news outlet ProPublica revealed that the software was biased against blacks (Angwin et al. 2016), it has become the primary example of the risks posed by algorithmic crime prediction overall. The controversy surrounding its use revolves around what the models measure and intend to measure, the accuracy of the predictions, and whether they might increase inequality and discrimination or otherwise compromise fairness (Mayson 2019; Rudin et al. 2020). The question of (un)fairness is of particular concern, given that risk assessment tools can lead to discriminatory outcomes based on race or ethnicity (Jordan and Bowman 2022). Furthermore, since the software is proprietary, the data and algorithms are not transparent, neither for the suspect nor for the judge (van Dijck 2022), a problem that was addressed in the case of Loomis v. Wisconsin[3] (2016). In this case, while admitting the system’s flaws, the Court claimed that it is up to judges to exercise discretion when assessing a risk score.

In high-stakes decisions such as criminal justice risk assessments, it is common to place emphasis on the decision-makers’ discretion in incorporating algorithmic advice into their decisions, to make their use acceptable even in light of flaws. However, human discretion does not necessarily improve outcomes. Decision-makers are susceptible to automation bias, a tendency to defer to automated systems, reducing the amount of independent scrutiny exhibited when deciding (Parasuraman and Manzey 2010). Similar issues arise when humans collaborate with predictive algorithms. Recent research has found that people are bad at judging the quality of algorithmic outputs and determining whether and how to override those outputs (Green 2022). In simulated pretrial and sentencing decisions, for instance, risk assessments made participants – including judges – place a greater emphasis on its results than on other relevant factors (Green and Chen 2021). It is thus likely that, instead of improving the issue of bias by promoting an ‘objective’ score, the incorporation of results into a decision may nonetheless result in biased outcomes.

The challenges discussed in this section do not necessarily entail a categorical rejection of employing AI for assisting in judicial decisions, but it does raise the question of which uses might be advantageous without presenting a risk to the fairness of a trial. Here, one possibility relates to its use for triaging, allocation, and workflow automation, facilitating some activities during the lifecycle of proceedings and minimizing the need for human input. For instance, one Higher Regional Court in Germany began using AI to help process the large volume of lawsuits relating to a scandal involving vehicle manufacturers. With more than 13,000 cases pending and around 600 entries added monthly, the AI is used to analyze the files and organize them according to the facts, but the judges remain responsible for processing the content, reviewing it, and making decisions (SWR 2022).

AI as a debiasing tool

After examining how AI has been used for decision-support in judicial settings, I will now explore the possibility of applying AI methods for helping detect and counteract judges’ implicit biases. One way of doing so is through predictive judicial analytics in the form of machine learning. As Chen (2019a) explains, biases are most likely to manifest in situations where judges are closer to indifference between options. These contexts of ‘judicial indifference’ are also ones where the highest levels of disparities in inter-judge accuracy are present – essentially conditions where judges are unmoved by legally relevant circumstances. A judge could be said to have strong preferences over legally relevant circumstances (which are covariates) when it is costly to depart from the legally optimal outcome, defined as the outcome that would be generated through consideration of legal facts alone. But a judge may also have weak preferences over legally relevant circumstances, meaning that there is a relatively low cost in departing from the legally optimal outcome. In such cases of legal indifference, a different set of covariates that are legally irrelevant (and thus should not predict a legal outcome) can be expected to have greater influence. In other words:

“If a judge can be predicted prior to observing the case facts, one might worry about the use of snap or pre-determined judgements, or judicial indifference. To put it differently, the preferences of judges over the legally relevant covariates may affect the influence of irrelevant features. A judge could be said to have weak preferences, meaning that there was a relatively low cost in departing from the legally optimal outcome. In such cases of legal indifference, irrelevant factors can be expected to have greater influence. Behavioral bias reveals when decision-makers are indifferent” (Chen 2019b, p. 16).

The accuracy of predictions depends on the number of judicial attributes or characteristics of the case that are analyzed by the system. In a study conducted by Dunn et al. (2017) on asylum courts, using only data available at the case opening (i. e., information on the judge’s identity and the nationality of the asylum seeker), researchers were able to predict case outcomes with an accuracy of 78 %. Through this notion of ‘early predictability’, ML could be used to automatically detect judicial indifference, alerting to situations where extralegal factors are more likely to influence a decision. This raises the question of which other sets of data could be incorporated into the legally irrelevant covariates to improve predictive accuracy, requiring an analysis of which are the main extralegal factors that influence judges when deciding. Based on an examination of the literature on social biases, initial contenders include race, gender, and ethnicity of the defendant and of the judge, the latter for the assessment of ingroup favoritism. And based on findings on cognitive biases, the number of (un)favorable previous decisions by the court, the comparison of caseloads between judges, and whether it is a specialized court (all being indicative of contrast effects) are variables likely to influence the way judges decide. Other factors besides biases include the time of day at which a decision is made (Shroff and Vamvourellis 2022; Danziger et al. 2011), and temperature (Chen and Loecher 2019; Heyes and Saberian 2019). These are of course only initial suggestions and do not fully encompass all the irrelevant covariates that can affect a decision. But the advantage of using machine learning techniques for this purpose is precisely that any sort of data can be used to feed the model, enabling patterns and trends to emerge without necessarily requiring a theoretical explanation for such.

In the in-depth study on the use of AI in judicial systems that accompanies CEPEJ’s ethical charter (2018), the commission analyzes the benefits and risks of different applications, encouraging their use to various degrees. While specific judge profiling is highly discouraged, among uses to be considered following additional studies is offering judges an assessment of their activities with an informative aim of assisting in decision-making. Indeed, offering judges’ feedback regarding their decisions is a fundamental step in debiasing, alongside other interventions such as the promotion of general bias awareness, training in rules and representations, exposure to stereotype-incongruent models, and the adoption of scripts and checklists (Wistrich and Rachlinski 2017) – all of which can be directed once bias is identified. Furthermore, AI offers a mechanism of detecting bias in real time, and could hence alert judges to situations where biases are likely to occur (e.g., after a string of positive decisions), allowing them to intervene before a biased decision takes place.

Concluding remarks

The adoption of digital technologies like artificial intelligence in judicial settings often comes from a desire to increase objectivity, accuracy, and consistency in decision-making, improving the quality of decisions traditionally made by humans. However, we have seen how the use of AI for decision-support in adjudication, albeit still not prevalent in CoE member states, can worsen issues already identified in the risk assessment tools that they seek to automate. These include the accuracy (or lack thereof) of their predictions, the reproduction of existing patterns of prejudice and bias, the lack of transparency and opportunities for defendants to challenge their outcomes, and the difficulty of decision-makers to properly evaluate assessments. Thus, instead of using AI to make decisions that traditionally pertain to judges (e.g., assessing the risk of recidivism), a different possibility was offered, namely, to employ the technology for debiasing purposes. AI can help identify the situations where judicial bias is likely to take place, based on the analysis of covariates that, despite being legally irrelevant, have been shown to influence judicial decisions, some of which were listed here. By identifying the instances in which bias commonly arises, it is possible not only to alert judges but also to target debiasing interventions, such as educating judges on the subject and offering feedback on their work, with the ultimate goal of ensuring objectivity and impartiality in their decisions.