The Alignment Problem: Machine Learning and Human Values

This chapter discusses the historical and evolving use of numerical models to replace human judgment in criminal justice, focusing on the parole system. The chapter opens with the account of Hinton Clabaugh, the chairperson of the Parole Board of Illinois in 1927, and Ernest Burgess, a sociologist from the University of Chicago. Clabaugh commissioned a study by prominent universities to assess the parole system, which had become unpopular and was perceived as overly lenient. Ernest Burgess played an essential role in this effort, collecting data on parolees to identify factors predicting parole success or failure, and categorizing them into eight restrictive social categories. Burgess developed one of the earliest predictive models in criminal justice, aiming to scientifically improve parole decisions. He proposed using “summary sheets” for parolees to help the Parole Board quickly assess risks based on various factors, introducing a more objective approach than the subjective judgments previously used. His work suggested that human behavior had predictable elements that could enhance parole decision-making.

The adoption of Burgess’s scientific approach was initially slow, but it gained traction over the decades. By 1951, Illinois published a “Manual of Parole Prediction” (54), announcing the progress made in the 20 years of using statistical models for parole decisions. These tools were intended to make parole decisions more consistent and fairer, reducing reliance on potentially biased human judgment.

However, states were slow to adopt them. By 1970, only two states in the US used these tools, but interest grew after Tim Brennan, a Scottish-born statistician working for Unilever, moved from applying statistical models to market products to working in a university research environment to address first educational concerns and later to improve consistency in prison inmate classification. Together with Dave Wells, another researcher in the field, Brennan founded the company Northpointe, aiming to reform the justice system using statistical models. By 2000, half of the states used statistical models for parole decisions, with tools like COMPAS, developed by Brennan and Wells, becoming standard by 2011. However, concerns about the ethical implications and fairness of such predictive tools began to surface, leading to critical scrutiny and debates about the reliance on algorithms in the justice system.

One journalist who covered such ethical implications of statistical risk assessment systems was Julia Angwin, working for the investigative journalism platform, ProPublica. Shocked by the lack of substantiation behind data-driven tools like COMPAS used nationwide for criminal risk assessment, Angwin investigated the systems’ reliability and biases, discovering racial disparities in risk assessments, namely a bias against Black defendants. While Northpointe defended its system, pointing to the method of calibration, which implies that the rating system is maintained throughout the data regardless of the defendants’ race, the disparity in prediction was still sustained.

To address ProPublica’s diagnosis of the use of the COMPAS system in the justice system, Cynthia Dwork, a Harvard computer scientist known for pioneering “differential privacy,” a system of collecting data while protecting individuals’ privacy, started researching how data fairness could address systemic issues like racism and sexism. Working together with computer scientist Amos Fiat, Dwork explored the idea of “fairness” and how it could be mathematically defined and implemented in technology. Dwork’s engagement with fairness in computer science led to a broader exploration of the ways data is used and its societal implications. Together with other researchers in the field, such as Moritz Hardt and Helen Nissenbaum, Dwork continued the work on fairness.

Also in response to ProPublica’s findings about COMPAS’s racial biases, Cornell University computer scientist Jon Kleinberg and economist Sendhil Mullainathan examined pretrial detention decisions using machine learning, comparing them with human judges. They explored algorithmic concerns related to racial biases. Around the same time, Alexandra Chouldechova developed a visual dashboard to analyze risk-assessment tools, pondering the various notions of fairness.

This research spurred a significant academic response with multiple papers analyzing the compatibility of different fairness definitions. Kleinberg and his team demonstrated that satisfying both ProPublica’s fairness criteria and COMPAS’s calibration simultaneously was mathematically impossible unless both groups committed repeat offenses at the same rates. Chouldechova reached similar conclusions about the impossibility of achieving equal false positive and negative rates across groups with different recidivism frequencies.

These findings suggest an inherent challenge in using machine learning for judicial predictions. The research highlights the complexities and potential biases embedded in risk-assessment tools. Researchers in the field, such as Moritz Hardt, emphasize the need for continuous scrutiny and adjustment of these algorithms to address and mitigate discrimination effectively from outside of the field of computer science. The broader academic and policy discussions triggered by these analyses continue to explore how best to balance technical fairness with broader societal justice.

Chapter 3 discusses the application of neural network models in the field of medicine and its implications. In the 1990s, Rich Caruana (at the time a graduate student at Carnegie Mellon) embarked on a critical project to predict patient survival rates for pneumonia using neural networks. This project, supervised by Tom Mitchell, aimed at improving hospital decisions on whether to treat pneumonia patients as inpatients or outpatients. The research involved a large interdisciplinary team and utilized a dataset of 15,000 patients, employing various machine-learning models. Caruana’s neural network emerged as the most accurate, surpassing other models and traditional statistical methods.

However, despite the neural network’s superior performance, Caruana and his team decided against its deployment due to the discovery of misleading correlations within simpler, rule-based models that were easier to interpret. For instance, one rule-based model suggested treating asthma patients as low-risk outpatients because historical data showed they had better survival rates. However, it was revealed that this was due to asthma patients receiving more attentive in-hospital treatments, including ICU care.

This discovery led to a broader discussion about the transparency and safety of using neural networks in healthcare. Caruana feared that the neural network, although powerful, might also learn similarly misleading correlations that were not as easily detectable as in rule-based systems. The potential for neural networks to accidentally learn and act on incorrect or harmful correlations without clear insight into their decision-making process posed significant risks. Thus, the decision to use a simpler and more interpretable model was a precaution against unintended and potentially dangerous outcomes based on opaque decision-making processes.

Caruana’s experience underlined the critical issue in AI and machine learning: the trade-off between accuracy and interpretability. It highlighted the importance of understanding the basis of a model’s decisions, particularly in fields like healthcare where decisions have significant consequences.

Other researchers in the machine learning community have grown increasingly concerned with the opacity of neural networks, often described as “black boxes” and their pervasive use in critical decision-making sectors, including defense, healthcare, and finance. The issue of machine learning models’ opacity became notably pressing as these models began to play a more significant role in strategic military and intelligence operations, leading to the development of models that users could more easily understand and trust.

Simultaneously, the European Union was advancing the General Data Protection Regulation (GDPR), which mandated that decisions made by algorithms, including loan approvals or parole denials, must be explainable to those affected. This regulation sparked significant concern among tech leaders about the feasibility of providing clear explanations for decisions derived from complex neural networks.

Discussing the transparency and necessity of complex AI models, Christian refers to the work of Robyn Dawes in the mid-20th century. Dawes, initially an ethics student, shifted to psychology due to his skepticism about the efficacy of intuitive methods like the Rorschach tests. His career pivot was influenced by a clinical error misdiagnosing a patient with a genetic condition as having a psychiatric delusion. This incident propelled him toward “mathematical psychology,” which means that he checked expert clinical judgment against simple mathematical models. The efficacy of straightforward statistical models often surpassed more complex and intuitive clinical assessments, suggesting that often, simpler models or even actuarial methods might be preferable for decision-making.

Further on, Dawes and his colleagues investigated why simple linear models were so effective in decision-making. Studies showed that even models mimicking a single expert’s judgment often outperformed the experts themselves. The researchers’ work pointed to the fact that the key to effective modeling lies in identifying the right variables to consider and then simply adding them up. This research implies that the right information for the models is not in individual decision-making but in the established practices of the field.

Carrying Dawes’s work further into present times, Cynthia Rudin, a computer scientist at Duke University, championed the use of simple, interpretable models over complex ones, particularly in the criminal justice system. In 2018, she demonstrated that a straightforward model could effectively predict recidivism, rivaling the more complex COMPAS system. Her approach advocates for models that are grounded in empirical data rather than clinical intuition. Rudin’s work extends to healthcare, where she criticizes existing models for being overly influenced by subjective clinical judgments rather than robust data. She emphasizes the importance of creating tools that enhance clarity and decision-making in practical settings, suggesting that even with vast amounts of data, simple models can often outperform their more complex counterparts.

As Christian explains, humans have visible sclera (the whites of the eyes), which evolutionary biologists believe are proof of the evolutionary need for cooperation through visible attention direction. In machine learning, the concept of “saliency” reflects this trait by identifying which parts of an image a system focuses on, enhancing the researchers’ understanding of its decision-making process. This method delivers surprising insights, such as systems focusing on unexpected areas of images, which challenge assumptions about their learning processes. Machine learning’s saliency methods reveal unintuitive focus areas, often emphasizing irrelevant training data aspects. In medical applications, training the machine through saliency methods has enhanced diagnostic accuracy and transparency, allowing networks to assist in complex disease detection with high precision. For example, a network retrained on a vast dataset of skin conditions outperformed dermatologists, proving its utility in enhancing global diagnostic capabilities and acting as a reliable second opinion. Such systems, however, require careful implementation to avoid misclassifications due to unexpected data cues.

However, saliency methods have not revealed anything about the “black box” (what happens inside a neural network model). Focusing on this problem, Matthew Zeiler and Rob Fergus at NYU developed deconvolution to visualize intermediate network layers, revealing how models process visual inputs. Their insights demonstrate both the potential and limitations of these models, leading to further innovations in machine learning visualization. These techniques, like Google’s DeepDream, allow for both artistic exploration and critical insights into how models perceive and categorize inputs, highlighting the importance of transparency and the ongoing challenge of understanding AI’s decision-making processes. Researchers such as Been Kim, a graduate student at MIT, advocate for incorporating human-computer interaction and cognitive science to enhance AI’s usability and trustworthiness. Such research work emphasizes the necessity of developing AI that communicates in human terms, with the final goal of making AI systems more interpretable and accountable.