By Dr. Mohit Nair, public health researcher, educator, and AI governance researcher dedicated to advancing equity in digital health and technology.

When OpenAI first released chatGPT in November 2022, it was no surprise to see how quickly it went viral, as users discovered its capabilities in everything from refining e-mails to generating trip itineraries, all with a simple text prompt.  

Since then, we have seen the advent of image generation models, chatbots, and all things “AI.” Start-ups, government agencies, and enterprises alike have rushed to examine potential use cases with AI to see how it can accelerate their own goals. While AI undoubtedly presents several opportunities, it also poses significant risks, particularly for historically marginalized communities. Without intentional efforts to center racial equity in AI development, we risk amplifying systemic inequities rather than alleviating them. 

How did we get here? 

Before we examine AI’s racial equity implications and societal impact, we must first define what we mean by “AI.” 

Tools like chatGPT are powered by large language models, such as GPT-4o, but they represent the tip of the iceberg when it comes to AI systems. Broadly, artificial intelligence refers to training machines to perform tasks traditionally done by humans, often using machine learning1 techniques, which analyze structured or unstructured data to make predictions or recommendations. 

Source: Shelby Temple1 

The next generation of AI technology went several steps further and leveraged something called deep learning2 with multiple layers of data processing to scale the level of complexity and performance.  

Generative AI is a more specialized subset of deep learning, designed specifically to create new content—hence the term “generative” AI. 

The development of these models has relied on controversial methods, such as web crawling.3 In this process, the works of artists, authors, and journalists, along with publicly available online content, were used to train and refine these models without proper recognition, compensation, or copyright protections. 

This is what makes it possible for an image-generation model like DALL-E to generate art “in the style of Hayao Miyazaki:” 

Source: DALL-E Prompt Book4  

This has already prompted several lawsuits, with companies like Reuters and the New York Times as well as comedians like Sarah Silverman suing OpenAI for copyright infringement.5 As the lawsuits wind their way through the courts and the dust settles, there will be major implications for copyright and IP protections for creatives moving forward. 

The perils of unguarded use of AI 

As the novelty of generative AI fades, we must confront its societal implications, particularly for marginalized groups. AI models often perpetuate and even amplify existing biases, as seen in policing, hiring, finance, and healthcare.  

The COMPAS (Correctional Offender Management Profiling for Alternative Sanctions) machine learning algorithm provides a perfect case study: it’s been in use long before generative AI and assesses the likelihood of a person being rearrested for a crime within two years. Though race is excluded as a factor, correlated variables lead to racial disparities. A ProPublica investigation6 analyzed over 10,000 cases in Broward County, Florida, comparing predicted recidivism rates with actual recidivism rates over a two-year period and found that “Black defendants were nearly twice as likely to be misclassified as higher risk compared to their white counterparts (45 percent vs. 23 percent)…[while] white violent recidivists were 63 percent more likely to have been misclassified as a low risk of violent recidivism, compared with black violent recidivists.”  

The known and unknown biases with predictive AI algorithms in the criminal-legal system combined with the documented practices around increased surveillance in Black and Brown neighborhoods7 should prompt significant concerns around how AI gets leveraged.  

Similar concerns can be found in the use of AI in employment. The labor market already experiences significant distortions based on racial bias in hiring practices: a seminal study by Bertrand and Mullainathan aptly titled “Are Emily and Greg More Employable than Lakisha and Jamal?8” found that white names with identical resumes receive 50 percent more callbacks for interviews compared to African American names. 

AI-driven hiring tools, trained on biased data, reinforce these disparities—Amazon’s AI recruiting tool, for example, was found to downgrade resumes that included women’s colleges or certain gendered language.9 

Studies by the Algorithmic Justice League10 show that facial recognition algorithms have higher error rates for Black and Asian individuals and perform especially poorly with women who have darker skin, due to unrepresentative training datasets. This can often lead to deadly consequences and false accusations directed at Black and Brown communities11, amplifying systemic disparities.  

In finance, AI-driven lending models have reinforced historical discrimination12, denying credit to underserved groups and perpetuating economic disparities. As the study author says: “It’s a self-perpetuating cycle…we give the wrong people loans and a chunk of the population never gets the chance to build up the data needed to give them a loan in the future.”  

In the health sector, a 2019 study13 published in Science uncovered how a widely used healthcare AI system systematically disadvantaged Black patients by misallocating resources. By using healthcare costs as a proxy for medical need, the algorithm assigned lower risk scores to Black patients, limiting their access to critical care. The result? Only 17.7% of Black patients received extra care, when nearly three times as many should have.  

We already know that climate change disproportionately impacts historically redlined communities in the United States and marginalized communities14 globally. AI technologies are heralded as promising a brighter future in preventing climate crises, diagnosing medical illnesses, and predicting disease outbreaks. But what happens when AI models that don’t adequately account for underrepresented populations fail to detect conditions in non-White populations or under-represent the degree of risk to marginalized populations? How do we reconcile the claimed future benefits of predicting climate catastrophes with the real and documented costs of training and running these models15 on our planet?  

Source: DALL-E (“Create an image of a CEO speaking to a corporate boardroom” prompted an immediate interpretation of a “male” CEO and predominantly white board members) 

The list of potential harms is seemingly endless, from a biased interpretation of male-dominated and white CEOs and boardrooms to erasure of Indigenous or underrepresented languages to AI-powered misinformation or disinformation campaigns.  

Can we harness the power of AI to design a more equitable future? 

AI systems do not necessarily have to perpetuate harm. Strengthening legal safeguards, engaging communities early, regulating harmful use cases, and promoting responsible AI ethics can foster more equitable interaction paradigms. 

First and foremost, legislation must keep pace with AI developments and provide a framework for regulating risk in AI systems. The EU AI Act16 provides a model by establishing transparency requirements for technology providers, categorizing AI systems by risk, and banning systems deemed “unacceptable.” 

In the U.S., the AI Bill of Rights17 outlines essential protections—the need for safe and effective systems, algorithmic discrimination protections, data privacy, notice and explanation, and human alternatives—but lacks any enforcement mechanism. It outlines what one should do, but fails to outline any policy governing instances where systems breach these principles.  

At the state level, Illinois’ Biometric Information Privacy Act18 and New York City’s regulations for automated employment decision tools19 present a good framework for expanding protections. The need of the hour is a systematic application at the federal level to govern the use of AI across all jurisdictions.  

Once these basic protections are in place and we reach a consensus on what “unacceptable” or “high” risk AI systems are and regulate their implementation, we can explore responsible corporate AI use and community-engaged or community-led AI governance models.  

What would it look like to engage historically marginalized communities in AI-enabled design thinking? Could it be used in partnership with communities to design social housing? 

What would it look like to explore community-led models of data stewardship to govern how the data of Black and indigenous communities gets used to actively redress bias within data systems? What would it look like to let community groups govern how their data gets used (e.g., restricting its use for policing, facial recognition, or surveillance purposes)? 

Yet another area where AI holds a lot of promise is fostering accessibility: AI systems that are designed to translate languages, read text aloud, convert images to text for disabled communities, or even provide real-time AI-generated live captions can create a more inclusive environment for all of us.  

Moving forward 

At the time of writing, the technology is already evolving at a pace that is difficult to keep up with. We are now in the era of autonomous AI agents20 that can plan and execute tasks without human intervention. 

The legal community has a unique opportunity to shape AI’s trajectory toward a more just and equitable future through stringent regulations focused on transparency, fairness, and equity. 

As a society, we must collectively decide whether we design and leverage AI tools to reduce civil liberties for already-vulnerable groups or leverage them to enhance creativity and agency.  

References 

  1. Temple S. “Stop Confusing AI with Generative AI.” Medium, 8 Oct 2024, https://generativeai.pub/stop-confusing-ai-with-generative-ai-understanding-the-key-differences-21da2b2d3374. 
  1. Holdsworth J, Scapicchio M. “What is deep learning?” IBM. 17 June 2024, https://www.ibm.com/think/topics/deep-learning  
  1. Kurusumuthu J. “What Role Do Web Crawlers Play in Building Large Language Models?” Medium, 25 July 2024, https://medium.com/@kjbtrs/what-role-do-web-crawlers-play-in-building-large-language-models-cb11d4966e37. 
  1. “DALL·E Prompt Book v1.” Pitch, 14 July 2022, https://pitch.com/v/DALL-E-prompt-book-v1-tmd33y/be622e75-5cd3-48f2-9beb-eb31a84f578a. 
  1. Varanasi L, Edmonds L. “A judge compared OpenAI to a video game company in its court battle with The New York Times.” Business Insider, 23 Nov 2024, https://www.businessinsider.com/openai-copyright-lawsuit-new-york-times-video-game-company-2024-11. 
  1. Larson J, Mattu S, Kirchner L, Angwin J. “How We Analyzed the COMPAS Recidivism Algorithm.” ProPublica, 23 May 2016, https://www.propublica.org/article/how-we-analyzed-the-compas-recidivism-algorithm. 
  1. Pettit B, Gutierrez C. “Mass incarceration and racial inequality.” American Journal of Economics and Sociology, 77.3-4 (2018): 1153-1182., https://pmc.ncbi.nlm.nih.gov/articles/PMC9540942/. 
  1. Bertrand M, Mullainathan S. “Are Emily and Greg more employable than Lakisha and Jamal? A field experiment on labor market discrimination.” American Economic Review, 94.4 (2004): 991-1013. 
  1. Dastin J. “Insight: Amazon Scraps Secret AI Recruiting Tool That Showed Bias Against Women.” Reuters, 10 Oct. 2018, https://www.reuters.com/article/world/insight-amazon-scraps-secret-ai-recruiting-tool-that-showed-bias-against-women-idUSKCN1MK0AG/. 
  1. Quach K. “We listened to more than 3 hours of US Congress testimony on facial recognition so you didn’t have to go through it.” The Register, 22 May 2019, https://www.theregister.com/2019/05/22/congress_facial_recognition/. 
  1. Hill K. “Wrongfully Accused by an Algorithm.” The New York Times, 24 June 2020, https://www.nytimes.com/2020/06/24/technology/facial-recognition-arrest.html. 
  1. Heaven WD. “Bias isn’t the only problem with credit scores—and no, AI can’t help.” MIT Technology Review, 17 June 2021, https://www.technologyreview.com/2021/06/17/1026519/racial-bias-noisy-data-credit-scores-mortgage-loans-fairness-machine-learning/. 
  1. Obermeyer Z, Powers B, Vogeli C, Mullainathan S. “Dissecting racial bias in an algorithm used to manage the health of populations.” Science. 25 Oct 2019;366(6464):447-53.  
  1. Barber III WJ. “Why Marginalized Communities Pay the Highest Price for Climate Change.” MIT Press, 25 Oct 2024, https://thereader.mitpress.mit.edu/why-marginalized-communities-pay-the-highest-price-for-climate-change/  
  1. Coleman J. “AI’s Climate Impact Goes Beyond Its Emissions.” Scientific American, 7 Dec 2023, https://www.scientificamerican.com/article/ais-climate-impact-goes-beyond-its-emissions/. 
  1. “Regulation (EU) 2024/1689 of the European Parliament and of the Council.” Eur-Lex, 13 June 2024, https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:32024R1689. 
  1. “Blueprint for an AI Bill of Rights.” The White House Archives, https://bidenwhitehouse.archives.gov/ostp/ai-bill-of-rights/. 
  1. “Biometric Information Privacy Act (BIPA).” American Civil Liberties Union of Illinois, https://www.aclu-il.org/en/campaigns/biometric-information-privacy-act-bipa. 
  1. Francis SRD, Zagger ZV. “New York City Adopts Final Rules on Automated Decision-Making Tools for AI Hiring.” The National Law Review, 8 April 2023, https://natlawreview.com/article/new-york-city-adopts-final-rules-automated-decision-making-tools-ai-hiring#google_vignette. 
  1. “Introducing Operator.” OpenAI, 23 Jan 2025, https://openai.com/index/introducing-operator/