Consider all that you as a person, or we as a culture, chose not to acknowledge — not to look at directly. The inconvenient truths, the inherited assumptions, the prejudices so ambient they stopped feeling wrong and became common sense. The "It is what it is." Carl Gustav Jung called this our Shadow. What we pay attention to is in the foreground and the Shadow is in the underground. The Shadow does not disappear when you ignore it. It goes underground. And from the underground, it governs.
Our AI models have a Shadow. You did not put it there. But it is there. And on the day it surfaces in your output, you will be the one holding the result.
This is not a hypothetical. In 2019, it was a patient.
The Study That Should Have Changed Everything
In October 2019, Dr. Ziad Obermeyer, a Professor at UC Berkeley's School of Public Health, published a study in Science that examined a commercial algorithm called Impact Pro, built by Optum, the health services arm of UnitedHealth Group. Hospitals and insurers across the United States were using Impact Pro to identify which patients should be enrolled in high-risk care management programs: the chronically ill, the medically complex, the people who needed more organised, specialised attention to stay out of emergency rooms.
The algorithm used healthcare costs to predict and rank which patients would benefit most from additional care. This seemed reasonable. Cost is a tidy proxy for need. If someone is spending a lot on healthcare, they probably need a lot of healthcare. Clean logic. Efficient. Implementable at scale.
Simply because you left the race variable out of the model does not guarantee by any means that your algorithm will not be racist.
— Dr. Ziad Obermeyer, UC Berkeley · NBC NewsThere Was One Problem
The data showed that healthcare provided to Black people cost an average of $1,800 less per year than the care given to a white person with the same conditions. Not because Black patients were healthier. Because they had less access. Because systemic barriers — cost, transportation, working hours, earned distrust of a system that had not always served them well — meant they consumed less care than their medical needs actually required.
The algorithm looked at this history and concluded: lower cost, therefore lower need, therefore lower risk score. Black patients assigned the same level of risk by the algorithm were considerably sicker than their white counterparts, as evidenced by signs of uncontrolled illnesses. The model had learned, with perfect statistical fidelity, to systematically underestimate the needs of an entire population. The racial bias reduced the number of Black patients identified for extra care by more than half.
The algorithm did not contain the word "race." It was, in the language its creators used, race-blind. It was also, in practice, anything but.
How cost became a proxy for race — four steps from reasonable to harmful
The Shadow's Favourite Hiding Place
This is what Jung meant when he said the Shadow operates through projection. The model was not biased the way a prejudiced person is biased — consciously, defensibly, with a story attached. It was biased the way a mirror is biased: it showed exactly what was put in front of it, including the parts of the room that history had tried to keep out of frame.
The algorithm could see what patients spent. It could not see why they spent less. It could not see the two jobs, the bus route that didn't reach the clinic, the appointment that was missed because missing work meant missing rent. The absence of a protected characteristic in the feature set is not the same as the absence of its influence. It finds another door. It always finds another door.
This is the Shadow's favourite hiding place: the proxy variable. Zip code standing in for race. Healthcare spending standing in for health. Job title history standing in for gender. The model never sees the protected characteristic directly. It doesn't need to.
It doesn't account for the cost of fresh produce. It may not account for the fact that someone does not have access to transportation but is working two jobs.
— Dr. Fay Cobb Payton, Rutgers UniversityThe Fix, and What It Required
The story does not end in the darkness, which is why Baba tells it. Obermeyer's team partnered with Optum to improve the algorithm. The company independently replicated the analysis on a national dataset of almost 3.7 million insured people. Retraining the algorithm to rely on both past healthcare costs and other metrics — including preexisting conditions — reduced the disparity by 84 percent. The percentage of Black patients served by the algorithm would increase from 17.5 percent to 46.5 percent.
The fix was not technically complex. It was not expensive. It required, above all, someone asking why — and then being given enough access to the algorithm to find out. That is explainability. Not as an academic concept. As the mechanism by which a physician who noticed that her sickest patients were not being reached could trace the path back to a single design decision, made years earlier, by people who had never imagined this particular consequence.
Three Questions You Can Ask Today
Baba is not asking you to become a machine learning engineer. The distance between where you are and where you need to be is shorter than you think, and it begins with three questions:
If you use AI tools that make decisions about people — ask the vendor for the model card. Ask what the model was trained on, what populations it was validated for, and what known limitations have been identified. If they cannot answer, that is itself an answer.
If you are a developer building or deploying models — run disaggregated evaluations before you ship. Do not look only at overall accuracy. Slice performance by every variable that could carry demographic weight. Where the numbers diverge across groups, the Shadow is present.
If you are a creator using AI to generate content or make recommendations — read the output with one question active: whose experience taught this model what normal looks like? The model's defaults were learned from somewhere. Knowing this does not mean distrusting the tool. It means editing with your eyes open.
The Shadow in the training data may not be your fault. But it is, from the moment you deploy, your responsibility. Power without examination is not capability. It is, in the precise Jungian sense, inflation — expanded beyond its capacity for accountability, waiting for the moment the Shadow reasserts itself.
The Shadow always reasserts itself. The question is whether you meet it in the audit, or in the outcome that cannot be undone.