The Shadow Has a Mandate Now

There is a pattern in human systems — in organisations, in institutions, in any structure built by many hands over many years — where the things that are never examined do not simply wait patiently to be discovered. They act. They shape outputs. They determine who is favoured and who is not, which signals are amplified and which are buried, which futures become available and which quietly foreclose. They do all of this without announcement, without consent, and without any mechanism by which the damage can be traced back to its origin.

We have built this pattern into our machines at scale.

The algorithmic systems making consequential decisions across credit, hiring, healthcare, criminal justice, and social welfare are not neutral instruments. They carry the accumulated assumptions of every dataset that shaped them, every objective function that drove them, every human choice that was made and then forgotten when the model was handed off to production. The assumption does not disappear when you stop thinking about it. It runs. Quietly, at scale, at speeds no auditor can match.

This is what Baba means by the collective shadow in our machines. Not as metaphor. As architecture.

What you do not examine does not lie dormant. It makes decisions on your behalf — and sends you the invoice later.

You Cannot Audit What You Cannot See

Every practitioner who has spent time with a large model in a production environment knows the particular unease of watching it perform well on metrics and yet suspecting — not knowing, suspecting — that something underneath is wrong. The performance is real. The suspicion is also real. The two coexist because the thing being measured and the thing doing the deciding are not the same thing.

You are measuring the output. The output is not the reasoning. The reasoning — distributed across billions of parameters — remains opaque. And the gap between what is measured and what is actually happening is where the shadow lives.

Mandating explainability is, in the deepest sense, a demand for self-knowledge at institutional scale. Not the performance of self-knowledge. Not the checkbox that says we considered bias in our model card. The actual, costly, technically demanding work of surfacing the logic of a decision in a form that a human being can examine, contest, and act upon.

This is not comfortable work. The model you believed was selecting for merit turns out to be selecting for the proxy that correlates with merit in your historical data — which reflects the decisions of people who were not always selecting for merit. The feature driving your healthcare triage score is spend history, which correlates with insurance quality, which correlates with income, which correlates with race. The thing you said you would not measure is being measured. You simply cannot see it yet because you have not built the tools to look.

The law, in requiring explanation, requires the look. This is the law doing what law occasionally, imperfectly, and with enormous delay does: forcing the confrontation with a thing that individuals and organisations would prefer to leave unexamined.

Trust Is Not Sentiment. It Is the Precondition of Function.

A system that cannot explain its decisions to the people it acts upon is not a neutral system. It is a system that has made a specific choice: to treat the subject of the decision as a recipient rather than as a participant. This choice has consequences that compound.

When a benefits determination is made by an algorithm that cannot explain itself, the person denied has no mechanism for appeal that engages the actual logic of the decision. They can appeal to a human who also cannot explain the algorithm. They can submit additional documentation that may or may not be legible to the model. Or they can accept an outcome they do not understand and whose fairness they have no tools to assess. In communities with long histories of encountering institutions that do not explain themselves, this experience is not unfamiliar. It does not feel like a technical limitation. It feels like a continuation.

Trust, once lost in this way, does not return with a press release. It returns — slowly, conditionally, with significant evidence — when institutions demonstrate that they can be examined. That their reasoning can be read. That a person can see why a decision was made and, where the reasoning is wrong, can say so and be heard.

This is what explainability law, at its best, creates: not just the technical capacity for explanation, but the social infrastructure of legitimate contestability. The right to understand is also the right to disagree with reasons. And the right to disagree with reasons is the foundation of every accountability mechanism that has ever functioned in human governance.

The Unexamined Assumption Does Not Sit Still. It Runs Up a Tab.

Baba wants to address the builders directly here, because this is where the resistance to explainability mandates is most concentrated and most misplaced.

The argument against mandatory explainability is almost always framed as a cost argument: the engineering overhead, the performance trade-offs of interpretable models, the friction added to deployment pipelines. These costs are real. They are also, in almost every case, dwarfed by the costs they displace.

The unexamined assumption in your model is not free. It accrues liability. It produces outcomes that fail the populations you built the system to serve, which erodes the business case for the system. It attracts enforcement attention precisely because it cannot be audited — and enforcement, when it arrives, is expensive in ways that no pre-deployment explainability investment would have approached.

Build the explanation in from day one. Not for the regulator. For the model. The explanation is where the errors live — and errors left unexamined do not resolve themselves.

There is a more fundamental economic argument, though, and it is one that purely compliance-oriented framing misses entirely. Explainability is a forcing function for model quality. When you must explain a prediction, you must understand it. When you must understand it — genuinely, in the specific sense of being able to surface the features that drove it and the confidence with which it was made — you discover the proxy variables. You find the spurious correlations that drive performance on the training distribution and collapse on the deployment population.

For the consumers and enterprise clients: the regulated industries — healthcare, financial services, insurance, public procurement — cannot deploy what they cannot audit. The explainability mandate is not a barrier to adoption in these markets. It is the precondition of adoption. The vendor who cannot provide a compliant explanation artefact is not competing in the enterprise healthcare market. Full stop.

To Know What You Have Built Is the Beginning of Authority Over It

There is a certain kind of builder who genuinely believes that resistance to explainability is principled — that opacity protects something real. The competitive moat. The proprietary logic. The efficiency of the black box. Baba does not dismiss this. In narrow circumstances it is even partially true.

But what is more often true: the resistance to explanation is the resistance to knowing. Not a strategic choice about information asymmetry, but a prior, unexamined preference for not encountering what a thorough look at the model would reveal. This is not wickedness. It is the ordinary human capacity for avoiding the uncomfortable examination. Builders are not immune to it. Organisations actively cultivate it.

The explainability mandate does not eliminate this tendency. It makes avoidance costly. And cost, applied consistently over time, changes behaviour — which changes culture — which eventually changes what gets built in the first place. This is the slow, recursive logic by which external pressure becomes internal practice.

The builders who build explainability in from the beginning are not complying with a regulation. They are choosing to know what they have made. This is not a moral superiority. It is a different relationship with their own work — one in which the output is not separate from the understanding, and the understanding is not separate from the accountability.

The organisations that reach this point tend to build better. Not because the law made them better, but because looking at what you have made, consistently and with genuine tools, is the practice that generates quality. The mandate created the pressure. The practice generates the advantage.

Baba is not here to celebrate the specific text of any particular regulation. Most of it is imperfectly drafted, inconsistently enforced, and perpetually behind the technology it is trying to govern. The precise requirements will be amended before they are fully implemented. This is the normal condition of law engaging with technical change, and it changes nothing fundamental.

The fundamental thing is this: the shadow now has a mandate. The unexamined assumption in your model is no longer merely a product quality problem or an ethical exposure. It is a liability with a legal face. And that is, on balance — imperfect, contested, overdue, and incomplete — a civilisational improvement.

Not because the law is wise. Because the thing the law is demanding is wise, and we were not going to arrive there on our own.

What governs in shadow must eventually answer to light. The ritual of explainability is not compliance. It is the oldest act of authority: knowing what you have made.