Imagine you are a medieval peasant who has just been handed a map drawn by the king's most learned cartographer. The map is beautiful — intricate, confident, covered in calligraphic flourishes. You are told it will guide you safely through the forest. You are not told that the cartographer has never set foot outside the palace, that the map was drawn from travellers' rumours, and that several of the forest paths depicted no longer exist.
You trust the map because the king paid for it. Because it looks authoritative. Because everyone else is using it. And because questioning it might suggest you are too dim to understand it.
This is, give or take a few centuries and several billion dollars, a precise description of our current relationship with Artificial Intelligence. We are deep in the forest. The map is confidently wrong in ways we cannot easily verify. And the cartographers are, at this very moment, raising their next funding round.
Part IThe Fountainhead: Where AI Money Came From
Artificial Intelligence is often narrated as a story of sudden emergence. This narrative serves the mythology of disruption that Silicon Valley has perfected over decades, but it is, factually, a lie. The intellectual foundations of modern AI stretch back to Alan Turing's 1950 paper, through decades of what became known as 'AI winters' — periods when investment dried up because the technology could not deliver on its promises.
What ended the last AI winter was not a philosophical breakthrough. It was a hardware one. When Hinton, Sutskever, and Krizhevsky demonstrated their AlexNet architecture winning ImageNet in 2012 — reducing error rate from 26% to 15.3% in a single year — the investment community paid attention. But paying attention is different from understanding. This distinction will haunt everything that follows.
Global AI startup investment grew 150× in 14 years. The 2024 figure includes Microsoft's projected $80B+ infrastructure capex alone. This is not R&D spend — it is the capital cost of building a moat.
Part IIThe Architecture of Opacity: How LLMs Became Black Boxes by Design
OpenAI was founded in December 2015 as a non-profit research laboratory. Its stated mission was to 'ensure that artificial general intelligence benefits all of humanity.' Its founding donors committed $1 billion. The choice to establish a non-profit was deliberate: the founders explicitly feared that a for-profit structure would create incentives that could compromise AI safety.
By 2019, OpenAI had restructured into a 'capped profit' model. Microsoft subsequently invested $1 billion, then in January 2023, a further sum reported at approximately $10 billion.
What is not debatable is the consequence: A laboratory that had published GPT-2's technical report with genuine safety concerns about releasing a powerful language model would, four years later, release GPT-4 with a technical report that explicitly declined to disclose training details, model architecture, or dataset composition, citing 'the competitive landscape and the safety implications of large models.'
The language of safety became the syntax of secrecy. And the market applauded.
The Economics of the Black Box
Training a frontier AI model costs money that would cause most countries' research budgets to faint. Estimates for training GPT-4 range from $63 million to $100 million in compute alone. Meta reportedly spent over $30 billion on AI infrastructure in 2023. Microsoft's capital expenditures on AI infrastructure are projected to exceed $80 billion in fiscal year 2025.
When you spend that kind of money on an asset, you do not publish a manual that tells your competitors how to build the same thing. The training data, the training methodology, the fine-tuning techniques, the RLHF approaches — each constitutes intellectual property. Publishing them is not merely generous; from a fiduciary standpoint, it may be negligent.
This is not cynicism. This is capitalism operating exactly as designed. The problem is that AI systems are not typical products. When a pharmaceutical company withholds its proprietary synthesis process, we can still test the drug's effects rigorously and independently. When an AI company withholds its training methodology, we often cannot meaningfully evaluate the system's reliability, its biases, its failure modes, or its safety properties.
The opacity is the product. And the product is being deployed at civilisational scale.
— XAI BabaThree layers of AI infrastructure — chips, cloud, models — each controlled by a tiny number of actors. The dependency is structural: most AI companies run on infrastructure owned by their competitors.
Part IIIThe Prehistory of the Problem: Opacity Was Never Born with AI
The opacity problem in AI is, in a very real sense, a shadow problem. The behaviours, incentives, and cultural assumptions that produced black-box AI systems did not spring fully-formed from Silicon Valley. They are the continuation of patterns that have characterised the relationship between capital, technology, and governance for over a century.
The Credit Score Precedent
Consider the FICO credit score, introduced in 1989. For most of its history, the precise formula used to calculate a credit score was a trade secret. Consumers could be denied mortgages based on a score whose calculation they could not examine and whose errors they could not effectively challenge. A 2007 report by the National Community Reinvestment Coalition documented systematic patterns in credit scoring that disadvantaged Black and Latino borrowers — patterns that could not be investigated or corrected precisely because the methodology was not disclosed.
Does this dynamic sound familiar?
The Financial Complexity Parallel
The 2008 global financial crisis offers perhaps the most instructive parallel. The instruments at the centre of that crisis — CDOs, Credit Default Swaps, tranches of mortgage-backed securities — were technically complex, rated by institutions with significant conflicts of interest, traded in markets with minimal disclosure requirements, and ultimately so opaque that the institutions selling them did not fully understand their own exposure.
The Financial Crisis Inquiry Commission concluded in 2011 that the crisis was 'avoidable' and resulted from 'widespread failures in financial regulation and supervision.' One does not need to be a prophet to observe that the preconditions for an equivalent conclusion, applied to AI, are already in place.
Part IVThe Real-World Consequences: What Opacity Actually Costs
Criminal Justice: The Algorithm Nobody Could Examine
The COMPAS algorithm, used across multiple US states to assess recidivism likelihood and inform sentencing, bail, and parole decisions, was found by ProPublica in 2016 to be nearly twice as likely to falsely flag Black defendants as future criminals compared to White defendants, while White defendants were more likely to be incorrectly flagged as low risk. Northpointe disputed the methodology — a dispute technically unresolved — but the dispute itself illustrated the core problem: because the algorithm's weights and training data were proprietary, independent evaluation was severely constrained.
The Wisconsin Supreme Court, in State v. Loomis (2016), upheld the use of COMPAS in sentencing while acknowledging that the defendant had no right to examine the proprietary methodology — a ruling legal scholars have widely criticised as inconsistent with due process guarantees. The opacity of an algorithm was thus shielded by legal precedent from the very scrutiny that might establish its legitimacy or reveal its flaws.
The opacity-to-harm pipeline operates the same way across domains: undisclosed training → unaudited deployment → harm surfaces → no mechanism for accountability. The feedback loop continues because the people bearing the cost are the people with least power to break it.
Part VWhere We Stand: A Reckoning, Not a Conclusion
The opacity of AI systems is not an accident. It is a product of a specific economic logic — the logic of venture capital seeking winner-takes-most dynamics, of corporate development seeking competitive moats — that has operated consistently and predictably throughout the development of the AI industry.
This economic logic has political consequences. The regulatory frameworks that have emerged are, with some exceptions, inadequate to the governance challenge. They have been shaped, in significant measure, by the industries they purport to regulate, and they consistently prioritise the interests of capital over the interests of those subject to AI systems.
A large language model is trained on human-generated text. Its outputs are, in a very precise sense, a reflection of the patterns, biases, preferences, fears, and aspirations encoded in the collective textual output of humanity — or, more accurately, of the portion of humanity whose texts were included in the training data, which skews heavily toward English-language sources, toward the digitally connected, and toward the socioeconomically privileged.
When we are surprised by a language model's biases, we are, in a very real sense, surprised by ourselves. The sexism that Amazon's hiring algorithm learned was not invented by the algorithm — it was extracted from Amazon's own historical hiring practices. The opacity problem in AI is a mirror problem. We have built systems that reflect our collective shadow with extraordinary fidelity, and then made it structurally difficult to look in the mirror.
The forest is real. The map is a model. And you, gentle reader, deserve to know the difference.
— XAI BabaThe solution is not to stop. The capabilities being developed in AI laboratories around the world are real, and in many applications they are genuinely beneficial. The question is not whether to build, but how: with what transparency, what accountability, what genuine understanding of failure modes, and what structural commitment to the interests of all those who will be affected.
Explainable AI is not a technical subfield. It is a set of demands — technical, regulatory, ethical, and cultural — about what kind of relationship we will have with the systems that are increasingly shaping human life. Those demands will not be met by engineers alone, however talented. They require a cultural shift: from the acceptance of confident maps drawn by palace cartographers, to the insistence on knowing who drew the map, what data they used, and what happens when the path they drew turns out not to exist.