AI, without the magic

Most explanations of AI either sell it or fear it. This one does neither. It is the version I give to executives who need to make a decision about it this quarter.

It does not know things

A large language model has read an enormous amount of text. When you ask it something, it works out the most statistically likely words to answer with. Here is a simple way to think about it: type "the woman is brushing her..." and it will say "teeth", because that is by far the most likely next word given everything it has read.

It is very good at this. It is usually right. But it is never certain, and it does not know the difference between confident and correct. It will produce a wrong answer in exactly the same tone as a right one. Every serious decision about AI in a business follows from that one fact.

(Technical readers will say the mechanism is more subtle than that, and it is. The point for the boardroom is not the mechanism, it is the consequence: the model commits to the most likely answer and sounds equally sure at 68 per cent as at 98.)

What that makes it good and bad at

Good: drafting, summarising, extracting, finding patterns in language and documents, and doing all of it tirelessly.

Bad: being a database, looking things up unless it has been built to, and being trusted to act on something important without a person or a system checking its work.

An "AI agent" is one of these models wrapped in a loop so that it can take actions rather than just answer. That is a real capability. It is also where the risk changes. A model that is occasionally confidently wrong is a nuisance in a chat window and a liability once it is allowed to act.

What aviation did

Aviation is the best example we have of an industry that became extraordinarily safe without waiting for the technology to become perfect. Engines still fail. What changed was everything built around that fact: checklists, black boxes, mandatory investigation of every incident, and a culture that assumes failure and designs for it.

The International Air Transport Association, the global airline industry body, counted seven fatal accidents across roughly forty million flights in 2024. Measured by distance travelled, flying is dramatically safer than driving. Not because engines stopped failing, but because the industry stopped pretending they would not.

AI is at the stage aviation was in before the checklist. The model will make mistakes. That is not a flaw to be fixed; it is a property of how it works. The only useful question about any AI system is: who catches the mistake, and when?

The four questions

Whenever someone shows me an AI system, I ask four things. They work for a chatbot, a coding agent, or a claims-processing pipeline.

1. What can it touch? Which data and which systems, and is that enforced by the platform or by a policy document?
2. What is the evidence for what it produced? Does each output point back to where it came from, or does it just sound right?
3. Who approves anything that matters before it takes effect?
4. What is the record? If I ask tomorrow what it accessed, what rules it followed and who signed off, can you show me a log?

If a vendor cannot answer those four with something you can inspect, the technology is not the risk. The vendor is.

None of this is an argument against using AI. It is an argument for using it the way we use anything powerful: with the controls built in, and with a person still in charge of the things that matter.