“That’s what the AI said.” I have been hearing this sentence more often for some time now, usually when someone asks what a decision was based on. It is said calmly, almost casually, and nobody objects. The answer was well written and struck the right tone.
The problem with that sentence is not that the answer might have been wrong. Perhaps it was right.
The problem is that nobody in the room knew the difference.
Fluent language and subject knowledge
Throughout our lives, we have learned to recognise expertise through language. Someone who explains a subject clearly, without hesitation and in an appropriate register has usually worked to understand it. That inference was useful for centuries because one was rarely possible without the other.
A language model separates precisely those two things. It calculates which piece of text is most likely to come next, based on everything it has read. To put it more provocatively: it does not know what it is talking about. But it knows very well how someone who does know would talk.
That is why the advice to “stay critical” falls short. We are not fooled by a lie, but by a convincing form. Scepticism is poorly equipped to deal with form. It responds to errors, not polished sentences.
And what does that do to us?
Much is written about the machine’s errors, known in the industry as “hallucinations”. The second effect is more dangerous. Hardly anyone writes about it.
People who regularly pass on answers they cannot work out for themselves lose the practice of working things out. To others, they appear more competent than before because they answer faster and express themselves better. The team soon regards them as “well informed”. Internally, the opposite is happening. Their competence is on credit: borrowed confidence that holds up as long as nobody asks the second question.
That is how artificial intelligence creates artificially intelligent people. It is not because of the tools. It is because they can take over the most demanding part of our work, and that was the part that made us competent.
Please do not misunderstand me. I am not a holdout. I very much enjoy working with technology and getting people interested in the things that interest me.
That is exactly why I am writing this. Tools with powerful capabilities demand more from their users, not less. A tool that looks good and is operated incorrectly causes more damage than one whose limitations are obvious. I do not want to warn anyone away from technology. I want us to learn how to operate it.
Where it becomes costly
An unchecked answer is harmless only as long as it is merely read. Once someone acts on it, it is no longer harmless. And people act on answers everywhere, in the workshop just as much as at a desk.
Take a technician given a tightening torque. 180 newton metres, neatly presented, without a warning that the value comes from a product series whose designation differs by a single letter. The bolt holds. Until it no longer does.
This is not an isolated case. A notice period without the clause it came from. A total nobody recalculated because it appeared in a complete sentence rather than a table. And a phrase from a meeting that five people subsequently repeat, without any of them still knowing where it came from.
These cases have something in common. The failure is not in the answer itself, but in the lack of traceability. It cannot be noticed because there is nothing against which to check it.
Using a tool and knowing how to operate it
The obvious reaction is to distrust AI answers on principle. That is just as convenient as blind trust, and just as unhelpful. Anyone who checks everything has gained nothing. Neither has anyone who checks nothing.
A tool is powerful when you understand what it does and know how to operate it. The difference between merely using it and operating it properly is the whole point. Anyone who can type can use AI. Operating it means knowing how to read the answer.
For me, that involves four things:
- Ask what the answer is based on before asking whether it is correct. Without a source reference, a statement is an assertion, no matter how good it sounds.
- Know what the tool is suited to. A language model writes well and calculates poorly. Anyone who needs a number must know where it comes from.
- Treat the absence of a result as a result. “There is no information available on this” is information. Replacing it with a plausible guess is the most costly step in the entire process.
- Do not pass on anything you are unwilling to stand behind.
The last rule is the simplest and the most effective.
The part that is not up to the user
It would be too easy to make this solely a matter of discipline.
Most systems do not even give users the material they would need to check an answer. They supply a paragraph without a source reference. Under those conditions, verification takes willpower and minutes, both of which are scarce in everyday work. Anything that requires effort tends not to happen.
That is a design decision, not a question of character. A system can make checking costly or easy. That is why NXTGN AI CORE follows three rules: every key statement carries its source reference. If the system expands a search through the taxonomy, it says so. And if no information is available, it says that no information is available, instead of providing a well-worded approximation. Incidentally, numbers are not generated by the model; they are calculated in the database.
That is enough promotion for now. It does not remove anyone’s responsibility anyway. It makes checking easy enough for it to happen.
My conclusion
Many tasks can be handed over: drafts, summaries, analyses and the first version of a text. You can save hours and reach a presentable result faster. Routine work, in other words.
But responsibility for a statement cannot be handed over. It remains with the person who makes it, and it does not diminish because the wording came from somewhere else. Anyone who passes on an answer stands behind it. That was true before AI, and it has not changed.
And, honestly, that is quite an uncomfortable way to work with a tool that makes everything so easy.
Another question has been on my mind for some time. How does a person still notice that they have not understood something?