Opinion | AI can help. It can also fool us. Here is how to use it with care

AI is now inside reports, briefings, audits, and even courtroom summaries. Used well, it saves time. Used blindly, it scales small errors into big ones.

A recent case in Australia shows the risk. A high-value government report, valued at approximately AU$440,000, was created using a generative AI tool. Reviewers later found fake academic references and even a quote that did not exist in the legal record. Following public scrutiny, the firm corrected the report, admitted that AI was used for a core task, and agreed to refund the final payment. The bigger loss was trust. The first version did not disclose AI use. Disclosure came only after outside experts raised concerns.

What went wrong

• Generative AI predicts likely words. It can sound confident while inventing facts. This is called hallucination.
• The firm used AI to fill “traceability and documentation gaps.” That is exactly where facts matter most.
• The failure was not only technical. It was a governance failure. No strong review. No early disclosure. No clear owner for accuracy.

Why this is a watershed

When a paid assurance report uses synthetic facts, the whole model of independent advice is at risk. If a client cannot trust citations, legal quotes, or footnotes, the conclusions fall over. The lesson is simple. AI can draft, but humans must verify every claim that matters.

How this kind of error spreads to the wider world

• Finance: fast trading algorithms can trigger each other and deepen a fall within seconds.
• Fairness: models trained on biased data can deny loans or jobs to the wrong people.
• Safety: driver assist systems have shown phantom braking. One wrong read of a sign can cause a crash.
• Democracy: Deepfakes in recent elections overseas have shifted public mood with fake voices and videos.
• Health: studies show notable diagnostic error rates when tools are trusted without careful checks.

Five rules to use AI with judgment

1. Say when you use it
   Every public report that uses AI should carry a clear note up front. No surprises later.
2. Ground the facts
   Use retrieval-based methods so the model pulls from a verified source at the time of writing. Every quote and citation should link back to a real document.
3. Audit before release
   High-stakes outputs need independent audits that test for hallucination, bias, and missing sources.
4. Put a named human in charge
   One accountable reviewer signs off on accuracy. No tool replaces the duty of care.
5. Match the rule to the risk
   Adopt a simple risk ladder like the EU uses. Ban the worst uses. Treat welfare, law, health, and finance as high-risk and require stronger controls.

The bottom line

AI is not the villain. Silent use and weak review are. If we use AI for first drafts and use people for final truth, we get the best of both worlds. If we skip checks, errors spread at machine speed, and the bill arrives later.

The promise of AI is real. So is the duty to be careful.