Reliance on AI risks turning us into ceremonial box tickers

23 September 2026

Business and economy, Business School

When the Government announced a four-month trial using AI to help write New Zealand's biosecurity standards, Biosecurity Minister Andrew Hoggard reassured the public that "key decisions remain with experts".

The reassurance sounds sensible, but raises a deeper question: how will experts know when an AI system is wrong if they increasingly rely on it to do the work through which expertise is developed?

The biggest risk from AI may not be that machines take over decision-making, but that people gradually lose the ability to challenge it.

Across workplaces, people are already using AI to draft documents, summarise information, classify data and make recommendations. These tools can save time and improve consistency. They may also reduce the opportunities people have to practise their own judgment.

An indigenous technology offers a useful analogue for the problem. Consider the quipu, or khipu, an Andean record-keeping system made from knotted cords that encoded information. Many survive, but much of the knowledge needed to interpret them hasn't. We can still see the records without understanding how they were meant to be read or acted on.

The goal should not be to keep humans "in the loop", but to keep them exercising the skills of interpretation, contestation, ethical reasoning, and accountability.

Organisations dependent on AI systems risk creating the same problem. They may accumulate vast quantities of outputs while gradually losing the expertise needed to question, interpret and challenge them.

A second Indigenous technology offers a useful way to think about an alternative, how organisations can use AI while still preserving human expertise.

Marshallese stick charts, made from coconut fronds and shells, helped navigators learn to read swells, currents, and islands. Navigators studied and memorised them on land. At sea, however, they navigated through situated experience, using judgment and taking responsibility for the course taken.

The charts weren't substitutes for expertise but tools for developing it. They provided a structure for recognising patterns while ensuring that interpretation, judgment and responsibility remained with the navigator confronting the actual conditions.

That is the model AI governance should aspire to: systems that strengthen human capability rather than replace it.

The goal should not be to keep humans "in the loop", but to keep them exercising the skills of interpretation, contestation, ethical reasoning, and accountability. If those skills are lost because people don't practise using them, the loop becomes ceremonial. The human might provide the tick of approval, but much of the analysis has already been delegated to the system.

The biosecurity pilot shows how this could happen. Experts may still make the final decision, but if AI is doing more of the heavy lifting, it can only start influencing how evidence is weighed and interpreted. Over time, the judgment we left to people may migrate into the technology itself.

Already, many organisations are deploying systems that generate recommendations humans struggle to explain. In such cases, accountability may remain with people on paper while judgment increasingly resides in software.

Drawing on the Marshallese stick chart analogy, our recent research on value-centric AI identifies three recurring risks: accountability drifts away from practical agency, human capability erodes, and decisions lose legitimacy. The quipu and stick chart make the contrast tangible. One preserves outputs that can become detached from understanding.

AI can process large volumes of information, detect patterns, simulate futures and generate options. But humans must still ask: What is the situation? Who is affected? What would be legitimate, fair, and responsible in this context?

The practical test is not simply whether a human remains "in the loop", but whether people remain in practice. Can they reconstruct how a recommendation was produced, interrogate the evidence, contest its assumptions, exercise meaningful override rights and explain the decision? If they cannot, nominal oversight becomes ceremonial.

Managers should make priorities such as fairness, legitimacy, and wellbeing explicit before systems are embedded. They should also define roles and boundaries, preserve assumptions and evidence, and establish escalation paths, override rights, and accountability for the final decision.

The quipu reminds us that records can become opaque when the communities and practices needed to interpret them disappear. The stick chart offers a complementary lesson: tools can strengthen situated expertise rather than replace it. The challenge is therefore not whether AI can become more intelligent than humans, but whether we can design AI systems that help humans remain capable, accountable, and wise.

A human signature tells us who approved an AI-generated outcome. It does not tell us whether the expertise needed to challenge that outcome still exists. The critical question for AI governance is not whether humans remain in the loop, but whether they remain capable of exercising judgment when it matters most.

Professor Alexander Richter is the director of Huanui, the Auckland Business School's AI Initiative which works with industry, government and communities to translate AI research insights into societal impact.

Dr Guy Bate is Thematic Lead in AI at the University of Auckland Business School.

This article reflects the opinion of the author and not necessarily the views of Waipapa Taumata Rau University of Auckland.

This article was first published on Newsroom, 23 September, 2026.

Media contact

Margo White I Research communications editor
Mob 021 926 408
Email [email protected]