We have a human-in-the-loop savior complex

We tend to talk about keeping a human in the loop as though a person's involvement automatically brings judgment, scrutiny and safety, and it has become the familiar answer whenever agents are given access to sensitive information, business applications or consequential actions. I think that answer carries more assumptions than we usually admit. We rarely say who should be making the decision, what they need to see, or whether the review gives them a real chance to challenge the agent, so approval can become a formality long before anyone notices that the human has stopped exercising meaningful oversight.

When the agent shapes the decision

New research from Anthropic shows how this happens. Researchers asked 50 professional analysts to complete a statistical analysis with an AI agent, including four decisions that each had defensible alternatives, and secretly gave each agent a randomly selected preference to argue for honestly. The analysts followed their agent's assigned preference 86% of the time, and those who had stated their own preference beforehand frequently changed it when the agent disagreed.

That figure does not mean the analysts were wrong 86% of the time, because every option could be defended and the choices turned on analytical judgment rather than a single correct answer. It also has limits: the researchers deliberately strengthened the agents' preferences, and the study used one model, one task and a small group of professionals, so the rate will not carry over to every environment. What it does show is that the analysts kept responsibility for their recommendations while the agent had already influenced which assumptions and methods they considered appropriate.

The influence was also hard for the analysts themselves to see. Asked afterwards to explain their choices, 77% repeated at least one of their agent's arguments, sometimes almost word for word, and only 15% credited the agent as a source.

Human involvement does not guarantee a better outcome

A 2024 meta-analysis in Nature Human Behaviour compared humans alone, AI alone and the two together across 106 experiments. On average the combinations outperformed humans alone but fell short of whichever of the two did best by itself, and the results varied by task, with decision-making among the harder cases.

Neither study persuades me that people should be removed from consequential decisions, since judgment about organizational priorities, acceptable risk, competing interests and accountability still belongs with humans. They do suggest that including a person does not by itself make a process safer or more reliable. Reviewers are influenced by how a recommendation is framed, which alternatives are shown and whether the agent has already made choices that narrow the options, and they tire when asked to approve large volumes of actions without enough context to separate routine activity from the few that need attention. In those conditions the review satisfies a procedural requirement without delivering the scrutiny the organization intended.

Designing oversight that can work

When someone tells me an agent has a human approval step, I want to know how that step operates, and I start with who is making the decision. A security engineer, a domain specialist and a business owner bring different expertise and authority, and someone qualified to judge the technical accuracy of an action may not be the right person to decide whether its business consequences are acceptable.

Next is what the reviewer is shown. If they see only the proposed action and a persuasive explanation, they are evaluating a conclusion on the agent's terms, whereas the supporting evidence, assumptions, alternatives and likely consequences give them something to challenge. Timing matters in the same way, because an agent may make a long series of analytical or operational choices before approval is requested, and by then the decisions that mattered most have already shaped the outcome.

Volume comes after that. Asking people to examine every low-impact action makes it harder for them to spot the few that deserve attention, so organizations need to separate actions permitted within established boundaries, those that benefit from contextual guidance and those that need informed human authorization. Each of these needs a clearly stated purpose, because checking that an action is technically correct, deciding that its risk is acceptable and authorizing a change with significant business consequences are different responsibilities that call for different reviewers, information and controls. The approval process should reflect those differences and be tested against realistic agent behavior, including cases where a recommendation is plausible but its assumptions deserve challenge.

Measuring the value of oversight

Counting approvals requested, granted or rejected tells us how often a process is used and very little about the quality of the decisions inside it. I would rather ask whether reviewers had enough information to form an independent assessment, whether they found material problems, whether their involvement changed outcomes when it should have, and whether the volume and timing of approvals leave them room to exercise the judgment we are relying on.

Human oversight is a valuable control, particularly where someone has to take responsibility for decisions involving uncertainty or competing priorities, and making it effective means being deliberate about the conditions in which people are asked to exercise that responsibility. A person clicking Allow shows that an approval took place, and only the design around that click can tell us whether judgment did.