Briefing: When AI makes the recommendation, who owns the judgement?

For several years, one phrase has provided reassurance whenever concerns are raised about decisions made by artificial intelligence (AI) that affect people – there will always be a human in the loop.

It’s a comforting thought, but I increasingly wonder whether we are asking the wrong question.

Having a human involved in a decision only provides meaningful protection if that person has the knowledge, experience and confidence to question what technology is telling them.

Simply inserting somebody between an algorithm and an outcome does not, by itself, constitute effective oversight – and I’d argue that this particularly matters in insurance.

Our profession has always depended on judgement. Whether it is an underwriter assessing an unusual risk, a claims professional responding to individual circumstances, or a broker interpreting a client’s needs, the best decisions are rarely the product of data alone. They also draw on experience, context and professional judgement.

However, AI potentially changes that relationship. Increasingly, technology will be able to analyse information, identify patterns and make recommendations at a speed and scale beyond an individual professional.

The question, therefore, isn’t simply whether insurance professionals should trust AI. It is knowing when to trust it, when to question it and, occasionally, when to override it.

This question has become more pertinent with the FCA’s latest publication on good and poor practices for AI in financial services, issued earlier this week (19 October).

Human judgement remains vital

The latest statement from the regulator reinforces that existing expectations around Consumer Duty, governance, operational resilience and senior management accountability apply to firms’ use of AI, alongside the importance of human oversight.

There is an important principle underneath this. Technology can support a decision, but it doesn’t remove responsibility for its consequences.

Imagine, for example, that an experienced underwriter receives an AI-generated recommendation to decline a risk. They have seen similar cases before and believe the recommendation is wrong. So, should they override it?

Now turn the question around. What happens when a professional overrides an AI recommendation, only for the technology to prove to have been correct?

Over time, will repeated experiences like this make people less willing to challenge the machine? The risk is that the human in the loop becomes little more than someone wielding a rubber stamp.

Experience takes time

This debate around the role of a human practitioner also leads to another challenge – professional judgement does not suddenly appear when someone becomes sufficiently senior. It must be developed.

Traditionally, insurance professionals have built judgement progressively, with expertise accumulated through exposure as much as through formal learning. 

They deal with straightforward cases, encounter exceptions, observe experienced colleagues handling difficult situations and gradually take responsibility for more consequential decisions.

But if technology increasingly handles routine analysis and provides recommendations on more complex cases, we need to think carefully about where the next generation will gain the experience that eventually enables them to challenge those recommendations.

This is the paradox. AI adoption cannot simply be treated as a technology implementation challenge, because it is a professional development challenge too. 

AI literacy will undoubtedly become important, but knowing how to operate technology is only part of what future competence will look like. 

Professionals will need to understand enough about the strengths and limitations of the systems they use to question their outputs intelligently. They must retain the confidence to recognise circumstances that data may not adequately capture.

Organisations also need cultures where challenging an automated recommendation is regarded as evidence of professionalism rather than an obstacle to efficiency. But, equally, organisations must avoid creating environments in which professionals are effectively punished whenever they depart from an AI recommendation which subsequently proves correct. The predictable result would be people following the machine rather than exercising genuine judgement.

For professional bodies, employers and regulators, this means thinking carefully about what competence looks like when humans and machines increasingly make decisions together. Technical expertise will remain important, but so will critical thinking, ethical reasoning, curiosity and the ability to explain and defend a decision.

Ultimately, the objective should not be to keep a human somewhere in the process simply so that we can say one was there. The real test is whether that human adds something uniquely valuable to the process.

Because one of the defining professional skills of the AI age may not be knowing the answer. It may be having the knowledge, confidence and judgement to recognise when the machine’s answer isn’t good enough.