Who is accountable when AI gets it wrong?
AI produces outputs that vary with each interaction and operates across systems at a pace no human oversight function can match. So who’s accountable when it goes wrong, asks Paul Loftus
When an automated system makes a decision that harms a customer, there is usually a clear legal answer to the question of who is responsible. By extension, under the UK’s Senior Managers and Certification Regime, a named individual remains legally accountable for outcomes in their area, however those outcomes are produced.
What is less clear is how that individual meaningfully discharges that responsibility when the system producing the outcome behaves differently from anything that has come before. Unlike previous systems, generative and agentic AI produces context-dependent outputs that vary with each interaction and operates across systems at a pace no human oversight function can match. This is the central governance challenge AI is presenting to the financial services industry.
That is, the benefits these innovations can deliver are contingent on the governance frameworks overseeing these tools being genuinely fit for the technology being deployed. Notwithstanding the overarching application of the Consumer Duty, it remains likely that governance failures may result in poor customer outcomes. Further, it is a feature of our regulatory environment that senior managers carry the legal responsibility for those outcomes. They also carry the responsibility for ensuring the governance frameworks under their watch are capable of preventing them.
That said, research from Zango found that oversight functions are struggling to keep pace with the adoption of AI, and that the governance frameworks firms are relying on were not built for it.
How can we govern AI in financial services?
Legal and compliance functions have to evolve accordingly. We must turn our attention and assessment to whether the control environment around AI systems is sound, whether human judgement is intervening at the right points and whether the firm can demonstrate that customer outcomes are being protected throughout. That requires lawyers, risk professionals and compliance teams working together in a way that many firms have not yet formalised.
Further, it means revisiting assumptions that have served as the foundation of client-facing processes for years. Assumptions like how information is gathered, how suitability is assessed, how decisions are reached and recorded. Those processes must be stress-tested against failure modes that did not exist when they were designed. This speaks to a reimagined governance model – one ready and able to address these novel challenges.
Financial services is a significant contributor to the UK economy, and the government’s growth agenda is substantially premised on AI driving productivity across sectors. The cost of getting this wrong is therefore considerable. A significant AI-related failure in a regulated firm – a material error in a customer outcome produced autonomously and at scale – may harm those directly affected, but it would also create the conditions for reactive, crisis-driven regulation of a kind that stifles exactly the innovation the sector needs.
AI will undoubtedly revolutionise processes and drive efficiencies within firms. At St James’s Place, we believe agentic AI will complement human-led processes and, if interoperability is achieved carefully and with precision – as we are working to do – the benefits for our clients, our firm, and the wider economy will be significant.
Paul Loftus is general counsel at St James’s Place