The real AI risk isn’t job losses, it’s who can see what
Beyond the fearmongering, it’s a more mundane AI threat that will be making waves in boardrooms, writes Paul Armstrong
The AI risk that will actually reach a UK boardroom this year won’t be that machines are taking everyone’s job, but the more awkward one about machines showing the wrong people things they were never meant to see. New research from Box.com launched today at their annual Boxworks conference, puts an uncomfortable figure on it, with nearly half of the organisations surveyed admitting an AI tool has already surfaced internal content a user should never have been able to reach. Worse, most of those same organisations can’t say with any confidence where their AI tools are even running.
The cause, in most cases, is structural rather than careless. Companies are connecting AI agents to their own knowledge at speed, pushing them out of pilots and into live workflows where they read, summarise and act across documents, inboxes and systems that accumulated their permissions haphazardly over years. Access controls that were merely untidy when a human had to go looking for a file turn genuinely dangerous once an agent can reach everything at once and hand a tidy answer to whoever asked the question.
The permission problem nobody priced in
Shadow IT, the (not new) habit of staff reaching for tools outside the sanctioned setup, returns here in a more potent form, because an employee no longer needs to copy a sensitive file to misuse it when an agent will quietly retrieve and repackage its contents on request. A single mis-set permission stops being an isolated mistake and becomes a repeatable leak, served politely and at scale to anyone who phrases the prompt well enough. Salary bands, redundancy lists, unannounced results, a half-finished acquisition memo: none of it needs to be hacked when an obliging assistant will fetch it for a colleague who simply asked nicely.
Boards that spent a decade defending against dreaded external breaches now face a more embarrassing threat from within, a tireless army of automated whistleblowers they built and deployed themselves, handing over whatever is asked for without guilt or fear, for the simple reason that nobody ever told them they shouldn’t.
Governance as the product
Box is focusing on the multi trillion-dollar market opportunity. Samantha Wessels, president of the EMEA business, argues that the firms winning with AI are not the ones ‘simply deploying more AI tools’ but the ones building the foundations underneath, the trusted content and the permissions and the governance that decide what an agent can reach and who gets to see the result.
“The companies seeing the best results won’t be the ones throwing more agents at the problem. They’ll be the ones who have done the more difficult, but much less glamorous, work underneath [of] knowing where their knowledge lives, who should access it and what decisions an agent should actually be allowed to make. The next battleground for enterprise AI is trusted, portable context that moves securely across whichever AI tools businesses choose to deploy,” Wessels said.
The key is not treating governance as a brake on innovation, that misses the point entirely because the same research has leaders agreeing, almost unanimously, that permissions and access controls are now critical to trustworthy enterprise AI, and that better governance is the thing letting them move faster rather than slower.
The practical work is unglamorous and overdue. Knowing what each agent can actually access, who can see its outputs, and where across the business it is running at all turns out to matter far more than any model-selection question a leadership team is likely to agonise over. Visibility has to come first, since an organisation can’t govern what it can’t see, and most, on the evidence of this research, currently can’t see very much at all, least of all the regulated firms for whom an exposure of this kind isn’t merely awkward but reportable.
The AI story that ends up on this year’s board agendas won’t be the cinematic one about redundant workforces, but the mundane one about who can see what, and which agent showed it to them. Firms that treat the access layer as seriously as the technology running on top of it will be at an advantage down the line, the message is clear, quietly fix the plumbing before a leak becomes a headline.
Paul Armstrong is founder of TBD Group and author of Disruptive Technologies