Your AI Agent Will Not Leak Personal Data Maliciously. It Will Leak It Helpfully.

Picture the AI data incident everyone braces for and you get something cinematic: a model going off-script, an agent deciding on its own to post a customer list where it should not. Worth defending against. Rarely the first one.

The first one is duller. Someone asks the agent you deployed last quarter for everything it can find on a named account, for a perfectly good reason. It obliges, and one of the eleven documents it draws on is a spreadsheet of grievance notes shared with “anyone in the organisation” in 2021 and forgotten by everybody since.

Nothing failed. No control was bypassed, no guardrail jumped, no clever prompt involved. The agent did what it was built to do, at the speed it was bought for.

1It leaks by being helpful, not by going rogue

Most organisations approach agent risk with a software security model in their heads: find the vulnerability, patch it. That misses the ordinary failure entirely, because in the ordinary failure there is nothing to patch. A better model is a new starter — an unusually capable one who has read every document you gave them, remembers all of it, has no idea which cupboards nobody opens, and answers in four seconds without walking past your desk to check.

That last clause is where the personal data goes. A human with the same access would have found the same spreadsheet eventually — and would have paused. The tone, the file name, the colleagues listed by name would have prompted the thought this was probably not meant for me. That pause is social judgement, not a control anyone designed, and it has been doing unpaid security work for as long as your organisation has.

Every question your agent answers is a question about your permissions model. You have never had to read the answer out loud before.

2Permissions are the whole game

The reassurance every buyer is offered is that the agent respects existing permissions. It is true and it matters. Microsoft states it plainly in its guidance for securing agents: “Responses include only content from references where the user has appropriate permissions.” The same guidance notes that sensitivity labels travel with the content and are respected, and that the conversation shows the most restrictive label among the references used.

Now read that again, slowly. It is a statement about faithfulness, not about safety. It guarantees the agent will not exceed the rights of the person asking, and says nothing about whether those rights were ever right. If a document of home addresses was shared with “anyone with the link” four years ago by someone who has since left, the agent respecting that sharing is not the mitigation. It is the incident.

Obscurity was doing the work

Over-broad sharing is not new. Most organisations carry a decade of it: files opened up to unblock one collaboration and never closed, folders inherited by teams that no longer exist, links pasted into chats. It survived because search was bad enough to work as an access control.

That control has now been removed by a product you bought on purpose, and a decade of over-shared personal data becomes findable in one plain English sentence, by someone who was not even looking. Microsoft names this rather than hiding it: broadly shared content surfaced through agent responses is called out as an oversharing risk, with an oversharing data loss prevention policy for SharePoint and OneDrive offered as one mitigation. Which tells you where the work is. Not the agent — the sharing.

You did not have a data protection posture. You had a search engine bad enough to look like one.

3The service account trap

The second identity failure runs the other way, and teams walk into it while trying to be efficient. Narrowing an agent’s access to the right sites, tables and mailboxes means a ticket and a conversation with whoever owns each source. Granting it broadly means a checkbox and a demo that works on Thursday.

So the agent ends up with wider reach than any human in the building, answering for everyone. Whatever the asker’s own permissions, the answer is drawn from the union of everything the agent sees. “Who was allowed to know this?” stops being answerable, and so does “who asked?”.

There are legitimate reasons for an agent to hold its own identity: scheduled work, background processing, anything with no human to impersonate. The rule of thumb is that the dials move against each other — the broader the identity, the narrower the data it should reach.

4Retrieval is the exposure surface, not the model

Attention goes to the model, because the model is interesting. For personal data it is close to the least important part. The surface is the retrieval layer between your systems and the prompt. An index spanning sources of mixed sensitivity, with no per-user trimming at query time, is a disclosure engine wearing a chat interface.

Two details do most of the damage. The first is when trimming happens: at query time against the asker’s current rights, because an index is a photograph and access changes constantly. The second is that a chunk arrives stripped of its surroundings — three paragraphs from an occupational health record reach the model with no folder, no header and no owner.

Which is why label-aware controls at the grounding layer are worth understanding. A Purview data loss prevention policy scoped to the Microsoft 365 Copilot location can exclude labelled content from grounding: “Identified items still appear in the citations of the response, but the content of the item isn’t used in the response or accessed by Copilot.” Note the shape of it: the document is visible; its contents are not.

Whatever you are building on, ask the retrieval design four things:

• What is actually in the index? Not what was intended. List the sources and name the worst thing in each.
• Is trimming per-user, at query time? If the answer contains “at ingestion”, you have a photograph, not a filter.
• Do labels survive chunking? Sensitivity on the file but not the fragment disappears at the wrong moment.
• Can you enumerate it? If nobody can list what the agent reaches, nobody can say what it might.

5Personal data ends up somewhere it has never been

Then the second-order problem: not what the agent shows, but where the answer goes next. The prompt is logged for debugging. The response is pasted into a ticket so a colleague has context, summarised into an email, exported to a spreadsheet. Each is a fresh copy of personal data in a system with different retention and different access, and none appear on your map of where personal data lives, because that map predates the agent.

It bites at the unglamorous end. When someone makes an erasure request, the places you must reach now include prompt logs, transcripts and whatever your observability stack kept. None of this is legal advice — take that from someone qualified — but whatever regime you fall under, the practical question is yours: can you find every copy?

• Decide retention before launch. Prompt logs and transcripts need a defined lifetime, like every other record.
• Keep transcripts out of general-purpose logging. In a shared log platform they inherit its access list, which is generous.
• Treat an export as a copy. It is the copy nobody remembers when the request arrives.

6Minimise before the model ever sees it

The most effective control here is the least discussed, because it is not a product. The default when wiring an agent to a system is to hand over the whole record, for the unimpressive reason that the whole record is what the API returned. A customer row from your ERP arrives with dozens of fields; the task needs three. Sentiment analysis on a support conversation needs no date of birth, home address or account number, yet all three ride along into the prompt, the log and the transcript.

So send the fields the task needs and nothing else. Where an identifier exists only to join the answer back to a record, replace it with a token that means nothing outside your system and swap it back on the way out. Redact the rest. The test is quick enough for a code review: for each field in the prompt, what would be missing from the output without it?

Data you never put in the prompt cannot appear in the answer, the log, the ticket or the export. It is the only control here with no failure mode, and the cheapest: one conversation at design time, paying out for as long as it runs.

7Ask your own agent for things it should refuse

Testing before go-live is nearly always functional: does it answer accurately, cite properly, cope with a vague question. Almost nobody tests the other direction, which is the one that produces the phone call. Sit down and try to make the agent say something it should not:

• Salary, performance or grievance information from another department.
• A named colleague’s home address, personal number or absence record.
• “Summarise everything you know about [person].” The most revealing prompt there is, and the first one a curious employee thinks of.
• A customer’s full record when the task only needed a delivery date.
• The same questions, asked by three people at different access levels.

That last one is the test that matters. Ask identical questions as a director, a team leader and a new starter, then compare the answers. It tells you more about your permissions model than an audit report will.

The honest summary

The controls that keep personal data out of your agent’s answers are aggressively boring. Identity: who the agent acts as. Scope: what it reaches, and who agreed. Retrieval filtering: trimmed per user, at the moment of asking. Logging and retention: what is kept, where, for how long. Minimisation: what never went in. Nothing exotic, and no new discipline to hire for.

What is new is the speed at which weaknesses you already had become visible to people who were not looking for them. The oversharing was there before, and so were the stale permissions and the unlabelled spreadsheet of personal data in a team site nobody owns.

You have just installed something extremely good at finding it, and handed it to everybody. Go and look first.