Enterprise leaders have spent years being told that AI will make their organisations faster and more efficient. This is true, but Peri Kadaster, Chief Communications Officer at Nearform, explains that there’s also a harder hitting truth sitting underneath the tech’s promise: modern AI can create sensitive knowledge from data that didn’t appear sensitive in the first place.
A user doesn’t need to disclose a medical condition for an LLM or other system to infer a potential health risk from their search behaviours. An employee doesn’t need to explicitly state financial anxiety for patterns in behaviour to suggest it. A person doesn’t need to reveal their political view for a model to assemble it from movement, language, search and device data.
Sensitive information might not be explicitly shared, but an organisation can still end up holding knowledge that feels personal, consequential and, to the individual, unexpected.
For executives, this isn’t a niche compliance issue – it’s a board-level trust issue that’s growing fast. Stanford’s 2025 AI Index reported a 56% year-on-year rise in AI-related privacy incidents, with 233 documented cases in 2024. Privacy risk is no longer just about losing data, as it’s also about systems inferring data and sensitive information without a user’s knowledge or consent.
The privacy contract has changed
The old privacy model wasn’t perfect, but it was familiar. Businesses collected information, and users consented (often by clicking ‘Agree’ at the bottom of a lengthy web form). Risk was contained within over-collection, misuse, poor retention or a data breach.
AI breaks that process, because inference is now part of the product. Models are designed to identify relationships that people can’t see – this is the very thing that makes it valuable in fraud detection, demand forecasting, logistics, clinical support and project delivery. However, it’s that same capability that allows them to draw conclusions that users may not expect, and leaders may not have properly governed.
This is where many enterprises are exposed. They can list the datasets they collect, where they store them, and who can access them – but very few can list the sensitive inferences their models might generate once those datasets are combined.
That gap can pose many issues. If a postcode becomes a proxy for income, typing patterns suggests stress levels, or a purchasing history indicates a health condition, organisations can end up handing sensitive insight, even when the original data fields seemed harmless – and treating that as ‘low risk’ because it’s been anonymised is not the best approach, especially in the long run.
Anonymised data doesn’t mean safe
Anonymisation was built for a world with fewer signals and less powerful cross-referencing. Today, behavioural data is rich, messy and highly linkable. When enough fragments are combined, re-identification becomes a predictable outcome.
Consent has similar limits. A long privacy notice may satisfy a process, but it rarely gives a person a clear view on what an AI system may later infer. There’s a difference between agreeing to share information and understanding the conclusions that could be built from it. Leaders who ignore that difference may stay technically compliant for a while, but the impact on trust will be long-lasting.
To address the evolving risks, enterprises need to stop treating privacy as a legal wrapper around engineering decisions. Legal, risk and communications teams do have a role, but decisions impacting privacy are being made much earlier, in system architecture, product design, data selection and model evaluation. Trust needs to be designed into the system from the very beginning, across all of these areas.
Inference needs ownership
The question for enterprise leaders is to determine which conclusions systems should be allowed to draw. That decision can’t be left to model capability alone. Some inferences clearly serve the user, such as those helping to detect fraud, improve safety or flag operational risk before a project drifts. But others serve the organisation, including silent profiling, secondary monetisation or hidden classification of people into risk groups. Both may be technically possible, but they’re not ethically or commercially equal.
This is where leadership discipline is needed. So, before an AI system reaches production, teams should be able to explain what it’s intended to predict, what else it could plausibly infer, and who benefits from these conclusions. If the answer is vague, the system isn’t ready to ship.
A useful test is whether a reasonable person would expect the inference. Another is whether the benefit justifies the legal, reputational or operational risks. If an inference doesn’t pass those tests, it should be constrained in the design phase – not politely discouraged in a policy document that gets overlooked during the delivery phase.
What leaders can do now
Waiting for regulation to keep pace with evolving AI privacy risks is a poor strategy. The companies that prioritise trust will have the advantage – because customers, employees and partners are already forming their own views of what responsible AI looks like.
AI will keep seeing patterns that humans miss- which is the point of the technology and one we can remain to admire (and desire). But the leadership task now is to decide which patterns should become business knowledge, which should be blocked, and how openly those choices are explained. The organisations doing this well will naturally move faster because they won’t be busy rebuilding trust after dire damage is done.
Ciaran Cosgrave is CEO of Nearform
