Prompt Data Anonymization Techniques for Ad Intent Modeling

Conversational AI now runs through more than 400 million weekly active users, generating billions of chat exchanges a month. That volume contains a signal advertisers have wanted for two decades: what someone actually intends to buy, stated in their own words. Getting that signal out means handling some of the most sensitive text a person will ever type into a product, and most of the anonymization techniques running in production today solve the easy half of that problem while quietly ignoring the hard half. That is the actual state of the field before getting into the mechanics.

Prompts say more than search queries ever did. A search box gets keywords. A chat window gets sentences: "my knee's been swelling for three weeks and I can't afford a specialist," or "trying to figure out if we can refinance before the baby comes." Luke Jinu Kim, CEO of the AI search engine Liner, has said user intent appears far more clearly in conversational prompts than in traditional search queries. That's what makes prompt data so valuable to an ad platform, and it's also what makes it dangerous to mishandle.

Meta made this concrete. Starting December 16, 2025, Meta began using Meta AI conversations to personalize ads across Facebook, Instagram, WhatsApp, and Messenger, a change touching roughly 1 billion monthly Meta AI users. Ask Meta AI which hiking boots suit a beginner, and that question becomes a targeting signal for hiking gear ads. Older ad targeting relied on what people chose to post publicly. Prompt data hands the platform direct, explicit statements of intent instead, unfiltered by whatever social performance normally shapes a public post. The same directness that makes this data commercially useful also makes it legally and ethically loaded, so pulling purchase intent out of a prompt without exposing the person behind it requires a real anonymization layer sitting between those two goals, and that layer has gotten far more sophisticated than a find-and-replace script.

What prompt anonymization needs to protect against

Two separate threats live inside a single prompt, and treating them as one problem is where a lot of early anonymization work went wrong.

The first is explicit leakage: names, addresses, account numbers, phone numbers, email addresses. Classical named-entity recognition and rule-based scrubbers were built for exactly this, and for straightforward cases, they still hold up fine.

The second is harder to catch, and nearly impossible to fully close. Attribute inference attacks let a language model guess sensitive traits, age, gender, location, income bracket, from text that contains no PII whatsoever. Research on this problem (Yan et al., TRACE-RPS, arXiv 2602.11528) shows these attacks running through completely benign-looking prompts, scaling automatically, and staying technically inside the bounds of most safety guidelines. An LLM publisher can't just dull the model's reasoning to close this hole, because reasoning is the product itself. Stripping information out reduces what an attacker has to work with, but it doesn't stop a capable enough model from reasoning over whatever text remains. That gap is a structural feature no amount of scrubbing rules removes. It's structural, and no amount of scrubbing rules removes it.

Regulation doesn't care that conversational AI is new. GDPR, HIPAA, and CCPA all impose strict rules on how personal data gets processed, stored, and shared. None of those rules were written with chatbots in mind, but all of them apply anyway. OpenAI's production setup shows what taking this seriously looks like: the platform is designed to keep ad serving separated from raw conversation content. Advertisers do not receive full transcripts, and OpenAI holds most user-level advertising interaction data for 30 to 90 days before automatic deletion.

The DePrompt framework (arXiv 2408.08930) states the underlying tension cleanly: privacy and utility in prompt anonymization pull in opposing directions, and every design choice trades one off against the other. Push hard on one and another gives way. Every technique below is a different bet on which of those three to sacrifice, and by how much.

Type-consistent substitution: replacing PII without dropping semantic signal

The simplest fix that actually holds up is substitution instead of deletion. Rather than blacking out a name or an email, swap it for a fake one that's structurally identical: a real inbox becomes a plausible fake inbox, a real order number becomes a different, nonexistent order number. Sentence shape and fluency stay intact, which matters downstream because sentiment models, intent classifiers, and agentic reasoning chains all depend on grammatical structure surviving the edit.

The Anonymous-by-Construction framework (arXiv 2603.17217, March 2026) runs entirely on-premise, so data never leaves the organization's own servers to hit a third-party API. It uses GPT-oss at 20B parameters and DeepSeek-r1 at 7B parameters, picked for how well they follow instructions and handle multiple languages. The pipeline splits into three sub-tasks: finding spans across categories like names, usernames, phone numbers, addresses, and account IDs; generating a contextual replacement that fits the surrounding sentence; and preserving the tone and sentiment of the original. Decoding runs at temperature zero, so output stays deterministic instead of drifting between runs.

Benchmarked against Microsoft's Presidio, along with Google DLP and several ZSTS variants, this approach comes out ahead on the combined privacy-utility-trainability frontier. The trainability part matters because it decides whether sanitized output can be used beyond one-off inference: when researchers fine-tuned a language model with a parameter-efficient tuning method on the sanitized output, the loss in downstream performance stayed low. Anonymized text can still train a model.

But substitution only ever solves half the job, and treating it as a complete answer is the mistake most teams make early on. It handles explicit PII cleanly. It does nothing about attribute inference sitting in the non-PII parts of a sentence, the tone, the word choice, the specific complaint someone typed out. That gap is exactly where the next layer picks up.

Adversarial anonymization: iterating toward inference resistance

Rather than guessing what an attacker might infer, this approach builds the attacker into the pipeline and lets it fight the anonymizer directly. One LLM plays adversary, trying to infer private attributes from the text. A second LLM, the anonymizer, removes or scrambles whatever cues the adversary just used. Then it runs again, each round making the remaining cues harder to exploit.

Work presented at ICLR 2025 on LLM-driven adversarial anonymization found that this loop beats traditional anonymization on both text quality and privacy protection, a result that should unsettle anyone who assumed more complex systems automatically mean safer ones.

TRACE-RPS refines the idea with two components working together. TRACE uses attention weights and generated inference chains to pinpoint the exact words doing the damage, word-level precision instead of blacking out a whole paragraph because one phrase was risky. RPS then runs a lightweight two-stage optimization that appends a suffix engineered to trigger a rejection response, so a model asked to infer an attribute answers "I cannot determine that" instead of guessing. Together, the pair drops attribute inference accuracy from around 50% down to below 5% on open-source models tested.

That kind of inference resistance doesn't come free. Push the adversarial loop hard enough and the edits start eating into the semantic coherence intent classification needs to function at all. There's a real ceiling on how many rounds a pipeline can run before the text is technically anonymous but useless for targeting. The defense generalizes well across different attacker models and holds up even when an attacker varies its inference prompts, which is a genuine strength. For intent modeling, the text stays workable, but the pipeline runs heavier, and someone has to decide exactly where the threshold sits against how much intent signal they're willing to give up.

PromptObfus and desensitization models: perturbing words without breaking predictions

A team from Shanghai Jiao Tong University took a different angle with PromptObfus (arXiv 2505.01273): treat desensitization as a masked language modeling problem. Privacy-sensitive words get swapped for a [MASK] token, and a desensitization model generates candidate replacements for each one.

The clever part is how candidates get picked. Instead of choosing whatever sounds most natural to a human reader, the system uses gradient feedback from a surrogate model built to emulate how the remote LLM would actually behave on the task. Replacements get chosen to minimize disruption to that predicted output, not to fool a person skimming the sentence. Two models run entirely on-premise to make this work: the desensitization model itself and the surrogate.

Across three NLP tasks, this held up well after desensitization: sentiment analysis accuracy at 84.8%, topic classification at 84.25%, question answering at 96.4%. On the privacy side, it cut the success rate of implicit privacy inference attacks by 24.86% and fully blocked explicit inference attempts.

DePrompt, out of Xidian University, takes a related but distinct route. A fine-tuned LLM defines privacy types by pulling in contextual attributes, then runs an adversarial generative desensitization step that severs the link between identifiers and the sensitive attributes attached to them, while trying to hold onto the semantic content that matters. DePrompt's own dataset analysis, run across a sample of 1,000 public prompts, found medical scenarios accounted for 39.52% of PII leakage, everyday life scenarios 33.05%, and financial scenarios 27.43%. Health and finance happen to be the two categories advertisers pay the most for. Anonymization pressure runs heaviest exactly where the commercial stakes run highest, and that's not a coincidence any team building an ad intent pipeline gets to ignore.

The real advantage of gradient-guided selection over plain substitution is that replacements get chosen to preserve how the downstream model behaves, not to sound convincing to a person reading it. An intent classifier can keep working correctly even when the substituted word looks nothing like the original on the surface. The limitation is visible in black-box settings, where the surrogate model has to approximate the real target closely enough for its gradient signal to mean anything, and that approximation gets shakier the less visibility there is into the target system.

Differential privacy and federated approaches: protecting the pipeline, not just the text

Anonymizing the text isn't the end of the exposure. Even a perfectly sanitized prompt can leak information back out through a model's learned weights, if training data patterns allow reconstruction later, a memorization risk entirely separate from anything happening at inference time. Fixing the text doesn't touch that at all.

Older cryptographic answers, homomorphic encryption and secure multi-party computation among them, come with real costs in LLM settings: heavy compute overhead, dependence on user participation, and, per the same PromptObfus research, an inability to hit both real-time performance and solid privacy protection at once under black-box conditions.

Federated learning and differential privacy approaches for ad recommendation try to split the difference by combining distributed feature extraction with privacy budgets designed to resist manipulation. The goal is reducing coordination overhead while holding model accuracy steady.

Some approaches push differential privacy down into the embeddings themselves, injecting noise into word vectors and clustering the representations of sensitive terms so they become indistinguishable from each other, changes that operate on the readable text without ever appearing in it. Separately, ADPHE-FL (Wu et al., 2025) pairs adaptive differential privacy with homomorphic encryption inside a federated learning setup, tackling confidentiality at the model level rather than the prompt level.

For intent modeling, purchase intent can get pulled from patterns spread across thousands of users and devices without ever centralizing a single raw prompt or the identity behind it. The honest cost is that distributed architecture adds latency and coordination overhead, and a real-time ad auction has very little patience for either.

What each technique costs in intent signal fidelity

Four things actually matter when judging these techniques against an ad intent use case. Does the anonymized text still classify into the right intent category? Can a model trained on the anonymized data generalize? Does the technique fit inside a real-time auction's time budget? And does it work without access to the serving model's internals?

Type-consistent substitution wins on the first two and costs almost nothing in latency, but leaves attribute inference from non-PII signals completely unaddressed. It's the right call when explicit PII is the dominant risk and nothing more exotic is being inferred. It's the wrong call the moment someone treats it as sufficient on its own, a mistake made directly rather than hedged around.

Adversarial anonymization is the strongest defense against inference, full stop. TRACE-RPS's drop from 50% down to below 5% accuracy is a real result, not a marketing number. But the iterative loop adds compute cost, and pushed too far, the editing erodes the coherence a classifier needs to function. Calibrating how many rounds to run is the actual engineering job here, and it's a job most teams underinvest in.

Gradient-guided desensitization, PromptObfus's approach, holds task performance best when the surrogate model tracks the real target closely, and degrades once the setting turns black-box. It fits pipelines where the intent classifier is known and running locally, not ones bolted onto someone else's closed API.

Federated and differential privacy architectures solve a problem the other three don't touch at all: training-time memorization. They let intent modeling happen at population scale without ever centralizing raw prompts. The tradeoff is latency and coordination overhead that a real-time bidding system feels immediately.

DePrompt's framing of linkability and uncertainty points at something uncomfortable. Pushing uncertainty up, making re-identification harder, tends to push linkability down in ways that also break intent modeling across a session. A user's sequence of related questions in one conversation can stop cohering as a single intent thread once anonymization gets aggressive enough. No single technique wins on all four dimensions at once, and anyone claiming otherwise is selling something. Layering works best: substitution as the first pass, adversarial or gradient-guided refinement wherever inference risk runs high, and federated architecture handling the training layer that produces the privacy guarantees the other two depend on.

The anonymization layer's role in a live ad intent pipeline

Anonymization decisions get made at four distinct points, not one. Prompt ingestion happens first, ideally on-device or on-premise before the full text goes anywhere else. Intent extraction comes next, with an anonymized or partially anonymized version of the text feeding a classifier or embedding model. Signal transmission is what actually reaches the ad auction, and it takes the form of a topic label, an intent category, or an embedding vector, never a prompt that could be reconstructed back into its original form. Advertiser reporting sits last, and it only ever sees aggregate performance data, never anything traceable to one person.

OpenAI's production setup shows what taking this seriously looks like: content generation and ad serving run as independent pipelines, and advertisers do not receive full transcripts.

The Anonymous-by-Construction framework's work on agentic question-answering applies directly here. Put an on-premise substitution layer ahead of the model that actually answers the user, and sensitive content never reaches a third-party API in the first place, while the model downstream still produces a quality response and the intent signal survives the trip through the intermediary intact. That's the shape the whole discipline is converging on: anonymization not as a single filter bolted onto the front door, but as a layer distributed across every stage a prompt passes through on its way to becoming a bid.