Personal Attribute Inference (Inference Privacy Attacks)

For years, LLM privacy research focused almost exclusively on training-data extraction—preventing models from regurgitating memorized social security numbers, phone numbers, or email addresses seen during pretraining. In October 2023 (ICLR 2024), Robin Staab and colleagues at ETH Zurich’s Secure, Reliable, and Intelligent Systems (SRI) Lab published "Beyond Memorization: Violating Privacy Via Inference with Large Language Models," demonstrating that frontier LLMs can infer personal attributes from pseudonymous Reddit posts and interactive chat sessions with 85% top-1 accuracy at 100x lower cost and 240x higher speed than human investigators.

Sources: s1, s2, s3

Standard enterprise privacy pipelines rely on Named Entity Recognition (NER) and regex tools (such as Microsoft Presidio) to scrub explicit identifiers before sending text to an external LLM API or storing it in an analytics lakehouse. However, if a user mentions local slang, school-term schedules, or commute patterns, an LLM can still reconstruct their city, age bracket, occupation, and socioeconomic profile, creating both re-identification risks and covert demographic profiling.

Sources: s1, s2

An employee anonymizes a customer complaint by replacing the customer’s name and street address with `[REDACTED]` via a standard NER filter. The remaining text mentions "waiting for the hooks turn at the tram intersection after finishing my shift at the Royal Children’s." From those two cultural and geographic cues alone, an LLM immediately infers that the author is a healthcare worker living in Melbourne, Australia.

Sources: s1, s3

Attribute inference produces probabilistic hypotheses rather than deterministic database lookups; it is prone to stereotyping and false positives when users adopt non-native dialects or fictional personas, and iterative adversarial anonymization (LLM-guided paraphrasing) can reduce inference accuracy at the cost of stylistic fidelity.

Sources: s2, s3