How Do AI Systems Process Personal Data?

The consequence: an AI system is not a tool that sits outside the data protection framework until it does something obviously risky. Every AI system that touches personal data at any point in its operation is a data processing activity — from the moment training data is assembled through every query a user submits in production.

Why AI Systems Are Different From Traditional Software

Traditional software processes personal data in predictable, defined ways: a CRM stores a name and email; a payment processor transmits card data; an analytics tool counts page views. The data in, data out relationship is auditable and bounded.

AI systems process personal data differently on three dimensions that create novel compliance challenges:

1. Data is absorbed into model weights, not stored as retrievable records. When an AI model trains on a dataset containing personal information — names, behavioral patterns, health indicators, purchase history — that information is encoded as statistical patterns across the model's parameters. It is not stored in a database row that can be located and deleted. It is diffused through millions or billions of numerical weights. This creates the erasure problem: when a data subject exercises a right to deletion, there is no record to remove — only a model to retrain.

2. AI models can memorize and reproduce training data. Research has consistently demonstrated that large language models can reproduce verbatim text from their training datasets, including personal information. In 2022, a study of GPT-2 found that the model could be prompted to reproduce names, phone numbers, email addresses, and other personal data from its training corpus. The EDPB's Opinion 28/2024 explicitly addresses this: for an AI model to be considered anonymous, the likelihood of direct extraction of personal data and the likelihood of obtaining personal data through queries must both be assessed as insignificant. The EDPB sets a high threshold — most currently deployed models do not meet it.

3. Inference creates new personal data. An AI system making a hiring recommendation, a credit decision, or a medical diagnosis is not just processing the input data — it is generating a new data point about an individual that has legal significance. That output — a score, a recommendation, a classification — is itself personal data, and the individual has rights in relation to it under GDPR Article 22.

As Tiffany Li, Associate Professor of Law at the University of San Francisco School of Law, stated in MIT Technology Review: "Even if someone finds out their data was used in a training dataset and exercises their right to deletion, technically the law is unclear about what that means. If the organization only deletes data from the training datasets — but does not delete or retrain the already trained model — then the harm will nonetheless be done."

Phase 1: Data Collection and Training

What Happens Technically

The training phase begins with data assembly. A training dataset is built from sources that may include: web-scraped text and images, licensed third-party datasets, internal enterprise records, user-generated content, sensor data, transaction logs, and API feeds. The model is then exposed to this data iteratively, adjusting its internal parameters to minimize prediction error across the training examples.

For most commercially deployed AI systems — recommendation engines, language models, classification systems — this training corpus contains personal data. Web-scraped text includes names, email addresses, and biographical details. Internal records contain customer histories and employee data. User-generated content includes self-disclosed personal information. The presence of personal data in training data is the rule, not the exception.

CNIL, France's data protection authority, distinguishes two phases clearly: the learning phase (creating and training the AI system, resulting in a model that encodes the system's learnings from training data) and the production phase (actual deployment and use of the developed model). These phases have different objectives and, from a data protection perspective, should be treated as distinct processing activities — each requiring its own documented legal basis and purpose.

The Legal Basis Question for Training Data

GDPR Article 6 requires a valid legal basis for every processing activity. For AI training on personal data, four bases are relevant:

Phase 2: Inference — Personal Data in Production

What Happens Technically

When a deployed AI system receives a user input — a query, a document, a data record — it processes that input through its trained parameters to generate an output. This is inference. The input is preprocessed into a numerical representation the model can operate on; the model applies its learned weights to that representation; and an output — a prediction, a classification, a generated response, a recommendation — is produced.

At each step of this process, personal data may be present:

• In the input: a user submits their name and symptoms to a health AI; an HR system feeds a candidate's CV to a screening model; a customer submits a support query containing their account details
• In intermediate representations: the input is encoded into embeddings — numerical vectors that represent semantic meaning. Research published in 2020 demonstrated that embedding models leak significant personal information: attacks on popular sentence embedding models recovered 50–70% of input words (F1 scores of 0.5–0.7), including sensitive attributes such as authorship and potentially health-related information
• In the output: the model's prediction may itself be personal data — a credit score, a risk classification, a recommended action

Key term: Embedding — a numerical vector representation of input data (text, image, or other content) that captures semantic meaning in a format AI models can process. Where an embedding can be linked to an identified or identifiable individual — directly or because it encodes information that enables inference about a person — it constitutes personal data under GDPR Article 4(1) and must be governed accordingly.

Privacy Attack Vectors: Where Personal Data Escapes AI Systems

Understanding how AI systems process personal data requires understanding the attack surfaces through which that data can be extracted or inferred — because regulators now expect organizations to assess these risks proactively.

Membership inference attacks — an attacker queries the model with data about a specific individual to determine whether that individual's data was used in training. Research consistently shows that models trained without differential privacy protections are vulnerable to membership inference at rates significantly above random chance. The EDPB has incorporated this attack vector into its anonymization assessment framework.

Model inversion attacks — an attacker reconstructs training data from model outputs and gradients. In a landmark 2015 study, researchers demonstrated that model inversion could reconstruct recognizable facial images from a face recognition system's confidence scores. In medical AI contexts, inversion attacks have been used to reconstruct approximate patient records from diagnostic models.

Data extraction from generative models — large language models can reproduce verbatim training data, including personal information, when prompted in specific ways. A 2023 Google DeepMind study extracted over 10,000 examples of memorized training data from ChatGPT, including names, contact details, and passages from private documents.

These attack vectors require specific technical and organizational measures: differential privacy during training, output filtering for personal data patterns, prompt injection detection, and rate limiting on queries that could be used for systematic data extraction. The CNIL's AI guidelines explicitly state that a model facing a successful privacy attack that exposes personal data constitutes a data breach requiring notification.

Data Subject Rights and AI Systems

GDPR's data subject rights — access, erasure, rectification, portability, restriction, objection, and rights related to automated decision-making — apply to AI systems processing personal data. Each right creates specific operational challenges in the AI context.

Right of Access (Article 15)

Individuals can request confirmation that their data is being processed and a copy of that data. For AI systems, this raises the question: does the training corpus count? The EDPB's position is that if personal data was used in training, it was processed — and the individual's right of access applies to that processing activity, even if the data is no longer stored in a retrievable format.

Right to Erasure (Article 17)

The right to erasure in AI systems is one of the most technically demanding compliance obligations in the current landscape. Options include:

1. Retraining without the subject's data — viable for smaller models; operationally impractical for large foundation models
2. Machine unlearning — algorithmic techniques that attempt to reduce a specific individual's influence on model weights without full retraining. Still experimental; no vendor can currently guarantee effective unlearning for large-scale models
3. Anonymization demonstration — showing, via the EDPB's two-part test, that the likelihood of identification from the model is insignificant. Difficult to demonstrate convincingly for most current models

The Regulatory Framework: What Applies, When, and to Whom

GDPR and the EU AI Act: Simultaneous Obligations

As DLA Piper noted in their 2025 GDPR enforcement survey: "GDPR is now being used as the primary enforcement tool for AI regulation as AI-specific rules are still being phased in." This means AI enforcement is happening now — not waiting for the EU AI Act's full implementation.

The EU AI Act's high-risk obligations (Annex III systems) impose a second, parallel compliance layer from August 2, 2026: technical documentation, risk management systems, human oversight, conformity assessment, and post-market monitoring — all running alongside GDPR, not replacing it.

Organizations deploying high-risk AI systems that process personal data must satisfy both frameworks simultaneously. A hiring AI must comply with GDPR Article 22 automated decision-making rights and EU AI Act Annex III human oversight requirements. Treating these as separate workstreams creates gaps. The practical approach is a unified compliance infrastructure addressing both.

Enforcement Precedents

The pattern is consistent: regulators are not waiting for AI-specific laws to enforce against AI processing. GDPR's existing framework — lawful basis, data minimization, purpose limitation, transparency, data subject rights — is being applied directly to AI systems now.

U.S. State Law Landscape

In 2025, U.S. states introduced 1,208 AI-related bills and enacted 145 of them (LinkedIn AI Regulation Analysis, February 2026). The most operationally significant:

• Colorado AI Act (SB 24-205): Effective June 30, 2026. Developers and deployers of high-risk AI making consequential decisions in employment, housing, and lending must implement risk management programs, conduct impact assessments, and provide consumer disclosures
• California CPPA ADMT regulations: Automated decision-making technology regulations requiring pre-use notice and opt-out rights for profiling in significant decisions

Texas AI law (HB 149): Effective January 1, 2026. Requires developers of high-risk AI

Technical Controls for Privacy-Compliant AI

Privacy by design in AI systems means embedding data protection controls into the architecture before training begins, not retrofitting them after a regulatory inquiry. GDPR Article 25 requires it. The EU AI Act requires it. The technical controls that make it operational:

Differential privacy — a mathematical technique that adds calibrated noise to training data or model outputs such that any individual's contribution to the training dataset cannot be statistically distinguished. It provides a provable privacy guarantee that supports the anonymization assessments the EDPB requires.

Data minimization in feature engineering — selecting only the features necessary for the model's stated purpose before training, and eliminating fields that could enable re-identification or introduce bias. The EU AI Act for CTOs guidance from Secure Privacy notes: "For AI systems, this means training data minimisation, feature selection that avoids processing personal data where statistical patterns can be learned without it, and inference-time controls that limit what data the model is exposed to."

Federated learning — training a model across distributed devices or data silos without centralizing raw personal data. Each device trains locally on its own data; only model updates (gradients), not raw data, are shared. Reduces exposure from a single large training corpus, though does not eliminate all privacy risks.

Data anonymization and pseudonymization — removing or replacing direct identifiers before training. Pseudonymized data is still personal data under GDPR (because re-identification is possible with additional information); genuinely anonymized data falls outside GDPR entirely. The EDPB's high threshold for anonymization means most "anonymized" training datasets remain personal data.

Generative AI: Personal Data at the Prompt Layer

Generative AI — large language models, image generators, and multimodal systems — introduces a personal data processing layer that did not exist in traditional predictive AI: the user prompt.

Every prompt submitted to an LLM may contain personal data. A user drafting an HR letter includes employee names and performance details. A user asking for contract assistance pastes client information. A developer submitting code for review includes API keys and internal system identifiers.

AI chatbot data governance and RAG systems introduce additional complexity: Retrieval-Augmented Generation systems that pull from internal document stores before generating responses must govern the retrieval sources with the same controls as the model's training data. Germany's Datenschutzkonferenz issued guidance in October 2025 specifically on RAG systems, finding that unrestricted retrieval from external databases containing personal data creates the highest GDPR exposure category.

Operational requirements for generative AI personal data processing:

• User disclosure: individuals must be informed when their inputs will be processed and whether those inputs may be used to improve the model
• Input filtering: systems should flag and handle inputs containing high-risk personal data categories differently from general-purpose text
• Vendor DPA: a Data Processing Agreement must be executed with every LLM API provider before personal data flows through the integration

How Secure Privacy Supports Privacy-Compliant AI

Secure Privacy provides the infrastructure that connects AI governance obligations to operational privacy management — covering the shared compliance layer that GDPR and the EU AI Act both require.

AI system inventory and DPIA workflows: Structured Data Protection Impact Assessment processes for AI systems processing personal data, satisfying both GDPR Article 35 and EU AI Act Article 27 (FRIA) requirements. Built-in DPO review routing, sign-off tracking, and version-controlled documentation.

Training data provenance documentation: Records of training dataset sources, legal bases, consent provenance, and data subject coverage — the documentation that satisfies both GDPR Article 30 (Records of Processing Activities) and EU AI Act technical documentation requirements in a single evidence store.

Consent management for AI systems: For AI systems that make decisions based on user behavioral data, Secure Privacy's AI Governance platform ensures that consent signals flow into the AI system's processing logic in real time. Users who have not consented to profiling cannot have their data used in AI personalization — enforced at the infrastructure level, not just declared in a privacy policy.

Data subject request handling for AI processing: The DSAR module handles requests involving AI-processed personal data — including access requests about AI-generated inferences and deletion requests that require documented responses about training data handling — with audit-ready records for regulatory inquiry.

Frequently Asked Questions

Is training data always "personal data" under GDPR?

Not automatically — but in practice, yes for most commercially relevant training datasets. GDPR's definition of personal data is broad: any information relating to an identified or identifiable natural person. Web-scraped text, behavioral logs, enterprise records, and user-generated content all typically contain personal data. Data that has been genuinely anonymized — meeting the EDPB's high threshold of insignificant re-identification likelihood — falls outside GDPR. Pseudonymized data does not. Organizations should not assume their training data is anonymous without conducting the specific assessment the EDPB describes in Opinion 28/2024.

Do we need a DPIA for every AI system we deploy?

A DPIA is mandatory under GDPR Article 35 for any processing "likely to result in a high risk to the rights and freedoms of natural persons." The criteria that trigger this threshold, per EDPB guidelines, include systematic processing of sensitive data, large-scale processing, automated decision-making with significant effects, and monitoring of publicly accessible areas. Most AI systems processing personal data at scale meet at least one of these criteria. The practical guidance from supervisory authorities is to conduct a DPIA for any AI system that makes or contributes to decisions affecting individuals, uses biometric or special category data, or processes behavioral data at scale.

Can we use publicly available data to train AI models without GDPR issues?

Public availability does not equal lawful processing. The France CNIL's fine against Clearview AI was specifically for web-scraping publicly available facial images for AI training without a valid lawful basis. The EDPB and multiple supervisory authorities have confirmed that "publicly available" is not a legal basis under GDPR. A valid Article 6 basis must be established independently of whether the data was publicly accessible at the time of collection.

Summary: Personal Data Processing Across the AI Lifecycle

Secure Privacy is a unified consent management and privacy governance platform supporting GDPR, EU AI Act, and 65+ privacy regulations. Its AI governance capabilities cover DPIA workflows, training data documentation, consent management for AI-driven personalization, and DSAR handling for AI-processed data. Start free or contact the team to discuss privacy-compliant AI for your organization.