AI Agents and the EU AI Act: Risk That Won't Sit Still

Agentic AI is here, and it’s moving fast. Most trust infrastructure isn’t ready to contain these fascinating but unhinged creatures we’re shaping, let alone govern them. The EU has been trying: the world’s first comprehensive AI regulation, the EU AI Act, is already going into effect. But it was written before LLMs took the world by storm.

Can we fit agents into this framework? Let’s find out.

What follows is a best effort exploration from a technical perspective. Involve legal experts before acting on any of this.

The EU AI Act, a Short Refresher

The EU AI Act entered into force in August 2024, with provisions rolling out in phases through 2027. It takes a risk-based approach: the higher the risk of your AI system, the stricter the obligations. Like GDPR, the reach is extraterritorial: if your AI system’s output is used in the European Union, you’re in scope.

The Act doesn't regulate AI. It regulates high-risk use cases.

— Carme Artigas

The Act sorts AI systems into four risk tiers:

Annex III lists what counts as high-risk: employment, education, credit scoring, law enforcement, critical infrastructure, access to essential services. And “putting into service” includes internal use: deploying AI for your own purposes doesn’t make you exempt.

Going back to our question: it turns out the Act doesn’t mention agents. General-purpose models (LLMs) got a last-minute chapter. Agents, which use those models to autonomously plan and act, are a layer the regulation didn’t anticipate.

GPAI, Systemic Risk, and RAG

The Act distinguishes three roles: provider, distributor, and deployer. Where you fall matters.

As mentioned, LLMs forced the EU to extend its draft. The Act now has a dedicated chapter for general-purpose AI models (Article 3(63)), split into two tiers depending on compute used during training:

• GPAI: must publish technical documentation, describe training data, and comply with copyright rules.
• GPAI with systemic risk (Article 51): models trained with 10²⁵+ FLOPs, powerful enough to pose risks across entire sectors or society. Think GPT-4, Claude, Gemini. On top of documentation: adversarial testing, incident reporting, cybersecurity protections.

Being classified as a GPAI provider has significant compliance impact. Luckily for agent builders, these obligations sit with model providers, not with you. Using Claude doesn’t make you responsible for Claude’s systemic risks. Since the threshold is based on compute, only substantial fine-tuning could make you a GPAI provider: at least one-third of the original training compute. Most enterprise fine-tuning doesn’t come close. And with LLMs, it turns out most customization happens through context engineering anyway.

RAG and prompt engineering don’t count. The July 2025 guidelines clarify that only significant modifications to the model weights trigger provider obligations. RAG, prompt engineering, orchestration, tool-calling frameworks don’t reach that bar. If you’re building an agentic platform with context engineering, you’re a distributor, not a provider.

One wrinkle: open-weight models get a partial pass on transparency obligations, but not if they cross the systemic risk threshold. Frontier open models like DeepSeek R1 and Qwen likely cross that line. If the original provider hasn’t complied with EU obligations, that risk may flow down to you as the first entity in the EU value chain. Something to consider before building on open models with no EU presence.

So at the model layer, we’re safe if we use a compliant provider and we don’t intensively fine-tune it. But what about the agent layer on top?

Agents Don’t Fit in a Box

The EU AI Act was written for traditional AI deployments: fixed pipelines with a known use case at build time. An HR screening tool? High-risk from day one. Classify it, file the conformity assessment, move on.

Some agents work the same way: a single clear goal, a known risk tier. But not all of them. What if the goal isn’t that clear? Give a general-purpose office assistant “handle my inbox” and it decides itself to draft an email (minimal risk), screen a job application (high-risk), then assess a customer complaint (potentially high-risk). Suddenly you’ve ended up in a high-risk use case. The risk tier depends on how open-ended the prompt is.

In the first case, each step has a clear scope: the AI transcribes, summarizes, and the result gets emailed with a disclaimer that it’s AI-generated. In the second, the agent doesn’t just generate: it acts on its own output. The summary becomes input for decisions the agent makes autonomously. That’s where risk compounds.

You can classify a tool at build time. You can’t classify an agent whose use case emerges at runtime. Harnesses can constrain this, but how do you constrain something you can’t anticipate?

This is what the Future Society report calls the multi-purpose problem: generic agents default to high-risk classification unless you explicitly exclude high-risk uses. The Act is permissive by design. But agents need closer attention precisely because they’re general-purpose.

The Shadow Agent Problem

It gets worse with low-code platforms. When employees build agents on tools like Power Platform or Copilot Studio, the company is still the provider. An employee builds an HR screening agent without a compliance assessment, and the company is non-compliant without knowing the system exists. This is why Article 4 (AI literacy) matters: staff need to understand what makes something high-risk. And compliance teams can’t assume they have visibility into what’s being built. Just like shadow IT before it, shadow agents will be one of the harder governance challenges to solve.

From Compliance to Infrastructure

The Act tells you what to achieve, not how. For high-risk systems: quality management (Art. 17), risk management throughout the lifecycle (Art. 9), record-keeping and traceability (Art. 12), human oversight by design (Art. 14).

None of this works without infrastructure. You need:

• Governance: who can build and deploy agents, what uses are permitted
• Audit: what did the agent do, why, with traceable decision logs
• Authorisation: what is this agent allowed to do right now, in this context

We can get creative and extend current systems like On-Behalf-Of delegation, but agents also need emerging patterns for problems like the confused delegate: an agent acting on behalf of a user, but without clear limits on what it’s allowed to do.

What’s Next

The excitement about agents is that they can do things we don’t define at build time. That’s also what makes them hard to govern: a double-edged sword. The EU AI Act is a fitting example of this tension. It gives us a risk framework, but that framework was built for systems that sit still. Agents don’t, and we don’t know what will flow through them.

Without being a legal expert, I’ve come to the conclusion that this will be a mess if not approached by design. Trusted AI agents require governance. Part of that is regulation, part of it is technical: auth, identity, scoping, audit trails, guardrails. Solutions are emerging, but deploying agents without them is in most cases inappropriate.

This is what I work on. Follow along, build with me, or get in touch if you need more insights or help.

For further reading, Ahead of the Curve: Governing AI Agents Under the EU AI Act by The Future Society dives deeper into agents under the Act.

Resources