Sovereign AI: Self-Hosted LLMs With Granular Access Control
Nov 7, 2026
Every prompt sent to a hosted AI service leaves your company. A pasted-in contract, a customer complaint or a salary spreadsheet all travel to another company’s servers, under that company’s terms. For plenty of tasks that’s fine. For a hospital, a law firm, a public body or a bank, it’s often why AI projects never leave the pilot stage.
Open-weight models now perform well on many business tasks, which opens up another option: run the model yourself. This guide covers what sovereign AI means in practice, where to run a self-hosted LLM, how AI access control works, and when self-hosting actually saves money.
What is sovereign AI?
Sovereign AI means keeping control over where your AI runs, who operates it and who can access its data, instead of leaving that to a vendor. It’s a vague term, so we break it into four questions:
Question | Hosted AI service | Self-hosted model |
|---|---|---|
Where is the data processed? | In the provider’s data centres | On your servers, or in a cloud region you choose |
Which law applies? | Often includes non-EU law, such as the US CLOUD Act | EU law, plus whatever applies to your provider’s owners and subcontractors |
Who can change the model? | The provider, at any time | Only you |
Who can read the logs? | You and the provider | Only people you authorise |
Hosting with an EU provider doesn’t make foreign law disappear by itself, so ownership, subcontractors and support access are worth checking. Self-hosting also reduces your reliance on outside AI providers and lets you decide where prompts are processed, which makes GDPR work simpler. You still assess your hosting, support and subprocessors as usual.
On-premise, EU cloud or private cloud: where to run a self-hosted LLM
Option | Data stays | Best for | Trade-off |
|---|---|---|---|
Your own GPU server | In your building | Strict confidentiality, steady load | Upfront hardware, someone to run it |
EU cloud provider, such as OVHcloud or Scaleway | With an EU company | Most organisations | Monthly rental |
Private deployment in Azure, AWS or Google Cloud | In an EU region, under the provider’s terms | Teams already on that cloud | Weaker on the “which law” question |
Laptops and edge devices | On the device | Offline work, field staff | Small models only |
AI access control: who can use which model and tool
Running a model is the easy part. The harder question is what it’s allowed to do. Modern AI assistants don’t just answer questions. They call tools: search the CRM, read a file share, draft an invoice, send an email. Each tool is a door into one of your systems, and each needs its own lock.
We put an LLM gateway between your people and the model. Every request passes through it, so the rules live in one place instead of inside each application.
Through that gateway, access can be as fine-grained as you need:
Per model. Legal gets the large model, everyone else the fast one.
Per tool. Sales can read the CRM through the assistant. Only account managers can update it. Interns’ assistants can’t call web-browsing tools.
Per record. The HR assistant answers policy questions for everyone, but only shows salary data to HR, using the permissions your systems already have.
Per action. Actions you choose, such as writing, sending or paying, wait for a person to approve them.
Per budget. Each team gets a usage allowance and a cap on simultaneous requests, so one runaway script can’t hog the GPUs.
Every call logged. Who used which model and which tools ran, stored on your infrastructure for audits. Full prompts and answers are kept only where justified, with restricted access and a retention period.
Here’s what a simple policy can look like for a mid-sized company:
Tool | Everyone | Finance | HR | IT admin |
|---|---|---|---|---|
Internal wiki | Read | Read | Read | Read |
CRM | No | Read | No | Read and write |
Accounting | No | Write, with approval | No | Read |
HR records | Own record | No | Read and write | No |
External email | No | With approval | With approval | Yes |
MCP security and prompt injection
Tools are connected through the Model Context Protocol (MCP), an open standard for plugging AI into business systems. MCP doesn’t enforce permissions by itself, so we set up each tool to check who is asking and apply that person’s rights in the underlying system, and we test it. This matters because of prompt injection, where a malicious email or document tries to instruct the model to leak data or take an action. Using each person’s own rights limits what an attack can reach. Approvals, restricted destinations such as no emails outside the company, and monitoring reduce the remaining risk. They don’t eliminate prompt injection, so we test what an attacker could still do within each user’s permissions.
Self-hosted LLM vs API: the cost question
Hosted AI services charge per token, roughly per word read and written. Self-hosting costs per hour of GPU, the specialised processor that runs the model, whether you use it or not. Which is cheaper depends mostly on volume, and also on which models you compare and what it costs to run the service.
An illustrative example, using public prices from late 2026. It compares a frontier model, one of the largest hosted models such as GPT or Claude, with an open model you run yourself:
Cost item | Hosted frontier model | Self-hosted open model |
|---|---|---|
Workload | 500 million tokens in, 100 million out, per month | Same |
Price basis | About $5 per million tokens in, $25 per million out | One rented H100 GPU at about $3 an hour |
Monthly cost | About $5,000, rising with usage | About $2,190 in GPU rental, before staff time |
That works out at about 230 tokens a second on average, which one H100 can plausibly serve with a mid-sized model. Peaks, response times and spare capacity for failover still need a benchmark on your real traffic before you commit. The comparison also leaves out the time someone spends running the server. At low volume the hosted service wins easily: with the same mix, 20 million tokens a month costs about $170, far less than a GPU sitting idle. Budget models priced under $1 per million tokens can also undercut self-hosting for simple tasks.
Self-hosting pays off when volume is high and steady, when the data can’t leave anyway, or when a smaller, custom-trained model can replace an expensive frontier one. GPU rental prices move with demand, in both directions, so check current rates before you budget, or ask us to run the numbers on your actual usage.
Where hosted models are still ahead
The best open models are very good, but on the hardest reasoning tasks the leading hosted models are still ahead. We usually recommend a hybrid. Sensitive and high-volume work runs locally, and the gateway sends the occasional hard, non-confidential task to a hosted model, under the same access rules and the same logs.
The tools we use
Layer | Tools |
|---|---|
Models | Qwen, Llama, Gemma, Mistral, or your own custom-trained model |
Serving | vLLM or SGLang on GPU servers, llama.cpp and Ollama on laptops |
Gateway | LiteLLM, with per-team keys, budgets and model routing |
Identity | Your existing single sign-on, such as Microsoft Entra ID, Google Workspace or Keycloak |
Tools | MCP servers that act with each user’s own permissions |
Chat interface | Open WebUI, or your own applications |
Policy and audit | Open Policy Agent, OpenTelemetry, Langfuse |
How a sovereign AI pilot works
Before the pilot we agree the success measure, the security requirements and the scope. How long it takes depends mostly on your data and the systems to connect.
Assess. Which use cases, which data, which rules. We map who should be able to do what before touching any hardware. It’s data governance applied to AI.
Pilot. One team and one or two tools, on rented EU GPUs, so nothing is bought upfront.
Harden. Single sign-on, policies, approvals, logging and monitoring, tested with deliberate misuse such as prompt injection attempts.
Roll out. More teams and tools, and a decision on whether to keep renting or buy hardware, based on the usage the pilot actually measured.
What it costs
The assessment is the first step and is priced separately. After it, we quote a fixed scope for the pilot and roll-out. Complete projects, assessment included, usually land between €75,000 and €250,000 excluding VAT, depending mostly on how many teams and systems the assistant connects to. Plan for ten weeks for a pilot with one team. Running costs after launch, mainly GPU rental and support, are quoted separately as a monthly fee.
Frequently asked questions
Do we need to buy GPUs?
No. Most organisations start on rented GPUs from an EU cloud provider. Buying hardware only makes sense once usage is steady and measured.
How do you deploy an LLM on premise?
Choose an open-weight model sized for your hardware, serve it with vLLM or SGLang on a GPU server, and put an LLM gateway with single sign-on in front of it. Then connect tools through MCP with per-user permissions, and add logging and monitoring. We usually test it with one team before a wider roll-out.
What is an LLM gateway?
A single entry point between your users or applications and the models. It checks who is calling, applies the rules for that person or team, sends the request to the right model, enforces budgets and logs the call. LiteLLM is a widely used open-source example.
Are open-weight models good enough?
For most business tasks, such as drafting, summarising, extracting, classifying and answering questions about internal documents, yes. For the hardest reasoning, a hybrid setup keeps a hosted model available for non-sensitive work. On a specific task, fine-tuning on a few hundred to a few thousand labelled examples can close much of the remaining gap.
Can our staff still use a ChatGPT-style chat?
Yes. A familiar chat interface sits on top of the gateway, so people get the same experience with your rules applied.
Does self-hosting make us compliant with the AI Act?
Not on its own. Your obligations depend on what the system does and your role in it. Self-hosting helps you keep the evidence, such as logs and model versions, in your own hands.
Want to know what a sovereign setup would cost, or how your current AI use holds up on security? Get in touch with a rough idea of your users and use cases, and we’ll run the numbers. New to the technology? Our interactive guide to how LLMs work is a good place to start.
CONTACT
