Custom LLMs: Fine-Tuning Open-Weight Models With Reinforcement Learning
Nov 7, 2026
A general-purpose model like ChatGPT or Claude gets most business tasks mostly right. The trouble is the last stretch, where your document formats, your product codes and the judgement calls your best people make without thinking all matter. Better prompts only get you so far. Past that point, the fix may be giving the model your documents, or teaching the model itself your task, which is what a custom LLM is for.
This guide explains when a custom LLM is worth building, how LLM fine-tuning works, what reinforcement learning adds on top of it, and how we train open-weight models that you end up keeping. New to how these models work? Start with our interactive guide to LLMs.
RAG vs fine-tuning: do you need a custom LLM?
Most teams don’t, at first. Think of it as a ladder where each rung usually costs more than the one below. Climb only when the rung below stops improving results, and know that some projects combine rungs. The second rung, retrieval-augmented generation (RAG), lets the model look up your documents when it answers, instead of learning them.
Approach | What changes | Effort | Good for |
|---|---|---|---|
Prompting | The instructions | Low | Prototypes, low volume |
Retrieval (RAG) | What the model can look up | Medium | Questions about your documents |
Fine-tuning | The model itself, by imitation | Higher | Fixed formats, house style, classification |
Reinforcement learning | The model itself, by trial and reward | Highest | Tasks with an answer you can check |
LLM fine-tuning vs reinforcement learning
Supervised fine-tuning shows the model hundreds or thousands of examples of an input and the ideal output, and it learns to imitate them. It works well, but results depend heavily on the quality and coverage of those examples, which someone has to collect and check.
Reinforcement learning (RL) works differently. The model attempts a task many times, a program scores each attempt, and training nudges the model towards whatever scored well. Nobody has to write the perfect answer. You only have to recognise one.
That makes RL a strong fit wherever a result can be checked automatically:
invoice extraction where the line items must add up to the total
a SQL query that has to run on your warehouse and return the right rows
a support-ticket label that can be compared with the one your team chose
a structured output that must pass your schema and business rules
What is GRPO?
The method we use most is GRPO (group relative policy optimisation), introduced by DeepSeek in 2024 and made famous by their R1 model in 2025. It scores a group of answers to the same question and rewards the ones that beat the group average. It does without the separate “critic” model that older methods such as PPO train alongside, which cuts memory needs. That helps make RL practical for a mid-sized company, not just an AI lab.
Why build on an open-weight (open-source) LLM
We build on open-weight models, whose trained files are published for anyone to run, such as Qwen, Llama, Gemma and Mistral, the last one made in France. Three reasons:
You keep the result. The trained model is a set of files, called weights, that you run wherever you like. No vendor can change, reprice or retire it, though the base model’s licence still applies.
Small can beat big. A model with 7 to 30 billion parameters, trained on one task, can match a frontier model, one of the largest hosted models such as GPT or Claude, on that task at a fraction of the cost per request.
It runs where you want. On your own servers, in an EU cloud or on a laptop. Our guide to sovereign, self-hosted AI covers how.
How LLM fine-tuning works, step by step
Define the task and the score. We pick one workflow and build a test set from your real data, usually a few hundred examples with known good answers. If those answers don’t exist yet, our data labelling teams can create them. If we can’t define good, we stop here.
Measure the baseline. We compare a frontier model and a few open models, with well-written prompts, on development examples. Sometimes that’s already enough, and we’ll tell you.
Fine-tune. We teach a base model your format and vocabulary with LoRA, a technique that trains a small add-on to the model instead of the whole thing.
Reinforce, if needed. If fine-tuning still falls short and answers can be scored reliably, we write reward functions that encode your rules and train with GRPO. A separate final test set stays out of model selection, prompts and reward design until the final comparison with the baseline.
Check for side effects. We re-run your edge cases and general benchmarks, to make sure the model didn’t get better at one thing by forgetting another.
Host and monitor. We put the model live on your servers or on EU infrastructure we run for you, behind your access controls, with logging so you can see how it does on real traffic.
What can go wrong
A model optimises exactly what you measure, not what you meant. Reward it for valid JSON and it may learn to return empty JSON. This is called reward hacking. We reduce the risk with several independent checks, held-out examples and people reading samples during training, and we keep monitoring after launch.
A custom model also needs upkeep. When a newer base model appears, we compare it with yours on the same test set before deciding whether retraining is worth it. And training data has to be clean and representative, which is often more work than the training itself. If your data isn’t ready, data engineering comes first.
We can host your custom LLM
A trained model is only useful once it’s running somewhere your applications can reach it. You don’t need GPUs or an ML team in-house for that. We offer three set-ups:
Option | Where it runs | Who looks after it |
|---|---|---|
Managed by Datazzle | GPUs with an EU cloud provider, dedicated to your deployment where agreed | We handle updates and monitoring, with capacity and retraining reviews agreed in the support plan |
On your infrastructure | Your own servers or your company’s cloud account | We install it and can keep running it, or hand over to your IT team |
Handover only | Wherever you choose | Your team, with our documentation and the test set |
When we run the deployment, we set up access controls with you, so you decide which people and applications can use it. For a handover, we document the controls your team needs. Our guide to sovereign, self-hosted AI explains how that works in detail.
What a custom LLM costs
Most custom LLM projects, from first test set to a model running in production, land between €50,000 and €150,000 excluding VAT. Where yours falls depends mostly on how clean your data is and how hard the answer is to check. Plan for four months from first call to production. We quote a fixed scope after a first call, and state which hosting and support costs continue after launch. Hosting is a monthly fee, mostly GPU rental plus support, and our sovereign AI guide works through what a rented GPU costs.
The tools we use
Step | Tools |
|---|---|
Base models | Qwen, Llama, Gemma, Mistral |
Fine-tuning | Hugging Face TRL, Unsloth, LoRA and QLoRA |
Reinforcement learning | TRL’s GRPO trainer, reward functions written in Python |
Evaluation | Held-out test sets, task-specific scorers, a model as judge for open-ended text |
Serving | vLLM on GPU servers, llama.cpp on small devices |
What you get at the end
the trained model files (the weights), stored where you choose
the test set and reward functions, to evaluate future models and support retraining
a report comparing your model with the baseline, task by task
a running deployment, hosted by us or on your infrastructure, or a handover to your team
Frequently asked questions
How much data do we need?
For fine-tuning, a few hundred to a few thousand good examples. For reinforcement learning, you need realistic inputs and a way to check answers, which can mean far fewer hand-labelled examples. We size this before any training starts, and if you don’t have labelled examples yet, we can produce them.
Will our data train someone else’s model?
No. Before starting, we agree in writing where training runs and how your data and the trained weights are handled. Training happens on infrastructure you control or in a dedicated environment.
Is a small custom model as good as GPT or Claude?
On the task it was trained for, it can be. On everything else, no. A custom model can cost less for one narrow task at enough volume, and keeps data private when it runs on infrastructure you control. A hosted model stays more versatile. We compare both before recommending training.
Can you host the model for us?
Yes. We can run it on GPUs with an EU cloud provider, dedicated to you where agreed, and look after updates, monitoring and capacity, or install it on your own servers. You keep the weights either way, subject to the base model’s licence. Moving the service later is possible, with some migration work.
Do you offer LLM fine-tuning as a service?
Yes. We can run the whole project, from the test set through fine-tuning and reinforcement learning to hosting, or any part of it alongside your own team.
Can we start without committing to training?
Yes. Building a test set from your data and measuring how existing models score on it is useful on its own, and tells you whether a custom model is worth building at all.
Have a task where a general model keeps almost getting it right? Tell us about it. We’ll tell you which rung of the ladder it needs, the likely cost and where it could run.
CONTACT
