Hosting an LLM in Europe with Ollama and a GPU: the 2026 practical guide

Running a language model in production without sending a single byte of data outside the European Union has become a concrete requirement for many technical teams. Regulatory pressure, combined with the extraterritorial reach of the US Cloud Act, is forcing a rethink of inference architectures built on APIs hosted across the Atlantic.

The European ecosystem has caught up. Scaleway now offers H100 SXM instances in its PAR2 region in Paris, and Ollama has established itself as a simple runtime for serving open-weight models such as Llama, Mistral or Qwen. This article explains how to combine the two to host an LLM in Europe in conditions that genuinely hold up in production.

Why host your LLM in Europe

A regulatory framework on the move

Two regulations now shape AI architecture decisions. The GDPR governs transfers of personal data outside the EU, and the AI Act adds a layer of obligations around documentation, transparency and risk management.

The AI Act timeline is still shifting. The obligations for general-purpose AI (GPAI) models have applied since 2 August 2025. The transparency obligations set out in Article 50 take effect in August 2026. However, the political agreement on the Digital Omnibus reached in May 2026 pushes back the obligations for high-risk systems (Annex III) to December 2027. The regulatory direction is clear even if some deadlines move: compliance by default is becoming the cheapest option in the long run.

On top of that comes the Cloud Act of 2018: it allows US authorities to request access to data held by a provider under US jurisdiction, including when that data physically resides in Europe. Choosing a European provider to host an LLM is therefore not just a matter of latency or cost: it is a legal lever over your exposure to extraterritorial law.

A mature ecosystem of open models

Open-weight models now cover most enterprise use cases: document summarisation, code generation, classification, structured extraction. Mistral, Llama, Qwen and DeepSeek are distributed under licences that allow self-hosted deployment.

That makes a hybrid approach possible: a proprietary API for non-sensitive tasks, and a self-hosted LLM in Europe for anything touching customer, HR or strategic data.

Ollama: the first runtime to consider

Ollama is an open source runtime that simplifies serving models in the GGUF format (which comes from llama.cpp). It handles downloading the weights and loading them onto the GPU, and exposes an HTTP API compatible with the OpenAI Chat Completions protocol.

What Ollama does well

• Fast start-up: one ollama run mistral command and the model is available.
• Automatic GPU detection: Ollama detects CUDA-compatible NVIDIA cards and attaches to them with no configuration.
• Native multi-GPU: Ollama automatically splits a model across several cards when the VRAM of a single one is not enough.
• Standardised API: OpenAI-compatible, which makes migrating existing code easier.
• Built-in quantisation: GGUF Q4_K_M or Q5_K_M models cut VRAM requirements sharply with limited quality loss.

Its limits in production

Ollama is not optimised to serve dozens of parallel requests at very high throughput — runtimes such as vLLM or TGI remain more efficient on that front. Ollama shines on moderate workloads: internal assistants, batch pipelines, business copilots. For demanding SLAs, you run several Ollama replicas behind a load balancer.

Choosing the right GPU on Scaleway Paris

Scaleway has expanded its GPU catalogue in the PAR2 region and remains one of the European references for LLM inference. The choice of instance depends mainly on the size of the model you are targeting and the level of quantisation you can accept. As prices are adjusted regularly (an upward revision took place in early June 2026), we refer you to the official pricing page for current figures.

L4: for small models and development

The L4 (24 GB of VRAM) is enough to run a Mistral 7B or a Llama 3 8B quantised to Q4. It is the entry-level instance for validating a POC or serving a low-traffic internal assistant.

L40S: the best value for production inference

The L40S is designed for generative inference. With 48 GB of VRAM per GPU, it can load a Llama 3 70B in Q4 across two cards, or a mid-sized model on a single GPU. It is the most relevant instance for the majority of production Ollama deployments. Scaleway offers it in configurations of 1, 2, 4 or 8 GPUs.

H100: very large models and fine-tuning

The H100 PCIe and H100 SXM offer 80 GB of VRAM per card. They are oversized for standard Ollama inference but become relevant for fine-tuning or for serving unquantised models at low latency. The arrival of H100 SXM in PAR2, hosted in the DC5 datacentre (powered by renewable energy certified by Guarantees of Origin), opens the door to European workloads that used to be the preserve of the US hyperscalers.

Summary table

• L4 (24 GB): Mistral 7B, Llama 3 8B Q4. POC and low traffic.
• L40S (48 GB): Llama 3 70B Q4 on 2 GPUs, or mid-sized models on a single GPU. Standard production.
• H100 PCIe (80 GB): unquantised Llama 70B, 7B-13B fine-tuning.
• H100 SXM (80 GB): very high throughput, heavy fine-tuning, multi-node.

OVH and Outscale are credible alternatives on the sovereignty side (HDS and SecNumCloud certifications), but with more limited GPU coverage for now. The choice depends as much on the surrounding ecosystem (object storage, private networking, certifications) as on the GPU price alone.

Quantisation: finding the right trade-off

GGUF quantisation compresses the model weights so they fit into more modest VRAM budgets. Orders of magnitude for a Llama 70B:

• Native FP16: around 140 GB of VRAM — impossible on a single consumer GPU.
• Q5_K_M: around 50 GB — fits on one H100 80 GB.
• Q4_K_M: around 40 GB — fits on two L40S 48 GB in multi-GPU.

The quality degradation caused by Q4_K_M or Q5_K_M depends on the model and the task. For conversational use and extraction tasks, the loss is often acceptable. For multi-step reasoning or complex code generation, validate your use case with a mini-benchmark on your real prompts before locking in the architecture.

Deploying Ollama on Fransys: the actual workflow

Fransys is a platform that orchestrates managed Kubernetes on European infrastructure (Scaleway, OVH, Hetzner, Outscale) through a visual interface. In practice, you build your infrastructure by dragging and dropping blocks onto a canvas — no YAML, no kubectl. To host Ollama on a Scaleway GPU, the process comes down to a few steps.

Step 1: create a cluster with GPU

When you create a cluster, the Fransys wizard asks whether you need GPUs. Answer Yes and the interactive map automatically filters the compatible regions. Scaleway currently exposes GPU nodes in Paris and Warsaw. Select Paris (PAR2), then add a node pool with a GPU node type — the options appear prefixed with gpu along with the card's specifications. AutoPilot can also handle sizing in budget mode.

Step 2: add a Docker block for Ollama

Drag a Docker block from the sidebar onto the canvas. At first, the interface shows only the essential settings: source, image, port. Choose:

• Source: External (the image comes from a public registry)
• Image: ollama/ollama
• Port: 11434

As soon as the image is filled in, the interface expands with the Resources, Storage and Environment variables tabs and the Ingress section. That is the principle behind Fransys blocks: configuration is revealed progressively, showing only what makes sense at each step.

Step 3: attach a persistent volume for the weights

A Llama 70B Q4 weighs around 40 GB. Without a persistent volume, Ollama would re-download the models on every redeployment. In the block's Storage tab, you define:

• A size suited to the models you plan to host (100 to 200 GB is a good starting point if you want to try several)
• The mount point: /root/.ollama

The volume survives redeployments, restarts and application updates. Note that automatic volume backups are still under development at Fransys — for models this is a non-issue (they can be downloaded again), but keep it in mind if you store other data on the same volume.

Step 4: configure the Ollama variables

In the Environment variables tab, a few settings make a real difference in production:

• OLLAMA_HOST=0.0.0.0:11434 so that the service listens on the container's exposed port.
• OLLAMA_KEEP_ALIVE=24h (or even -1) to keep the model warm in VRAM. The 5-minute default is too short for production: you pay the loading time at every traffic spike.
• OLLAMA_NUM_PARALLEL to tune the number of requests processed in parallel according to the available VRAM.
• OLLAMA_MAX_LOADED_MODELS if you serve several models from the same instance.

Step 5: exposure and SSL

On Fransys, exposure is automatic. As soon as the block is deployed, a *.fransysapp.com URL is assigned, with a Let's Encrypt SSL certificate generated and renewed automatically. For a custom domain (e.g. llm.yourcompany.com), you add it in the Ingress section, create a CNAME or an A record on the DNS side, and Fransys takes care of the certificate. The load balancer and health checks are handled at cluster level, so there is nothing to configure on the traffic side.

Step 6: handle authentication

To be upfront about an important point: Ollama ships with no native authentication. Exposing your instance to the Internet without access control is out of the question — you would be handing your GPU to anyone. Several approaches are possible:

• Application sidecar: a second Docker block (an authentication middleware that you maintain) in front of Ollama, which validates an API key or a JWT before proxying.
• Network auth: restrict access to a corporate VPN or a Zero Trust solution (Tailscale, Cloudflare Access) pointing at the Fransys URL.
• Application-level auth: if the instance is only called by your backend, keep the URL non-public and route calls solely from your application, itself hosted on the same Fransys cluster (internal communication).

The simplest option for internal use remains network auth via Zero Trust. For a product aimed at external users, the application sidecar is more robust.

Going to production: scaling and observability

Once the Ollama block is in place, running it at scale relies on the platform's native building blocks.

Scaling

Ollama can be scaled horizontally by increasing the number of replicas and GPU nodes, with the Fransys load balancer distributing traffic automatically. Be careful: each replica consumes its own VRAM, and therefore its own dedicated GPU. GPU scaling is expensive — it is often better to size generously from the start than to multiply small instances.

Observability

The block's Metrics and Logs tabs report CPU/RAM consumption and Ollama logs in real time. For detailed GPU monitoring (utilisation, memory, temperature), you can log the output of nvidia-smi as a periodic task or push the metrics to an external Prometheus.

Costs

A GPU rented by the hour runs 24/7: at low volume, a managed European API (Mistral, Albert) is often still cheaper. The tipping point depends on your inference volume and the sensitivity of your data. Do the maths before locking in the architecture.

Limits and trade-offs to be aware of

Self-hosting an LLM is not magic. A few points deserve honesty upfront:

• Fixed vs variable cost: a GPU instance runs continuously, even at night. Break-even against a managed API does not come immediately.
• Quality of open models: on very complex tasks (long reasoning, expert-level code generation), proprietary models keep a lead over 70B open-weight models.
• GPU availability: H100s can be temporarily unavailable during demand peaks. Plan a fallback strategy.
• Authentication: Ollama offers none natively. It is up to you to add an access control layer.
• Updates: GPU drivers, Ollama versions, models. On Fransys, node management is handled for you; keeping Ollama itself up to date remains your responsibility.

The pragmatic approach is often a mix: a self-hosted LLM for sensitive data and predictable volumes, and a European API to absorb peaks or handle non-confidential requests.

Conclusion: a European stack that is now realistic

Combining Ollama with a Scaleway GPU in the PAR2 region now makes it possible to host an LLM in Europe in technically credible conditions, at a controlled cost, and with minimal legal exposure to the Cloud Act. The maturity of the open-weight ecosystem makes this option viable for the majority of internal enterprise use cases.

The key decision is less about technology than about approach: build a DevOps team to run raw Kubernetes, or rely on a platform that abstracts that complexity away. Fransys is the second approach, with deployment through visual blocks on European GPUs and a strong focus on the sovereignty of the underlying infrastructure.

Try Fransys for free to deploy your Ollama instance on a European GPU.