Why in-house hosting is enough for models from 7B to 70B
Most mid-market workloads do not require a US cloud or a 400-billion-parameter model. Open-weight models such as Llama, Mistral, Qwen or the German Teuken, in the 7B to 70B range, handle extraction, classification, summarisation and citation-grounded RAG reliably. We run them with vLLM on our own GPU servers in Germany. PagedAttention manages the KV cache efficiently, continuous batching keeps the GPUs busy, and tensor parallelism spreads larger models across multiple cards. The result is response latency and throughput that hold up in production, without paying per token.
In front of the models sits a LiteLLM gateway as a single point of access. It standardises the API, separates tenants, enforces rate limits and logs every call for your audit trail. You book either a dedicated instance or a hardened tenant with logical isolation. In both cases your data stays in Germany and is never used to train third-party models.
The second argument for in-house hosting is cost. Token-based billing becomes unpredictable as usage grows. Owned capacity turns this into planned fixed cost: you pay for GPU hours and storage, not for each individual request. At steady utilisation, total cost of ownership falls below that of a commercial API subscription.
- Data residency: storage and inference exclusively on servers in Germany, with no data flowing to US clouds.
- Hardware: various GPU systems, sized to model footprint and load profile.
- Operations: SLA, monitoring of latency and utilisation, and hardening with BSI IT-Grundschutz as the reference framework.
- Regulated sectors: an architecture built around GDPR, the EU AI Act and ISO 27001 as requirements we implement on your behalf.
- Scaling: horizontal growth by adding further GPU nodes, without changing provider.