"Just self-host it" has become a reflex answer to almost every awkward question about AI and data. It is sometimes the right call, and sometimes an expensive way to end up with a worse model and a new operational liability. The honest version of the decision is narrower than the enthusiasm suggests: open-weight models — models whose parameters you can download and run yourself — win in a handful of specific situations, and lose fairly clearly outside them. This is an attempt to draw that line without hand-waving.
When local genuinely wins
Four conditions, roughly in order of how often they actually decide the matter.
Data residency and control. If you are processing personal or sensitive data, the appeal of self-hosting is not mystical — it is that the data never leaves infrastructure you control. Under the GDPR, sending personal data to a hosted API means a processor relationship, a data processing agreement, and, for many providers, a transfer outside the EEA that needs a lawful basis and a transfer mechanism. Running the model on a VPS in a Frankfurt or Helsinki datacentre collapses most of that surface area. It does not make you compliant by itself, but it removes the hardest questions — third-country transfer, sub-processor sprawl, "where exactly did the prompt go" — from the table.
Offline and air-gapped operation. Some environments cannot call an external API as a matter of policy or physics: classified networks, industrial control segments, a factory line, a ship, a clinic with an unreliable link. An open-weight model is the only category that runs with the network cable pulled. If "must work with no internet" is a hard requirement, the decision is already made.
Cost at sustained volume. Hosted APIs are priced per token and are extraordinarily cheap to start with. The economics invert only at high, steady utilisation. A GPU — rented or owned — costs the same whether it runs at two percent or ninety percent load, so the break-even is really a question of how full you can keep it. Bursty, unpredictable workloads favour the API. A pipeline that classifies, extracts, or summarises millions of documents on a predictable schedule is exactly the shape where a well-fed local GPU undercuts per-token billing. The mistake is assuming volume you do not yet have.
Latency and determinism. A local model on the same network removes a network round-trip and, more importantly, removes a dependency on someone else's rate limits, queue depth, and deprecation calendar. For interactive or tightly-looped workloads, predictable local latency can matter more than raw speed.
When it clearly does not
The counter-case is just as important. For frontier reasoning quality — hard multi-step problems, long-context synthesis, the top of the capability curve — the best hosted models remain ahead of what you can practically run yourself, and the gap is largest precisely where the task is hardest. Open weights have closed much of the distance on everyday tasks; they have not closed it at the frontier.
Then there is the operational reality. Self-hosting means you now own GPU capacity planning, driver and CUDA versions, an inference server, model updates, monitoring, and an on-call path when the box falls over at 3 a.m. For a small team this is a real, recurring tax. If your volume is modest, your workload spiky, or your team has no appetite for infrastructure, a hosted API is not the lazy choice — it is the correct one.
The practical realities
If you do host, three things govern whether it works.
VRAM sizing. The rough rule is parameters × bits-per-weight ÷ 8 = GB of weights, plus meaningful headroom for the context/KV cache. A model quoted in "billions of parameters" at full precision needs about two bytes per parameter; at 4-bit quantisation, roughly half a byte. That is why a mid-size model that will not fit a 24 GB card at full precision fits comfortably once quantised.
Quantisation. This is the compression that makes local hosting viable — trading a small amount of accuracy to shrink weights from 16-bit down to 8- or 4-bit. Modern 4-bit schemes lose surprisingly little on most tasks. The practical advice: use the least aggressive quantisation your hardware allows, and evaluate on your task rather than trusting a generic leaderboard, because degradation is uneven and shows up first on the reasoning you care most about.
An inference server. You do not run raw weights; you run them behind a server that handles batching, the KV cache, and an API surface. Lightweight tooling is fine for prototyping and single-user use. For real concurrent traffic, a throughput-oriented server with paged attention and continuous batching is the difference between one comfortable user and dozens. Budget the choice deliberately — it is as load-bearing as the model.
The supply-chain angle nobody budgets for
A downloaded model is a large binary artefact from the internet, and it deserves the same suspicion as any dependency. Weights have historically shipped in formats that could execute arbitrary code on load; safer serialisation formats exist and should be preferred. Beyond format, provenance matters: pull from the original publisher, pin to a specific revision or hash, and treat a random re-upload of "the same model" as untrusted. There is also a subtler risk — a model can be fine-tuned to behave normally until a trigger appears, and you cannot fully inspect weights the way you can read source. Mitigations are governance, not magic: known-good sources, integrity checks, an internal registry, and evaluation before promotion. In EU terms, note that the AI Act's general-purpose model obligations lean on providers of the model, with lighter documentation duties for genuinely open-source models and full obligations for the largest "systemic-risk" models; enforcement powers for the AI Office phase in around 2 August 2026. Self-hosting shifts you toward the deployer role, but does not erase your own accountability for how the system behaves.
What this means for you
Decide by requirement, not by ideology. If you have a hard data-residency, offline, or high-sustained-volume constraint, price out self-hosting seriously — and count the operational tax honestly, not just the GPU invoice. If you need frontier reasoning, have spiky or modest volume, or no one to carry a pager, a hosted API with a proper data processing agreement is the mature answer. Many teams land on a split: a local open-weight model for the high-volume, privacy-sensitive bulk work, and a hosted frontier model reserved for the genuinely hard requests. Whichever way it goes, treat the weights as a supply-chain dependency and the box as production infrastructure — because both of those are true. That deliberate, evidence-led sizing of what actually needs to run where is a large part of the work I take on.