Local LLM vs. Cloud AI APIs: When Should a Business Self-Host AI?

Sooner or later, most businesses planning an AI system ask us the same question: should this run on our own hardware, or should we pay for a model through an API from a provider like OpenAI, Anthropic or Google?

There isn't one right answer. We build both: systems that call hosted models such as Claude and Gemini, and systems that run open-source models on a company's own servers. This guide covers how we decide between them, including the trade-offs that tend to get left out.

The short answer

Start with a cloud API unless you have a specific reason not to. Self-host when at least one of these is true:

• Your data can't leave your environment because of client contracts, regulation or your own risk policy.
• You have steady, high-volume work where per-token fees add up and predictable costs matter.
• The system has to work offline or inside a closed network.
• You need to control which model runs, and keep it unchanged for as long as you choose.

If none of those apply, a hosted model will usually give you better answers for less effort.

What you're choosing between

A cloud AI API. Your application sends text, documents or audio to a provider's model over the internet and pays per token, roughly per word processed. The provider runs the hardware, improves the models and handles scaling.

A self-hosted (local) LLM. You download an open-weight model, for example from the Qwen, Llama, Mistral or Gemma families, and run it on hardware you control: a server in your office, your data centre or your own private cloud account. Tools like Ollama and llama.cpp make this practical on a single machine. vLLM is a common choice when many people use the system at the same time.

Side by side

The cost question: buying tokens vs. owning hardware

This is where most of the debate happens, and where the numbers are easiest to get wrong.

With an API, you pay for what you use. Every request is billed by the tokens going in and coming out, so the bill grows in step with usage. There's nothing to pay when the system is idle. Several providers also discount repeated context (prompt caching) and work that can wait (batch processing), and API prices for a given level of quality have tended to fall over time.

With self-hosting, you pay for capacity. The main costs are:

• Hardware, bought or rented
• Power and hosting
• The time to run it: updates, monitoring, fixing failures and testing new models. This is the cost people most often leave out.

Once that's paid for, each extra request costs almost nothing. But you pay the same whether the server is busy or sitting idle.

The break-even is simple to estimate. Take the hardware cost and divide it by your monthly saving, which is your API bill minus the monthly cost of running the server yourself. The result is how many months until self-hosting pays for itself.

Suppose a document-processing workload costs $2,000 a month in API fees, a suitable server costs $12,000, and running it costs $500 a month all-in. The saving is $1,500 a month, so the server pays for itself in about 8 months. That only holds if the volume stays steady and a local model is accurate enough for the task.

Now suppose the same workload costs $200 a month through an API. Running the server costs more than the API bill, so it never pays back. At that volume, the API wins on cost alone.

Small API bills are also easy to cut further. On one of our voice agents for product support, sending each step of a call to the cheapest model that handles it well brought the average call to about $0.27.

When a cloud API is the better choice

This surprises some clients: for most first AI projects, we recommend a hosted model.

• You need the best answers on hard problems. Multi-step reasoning, long documents and complex tool use are where the largest hosted models still lead.
• You're still finding out what works. A prototype on an API can be in front of real users within days. If the idea doesn't work out, you haven't bought hardware.
• Volume is low or unpredictable. A few hundred requests a day costs very little through an API.
• The provider's data terms are acceptable to you. Major providers offer business agreements, and many don't train on business API data by default. Read the terms for your provider and region; for many businesses they're enough.

When self-hosting makes sense

• Your data can't leave. Client contracts, regulation or internal policy may rule out sending documents to a third party, even one with good terms. With a local model, the text the model reads is processed on your hardware.
• The workload is steady and large. If the system processes thousands of documents every day, per-token fees become a real line item. Hardware is a fixed cost you can plan around.
• It has to run in a closed network. Sites with restricted or no internet access can't depend on an external API.
• You want stability. Hosted models get updated and retired, and their behavior can shift when they do. A self-hosted model behaves the same next year as it does today, unless you change it.

The middle ground

The choice isn't strictly one or the other.

• Private cloud GPUs. Rent GPU servers in your own cloud account and run open models there. Your data stays in an environment you control, you don't buy hardware, and you pay by the hour. This is often the easiest way to test self-hosting before committing.
• Models hosted by your cloud provider. Major cloud platforms offer hosted models, including some open ones, in a region you choose and under your existing cloud agreement. For some businesses, that settles the data question without running anything themselves.
• Hybrid routing. Send sensitive documents to a local model and everything else to an API. This works well, as long as the system can reliably tell which is which.

What self-hosting actually takes

Choosing a model

Open-weight models come in sizes measured in billions of parameters. Bigger models are generally more capable and need more memory. For focused business tasks, such as pulling fields out of documents, sorting emails or drafting replies from a template, a mid-sized model is often enough.

Test with your own examples before you decide. Public benchmarks don't tell you how a model handles your documents.

Hardware

The main constraint is memory. Most local deployments run models compressed to around 4-bit precision (quantized), and at that level these are rough sizes for the model alone:

• 7–8 billion parameters: about 5 GB. Fits on a single consumer graphics card.
• 14 billion: about 9 GB.
• 30–32 billion: about 20 GB. A high-end consumer card or a workstation card.
• 70 billion: about 40 GB or more. Usually two cards or a data-centre GPU.

Leave headroom on top of that for the context, the text the model holds in memory while it works. Long documents need more.

A GPU isn't strictly required. llama.cpp runs models on an ordinary CPU, just more slowly. We set up our private deployments to use a GPU where one is available and fall back to the CPU where it isn't, so the same deployment works on both.

Operations

Someone has to keep the system running: install updates, watch performance, restart services that fail and test a new model before switching to it. We hand deployments over as Docker packages with documentation so a client's IT team can run them, but it's still work an API provider would otherwise do for you.

The catch: "local" doesn't mean nothing leaves

This is the part most often misunderstood. Running the model locally keeps the model's processing in your environment. It says nothing about the rest of the system.

If your private AI assistant sends emails, searches the web, syncs with a cloud CRM or calls a hosted text-recognition service, information leaves through those connections. Before we build, we map every connection: what data it carries, where it goes and whether that's allowed. Then we agree which external services, if any, the system may use.

A private deployment also needs the ordinary controls of any business system: who can use it, which documents each person can see, which actions need approval and what gets logged.

How to decide

1. Pick one task. Choose something specific and frequent, like extracting fields from one type of document or sorting incoming email. Not "an AI for the whole company".
2. Collect real examples. Gather 50 to 100 representative cases with the correct answers. They become your test set.
3. Test both options. Run the examples through a strong hosted model and one or two local models. Compare accuracy, speed and cost per task.
4. Do the cost maths. Estimate your monthly volume, price it on the API, and compare that with the hardware and running costs of the local option.
5. Decide on evidence. If a local model meets the bar and the numbers work, plan the hardware. If not, use the API for now under the right data agreements, and test again as open models improve.

Questions we hear often

Is a local model as good as GPT or Claude?

On complex reasoning, usually not yet. On narrow, well-defined tasks with clear instructions, a good open model is often close enough. The only reliable answer comes from testing it on your own examples.

Is self-hosting cheaper?

At high, steady volume it can be. At low volume it rarely is, because the hardware costs the same whether it's busy or idle. Count the time it takes to run and maintain the system, too.

Can we start with an API and move to a local model later?

Yes, if the system is designed for it. Keep the model behind one interface in your code and keep your test examples. Switching then means changing configuration and re-running the tests, not rebuilding the system.

Do we need a GPU?

Not always. Small models run acceptably on a modern CPU for low-volume or background work. For several people using the system at once, or for larger models, a GPU makes a big difference.