I frequently speak with people who are enthusiastic about cloud providers for GenAI models. They have a Claude subscription, use Copilot for work or have connected their phone to Gemini. For these power users, LLMs have become far more than a chatbot: with the right integrations, these models can code independently, reply to emails and maintain a blog. Within organizations, these early adopters increasingly often play a major role in innovation. Their opinion therefore has a major impact on the implementation of novel technology.

Local models are not good enough for me to do my work.
That describes the reaction I receive when I ask whether they have tried out local models.

Until recently, this would have been the case for most people. The infrastructure could be challenging to set up, and while the models would generate words, reliability was insufficient. That is rapidly changing. In this post I introduce a few reasons why you would (not) want to use local AI.

Open versus closed

Nearly all models that can be run locally are called "open weight". The most popular way to use them is to download a GGUF file (for example through https://huggingface.co/) and to run it with llama.cpp . The GGUF file itself contains instructions about model configuration and contains the "weights". Because those are publicly available, we refer to them as "open weights". Open-weight is not the same as open-source. Most labs that produce open-weight models, do not make the source code, data and methodology available that is required to train the models.

Open-weight models do not necessarily need to be run locally. A hefty VM from a cloud hoster is also suitable. Hence, local versus cloud does not appropriately frame the discussion: this is about open versus closed.

Commercial providers of GenAI models that develop their own models, do not always share the model files. These models are called "closed weight". As the model files are not shared, these models cannot be deployed on a local setup.

The most advanced models are called "frontier models". In 2024, nearly all of these models were closed-weight. Today, that is debatable: at the time of writing (october 2026) public benchmarks on (for example) Qwen3.8 Max show that this open weight model ranks 2nd on the benchmark for following instructions. For the multimodal ranking, based on the combination of visual information and text, the open-weight Kimi K3 model even occupies the 1st position. While aggregate rankings are still led by closed-weight models, open-weight models are increasingly closing in. For specific tasks, they might even outperform closed-weight models.

These rankings show that recent open weight models are more than good enough for many use cases. When it comes to specific tasks, they can even outperform closed weight models. This means it is already interesting to deploy multiple models in parallel so that it is possible to take advantage of their strengths. A testing setup makes these evaluations possible.

Costs

In contrast to the relative cost-effectiveness of GenAI subscriptions, the price of tokens for API-based tools can become challenging to manage. A major problem here is that the cost of tokens scales linearly with increased usage. With self-managed hardware this is not the case. There is an initial investment, followed by periodic depreciation of the hardware. In the long term, access to self-managed hardware can make the costs of AI-based integrations more predictable. Theoretically, there is also a break-even point: if the hardware lasts long enough, the cost per token can become lower than it would have been with usage-based billing.

Theoretically - because that break-even point is currently difficult to calculate. The demand for GenAI has led to an enormous demand for RAM, storage and GPUs. And this has had a significant impact on prices, delaying the break-even point. Moreover, hardware availability is not keeping pace with model development. While more and larger open weight models are becoming available, the hardware to deploy those models is only getting scarcer. There is also a risk that the future need for better models causes a shorter effective lifespan for self-managed hardware.

A second complication has to do with the flexibility of AI use cases. This makes it challenging to calculate the added value of AI integrations in (business) processes. Moreover, many of those integrations are still in development. This means that it is unclear how generated tokens will contribute financially. If that contribution is low, any investment in AI integrations is illogical - regardless of it being an investment in local or cloud systems.

One of the major factors depressing the cost-effectiveness of self-managed hardware is its usage. A server in standby costs a lot, but earns nothing. In that case, it does not matter how expensive a cloud provider's tokens are: on usage-based billing, customers are not billed for unused tokens. A unused server still comes with costs. In fact, those costs are so significant that cost-effectiveness can be challenging to guarantee even with 100% usage.

Are the processes that will be supported by GenAI integrations in a mature state? Is it possible to compare the cost of tokens to the cost of hardware and maintenance? In that case, it can be worthwhile to carry out an analysis. In all other cases it can still be interesting to work with open weights models, but that won't be because of possible savings.

Stability

Cloud providers make choices to improve the usability of their GenAI services and to control costs. For example, providers can choose to limit the number of models they offer. They can also deploy convenient integrations with cloud systems that do not clarify how access management is handled. In practice, this means a high frequency of changes that can potentially drastically affect end users. A model that performs perfectly in an integration could therefore suddenly start showing different behavior after an update.

With open-weight models, this is the responsibility of the maintainer. Models will never be automatically phased out and integrations are not invisibly deployed. That has advantages as well as disadvantages.

When processes are relatively optimized, some form of change and quality management usually exists. Processes are documented; training materials may be available; and the process outcomes are clearly defined. If these processes need to be changed, involved people can accurately judge the most important risks. If something goes awry, it is immediately clear what the consequences are.

Are the planned GenAI integrations related with these kinds of processes? In that case, it makes sense to demand the same level of risk management. If the process has a clear and measurable outcome, there needs to be a strategy for ensuring the AI integrations does not negatively affect those outcomes. If changes to the process cannot be made by unauthorized individuals, that should probably also be the case for the AI integrations. And if the process requires multiple suppliers to avoid problems in case one of the suppliers becomes unavailable, it makes sense to make the server behind the integrations interchangeable as well.

For smaller organizations in particular, there is a risk that IT processes are not sufficiently automated. That can cause major problems because it creates a barrier for secure and efficient access management. In the same way that individuals in an organization cannot have unbridled access to IT resources, there needs to be a policy for AI integrations. Investments in effective systems administration start repaying themselves very quickly in such situations.

The choice for GenAI infrastructure is therefore strongly dependent on the type of process that needs to be supported and the available technical capacity. If there is no clarity (yet) on the role of GenAI in processes, it pays off to remain flexible with integrations and changes resulting from external providers. Are there solid arguments that support a requirement like local data processing or durable and reliable processes? In that case, it is worthwhile to research the benefits and costs of open weight infrastructure. A small local test setup can be helpful in the design and evaluation of AI integrations.

Expertise

While adoption of GenAI technology can be an objective in its own right, it is usually subservient to existing objectives. Often, there are already processes in place that have to do with those objectives. Those processes do not appear out of thin air: they depend on the expertise of people who design and execute them.

Truly effective GenAI integrations in such processes can require creative solutions. Maybe there needs to be a connection to a database. Or there needs to be a more effective way to access internal documentation. Service providers often provide their own solutions, sometimes as part of their existing services, other times as an additional service. In most cases, those services have been designed to accommodate the demands of the majority of end users.

For processes that depend on expertise, it can be the case that standardized solutions do not suffice. That creates an opportunity to collaborate with the experts to design a custom solution. Such custom solutions are expensive, creating a need to product the durability of the return on investment. Introducing dependencies on external providers can endanger that durability. Moreover, it can introduce security risks that can be avoided with self-managed infrastructure.

It is these types of processes that are the most interesting targets for investment. With sufficient expertise available, it becomes possible to estimate the added value and risks of the new solution. Performance can be measured more reliably and requirements for a successful deployment can be identified. For example, are there requirements in terms of productivity or security? How is quality defined?

It is possible that there are solutions from external providers that do meet objectives. Is this desirable? Using such solutions can result in the erosion of internal expertise and to vendor lock-in. Building knowledge to effectively leverage technology is challenging to develop, especially when it comes to processes that are part of a unique niche. The infrastructure determines which knowledge is protected.

TLDR;

The choice for open- versus closed-weight depends on:

• An accurate and complete overview of costs
• Requirements for risk management
• The role of internal expertise

Cloud-based GenAI solutions are nearly always more cost-effective in the case of general approaches where the context is unclear.

Interested in the opportunities and risks of GenAI in your organization, and the infrastructure that best fits this profile? Reach out for a quote..