Open-weight models are the way to use AI without data ever leaving your walls. In return, infrastructure responsibility moves to you.
01. When It Makes Sense
In three cases: when data cannot leave under any condition, when volume is high enough that per-document cost dominates, or when you need deep customisation of the model.
02. When It Doesn't
For an organisation making a few thousand calls a month, self-hosting is almost always more expensive. GPU rental, operational time and upgrade load exceed the closed model's invoice.
03. Hardware
Model size determines memory. Mid-sized models can run on a single strong GPU; large ones want multiple. If memory is short the model runs but slows down, which kills the user experience.
04. The Serving Layer
Running the model isn't enough; you need a serving layer that queues, batches and scales requests. Without it the system stalls under concurrent users.
05. Upgrades
The open model ecosystem moves fast. A model that was good six months ago can be mediocre today. An abstraction layer that makes model swaps easy should be built from the start.
06. The Hybrid Approach
The common answer is using both: sensitive data on your own model, general work on a closed one. If the application was written to allow model swaps, that switch becomes a configuration matter.