Introduction
Self-hosting an open-weight LLM used to be a hobbyist project. In 2026 it is a mainstream engineering decision. Teams choose it for data control, predictable cost at scale, and the freedom to customize inference. But the gap between a demo and a production deployment is wide, and most of the work happens before you ever load a model.
The good news is that the ecosystem has matured. Serving frameworks handle batching, quantization and multi-GPU sharding out of the box. Model families now span a wide range of sizes, from small dense models that fit on a single consumer card to massive mixture-of-experts systems that need a rack. The hard part is matching your workload to the right model, hardware and licence.
This guide walks through the decisions in order: pick a model size, size the hardware, choose a serving stack, check the licence, and decide whether self-hosting is actually cheaper than an API. Along the way we will point to concrete open-weight releases you can evaluate today.
Key Concepts
Open weights: the trained model parameters are publicly downloadable, so you can run inference on your own hardware. The licence may still restrict commercial use, redistribution or acceptable-use policies.
Quantization: reducing the precision of weights and activations (for example to 8-bit or 4-bit) to cut memory use and speed up inference, usually with a small quality trade-off.
KV cache: the memory that stores attention keys and values for tokens already processed. It grows with context length and batch size, and it often determines how many concurrent users a GPU can serve.
Serving stack: the runtime that loads the model and exposes an API, handling batching, scheduling and memory management. Examples include vLLM, TGI and llama.cpp-based servers.
Deep Dive
Step 1: Pick a model size that matches your hardware
Start from the hardware you can actually get, not from the biggest model on a leaderboard. A 12B dense model is a comfortable single-GPU target. Gemma 4 12B, released by Google DeepMind in June 2026, is an open-weights multimodal model with roughly 11.95B parameters and a context window of up to 256K tokens. It handles text and image input, with audio supported on some sizes, and covers over 140 languages. That makes it a strong default for a first self-hosted deployment.
If you need more capability and have more memory, a 30B dense model is the next step. Granite 4.1 30B, released by IBM in April 2026, is part of a dense family trained on roughly 15T tokens with long-context extension up to 512K tokens. IBM states the models are released under the Apache 2.0 licence, which simplifies commercial use. At the top end, DeepSeek V4.1 Flash is a 748B-parameter mixture-of-experts model with a 552B backbone and support for contexts up to one million tokens. That is a multi-node deployment, not a single-GPU project.
Step 2: Size the hardware around memory, not just FLOPs
The first constraint is memory. Weights must fit in VRAM (or be sharded across GPUs), and the KV cache must fit alongside them. A rough rule: a 12B model in 16-bit needs about 24 GB just for weights, so a 24 GB or 48 GB GPU is a realistic target. Quantizing to 8-bit or 4-bit cuts that substantially and is usually the first lever you pull.
The second constraint is throughput. Batch size and context length drive KV-cache memory, so long-context workloads need more headroom. If you plan to serve many concurrent users, measure tokens per second under realistic load rather than trusting a single-stream benchmark. For agent workloads that are input-heavy, prefill cost and cache capacity matter as much as generation speed.
Step 3: Choose a serving stack
For GPU serving, vLLM and TGI are the common choices. They provide continuous batching, paged attention and OpenAI-compatible endpoints, which means your application code can switch between a hosted API and your own server with minimal changes. For CPU or Apple-silicon experimentation, llama.cpp and its server are convenient, especially with quantized GGUF weights.
Whichever stack you pick, wrap it in the usual production scaffolding: health checks, request timeouts, rate limits, structured logging and a queue for long-running jobs. Model servers are stateful and memory-hungry, so treat them like databases rather than stateless web apps.
Step 4: Check the licence before you ship
Open weights is not the same as open source. Read the model card and licence for each candidate. Some releases are unrestricted, some use Apache 2.0, and others add acceptable-use terms or attribution requirements. If you plan to redistribute the model, offer it as a hosted service, or fine-tune it, confirm those rights explicitly. Keep a record of the licence version you deployed.
Practical Application
A sensible first project is a private document assistant. Pick a 12B-class model, serve it with vLLM behind an internal API, and connect it to a retrieval layer over your own files. Measure latency and quality on a small evaluation set before expanding. If you need a hosted reference point, you can compare behaviour in the AIPortalX Playground before committing hardware.
For teams building agentic workflows, the AI agents tooling category is a useful checklist of the surrounding infrastructure — tool calling, retries, observability — that a self-hosted model still needs. And if your workload is primarily question answering over long documents, the question answering task page lists models suited to that pattern.
Common Mistakes
Buying hardware before benchmarking the model. Measure tokens per second and quality on your actual workload first; a smaller quantized model may be enough.
Ignoring the KV cache. Teams size for weights and then discover that long contexts or high concurrency exhaust memory. Budget for cache from day one.
Assuming 'open weights' means unrestricted commercial use. Check the licence and acceptable-use terms for every model you deploy.
Forgetting idle cost. A GPU that sits idle still costs money. If utilization is low or spiky, an API is often cheaper than a dedicated box.
Skipping evaluation. Self-hosting changes the model, the prompt format and the sampling defaults. Re-run your quality checks after every change.
Next Steps
Start small: pick one 12B-class model, serve it on a single GPU with vLLM, and run a week of internal traffic. Track latency, memory headroom and quality. If that works, scale up to a 30B model or add a second node. Keep the API path available as a fallback for overflow and experimentation.
Before you commit, review the licence for each candidate model, document your evaluation results, and compare total cost against a hosted API at your expected utilization. For more model options and deployment notes, browse the AIPortalX deployment resources.
