Defining the Memory Budget: Weights, KV Cache, and Overhead
The first step in the protocol is determining the actual memory footprint of a model, which is not limited to quantized parameters. The total required memory is the sum of four independent components: model weights, the key-value cache (KV cache), transient activations, and the inference framework overhead.
A common error is applying a fixed 20% margin over the weights, which is inaccurate because these terms scale differently. For example, in a 7B parameter model with Q4_K_M quantization, the weights occupy approximately 4.9 GB (0.61 bytes per parameter, not 0.5 as suggested by the simplified 4-bit formula). To this, one must add the KV cache, which grows linearly with context length. For an 8,000-token context, the KV cache can add between 1 and 2 GB, depending on the attention architecture (GQA or MHA) and cache precision (FP16 or FP8).
The memory budget should distinguish the model file from the resources needed while it is running. Loading the weights is only part of the requirement: the active context, temporary calculations, and runtime also consume memory. An estimate that accounts only for the download size can therefore describe a model that fits on disk but cannot complete the intended workload in memory. The practical question is whether the full configuration fits while processing the context and number of simultaneous requests that the application actually needs.
The context estimate also needs an explicit operating condition. A short conversation does not exercise the same cache requirement as a long input, and a successful initial load does not establish the maximum context that the machine can sustain. Keeping weights, cache, activations, and framework overhead separate makes a failed test easier to diagnose. It also prevents a margin chosen for one configuration from being treated as a reliable rule for every model or attention architecture.
Memory Breakdown for a 7B Model in Q4_K_M (8k Context)
Estimation based on the VRAM calculation methodology for self-hosted LLMs.
| Component | Estimated Size | Notes |
|---|---|---|
| Weights (Q4_K_M) | ~4.9 GB | 0.61 bytes/parameter (4.9 effective bits) |
| KV Cache (8k tokens) | ~1.5 GB | Depends on n_kv_heads and head_dim |
| Overhead (CUDA/Framework) | ~1.0 GB | Fixed minimum per GPU |
| Total Minimum | ~7.4 GB | Requires 8 GB or higher GPU |
Selecting Quantization: Effective Precision vs. Nominal Bits
The choice of quantization must be based on effective bits, not the file name. GGUF formats mix precisions: Q4_K_M equates to about 4.9 effective bits (0.61 bytes/parameter), while Q8_0 is 8.5 bits (1.06 bytes/parameter). Assuming 4.0 bits for Q4 underestimates the weights by approximately 20%, which can lead to insufficient hardware selection.
Quality degradation varies by task. Q4_K_M offers an acceptable balance for most general use cases, with an estimated quality loss between 1% and 3% compared to FP16. Q5_K_M is recommended if additional VRAM is available and higher precision is required for reasoning or code. Q8_0 approaches original quality but doubles memory consumption compared to Q4, making it viable only in high-end hardware or for critical evaluation tasks.
Quantization names are useful identifiers, but they are not a complete memory specification. Mixed-precision formats preserve different parts of the model at different precisions, so the effective storage requirement differs from a calculation that assigns the nominal bit count uniformly to every parameter. Compare the actual artifact and its documented configuration rather than assuming that similarly named downloads have identical requirements. Any estimate still needs to be checked alongside the runtime memory described above.
The quality trade-off must be evaluated on the intended task. A general estimate of degradation does not guarantee the same effect on reasoning, code, or every language. A useful comparison keeps the model, prompts, runtime, and evaluation criteria consistent while changing the quantization being tested. The result then describes that comparison rather than a universal ranking. Additional memory may make a higher-precision artifact practical, but it does not replace checking whether the difference matters for the application.
Configuring the Inference Runtime: Ollama, LM Studio, or vLLM
The selection of the inference engine depends on the workload. For a single user on a desktop machine, Ollama and LM Studio (based on llama.cpp) are comparable in performance and offer an OpenAI-compatible API on localhost. Ollama stands out for its ease of use and model management with a single command, while LM Studio offers a more complete graphical interface for experimentation.
For production environments with multiple simultaneous users, vLLM is superior due to its use of paged attention and continuous batching. In concurrency tests, vLLM can serve up to 44 times more tokens per second than llama.cpp under high load. However, for a single user, the difference is minimal, so the choice should prioritize ease of auditing and integration with the existing workflow.
The workload determines which runtime features matter. Desktop experimentation emphasizes convenient installation, model management, and inspection of individual responses. A service with simultaneous users also needs to consider how requests share the available resources. A throughput advantage measured under concurrency should not be read as an equivalent reduction in the response time of a single desktop conversation. The cited benchmark describes particular conditions; the actual integration needs its own measurement with the chosen hardware and workload.
An API that follows a familiar interface can simplify integration, but the interface alone does not demonstrate equivalent performance or privacy. The configuration should retain enough information to reproduce the test: the selected artifact, quantization, runtime, context, and workload. Changing several of these together makes the source of a difference harder to identify. Keeping the configuration explicit helps distinguish a model limitation from a runtime or resource constraint.
Local Inference Runtime Comparison
Selection criteria based on use case.
| Runtime | Ideal Use Case | Key Advantage | Limitation |
|---|---|---|---|
| Ollama | Personal Use / Development | Ease of installation and local API | Less control over advanced parameters |
| LM Studio | Experimentation / RAG | Graphical interface and vector management | Closed-source application (open CLI) |
| vLLM | Production / High Concurrency | Maximum throughput and continuous batching | Configuration complexity and GPU requirements |
Network Isolation Audit: Verifying Privacy
Network isolation is a security strategy that involves dividing the network into independent segments to prevent unauthorized access or data leakage. In the context of local inference, the goal is to ensure that the inference process does not establish unauthorized outbound connections, keeping data within the jurisdiction controlled by the operator.
To empirically verify isolation, traffic monitoring tools such as Wireshark or tcpdump should be used to capture the connections of the inference process. Additionally, firewall rules can be configured to block all outbound traffic except necessary connections (if any). In corporate environments, this aligns with managed network isolation practices, where outbound access is controlled via private endpoints and service tags.
Local execution and verified network isolation answer different questions. Running inference on a local machine identifies where the computation occurs; observing and controlling its connections helps establish whether it sends data elsewhere under the tested conditions. Traffic capture and firewall rules therefore complement the deployment choice. A capture that shows no outbound traffic during a limited test is an observation of that run, while enforced restrictions control which connections are permitted. Neither should be replaced by an assumption based only on the word local.
Network Audit Process for Local Inference
Steps to certify that the stack does not expose data.
- 01Identify the PID of the inference process (e.g., ollama, vllm).
- 02Run a network monitor (e.g., tcpdump -i any host <destination_IP>) to capture outbound traffic.
- 03Verify that there are no connections to unauthorized external domains.
- 04Configure firewall rules (iptables/nftables) to block unwanted outbound traffic.
- 05Repeat the test with varied workloads to ensure consistency.
Artifact Integrity Validation and Smoke Test
Before integrating the model into the workflow, it is crucial to verify the integrity of the downloaded artifact. This involves comparing the checksum of the GGUF or safetensors file with the one published by the provider or community. A discrepancy indicates possible corruption or manipulation of the file.
Once integrity is validated, a minimal smoke test must be executed: generate a simple response to confirm that the model loads correctly into memory and produces coherent output. This test should be performed in a network-isolated environment to not compromise the privacy audit. If the test fails, compatibility between the quantization, runtime, and hardware must be reviewed before proceeding.
An integrity check compares the downloaded file with the expected artifact. Matching the published checksum helps detect a changed or corrupted download, but it does not establish the quality of the model's answers or the safety of its software dependencies. The smoke test addresses another question: whether this artifact loads and produces a response in the configured environment. Passing that minimal test is a starting point for workload evaluation, rather than evidence that longer contexts, concurrency, or every task will work correctly.
Keep exploring
Sources consulted
Corrections and transparency
If you spot incorrect or outdated information, send us a correction with the page and source we should review.
Submit a correction