Evaluating Quantized Local Models: Measuring Quality and Speed on Your Hardware

Why Measure Before Choosing

Quantization reduces the memory required to load weights, but it does not by itself indicate how much quality will change for a specific task. Nor is knowing the parameter count of a model enough to anticipate speed on your machine: the processor, available memory, inference configuration, context, and system load all play a role. The useful decision comes from measuring alternatives with inputs representative of real usage.

A 2025 study on six models run locally with Ollama in an academic information retrieval system compared accuracy, latency to the first token, and generation speed. That work shows a pertinent empirical approach, but it does not prove that a specific GGUF quantization preserves quality across other models, tasks, or hardware. The following guide turns that principle into a custom protocol: first define what 'getting it right' means and what performance you need; then compare each configuration against an explicit reference.

1. Define the Task and Baseline

Start by describing what the model will do in production: who uses it, what it receives as input, what output it must produce, which errors are costly, and how long it can take. An evaluation of field extraction requires measuring value matches; a document-based response task needs to check fidelity to sources; a code generator requires running tests, not just judging if the response seems plausible. If you mix different tasks into a single score, you may hide critical failures.

Prepare a test set that represents the expected distribution: frequent input types, edge cases, ambiguous inputs, and difficult examples. Include a verifiable reference for each case—a correct answer, expected fields, automated tests, or an evaluation rubric—and record where it comes from. Separate a development set, which you can use to tune prompts or parameters, from an evaluation set that you should not use for repeated optimization. This reduces the risk of choosing a configuration that only works well on already-seen examples.

2. Measure Quality for the Real Work

Quality is not a single metric. For responses with a known solution, calculate accuracy or hit rate; for extraction, exact match per field and per record; for classification, report precision and recall when false positives and negatives have different costs. In open generation tasks, use an explicit rubric—for example, factuality, coverage, clarity, and utility—and keep error examples to interpret the score. An automatic or model-based assessment can help review many cases, but it should not be treated as ground truth without checking its validity for that use.

Measure format as an independent criterion when the system requires JSON, a fixed structure, a tool call, or length limits. Check if the output can be parsed by the real validator and if it respects mandatory fields, types, enumerations, and constraints. Also record consistency between runs when sampling is not deterministic: a single correct response does not prove the system is reliable. For high-risk tasks, explicitly count critical errors and establish a rejection or human review rule before evaluating.

3. Measure Speed, Latency, and Memory

Record at least the latency to the first token and the generation speed in tokens per second. The former approximates how long a person waits before seeing a response; the latter describes subsequent production. Add the total time to complete the response, because a long output can take a long time even if the generation rate is high. Indicate which tool reports each measure and what it includes: initial model load, prompt preparation, input processing, or only generation.

Note memory consumption during inference and the context tested. The size of the GGUF file does not automatically equal the total memory required: context, attention cache, runtime, and the operating system can also influence it. Compare the same model and the same task with identical context limits, generation parameters, and hardware. If a configuration does not fit in memory and resorts to swapping to storage or a different execution path, record it: it is no longer an equivalent comparison.

4. Run a Controlled Comparison

To attribute differences to the evaluated configuration, keep the input set, prompt template, token limit, temperature, and other relevant parameters constant. Note the model name and revision, the source of the GGUF file, and its exact quantization. Also record versions of Ollama or LM Studio, operating system, processor, accelerator, memory, and layer offloading options. In the protocol, specify if the model was already loaded or if you also measure the initial load.

Run a preparation pass first if you need the model to load and stabilize, and do not mix it with regular measurements unless startup is part of the operational requirement. Repeat each test several times; report the median and variability, not the most favorable run. If the application uses stochastic outputs, fix the seed when the tool allows it or evaluate several samples with the same policy. Alternate the order of configurations if thermal or system load may change during the session. Repeat tests in conditions close to deployment, not just with a freshly rebooted machine.

5. Decide Between Quantization and Model Size

First evaluate if each option meets the pre-defined quality and performance requirements. Discard configurations that fail a critical criterion, even if they excel in another metric. Among those that pass, compare total time, initial latency, memory, and format stability. If a more aggressive quantization of the large model maintains necessary quality and fits in the hardware, it may be preferable; if it loses precision in a sensitive task, a smaller model with more conservative quantization may yield better results. There is no general rule that resolves that choice without measuring.

It is also reasonable that the best option is a different model than initially planned, or that the workflow routes simple tasks to a fast model and reserves a more capable one for difficult cases. The academic Ollama study mentioned earlier found differences in speed and accuracy between models in its own RAG system; its results should not be transferred as a guarantee to another configuration. To justify a technical decision, keep results by task type and explain what trade-offs the team accepts. If two alternatives are very close, consider the uncertainty of the measurements and repeat the test before fixing a conclusion.

6. Limits and Best Practices

A small set may miss rare errors; one too similar to development examples may favor the configuration tuned to those cases. An aggregated score can also hide failures by language, length, document type, or minority class. Examine broken-down results and keep error examples. If a difference is within run-to-run variation, do not describe it as a conclusive improvement without more repetitions or evidence.

The results of a research published in an academic RAG environment are evidence about that work and its configuration, not an evaluation of GGUF or a universal recommendation. Similarly, generic figures on a supposed fixed quality loss per quantization level do not replace a test with the specific model and task. Document the source of files, avoid including personal data in test prompts without authorization, and establish human review or rejection mechanisms when an error may cause harm. After deployment, periodically review a sample of real usage and repeat the protocol if the model, runtime, hardware, or input pattern changes.

Keep exploring

Corrections and transparency

If you spot incorrect or outdated information, send us a correction with the page and source we should review.