Model Quantization Guide: Foundations to Production Serving

Viacheslav Dubrov

2026-07-05 · Updated 2026-07-22 · 23 min · Model Engineering · AI Infrastructure

Quantization uses fewer bits to represent model values. A serving path may quantize weights, activations, the KV cache, or some combination, and each target solves a different serving problem.

Choose the target from the current bottleneck: weight memory, activation compute, KV-cache size, kernels, hardware, calibration data, or quality. A 4-bit model may fit in VRAM yet run slowly with an unoptimized kernel, as the JarvisLabs vLLM benchmark demonstrates. FP8 works well on NVIDIA Hopper hardware with a compatible runtime and offers no native benefit on unsupported GPUs. TensorRT-LLM’s hardware matrix shows the supported paths. For long contexts or high concurrency, KV-cache quantization may save more memory than weight quantization.

If you are an ML or platform engineer choosing a quantization path for a model you need to serve, this guide is for you. After reading, you should be able to map a bottleneck to a candidate format, runtime, and validation plan before deployment.

For the short artifact and method comparison, see LLM Quantization Formats.

1. Start with the bottleneck

Ask what limits the workload before choosing a bit-width. The answer may be model weight memory, prefill compute, decode bandwidth, or the KV cache rather than numerical precision by itself.

Common bottlenecks and their starting points:

Use this table as the map. The sections below explain why those starting points differ.

Notation used in serving recipes

• W{x}A{y} tells you the precision for supported weight and activation math, usually GEMM paths in serving engines. W4A16 stores weights in 4-bit form and keeps activations in 16-bit precision. W8A8 uses 8-bit weights and activations in supported compute paths, but it does not automatically define the persistent storage dtype of every runtime tensor.
• FP8, INT8, INT4, NF4 are number formats. They decide what values can be represented.
• GPTQ, AWQ, SmoothQuant, QuaRot are algorithms. They decide how to map a trained model into a lower-precision format.
• GGUF is a file format that stores tensors and metadata for GGML and llama.cpp-style runtimes. A GGUF file can contain unquantized tensor types such as F16, BF16, or F32, as well as quantized encodings. Presets include Q4_K_M, Q5_K_M, Q8_0, IQ*, TQ*, and MXFP4. The tensor encoding determines the quantization choice. GGUF alone does not describe a CUDA-style FP8 W8A8 serving recipe.
• KV cache is the attention cache used during generation. It stores prior keys and values so the model doesn’t recompute the whole conversation at every token.
• KV-cache quantization stores cached key/value activation tensors in a lower-precision cache format such as FP8, INT8, INT4, or INT2, depending on runtime support. This is different from prefix caching, PagedAttention, or offload, which decide whether cache entries are reused, how they are allocated, or where they live.
• GEMM means general matrix multiply. Most transformer inference time is spent doing matrix multiplication.

2. Quantization is controlled rounding

Quantization maps high-precision values into a smaller set of representable values, which is the core definition used by both Hugging Face Optimum and TensorRT-LLM. You save memory and bandwidth. You also introduce rounding error.

INT4 provides only 16 discrete values, so mapping BF16 weights into that grid creates rounding error. Methods such as GPTQ, AWQ, and SVDQuant focus on preserving outliers and reducing reconstruction error. A good mapping saves memory with little quality loss. A poor one damages reasoning, instruction following, or visual fidelity.

Symmetric and asymmetric mapping

Following the affine mapping used in common quantization guides, quantization maps a continuous float value $x \in [\beta, \alpha]$ to a discrete grid.

• $x$ is the original high-precision value.
• $x_q$ is the quantized value.
• $s$ is the scale, or step size.
• $z$ is the zero point, the integer location that represents 0.0.
• $[q_{\min}, q_{\max}]$ is the target integer range. Signed 4-bit values often use $[-7, 7]$.

Symmetric quantization centers the grid around zero and sets $z = 0$:

This is hardware-friendly because runtime math doesn’t need to subtract a zero-point offset. PyTorch’s quantization stack exposes these affine scale and zero-point choices as primitive quantization parameters in torchao.

Asymmetric quantization shifts the grid to cover skewed ranges:

That shifted grid can preserve positive-only activations better, but the offset adds work unless the kernel handles it well.

Scale granularity matters

The scale factor can cover a whole weight tensor, one channel, or a small group of values. vLLM’s FP8 KV-cache docs use the same distinction between per-tensor and per-attention-head scale strategies. Smaller groups usually preserve quality better, but they require more scale metadata.

Weights are static, so their scales can be computed offline before you load the model. Activations change with every token, which makes their ranges workload-dependent.

The KV cache sits between those cases. Keys and values begin as runtime activation tensors: each layer computes them from hidden states during the forward pass. Once generated, they stop being transient matmul intermediates and become persistent serving state that attention reads for later tokens.

A serving engine can store that state in lower precision and keep scale metadata alongside it. vLLM’s Quantized KV Cache, TensorRT-LLM’s FP8 KV Cache, and Transformers QuantizedCache all expose this storage choice.

Cache storage precision remains separate from the activation precision used inside linear kernels. “KV-cache quantization” names one cache optimization, not every technique that reuses, allocates, or moves cache entries.

PTQ and QAT happen at different stages

Post-training quantization, or PTQ, compresses a trained model after the fact. Quantization-aware training, or QAT, exposes the model to quantization noise during training so it can adapt.

Calibration data has to look like the traffic you will serve. vLLM’s KV-cache calibration path, for example, uses a curated dataset through llm-compressor. If production prompts are long RAG traces, short Wikipedia paragraphs will give you neat benchmark numbers and a broken deployment. The model will tune its activation scales around short text, then encounter different activation patterns when real long-context prompts arrive. Long-context quantization evaluations measure this risk directly.

3. Number formats dictate hardware requirements

The number format defines what values the model can represent in memory. Efficient computation requires runtime kernels and hardware support for the same bit-width and format. TensorRT-LLM documents both the recipe list and hardware support matrix.

BF16 and FP16 both use 16 bits, but they distribute precision differently and therefore fail differently. BF16 keeps the 8-bit exponent range of FP32 and is harder to overflow. FP16 has more mantissa bits and a narrower exponent range, so activation spikes need more care. The Kurtic et al. evaluation explicitly uses BF16 as the baseline when comparing FP8, INT8, and INT4 serving formats.

FP8 has two common variants. E4M3 gives more precision and is usually used for forward weights and activations. E5M2 gives more dynamic range and is more useful for gradients or volatile activation paths. vLLM exposes both FP8 E4M3 and E5M2 KV-cache dtypes. In Kurtic et al.’s ACL 2025 study, “Give Me BF16 or Give Me Death,” FP8 W8A8 was effectively lossless on the Llama-3.1 family over more than 500,000 evaluations. The result covers that model family, evaluation suite, and serving setup. Each deployment still needs its own quality gate.

Blackwell adds microscaling formats such as MXFP8 and NVFP4. Instead of one scale for a whole tensor or row, microscaling uses tiny blocks. NVIDIA’s NVFP4 explainer describes 4-bit floating-point values in blocks of 16, with FP8 scale factors and a higher-level FP32 scale. This approach aims to provide a near-INT4 footprint with floating-point behavior. However, it requires a matching hardware architecture, compiler, and runtime support, which is why TensorRT-LLM lists FP4 and FP8 support by GPU generation.

4. Weight-only vs. weight-activation quantization

The notation WxAy describes the precision of weights and activations, which stress the GPU in different ways during inference.

During prefill, the model processes the input prompt. This phase is usually compute-bound because the GPU is doing large matrix multiplications, which is why W8A8 FP8/INT8 recipes matter for throughput-oriented serving.

During decode, the model generates one token at a time. This phase is often memory-bandwidth-bound because the GPU keeps loading weights from VRAM to produce the next token. Weight-only papers such as GPTQ and AWQ target that pressure by reducing weight bytes.

W4A16 compresses weights and keeps activations in BF16 or FP16. The GPU loads fewer weight bytes, then dequantizes the weights back into a higher-precision form for the multiply. This helps decode and fit-to-memory problems. Compute-bound prefill may see little benefit because the matrix math still runs in 16-bit.

W8A8 compresses weights and the activation tensors used by supported matmul kernels. If the hardware has native low-precision tensor cores, the serving engine can run matrix math directly in FP8 or INT8. FP8 can therefore help high-throughput serving by reducing memory traffic and using faster arithmetic. The KV cache has its own storage setting, so check the runtime’s cache dtype or cache implementation separately.

If the model barely fits in VRAM, start with weight-only quantization to reduce the memory footprint. If the model fits but struggles with throughput under high batch loads, evaluate FP8 or INT8 W8A8 to accelerate the compute phase. If memory issues only appear during long conversations, first estimate the KV-cache term. Test KV-cache quantization when that term dominates. Enable prefix caching when repeated prefixes dominate.

5. Algorithms vs. runtime kernels

Quantization algorithms (like GPTQ or AWQ) define how the model’s weights are mapped to lower precision. Runtime kernels (like Marlin or vLLM’s custom kernels) are the low-level GPU code that executes the matrix multiplication. A highly compressed model will only run fast if there is an optimized kernel for its specific quantization format.

JarvisLabs’ vLLM benchmark on Qwen2.5-32B-Instruct with an NVIDIA H200 makes the kernel effect visible:

Don’t copy these numbers to your own stack. They come from one model, one GPU class, and one software setup. They show a narrower point: the algorithm name on the checkpoint doesn’t tell you how fast serving will be.

For example, AWQ and Marlin-AWQ both use the same 4-bit weights. The Marlin implementation is much faster because its CUDA kernel fuses dequantization and matrix multiplication into a single, highly optimized GPU operation.

Benchmark the baseline and the compressed variants under the same prompt mix with the same tool, such as vllm bench serve:

vllm bench serve \
  --model ./outputs/Qwen2.5-32B-Instruct-AWQ-W4A16 \
  --dataset-name sharegpt \
  --num-prompts 200 \
  --input-len 1024 \
  --output-len 256

Track throughput, TTFT, inter-token latency, memory use, and task quality. When some of those move in opposite directions, that trade-off is what you need to see before shipping.

The algorithm menu

Use this table as a map, not as a ranking:

The toolchain is actively evolving. AutoGPTQ was archived in April 2025, and AutoAWQ was archived and officially deprecated in May 2025. For new compressed-tensors checkpoints consumed by vLLM, start with llm-compressor. Use GPTQModel when you need the active GPTQ path with Marlin, Machete, MoE memory options, or disk offload.

Pruning and distillation also reduce serving cost through separate workflows. 2:4 structured sparsity removes weights in a pattern that NVIDIA sparse tensor cores can use. Distillation trains a smaller student model to copy a larger one, which can work well for narrow tasks. Include either path in the shortlist only when the project can support the extra pruning or training work.

6. Serving memory is more than weights

The compressed checkpoint is only part of the serving-memory footprint. Size the full runtime before deciding whether weight quantization is enough. PagedAttention identifies the KV cache as a major serving-memory term.

Offline quantization can be layer-bounded. Tools such as llm-compressor can load a transformer block, run calibration and quantization math, write the compressed block, and move on. That keeps peak GPU memory closer to the largest active layer plus calibration buffers. You still need CPU RAM and disk for the source checkpoint, but the GPU doesn’t always hold the full BF16 model.

Peak GPU memory during offline quantization can look more like this:

Serving is stricter. The whole compressed checkpoint must stay resident with the KV cache and runtime buffers. The KV cache grows with context length and active batch size:

Where:

• $L$ is the layer count.
• $H_{\text{kv}}$ is the number of key-value attention heads. Grouped-query attention reduces this by letting many query heads share fewer KV heads.
• $D$ is the dimension of each head, often 128 or 256.
• $S_{\text{ctx}}$ is prompt tokens plus generated tokens.
• $B_{\text{batch}}$ is the active serving batch.
• $\text{BytesPerValue}$ is 2 for BF16 or FP16 and 1 for FP8 or INT8. vLLM’s FP8 KV-cache mode is the serving-stack example this article uses.

KV-cache quantization changes storage, while reuse changes allocation

The KV cache can be quantized during inference. Each decode step produces K and V activation tensors for the new token. A W8A8 model may already use FP8 or INT8 for supported projection math, but the cache remains a separate storage object.

Many serving stacks keep that object in the model or cache dtype. To change it, enable a KV-cache dtype, use a checkpoint with cache scales, or choose a quantized cache implementation.

When KV-cache quantization is enabled, the engine writes entries as a lower-precision representation plus scales. Later attention either dequantizes the cache inside its kernel or, on some backends, performs part of the attention operation in the quantized domain.

vLLM’s stable Quantized KV Cache docs expose this directly with kv_cache_dtype="fp8" or --kv-cache-dtype fp8. vLLM supports FP8 E4M3 and E5M2 cache formats, plus per-tensor and per-attention-head scale strategies. It offers three ways to get the scales: defaults, warmup-time estimation, and dataset calibration through llm-compressor. With FlashAttention 3, vLLM can also run attention operations in the FP8 domain by quantizing queries in addition to keys and values.

TensorRT-LLM exposes FP8 KV cache through KvCacheConfig(dtype='fp8') and lists FP8 KV cache and NVFP4 KV cache as separate quantization recipes from weight/activation quantization. Hugging Face Transformers also has a QuantizedCache path via cache_implementation="quantized", with hqq supporting int2, int4, and int8 cache formats and quanto supporting int2 and int4.

Ordinary KV caching stores prior keys and values to avoid recomputing them. Prefix caching reuses cache blocks across requests with the same prefix. PagedAttention reduces fragmentation and improves allocation, while KV offload moves cache blocks between memory tiers. These combinations are runtime-dependent. Hugging Face’s QuantizedCache does not support offloading. vLLM documents its quantized KV cache separately from prefix caching and other cache-management features. Verify each combination in the runtime you deploy.

The quality risk also differs from weight-only PTQ. KV-cache quantization injects error into the attention state read at every later decode step. Test long-context retrieval, multi-turn behavior, safety and refusals, tool-use formatting, and output latency separately. KVQuant, KIVI, and the vLLM FP8 KV-cache study all evaluate KV-cache quantization as its own problem.

Model size and context length alone do not determine whether cache compression beats another round of weight compression. Calculate cache bytes with the equation above using the model’s KV-head count, head dimension, active batch, and cache dtype. Then compare that result with the bytes saved between two named weight formats, such as BF16 and INT4. If the cache is larger, testing FP8 KV-cache quantization may free more serving memory than shrinking the weights again.

7. Hardware narrows the menu

Weight footprint is easy to estimate from parameter count and storage precision, the same sizing idea used in serving-memory discussions around KV cache:

Mixture-of-experts models may activate fewer parameters per token, but the full set of weights still needs to live somewhere unless the runtime supports offload. TensorRT-LLM’s quantization support matrix treats MoE model families as deployment targets with their own supported recipes.

Your deployment hardware restricts which quantization formats are viable:

• CPU serving depends on vector instructions such as AVX-512 or AMX. A GGUF file loaded through llama.cpp is the practical path.
• Apple Silicon uses unified memory, so local models can use a large shared RAM pool instead of dedicated VRAM. GGUF and llama.cpp remain the common local-runtime path because GGUF is built for GGML executors.
• NVIDIA Ampere supports INT8 tensor-core serving paths but not native FP8 W8A8 tensor-core math. Common choices are W4A16 weight-only quantization or static INT8, matching the TensorRT-LLM hardware support matrix.
• NVIDIA Ada and Hopper support FP8 serving paths in TensorRT-LLM. FP8 W8A8 serving is worth testing on these GPUs.
• NVIDIA Blackwell adds NVFP4 and microscaling support, but the software path still matters. Treat early low-bit floating-point stacks as version-sensitive.

8. Calibration and evaluation before deploy

A model that loads has passed a smoke test. Deployment requires quality and serving checks for the target workload. Recent quantization evaluations report different outcomes for LLM serving, long-context tasks, and reasoning-heavy models.

For calibration, use prompts that look like production:

• Include RAG traces, SQL queries, agent histories, code tasks, tool-call payloads, and system prompts from the target workload. Static PTQ depends on calibration data matching the production distribution.
• Match sequence lengths. Short single-turn prompts won’t expose long-context activation behavior.
• Keep embed_tokens and lm_head in higher precision if the method or runtime allows it, a common exclusion pattern in LLM Compressor recipes.
• Use enough samples to stabilize activation ranges. The vLLM KV-cache example sets NUM_CALIB_SAMPLES = 512. Treat that as one documented example, not a universal count. The right sample count depends on the method, model, sequence length, and production workload.
• Redact secrets and private user data before using production logs.

For evaluation, test both language quality and serving behavior:

• Perplexity on a standard corpus catches broad language degradation, but the JarvisLabs benchmark is a useful reminder that perplexity and throughput can move differently.
• Domain tasks catch failures that perplexity hides. Use HumanEval for coding, MMLU for broad knowledge, and AIME or MATH-500 for mathematical reasoning when those domains matter.
• Format checks matter for agentic systems. Test JSON schema compliance, markdown output, tool-call shape, and refusal behavior because quantized model evaluations can miss application-level failures even when aggregate benchmark accuracy looks stable.
• Long-context tests catch KV-cache quantization damage. Needle-in-a-haystack is crude, but long-context quantization results show why those checks belong in the deploy gate.
• Load tests should report throughput, TTFT, inter-token latency, max batch capacity, and peak memory. vLLM exposes these measurements through vllm bench serve.

Evaluate reasoning-heavy models rigorously. Sub-4-bit or unrotated W4A4 quantization can hurt reasoning accuracy even when baseline perplexity looks stable, which is the core warning in the quantized reasoning-model study.

9. Companion repository workflow

The companion repository, slavadubrov/model-compression-demo, is meant to make the decision process reproducible. It uses uv and focuses on planning, recipes, dry runs, and benchmark configs grounded in the same sources used here: vLLM, LLM Compressor, TensorRT-LLM, and the algorithm papers.

The public README at revision 8b45003849e830bed2ff341a9f027b017d932c1f was checked on 2026-08-16. The companion checkout is not present in this workspace, so I could not run its CLI here. The commands below are illustrative until you run them from that pinned checkout, and the benchmark plan still needs the target serving hardware.

Do not copy the FP8 recipe output from that pinned revision. Its recipe --algorithm fp8-dynamic command emits an internally inconsistent model and output path. The command remains omitted here until the companion is repaired.

Clone it and inspect the supported algorithms:

git clone https://github.com/slavadubrov/model-compression-demo.git
cd model-compression-demo
git checkout 8b45003849e830bed2ff341a9f027b017d932c1f
uv run python demo.py list-algorithms

Start with planning and sizing:

uv run python demo.py plan \
  --model-preset qwen3-8b \
  --goal fit-memory \
  --hardware ampere \
  --context 4096 \
  --concurrency 4

uv run python demo.py estimate \
  --model-preset qwen3-8b \
  --scheme w4a16 \
  --context 4096 \
  --concurrency 4

uv run python demo.py plan \
  --model-preset qwen3-0.6b \
  --hardware cpu

Then generate a recipe and preview quantization before spending GPU time:

uv run python demo.py recipe --algorithm gptq-w4a16

uv run python demo.py quantize --dry-run
uv run python demo.py quantize \
  --algorithm gptq-w4a16 \
  --model Qwen/Qwen3-8B \
  --dry-run

For serving and benchmark planning:

uv run python demo.py serve-command \
  --algorithm fp8-dynamic \
  --fp8-kv-cache \
  --enable-prefix-caching

uv run python demo.py benchmark-plan \
  --model Qwen/Qwen3-8B \
  --algorithms gptq-w4a16,rtn-w8a16,fp8-dynamic \
  --dataset-name sharegpt \
  --num-prompts 200 \
  --input-len 1024 \
  --output-len 256 \
  --output-json reports/quantization-benchmark-plan.json

Finally, compare the base and compressed models against explicit thresholds:

uv run python demo.py quality-eval \
  --base-model Qwen/Qwen3-8B \
  --compressed-model outputs/Qwen3-8B-W4A16 \
  --mode all \
  --lm-eval-task hellaswag \
  --lm-eval-limit 50 \
  --max-perplexity-delta-pct 5 \
  --output-json reports/qwen3-8b-w4a16-quality.json

Run the work in order: plan the target, estimate memory, dry-run the recipe, benchmark serving, then compare quality against thresholds. That sequence mirrors the separation in this article between memory sizing, runtime benchmarking, and quality evaluation.

10. Diffusion models need a separate path

Diffusion and diffusion-transformer pipelines have different activation behavior from autoregressive LLMs. SVDQuant treats diffusion quantization as a separate activation-outlier problem.

Autoregressive LLMs generate one token at a time. Diffusion models run repeated denoising steps, and their activation distributions shift during the process. A standard 4-bit LLM quantization pass may save diffusion-model memory while introducing severe visual artifacts. Diffusion-focused methods such as SVDQuant / Nunchaku and NVIDIA ModelOpt diffusion quantization address that different activation pattern.

Use the following as conservative heuristics, not universal defaults. The pipeline, component, model, and runtime each need separate tests:

1. Keep the VAE in 16-bit for the first comparison. This conservative heuristic reduces one source of image artifacts. Test lower precision only when the method and target pipeline validate it.
2. Try the DiT or U-Net backbone first because it often holds the largest parameter share. This is a heuristic, so verify memory, latency, and image quality for the target pipeline. Diffusion quantization methods take the same component-level approach.
3. Treat text encoders separately. Quantizing T5-XXL or CLIP may affect prompt alignment or text rendering in a given pipeline, so evaluate them independently rather than assuming generic transformer behavior.
4. Use diffusion-aware methods such as SVDQuant when activation outliers are the main issue.
5. Evaluate with images, not text metrics. Check prompt adherence, text rendering, skin tones, color balance, fine detail, latency, and VRAM.

If the evaluation set only contains simple or common prompts, you will miss edge-case failures. Include hard cases: small text, hands, repeated objects, structured layouts, and prompts with negative constraints, because diffusion quantization failures show up visually rather than in language-model perplexity.

11. Production defaults

For enterprise LLM serving, start with a BF16 baseline in the exact serving engine you plan to use. If throughput is the goal and the hardware supports it, test FP8 W8A8. If the model doesn’t fit, test AWQ or GPTQ W4A16 with Marlin-class kernels. If long context or concurrency is the problem, test FP8 KV-cache quantization. If repeated prefixes are the problem, also enable prefix caching. Ship the compressed version only when quality and serving benchmarks both pass.

For local and edge inference, start with a GGUF file using Q4_K_M or Q5_K_M. Move to a Q8_0 GGUF when memory allows and quality matters more than footprint. Going below 4-bit is a last resort, not a default.

For fine-tuning, use NF4 with QLoRA to train adapters cheaply. Evaluate the adapter in the application before merging. After merging, export to the serving artifact you actually need: a llama.cpp-compatible GGUF file, an AWQ/GPTQ/compressed-tensors checkpoint, an FP8 serving checkpoint, or BF16.

For diffusion, start with those conservative heuristics and test each pipeline/model/runtime combination visually. Text perplexity won’t tell you whether an image pipeline broke, so use diffusion-specific evidence such as SVDQuant and visual evaluation.

References

• JarvisLabs vLLM Benchmarks: JarvisLabs, vLLM Quantization Complete Guide and Benchmarks, 2026. JarvisLabs.
• Hugging Face Optimum Quantization Guide: Hugging Face, Quantization conceptual guide. Docs.
• vLLM Quantization Docs: vLLM project, Quantization. Docs.
• vLLM Quantized KV Cache Docs: vLLM project, Quantized KV Cache. Docs.
• vLLM Benchmark Docs: vLLM project, vllm bench serve. Docs.
• LLM Compressor Docs: vLLM project, LLM Compressor. Docs.
• GPTQModel: ModelCloud, GPTQModel. GitHub.
• TensorRT-LLM Quantization: NVIDIA, TensorRT-LLM Quantization. Docs.
• NVIDIA NVFP4: NVIDIA, Introducing NVFP4 for Efficient and Accurate Low-Precision Inference. Blog.
• Hugging Face QuantizedCache: Hugging Face, Cache strategies: Quantized cache. Docs.
• NVIDIA Model Optimizer: NVIDIA, Model Optimizer. GitHub.
• torchao Quantization: PyTorch, torchao quantization overview. Docs.
• bitsandbytes Quantization: Hugging Face, bitsandbytes. Docs.
• HQQ: Dropbox, Half-Quadratic Quantization. GitHub.
• PEFT: Hugging Face, Parameter-Efficient Fine-Tuning. Docs.
• Ollama: Ollama local model runtime. Website.
• LM Studio: LM Studio local AI runtime. Website.
• AutoGPTQ status: AutoGPTQ repository, archived April 2025. GitHub.
• AutoAWQ status: AutoAWQ repository, archived and deprecated May 2025. GitHub.
• GPTQ: Frantar et al., GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers, NeurIPS 2023. arXiv:2210.17323.
• Marlin: Frantar et al., MARLIN: Mixed-Precision Auto-Regressive Parallel Inference on Large Language Models, arXiv:2408.11743. arXiv:2408.11743.
• AWQ: Lin et al., AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration, MLSys 2024. arXiv:2306.00978.
• SmoothQuant: Xiao et al., SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models, ICML 2023. arXiv:2211.10438.
• QuaRot: Ashkboos et al., QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs, NeurIPS 2024. arXiv:2404.00456.
• SpinQuant: Meta AI Research, SpinQuant: LLM Quantization with Learned Rotations, arXiv:2405.16406. arXiv:2405.16406.
• QLoRA / NF4: Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs, NeurIPS 2023. arXiv:2305.14314.
• SVDQuant / Nunchaku: MIT HAN Lab, SVDQuant: Absorbing Outliers by Low-Rank Components for 4-Bit Diffusion Models, ICLR 2025. arXiv:2411.05007, Nunchaku.
• vLLM PagedAttention: Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention, SOSP 2023. arXiv:2309.06180.
• Grouped-Query Attention: Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, EMNLP 2023. arXiv:2305.13245.
• vLLM FP8 KV Cache: Kubler, Kurtic, Wilkinson et al., The State of FP8 KV-Cache and Attention Quantization in vLLM, vLLM Blog, April 2026. vLLM Blog.
• KIVI: Liu et al., KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache, ICML 2024. arXiv:2402.02750.
• KVQuant: Hooper et al., KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization, NeurIPS 2024. arXiv:2401.18079.
• LLM Serving Evaluation: Kurtic et al., “Give Me BF16 or Give Me Death”? Accuracy-Performance Trade-Offs in LLM Quantization, ACL 2025. arXiv:2411.02355.
• Long-context Quantization Evaluation: Mekala et al., Does quantization affect models’ performance on long-context tasks?, arXiv:2505.20276. arXiv:2505.20276.
• Reasoning Evaluation: Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Models, arXiv:2504.04823. arXiv:2504.04823.
• SlideSparse: SlideSparse: Fast and Flexible (2N-2):2N Structured Sparsity, arXiv:2603.05232v1. arXiv:2603.05232v1.
• HumanEval: OpenAI, HumanEval. GitHub.
• MMLU: Hendrycks et al., Measuring Massive Multitask Language Understanding. arXiv:2009.03300.
• MATH-500: Hugging Face H4, MATH-500. Dataset.
• GGUF and llama.cpp: ggml-org, GGUF file format and llama.cpp. GGUF, llama.cpp.
• Hugging Face GGUF Docs: Hugging Face, GGUF. Docs.
• Reference Repository: slavadubrov/model-compression-demo.