Open-Weight LLM Variants, Quantization, and Formats: Instruct, MoE, GGUF, GPTQ, and AWQ
Names such as Model-32B-A3B-Instruct-AWQ look dense because they combine several independent decisions: family and size, architecture, training role, and quantization. A repository may package those weights as sharded Safetensors, while a community conversion of the same checkpoint appears as Q4_K_M.gguf.
Those labels do not share one category: GPTQ and AWQ are quantization methods, GGUF is a container and runtime ecosystem, and MoE is an architecture. Reading the layers separately makes the download choice much easier.
TL;DR. Select a checkpoint on task quality, license, language, context, and interface behavior. Then select a runtime that supports its architecture. Only then choose a weight representation and quantization that fit measured memory and latency. GPTQ and AWQ are quantization methods, Safetensors and GGUF are containers, and MoE is an architecture—not a promise that all weights fit in “active parameter” memory.
Read a model artifact in six layers
| Layer | Example | Question it answers |
|---|---|---|
| Family and revision | Model-3.1, commit hash | Which weights and tokenizer contract? |
| Training role | Base, Instruct, reasoning-tuned, distilled | What behavior was optimized? |
| Architecture | Dense, MoE, total and active parameters | Which kernels and memory layout are required? |
| Numeric representation | BF16, FP8, GPTQ 4-bit, AWQ 4-bit | How are tensors represented or quantized? |
| Container and layout | Safetensors shards, GGUF | How are tensors and metadata packaged? |
| Runtime | Transformers, vLLM, llama.cpp | Which loader and hardware path execute it? |
These layers are not mutually exclusive. A checkpoint can be distilled, instruction-tuned, reasoning-tuned, and MoE at the same time.
“Open-source” also deserves care. Many downloadable models are open-weight under licenses that do not meet an open-source definition or that impose use restrictions. Read the model card and license before architecture or benchmark comparisons.
Training-role labels describe behavior, not capability guarantees
Base
A base checkpoint is trained primarily for next-token prediction. It is useful for continued pretraining, controlled research, or adaptation where you want to own the instruction behavior. It may complete a prompt rather than answer it.
Do not assume every fine-tune should start from base. An instruct checkpoint can be the better initialization when its existing behavior aligns with the target and your evaluation confirms that it does not fight the new objective.
Instruct or chat
These checkpoints receive post-training intended to improve instruction following and conversation. The exact recipe may include supervised fine-tuning, preference optimization, reinforcement learning, distillation, or combinations—not necessarily classic SFT plus RLHF.
Use an instruct checkpoint as the first assistant baseline. Confirm its chat template, supported tool-call format, system-message behavior, and refusal characteristics. “Instruct” alone does not guarantee reliable JSON or tool use.
Reasoning-tuned
Reasoning-oriented checkpoints are optimized on tasks or trajectories that reward multi-step problem solving. Some expose reasoning text, some separate it through a serving parser, and some present only an answer. Longer generation does not guarantee faithful reasoning or fewer hallucinations.
Adopt one when it improves the hard slices that matter after accounting for output tokens, latency, and verification. Routine extraction or classification may become slower without becoming better.
Distilled
Distillation transfers behavior from a teacher or teacher-generated data into another model. The student may be smaller, the same size, or structurally different. There is no stable “70–80% quality at half the size” rule: retained quality depends on teacher, data, objective, student capacity, and evaluation.
Treat Distill as provenance about training, then benchmark it like any other checkpoint.
Architecture labels describe execution
Dense models
Most parameters participate in each token’s forward pass. Parameter count is a rough indicator of weight storage, but runtime memory also includes KV cache, activations or workspace, allocator overhead, and sometimes duplicated or sharded state.
Mixture of Experts
An MoE layer routes each token to a subset of expert feed-forward networks. Names such as A3B often mean approximately three billion parameters are active per token, but naming conventions are family-specific. Read the model card for total parameters, active parameters, expert count, and routing design.
Active parameters mainly describe compute. Unless the runtime offloads experts, all expert weights still need storage and usually device memory across the serving topology. A 30B-total, 3B-active model does not automatically fit like a dense 3B model.
Runtime support is also architecture-specific. Confirm model implementation, expert-parallel or tensor-parallel support, quantization kernels, and maximum context before downloading.
Containers and quantization are different layers
Safetensors
Safetensors is a safe tensor serialization format commonly used in Hugging Face repositories. A model may have multiple .safetensors shards plus configuration, tokenizer, and generation files. Those tensors can be BF16/FP16 or pre-quantized with a method such as GPTQ or AWQ.
The extension alone does not tell you precision or runtime compatibility. Inspect config.json, quantization configuration, model card, tensor dtype, and runtime documentation.
GGUF
GGUF packages tensors and metadata for the ggml/llama.cpp ecosystem. llama.cpp requires GGUF and supports backends including Metal, CUDA, HIP, Vulkan, and CPU paths. Model architecture and conversion quality still determine compatibility.
GGUF is a container. It can hold high-precision or quantized tensors. Multimodal models may also need a separate projector or encoder file; “one GGUF file contains everything” is not a universal rule.
GPTQ and AWQ
GPTQ and AWQ are post-training weight-quantization methods, not file extensions. Their artifacts commonly use Safetensors plus method-specific configuration. Serving engines need kernels compatible with the method, bit width, group size, model architecture, and hardware.
Neither method is a universal quality winner. Calibration data, implementation, kernel path, and task matter. Current Transformers, TGI, and vLLM support several quantization backends, but their matrices change; verify the pinned runtime rather than relying on a static blog table.
Quantization math is a lower bound, not capacity planning
For (P) weight parameters at (b) bits, raw weight storage is approximately:
[ \text{weight bytes} \approx \frac{P \times b}{8} ]
A 13B model at nominal four-bit weights therefore starts near 6.5 GB. It will not necessarily run in 6.5 GB. Scales, zero points, higher-precision tensors, embeddings, metadata, runtime buffers, KV cache, and fragmentation add memory.
Context and concurrency can dominate the difference between “loads” and “serves.” Measure peak memory with the real maximum sequence, batch policy, cache dtype, and parallelism.
Quantization can reduce memory and sometimes improve speed, but low-bit kernels may also be slower on unsupported hardware. Compare task quality and end-to-end throughput—not file size alone.
Decode GGUF quantization names carefully
In Q4_K_M:
Q4indicates a four-bit quantization family, not exactly four bits for every tensor.Kidentifies the K-quant scheme.Midentifies a mixed recipe that uses different tensor types for selected weights. It does not mean “medium block size.”
The current llama.cpp quantization documentation shows that recipes can vary by architecture and tensor category. An importance matrix may also guide which weights retain more precision.
Avoid universal claims that Q4_K_M is indistinguishable from BF16 or that a larger Q3 model always beats a smaller Q8 model. Use a small ladder for the exact checkpoint:
- high-precision or trusted reference artifact
- one candidate near the memory limit
- one smaller candidate with more headroom
Run the same prompts, structured-output checks, long-context cases, and latency test on all three.
A selection workflow that survives new formats
1. Fix the task contract
Define language, modality, context length, tool or schema interface, safety constraints, license requirements, and evaluation slices. Compare checkpoints in a representation accurate enough to avoid quantization deciding the first round.
2. Select the checkpoint
Choose the smallest checkpoint that clears the non-negotiable quality and behavior gates. Record the exact repository and revision, tokenizer, chat template, and any required reasoning parser.
3. Select the runtime
Check architecture support, hardware backend, parallelism, quantization kernels, structured output, adapters, and operational interface. For local GGUF inference, llama.cpp is the reference runtime. For GPU serving, compare current vLLM, TGI, Transformers, or specialized engines against the actual artifact.
4. Establish a measured memory budget
Include weights, KV cache, runtime workspace, expected concurrency, and headroom for the operating system or co-located processes. A file fitting in RAM or VRAM is necessary, not sufficient.
5. Choose and validate a representation
Prefer publisher-provided artifacts with documented calibration and provenance. If using a community conversion, record the source revision, converter revision, quantization recipe, calibration or importance data, and hashes.
6. Benchmark the release unit
Measure task quality, schema/tool-call correctness, time to first token, output throughput, peak memory, and failures at target context and concurrency. Re-run after changing any checkpoint, runtime, kernel, or quantization setting.
Worked name
Suppose a repository is named:
Acme-32B-A3B-Instruct-AWQRead it as a set of questions:
Acme: which family, license, and revision?32B: total weights or another publisher convention?A3B: how does this family define active parameters?Instruct: which post-training recipe and chat template?AWQ: what bit width, group size, calibration, and supported kernels?- repository files: Safetensors shards, configs, tokenizer, and custom code?
- target runtime: does the pinned version support this exact architecture and quantization?
The name is an index into documentation, not a complete deployment specification.
Conclusion
Model selection becomes less confusing once labels stop sharing one bucket. Training role tells you what behavior was optimized. Architecture tells you how computation is organized. Quantization tells you how some tensors were approximated. Containers tell you how artifacts are stored. Runtimes determine what executes efficiently on your hardware.
Choose in that order, preserve exact provenance, and let one task evaluation compare release artifacts. A familiar suffix is not evidence that the model fits, runs fast, or retains the behavior you need.