Laatst bijgewerkt:
Full-precision LLMs are expensive to store, fine-tune, and serve. Quantization and parameter-efficient fine-tuning (PEFT) methods like LoRA make open-weight models practical for enterprise deployment. This guide explains what these techniques are, how they work together, and when to use each.
Quantization reduces the numerical precision of model weights and activations. Instead of storing each parameter as a 32-bit or 16-bit float, quantized models use 8-bit, 4-bit, or even lower precision representations. Benefits include smaller model files, lower GPU memory requirements, and faster inference on the same hardware. Trade-offs include potential accuracy loss. Aggressive quantization can degrade reasoning quality on complex tasks, though modern methods (GPTQ, AWQ, GGUF) preserve performance surprisingly well for many workloads.
FP16 / BF16: Half-precision training and inference. Standard for many production deployments. INT8 / INT4: Integer quantization for inference. Dramatically reduces memory at some quality cost. GGUF: Popular format for running quantized models locally via llama.cpp and similar runtimes. GPTQ / AWQ: Post-training quantization methods that optimize weight rounding to minimize accuracy loss. Teams choose formats based on hardware, latency targets, and acceptable quality degradation for their domain.
LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method. Instead of updating all billions of model weights during fine-tuning, LoRA adds small trainable adapter matrices to selected layers. The base model weights stay frozen. Only the LoRA adapters learn task-specific or domain-specific behavior. Benefits: Train adapters with a fraction of the compute and memory of full fine-tuning. Store many domain adapters for one base model and swap them at serving time. Reduce catastrophic forgetting of base model capabilities. LoRA is the most widely used PEFT method in open-weight LLM customization.
PEFT (Parameter-Efficient Fine-Tuning) is the broader category of methods that adapt large models by training only a small subset of parameters. LoRA is the most common PEFT technique. Others include: Prefix tuning: Learnable prefix tokens prepended to inputs. Adapter layers: Small bottleneck modules inserted between transformer layers. Prompt tuning: Soft prompts optimized in embedding space. PEFT makes domain customization accessible to teams without the budget to fully retrain frontier-scale models.
Production stacks often combine both: 1. Start with an open-weight base model. 2. Fine-tune LoRA adapters on domain data. 3. Merge adapters or serve them alongside the base weights. 4. Quantize for efficient inference on target hardware. Platforms like Fireworks, vLLM, and local runtimes support multi-LoRA serving: one quantized base model with many domain adapters loaded in parallel. For sovereign enterprise deployment, this stack enables inspectable open weights, domain specificity, and cost-efficient serving without retraining from scratch.