All Tools
G
LLMFreeOpen Source
GPTQ
Post-training LLM quantization with GPU acceleration
MIT
ABOUT
Large language models require enormous GPU memory — a 70B parameter model in fp16 needs 140GB, far beyond a single consumer GPU. GPTQ (Post-Training Quantization for GPTs) compresses LLMs to 3-4 bits per parameter with minimal accuracy loss, reducing memory requirements by 4-5x. GPTQModel is the maintained, hardware-accelerated implementation supporting NVIDIA CUDA, AMD ROCm, and Intel GPU backends. It integrates with HuggingFace, vLLM, and SGLang for seamless deployment of quantized models on consumer and enterprise hardware.
INSTALL
pip install gptqmodelINTEGRATION GUIDE
1. Quantize a 70B Llama model to 4-bit and run it on a single A100 GPU
2. Deploy quantized LLMs on consumer GPUs (RTX 3090/4090) without losing significant accuracy
3. Apply INT3 quantization to reduce model serving cost in production by 4x
4. Integrate GPTQ quantized models with vLLM for high-throughput inference
5. Batch-quantize multiple model variants with per-layer sensitivity analysis for optimal compression
TAGS
quantizationcompressionllmgpuoptimizationinferencemodel-optimization