All Tools
G
Fine-tuningFreeOpen Source
GALORE
Low-rank gradient projections for cheaper LLM training
Apache-2.0
ABOUT
AdamW stores optimizer states that dwarf the model weights, so full finetuning OOMs on a single GPU. GaLore projects gradients to a low-rank subspace, cutting optimizer memory while staying closer to full-rank training than parameter-efficient adapters alone.
INSTALL
pip install galore-torchINTEGRATION GUIDE
1. Full-parameter finetune a 7B model on fewer GPUs by shrinking optimizer state
2. Combine GaLore with 8-bit optimizers for even tighter memory budgets
3. Compare GaLore against LoRA on the same downstream task
4. Train larger models on a workstation that previously only fit LoRA
TAGS
pythonpytorchoptimizerloramemorytrainingopen-source