All Tools
F
Fine-tuningFreeOpen Source
FAIRSCALE
PyTorch extensions for large-scale training
BSD-3-Clause
ABOUT
Fitting large models on limited GPUs needs sharding that vanilla PyTorch used to leave to research forks. FairScale brought FSDP, pipeline parallel, and ZeRO-style optimizer sharding into reusable PyTorch APIs so teams can scale training without a full Megatron stack.
INSTALL
pip install fairscaleINTEGRATION GUIDE
1. Wrap a PyTorch model in FSDP to shard parameters across GPUs
2. Pipeline-parallel a deep transformer when a single GPU cannot hold it
3. Shard optimizer state to cut peak memory during LLM finetuning
4. Prototype distributed training before moving to TorchTitan or Megatron
TAGS
pythonpytorchfsdpdistributedtrainingopen-source