All Tools
G
Fine-tuningFreeOpen Source
GPT-NEOX
Megatron plus DeepSpeed for large transformer training
Apache-2.0
ABOUT
Training GPT-scale models needs tensor and pipeline parallelism that a plain PyTorch loop does not provide. GPT-NeoX packages Megatron-style parallelism with DeepSpeed so research labs can train and continue pretraining large autoregressive models on multi-GPU clusters.
INTEGRATION GUIDE
1. Pretrain or continue-pretrain a GPT-style model on a GPU cluster
2. Study model-parallel training configs used for GPT-NeoX-20B
3. Fine-tune a large decoder model with DeepSpeed ZeRO and Megatron splits
4. Benchmark multi-GPU training throughput before moving to a custom trainer
TAGS
pythontrainingmegatrondeepspeedgptopen-source