All Tools
P
LLMFreeOpen Source
PETALS
Run LLMs at home in a peer-to-peer swarm
MIT
ABOUT
Running 100B+ parameter models normally requires high-end GPUs or expensive cloud instances, making them inaccessible to most developers and researchers. Petals solves this by splitting models across a community-run peer-to-peer network, allowing inference and fine-tuning on consumer-grade GPUs or even free Google Colab instances without provisioning massive hardware.
INSTALL
pip install petalsINTEGRATION GUIDE
1. Generate text with distributed Llama 3.1 (405B), Mixtral, Falcon, or BLOOM models from home
2. Fine-tune or prompt-tune massive language models for custom tasks on consumer GPUs
3. Build chatbots and interactive applications backed by 100B+ parameter models cheaply
4. Execute custom inference paths through models and inspect hidden states for research
5. Deploy private swarms for sensitive data that should not leave your organization
TAGS
pythondistributedinferencefine-tuningp2pllm-servingbloom