All Tools
P
LLMFreeOpen Source
PYTHIA
Suite of interpretability-focused LLMs for studying training dynamics
Apache-2.0
ABOUT
Studying how large language models acquire knowledge during training is difficult without access to intermediate training checkpoints and fully reproducible training setups. Pythia solves this by releasing 16 models at various scales with 154 checkpoints each — all trained on identical data in the same order — providing a standardized research platform for studying learning dynamics, memorization, and emergent behaviors across model sizes and training time.
INSTALL
pip install transformersINTEGRATION GUIDE
1. Study how knowledge and capabilities emerge during LLM training using 154 intermediate checkpoints per model
2. Benchmark interpretability techniques across 16 model scales from 14M to 12B parameters
3. Investigate memorization, bias, and safety properties of language models throughout the training process
4. Reproduce and extend interpretability research with fully open data, models, and training code
TAGS
llmtext-generationopen-sourceinterpretabilityresearchtransformerstraining-dynamics