All Tools
R
DataFreeOpen Source
REDPAJAMA
Open dataset with 30T tokens for LLM training
Apache-2.0
ABOUT
Training state-of-the-art large language models requires massive, diverse, and high-quality text data, but creating such datasets from scratch is prohibitively expensive and time-consuming. RedPajama provides a fully open dataset with 30 trillion tokens from diverse web sources, processed with deduplication, quality filtering, and comprehensive quality signals. It enables researchers and companies to train competitive LLMs without the data engineering burden, and its permissive Apache-2.0 license avoids legal restrictions that limit other open datasets.
INTEGRATION GUIDE
1. Pre-train large language models on high-quality open data without licensing restrictions
2. Combine RedPajama with domain-specific data for continued pre-training and domain adaptation
3. Use quality signal annotations to experiment with data filtering strategies for improved training
4. Benchmark model performance against a standardized, reproducible pretraining dataset
5. Augment retrieval datasets with diverse, deduplicated web text for RAG system training
TAGS
datasettraining-datallmopen-datapretrainingdeduplication