All Tools
L
RAGFreeOpen Source
LLMLINGUA
Compress prompts 20x without losing quality
MIT
ABOUT
Long prompts waste tokens and increase LLM latency and cost — a 10,000-token prompt with verbose context and task descriptions contains significant redundancy. LLMLingua uses a small language model to identify and remove superfluous tokens and phrases, compressing prompts by up to 20x with minimal accuracy loss. It also compresses the KV cache for even greater speedups during generation.
INSTALL
pip install llmlinguaINTEGRATION GUIDE
1. Compress retrieval-augmented generation contexts before passing them to an LLM for faster answers at lower cost
2. Reduce prompt engineering budget by stripping verbose instructions while keeping task semantics intact
3. Speed up batch inference on long-document tasks like summarization and question answering
4. Compress multi-turn conversation histories to fit within context windows without losing state
TAGS
pythonprompt-compressionllminference-optimizationragtoken-efficiency