All Tools
F
Fine-tuningFreeOpen Source
FLASH LINEAR ATTENTION
Hardware-efficient linear attention kernels for state-space models
MIT
ABOUT
Linear attention and state-space models (Mamba, GLA, DeltaNet) offer theoretical efficiency over standard quadratic attention, but their custom CUDA kernels are difficult to implement and optimize. Flash Linear Attention provides a unified collection of Triton-based kernels for these emerging architectures — Mamba-2, GLA (Gated Linear Attention), DeltaNet, and more — with hardware-aware optimizations for modern GPUs. Each kernel is benchmarked and tuned for throughput and memory efficiency, reducing development time for researchers and engineers working on sub-quadratic attention mechanisms.
INSTALL
pip install flash-linear-attentionINTEGRATION GUIDE
1. Train a linear attention language model with the Mamba-2 kernel for faster sequence processing
2. Replace the standard attention in a transformer with GLA kernels for improved long-context efficiency
3. Benchmark linear attention variants against softmax attention for a specific GPU architecture
4. Integrate DeltaNet recurrent kernels into an existing transformer training pipeline
5. Experiment with hybrid attention architectures combining linear and quadratic attention layers
TAGS
attentiontransformerkerneltritonmambastate-space-modelcudaoptimization