All Tools
H
Dev ToolsFreeOpen Source
HAPROXY
High-performance load balancer for reliable traffic distribution
GPL-2.0
ABOUT
Model serving and inference infrastructure need to distribute traffic across multiple server instances to handle scale, maintain availability during failures, and route to different model versions. Without a capable load balancer, client-facing services become single points of failure and can't scale horizontally. HAProxy solves this by providing a battle-tested TCP/HTTP proxy that distributes traffic based on health checks, implements sophisticated routing rules, enforces rate limits, and provides detailed metrics — all with sub-millisecond per-request overhead and the ability to handle millions of concurrent connections on modest hardware.
INSTALL
brew install haproxyINTEGRATION GUIDE
1. Distribute inference requests across a pool of model server instances with health-check-aware load balancing
2. Route requests to different model versions based on URL paths or headers for A/B testing and canary deployments
3. Implement rate limiting and access control for public-facing LLM API endpoints
4. Terminate TLS for internal model serving infrastructure and forward plain HTTP to backend services
5. Provide a unified ingress point for multiple model serving frameworks (Triton, vLLM, TGI) with detailed request metrics
TAGS
load-balancingreverse-proxytraffic-managementhigh-availabilitynetworkinginfrastructure