All projects

Project · 2026

LLM Gateway

An OpenAI-compatible reverse proxy in Go with unbuffered SSE streaming and per-key rate limiting.

View on GitHub

Highlights

  • Added weighted replica failover that retries only before the first byte; client disconnects cancel the upstream request.
  • Zero errors at 6,000 req/s with 6.7 ms p99 and 0.5 ms added overhead; race-clean, all benchmarks reproducible.

Stack

  • Go
  • net/http
  • SSE
  • Prometheus