key routing persists across restarts

2026-07-21 16:13:35 -05:00
parent d59673e0c5
commit 0755fb9d78
6 changed files with 49 additions and 7 deletions
@@ -37,7 +37,7 @@ The global context limit defaults to 256k tokens and can be changed with `MAX_CO

 Inference requests are limited per client IP to 2 requests per second, 100 requests per five hours, and $15 of reported upstream cost per five hours. Railway's `X-Real-IP` header is used to identify clients.

-Set `REDIS_URL` to persist and share these limits across process restarts and multiple proxy instances. The Redis backend stores only the active five-hour window and uses atomic operations, so concurrent instances enforce one shared limit. Without `REDIS_URL`, limits remain in memory as before.
+Set `REDIS_URL` to persist and share these limits across process restarts and multiple proxy instances. The rate-limit backend stores only the active five-hour window and uses atomic operations, so concurrent instances enforce one shared limit. Redis also keeps the upstream API-key round-robin cursor in a counter, so routing continues across restarts and is shared by every proxy instance. Without `REDIS_URL`, both limits and key routing remain in memory as before. Use `REDIS_KEY_ROTATION_KEY` to change that counter's key when sharing a Redis database.

 ```sh
 REDIS_URL=redis://localhost:6379