Why your LLM API is slow: the hidden trade-off between broker and client parallelism

The LLM API performance trade-off is a critical factor in application design. Developers often optimize for throughput by increasing the number of parallel requests to the API server. While this improves server-side throughput (requests per second), it increases latency (time per request) due to resource contention on the server. The client must wait longer for each request to complete, even though the server is handling more requests overall. \n\nWhy this matters for us: Brown developers and teams often face this trade-off when building apps that use LLM APIs. Optimizing for throughput can lead to slower user experiences and unhappy customers. The fix is simple: profile the API's performance and adjust the concurrency level based on the application's requirements (throughput or latency). But many teams don't profile and end up with suboptimal performance.