Efficient Request Queueing – Optimizing LLM Performance

Serving LLMs to many applications and users in parallel is challenging because they compete for limited GPU resources. This article is the first in a series on LLM performance, based on our experience with serving self-hosted LLMs at TNG Technology Consulting GmbH. In the first part, we focus on the impact of queuing and discuss different scheduling strategies.

An inference engine like vLLM or HuggingFace TGI consists of

Why do we need a queue here? Because calculations on the GPU are more performant and resource-efficient when they are done batch-wise instead of isolated for individual requests. This backend queue allows the scheduler to pick multiple requests and put them on the same batch to be processed.

Recently, vLLM added a priority-based scheduling option instead of sticking with FIFO. For this new feature, requests are tagged with a priority before they are sent to the backend. And vLLM would regularly check if any higher-priority request is queuing and then sort all requests by priority (both the running batch and the waiting queue). This would not only let the high-priority request literally jump the queue but even move it directly to the processed batch. The price is (aside from some overhead due to sorting) that lower priority requests would be evicted from the running batch and put back in the waiting queue again.

vLLM understands the request priority as a continuous number. This does not only allow you to distinguish between default, high priority (interactive AI assistant) and low priority (batch API) but you could even make smaller decrements for every request that the user has already sent to the backend and where the response is pending. For example, users B and C send only single requests, which will have priority zero. User A sends four requests at once; the first one has priority zero, too, but the next ones will have priorities one, two, and three, and will be processed later (higher number = lower priority).

The solution can also be extended by having not only one queue per user (and model) but several queues with different priorities . For example, interactive applications like TNG's AI Assistant with chat interface should have higher priority because users who don't see any progress within five seconds will think the application is broken. Users who generate code reviews for tens of files and several thousand lines of code, however, will expect the LLM requests to take some time. And some use cases (like benchmark runs, scheduled via a batch API) should have such low priority that they don't disturb other use cases and are just scheduled when nothing else is running.

Note that typically each inference engine serves only a single model, and we have multiple deployments running for different models in parallel.

When a single user "A" sends a large number of requests, they will quickly fill up the queue. Other users ("B" and "C") that send their requests only shortly after are blocked from using the same model (until all requests from "A" have been processed). Note that the picture focuses on vLLM as inference engine, but the problem is more general and applies to other backends as well.

At TNG, users don't send requests directly to the vLLM backend but to an API Server (what we call "LLM-Server"). Here we can have separate queues for each user (and model), and a scheduler that is not FIFO (first in - first out) but goes round-robin through all user queues. This achieves some "fair scheduling" : for example, in the diagram users B and C send their request a bit later, when user A's first request has already been scheduled. At that time, three requests from user A had been waiting already for some time, but user C has to wait for only one request from user A to be completed.