LLM Serving and the Bus That Never Stops

I used to think batching requests for a machine learning model was a solved problem. I have hosted models and served requests before. Put a few requests together, send them through the GPU, and get better throughput. Simple. Without batching, a GPU is like a giant bus carrying one passenger. It still gets to the destination, but it is an expensive way to move one person. LLMs make this familiar problem weird because every passenger is going to a different stop.

The Idea

For something like an image classifier, batching is straightforward. You put requests in a queue, wait until there are 4 or 8 of them, run the batch through the GPU, and return the results. Each request takes a single trip through the model.

input -> model -> output

Embedding models work the same way. You pass in some data, the model runs once, and you get a result. An LLM does not quite work like this. It generates one token at a time.

prompt -> prefill
token 1 -> decode
token 2 -> decode
token 3 -> decode
...

So serving an LLM is not a single call to model(input). The server has to schedule the requests again after every token. If it treats them like ordinary inference requests, some requests finish early while the others keep the whole batch alive.

With LLM serving, the batch can change after every generated token.

Prefill, Decode, and the KV Cache

An LLM request has two phases. During prefill, the model reads the prompt and builds its attention state. During decode, it generates a token, feeds that token back into the model, and predicts the next one.

Decode continues until the model produces an end token or reaches a limit. Some answers take 10 tokens and others take 1,000. The server does not want to recompute the entire prompt and output history every time, so it keeps the attention keys and values in a KV cache on the GPU. This cache is fast, but GPU memory is finite. The scheduler has to juggle both compute and cache space.

Static Batching

Imagine three requests arrive together.

A -> needs 8 output tokens
B -> needs 2 output tokens
C -> needs 6 output tokens

Static batching puts them together and waits for the longest request to finish before admitting another batch.

Step 1: A B C
Step 2: A B C  # B is done
Step 3: A _ C
Step 4: A _ C
Step 5: A _ C
Step 6: A _ _  # C is done
Step 7: A _ _
Step 8: A _ _  # A is done

After B finishes, its position in the batch is useless. The same thing happens to C two steps later. No waiting request can take either position until A is done, so the batch becomes more empty with every step.

The Tour Bus vs. The City Bus

The way I understand the difference is to picture two kinds of buses.

A tour bus leaves with a fixed passenger list. If someone gets off at the second stop, that seat stays empty for the rest of the tour. The bus only takes a new group after it returns to the station. This is static batching.

A city bus keeps running the same route. People get off at each stop and new people take their seats. This is in-flight batching.

For an LLM server, the bus is the active batch. A seat represents GPU compute and some KV cache memory. A request gets off when it finishes generating. If there is enough memory, the scheduler can put a waiting request in that newly empty seat.

LLM serving intuition

The bus that changes passengers while moving

Each seat is a batch slot backed by GPU memory and KV cache. Each tick is one generation iteration.

In-flight batch

The bus is already moving, but the scheduler keeps swapping riders at token boundaries when capacity opens up.

3/3 active seats

requestA

7 tokens left

requestB

1 tokens left

requestC

5 tokens left

Token iteration

Bus stop queue

nobody waiting

0 idle seats in this toy loop

The important part is that the passenger list is no longer fixed. It can change between tokens.

In-flight Batching

In-flight batching is also called continuous batching or iteration-level batching. Instead of scheduling a whole request at once, engines such as vLLM and TensorRT-LLM schedule one generation step at a time.

After each step, the scheduler has a few decisions to make:

Which requests are still active?
Which requests just finished?
What new requests are waiting?
Is there enough KV cache space?
Should a waiting request join the next step?

It then builds the next batch from the requests that are still generating and any new requests it can fit. One forward pass generates one token for each active decode request, and then the scheduler gets another chance to rearrange the batch.

Real serving engines have more to worry about, including prompt prefills, priorities, multiple GPUs, and fairness. But this loop is the part I had been missing. The scheduling decision happens over and over, not only when a request first arrives.

Why It Matters

The obvious way to get more throughput is to increase the batch size. That works, but now a request may sit in a queue while the server waits to fill the batch. For a chatbot, that delay is noticeable. I do not mind watching an answer stream for a few seconds, but staring at an empty screen before the first token makes the application feel stuck.

The server is trying to produce more tokens per second while keeping that first-token delay reasonable. It also has to stay within the KV cache budget, and a long answer should not hold up every short one behind it.

In-flight batching helps because the server can fill a slot as soon as it becomes available. There is still a lot of difficult engineering in deciding which request gets that slot. But the basic trick is surprisingly simple: do not wait for the whole bus to come back when one passenger has already gotten off.