inference

LLM Serving and the Bus That Never Stops

In-flight batching is the trick that keeps LLM serving from wasting GPU seats.