Session abstract
What you’ll learn
You shipped your first AI feature. It worked. Then traffic grew, and suddenly your $4/hour GPU is sitting idle most of the time, your p99 latency is awful, and your cloud bill is unhinged. What happened? This talk is about the layer between your model and your users — the inference serving layer that's quietly become one of the most interesting infrastructure problems of the AI era. We'll start with the naive solution every team reaches for first (static batching), see exactly why it falls apart under real load, and then unpack the elegant insight behind continuous batching — the technique powering vLLM, TGI, Baseten, and every serious inference platform today. By the end you'll know: - Why a GPU running an LLM is idle far more than it's working - The difference between static and continuous batching, and why one wins - A mental model for PagedAttention and how it makes any of this possible - When this matters (you're self-hosting) and when it doesn't (you're calling an API) No PhD required. If you've ever called an LLM API and wondered what's behind the curtain, this talk is for you.