Serving large language models (LLMs) in production can incur substantial costs, which has prompted recent advances in inference system optimizations. Today, these systems are evaluated against conventional latency and throughput metrics (eg. TTFT, TBT, Normalised Latency and TPOT). However, these metrics fail to fully capture the nuances of LLM inference, leading to an incomplete assessment of user-facing performance crucial for real-time applications such as chat and translation. In this talk, we will discuss the pitfalls of current performance metrics in evaluating LLM inference systems and then discuss Etalon, a comprehensive performance evaluation framework that includes fluidity-index - a novel metric designed to reflect the intricacies of the LLM inference process and its impact on real-time user experience.
Amey Agrawal is a Ph.D. student at Georgia Tech advised by Prof. Alexey Tumanov, specializing in systems for foundation models. His work on Sarathi-Serve and Vidur have seen significant industry adoption along with publications at OSDI and MLSys. Amey’s research aims to make large language models more accessible and cost-effective. Previously, he contributed to Microsoft’s planet-scale AI infrastructure project. His innovations span from improving GPU utilization to developing holistic LLM evaluation frameworks, driving forward the field of AI systems.