LMSYS Org
@lmsysorg
Serving GLM5.2 NVFP4 Agentic Workload with SGLang: How We Reached 500 TPS on 8xB300 at bs=1
In this deep dive, we explain how SGLang reaches 500+ tok/s/user at bs=1 on 8xB300, with 18 to 34% higher single-user interactivity within two weeks since day-0, and 6 to 11% better peak
In this deep dive, we explain how SGLang reaches 500+ tok/s/user at bs=1 on 8xB300, with 18 to 34% higher single-user interactivity within two weeks since day-0, and 6 to 11% better peak
0:47 4 138