Banghua Zhu
@BanghuaZ
Learning from SGLang GOAT about deploying K3 lol
SGLang defaults with bf16 kv cache for better precision. If you test fp8 kv cache and find no accuracy drop in your environment, it will greatly speed up your inference.
SGLang defaults with bf16 kv cache for better precision. If you test fp8 kv cache and find no accuracy drop in your environment, it will greatly speed up your inference.
Byron Hsu@hsu_byron · Aug 1Scaling Kimi K3 on H200s with @sgl_project @lmsysorg
While most discussion on X focuses on Blackwell, I thought it would be useful to share what we learned while tuning Kimi K3 for throughput-oriented workloads on H200s.
- TP+EP beat DP+EP for K3. Only about 1/4 of its layers
Open quoted post →While most discussion on X focuses on Blackwell, I thought it would be useful to share what we learned while tuning Kimi K3 for throughput-oriented workloads on H200s.
- TP+EP beat DP+EP for K3. Only about 1/4 of its layers
1 134