Banghua Zhu

@BanghuaZ

Cofounder & CTO @radixark

Seattle, WA banghua.me Joined August 2018

8.9KFollowers

1.5KFollowing

1.1KPosts

14Media

6.4KLikes

@BanghuaZ reposted

@sgl_project

We brought SGLang to NVIDIA Vera Rubin and accelerated Kimi K3 inference.
Working closely with @NVIDIA, we optimized attention, MoE, and speculative verification kernels on early-access Rubin hardware.

Highlights:
• Up to 20% faster FP8 MLA at batch 1 / 128K context
• 20% faster KDA verification, with bitwise-identical output
• 5.9% end-to-end inference speedup from MoE tail fusion, removing 276 kernel launches per decode step

SGLang also powers rollouts for Miles' end-to-end RL training on Rubin, including agentic RL with 64 concurrent sandboxes on the Vera CPU.

Full results and engineering details 👉 lmsys.org/blog/2026-10-0…

14 21 9 132 23.9K 44

@BanghuaZ reposted

@ChengWan17

We just landed layer_boundary in @sgl_project. The biggest win: feature compatibility across models.

Instead of re-implementing TP/DP/CP communication, residual handling, token movement, and normalization for every model, we put these semantics behind a shared stage boundary.

Now features like Attention DP × TP × CP can be integrated once at the boundary level and reused across many model architectures.

Less model-specific plumbing. Much broader feature compatibility.

5 20 2 87 5.4K 26

@BanghuaZ reposted

@sgl_project

Meet the SGLang Summit 2026 speakers.
🗓️ Nov 12–13 · 📍 Fort Mason, San Francisco
🎟️ Early bird pricing ends Oct 20: sglang.io/summit

Two days to connect with the people building frontier AI infrastructure, from silicon to models and apps. Main stage keynotes, plus breakout deep dives and hands-on labs you can only catch in person.

Featuring @LipBuTan1 (Intel), @lilianweng (OpenAI), @soumithchintala (Thinking Machines), @AravSrinivas (Perplexity), Bill Jia (Google), @ctnzr (NVIDIA), @luisceze (NVIDIA), Ujval Kapasi (NVIDIA), @ibab (River AI), @ericzelikman (humans&), @tachim (Harmonic), @marksaroufim (Core Automation), @tydsh (Recursive), Qi Jin (Cerebras), @cHHillee (Thinking Machines), @SimonShaoleiDu (Apodex), @zianglih (humans&), @hsu_byron (Periodic Labs), @JiantaoJ (NVIDIA), Vinay Kumar (DigitalOcean), @lm_zheng (Meta), @ying11231 (RadixArk), @BanghuaZ (RadixArk) and more.

0 8 3 50 64.7K 7

@BanghuaZ

New benchmark on inference built out of SGLang! Congrats @JiantaoJ and team!

Jiantao Jiao@JiantaoJ · Sep 23

🚀 New benchmark alert: SWE-Serve (research.nvidia.com/benchmarks/swe…)

Can AI agents develop inference engine and make it serve real models?

Our team at @nvidia built 53 tasks from real @sgl_project engineering work to find out. The chart shows why live-serving tests matter. 🧵

Open quoted post →

1 2 0 57 7.1K 10

@BanghuaZ

Scaling law has unlocked capability in math & coding, and now materials science through high-throughput labs. Really an exciting time to live!

Thrilled that SGLang and Miles help power Neon in inference and training, and to see Neon surpass GPT-6 Astra on material science!

Congrats to @LiamFedus, @hsu_byron, @vwxyzjn, and the entire @periodiclabs team! 🚀

Liam Fedus@LiamFedus · Sep 15

We built high-throughput materials labs in Menlo Park to create a loop between experiments and models. The labs generate fresh data, the models learn from it, and then help us decide what to try next.

Using only 1,300 H200s, plus months of our experimental data, we mid-trained and RL’d an open-source model to surpass GPT-6 Astra on our analysis benchmark. We call it Neon.

This is real footage from our lab. We’re focusing first on hard problems in materials science, including superconductors, magnets, and semiconductor materials.

Read our blog posts below.

0:16

Open quoted post →

2 6 1 59 10.5K 8

@BanghuaZ

Amazing results. Congrats @zhijianliu !

Zhijian Liu@zhijianliu · Sep 3

Hmm, another inference company topping the leaderboard?

Oh wait, that's us! Speed of light on Artificial Analysis for four major open-source models. ⚡️

Open quoted post →

1 0 0 14 5.5K 3

@BanghuaZ

The first language model from Rednote dots studio lab! (Yes, rednote from China is also starting to release open model lol.)

dots studio@dotsstudioai · Aug 14

Introducing dots3-note preview — a small but mighty step toward long-horizon agency in real life.
🔹 280B MoE with 16B active parameters, a 512K context window, and multimodal understanding across text, vision, and audio
🔹 Introduces TEMPO, a new RL approach for long-horizon agent training through self-critiquing and test-time-scaled value estimation
🔹 Built to reason, explore unfamiliar environments, update memory over time, and combine multimodal perception with coding and tool use to solve complex tasks
🔹 Open weights on Hugging Face, alongside two open benchmarks for real-life agents: VibeSearchBench and VibeLifeBench
Competitive with much larger models across reasoning, agentic, and multimodal evaluations.
🔗 Tech blog: studio.dots.ai/dots/dots3-en.…
🔗 Model weights: huggingface.co/dots-studio/do…
🔗 Github: github.com/studio-dots-ai…

Open quoted post →

2 0 0 57 7.3K 10

@BanghuaZ

Congrats @MiniMax_AI on the new music generation model! Very cool to see more modality being introduced in open source. You can serve Music3 now with SGLang at day-0!

MiniMax (official)@MiniMax_AI · Aug 13

🎵MiniMax-Music3
Next-Generation Open-Weights Production-Ready & Versatile Music Model

huggingface.co/MiniMaxAI/Mini…

Open quoted post →

1 1 0 13 2.2K 2

@BanghuaZ

Great to see really strong video generation model being open sourced! Congrats @MiniMax_AI!!

LMSYS Org@lmsysorg · Aug 3

@MiniMax_AI H3 is live in SGLang Diffusion, with day-0 serving support 🎬

This open model matches Seedance 2.0 at 1/3 the cost, or $0 if you run it locally on 2x 5090 or 1 RTX 6000.

With SGLang Diffusion, you can build visual concepts, motion design, e-commerce creatives, precise video edits, animation and stylized visuals, all locally on your own machine. No API bill, no waitlist.

H3 takes text, image, video and audio in a single context, and generates 5-15s clips at native 2K, 24fps with native stereo audio.

SGLang Diffusion also runs on @NVIDIAAI Blackwell and Hopper, and @AMD MI355X / MI300X.

Let's create something with MiniMax H3 👇

0:31

Open quoted post →

6 1 0 30 3.5K 6

@BanghuaZ

Learning from SGLang GOAT about deploying K3 lol

SGLang defaults with bf16 kv cache for better precision. If you test fp8 kv cache and find no accuracy drop in your environment, it will greatly speed up your inference.

Byron Hsu@hsu_byron · Aug 1

Scaling Kimi K3 on H200s with @sgl_project @lmsysorg

While most discussion on X focuses on Blackwell, I thought it would be useful to share what we learned while tuning Kimi K3 for throughput-oriented workloads on H200s.

- TP+EP beat DP+EP for K3. Only about 1/4 of its layers use gated MLA, so DP attention provides less benefit from avoiding KV-cache duplication. Having each DP rank process every attention head also substantially increased latency. This differs from our K2.6 and GLM setups, where we generally prefer DP+EP.

- FP8 KV cache worked well. We measured no quality or throughput regression in our evaluations, while effectively doubling KV-cache capacity.

- TP32+EP32 provided significantly more cache capacity: approximately 4M KV-cache tokens per replica, compared with about 800K using TP16+EP16.

- For low cache-hit workloads, consider disabling radix cache. Prefix caching requires four Mamba-state slots per sequence in our setup; without radix cache, each sequence needs only one.

- Non-speculative decoding won at high concurrency. Above roughly 64 concurrent requests, draft-and-verify overhead outweighed the savings from accepted speculative tokens in our tests.

- A few caveats: symmetric memory caused instability in our multi-node environment. We also observed hangs when combining speculative decoding, HiCache, and high KV pressure. Both issues are still under investigation.

We’ve been running this configuration at full load across roughly ~O(1000) chips for several days. Aside from hardware failures, it has been reliable.

Feedback, discussion, and better configurations are very welcome! Sharing my launch command in the first comment.

h/t to @hanming_lu @liin1211 @0xishand @ChengWan17 @NealBayya for the discussion

Open quoted post →

1 10 0 134 15.6K 59

@BanghuaZ

We've received a lot of feedback around the K3 DSpark perf degradation for long context cases, now we present a new DSpark checkpoint with support up to 1M context without degrading acceptance length!

1 3 0 23 2K 1

@BanghuaZ

Excited to work with Google cloud @googlecloud on bringing SGLang to TPU!

RadixArk@radixark · Jul 30

RadixArk and Google Cloud are joining forces with the SGLang community to make TPU a drop-in, cost-efficient path to frontier inference.

SGL-JAX already serves the major open model families on the latest TPU generations: Gemma, Qwen, DeepSeek, GLM, Kimi, Ling, MiniMax, MiMo, Grok, plus Wan and Flux for video and image generation.

With this partnership, TPU unlocks SGLang's production feature set: 5D parallelism, Radix Cache, HiCache, quantization, speculative decoding, on TPU Pallas kernels co-built by Google, RadixArk, and the SGLang community. SGL-torchtpu opens a PyTorch-native path to all of it.

SGLang on TPU comes with the same API and the same features, so for teams already running it, getting started is simple. Going forward, TPU support for new open models will land alongside the rest, and that carries forward to each new TPU generation.

Full announcement and SGL-JAX repo 👇

Open quoted post →

6 0 0 44 3.4K 5

@BanghuaZ

Congrats on the seed round and new model release from @FishAudio team! SGLang community are proud to have you and support you!

Fish Audio@FishAudio · Jul 28

Today we’ve raised $52M Seed and we are announcing the public launch of S2.1 Pro.

>It can clone a voice from 5 seconds of audio
>2x faster than Cartesia & 1/6th the cost of Eleven Labs
>most expressive model with word level control over emotion, intonation, pacing etc

We support frontier AI companies including HeyGen, LiveKit, Retell, Sanas, and OpenArt all run our model in production.

If you're a business and we can't cut your voice AI costs by 50%, we'll give you 1 year of Fish Audio for free.
Book a demo: s.fish.audio/tmapke

To celebrate our first birthday, we'll give you 1 month of S2.1 Pro for free. Like, retweet, and comment “Fish” to get it.

2:24

Open quoted post →

0 0 0 21 3.4K 2

@BanghuaZ

Congrats @jiayq for coming out of stealth! In the era of agent, the definition of software might change drastically. Would be amazing to see how one can turn people’s intent into reliable software system. And congrats on the achievement of GLM 5.2!

Yangqing Jia@jiayq · Jul 28

We are starting Intent Lab, building an autonomous team we call "fleet" that turns intent into production software. Today we are sharing some early results: the fastest GLM5.2 inference engine, one shot database creation, and a fully verified agent filesystem.

intentlab.ai/blog/turn-your…

Open quoted post →

1 0 0 37 7K 6

@BanghuaZ

SGLang delivers a good speed for K3 at day-0. Behind it were many sleepless nights across the SGLang ecosystem, and an incredible relay race across the ecosystem.

@nvidia and @AMD put serious engineering power into kernels and hardware enablement. @KVCache_AI pushed support across different hardwares, PD disaggregation, and HiCache.

Then @modal jumped straight into co-development and trained their ridiculously fast Dflash model. (They’ve already reported 460 tok/s. Huge congratulations!)

@baseten tested the SGLang implementation, gave us invaluable feedback, and submitted PRs to improve tool calling. @digitalocean provided compute when we needed it.

Meanwhile, @Kimi_Moonshot, @googlecloud, @nebiustf, @fal, @runpod, @DeepInfra, @gmi_cloud, and many others we can’t name yet were racing to test, optimize, and support Kimi K3 with SGLang.

That’s the magic of open source: not one team doing everything, but many teams picking up the baton and running with it.

And for a little extra fun, we used K3 to create the entire launch video and its companion game. AI does make life much easier lmao.

LMSYS Org@lmsysorg · Jul 27

SGLang day-0 speed on Kimi K3: 423 tok/s (measured on gsm8k), plus RL support ready in Miles @radixark!

How the largest open-source model runs this fast: we natively implemented and deeply optimized K3’s new architecture with fused KDA decode kernels, DP attention, DSpark, PD disagg, and KDA-aware prefix caching. We've passed Kimi Vendor Verifier and are ready for production!

Thanks to @Kimi_Moonshot, @nvidia, @AMD, @KVCache_AI, @modal, and @baseten for building this with us, and to @googlecloud, @nebiustf, @fal, @digitalocean, @runpod, @DeepInfra and @gmi_cloud for serving K3 on SGLang.

Blog, cookbook, benchmarks in the comments.

P.S. This demo video? Kimi K3 made it itself. Play the game 👇

0:53

Open quoted post →

4 4 0 46 27K 5

@BanghuaZ

@lmsysorg and @radixark have co-signed the Open Weights letter. We believe that AGI should not be built a thousand times behind a thousand walls.

I’ve had the privilege of working with @JensenHuang and the Nemotron team, and I’ve seen firsthand how relentlessly NVIDIA pushes open AI forward, from open pre-training and post-training datasets, to Nemotron and Cosmos, and now the Nemotron Coalition.

To me, the logic is simple: if AGI can benefit all of humanity, we should build as much of it as possible together. When every team rebuilds the same foundations in isolation, we waste talent, compute, energy, and time. When we share models, data, systems, and lessons openly, every breakthrough becomes a starting point for thousands of others.

This is also why @lmsysorg and @radixark exist: to reduce duplicated work, bring resources together, and push the open frontier forward, from Vicuna, @arena, FastChat, @sgl_project , FlashInfer, and SpecForge, to Miles and the open-source projects still to come.

Jensen Huang@JensenHuang · Jul 24

For my first post, I’m sharing a letter @NVIDIA signed on why open models matter.

AI will transform every industry, power every company, and be built by every country.

Open models strengthen safety and cybersecurity, accelerate innovation and diffusion, and enable sovereignty.

The world needs both frontier closed models and frontier open models.

images.nvidia.com/pdf/Open-Weigh…

Open quoted post →

4 5 0 55 51.1K 6

@BanghuaZ

New Dspark checkpoint for Inkling is here!

RadixArk@radixark · Jul 22

We trained a DSpark speculator for Inkling NVFP4, built end-to-end with SpecForge on a live SGLang target engine.

- 1.89x decode throughput over non-spec at bs=64 (8xB200, TP8)
- 7–14% faster than Inkling's built-in MTP at the same batch size
- 3.66 mean accept length, up to 4.76 on GSM8K

Under the hood:
1️⃣ 5-layer Qwen3-style DFlash parallel-draft backbone
2️⃣ Rank-256 Markov logit-bias head for intra-block dependencies
3️⃣ Per-position confidence head predicts acceptance
4️⃣ Distilled online from 400K Inkling regenerations

Try it out now! Serving command in the comment.

Open quoted post →

0 3 1 17 2.7K 5

@BanghuaZ

Big fan of TML’s mission to build AI that extends human will and judgment, and reimagine how humans and AI collaborate.

With Inkling, released with full weights and built for customization, we see a great future of US open models. Congrats, @thinkymachines!

Thinking Machines@thinkymachines · Jul 15

Today, we are introducing Inkling.

Inkling reasons efficiently across text, image, and audio modalities. We are making the full weights available.

thinkingmachines.ai/news/introduci…

Available today for fine-tuning on Tinker. Play with it in the Inkling Playground. 🧵

Open quoted post →

0 2 0 46 1.7K 2

@BanghuaZ

You probably didn’t have Thinking Machines Lab dropping a 975B open model on your bingo card.

Inkling takes text, images, and audio as input, and it looks insanely strong.

SGLang already provides broad support for Inkling: PD disaggregation, multi-LoRA serving, HiCache, plus a DFlash checkpoint trained by @modal. Thanks for the amazing collaborations @thinkymachines @radixark @sgl_project.

Enjoy 🚀

LMSYS Org@lmsysorg · Jul 15

Inkling, @thinkymachines' first open model, dropped today: 975B total / 41B active MoE, up to 1M context, reasoning natively over text, images, and audio.

Serving and RL support are already live: you can run and shape it on an open stack, starting now.

Day 0 support on SGLang @sgl_project and Miles @radixark👇

- Inkling's new architecture (ShortConv, attention with relative positional embedding, shared expert sink MoE) is natively implemented and deeply optimized, with prefill full CUDA graph and MXFP8 KV cache

- Full parameter and LoRA RL in a customized Megatron backend, train inference consistency via customized kernels, routing replay, and cross-runtime parameter synchronization

- DFlash speculative decoding from @modal for low-latency serving

Launching now, blog and cookbook in the comments ⬇️

1:01

Open quoted post →

2 14 1 97 9.9K 10

@BanghuaZ

Since the release of GLM 5.2, SGLang team have been constantly working on pushing the performance to the limit, building in the open, and sharing it with everyone. Now you can achieve 535 token/s with latest version of SGLang!

Those won't happen without great efforts and support from @radixark @NVIDIAAI and of course @zai_org. Huge respect to @jietang and team for bringing such an amazing model to the world!

LMSYS Org@lmsysorg · Jul 14

Serving GLM5.2 NVFP4 Agentic Workload with SGLang: How We Reached 500 TPS on 8xB300 at bs=1

In this deep dive, we explain how SGLang reaches 500+ tok/s/user at bs=1 on 8xB300, with 18 to 34% higher single-user interactivity within two weeks since day-0, and 6 to 11% better peak throughput at high concurrency, benchmarked on a real multi-turn agentic coding workload. Our new TopK-V2 kernel is 2.33x faster at 80K ISL, scaling to 10.17x at 1M ISL, keeping interactivity essentially flat out to 1M tokens.

Part of the story is the architecture itself. GLM-5.2 applies IndexShare to its DSA layers and ships a stronger MTP head reusing IndexShare and KVShare. The rest comes from our serving optimizations.

Special thanks to @NVIDIAAI for the help in day-0 support of GLM-5.2 NVFP4, and to @Zai_org for IndexShare in SGLang!

0:47

Open quoted post →

2 4 0 59 6.3K 13

Banghua Zhu (@BanghuaZ) — posts, photos & videos | PeekX