Daily

Introducing Saaras V4, our most capable speech to text model yet.

It delivers strong performance across both English and Indian languages, with better accuracy over a much wider range of speech.

Saaras V4 was built with specific attention to noise, accents, dialects and mixed-language speech.

Read more in the blog: links.sarvam.io/bqvvyS

We released Sonic-3.5 and Ink-2, the #1 streaming models for text to speech and speech to text you can use in your voice agents today.

New architectures enable new frontiers for speed and quality.

We're now the only provider to have #1 models for both speaking and listening.

Claude Sonnet 4.6 scores 100%, with a median TTFT of 850ms, on our standard LLM Voice Agent performance benchmark.

It's currently the fastest model that saturates this benchmark.

I also re-ran the numbers for the whole leaderboard, and Claude Haiku 4.5 scored 98% with a TTFT of 637ms. This puts Haiku in front of GPT 5.1 in the rankings, and a bit better in "intelligence" than GPT 4.1, but 100ms slower.

This is the first time we've had an Anthropic model that's a really good fit for most of our voice agent use cases. And now we have two!

Claude models have always had great instruction following, tool calling, and conversational dynamics. But they've been slower than the other SOTA models. That's changed.

One reason to re-run a benchmark like this is that latency changes. We continuously monitor latency for all the models we regularly use. But a specific run of a long-format benchmark like this is a bit different than our standard monitoring.

Another reason, though, is that models like Claude, Gemini, and the GPT family are hosted systems and they evolve. A good rule of thumb is that changes in model behavior are probably your own code rather than real changes on the provider side. But that's not always true. And this performance jump for Claude Haiku 4.5 over the past two months is dramatic.

I recently fixed some corner cases in tool call handling and improved the judging prompts in this benchmark. So I'll re-run Claude Haiku 4.5 against the benchmark code from 2 months ago, at some point, because I'd like to understand whether I previously had bugs that unfairly penalized Haiku. But either way, whether the model has gotten better or we've ironed out some issues with the benchmark, Haiku is impressive and is worth experimenting with if you are a voice AI developer.

Voice AI turn taking is a solved problem.

The single most common complaint about voice AI, today, is that agents interrupt too often. But the voice agents I build for myself now respond quickly and interrupt me less often than the people I talk to every day. (I actually measured this.)

@mark_backman made a @pipecat_ai PR two weeks ago that was the last piece of the puzzle for turn taking so good that I no longer ever think about it.

The approach combines three layers of processing:

1. Voice activity detection, with a short (200ms) trigger.
2. A native audio turn detection model that's small, fast, and runs on CPU. This model captures audio nuances like inflection and filler sounds that don't get transcribed.
3. A prompt mixin for the conversation LLM that decides turn completion based on conversation context.

None of these are new. We've been using VAD for a long time. We trained the first version of the Pipecat Smart Turn native audio model in December 2024. And we've been experimenting with prompt-based large model turn detection (sometimes called "selective refusal") for more than a year.

Now, the Smart Turn model and the SOTA LLMs we're using in voice agents have both gotten so good that using them together feels like we've finally "solved" turn detection.

Mark also figured out how to elegantly apply a "single-token tagging" technique to this problem. We sometimes use single-token tagging in place of tool calling, when we need a near-zero latency programmatic trigger. Mark's Pipecat mixin defines three single-token characters and prompts the LLM to output exactly one of them at the beginning of every response.

- ✓ means the agent should respond normally (immediately)
- ○ is a "short incomplete" - the agent should wait 5 seconds
- ◐ is a "long incomplete" - the agent should wait 10 seconds

The wait times, and the details of the prompt, are configurable, of course.

Watch the video to see me talk to an agent that handles all my various pauses and inflections, plus phrases like "let me think," pretty much the way a person would handle them, in terms of response latency.

Also, in the second half of the video, I ask the agent to adjust its response pattern because I'm going to tell it a phone number. This kind of "in-context" adjustment of response wait times is really useful.

The LLM in the video is GTP-4.1. We've tested the prompt and single-token adherance with GPT-4.1, Gemini 2.5 Flash, Anthropic Claude Sonnet 4.5, and AWS Nova 2 Pro. Note that older models in all these families (and, in general, smaller open weights models) aren't able to reliably output these single-token tags. But the new models we're using these days are pretty amazing.

Drop 5/14: Introducing Bulbul V3, our latest text-to-speech model. It raises the bar for how human it sounds, while being super robust.

In an independent third-party human listening study, Bulbul V3 delivers the highest listener preference, and low error rates across use-cases and languages. See details in our blog, but first watch the video. sarvam.ai/blogs/bulbul-v3

WeaveHacks is back!

Jan 31-Feb 1 at W&B HQ in SF.

This time we're building self-improving agents.

We've seen @GeoffreyHuntley's Ralph and @Steve_Yegge's Gas Town push the boundaries of what agents can do. Now it's your turn to build what comes next.

Details below. 👇

Just launched #CES2026, the new open-source NVIDIA Nemotron Speech ASR model is here to solve latency drift and redundant compute.

Its cache-aware streaming architecture eliminates the need for buffered inference, giving you stable, sub-100ms latency (24ms median T-T-F) and up to 3x more throughput on your GPU.

🤗 Read the technical blog with real-world results from @trydaily and @modal on @HuggingFace: nvda.ws/3Lt8m3Q

We're making Inworld TTS free until the end of the year (!)

We were feeling in the holiday spirit today, and after seeing the community rate our TTS at #1 on leaderboards and help us grow 100% week on week, we wanted to gift something and give every builder the chance to try what's topping the benchmarks.

Merry Christmas, Happy holidays. Inworld TTS is free this month.

Look out next week: we're going to redefine what #1 means.

Join us for InferSights by Simplismart and @pipecat_ai 🎉

Learn how to build real-time voice agents with Pipecat and optimize open weight models for fast, production-ready voice applications.

📅 Dec 11 | ⏰ 5 PM PT

Register here: luma.com/cvnyuvrq

Pipecat Thanksgiving day release. 🦃

Some highlights: Deepgram AWS SageMaker realtime speech-to-text support, improved text aggregation, simplified and more powerful error handling, new MiniMax Speech 2.6 HD and Turbo models.

SageMaker is AWS's AI platform for deploying and using machine learning models at scale. AWS has brand new support for streaming data in and out of models hosted on SageMaker, which is great for voice AI use cases. This Pipecat release includes a generic base class for SageMaker "bidirectional streaming," plus a new `DeepgramSageMakerSTTService` class.

Text aggregation and error handling are important fundamental jobs that a realtime agent framework needs to do well for the widest possible range of models, APIs, and use cases.

Different APIs chunk streaming text differently. For different use cases, you might want different aggregation strategies. (For example, feed one sentence of LLM output at a time to your voice generation service.) And managing multi-turn context as accurately as possible requires different strategies depending on what the APIs you are using can do. (For example, whether your TTS model can give you word-level timestamps or not).

Good error handling requires both managing the very different approaches to error handling that different services have, and giving developers good application-level ways to catch, respond to, and log errors.

The more services Pipecat supports, and the more different kinds of things people use Pipecat for, the more work these abstraction layers need to do! This Pipecat release includes several new text aggregation and error handling frame types and methods. The goal of these improvements is to make common use cases work better with less application-level code required, while also making it easier to build robust error handling for complex applications.

Finally, the MiniMax Speech models are getting great reviews. Thank you to the MiniMax team for the implementation!

What an insane workshop today!

The energy, the questions, the discussions, everything was on point.

Thank you to everyone who joined, learned, and pushed the limits with us!