Paddy Srinivasan
@paddix
Your TTFT benchmark is only as good as your stream instrumentation.
Reasoning models can emit reasoning_content before content. Watch only content, and you can miss tokens the model is already generating.
My takeaway: measure first streamed token and first answer token
Reasoning models can emit reasoning_content before content. Watch only content, and you can miss tokens the model is already generating.
My takeaway: measure first streamed token and first answer token
digitalocean.comWhat real time-to-first-token looks like on DigitalOcean Serverless Inference | DigitalOcean
0 6