Ryan Brooks

@rbro112

Sentry

Staff Eng @sentry on AI/ML (all things evals)

Formerly employee 1 @emergetools (YC W21), mobile @airbnb

github.com/rbro112 Joined June 2020

345Followers

212Following

271Posts

38Media

3.4KLikes

@rbro112 reposted

@sergical

fun fact

Ryan and I prototyped a part of the internal evals platform we are calling Signals during our yearly hackweek in august.

Ryan's been cooking since then and I'm excited to catch up and chat about shipping evals and observing our agents at @sentry

Ryan Brooks@rbro112 · 15h

Come join @sergical and I next Wednesday Oct 14 @ 10am PT!

We'll be chatting about how we debug our agents at @sentry and I'll share how I come up with eval scenarios using our agent traces.

Tune in sentry.io/resources/agen…

Open quoted post →

0 2 0 4 348 0

@rbro112

Come join @sergical and I next Wednesday Oct 14 @ 10am PT!

We'll be chatting about how we debug our agents at @sentry and I'll share how I come up with eval scenarios using our agent traces.

Tune in sentry.io/resources/agen…

1 1 2 10 939 2

@rbro112

A fun slide I shared in my @AIconference talk last week (already outdated!), Sentry eval runs are really taking off

0 0 0 11 320 2

@rbro112

The “Software factory” term gives me the ick

1 0 0 4 342 0

@rbro112

Those who manage large eval datasets - how are you guys creating scenarios?

Do you take from prod data? Have AI generate synthetic?

Finding a methodical solution here hard to come up with

1 0 0 8 423 0

@rbro112

It’s been a week since I moved one of eval datasets from Gemini 3.1 to Jev by @typesafeai for LLM judging. The results so far:

- No meaningful change in scoring accuracy
- ~200× cheaper (~$0.01 → ~$0.00005 per judge)
- ~50× faster (~10s → ~0.2s median)
- ~50% fewer input tokens, 87% fewer output tokens

But not everything’s perfect, more details in the thread

Ryan Brooks@rbro112 · Sep 18

👀👀

Open quoted post →

16 15 2 110 17.3K 95

@rbro112

Working to get our agent tracing -> eval pipeline flowing.

It's manual right now, but using our own agent traces to build our eval dataset is the future.

Milin Desai@virtualmilin · Sep 24

Agree and we are doing this ourselves and helping our users do it too. An easy way to setup and use agent tracing data with your eval.

A little cookbook on this topic and there is a lot more to come around our own learnings and more cc @rbro112

sentry.io/cookbook/agent…

Open quoted post →

1 0 1 13 2.1K 11

@rbro112

Posts like these resonating makes me happy that some sanity is coming back to the engineering world

ThePrimeagen@ThePrimeagen · Sep 23

It does seem like pushing back against management as software devs in the age of AI is a lost art.

I have never had a job or a project that wasn't in the shape of "what corners can we cut so we can ship this last week? Is that a prototype, welcome to production..."

It was my job to push back and set pace and I don't see how that is not the case still today

Open quoted post →

1 0 0 4 1.7K 0

@rbro112

Come help me build the best eval platform for fixing bugs

Indragie Karunaratne@indragie · Sep 23

We’re also hiring engineers to help build our eval platform and enable every product team at Sentry to ship AI features with confidence: sentry.io/careers/95d2ee…

Open quoted post →

0 1 0 8 2.3K 10

@rbro112

We’re investing heavily in evals at @sentry. It’s early but we’re seeing a culture change from shipping vibes to shipping with clear data using our evals platform.

Will share more in a future blog post, but some stats from our devs in the past month:
- ~1800 eval runs triggered directly on PRs
- 1500 eval scenarios ran across 17 AI surfaces in our product
- Cost/token reductions of ~20-50% across a few of our surfaces
- Eval passing improvements of ~10-20% from some core “code mode” changes launching soon (shoutout @grichadev)

All this from essentially 0 back when we started this effort in May.

Lenny Rachitsky@lennysan · Sep 22

Evals have been coming up more and more in my conversations with podcast guests and PM friends.

Nearly half of the 25 awesome PM job openings I shared last week ask for experience writing evals. And leading companies keep sharing what investing in evals bought them:
— @tryramp took its automatic receipt collection from 35% to 83% accuracy.
— @Shopify shipped an AI workflow builder that's 2.2x faster and 68% cheaper than the frontier-model setup it replaced.
— @harvey__ai rebuilt its AI contract reviewer, nearly doubling its internal quality score.
— @cursor_ai tuned its Auto Balance routing, with much higher user satisfaction at 41% lower cost.

So I asked the 🐐s of evals, @HamelHusain and @sh_reya, to write an advanced sequel to their very popular "Building eval systems that improve your AI product." Drawing on their work with 50+ AI companies, they share the key step most teams skip, what you should (and shouldn't) automate, and a free plugin that lets a coding agent do most of the heavy lifting.

Read it here: lennysnewsletter.com/p/advanced-eva…

Open quoted post →

12 8 4 163 32.8K 196

@rbro112

👀👀

1 0 1 10 16.5K 0

@rbro112

Come see me talk in a few weeks!

The AI Conference@AIconference · Sep 12

Join @rbro112 from @sentry at The AI Conference 2026 for “Your Evals Probably Aren’t Dev Friendly.”

Rather than focusing on how to create evals, Ryan will share strategies Sentry has used to make its eval platform more actionable and useful for developers, plus practical tips teams can bring back to their own workflows.

See you in San Francisco!

0:24

Open quoted post →

1 4 1 13 1.8K 0

@rbro112

A quiet ship, we've been dogfooding our own Agent Tracing product for months. It's really helped us debug our agents and help me personally build interesting eval cases.

Take a peek 👇

Sentry Changelog@SentryChangelog · Sep 11

Agent Tracing is now GA

Sentry's Agent Tracing is now GA — trace every model call, tool execution, and handoff inside your existing traces, and read the full user-agent exchange in Conversations. sentry.io/changelog/agen…

Open quoted post →

2 2 0 6 1.4K 2

@rbro112

Not the result I wanted but at least college football is back

1 0 0 4 119 0

@rbro112

Why did AI reinvent terms for testing & analytics?

Testing -> "offline evals"
Metrics & analytics -> "online evals"

Nothing has really changed in practice other than using non-deterministic (LLM) scoring/judging.

0 0 0 2 93 0

@rbro112

Slopped too close to the sun recently, so I deleted ~5-10k loc and make sure to read+review everything now.

Feels like I actually know what's my code is doing again, crazy concept

6 1 0 23 2.3K 1

@rbro112

@sentry's got that

Boris Cherny@bcherny · Aug 21

Coding is solved, bugs are not yet solved. Fix incoming

Open quoted post →

1 1 0 11 826 0

@rbro112

Real good post here. This is what our evals at Sentry have naturally converged to - guardrails, goals and metrics - each serving a slightly different purpose.

Figuring out how to get devs to use the results is the tougher problem

Lotte@lotte_verheyden · Aug 18

Before you write good evals, you have to pick the right things to evaluate. Main take-aways:
- source metrics from failures you see in traces
- keep the set small and maintainable
- only keep a metric if you'd take action when it moves x.com/i/article/2089…

Open quoted post →

0 0 0 6 362 0

@rbro112

But seriously, who's building the GitHub replacement? Can't take it anymore

1 0 0 5 255 0

@rbro112

I usually do a lunchtime workout and knocked it out early today to see if I could avoid the @github downtime.

I come back and we’re now pushing 5hrs with no relief in sight!?

0 0 0 4 194 0

Ryan Brooks (@rbro112) — posts, photos & videos | PeekX