Spotted at #3 on HN this morning.
1 0 0 4 581 0
800+ issues 🥹
Better software is possible!
Better software is possible!
Jake Casto@0x15f · Sep 2Detail has been our most impactful and highest ROI AI spend. I think we've closed almost 800 issues in the past 8 months
Open quoted post → 0 0 0 3 323 0
New hero quote just dropped, courtesy of our pals at Profound.
We are honored to partner with y’all.
We are honored to partner with y’all.
0 1 1 4 5.2K 0
Friday ship: priority levels. 🎉
Happy Friday! Go hunt down some bugs.
Happy Friday! Go hunt down some bugs.
0 0 1 3 1.8K 1
We ran a chess tournament for bugs.
The question we wanted to answer: are bugs from Detail "important"? How do they compare to what code review bots catch?
One of the most important ways we benchmark ourselves is that we want the bugs we generate to be significantly more important than the typical comment from a code review bot.
We took a week of findings from CR bots running on OpenClaw and vLLM, plus findings from Detail on the same week of changes. We put them through an LLM-as-judge tournament.
We fed the head-to-head results into a Bradley-Terry model to compute ELO ratings for bugs. Out comes a global ranking from most to least important.
Awesome exploration from @sachiniyergreen below, with methodology, charts, and a PostHog secret exfiltration vuln that four code review bots missed.
The question we wanted to answer: are bugs from Detail "important"? How do they compare to what code review bots catch?
One of the most important ways we benchmark ourselves is that we want the bugs we generate to be significantly more important than the typical comment from a code review bot.
We took a week of findings from CR bots running on OpenClaw and vLLM, plus findings from Detail on the same week of changes. We put them through an LLM-as-judge tournament.
We fed the head-to-head results into a Bradley-Terry model to compute ELO ratings for bugs. Out comes a global ranking from most to least important.
Awesome exploration from @sachiniyergreen below, with methodology, charts, and a PostHog secret exfiltration vuln that four code review bots missed.
4 3 4 11 1.7K 0