Ofir Press
@OfirPress
Oh and there's step 4 which is the hardest:
Make the data collection scalable.
A benchmark is great if it's easy to crawl training data for it automatically from the internet, as in SWE-bench, ProgramBench and SWE-Sweep. This makes benchmark design even harder.
Make the data collection scalable.
A benchmark is great if it's easy to crawl training data for it automatically from the internet, as in SWE-bench, ProgramBench and SWE-Sweep. This makes benchmark design even harder.
Ofir Press@OfirPress · Oct 1Designing a good benchmark means you:
1. Found an ability you want future AI to have
2. Expressable as a verifiable set of tasks
3. That is challenging for current frontier AI
Doing this is extremely hard, and is becoming harder every day. Really excited we found SWE-sweep!
Open quoted post →1. Found an ability you want future AI to have
2. Expressable as a verifiable set of tasks
3. That is challenging for current frontier AI
Doing this is extremely hard, and is becoming harder every day. Really excited we found SWE-sweep!
4 39