Oren Melamed
@OrenMe
Watched a team ship a “great” skill last month.
It worked in the demo.
It quietly got worse every time the model changed underneath it.
No evals. No version. No way to say “v12 is worse than v11, roll it back.”
We would never treat a library like that.
We treat prompts and
It worked in the demo.
It quietly got worse every time the model changed underneath it.
No evals. No version. No way to say “v12 is worse than v11, roll it back.”
We would never treat a library like that.
We treat prompts and
7 10