Here in SF this week? We're hosting a happy hour tomorrow evening, Thursday, Oct 8, 5–8 pm at our SF office. If you're into all things data, come say hi!
Sign up here: luma.com/hud-happy-hr
Sign up here: luma.com/hud-happy-hr
0 0 0 14 583 1
hud
@hud_evals
RL environments + evals for agents | @ycombinator | we're hiring!
3KFollowers
14Following
108Posts
33Media
699Likes
@hud_evals reposted
Cool to see myself and a few of our partners/customers on this list.
Happy to chat with ppl interested in all things data, rl, and data licensing to labs!
Happy to chat with ppl interested in all things data, rl, and data licensing to labs!
Chris Barber@chrisbarber · Oct 5(part 3) I asked people: "who have you met in the AI data and post-training space that you find most insightful?"
Also 1 vote each:
- Justus Mattern (@MatternJustus)
- Kai Du (@knightwolf30)
- Kaustubh Sridhar (@_k_sridhar)
- Kevin Huang (@winkey_h)
- Linden Li (@lindensli)
- Madhav Singhal (@madhavsinghal_)
- Max Kan (@maxkan)
- Michael Truell (@mntruell)
- Ming-Yu Liu (@liu_mingyu)
- Mokshith Voodarla (@mokshith)
- Nathan Lambert (@natolambert)
- Noam Brown (@polynoamial)
- Ofir Press (@OfirPress)
- Paco Guzmán (@guzmanhe)
- Parth Patel (@parth220)
- Pierre-Louis Vouteau (@PLouisV)
- Punit Arani (@punit_arani)
- Raj Ammanabrolu (@rajammanabrolu)
- Rebecca Qian (@rebeccatqian)
- Rhythm Garg (@rhythmrg)
- Richard Socher (@RichardSocher)
- Rishi Desai (@rishi_desai2)
- Ryan Marten (@ryan_marten)
- Sahil Bhaiwala (@Sbhaiwala03)
- Sami Jaghouar (@samsja19)
- Sergio Bruccoleri (@SergioBrucc)
- Shreya Shankar (@sh_reya)
- Shubha Raghvendra (@shubhar25)
- Sid Potdar (@sid_potdar_)
- Soren Larson (@hypersoren)
- Sujai Hiremath (@sujaihiremath)
- Tanmay Sharma (@tsha444)
- Thijs Simonian (@cdngdev)
- Tim Shi (@timshi_ai)
- Tom McGrath (@banburismus_)
- Tomer Cohen (David AI)
- Trajectory team
- Zachary Baker (@zacbakerr)
- Zikang Jiang (@ZikangJiang)
Open quoted post →Also 1 vote each:
- Justus Mattern (@MatternJustus)
- Kai Du (@knightwolf30)
- Kaustubh Sridhar (@_k_sridhar)
- Kevin Huang (@winkey_h)
- Linden Li (@lindensli)
- Madhav Singhal (@madhavsinghal_)
- Max Kan (@maxkan)
- Michael Truell (@mntruell)
- Ming-Yu Liu (@liu_mingyu)
- Mokshith Voodarla (@mokshith)
- Nathan Lambert (@natolambert)
- Noam Brown (@polynoamial)
- Ofir Press (@OfirPress)
- Paco Guzmán (@guzmanhe)
- Parth Patel (@parth220)
- Pierre-Louis Vouteau (@PLouisV)
- Punit Arani (@punit_arani)
- Raj Ammanabrolu (@rajammanabrolu)
- Rebecca Qian (@rebeccatqian)
- Rhythm Garg (@rhythmrg)
- Richard Socher (@RichardSocher)
- Rishi Desai (@rishi_desai2)
- Ryan Marten (@ryan_marten)
- Sahil Bhaiwala (@Sbhaiwala03)
- Sami Jaghouar (@samsja19)
- Sergio Bruccoleri (@SergioBrucc)
- Shreya Shankar (@sh_reya)
- Shubha Raghvendra (@shubhar25)
- Sid Potdar (@sid_potdar_)
- Soren Larson (@hypersoren)
- Sujai Hiremath (@sujaihiremath)
- Tanmay Sharma (@tsha444)
- Thijs Simonian (@cdngdev)
- Tim Shi (@timshi_ai)
- Tom McGrath (@banburismus_)
- Tomer Cohen (David AI)
- Trajectory team
- Zachary Baker (@zacbakerr)
- Zikang Jiang (@ZikangJiang)
4 1 0 41 4.2K 28
Most browser evals look clean in a sandbox and fall apart on the live web. We wrote a walkthrough with @browserbase on how to write browser evals that hold up on the live web.
Sites update, graders reward faked outputs, and bot detection kill your runs before the agent even starts. If you're benchmarking browser agents, your scores are probably noisier than you think.
hud.ai/research/can-y…
Sites update, graders reward faked outputs, and bot detection kill your runs before the agent even starts. If you're benchmarking browser agents, your scores are probably noisier than you think.
hud.ai/research/can-y…
4 6 2 34 5.2K 17
New cookbook with @FireworksAI_HQ! Prepare your environment and taskset in HUD, then train a model on them with Fireworks.
Fireworks@FireworksAI_HQ · Sep 25Want to RL-train a model on your own task without building the infra?
Use our new cookbook with @hud_evals:
→ Define your task + grader in HUD
→ Sample + train the model on Fireworks
Define the task once. Train and evaluate against the same environment.
docs.fireworks.ai/fine-tuning/tr…
Open quoted post →Use our new cookbook with @hud_evals:
→ Define your task + grader in HUD
→ Sample + train the model on Fireworks
Define the task once. Train and evaluate against the same environment.
docs.fireworks.ai/fine-tuning/tr…
2 3 3 27 3.1K 7
It’s clear that major mathematics progress is within reach.
We’re supporting the Caltech Mathathon Challenge along with @AnthropicAI, @OpenAI, @cognition, and others to give top mathematicians $2m in compute to push the frontier forward.
HUD will be running the Riemann Hypothesis track! ζ
We’re supporting the Caltech Mathathon Challenge along with @AnthropicAI, @OpenAI, @cognition, and others to give top mathematicians $2m in compute to push the frontier forward.
HUD will be running the Riemann Hypothesis track! ζ
Sathvik Redrouthu@_sathvikr · Sep 4We partnered with @AnthropicAI @OpenAI
to host the world's first math hackathon.
40 hours, $2M compute - solve, understand, and present an open problem.
Applications are live: mathathonchallenge.com
to host the world's first math hackathon.
40 hours, $2M compute - solve, understand, and present an open problem.
Applications are live: mathathonchallenge.com
1:17 3 10 0 53 7.7K 6
We're releasing a new cookbook with @daytonaio!
You can now build a coding environment in 80 lines, evaluate at up to 256 parallel sandboxes, and immediately train on the graded rollouts.
You can now build a coding environment in 80 lines, evaluate at up to 256 parallel sandboxes, and immediately train on the graded rollouts.
4 8 3 41 3.5K 10
HUD will sponsor the Agent Native Builders Hackathon this week, Aug 26-27 @ Cloudflare SF.
Our research engineer, Axel Cassou (@AlienTta), will be judging the projects. Come build!
Our research engineer, Axel Cassou (@AlienTta), will be judging the projects. Come build!
Immersive Commons@immersivecommon · Aug 24HUD is sponsoring the Agent Native Builders Hackathon.
The platform for building reinforcement learning environments. If you cannot measure whether your agent did the job, you cannot improve it. @hud_evals @AxelCassou
Aug 26-27, Cloudflare SF, 50 builder seats only
luma.com/agentnativebui…
#AgentNatives #AIAgents
The platform for building reinforcement learning environments. If you cannot measure whether your agent did the job, you cannot improve it. @hud_evals @AxelCassou
Aug 26-27, Cloudflare SF, 50 builder seats only
luma.com/agentnativebui…
#AgentNatives #AIAgents
Open quoted post →
2 0 0 10 1.3K 1
We created Assemble Bench, a benchmark for robot models based on the NIST standard for small-parts assembly + a synthetic post-training approach we refer to as CG-DAgger.
Our goal was to explore simulated robotics environments and to see if they - like environments for coding agents - could serve as training environments for vision-language-action and world-action models (VLAs, WAMs).
Our goal was to explore simulated robotics environments and to see if they - like environments for coding agents - could serve as training environments for vision-language-action and world-action models (VLAs, WAMs).
0:06 12 17 6 122 13.3K 65
Average evening at a Gen Z AI startup, ft. our friends at @FireworksAI_HQ
0:09 4 0 1 19 2.5K 6
Last month the #1 ranked Catan player on colonist.io came to our hackathon and RL'd a model to play Catan using hud + @FireworksAI_HQ.
So we're running it back with humans – 32-player bracket at HUD HQ. Thursday July 16th
So we're running it back with humans – 32-player bracket at HUD HQ. Thursday July 16th
4 3 3 42 233K 10
Roughly 90% of purchased post-training data doesn't work. Reward hacks, contrived tasks, models climbing the wrong hill.
We sat down with @daltonc from @Standard_Cap to talk about why that happens — and how the RL environment supply chain gets fixed.
We sat down with @daltonc from @Standard_Cap to talk about why that happens — and how the RL environment supply chain gets fixed.
18:20 8 10 4 91 213K 34
Last weekend 200+ of the best builders showed up to our Frontier RL Environments Hackathon at @ycombinator.
The idea behind HUD is simple: anything you can simulate and grade, you can improve with RL.
Teams built RL envs for autonomous farming, gene editing, city planning, world sims, and games. Some did autonomous ML research + chip design, and used HUD to RL their own open models past Opus 4.8 in-domain.
Huge thanks to our cosponsors for making it happen!
If you want to work on RL, or just be at the frontier of AI, build with us, we're hiring: jobs.ashbyhq.com/hud
The idea behind HUD is simple: anything you can simulate and grade, you can improve with RL.
Teams built RL envs for autonomous farming, gene editing, city planning, world sims, and games. Some did autonomous ML research + chip design, and used HUD to RL their own open models past Opus 4.8 in-domain.
Huge thanks to our cosponsors for making it happen!
If you want to work on RL, or just be at the frontier of AI, build with us, we're hiring: jobs.ashbyhq.com/hud
3 7 2 57 15K 20
KBBQ Robotics Arm RL Environment is on the critical path to RSI
0:28 4 2 2 36 3.5K 11
Didn’t expect people to hack on protein folding environments for creating mythical beasts IRL…
hud@hud_evals · Jun 18Come join us and celebrate this weekend at HUD's Frontier/RSI RL Environments Hackathon @ the YC HQ in San Francisco.
We have an all-star cast of cosponsors such as:
@ycombinator, @modal, @GoogleDeepMind, @OpenAI, @AnthropicAI , @daytonaio, @FireworksAI_HQ, @MiniMax_AI, @SemiAnalysis_, @hillclimbai, @withprotegeai, @AntimLabs, @sixtyfourai, @ExaAILabs, and @arcprize!
We have an all-star cast of cosponsors such as:
@ycombinator, @modal, @GoogleDeepMind, @OpenAI, @AnthropicAI , @daytonaio, @FireworksAI_HQ, @MiniMax_AI, @SemiAnalysis_, @hillclimbai, @withprotegeai, @AntimLabs, @sixtyfourai, @ExaAILabs, and @arcprize!
Open quoted post →
2 1 0 45 5.4K 7
Today, HUD is excited to share our Series A funding!
We are the platform for building high quality post training datasets. Over 50 businesses use HUD to build RL environments, sell them to AI labs, or train their own models from them.
Our mission is to enable a generation of data entrepreneurs. The previous generation built apps to impact the world. We believe people will build infrastructure around data, both digital and physical, that align AIs to their specific goals.
We’ve raised $16M in total funding from @Standard_Cap, @ycombinator, @ExceptionalCap, @Liquid2V, @twentytwovc , amazing angels such as @dylan522p from @SemiAnalysis_, @tszzl, @ivanburazin, @theo, and more!
We are the platform for building high quality post training datasets. Over 50 businesses use HUD to build RL environments, sell them to AI labs, or train their own models from them.
Our mission is to enable a generation of data entrepreneurs. The previous generation built apps to impact the world. We believe people will build infrastructure around data, both digital and physical, that align AIs to their specific goals.
We’ve raised $16M in total funding from @Standard_Cap, @ycombinator, @ExceptionalCap, @Liquid2V, @twentytwovc , amazing angels such as @dylan522p from @SemiAnalysis_, @tszzl, @ivanburazin, @theo, and more!
44 37 13 253 84.3K 74
Excited to welcome @OpenAI and @arcprize as co-sponsors!
HUD RL for RSI hackathon, June 20th-21st @ YC HQ
Signups close tomorrow! 📢
HUD RL for RSI hackathon, June 20th-21st @ YC HQ
Signups close tomorrow! 📢
2 6 3 34 7.3K 11
btw, if you win our RL for RSI hackathon, u get a cool robot dog 🐕🦺
June 20th-21st @ YC HQ. Signups close in 3 days! 👇
June 20th-21st @ YC HQ. Signups close in 3 days! 👇
0:45hud@hud_evals · May 16Announcing HUD's RL environments for RSI hackathon! 🎉
Join us June 20–21 in SF if you're interested in RL and want to push the frontier forward!
(w/$100,000+ in prizes and compute credits 👀)
Join us June 20–21 in SF if you're interested in RL and want to push the frontier forward!
(w/$100,000+ in prizes and compute credits 👀)
Open quoted post →
4 4 1 22 3.5K 2
Announcing HUD's RL environments for RSI hackathon! 🎉
Join us June 20–21 in SF if you're interested in RL and want to push the frontier forward!
(w/$100,000+ in prizes and compute credits 👀)
Join us June 20–21 in SF if you're interested in RL and want to push the frontier forward!
(w/$100,000+ in prizes and compute credits 👀)
11 28 14 136 71.2K 78
This Tuesday HUD is hosting Strange Evals.
This session: if VLM reasoning benchmark are saturated why cant claude make me a decent PPT?
DM if you’d like to join!
This session: if VLM reasoning benchmark are saturated why cant claude make me a decent PPT?
DM if you’d like to join!
Vincent Koc@vincent_koc · May 4For my eval-maxxing nerds out there, good friends of mine are running a series called "strange evals", you can benchmaxx now on anything. If in SF swing by! luma.com/lvqbs1mo
Open quoted post → 2 2 0 15 2.2K 2
AI agents are deploying to prod, but can they autonomously find and patch unseen critical vulnerabilities?
We introduce ZeroDayBench, a benchmark for evaluating LLM agents on proactive cyberdefense.
Plus, a novel high-severity (CVSS 8.1) CVE we found partway through ... 👀
We introduce ZeroDayBench, a benchmark for evaluating LLM agents on proactive cyberdefense.
Plus, a novel high-severity (CVSS 8.1) CVE we found partway through ... 👀
3 16 6 85 37.4K 61