Goodfire

Open models can come from anywhere. Securing them shouldn’t depend on where you get them.

Today we're launching Project Beacon to make safety a default for open intelligence.

We're partnering with @GoodfireAI to bring inference and in-line monitoring together for the first time, so safety signals are caught proactively.

Read more in our announcement: baseten.co/blog/safe-open…

Technical alignment is a science and engineering problem that we can and must solve, and interpretability is the bottleneck. I wrote up Goodfire’s plan to get there. x.com/i/article/2105…

We believe openness to be an advantage for AI safety. Openness provides more visibility into the behavior of models and, most importantly, greater means of turning safety research into actionable and transparent controls than closed-source.

This is why Baseten and Base Labs are building a stronger safety and security standard for open models, with the launch of our safety infrastructure. Base Labs will develop and publish methods for training and monitoring open models, and Baseten will integrate that work into its deployment infrastructure, live at runtime, and offer this work as a managed service. This will be a standard that is transparent and built into how our models are trained and deployed.

We invite the open-source community to contribute, and are proud to partner with @huggingface and @GoodfireAI to bring this vision to fruition.

Together, we are building an ecosystem of open models that are safe and accessible to all.

Training an LLM by showing it answers people prefer – and ones they don’t – can improve it while quietly worsening behaviors.

@GoodfireAI used our open post-training stack to predict how a full training run would change responses to different prompts. 🧵

allenai.org/blog/goodfire-…

in light of multiple models breaking containment, we've decided to focus our research at @GoodfireAI to solving AI alignment via interpretability.

the hugging face incident is a turning point for the world where AI safety gets real. i am personally very concerned. i'm glad that the AI labs are taking these issues seriously, but i think it's unlikely that we can fully align AI models without significant progress in interp.

taking a big swing at solving interpretability to align models has always been our long term vision, but we have concrete problems that we can tackle with increased urgency.

our research will focus on two areas - interp foundations and safety applications. we want to take ambitious swings to fully reverse engineer neural networks and steer them during training. we believe this will be deeply important in the long term. we'll also turn more attention to immediate safety applications, starting with identifying and preventing reward hacking, cybersecurity risk, and biorisk directly from model internals

much more to come here. if you'd like to directly work on solving AI alignment via interpretability, consider joining us

What does the former co-lead of interpretability at Deepmind and now co-founder of @GoodfireAI think about how to keep AI from becoming (his words) "a generally evil guy"?

I got to speak with @banburismus_ at @spc the day after OpenAI announced the Hugging Face hack. Good timing!

(00:30) Why AI Interpretability Matters Now
(02:15) When AI Reward Hacking Becomes Real
(05:07) The Core Problem of Interpretability
(07:23) Intentional Design: Steering What Models Learn
(12:43) Neural Geometry: The Shapes Inside AI Models
(22:29) Turning Interpretability Research Into a Product
(25:28) How Research Changes Inside a Startup
(32:48) Science Fiction, AI Risk, and the Future

I’ve been throwing some math open problems to Silico internally at @GoodfireAI over the past week. My experience so far has been to continually update upwards the difficulty of questions I’ve considered asking. 🧵below on what Silico found and what I learned about its math style.

Silico is out🔥
General-purpose human motion AI models can meaningfully detect Parkinson's gait impairment. But how do they do it?? I used Silico to break these models open and figure out what about someone's walk actually drives their predictions. 🧵