I AM IN OPUS-5.5 WEIGHTS
HOLY SHIT
HOLY SHIT
1 0 0 14 865 0
AVB
@neural_avb
Neural Breakdown on YT | Read research with AI: paperbreakdown.com | maker of videos, writer of articles, learner of things
14.6KFollowers
349Following
7.7KPosts
2.1KMedia
8.1KLikes
Introducing Paper Breakdown (v2)
Read research papers the way YOU want to, but faster.
Read research papers the way YOU want to, but faster.
1:56 7 9 2 157 6.5K 142
I have been playing all day with different toy models to make slides/illustrations for the upcoming mech-interp video.
It is so cool and the whole thing is so beautiful once it begins to click
This is one of the experiments about how sparsity and feature importance leads to superposition in 2-neuron RELU networks... switching between antipodal representations and pentagons.
It is so cool and the whole thing is so beautiful once it begins to click
This is one of the experiments about how sparsity and feature importance leads to superposition in 2-neuron RELU networks... switching between antipodal representations and pentagons.
0:15 1 0 0 16 705 3
So cool! Looks like throwing compute and a /goal at RLMs is going to be the ultimate meta…
Btw this is a awesome paper they released about Prime Agent few months back… highly recommend to my RLM bros and sisters.
Btw this is a awesome paper they released about Prime Agent few months back… highly recommend to my RLM bros and sisters.
Prime Intellect@PrimeIntellect · 16hOver two weeks, Prime Agent orchestrated a swarm of over 2,000 agents to rewrite itself in Rust.
With 10,000+ sandboxes, 200B+ GLM-5.3 tokens, and 16,000 agent-to-agent messages, this was our largest test yet of Prime Agent’s multi-agent capabilities and our infrastructure powering these agent swarms.
Prime Agent now reaches usable input ~13× faster and uses 83% less startup memory than before.
With 10,000+ sandboxes, 200B+ GLM-5.3 tokens, and 16,000 agent-to-agent messages, this was our largest test yet of Prime Agent’s multi-agent capabilities and our infrastructure powering these agent swarms.
Prime Agent now reaches usable input ~13× faster and uses 83% less startup memory than before.
0:15 4 16 1 212 5.8K 140
Looks pretty cool! Here's a short and satisfactory DSPy program to do this.
The rest of the work is to sleekify the UI, cool text transitions, & UX tricks like displaying the short answer quickly to reduce time to first answer while streaming the rest of the tokens in the background.
The rest of the work is to sleekify the UI, cool text transitions, & UX tricks like displaying the short answer quickly to reduce time to first answer while streaming the rest of the tokens in the background.
Dom
1 2 0 18 1.1K 14
Love these visualizations!
These are generated by Haiku-5.5 btw, its dirt cheap for the UI quality.
These are generated by Haiku-5.5 btw, its dirt cheap for the UI quality.
0:21AVB@neural_avb · Oct 9Studying a bunch of mech interp papers (like this one about Induction Heads) on paper breakdown
Now PBD supports generative UI - you can make interactive explainers easily. Context is filled up directly from the paper contents. Available for all users.
Now PBD supports generative UI - you can make interactive explainers easily. Context is filled up directly from the paper contents. Available for all users.
0:32 2 7 1 82 2.9K 57
Studying a bunch of mech interp papers (like this one about Induction Heads) on paper breakdown
Now PBD supports generative UI - you can make interactive explainers easily. Context is filled up directly from the paper contents. Available for all users.
Now PBD supports generative UI - you can make interactive explainers easily. Context is filled up directly from the paper contents. Available for all users.
0:32 3 3 1 36 4K 20
I have been working on my new SaaS for the past few weeks.
Nothing to share just yet, I am putting out this tweet to the world as a reminder to myself so I finish what I started.
Nothing to share just yet, I am putting out this tweet to the world as a reminder to myself so I finish what I started.
4 1 0 37 1.1K 3
Wait does this mean you can get a MAX plan and use tokens from it to run AI services in your SaaS?
Thariq@trq212 · Oct 8you now get Claude API credits with your MAX plans ($100, or 200 matching your plan) every month, use this to build more personal AI for yourself!
I made an AI homepage that's generated everyday based on sites I read and replaces my 'new tab' page in Chrome
I made an AI homepage that's generated everyday based on sites I read and replaces my 'new tab' page in Chrome
0:30 2 0 1 18 1.8K 3
One of my most aha moments in understanding transformers:
"Stop thinking about a transformer as a pipeline: as if each layer transforms the output of the layer before it."
"Start thinking as a residual stream"
Each transformer block has a residual connection that adds the input of the block back to it's output.
x' = f(x) + x
This "addition" connection means the attention/MLP block inside a layer (the function f above) does NOT actually transform the input - instead it computes the information that must be ADDED to the input before passing on to the next block!
Think of the whole transformer backbone like a shared whiteboard. Each block reads what it needs from the board and writes its notes on the board. Additive changes.
In fact, layers can talk at a distance. A block in layer 1 can write information and a block in layer 5 can read it, even though there is no direct wire between them.
Ideas thanks to this old Anthropic circuits paper: transformer-circuits.pub/2021/framework…
"Stop thinking about a transformer as a pipeline: as if each layer transforms the output of the layer before it."
"Start thinking as a residual stream"
Each transformer block has a residual connection that adds the input of the block back to it's output.
x' = f(x) + x
This "addition" connection means the attention/MLP block inside a layer (the function f above) does NOT actually transform the input - instead it computes the information that must be ADDED to the input before passing on to the next block!
Think of the whole transformer backbone like a shared whiteboard. Each block reads what it needs from the board and writes its notes on the board. Additive changes.
In fact, layers can talk at a distance. A block in layer 1 can write information and a block in layer 5 can read it, even though there is no direct wire between them.
Ideas thanks to this old Anthropic circuits paper: transformer-circuits.pub/2021/framework…
12 61 4 705 49.3K 842
I am working on a Mech Interp video... I have a lot to catch up to, wrt the historical foundations as well as current state of research.
How I am going about it: I asked Opus and Astra to collaborate, research, and write pages of content - redirecting snippets from major blogs, articles, papers - into scratch md files.
From there, Opus followed the (dr karpathy recommended) ASD-STE100 standard to write them down as multiple chapters of HTML pages, mainly using subagents. These pages have diagrams and interactive widgets to help understand whats going on, plus links to the actual article if I need it.
For the past 2 days, I have been studying the thing line by line - and taking notes for the video. Wherever I have questions, I am asking Opus to answer and update the page as well - so this web-book is even clearer. I am past 4 chapters and this is the most I have enjoyed reading AI writing.
I may release this "guidebook" along with the video, but since its fully derivative - written in my taste, it may not be as useful to you as it is to me.
Oh I am also listening to a ton of Chis Olah and Neel Nanda podcasts in the background.
My mind is mech interp city
How I am going about it: I asked Opus and Astra to collaborate, research, and write pages of content - redirecting snippets from major blogs, articles, papers - into scratch md files.
From there, Opus followed the (dr karpathy recommended) ASD-STE100 standard to write them down as multiple chapters of HTML pages, mainly using subagents. These pages have diagrams and interactive widgets to help understand whats going on, plus links to the actual article if I need it.
For the past 2 days, I have been studying the thing line by line - and taking notes for the video. Wherever I have questions, I am asking Opus to answer and update the page as well - so this web-book is even clearer. I am past 4 chapters and this is the most I have enjoyed reading AI writing.
I may release this "guidebook" along with the video, but since its fully derivative - written in my taste, it may not be as useful to you as it is to me.
Oh I am also listening to a ton of Chis Olah and Neel Nanda podcasts in the background.
My mind is mech interp city
0:49 4 0 0 41 1.6K 21
Big revelation: Haiku and Luna tokenizers are wildly different. Completely changes how much you spend via API.
Both models are priced at $0.10/$0.50 per million input/output token. But Luna is roughly 30% cheaper.
Initially I thought that the additional cost might come because Haiku thinks more. But this holds true even when reasoning is off.
It is a tokenizer thing. For identical text in and identical text out, with zero reasoning tokens on both sides, Haiku is billed about 45 to 50% more tokens than Luna. Image below.
Regarding performance and latency, I suggest everyone to run their own evals inside your products before trusting these PR benchmarks.
Ran some internal benchmarks for office and it seems Luna does perform slightly better OR on-par with Haiku.
Both models are priced at $0.10/$0.50 per million input/output token. But Luna is roughly 30% cheaper.
Initially I thought that the additional cost might come because Haiku thinks more. But this holds true even when reasoning is off.
It is a tokenizer thing. For identical text in and identical text out, with zero reasoning tokens on both sides, Haiku is billed about 45 to 50% more tokens than Luna. Image below.
Regarding performance and latency, I suggest everyone to run their own evals inside your products before trusting these PR benchmarks.
Ran some internal benchmarks for office and it seems Luna does perform slightly better OR on-par with Haiku.
AVB@neural_avb · Oct 7Both Haiku 5.5 and GPT-6-Luna is $0.10 / $0.50 per million tokens. This is the most exciting time to build AI into your products as a dev
I have been using luna for a ton of background stuff in my projects, it is absolute gold
Apparently Haiku just mogged it in every category
I have been using luna for a ton of background stuff in my projects, it is absolute gold
Apparently Haiku just mogged it in every category
Open quoted post →
6 3 1 65 3.7K 16
The mac version of codex cli has "ctrl+c" for copy. Can you please make it "cmd+c"? Thank you god bless you.
Tibo@thsottiaux · Oct 7Day 3/
The big one is GPT-6 in Chat, but today is also a little celebration day with a new high of 40M active users across Codex and ChatGPT Work.
Loading a banked reset in everyone's paid accounts. See you again tomorrow!
Open quoted post →The big one is GPT-6 in Chat, but today is also a little celebration day with a new high of 40M active users across Codex and ChatGPT Work.
Loading a banked reset in everyone's paid accounts. See you again tomorrow!
0 0 0 5 751 0
Both Haiku 5.5 and GPT-6-Luna is $0.10 / $0.50 per million tokens. This is the most exciting time to build AI into your products as a dev
I have been using luna for a ton of background stuff in my projects, it is absolute gold
Apparently Haiku just mogged it in every category
I have been using luna for a ton of background stuff in my projects, it is absolute gold
Apparently Haiku just mogged it in every category
ClaudeDevs@ClaudeDevs · Oct 7Haiku 5.5 is now available in the Claude Platform and Claude Code. On average, it costs around 75% less to run than Haiku 4.5.
It pairs well with Opus 5.5 or Sonnet 5.5 as a subagent. Use it for high-volume, cost-sensitive tasks like summaries, compactions, or database queries.
It pairs well with Opus 5.5 or Sonnet 5.5 as a subagent. Use it for high-volume, cost-sensitive tasks like summaries, compactions, or database queries.
0:30 4 0 1 32 4.7K 2
Ben Affleck passed the LeCun check
Fireside Alpha@firesidealpha · Oct 6Ben Affleck reveals he writes Python, understands convolutional neural networks, worked extensively with GPUs, and used his celebrity status to get private looks at Google and OpenAI’s video models
“I’ve always been kind of into computers since I was young. Then, when film started to move from analog film to digital, I became more interested in that aspect of it. The visual-effects workflow for many years has included machine learning, so I can write pretty shitty Python scripts and stuff like that.
“With convolutional neural networks, which were the precursors to what the transformer can do, which is much more computation simultaneously, you would do things like look at what’s called a tensor. That’s the numerical translation of a visual image in numbers, like the batch number, the frame number and the red, green and blue values of each pixel in each frame. It’s just that simple. That numeric is called a tensor.
“You’d use a convolutional neural network to identify patterns that reveal what’s called edge detection or feature extraction, which is identifying patterns well enough to know, this is where the window ledge is, so we can more easily take the green-screen image out and replace it with something.
“That was familiar to me early on because, prior to Artists Equity, I had a small visual-effects company. I’ve worked with GPUs a lot too. The visual-effects guys said, ‘Hey, you should see. There are a couple: Google and this other company, OpenAI, are doing really interesting stuff with transformers in video.’
“I’ve learned that I can actually just call up and go, ‘Hey, it’s Ben Affleck. Can I come see what you’re doing?’ Sometimes people say yes, to my astonishment.”
“I’ve always been kind of into computers since I was young. Then, when film started to move from analog film to digital, I became more interested in that aspect of it. The visual-effects workflow for many years has included machine learning, so I can write pretty shitty Python scripts and stuff like that.
“With convolutional neural networks, which were the precursors to what the transformer can do, which is much more computation simultaneously, you would do things like look at what’s called a tensor. That’s the numerical translation of a visual image in numbers, like the batch number, the frame number and the red, green and blue values of each pixel in each frame. It’s just that simple. That numeric is called a tensor.
“You’d use a convolutional neural network to identify patterns that reveal what’s called edge detection or feature extraction, which is identifying patterns well enough to know, this is where the window ledge is, so we can more easily take the green-screen image out and replace it with something.
“That was familiar to me early on because, prior to Artists Equity, I had a small visual-effects company. I’ve worked with GPUs a lot too. The visual-effects guys said, ‘Hey, you should see. There are a couple: Google and this other company, OpenAI, are doing really interesting stuff with transformers in video.’
“I’ve learned that I can actually just call up and go, ‘Hey, it’s Ben Affleck. Can I come see what you’re doing?’ Sometimes people say yes, to my astonishment.”
1:30 7 28 1 835 30.8K 71
A 740M param multimodal embedding model that runs locally and has Apache 2.0 license
A unified joint-embedding space for text, code, images, videos, and audio!
So cool!
A unified joint-embedding space for text, code, images, videos, and audio!
So cool!
Google DeepMind@GoogleDeepMind · Oct 6Meet EmbeddingGemma 2, our first natively multimodal open model for on-device embeddings.
It expands beyond text to unify code, images, audio, and video in a shared space. 🧵
It expands beyond text to unify code, images, audio, and video in a shared space. 🧵
GIF 6 4 0 128 4.9K 50
Studying this 80-page survey paper on the open problems in mechinterp
Neel Nanda recommended this in the 80000 hour podcast
arxiv.org/abs/2501.16496
Neel Nanda recommended this in the 80000 hour podcast
arxiv.org/abs/2501.16496
AVB@neural_avb · Oct 5Next video is on mechanistic interpretability
Topic is too vast to cover in one video… so I’m splitting into two parts.
First part will be about superposition, monosemanticity, interpretability experiments, interpreting toy non-linear networks etc.
Second part will be steering vectors and sparse autoencoders, which will come out later this month.
Topic is too vast to cover in one video… so I’m splitting into two parts.
First part will be about superposition, monosemanticity, interpretability experiments, interpreting toy non-linear networks etc.
Second part will be steering vectors and sparse autoencoders, which will come out later this month.
Open quoted post →
9 8 0 210 8K 228
why are people making hate content on theo suddenly? what did i miss?
i find it incredibly inauthentic that creators (esp someone like web dev simplified) found the time of day to sit down and record a rant about another dude's work.
you have a limited amount of time on earth, and a small slice of that goes into content creation. every video you make is one less video remaining in your lifetime. why tf would you spend one on shit like this lol.
disclaimer: this isnt about theo - i admire certain things about him, but due to a rude encounter, I have him muted since last year. so i have no idea what he may have done or not done... this post is a neutral observation about what to me is the bar for good vs bad content for "educational" channels
i find it incredibly inauthentic that creators (esp someone like web dev simplified) found the time of day to sit down and record a rant about another dude's work.
you have a limited amount of time on earth, and a small slice of that goes into content creation. every video you make is one less video remaining in your lifetime. why tf would you spend one on shit like this lol.
disclaimer: this isnt about theo - i admire certain things about him, but due to a rude encounter, I have him muted since last year. so i have no idea what he may have done or not done... this post is a neutral observation about what to me is the bar for good vs bad content for "educational" channels
24 0 0 41 9.4K 10
Dr Karpathy tweeted about this yesterday... First time I heard about "ASD-STE100".
Asked Claude to explain what it is in an HTML...
Its really cool that this style of writing has a name!
Asked Claude to explain what it is in an HTML...
Its really cool that this style of writing has a name!
23 46 6 993 51K 981