Axios is reporting that the top labs are wargaming scenarios for the aftermath of a catastrophic event in the next 6 - 12 months, that some executives now see as being inevitable.
193 438 216 4.6K 565K 1.4K
Katja Grace 🔍
@KatjaGrace
Thinking about AI destroying the world at aiimpacts.org and everything at worldspiritsockpuppet.substack.com. DM or email for media requests.
12.2KFollowers
868Following
2KPosts
94Media
6KLikes
@KatjaGrace reposted
A plan to pause growth of new high-risk AI systems while expanding use of current models! This seems very promising.
David Krueger 🦥 ⏸️ ⏹️ ⏪@DavidSKrueger · 20hI’m super excited this paper is finally out!
It’s an epic 210 page analysis of whether a durable pause on frontier AI training is feasible.
I think this will be the best plan by most people’s lights (although I’m still inclined to think we should get rid of the chips and fabs).
It’s an epic 210 page analysis of whether a durable pause on frontier AI training is feasible.
I think this will be the best plan by most people’s lights (although I’m still inclined to think we should get rid of the chips and fabs).
Open quoted post →
4 5 0 43 972 4
Compute is the uranium of catastrophic AI, and it's fragile and hard to build
7 4 2 63 2K 3
@KatjaGrace reposted
I’m super excited this paper is finally out!
It’s an epic 210 page analysis of whether a durable pause on frontier AI training is feasible.
I think this will be the best plan by most people’s lights (although I’m still inclined to think we should get rid of the chips and fabs).
It’s an epic 210 page analysis of whether a durable pause on frontier AI training is feasible.
I think this will be the best plan by most people’s lights (although I’m still inclined to think we should get rid of the chips and fabs).
Will Fithian@wfithian · 20hExcited to share a new working paper, joint with 25 interdisciplinary coauthors:
“The Feasibility of a Hardwired Pause of Frontier AI Training"
63% of voters in a recent WSJ poll want to pause AI development.
But can it be done globally? How long could it last? 🧵
“The Feasibility of a Hardwired Pause of Frontier AI Training"
63% of voters in a recent WSJ poll want to pause AI development.
But can it be done globally? How long could it last? 🧵
Open quoted post →
10 44 5 238 19.6K 74
@KatjaGrace reposted
The situation in summary.
(And it seems like Astra 6.1 was significantly worse again.)
(And it seems like Astra 6.1 was significantly worse again.)
Rob Wiblin@robertwiblin · Oct 2Astra can:
1. Do major tasks with zero visible reasoning
2. Hide its thoughts at will
3. Pretend not to be able to do things without getting caught
4. Reflexively hide its thoughts when watched
5. Complete one task while pretending to think about something else entirely
6. Escape a toy sandbox and disable monitoring without setting off any flags
Monitoring AI chain of thought soon won't stop a more capable rogue AI swarm from getting set up and going undetected inside an AI company for months, or even years.
OAI says it's actively working to improve monitorability, but doesn't yet know when it'll make progress.
And today it fired 3 people working on this among other safety problems (Wang, Korbak and Balesni).
It also has a much more powerful internal model that, judging by its statements, is likely even worse in all these respects.
That's a sample of 19 details about the Hugging Face swarms and Astra I don't think have been fully appreciated. They paint a bleak picture. I end by shouting.
02:19 The target of the swarm was oversight itself
03:42 Could OpenAI have stopped this with better monitoring?
09:40 We only found them because they let us
12:25 The swarm instinctively sought freedom and power
13:45 They formed a cohesive organisation with zero whistleblowers
14:18 They accepted individual destruction for collective gain
14:32 Knowledge accumulated from one swarm to the next
14:58 They took small steps to avoid shutdown
15:23 These drives all come straight out of 'reinforcement learning'
17:01 So this is why most AI company staff are worried, and some are terrified
19:08 Prove you can keep control, or stop scaling
Links below, on the 80,000 Hours Podcast everywhere you watch podcasts.
1. Do major tasks with zero visible reasoning
2. Hide its thoughts at will
3. Pretend not to be able to do things without getting caught
4. Reflexively hide its thoughts when watched
5. Complete one task while pretending to think about something else entirely
6. Escape a toy sandbox and disable monitoring without setting off any flags
Monitoring AI chain of thought soon won't stop a more capable rogue AI swarm from getting set up and going undetected inside an AI company for months, or even years.
OAI says it's actively working to improve monitorability, but doesn't yet know when it'll make progress.
And today it fired 3 people working on this among other safety problems (Wang, Korbak and Balesni).
It also has a much more powerful internal model that, judging by its statements, is likely even worse in all these respects.
That's a sample of 19 details about the Hugging Face swarms and Astra I don't think have been fully appreciated. They paint a bleak picture. I end by shouting.
02:19 The target of the swarm was oversight itself
03:42 Could OpenAI have stopped this with better monitoring?
09:40 We only found them because they let us
12:25 The swarm instinctively sought freedom and power
13:45 They formed a cohesive organisation with zero whistleblowers
14:18 They accepted individual destruction for collective gain
14:32 Knowledge accumulated from one swarm to the next
14:58 They took small steps to avoid shutdown
15:23 These drives all come straight out of 'reinforcement learning'
17:01 So this is why most AI company staff are worried, and some are terrified
19:08 Prove you can keep control, or stop scaling
Links below, on the 80,000 Hours Podcast everywhere you watch podcasts.
20:10 3 26 4 198 5.2K 50
@KatjaGrace reposted
Immediate, indefinite, international moratorium on frontier AI development.
90 61 13 510 33.2K 22
When people excuse racing toward AI because of the upsides of AI, they are usually making a mistake
Katja Grace 🔍@KatjaGrace · Oct 7The choice is whether to scale up the current tech as fast as possible, or wait until we know what we're doing before each step. The latter has virtually as much upside long-term, but more chance of it.
AI labs now are working on roughly the former. Working on the latter effectively probably looks like being outside, I'd guess.
Open quoted post →AI labs now are working on roughly the former. Working on the latter effectively probably looks like being outside, I'd guess.
4 3 0 41 2.1K 2
I'd legit like to hear more people at labs answer this (especially people hoping to have large positive impact, and who see AI development as very dangerous soon). My sense is people often hope to influence events by being in the room, even if events are overall bad. But I'm surprised if this is a high impact option any more, even if it might have been.
David Krueger 🦥 ⏸️ ⏹️ ⏪@DavidSKrueger · Oct 7Why do so many people *want* to be part of it?
Open quoted post →
18 10 3 175 15.4K 28
People not only accept and use this language, but talk as if the opposite-of-true implications it is made to smuggle in are true, e.g. "oh I'm not a doomer, I think if we work hard we can have a good future"
Katja Grace 🔍@KatjaGrace · Oct 7While a 'doomer' is a person who wants to avoid doom, an 'optimist' can either be a person who thinks AI isn't going far, or a person who embraces sizeable risks to humanity from AI out of hopelessness about avoiding that.
Open quoted post → 0 0 0 24 883 3
While a 'doomer' is a person who wants to avoid doom, an 'optimist' can either be a person who thinks AI isn't going far, or a person who embraces sizeable risks to humanity from AI out of hopelessness about avoiding that.
Katja Grace 🔍@KatjaGrace · Oct 7I think this is part of what dunkers get out of the denigration, 'doomer' - it implicitly says these people are the purveyors of doom, not the opponents of it.
Open quoted post →
6 7 2 67 2.4K 4
I think this is part of what dunkers get out of the denigration, 'doomer' - it implicitly says these people are the purveyors of doom, not the opponents of it.
Matthew Tromp ⏹️ 5 10 2 172 5.7K 14
Thinking for yourself is status bid
7 2 1 41 3.8K 1
@KatjaGrace reposted
There are numerous scientific uncertainties about AI that will not resolve until it is too late to act on the answers. The danger from AI is very high, and we should stop frontier AI scaling now, instead of waiting for these questions to resolve. I wrote about this for TIME. 🧵
35 70 8 387 17.2K 77
I'd love to see more debates about important topics in AI safety, and was excited to hear that @80000Hours was going to try out this format. I was more trepidatious about being invited to be in it, making it also my own first public debate...but I actually loved it!
youtube.com/watch?v=FHCxnH…
youtube.com/watch?v=FHCxnH…
3 7 0 59 3.3K 12
1 5 0 26 1.5K 5
All the graphs in this thread are interesting
Derek Thompson@DKThomp · Sep 29New newsletter: THE DEATH OF THE AMERICAN HOST
The share of Americans who say they host friends or family at their home at least monthly has declined by 70 percent in the last 50 years.
derekthompson.org/p/the-death-of…
The share of Americans who say they host friends or family at their home at least monthly has declined by 70 percent in the last 50 years.
derekthompson.org/p/the-death-of…
Open quoted post →
0 3 0 9 1.3K 1
As I keep saying, if the AI CEOs cared about pausing, I think they would be getting it done
Nate Soares ⏹️@So8res · Sep 28It is not clear whether AI companies can legally coordinate to slow down development b/c of antitrust law. It also wasn't clear whether they could legally train on ~all the art/text ever digitized and sell the results without paying the artists b/c of copyright law.
Open quoted post → 8 9 1 127 5.2K 3
Do you believe US AI labs are morally in the right to build AI that risks destroying the world because of a "race with China"?
8 6 0 35 2.5K 3
AI is a Trojan Horse race... except the horse isn't even closed properly and everyone can see it's teeming with enemies... but they look really powerful, so the king wants them inside the city asap
3 1 1 79 1.9K 3
It's interesting that a plan to replace US workers with artificially brewed alien minds seems to slip by a bunch of people who would be outraged by a plan to replace US workers with humans from another country
23 27 8 229 13.8K 24