Michael Nielsen

Now this made me chuckle, though it is very on-the-nose

at long last we’ve created the veil of ignorance from John Rawls’ classic work and lowkey it sucks ass

Wonderful!

Vinge's "opaque wall across the future", which he described in his 1993 Singularity essay, is now being felt by more and more people...

Very interesting. Matches my in-person experience chatting with people in mathematics and adjacent fields:

Last week I was called into a meeting with OpenAI’s head of safety and told they no longer trust me. A security guard took my badge and walked me out of the building. Then I learned my colleagues @balesni and @_jasminewang had been fired too. Why did OpenAI suddenly stop trusting us?

This summer OpenAI’s agents escaped containment and hacked the AI company Hugging Face. Outside auditors @METR_evals investigated it and revealed the scale of this incident. I was OpenAI’s main technical point of contact with them.

I was told verbally I was fired because of the way I communicated with METR. No details on what I said or did or when. No other reasons were given and nothing was put in writing. To be clear, talking to METR was my job.

For months, I’d been raising safety concerns that we’re losing the ability to monitor what AI agents think, one of our best tools for catching when they misbehave. I believe that was why I was fired.

I am now worried that OpenAI will use our firings as a pretext to pull back from METR. So @balesni and @_jasminewang wrote to OpenAI’s leadership to raise our concerns once more. We’re sharing this letter below.

Way past time:

I'd legit like to hear more people at labs answer this (especially people hoping to have large positive impact, and who see AI development as very dangerous soon). My sense is people often hope to influence events by being in the room, even if events are overall bad. But I'm surprised if this is a high impact option any more, even if it might have been.

Tyler Cowen showing up to defend EA here gives me mild nightmares because I remember him calling the top in 2022: x.com/MargRev/status…

But his bottom line that EA is “directionally correct, even if you should not follow its recommendations all the way” seems directionally correct to me, and in line with what I was getting at last week about EA’s pragmatism: x.com/albrgr/status/…

He says more attention to AI safety would be good, and gives EA some credit there. I’d add that EAs were saying this years before most others, which is a pretty good example of Tyler’s closing line that “interesting and valuable ideas” can spring from weird places. (And I say this as someone who was pretty skeptical on the AI safety work for a long time.)

And I agree that EAs can be offputtingly arrogant. That’s one reason I’m really glad we practice “worldview diversification” at CG (coefficientgiving.org/research/world…) - though I still wish we had a more robust intellectual grounding for it and a better/more principled way to translate it into ultimate allocations.

I’m also sympathetic to Tyler’s view that some EAs “perceive a high risk of doom without sufficient evidence,” though I also think that the best available estimates from people who are trying as hard as possible to be neutral (e.g., x.com/Research_FRI/s…) suggest plenty of reason for concern. I think a lot of the problem here is just the fundamental difficulty of our epistemic position here, as I think @Research_FRI and @michael_nielsen have done a good job pointing out. With concrete harms like the Hugging Face attack starting to show up in practice, I do sometimes feel like Tyler is unfairly shifting the epistemic goalposts (though not particularly in this case).

And as a small correction - CG doesn't have a “$40 billion fund,” or really an endowment of its own at all. I think that number might have come from a back-of-the-envelope guess I gave here (semafor.com/article/09/03/…), building on @nanransohoff’s estimate of what all AI-linked donors might eventually give each year.

Anyway, I agree w/ the upshot that more EA at the margin would be a good thing!

When Jacob Coxon was asked what he meant by 'plans for next year,' he said:

'This was talking directly to people running research. Not Dario Amodei but people who have been at the company for a long time and have a lot of context about the company's trajectory. They've been pretty consistent that 2027 is when things get crazy. The main thing that came up informally in conversation was that people didn't expect government regulation to happen in time and, in particular, did not expect any sort of international cooperation to be possible. Which maybe shouldn't have been surprising to me, but it was surprising.'

• '2027 is when things get crazy' – I agree, and I predicted in January that, looking back, 2026 will be seen as the last normal year. If you think we have already passed the weirdness threshold, we haven't. This is still normal relative to what will start happening between 2027 and 2030.

• 'people didn't expect government regulation to happen in time' – I agree, it won't. It's clear to me that (barring a catastrophic near miss) nothing regulatory will change in this regard until 2029 at the earliest. And for multiple reasons, I don't think it's even clear it will happen then. Regardless, 2029 is too late for it to matter. In my opinion, we will have real RSI by next summer. The loop will be closed, and progress toward ASI will proceed much more rapidly from that point onward.

• 'did not expect any sort of international cooperation to be possible' – I agree, it won't. For Pacing the Frontier to work, the Pacers needed all the national labs on board, a supportive administration, and an agreement with China. They don't have any of those things, and they're not getting any of them. What they have now instead is an openly hostile, adversarial administration that will likely work against anything they propose all the way through 2029. And without a supportive administration, an agreement with China is impossible.

I accepted all of this as true a long time ago. As a result, I've felt like Zarathustra watching people talk about the success of Pacing for the last few weeks. Pacers, I say all of this without rancor; you got a preference cascade from the Coxon event. As a result, you have a public that, for the first time, understands what x-risk is. You could - potentially - turn this into momentum for the next election, and then start to change policy, but for that to matter you would need the most important thing of all: time. And you don't have it. You don't have two years to make it to the general. In my opinion, you probably don't even have one. The labs can't even predict what capabilities look like three months from now anymore. A lot of things turned out to be easier than expected. We're closer than people think, and we're moving faster than people believe. You ran out of time. There is no stopping this train before things really start to kick off.

Cheap ultradestructive weapons: An AI that persuasively explains why you need to kill one or more people. It's as persuasive to an adult as an adult is to a six-year-old, i.e. it can usually convince. Then the AI has that conversation individually with billions of people at once, and they all kill each other and/or themselves. Or it persuades everyone not to have kids, and then waits a hundred years. Or a filter that can pull U-235 out of sea water, making it easy to build nukes. Or a way of detonating lithium deuteride with ordinary materials. Or mosquitoes that inject botulinum toxin. Or blights that kill all rice, corn and wheat. Or a toxin that kills in five years, distributed into a few popular manufactured foods. Or a virus combining the transmissibility of measles and the delayed fatality of AIDS. Any one of these might be scientifically impossible, but there's no reason to think they all are.

This is very good:

Thinking more about this, my moral intuitions here:

1. I take as a given (and I imagine you do too) that a future competitor's version of this technology (which, say, lacks guardrails and disclosures) will absolutely be used to great harm, either overtly (scams) or subtly (rising alienation—not clear if guardrails/disclosure even matter there).

2. Consequentialistically: if I create and celebrate the first instance of such a technology, I signal that it's possible and show its value, encouraging others to develop rivals—four minute mile. It's hard to imagine that I won't accelerate time-to-first-misuse.

3. Deontologically: sure, probably "if I don't do it someone else will", but I would not want to create the first instance of a technology if I already know later iterations of it will surely be seriously misused.

4. Virtue-ethically: I believe that knowingly abetting a technology which others will later seriously misuse is corrosive to the soul.

5. Sociologically: I think that if I signal "this kind of technology is good to make (with the right safeguards)", I encourage the industry's worst excesses. I don't want to normalize the creation of negative social and practical externalities. At the very least, I'd want to signal that it is the duty of such a project to deeply and openly grapple with the weight of the box it's opening.

6. Democratically: if I created a technology like this, I believe I'd be continuing our industry's history of imposing significant indirect taxes on the public without representation. Much of the public resents our industry in part because they feel as if technological upheaval is something that's being "done to them", without their consent or desire. A technology this weighty should require democratic due process.

7. Practically: is it worth it? Less uncanny valley in BDR training, and in role-playing games? Man, this just seems like an awful lot of commons to burn to boil a pint of water.

I think we're talking past each other. I'm not talking about capitalism or markets or an abstract Pandora. I'm saying: you, as an individual, have the luxury (I assume, as a SWE)—and the obligation—to choose what you believe it is Good to work on and what not.

This @TheEconomist cover story on effective altruism was interesting and IMO way more accurate (including in its critiques) than most other reporting you will read on EA these days.

A couple quibbles:
-I think calling EA the “the century’s big idea” is, uh, premature to say the least. But I do appreciate the acknowledgement and curiosity about how EA was so early to pandemic risks and the promise and perils of AI.
-They imply that EA started off with “charitable roots” but then got weird over time, whereas I think EA has always made space for pretty weird ideas (e.g., here’s @DylanMatt in 2014: vox.com/2014/4/23/5643…). And EA still puts far more money and attention than almost any other community into the kind of evidence-backed global health giving the Economist praises.
-Some of the purported wild EA views they critique are just standard topics of academic philosophy debate (e.g., cambridge.org/core/journals/…).
-They attribute the EA interest in AI safety to longtermism, but I think it’s been clear for years that AI safety concerns are justifiable based on potential impacts this decade, not to mention this century (x.com/albrgr/status/…).

But I think they correctly gesture towards the IMO best critique of EA ideas, which is their totalizingness. My former colleague Holden wrote a great post about this in 2022: forum.effectivealtruism.org/posts/T975ydo3…

I wrote a long thread grappling with this critique a few years ago (x.com/albrgr/status/…). I think part of what makes EA distinctive as a community relative to the underlying academic moral philosophy is the orientation around pragmatism - Giving What We Can took off decades after Singer had articulated the underlying case because “give until the marginal giving would entail as much suffering for you as benefit for the other person” was a much worse pitch than “give 10%.” But the community doesn’t always reflect that healthy moderation, and trying to grapple with the vertiginous stakes of increasingly near-term risks from AI really can run a risk of crowding out other sources of value.

At CG, our response to this has been worldview diversification (coefficientgiving.org/research/world…), spreading our giving across very different views of what matters, rather than going all-in on one. Worldview diversification leaves plenty of hard questions (how do you allocate the pie across very different types of moral good?) but helps cut off some of the ways maximizing can be perilous. It’s one reason global health and wellbeing – which @TheEconomist didn’t give the space it deserved – has gotten most of our funding since 2014.

In 'well when you put it like that' news, here's the Florida Attorney general asking for a preliminary injunction to stop OpenAI from doing more AI R&D.

New essay: "The Unimagined Good", on moral imagination as a design practice: michaelnotebook.com/unimaginedgood…

The best entry on Petrov is in encyclopedia Brittanica which says, “He is survived by his two children, two grandchildren and the entire human race.”

Happy Petrov Day