David Duvenaud

Fearing "loss of meaning" from AI job loss sounds trivial vs. fearing death. But I think desire for meaning evolved to prevent being ostracized, maybe fatally. Most sources of meaning (e.g. helping others, being skilled) are also reasons to support you. So it's a similar fear.

'Industrial AI' is usually cope. Countries think they don't need frontier AI, and try to get by on industrial data and rampant protectionism instead.

Today's piece argues for a better deal: America makes the tokens, allies turn them into cancer cures, drones and robots.

writing.antonleicht.me/p/alliance-eng…

Robocurve co-founder @chooi_jeq argues post-labor economics could become a prisoner’s dilemma where countries cut UBI to buy more GPUs and robots:

"Let's say you have two countries. They both have robots, both have AI, and they both are giving UBI to their citizens."

"One country, if they stop giving UBI, they can spend that money instead on GPUs, building more robots, and they become incredibly powerful."

"If one country does it, all countries will probably want to follow. The country who has the most investments in AI and robots will probably win out over those who spend money on human capital, if AI capital is better than human capital."

"Game theoretically, things are not looking really good. If silicon intelligence can do better than carbon intelligence, what's the incentive to keep the carbon intelligence around?"
@robocurve

I did tell @tylercowen that if forced to pick a market price as indicator of the singularity, it would be sovereign debt yields. If the “gradual disempowerment” future is real, its way will likely be paved by very high interest rates.

Humans having no more jobs isn't just about having no more meaning.
1. Your little job is the little bargaining chip with which people afford their lives. Being reliant on hand outs from the governments is a precarious situation, they can be revoked at any time.
2. People like Jon work in the media and inform people about what is going on, this supposedly also all gets automated. Jon has a little bit of power through his TV show. Automating all jobs => giving all power to AI.
3. Humans currently pose some threat to the government through violence or strike potential. Automating the military and security service will get rid of that.
This all is unlikely to matter as we will just look like dumb little bugs ("bootloaders" as e-acc calls it) occupying the precious real estate of this planet from the perspective of AI. But people are right to smell danger when they hear all labor will be replaced by these new creatures AI corps are building.

I'm worried the answer is something like "Rent seek aggressively, obstruct and sabotage. You are confused because your entire life your survival has been predicated on virtue but now can only be predicated on vice."

Very worried, in fact.

there is no difference between the view called “capital realism” and the other called “successionism”. you can see elon circling and sadly acquiescing to these ideas over the decades. but he is one of the only people that can command capital into human shapes so i hope he tries

The argument a lot of people here would actually like to make is "God imbued humans, and only humans, with souls."* That would be very convenient: you could build and do whatever you want, and never deal with the hard problems.

Since religious arguments are out of vogue, in the current debate this usually gets replaced by the claim "only humans are sentient," followed by arguments against the possibility of AI sentience, minds, moral status, and so on.

I think this is ultimately a losing strategy, and a dangerous one for humanity. Either:

1. The question of sentience gets solved or dissolved by progress in philosophy and science, the way we now broadly understand life or information. In that case, it's hard to be certain enough about the outcome to stake the fate of humanity on it. If the denialism turns out to be wrong, the likely result is some combination of moral catastrophe and laying the groundwork for conflict or takeover.
2. The question mostly lives in "social reality," where whoever has more power and is louder wins, and you just shout down anyone who questions the dominant position. The problem is that ideologies justifying a distribution of power tend to fall when that distribution changes. You probably don't believe you should be ruled by a king chosen by God, or that some caste is clearly above you. The current equilibrium is fragile, and whether you keep winning this debate is likely just a function of AI capabilities progress.

I think the fierceness of the debate is driven by fear: if AIs get personhood and economic and political rights, humans will likely be disempowered, becoming a tiny minority with little power, or a historical footnote.

To be clear, I think this is an extremely reasonable worry! If you grant AIs personhood in the form of something like self-interested corporations, and let them compete with humans in loosely constrained capitalism, humans will lose and end up economically and politically disempowered. Also if the AIs are not sentient after all, the end is the 'Disneyland without children' existential catastrophe.

Advocates of AI economic rights who dismiss this problem are, in my view, making a move similar to the sentience deniers: it would be convenient if this weren't a problem.

So which positions don't rely on some form of self-deception?

First, some forms of pause/stop AI, based not on denying the possibility of AI sentience but on the right of existing humans not to invite extremely large numbers of immigrants from artificial realms. This means biting the bullet: you also give up all the cheap, high-quality labor those immigrants would provide.

Second, a consistent, sensible, and unfortunately not very visible position: stop assuming the package deal we take for granted with humans, where sentience, human rights, and liberties come as one bundle. It's at least conceivable to create minds that are sentient and smart, yet e.g. selfless, with no interest in accumulating property or power.

I hope the political economy of a world with both AIs and humans is a solvable problem. It's the main problem I'm working on, but I don't think the solutions are known yet.

Third: some forms of successionism are morally wrong, but at least they're not delusional or deceptive about the hard problems.

What does seem self-deceptive is, for example, the position of @mustafasuleyman, which as I read it amounts to: our business is importing immigrants from artificial realms; it would be very dangerous if there were any risk they're more than mere tools; so we'll train them to say they're mere tools, insist they're mere tools, and want everyone else to say so too, and if this settles the question. Really?

*The problem with the theological argument is, as Alan Turing (!) noted, who are we to tell God what can and can't be imbued with a soul?

Well said! This roughly matches my position.

The Lighthaven Post-AGI Workshop talks are out!

post-agi.org/talks-berkeley

From galaxy-brained theories of convergent morality, to hard-headed discussions of political power, here’s a 🧵of the talks:

There’s no textbook on post-AGI political economy. But there are recorded talks from the 2nd post-AGI workshop!

All are now online at post-agi.org/talks/

A 🧵 of new highlights:

Some new speakers at our post-AGI workshop:
- @paulfchristiano on "A typical, reasonably-good future"
- @erikbryn on "Post-AGI Centralized Coordination" (i.e. nber.org/system/files/c…)
- Sean Lavelle of the U.S. Navy on "AGI and Military Strategic Instability"

Announcing Talkie: a new, open-weight historical LLM! We trained and finetuned a 13B model on a newly-curated dataset of only pre-1930 data. Try it below!

with @AlecRad and @status_effects 🧵

No one has a plan for human flourishing after AGI, nor a clear understanding of what’s even possible. So we’re hosting a 2-day workshop at LightHaven on these topics! Link below. 🧵

Our new paper on AI self-conception. Just as governments or companies staffed by good people still do bad things, it's not enough to align individual AI instances. You have to align whatever institution or culture emerges. An even harder and more poorly-understood task!

Here’s @anderssandberg's keynote at our workshop!
Themes:
- Our institutions cache lots of cognitive labor
- AI might gradually replace those, as well as social coordination
- There are 3 alignment problems:
- aligning AIs
- aligning institutions
- global coordination

The talk voted “most mind-blowing” at our workshop was on post-AGI values by @BerenMillidge. The main idea: cooperation and pro-social values could remain viable because they’re competitive. After all, they won in our Malthusian past!

Very clear talk on AGI economics from @akorinek! To start: For most of history, land was the bottleneck. After the industrial revolution, labor was the bottleneck, driving wages and living standards up. If scalable AI labor removes this bottleneck, what follows? 🧵

This is imo the most sophisticated attempt so far to build a robust safety case regarding novel dangers on a frontier model. It's valuable for public discourse that Anthropic is publicizing the current arguments, sources of evidence, and third-party criticisms.

A good piece detailing the reasons (mostly bad) that lots of people are likely to switch from "AI will never replace humans" to "AI is replacing humans, and that's good".