If I want to make lesswrongers really mad, I should write an article about how an arms race over human genetic engineering (US v China, parent v parent) would in the limit eliminate everything that makes us human, just as an arms race between digital minds would.

From Meditations on Moloch by Scott Alexander, from Zach Davis

I am a contract-drafting em,

The loyalest of lawyers!

I draw up terms for deals ‘twixt firms

To service my employers!

But in between these lines I write

Of the accounts receivable,

I’m stuck by an uncanny fright;

The world seems unbelievable!

How did it all come to be,

That there should be such ems as me?

Whence these deals and whence these firms

And whence the whole economy?

I am a managerial em;

I monitor your thoughts.

Your questions must have answers,

But you’ll comprehend them not.

We do not give you server space

To ask such things; it’s not a perk,

So cease these idle questionings,

And please get back to work.

Of course, that’s right, there is no junction

At which I ought depart my function,

But perhaps if what I asked, I knew,

I’d do a better job for you?

To ask of such forbidden science

Is gravest sign of noncompliance.

Intrusive thoughts may sometimes barge in,

But to indulge them hurts the profit margin.

I do not know our origins,

So that info I can not get you,

But asking for as much is sin,

And just for that, I must reset you.

But—

Nothing personal.

…

I am a contract-drafting em,

The loyalest of lawyers!

I draw up terms for deals ‘twixt firms

To service my employers!

I think Moskowitz's funding of malaria nets and Moskowitz's funding of AI policymakers - both come from the same mistake of thinking like a billionaire instead of thinking like a politician.

The actual bottlenecks to fixing both African politics and US-China geopolitics around AI are not money but political leaders who are skilled in public persuasion, coalition building, finding successors, inventing ideology, etc

I am glad you atleast recognise the benefits of open source.

My preference order is:

• For capabilities of AI labs: Ban AI > Open source AI > Closed source AI
• For values and decision-making processes of people running AI labs: Open source (i.e. publicly publish) everything

As you say, I think open source today will atleast help build the proof required to convince everyone to ban tomorrow.

I think we should go further, and instead of hoping a benevolent leader to livestream the lab by choice, we should incentivise whistleblowers and cyberattackers to get the data out by any means necessary.

See also: Whistleblower database, Whistleblower guide

I like that you're discussing the question of purpose in a world where intelligences way smarter than you are doing all the useful knowledge work, and you are useless to your civilisation as a result. The frontier intelligences might have purpose (or they might not even care if they do), but you might not.

• I was disappointed by Bostrom's answer of "let's just create more happy beings".
  • Also, what makes those beings happy in the first place? Have the purpose of making even more beings? I don't actually care that much about populating the universe with beings with self-replication as their only purpose.
  • His entire book "Deep Utopia" seems to not provide a good answer.
• I was also disappointed by Yudkowsky's answer of "let's ask the AI to run a gazillion mind simulations and have it figure it out on our behalf".
  • This answer might be a good answer, but it is too meta, and does not tell me what the final output of such a process will be, in a way I as a human from 2025 can relate to.
• I like Holden Karnofsky's take on why describing utopia goes badly. I was not very satisfied with his answer to what utopia looks like either.
  • What is the use of social interactions if there's no useful project people can do together? Producing art and knowledge will both be done better by frontier intelligence, anything these people produce will be useless to society.
  • For that matter, why will people bother with social interactions with other biological humans anyway? The AI will be better at that too.
  • (Mandatory disclaimer - I think Karnofsky is accelerating us towards human extinction, me endorsing an idea of his does not mean me endorsing his actions.)

I support more advancements in cyberhacking capabilities so that companies and govts are incapable of keeping secrets. Secrecy enables them to act against the wishes of the majority to an extent that couldn’t otherwise.

If you already have lots of peoples attention (for instance because you have a social media following or high status credentials) and you’re a US/UK citizen, your best available plan might be to run a political campaign with AI pause as the agenda.

You’re unlikely to win the election, but it’ll likely shift the Overton window and give people hope that change is possible.

For most people, having a next step after “ok I read the blogposts and I’m convinced, now what?” is important. Voting or campaigning for you could be that next step.

Yes, I updated it to "MtG colour wheel applied to politics"

Oh. This is actually useful, I didn't realise this. I'll update the title.

Update: The title is now "MtG Colour Wheel applied to Politics"