Henry Prowbell

Wiki Contributions


You are however only counting one side here


In that comment I was only offering plausible counter-arguments to "the amount of people that were hurt by FTX blowing up is a rounding error."

How to model all the related factors is complicated. Saying that you easily know the right answer to whether the effects are negative or positive in expectation without running any numbers seems to me unjustified. 

I think we basically agree here.

I'm in favour of more complicated models that include more indirect effects, not less.

Maybe the difference is: I think in the long run (over decades, including the actions of many EAs as influential as SBF) an EA movement that has strong norms against lying, corruption and fraud actually ends up more likely to save the world, even if it gets less funding in the short term. 

The fact that I can't predict and quantify ahead of time all the possible harms that result from fraud doesn't convince me that those concerns are unjustified.

We might be living in a world where SBF stealing money and giving $50B to longtermist causes very quickly really is our best shot at preventing AI disaster, but I doubt it. 

Apart from anything else I don't think money is necessarily the most important bottleneck.

If you believe that each future person is as valuable as each present person and there will be 10^100 people in the future lightcone, the amount of people that were hurt by FTX blowing up is a rounding error.


But you have to count the effect of the indirect harms on the future lightcone too. There's a longtermist argument that SBF's (alleged and currently very likely) crimes plausibly did more harm than all the wars and pandemics in history if...

  • Governments are now 10% less likely to cooperate with EAs on AI safety
  • The next 2 EA mega-donors decide to pass on EA
  • (Had he not been caught:) The EA movement drifted towards fraud and corruption
  • etc.

The frequency with which datacenters, long range optical networks, and power plants, require human intervention to maintain their operations, should serve as a proxy to the risk an AGI would face in doing anything other than sustaining the global economy as is.


Probably those things are trivially easy for the AGI to solve itself e.g. with nanobots that can build and repair things.

I'm assuming this thing is to us what humans are to chimps, so it doesn't need our help in solving trivial 21 century engineering and logistics problems.


The strategic consideration is: does the upside of leaving humans in control outweigh the risks. Humans realising you've gone rogue or humans building a competing AGI seem like your 2 biggest threats... much bigger considerations than whether you have to build some mines, power plants, etc. yourself.

keeping alive a form of intelligence with very different risk profiles might be a fine hedge against failure

Probably you keep them alive in a prison/zoo though. You wouldn't allow them any real power.

I haven’t looked into his studies’ methodologies, but from my experience with them, I would put high odds that the 65% number is exaggerated.


From his sales page

"In our scientific study involving 245 people...

65% of participants who completed The 45 Days to Awakening Challenge and Experiment persistently awakened.


Another couple hundred people entered the program already in a place of Fundamental Wellbeing..."

Sounds like he's defining enlightenment as something that ~50% of people already experience.

Elsewhere he describes 'Location 1' enlightenment as a background sense of okayness that doesn't include any kind of non-dual experience and can be interrupted by negative thoughts and emotions.

I can believe that people agreeing with a statement like 'underneath everything I feel that I'm fundamentally okay' might be measuring something psychologically important – but it isn't what most people mean when they use words like enlightenment or persistently awakened.

(P.S. thanks for writing this up and really happy that you got so much from the loving-kindness meditation!)

Does anybody know if the Highlights From The Sequences are compiled in ebook format anywhere?

Something that takes 7 hours to read, I want to send to my Kindle and read in a comfy chair.

And maybe even have audio versions on a single podcast feed to listen to on my commute.

(Yes, I can print out the list of highlighted posts and skip to those chapters of the full ebook manually but I'm thinking about user experience, the impact of trivial inconveniences, what would make Lesswrong even more awesome.)

I love his books too. It's a real shame.

"...such as imagining that an intelligent tool will develop an alpha-male lust for domination."

It seems like he really hasn't understood the argument the other side is making here.

It's possible he simply hasn't read about instrumental convergence and the orthogonality thesis. What high quality widely-shared introductory resources do we have on those after all? There's Robert Miles, but you could easily miss him.

I'm imagining the CEO having a thought process more like...

- I have no idea how my team will actually react when we crack AGI 
- Let's quickly Google 'what would you do if you discovered AGI tomorrow?'*
- Oh Lesswrong.com, some of my engineering team love this website
- Wait what?!
- They would seriously try to [redacted]
- I better close that loophole asap

I'm not saying it's massively likely that things play out in exactly that way but a 1% increased chance that we mess up AI Alignment is quite bad in expectation.

*This post is already the top result on Google for that particular search

I immediately found myself brainstorming creative ways to pressure the CEO into delaying the launch (seems like strategically the first thing to focus on) and then thought 'is this the kind of thing I want to be available online for said CEOs to read if any of this happens?'

I'd suggest for those reasons people avoid posting answers along those lines.

Somebody else might be able to answer better than me. I don't know exactly what each researcher is working on right now.

“AI safety are now more focused on incidental catastrophic harms caused by a superintelligence on its way to achieve goals”

Basically, yes. The fear isn’t that AI will wipe out humanity because someone gave it the goal ‘kill all humans’.

For a huge number of innocent sounding goals ‘incapacitate all humans and other AIs’ is a really sensible precaution to take if all you care about is getting your chances of failure down to zero. As is hiding the fact that you intend to do harm until the very last moment.

“rather than making sure artificial intelligence will understand and care about human values?”

If you solved that then presumably the first bit solves itself. So they’re definitely linked.

Load More