Richard Ngo's case for why artificial general intelligence may pose an existential threat. Written with an aim to incorporate modern advances in machine learning, without taking any previous claims about AGI risk for granted.

Recent Discussion

This is a linkpost for https://forumkarma.com/

I've been posting and commenting pretty frequently over the last few months, and I was curious about some stats. What started as a few GraphQL queries and some Python scripting turned into an interactive web app:

Enter a username, and it will give you some stats and a graph, broken down by post and comment karma. You can use the slider to adjust the date range, and the stats are automatically recalculated for the selected time period.

Another feature is the list of "Gems" - comments with at least a few votes that have the highest net karma score, i.e. comments that received strong upvotes from high-karma users, and few downvotes. I found that this often gives a better sense of a user's best comments than just looking at...

Neat! I found it interesting that 8/10 of my top comments by karma are from pre-LW 2.0. At least some of that is because the rationality quotes threads were good for karma farming, but apparently there were also just way more votes being cast.

This will be slightly inaccurate for contributions from before strong voting was introduced, and more inaccurate for contributions from before self-voting was introduced.

Not important, but I guess there'll also be some inaccuracies to do with vote strength changing. (Out of interest, do you calculate vote strength based on current karma, or their fuzzily-back-computed karma at the time they made the comment/post?)

I've written up a rationality game which we played several times at our local LW chapter and had a lot of fun with. The idea is to put Aumann's agreement theorem into practice as a multi-player calibration game, in which players react to the probabilities which other players give (each holding some privileged evidence). If you get very involved, this implies reasoning not only about how well your friends are calibrated, but also how much your friends trust each other's calibration, and how much they trust each other's trust in each other.

You'll need a set of trivia questions to play. We used these

The write-up includes a helpful scoring table which we have not play-tested yet. We did a plain Bayes loss rather than an adjusted Bayes loss when we played, and calculated things on our phone calculators. This version should feel a lot better, because the numbers are easier to interpret and you get your score right away rather than calculating at the end.

For the last couple of years, the Russian-speaking LW community has been running the AAG online, using this Google Sheets template: https://docs.google.com/spreadsheets/d/1tm4AYBMs8N-ZkdJJeNezG6H6tIPkQ5tHJmiFx8n3_Xo/edit

It supports and calculates points for 2-6 players.

The participants add and update their probabilities and see the history and the points they’ll get.

Feel free to use it!

The game gets better if multiple teams compete for the largest total amount of points instead of individual players competing with each other for individual points (use mult... (read more)

I'm excited to share a special opportunity to create a systemic impact: a statewide approval voting ballot initiative in Missouri. This would affect all elections throughout the state including federal and presidential. Approval voting favors consensus candidates and a more accurate representation of the public's support. This is critical if we want a government to behave in our interests on policies that concern our well-being.

The organization leading this charge is Show Me Integrity, where I'm currently doing a fellowship and assisting with fundraising efforts. Show Me Integrity has successfully passed a ballot initiative before, showing their ability to succeed on this kind of scale. They also successfully ran the ballot initiative for approval voting in St. Louis.

Why is this important?

Approval voting is a method that allows voters...

1mlinksva6h
Great cause, godspeed! Interestingly it appears Show Me Integrity and Missouri Agrees have rebranded approval voting as freedom voting.  https://www.showmeintegrity.org/freedomvoting [https://www.showmeintegrity.org/freedomvoting]  https://www.missouriagrees.org/learn-more [https://www.missouriagrees.org/learn-more] Or if the rebranding already existed elsewhere, I'm idly curious about its origin. No mention yet in https://en.wikipedia.org/wiki/Approval_voting [https://en.wikipedia.org/wiki/Approval_voting] 

If I understand their draft language it looks problematic. It seems like they designed it so that people who lose primaries generally have no chance to appear on the ballot. I don't see a good reason to give political parties that much power over ballot access. 

Having a system where an incumbent who loses a primary can't appear on the ballot means that the benefit of protecting incumbents from extremist primary challenges disappears.

Another alternative would be to allow the top two candidates from each party primary ballot access for the general election. 

Let's say you have a few million tabs open in your mobile Chrome browser, because you never close anything, but now your browser is getting slow and laggy. You want to stick the URLs of those tabs somewhere for safekeeping so that you can close them all.

There's a lot of advice on doing this on the Internet, most of which doesn't work.

Here's a method that does work. It's a bit of a hack, but gives good results:

  • Enable developer tools on your Android phone: go to Settings -> About phone, scroll down to "Build number", and tap it repeatedly until it tells you you're a developer. (Seriously.)
  • Enable USB debugging on your phone: go to Settings -> System -> Developer options and make sure the "USB debugging" slider is
...

FYI, in ther answer you linked to, there is another, way easier way of doing it (& it worked for me):

tl;dr:

  • have the Android command line tools installed on a development machine, and USB debugging enabled on your device. The device does not need to be rooted
  • adb forward tcp:9222 localabstract:chrome_devtools_remote
  • wget -O tabs.json http://localhost:9222/json/list

For the purposes of this post, the anthropic shadow is the type of inference found in How Many LHC Failures Is Too Many?.

"Anthropic principle! If the LHC had worked, it would have produced a black hole or strangelet or vacuum failure, and we wouldn't be here!"

In other words, since we are "blind" to situations in which we don't exist, we must adjust how we do bayesian updating. Although it has many bizarre conclusions, it is more intuitive than you think and quite useful!

There are many similar applications of anthropics, such as Nuclear close calls and Anthropic signature: strange anti-correlations.

This actually has implications for effective altruism. Since we are so early into humanity's existence, we can infer from the anthropic shadow that humans will probably soon die out....

2dr_s2h
I mean, wouldn't that timeline be more like "my consciousness keeps somehow hanging onto a husk of a body out of increasingly unlikely coincidences as the universe approaches heat death"? Quantum Immortality strikes me as a terrible world that I really hope isn't true.

Yes that's what I take would happen too unless I'm misunderstanding something? Because it would seem far more probable for *just* your consciousness to somehow still exist, defying entropy, than for the same thing to happen to an entire civilization (same argument why nearly all Boltzmann brains would be just a bare "brain").

1dr_s2h
I either am not following, or disagree. You're just making this anthropic ghost a non-agent, but not unable to experience (dis)utility. If I know that a certain choice leads to death, but I expect that after death I'll still experience stuff, what I decide to do depends on what I imagine death to be like - is it going to be anthropic Heaven or anthropic Hell? Or just a Purgatory? This tangibly affects expected utility before death and thus my behaviour. It also changes the odds of there being a "me" that experiences a past with a deadly event in it - it will be a powerless me, but still a sentient one. The anthropic shadow approach requires death to be the end of sensation, not just of agency.

A podcast interview (posted 2023-06-29) with noted AI researcher Douglas Hofstadter discusses his career and current views on AI.

Hofstadter has previously energetically criticized GPT-2/3 models (and deep learning and compute-heavy GOFAI). These criticisms were widely circulated & cited, and apparently many people found Hofstadter a convincing & trustworthy authority when he was negative on deep learning capabilities & prospects, and so I found his comments in this most recent discussion of considerable interest (via Edward Kmett).

Below I excerpt from the second half where he discusses DL progress & AI risk:

    • Q: ...Which ideas from GEB are most relevant today?

    • Douglas Hofstadter: ...In my book, I Am a Strange Loop, I tried to set forth what it is that really makes a self or a soul. I like to use the word "soul", not in the religious

...
14mishka5h
I felt exactly the same, until I had read this June 2020 paper: Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention [https://arxiv.org/abs/2006.16236]. It turns out that using Transformers in the autoregressive mode (with output tokens being added back to the input by concatenating the previous input and the new output token, and sending the new versions of the input through the model again and again) results in them emulating dynamics of recurrent neural networks, and that clarifies things a lot...

Yeah, there's obviously SOME recursion there but it's still surprising that such a relatively low bandwidth recursion can still work so well. It's more akin to me writing down my thoughts and then rereading them to gather my ideas than the kind of loops I imagine our neurons might have.

That said, who knows, maybe the loops in our brain are superfluous, or only useful for learning feedback purposes, and so a neural network trained by an external system doesn't need them.

7Chris_Leong3h
In what sense do they emulate these dynamics?
9Cole Wyeth8h
It has become slightly more plausible that Melanie Mitchell could come around. 
To get the best posts emailed to you, create an account! (2-3 posts per week, selected by the LessWrong moderation team.)
Subscribe to Curated posts
Log In Reset Password
...or continue with

This is my fifth attempt at writing this post. I’m starting to think that I’ve already spent way too much time on this topic, which I’m convinced is valuable, but maybe not so valuable as to spend 20 hours perpetually rewriting a post about it. So obviously my solution is to rewrite it again, but this time in bullet points. 

Here’s a tl;dr: There are some habits people can pick up that are very cheap, and may have positive effects, but these effects are too small to reliably notice consciously. Hence these habits are often neglected. In this post I argue to take some of these habits more seriously, and if they’re low-cost enough for you to implement, stick to them even absent of any feeling of them being useful.


  • One
...

One of the main ways I managed to instill good habits in myself is to both use optimal paths to good habits, and closing optimal paths to sub-optimal habits. The trick is to make a good habit easier than it is annoying, and a bad habit more annoying than it is preferable.

Examples:

Hydration - I simply place a 2l water bottle by the apartment door every evening. It becomes impossible for me to leave the house without picking it up, and once it is in my hand, Im so much more likely to drink from it and take it with me than forget.

Exercise: I bought dumbbells ... (read more)

1Celenduin5h
What is "physical fiction"?

When trying to improve the world via philanthropy, there are compelling reasons to focus on nurturing individual talent rather than supporting larger organizations, especially those with nebulous and unquantifiable goals.

Tyler Cowen's Emergent Ventures is a prime example of this approach, providing grants to individual entrepreneurs and thinkers who aim to make a significant societal impact. When asked how his approach to philanthropy differs from the Effective Altruist approach, Cowen answers: 

I’m much more “person first.” I’m willing to consider, not any area—it ought to feel important—but I view it as more an investment in the person, and I have, I think, more faith that the person’s own understanding of what’s important will very often be better than mine. That would be the difference.

This model has been effective in...

16David Hornbein7h
If you make programmer money and a bunch of your friends are working on weird projects that take two hours to explain and justify—and I know that describes a lot of people here—then you're in an excellent position to do this. Essentially it's legibility arbitrage.

I would expect that one of the key reasons why many people would not do this, is because it's socially weird and they are uncertain about how to handle how that changes their social relationship to the people around them.

Especially, given that many programmers are more on the shy side, writing a check to a GiveWell-recommended charity is easier. I think it would be valuable if someone who acts like that would write more about their experience doing it, so that people have an easier model to copy.

14David Hornbein7h
I've had very good results from offering unsolicited no-strings $X,000 gifts to friends who were financially struggling while doing important work. Some people have accepted, and it took serious pressure off them while they laid the foundations for what's become impressive careers. Some turned me down, although I like to imagine that knowing they had backup available made them feel like they had more slack. It's a great way to turn local social knowledge into impact. The opportunities to do this aren't super frequent, but when it arises the per-dollar impact is absolutely insane. Some advice for anyone who's thinking of doing this: —Don't mention it to anyone else. This is sensitive personal stuff. Your friend doesn't want some thirdhand acquaintance knowing about their financial hardship. Talking about it in general terms without identifying information is fine. —Make gifts, not loans. Loans often put strain on relationships and generally make things weird. People sometimes make "loans" they can't afford to lose without realizing what they're doing, but you're not gonna give away money without understanding the consequences. Hold fast to this; if the recipient comes back three years later when they're doing better and offers to pay it back (this has happened to me), tell them to pay it forward instead. —The results will only be as good as your judgements of character and ability. Everyone makes mistakes, but if you make *big* mistakes on either of those, then this probably isn't your comparative advantage.
1NinaR5h
This sounds great - I think many underestimate the effectiveness of this kind of direct support. When giving money directly to talented and well-motivated people you know personally, you are operating with much more information, there are no middlemen so it’s efficient, and it promotes prosocial norms in communities. They can also redistribute if they think it’s wise at some point - as you mentioned, paying it forward.

[Thanks to Charlie Steiner, Richard Kennaway, and Said Achmiz for helpful discussion.]

[Epistemic status: my best guess after having read a lot about the topic, including all LW posts and comment sections with the consciousness tag]

There's a common pattern in online debates about consciousness. It looks something like this:

One person will try to communicate a belief or idea to someone else, but they cannot get through no matter how hard they try. Here's a made-up example:


"It's obvious that consciousness exists."

-Yes, it sure looks like the brain is doing a lot of non-parallel processing that involves several spatially distributed brain areas at once, so-

"I'm not just talking about the computational process. I mean qualia obviously exists."

-Define qualia.

"You can't define qualia; it's a primitive. But you know what I mean."

-I...

I tend to think that, regardless of which camp is correct, it's unlikely that the difference is due to different experiences, and more likely that one of the two sides is making a philosophical error. Reason being that experience itself is a low-level property, whereas judgments about experience are a high-level property, and it generally seems to be the case that the variance in high-level properties is way way higher.

E.g., it'd be pretty surprising if someone claimed that red is more similar to green than to orange, but less surprising if they had a stra... (read more)

3Signer4h
I agree that the epistemic status of experience is important, but... First of all does anyone actually disagree with concrete things that Dennett says? That people are often wrong about their experiences is obviously true. If that was the core disagreement, it would be easy to persuade people. The only persistent disagreement seems to be about whether there is something additional to the physical explanation of experience (hence the zombies argument) or whether fundamental consciousness is even coherent concept at all - just replacing absolute certainty with uncertainty wouldn't solve it, when you can't even communicate what's your evidence is.
2Rafael Harth3h
Yes; there are definitely people who disagree with most things Dennett says, including how exactly you can be wrong about your experience. Don't really want to get into the details here since that's not part of the post.
3Rafael Harth5h
Yeah, I agree with both points. I edited the post to reflect it; for the whole brain vs parts thing I just added a sentence; for the kind of access thing I made it a footnote and also linked to your comment. As you said, it does seem like a refinement of the model rather than a contradiction, but it's definitely important enough to bring up.