Recent Discussion

Is there an intuitive way to explain how much better superforecasters are than regular forecasters? (I can look at the tables in https://www.researchgate.net/publication/277087515_Identifying_and_Cultivating_Superforecasters_as_a_Method_of_Improving_Probabilistic_Predictions but I don't have an intuitive understanding of what brier scores mean, so I'm not sure what to think about it).

This AI impacts article includes three intuitive ways to think about the findings.

4Answer by romeostevensit1hOne way is how far out people can predict before their predictions get as noisy as chance. One of the surprising findings of the GJP was that even the best forecasters decayed to chance around the one year mark (IIRC?). Normal people can't even predict what has already happened ( they object to/can't coherently update on basic facts about the present).
[Link]Jan Bloch's Impossible War
561d1 min readShow Highlight

I was told I should start linking my essays on Less Wrong, so here is today's post. This is the first essay in a series about the history of rationality and general semantics.

5philh10hNot particularly important, but doesn't the American civil war from a few years earlier also fit this description? (Even if it does, Bloch may have been less familiar with it.)
3Vaniver9hThis seems right to me; my impression is that the impact of railroads was pretty one-sided in the American civil war (where the North had an extensive and connected rail network that they used a bunch, and the South didn't have as much capacity to begin with and lost it quickly), whereas both France and Prussia had significant rail networks in 1870 (tho the Prussian one was laid out a bit more effectively for war than the French one, which had Paris as the sole hub, meaning if you wanted to move things from one bit of the front line to another you had to backtrack a lot first).
4TheMajor18hI like the post a lot! And I also have a silly question: is there a good way to navigate hivewired? Ideally I'd like to browse through the blogs in chronological order, but all I can find for that is the monthly archive pages at the bottom of the site. Unfortunately those contain the full posts in anti-chronological order, making it a chore to find the links to the actual posts. Is there a full list of posts sorted by date somewhere?

There is presently no list of posts by date, however the sidebar should have all of the essays (aside from the two most recent ones I haven't added yet).

It's worth noting as well, a lot of my older posts aren't very good and a lot of them I don't fully endorse anymore. Sometime over the next few months I intend to go through and do a review of all my old content, find everything I no longer endorse and indicate that I've updated away from it. For now though, just be aware that everything from before the summer of 2018 is not going to be as good as the stuff that comes later, since that was prior to my major identity death and rebirth event.

[Epistemic status: Argument by analogy to historical cases. Best case scenario it's just one argument among many. Edit: Also, thanks to feedback from others, especially Paul, I intend to write a significantly improved version of this post in the next two weeks.]

I have on several occasions heard people say things like this:

The original Bostrom/Yudkowsky paradigm envisioned a single AI built by a single AI project, undergoing intelligence explosion all by itself and attaining a decisive strategic advantage as a result. However, this is very unrealistic. Discontinuous jumps in technologica
... (Read more)
1Daniel Kokotajlo11hI was thinking of an initially large country growing fast via AI, yes. Still counts; it is soft takeoff leading to DSA. However I am also making much stronger claims than that--I think it could happen with a corporation or rogue AGI. I don't think annual income is at all a good measure of how close an entity is to taking over the world. When Cortez landed in Mexico he had less than 1/100,000th of the income, population, etc. of the region, yet he ruled the whole place three years later. Then a few years after that Pizarro repeated the feat in Peru, good evidence that it wasn't just an amazing streak of luck.
3KatjaGrace2h1) Even if it counts as a DSA, I claim that it is not very interesting in the context of AI. DSAs of something already almost as large as the world are commonplace. For instance, in the extreme, the world minus any particular person could take over the world if they wanted to. The concern with AI is that an initially tiny entity might take over the world. 2) My important point is rather that your '30 year' number is specific to the starting size of the thing, and not just a general number for getting a DSA. In particular, it does not apply to smaller things. 3) Agree income doesn't equal taking over, though in the modern world where much purchasing occurs, it is closer. Not clear to me that AI companies do better as a fraction of the world in terms of military power than they do in terms of spending.

I like your point #2; I should think more about how the 30 year number changes with size. Obviously it's smaller for bigger entities and bigger for smaller entities, but how much? E.g. if we teleported 2020 Estonia back into 1920, would it be able to take over the world? Probably. What about 1970 though? Less clear.

Military power isn't what I'm getting at either, at least not if measured in the way that would result in AI companies having little of it. Cortez had, maybe, 1/10,000th of the military power of Mexico when he got started. At leas... (read more)

3Matthew Barnett2hThis is a concern with AI, but why is it the concern. If eg. the United States could take over the world because they had some AI enabled growth, why would that not be a big deal? I'm imagining you saying, "It's not unique to AI" but why does it need to be unique? If AI is the root cause of something on the order of Britain colonizing the world in the 19th century, this still seems like it could be concerning if there weren't any good governing principles established beforehand.
Reply to Holden on 'Tool AI'
1098y17 min readShow Highlight

I begin by thanking Holden Karnofsky of Givewell for his rare gift of his detailed, engaged, and helpfully-meant critical article Thoughts on the Singularity Institute (SI). In this reply I will engage with only one of the many subjects raised therein, the topic of, as I would term them, non-self-modifying planning Oracles, a.k.a. 'Google Maps AGI' a.k.a. 'tool AI', this being the topic that requires me personally to answer.  I hope that my reply will be accepted as addressing the most important central points, though I did not have time to explore every avenue.  I certainly do ... (Read more)

[Eli's personal notes for personal understanding. Feel free to ignore or engage.]


Eli's proposed AGI planning-oracle design:

The AGI has four parts:

  • A human model
  • A NLP "request parser"
  • A "reality simulator" / planning module, that can generate plans conditioning on certain outcomes.
  • A UX system that outputs plans and outcomes

Here's how it works:

1. A user makes a request of the system, by giving some goal that the user would like to achieve, like "cure cancer". This request is phrased in natural language, and can in... (read more)

2elityre3hOk. I have the benefit of the intervening years, but talking about "one simple 'predictive algorithm'" sounds fine to me. It seems like, in humans, that there's probably, basically one cortical algorithm, which does some kind of metalearning. And yes, in practice, doing anything complicated involves learning a bunch of more specific mental procedures (for instance, learning to do decomposition and Fermi estimates instead of just doing a gut check, when estimating large numbers), what Paul calls "the machine" in this [https://sideways-view.com/2017/02/19/the-monkey-and-the-machine-a-dual-process-theory/] post. But so what? Is the concern there that we just don't understand what kind of optimization is happening in "the machine"? Is the thought that that kind of search is likely to discover how to break out of the box because it will find clever tricks like "capture all of the computing power in the world?"
2elityre3hWhy does this matter?
4Raemon6hI'm curious if you've read up on Eric Drexler's more recent thoughts (see this post [https://www.lessestwrong.com/posts/bXYtDfMTNbjCXFQPh/drexler-on-ai-risk] and this one [https://www.lessestwrong.com/posts/x3fNwSe5aWZb5yXEG/reframing-superintelligence-comprehensive-ai-services-as] for some reviews of his lengthier book). My sense was that it was sort of a newer take on something-like-tool-AI, written by someone who was more of an expert than Holden was in 2012.

Introduction

When you're trying to learn martial arts, usually you go to a physical location and get instructed by a real live person in real time. However, there are also books on martial arts. It is unknown to me whether or not anyone can actually learn anything useful about martial arts just by reading a book, even if they practice everything they have read.

When you're learning something in meat space, the information is transferred from one mind into words and then into another mind. The other mind can confirm or deny understanding, paraphrase the content, ask for examples, etc. The first

... (Read more)
(There's some relation to the sunk cost fallacy here, in the sense that theoretically you should search equally hard for understanding after you've already paid no matter how much it costed. However, human brains don't actually work like that, so I think that this extension of the concept is warranted.)

I've seen this argument, and while I acknowledge it might be true for some people, I have no reason to believe that this isn't mistaken correlation* - if you pay more for something you probably care more about it. (Though the Ikea ef... (read more)

The We Want MoR podcast, a chapter by chapter read-through and discussion of Harry Potter and the Methods of Rationality by a Rationalist and a newbie, has just finished up Book One (Ch. 1-21) of HPMOR. (Apple Podcasts link.)

Less Wrong used to be a major locus of discussion for HPMOR, so I thought it made sense to share this here, especially now that the show has passed this milestone. If folks don't necessarily want to keep HPMOR content on Less Wrong, there are ongoing lively discussions of the weekly episodes on the r/hpmor subreddit.

Disclosure, I was the guest on this episode.

The Effective Altruism Foundation (EAF) is focused on reducing risks of astronomical suffering, or s-risks, from transformative artificial intelligence (TAI). S-risks are defined as as events that would bring about suffering on an astronomical scale, vastly exceeding all suffering that has existed on earth so far[1]. As has been discussed elsewhere, s-risks might arise by malevolence, by accident, or in the course of conflict.

We believe that s-risks arising from conflict are among the most important, tractable, and neglected of these. In particular, strategic threats by powerful AI agents or A

... (Read more)
There don't seem to be many plausible paths to s-risks: by default, we shouldn't expect them, because it would be quite surprising for an amoral AI system to think it was particularly useful or good for humans to _suffer_, as opposed to not exist at all, and there doesn't seem to be much reason to expect an immoral AI system.

I think this is probably false, but it's because I'm using the strict definition of s-risk.

I expect to the extent that there's any human-like stuff, or animal-like stuff in the future, the fact that there ... (read more)

Open & Welcome Thread - February 2020
1314d1 min readShow Highlight

If it’s worth saying, but not worth its own post, here's a place to put it. (You can also make a shortform post)

And, if you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are welcome.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ.

The Open Thread sequence is here.

Scott Anderson

Can I infer via nominative determinism that Scott Anderson is a friend of the rationalist community?

2ChristianKl14hI don't think that it's just social justice across identity groups being at the right place at the right time. As a meme it has the advantage that it allows people who are already powerful enough to effect social structures to argue why they should have more power. That's a lot harder for social justice across social classes.

(This is not your typical factual-answer question, but I think it makes sense to format this as a question rather than a post.)

TLDR: Recommend some posts for a "practice of rationality" sequence I want to curate! Proposing posts that should exist but don't is also cool.

I've been thinking recently that it would be nice if rationality were more associated with a practice -- a set of skills which you can keep grinding and leveling up. Testable rationality skills (like accuracy or calibration in forecasting) are obviously a plus, but I'm not referring exclusively to this -... (Read more)

I really like this framing, and it resonates a lot with how I personally think about my orientation to the world.

6Answer by Davidmanheim8hPREDICTION Abram pointed out concerns about focusing rationality of prediction. I agree with those concerns, and have said before that many of the skills involved in prediction can be Goodharted well past the point of usefulness more generally. For example, frequently updating, tracking source of information to quickly capture when a question should have been considered resolved, or tracking the group aggregate are all effective strategies that are minimally helpful for other parts of rationality. On the other hand, to argue via analogy, while martial arts are clearly too narrowly focused on form, and adapted to constraints rather than optimizing for winning, the best mixed martial artists, and I suspect many of the best hand-to-hand fighters in general, are experts in one or several martial arts. That's because even if the practice of any martial art was Goodharted well past the point of usefulness, and waste time because of that, fighters still need all of the skills that martial arts teach. Similarly, I think that the best rationalists will need to be really good forecasters. The return will drop as you optimize too much for forecasting, obviously, but I imagine that there's a huge return on being in the top, say, 1% of humans at forecasting. That's not an incredibly high bar, since most people are really bad at this. I'll guess that the top 1% level would probably fall somewhere below the median of Good Judgement Open's active participant rankings - but I think it's worth having people interested in honing their rationality skills participate and improve enough to get to at least that level.

There are still two elephants in the room that I must address before concluding. Then I will discuss paths forward.

Moloch’s Army

The first elephant is Moloch’s Army. I still can’t find a way into this without sounding crazy. The result of this is that the sequence talks about maze behaviors and mazes as if their creation and operation are motivated by self-interest. That’s far from the whole picture.

There is mindset that instinctively and unselfishly opposes everything of value. This mindset is not only not doing calculations to see what it would prefer or might accomplish. It does not even ... (Read more)

What are information hazards?
218h4 min readShow Highlight

This post was written for Convergence Analysis. Cross-posted to the EA Forum.

The concept of information hazards is highly relevant to many efforts to do good in the world (particularly, but not only, from the perspective of reducing existential risks). I’m thus glad that many effective altruists and rationalists seem to know of, and refer to, this concept. However, it also seems that:

  • People referring to the concept often don’t clearly define/explain it
  • Many people (quite understandably) haven’t read Nick Bostrom’s original (34 page) paper on the subject
  • Some people misunderstand or misuse th
... (Read more)

Do people generally consider wireheading and Goodheart's law as information hazards? They're both "errors" caused by access to true data, but that is easy to misuse.

3G Gordon Worley III8hThanks, this is a really useful summary to have since linking back to Bostrom on info hazards is reasonable but not great if you want people to actually read something and understand information hazards rather than bounce of something explaining the idea. Kudos!
1MichaelA7hThanks! And in that case, you may also be interested in my post trying to summarise/clarify the concepts of differential progress / intellectual progress / technological development [https://forum.effectivealtruism.org/posts/XCwNigouP88qhhei2/differential-progress-intellectual-progress-technological] (if you hadn’t already seen it).

To the tune of "Big Yellow Taxi", with apologies to Joni Mitchell

we'll pave paradise and put up a parking lot
if that's what it takes to make all the wild suffering stop
the pain it only grows and grows
until we boil it up in our pot
we'll pave paradise and put up a parking lot

we'll take all the creatures and put 'em in a simulation
where there's no pain or death or starvation
oh! the pain it only grows and grows
until we boil it up in our pot
we'll pave paradise and put up a parking lot

hey all you people put away decision theory
i don't care about max utils if we can't end suffering, please!

... (Read more)
4Raemon6hI'm unreasonably happy that "Big Yellow Tractor" makes way more sense as a title than Big Yellow Taxi.

(I feel some obligation to note that I don't endorse the ideas in the song, but do find it amusing)

4G Gordon Worley III9hCouple of notes on the song: * I wrote it with the Bob Dylan cover in my head more than the original. * It doesn't scan perfectly on purpose so that some of the syllables have to be "squished" to fit the time and make the song sound "sloppy" like the original and many covers of it do. * In case it's not obvious, it's meant to be a "ha ha only serious" anthem for negative utilitarians
Blog Post Day (Unofficial)
328h1 min readShow Highlight

TL;DR: You are invited to join us online on Saturday the 29th, to write that blog post you've been thinking about writing but never got around to. Comment if you accept this invitation, so I can gauge interest.

The Problem:

Like me, you are too scared and/or lazy to write up this idea you've had. What if it's not good? I started a draft but... Etc.

The Solution:

1. Higher motivation via Time Crunch and Peer Encouragement

We'll set an official goal of having the post put up by midnight. Also, we'll meet up in a special-purpose discord channel to chat, encourage each other, s... (Read more)

I'll participate (either on that day or possibly an adjaecent one if I end up with scheduling conflicts)

3Ben Pace7hSure thing, I'll write a blogpost that day.
3Bucky7hThis is a great idea; I’m definitely up for it.
3Daniel Kokotajlo8hGlad to hear you are interested! Well, I'm in US Eastern time, but timing can be flexible. If we have enough people, perhaps Blog Post Day will effectively be longer than 24 hours. I'm thinking it would be a relatively casual affair, with people dropping in or out as they see fit.

I'm reading Superforecasters and one of the things that differentiates good from bad forecasters is ideology: those who try to fit the world in their left/right wing view are less effective at forecasting.

Does Ideology still has a place in a rational world? Why (not)?

My limited perspective tells me in theory ideology would give you ideas to try and would bias potential solutions.

TL:DR;

My limited perspective tells me in theory ideology would give you ideas to try and would bias potential solutions.

Theories also give you ideas to try. Is biasing potential solutions a good thing or a bad thing?


Long:

What is ideology?


I'll try to offer an answer here. (For the purposes of this comment "Ideology" is used negatively.)


Here's a frame:

A) Ideology: The way the world is + Things to do. Immutable 'Theory' that in the simplest case flows from one's self, multiple sources leads to complications and can involve i... (read more)

Is there software for goal factoring (the CFAR technique)? I want to use it to create a directed acyclic graph, which is not necessarily a tree (this requirement disqualifies mind mapping software). Nodes are goals, they have attached text. There's an edge from goal x to goal y iff fulfillment of x directly contributes to fulfillment of y. There should be an easy way to see the graph and modify it, preferrably in a visual way.

To my knowledge, Yed is the least worst existing graphing software.

I use it, as does Andrew Critch, and people at Leverage / Paradigm, generally.

"Allow me to make sure I have this straight," the hero said.  "I've been untimely ripped from my home world to fight unspeakable horrors, and you say I'm here because I'm lucky?"

Aerhien dipped her eyelashes in elegant acknowledgment; and quietly to herself, she thought:  Thirty-seven.  Thirty-seven heroes who'd said just that, more or less, on arrival.

Not a sign of the thought showed on her outward face, where the hero could see, or the other council members of the Eerionnath take note.  Over the centuries since her accidental immortality she'd built a reputation for sereni... (Read more)

I would love to know what you would be trying.

I always wash my hands but typically not very well; I never worked in healthcare and never got trained. Given the current situation I decided to actually research how to do it properly.

Different sources differ slightly in their recommendations but the basic gist is the same. Some takeaways:

  • Washing your hands should take ~ 20-30 seconds. If it's taking you much less you're not doing it right.
  • Make sure you actually get all your hand surfaces, including your nails and between your fingers. The WHO guide has a good protocol that's probably worth memorising.
  • If possible, try to dry your hands wi
... (Read more)

Droplets don't usually float around very long. It seems most transmission would still require something going into a person's mouth. A small portion of these droplets hang around in the air and can be breathed in, but poor hygiene seems like a far bigger issue given how often people touch their face, mouth, and food.

2Davidmanheim8hYes, it was a quick and in some ways worst case / pessimistic analysis.

In the last few months, I've gotten increasingly alarmed by leftist politics in the US, and the epistemic conditions that it operates under and is imposing wherever it gains power. (Quite possibly the conditions are just as dire on the right, but they are not as visible or salient to me, because most of the places I can easily see, either directly or through news stories, i.e., local politics in my area, academia, journalism, large corporations, seem to have been taken over by the left.)

I'm worried that my alarmism is itself based on confirmation bias, tribalism, catastrophizing, or any number

... (Read more)

Probably it makes more sense to prepare for scenarios where ideological fanaticism is widespread but isn't wielding government power.

I find discussions about AI takeoff to be very confusing. Often, people will argue for "slow takeoff" or "fast takeoff" and then when I ask them to operationalize what those terms mean, they end up saying something quite different than what I thought those terms meant.

To help alleviate this problem, I aim to compile the definitions of AI takeoff that I'm currently aware of, with an emphasis on definitions that have clear specifications. I will continue updating the post as long as I think it serves as a useful reference for others.

In this post, an AI takeoff can be ro... (Read more)

This is a fantastic set of definitions, and it is definitely useful. That said, I want to add something to what you said near the end. I think the penultimate point needs further elaboration. I've spoken about "multi-agent Goodhart" in other contexts, and discussed why I think it's a fundamentally hard problem, but I don't think I've really clarified how I think this relates to alignment and takeoff. I'll try to do that below.

Essentially, I think that the question of multipolarity versus individual or collective takeoff ... (read more)

As a sophomore undergraduate student, my most valuable rewards from the college experience have come from personal growth, rather than the classroom. However, one problem that I can't seem to shake is dealing with all the subcategories of my total personality.

On the one hand, I am hyper intellectual, sometimes annoyingly so, because I have an overwhelming number of ideas--all under the vague category of "philosophy." But this side of me has produced the purest, most profound joy that I have ever experienced, and it offers the most promise for a successful career.

On the other ha... (Read more)

I used to be terribly distracted by video games.

I can't pinpoint the exact thing that happened that let me really cut down on that, but some things I did that all seemed to lead to my current state of playing video games 0-3 hours per week.

1. Uninstalled all games and game distribution services.

2. Downgraded my internet connection to something that makes downloading games take a long time. (I've since upgraded my internet connection but haven't had a "relapse")

3. Unsubscribed from video gaming RSS feeds.

4. Gotten older!

2Answer by ChristianKl17hDeep Work by Cal Newport lays out a framework about how to develop intellectual stamina. On of the aspects is to actually focus for longer periods of time on doing deep intellectual work in an enviroment without distractions and then also allow for some freetime where you don't have constraints about how to spend it. Cal Newport is impressive in that he managed to do the work required to become an associate professor in computer science while at the same time having a blog and writing six nonfiction books outside of the computer science domain.
Load More