All of KatjaGrace's Comments + Replies

What do ML researchers think about AI in 2022?

We didn't do rounding though, right? Like, these people actually said 0?

That doesn't mean it wasn't rounding though. People 'helpfully' round their own answers all the time before giving them. '0' as a probability simply means that it isn't going to happen, not necessarily that it couldn't, while '100' then means something that will definitely happen (though it may not be a logical necessity). In some cases '1%' could actually be lower than '0%' since they are doing different things (and 1% is highly suspect as a round number for 'extremely unlikely' too.) Ditto for '99%' being higher than '100%' sometimes (while 99% is also a suspicious round number too for, 'I would be very surprised if it didn't happen'.) I don't think that it would necessarily be telling, but it might be interesting to look at it without numbers from '99%-100%' and '0%-1%', and then directly compare them to the results with those numbers included..
2Zach Stein-Perlman4d
Survey supports ‘long covid is bad’ hypothesis (very tentative)

I did ask about it, data here (note that n is small):

Why do people avoid vaccination?

Yeah, I meant that early on in the vaccinations, officialish-seeming articles said or implied that breakthrough cases were very rare (even calling them 'breakthrough cases', to my ear, sounds like they are sort of more unexpected than they should be, but perhaps that's just what such things are always called). That seemed false at the time even, before later iterations of covid made it more blatantly so.  I think it was probably motivated partly by desire to convince people that the vaccine was very good, rather than just error, which I think is questionable behavior.

Ah, got it. Thanks!
Long covid: probably worth avoiding—some considerations

I agree that I'm more likely to be concerned about in-fact-psychosomatic things than average, and on the outside view, thus probably biased in that direction in interpreting evidence. Sorry if that colors the set of considerations that seem interesting to me. (I didn't mean to claim that this was an unbiased list, sorry if I implied it. )

Some points regarding the object level:

  1. The scenario I described was to illustrate a logical point (that the initially tempting inference from that study wasn't valid). So I wouldn't want to take the numbers from that hypot
... (read more)

Good points. Some responses:

  • I put a lot more trust in a single study with ground-truth data than in a giant pile of studies with data which is confounded in various ways. So, I trust the study with the antibody tests more than I'd trust basically-any number of studies relying on self-reports. (A different-but-similar application of this principle: I trust the Boston wastewater data on covid prevalence more than I trust all of the data from test results combined.)
  • I probably do have relatively high prior (compared to other people) on health-issues-in-general
... (read more)
COVID and the holidays

I thought rapid tests were generally considered to have a much lower false negative rate for detecting contagiousness, though they often miss people who are infected but not yet contagious. I forget why I think this, and haven't been following possible updates on this story, but is that different from your impression? (Here's one place I think saying this, for instance: On this story, rapid tests immediately before an event would reduce overall risk by a lot.

Ah, good point. The main reason I don't automatically make a huge adjustment for this is that it seemed that there were still ~25% false negatives at peak contagiousness by PCR according to some studies. And 40% false negatives a few days later. All sort of things are possible, like that these studies included many asymptomatic and acontagious cases, but having seen anecdotal corroboration of this phenomenon, I'm inclined to think something weird is going on. But I should give some weight to it though—perhaps 2x less ineffective when compared to contagiousness, so maybe 60% efficacy for rapid and 75% for PCR?
Beyond fire alarms: freeing the groupstruck

Agree the difference between actors and real companions is very important! I think you misread me (see response to AllAmericanBreakfast's above comment.) 

Your current model appears to be wrong (supposing people should respond to fire alarms quickly).

From the paper:

"Subjects in the three naive bystander condition were markedly inhibited from reporting the smoke. Since 75% of the alone subjects reported the smoke, we would expect over 98% of the three-person groups to contain at least one reporter. In fact, in only 38% of the eight groups in this condit... (read more)

Oh, very interesting. I will take a look at the paper in the next few days. That does seem like it contradicts with my beliefs!
Beyond fire alarms: freeing the groupstruck

Sorry for being unclear.  The first video shows a rerun of the original experiment, which I think is interesting because it is nice to actually see how people behave, though it is missing footage of the (I agree crucial) three group case. The original experiment itself definitely included groups of entirely innocent participants, and I agree that if it didn't it wouldn't be very interesting. (According to the researcher in the footage, via private conversation, he recalls that the filmed rerun also included at least one trial with all innocent people,... (read more)

Ah, now I see. My bad. By the end of the video, I'd lost the nuance that you never stated the clip was of the 3 innocent subjects portion of the experiment. Might be worth signposting that explicitly? I'll retract my comment above, but I also suspect that others may make the same mistake that I did. On introspection, I think the issue is that I read the "3 innocent subjects" bit, visualized it in my mind, got interested to see it play out, and spent 8 minutes waiting to see it on the video clip. Not seeing it, I just immediately thought, "oh, must have been an incorrect description of the video," rather than going back to carefully think about your wording. So definitely my bad, but also a mistake I think some others are likely to make, and which may be worth anticipating as a writer. Note to self for any future blog posts with embedded video!
Ask Not "How Are You Doing?"

I think I would have agreed that answering honestly is a social gaffe a few years ago, and in my even younger years I found it embarrassing to ask such things when we both knew I wasn't trying to learn the answer, but now I feel like it's very natural to elaborate a bit, and it usually doesn't feel like an error. e.g. 'Alright - somewhat regretting signing up for this thing, but it's reminding me that I'm interested in the topic' or 'eh, seen better days, but making crepes - want one?' I wonder if I've become oblivious in my old age, or socially chill, or ... (read more)

Feedback is central to agency

To check I have this: in the two-level adaptive system, one level is the program adjusting its plan toward the target configuration of being a good plan, and the other level is the car (for instance) adjusting its behavior (due to following the plan) toward getting to a particular place without crashing?

Taboo "Outside View"

Fwiw I'm not aware of using or understanding 'outside view' to mean something other than basically reference class forecasting (or trend extrapolation, which I'd say is the same). In your initial example, it seems like the other person is using it fine - yes, if you had more examples of an AGI takeoff, you could do better reference class forecasting, but their point is that in the absence of any examples of the specific thing, you also lack other non-reference-class-forecasting methods (e.g. a model), and you lack them even more than you lack relevant refe... (read more)

7Daniel Kokotajlo1y
Thanks for the pushback! Fair enough if your experience is different than mine. Lots of people I've talked to seem to have had experiences similar to mine. To be clear it's not just that I've seen other people abusing the concept, I think I've caught myself doing it too on several occasions. I think quite a lot of things can be thought of as a form of reference class forecasting if you stretch it enough, but that's true of a lot of things, e.g. quite a lot of things can be thought of as a form of modelling, or logical deduction, or intuition, if you stretch those concepts far enough. I guess we are getting into the question of what the charitable interpretation of their statement is. We could interpret it in the way you mentioned, but then it would be a pretty big and obvious non sequitur -- Obviously reference classes work less well the fewer examples of the thing (and things similar to the thing) you have, but part of what's interesting about other things (e.g. deduction, gears-level models) is that they often work fine in completely novel cases. For example, the first human landing on the Moon was not achieved via reference classes, but via gears-level modelling. Do you follow it in the case where "outside view" has the expansive, ambiguous new meaning I've complained about? I feel like it's pretty clear that's not what Tetlock meant. In the case where "outside view" means reference classes, here are my complaints: 1. I don't think the "only use inside view if you feel like you are a qualified expert" bit is justified by Tetlock's advice. He tells everyone to adjust with inside view, and he has all this evidence about how experts suck when they use inside view. 2. "Have some outside views, and some inside views, and then aggregate them" is not the same thing as "start with an outside view, then adjust it, then get an entirely different perspective and repeat, then aggregate the perspectives." And note that even that is not exactly what he said IMO; my summa
Holidaying and purpose

I too thought the one cruise I've been on was a pretty good type of holiday! A giant moving building full of nice things is so much more convenient a vehicle than the usual series of planes and cabs and subways and hauling bags along the road and stationary buildings etc.

Yeah, exactly my thoughts. For anyone considering a cruise, I will note that different cruise lines cater to people looking for different kinds of things. Some are more aimed at people looking for non-stop partying, others at people looking for a more relaxed on-board atmosphere.
Coherence arguments imply a force for goal-directed behavior

I wrote an AI Impacts page summary of the situation as I understand it. If anyone feels like looking, I'm interested in corrections/suggestions (either here or in the AI Impacts feedback box).  

Looks good to me :)

Coherence arguments imply a force for goal-directed behavior

A few quick thoughts on reasons for confusion:

I think maybe one thing going on is that I already took the coherence arguments to apply only in getting you from weakly having goals to strongly having goals, so since you were arguing against their applicability, I thought you were talking about the step from weaker to stronger goal direction. (I’m not sure what arguments people use to get from 1 to 2 though, so maybe you are right that it is also something to do with coherence, at least implicitly.)

It also seems natural to think of ‘weakly has goals’ as some... (read more)

4Rohin Shah1y
Thanks, that's helpful. I'll think about how to clarify this in the original post.
Coherence arguments imply a force for goal-directed behavior

Thanks. Let me check if I understand you correctly:

You think I take the original argument to be arguing from ‘has goals' to ‘has goals’, essentially, and agree that that holds, but don’t find it very interesting/relevant.

What you disagree with is an argument from ‘anything smart’ to ‘has goals’, which seems to be what is needed for the AI risk argument to apply to any superintelligent agent.

Is that right?

If so, I think it’s helpful to distinguish between ‘weakly has goals’ and ‘strongly has goals’:

  1. Weakly has goals: ‘has some sort of drive toward something,
... (read more)
I get the feeling that a one-dimensional view of directed/purposeful behavior (weak -> strong) is constraining the discussion. I do not think we can take as obvious the premise that biologically-evolved intelligence is more strongly directed than other organisms, which, in many different ways, are all strongly directed to reproduction, but it seems there are more dimensions to intelligent purposeful behavior. There are some pathological forms of directedness, such as obsessive behavior and drug addiction, but they have not come to dominate in biological intelligence. We generally pursue multiple goals which cannot all be maximized together, and our goals are often satisfied with some mix of less-than-maximal outcomes. So, even if coherence does imply a force for goal-directed behavior, we have some empirical evidence that it is not necessarily directed towards pathological versions - unless, of course, we have just not been around long enough to see that ultimately, it does!

Yes, that's basically right.

You think I take the original argument to be arguing from ‘has goals' to ‘has goals’, essentially, and agree that that holds, but don’t find it very interesting/relevant.

Well, I do think it is an interesting/relevant argument (because as you say it explains how you get from "weakly has goals" to "strongly has goals"). I just wanted to correct the misconception about what I was arguing against, and I wanted to highlight the "intelligent" --> "weakly has goals" step as a relatively weak step in our current arguments. (In my ori... (read more)

Animal faces

Good points. Though I claim that I do hold the same facial expression for long periods sometimes, if that's what you mean by 'not moving'. In particular, sometimes it is very hard for me not to screw up my face in a kind of disgusted frown, especially if it is morning. And sometimes I grin for so long that my face hurts, and I still can't stop.

(Lesswrong version here: [] )
Tentative covid surface risk estimates

It doesn't seem that hard to wash your hands after putting away groceries, say. If I recall, I was not imagining getting many touches during such a trip. I'm mostly imagining that you put many of the groceries you purchase in your fridge or eat them within a couple of days, such that they are still fairly contaminated if they started out contaminated, and it is harder to not touch your face whenever you are eating recently acquired or cold food.

Wordtune review

Yes - I like 'application' over 'potentially useful product' and 'my more refined writing skills' over 'my more honed writing', in its first one, for instance.

Neck abacus

I grab the string and/or some beads I don't want to move together between my thumb and finger on one hand, and push the bead I do want to move with my thumb and finger of the other hand. (I don't need to see it because I can feel it and the beads don't move with my touching it.) I can also do it more awkwardly with one hand.

Neck abacus

Thanks for further varieties! I hadn't seen the ring, and have had such a clicker but have not got the hang of using it non-awkwardly (where do you put it? With your keys? Who knows where those are? In your pocket? Who reliably has a pocket that fits things in? In your bag? Then you have to dig it out..)

Good point regarding wanting to know what number you have reached. I only want to know the exact number very occasionally, like with a bank account, but I agree that's not true of many use cases.

Unpopularity of efficiency

I haven't read Zvi's post, but would have thought that the good of slack can be cashed out in efficiency, if you are optimizing for the right goals (e.g. if you have a bunch of tasks in life which contribute to various things, it will turn out that you contribute to those things better overall if you have spare time between the tasks).  

If you aren't in the business of optimizing for the ultimately right goals though, I'd think you could also include slack as one of your instrumental goals, and thus mostly avoid serious conflict e.g. instead of turnin... (read more)

There is probably a process to burn slack to get efficiency and to use efficiency to create slack. I am somewhat skeptical that just having spare time would make overall time spent less. The way I would imagine a slack approach working out for greater efficiency would be like having a drinking pause and conversing with a buddy who gives a tip about the cookie making that makes it go more smoothly. I would also think that instead of using only half the time a slack approach would be just bake cookies to achieve mastery better by being deliberate and slow instead of setting tighter and higher bars. Like pausing to wonder about the philosophy of baking. I have also this analog about anticipating things go wrong. In a military setting having a reseve can be used to address when a default operation goes wrong. A optimization focused mind might think that they need to assign the minimum amount of soldiers to get each task done so that reserve is as big as it can get so it can more forcefullly address problems. But overassigning soldiers to tasks makes each of them less likely to fall. So being slack about it could mean that you want a reserve so that there is flexibility if main plan goes awry but you want the main plan to be flexible enough so it doesn't brittle immidietly on the first hiccups. If it was important that the cookies are made (big party or something) I would probably do them slowly rather than doing them quick and then idling. For big things the difference of being able to withstand 0,1 or 2 catasrophes is pretty big. If you do it quick and have some probability of having to do it from scratch again there is some probablity of spending double time on it. So one approach of increasing the expectancy of success would be to buff up the reliability rather than the number of shots. If you can aim that one bullseye it doesn't matter how many arrows per second you can shoot at the target. One could think this as bullseyes per arrow in eficiency terms. But one c
Li’l pots

Thanks. What kind of gloves do you suggest?

Latex or nitrile, like you'd find at a hospital. Most grocery stores will sell reusable rubbery gloves for cleaning, but those are usually too thick and oversized to get the fine motor control I want when cooking.
Blog plant

I actually know very little about my plants at present, so cannot help you.

Blog plant

It is irrigation actually, not moisture sensors. Or rather, I think it irrigates based on the level of moisture, using a combination of tiny tubes and clay spikes that I admittedly don't fully understand. (It seems to be much better at watering my plants than I am, even ignoring time costs!) I do have to fill up the water container sometimes.

What technologies could cause world GDP doubling times to be <8 years?

I meant: conditional on it growing faster, why expect this is attributable to a small number of technologies, given that when it accelerated previously it was not like that (if I understand)?

3Daniel Kokotajlo2y
Another, somewhat different reply occurs to me: Plausibly the reason why growth rates have been roughly steady for the past sixty years or so is that world population growth has slowed down (thanks to education, birth control, etc.). So on this view, there's technological growth and there's population growth, and combined they equal GWP growth, and combined they've been on a hyperbolic trajectory for most of history but recently are merely on an exponential trajectory thanks to faltering population growth. If this story is right, then in order for GWP growth to accelerate again, we either need to boost population growth, or boost technological growth even more than usual, i.e. even more than was the case in previous periods of GWP acceleration like the industrial revolution or the agricultural revolution or, well, literally any period. So, I'd conclude that it's unlikely for GWP growth to accelerate again, absent specific reasons to think this time will be different. AGI is one such reason. The other answers people are giving are other reasons (though I don't find them plausible.)
1Daniel Kokotajlo2y
Ah, OK. Good point. I think when it accelerated previously, it was the result of a small number of technologies, so long as we are careful to define our technologies broadly enough. For example, we can say the acceleration due to the agricultural revolution was due to agriculture + a few other things maybe. And we can say the acceleration due to the industrial revolution was due to engines + international trade + mass-production methods + scientific institutions + capitalist institutions + a few other things I'm forgetting. I'm looking for something similar here; e.g. Abram's answer "We automate everything, but without using AGI" is acceptable to me, even though it's only a single technology if we define our tech extremely broadly.
What technologies could cause world GDP doubling times to be <8 years?

If throughout most of history growth rates have been gradually increasing, I don't follow why you would expect one technology to cause it to grow much faster, if it goes back to accelerating.

2Daniel Kokotajlo2y
I currently don't expect it to grow much faster, at least not until we have AGI. Is your question why I think AGI would make it grow much faster? Roughly, my answer is "Because singularity." But if you think not even AGI would make it grow much faster -- which in this context means >9% per year -- then that's all the more reason to think "Economy doubles in 4 years before the first 1-year doubling" is a bad metric for what we care about. (To clarify though, I don't intend for the question to be only answerable by single technologies. Answers can list several technologies, e.g. all the ones on my list.)
Why are delicious biscuits obscure?

Thanks for contributing data! :D 

Why are delicious biscuits obscure?

They are meant to be chewy, not crumbly.

Why are delicious biscuits obscure?

Making them tastier, though not confident about this - originally motivated by not having normal flour, and then have done some of each, and thought the gluten free ones were better, but much randomness at play. 

I did mean 'white' by 'wheat'; sorry (I am a foreigner). I haven't tried anything other than the gluten free one mentioned and white wheat flour.

Automated intelligence is not AI

>Someone's cognitive labor went into making the rabbit mold, and everything from there on out is eliminating the need to repeat that labor, and to reduce the number of people who need to have that knowledge.

Yeah, that's the kind of thing I had in mind in the last paragraph.

My dad got stung by a bee, and is mildly allergic. What are the tradeoffs involved in deciding whether to have him go the the emergency room?

In such a case, you might get many of the benefits without the covid risks from driving to very close to the ER, then hanging out there and not going in and risking infection unless worse symptoms develop, but being able to act very fast if they do.

Soft takeoff can still lead to decisive strategic advantage

1) Even if it counts as a DSA, I claim that it is not very interesting in the context of AI. DSAs of something already almost as large as the world are commonplace. For instance, in the extreme, the world minus any particular person could take over the world if they wanted to. The concern with AI is that an initially tiny entity might take over the world.

2) My important point is rather that your '30 year' number is specific to the starting size of the thing, and not just a general number for getting a DSA. In particular, it does not apply to sma... (read more)

1Daniel Kokotajlo2y
I like your point #2; I should think more about how the 30 year number changes with size. Obviously it's smaller for bigger entities and bigger for smaller entities, but how much? E.g. if we teleported 2020 Estonia back into 1920, would it be able to take over the world? Probably. What about 1970 though? Less clear. Military power isn't what I'm getting at either, at least not if measured in the way that would result in AI companies having little of it. Cortez had, maybe, 1/10,000th of the military power of Mexico when he got started. At least if you measure in ways like "What would happen if X fought Y." Probably 1/10,000th of Mexico's military could have defeated Cortez' initial band. If we try to model Cortez' takeover as him having more of some metric than all of Mexico had, then presumably Spain had several orders of magnitude more of that metric than Cortez did, and Western Europe as a whole had at least an order of magnitude more than that. So Western Europe had *many* orders of magnitude more of this stuff, whatever it is, than Mexico, even though Mexico had a similar population and GDP. So they must have been growing much faster than Mexico for quite some time to build up such a lead--and this was before the industrial revolution! More generally, this metric that is used for predicting takeovers seems to be the sort of thing that can grow and/or shrink orders of magnitude very quickly, as illustrated by the various cases throughout history of small groups from backwater regions taking over rich empires. (Warning: I'm pulling these claims out of my ass, I'm not a historian, I might be totally wrong. I should look up these numbers.)
3Matthew Barnett2y
This is a concern with AI, but why is it the concern. If eg. the United States could take over the world because they had some AI enabled growth, why would that not be a big deal? I'm imagining you saying, "It's not unique to AI" but why does it need to be unique? If AI is the root cause of something on the order of Britain colonizing the world in the 19th century, this still seems like it could be concerning if there weren't any good governing principles established beforehand.
Soft takeoff can still lead to decisive strategic advantage

The time it takes to get a DSA by growing bigger depends on how big you are to begin with. If I understand, you take your 30 years from considering the largest countries, which are not far from being the size of the world, and then use it when talking about AI projects that are much smaller (e.g. a billion dollars a year suggests about 1/100,000 of the world). If you start from a situation of an AI project being three doublings from taking over the world say, then most of the question of how it came to have a DSA seems to be the question of how it grew the... (read more)

1Daniel Kokotajlo2y
I was thinking of an initially large country growing fast via AI, yes. Still counts; it is soft takeoff leading to DSA. However I am also making much stronger claims than that--I think it could happen with a corporation or rogue AGI. I don't think annual income is at all a good measure of how close an entity is to taking over the world. When Cortez landed in Mexico he had less than 1/100,000th of the income, population, etc. of the region, yet he ruled the whole place three years later. Then a few years after that Pizarro repeated the feat in Peru, good evidence that it wasn't just an amazing streak of luck.
LW For External Comments?

This sounds great to me, and I think I would be likely to sign up for it if I could, but I haven't thought about it for more than a few minutes, am particularly unsure about the implications for culture, and am maybe too enthusiastic in general for things being 'well organized'.

I don't think so. :)
Pieces of time

Oh yeah, I think I get something similar when my sleep schedule gets very out of whack, or for some reason when I moved into my new house in January, though it went back to normal with time. (Potentially relevant features there: bedroom didn't seem very separated from common areas, at first was sleeping on a pile of yoga mats instead of a bed, didn't get out much.)

jacobjacob's Shortform Feed

I think random objects might work in a similar way. e.g. if talking in a restaurant, you grab the ketchup bottle and the salt to represent your point. I've only experimented with this once, with ultimately quite an elaborate set of condiments, tableware and fries involved. It seemed to make things more memorable and followable, but I wasn't much inclined to do it more for some reason. Possibly at that scale it was a lot of effort beyond the conversation.

Things I see around me sometimes get involved in my thoughts in a way that seems related. For ... (read more)

Realistic thought experiments

No, never heard of it, that I know of.

Berkeley: being other people

I'm pretty unsure how much variation in experience there is—'not much' seems plausible to me, but why do you find it so probable?

Moloch in whom I sit alone

I also thought that at first, and wanted to focus on why people join groups that are already large. But yeah, lack of very small groups to join would entirely explain that. Leaving a group signaling not liking the conversation seems like a big factor from my perspective, but I'd guess I'm unusually bothered by that.

Another random friction:

  • If you just sit alone, you don't get to choose the second person who joins you. I think a thing people often do rather than sitting alone is wander alone, and grab someone else also wandering, or have plausible deniability that they might be actually walking somewhere, if they want to avoid being grabbed. This means both parties get some choice.
Moloch in whom I sit alone

Aw, thanks. However I claim that this was a party with very high interesting people density, and that the most obvious difference between me and others was that I ever sat alone.

Epistemic Spot Check: The Dorito Effect (Mark Schatzker)

I share something like this experience (food desirability varies a lot based on unknown factors and something is desirable for maybe a week and then not desirable for months) but haven't checked carefully that it is about nutrient levels in particular. If you have, I'd be curious to hear more about how.

(My main alternative hypothesis regarding my own experience is that it is basically imaginary, so you might just have a better sense than me of which things are imaginary..)

Epistemic Spot Check: The Dorito Effect (Mark Schatzker)

A page number or something for the 'more seasoned' link might be useful. The document is very long and doesn't appear to contain 'season-'.

The 'blander' link doesn't look like it supports the claim much, though I am only looking at the abstract. It says that 'in many instances' there have been reductions in crop flavor, but even this appears to be background that the author is assuming, rather than a claim that the paper is about. If the rest of the paper does contain more evidence on this, could you quote it or something, since the paper is expensive to see?

Re: seasoning. Page 19: "Miscellaneous foods including spices generally increased from 10 pounds per capita in 1909 to 13 pounds per capita in 2000. Spices were not added to the food supply until 1918. The use of spices increased more than fivefold from one-half pound per capita in 1918 to 2.59 pounds per capita in 2000 (data not shown)." Have contacted you out of band with a copy of the paper, which does indeed go into more detail than the abstract.
Reframing misaligned AGI's: well-intentioned non-neurotypical assistants
I am somewhat hesitant to share simple intuition pumps about important topics, in case those intuition pumps are misleading.

This sounds wrong to me. Do you expect considering such things freely to be misleading on net? I expect some intuition pumps to be misleading, but for considering all of the intuitions that we can find about a situation to be better than avoiding them.

6Ben Pace4y
I feel like there are often big simplifications of complex ideas that just convey the wrong thing, and I was vaguely worried that in a field primarily dominated by things that are hard-to-read, things that are easy to understand will dominate the conversation even if they're pretty misguided. It's not a big worry for me here, but it was the biggest hesitation I had.
Not sure what Ben meant, but my own take is "sharing is fine, but intuition pumps without rigor backing them are not something we should curate regularly as an exemplar of what LW is trying to be"
Will AI See Sudden Progress?

Thanks for your thoughts!

I don't quite follow you on the intelligence explosion issue. For instance, why does a strong argument against the intelligence explosion hypothesis need to show that a feedback loop is unlikely? Couldn't we believe that it is likely, but not likely to be very rapid for a while? For instance, there is probably a feedback loop in intelligence already, where humans with better thoughts and equipment are effectively smarter, and can then devise better thoughts and equipment. But this has been true for a while, and is a fairly slow process (at least for now, relative to our ability to deal with things).

1Charlie Steiner4y
Yeah, upon rereading that response, I think I created a few non sequiturs in revision. I'm not even 100% sure what I meant by some bits. I think the arguments that now seem confusing were me was saying that by putting an intelligence feedback loop in the reference class of "feedback loops in general" and then using that to forecast low impact, the thing that is doing most of the work is simply how low impact most stuff is. A nuclear bomb (or a raindrop forming, or tipping back a little too far in your chair) can be modeled as a feedback loop through several orders of magnitude of power output, and then eventually that model breaks down and the explosion dissipates, and the world might be a little scarred and radioactive, but it is overall not much different. But if your AI increased by several orders of magnitude in intelligence (let's just pretend that's meaningful for a second), I would expect that to be a much bigger deal, just because the thing that's increasing is different. That is, I was thinking that the implicit model used by the reference class argument from the original link seems to predict local advantages in AI, but predict *against* those local advantages being important to the world at large, which I think is putting the most weight on the weakest link. Part of this picture I had comes from what I'm imagining as prototypical reference class members - note that I only imagined self-sustaining feedback, not "subcritical" feedback. In retrospect, this seems to be begging the question somewhat - subcritical feedback speeds up progress, but doesn't necessarily concentrate it, unless there is some specific threshold effect for getting that feedback. Another feature of my prototypes was that they're out-of-equilibrium rather than in-equilibrium (an example of feedback in equilibrium is global warming, where there's lots of feedback effects but they're more or less canceling each other out), but this seems justified. I would agree that one can imagine som
Making yourself small

My example for high status/small was an esteemed teacher unexpectedly dropping in to see to see their student perform, and entering silently and at the last minute, then standing quietly at the back of the room by the door.

Person-moment affecting views

I also think they are probably wrong, but this kind of argument is a substantial part of why. So I want to see if they can be rescued from it, since that would affect their probability of being right from my perspective.

Do you think there are more compelling arguments that they are wrong, such that we need not consider ones like this? (Also just curious)

I think this is quite devastating analysis for them, even if you would take a "person" to be well defined object. The Person-Affecting Restriction, Comparativism, and the Moral Status of Potential People, by Gustaf Arrhenius [] for example see fig. 2 and related argument. Basically, you have 3 worlds (A;B;C), with populations ([x,y];[y,z];[z,x]). You set the welfare of the populations such that So you would have to sacrifice transitivity to "rescue" PAW. Another argument may be from physics - according to many-worlds interpretation of QM, there exists a world where I was not born, because some high energy particle damaged a part of DNA necessary for my birth. Hence, for each person there exists a world where he does not exists. Taken ad absurdum, nobody has moral value.
I'm not sure if "wrong", "incoherent", or just "incomplete", but this is one major hole in strict person-affecting views. When comparing two future universes, are you disallowed from having a preference if NEITHER of them contains any entity (or consciousness-path or whatever you say is "person" across time) from the current universe? 200 years from now has ZERO person-overlap with now. Does that mean nothing matters?
Load More