All of Taran's Comments + Replies

Also, the specific cycle attack doesn’t work against other engines I think? In the paper their adversary doesn’t transfer very well to LeelaZero, for example. So it’s more one particular AI having issues, than a fact about Go itself.

Sure, but refutations don't transfer to different openings either, right?  I feel like most game-winning insights are contingent in this sense, rather than being fundamental to the game.

EDIT: also, I think if you got arbitrary I/O access to a Magnus simulator, and then queried it millions of times in the course of doing Al

... (read more)

Like I said, I feel like I hear it a lot, and in practice I don't think it's confusing because the games that get solved by theorists and the games that get "solved" by AIs are in such vastly different complexity regimes.  Like, if you heard that Arimaa had been solved, you'd immediately know which sense was meant, right?

Having said that, the voters clearly disagree and I'm not that attached to it, so I'm going to rename the post.  Can you think of a single adjective or short phrase that captures the quality that chess has, and Starcraft doesn't,... (read more)

I see where you're coming from, but I don't think the exploit search they did here is fundamentally different from other kinds of computer preparation. If I were going to play chess against Magnus Carlsen I'd definitely study his games with a computer, and if that computer found a stunning refutation to an opening he liked I'd definitely play it. Should we say, then, that the computer beat Carlsen, and not me? Or leave the computer aside: if I were prepping with a friend, and my friend found the winning line, should we say my friend beat Carlsen? What if ... (read more)

Conditionally on him continuing to play the opening, I would expect he has a refutation to that refutation, but no reason to use the counter-refutation in public games against the computer. On the other hand, he may not want to burn it on you either.
That’s fair, but I still think saying “Go has been unsolved” is importantly misleading: for one, it hasn’t been solved! Also, the specific cycle attack doesn’t work against other engines I think? In the paper their adversary doesn’t transfer very well to LeelaZero, for example. So it’s more one particular AI having issues, than a fact about Go itself. (EDIT: As Tony clarifies below, the cyclic attack seems to be a common failure mode amongst other top Go AIs as well, so I retract this paragraph -- at the very least, it seems to be a fact about ~all the top Go AIs!) EDIT: also, I think if you got arbitrary I/O access to a Magnus simulator, and then queried it millions of times in the course of doing AlphaZero style training to derive an adversarial example, I’d say it’s pretty borderline if it’s you beating beating him. Clearly there’s some level of engine skill where it’s no longer you playing!

In a game context you're right, of course.  But I often hear AI people casually say things like "chess is solved", meaning something like "we solved the problem of getting AIs to be superhumanly good at chess" (example).  For now I think we have to stop saying that about go, and instead talk about it more like how we talk about Starcraft.

To which one should reply: 'oh really, is it a draw or a win for white?'

Man, that’s such a terrible way to say it, given “solved game” has a pre-existing technical meaning.

But in this case it’s probably because Eliezer wanted to get it under 280 characters, lol

I don't like it. "The problem of creating AI that is superhuman at chess" isn't encapsulated in the word "chess", so you shouldn't say you "solved chess" if what you mean is that you created an AI that is superhuman at chess. What it means for a game to be solved is widely-known and well-developed[0]. Using the exact same word, in extremely similar context, to mean something else seems unnecessarily confusing.  [0] See

Well, I'm hardly an expert, I've just read all the posts.  Marcello summed up my thinking pretty well.  I don't think I understand how you see it yet, though.  Is is that the adversary's exploit is evidence of a natural abstraction in Go that both AIs were more-or-less able to find, because it's expressible in the language of live groups and capturing races?

You can imagine the alternative, where the "exploit" is just the adversary making moves that seem reasonable but not optimal, but then KataGo doesn't respond well, and eventually the adversary wins without there ever being anything a human could point to and identify as a coherent strategy.

In the paper, David Wu hypothesized one other ingredient: the stones involved have to form a circle rather than a tree (that is, excluding races that involve the edge of the board).  I don't think I buy his proposed mechanism but it does seem to be true that the bait group has to be floating in order for the exploit to work.

Interesting point about the scaling hypothesis.  My initial take was that this was a slightly bad sign for natural abstractions: Go has a small set of fundamental abstractions, and this attack sure makes it look like KataGo didn't quite learn some of them (liberties and capturing races), even though it was trained on however many million games of self-play and has some customizations designed to make those specific things easier.  Then again, we care about Go exactly because it resisted traditional AI for so long, so maybe those abstractions aren... (read more)

The comments in that post are wrong, the exploit does not rely on a technicality or specific rule variant.  I explained how to do it in my post, cross-posted just now with Vanessa's post here.

Any time you get a data point about X, you get to update both on X and on the process that generated the data point.  If you get several data points in a row, then as your view of the data-generating process changes you have re-evaluate all of the data it gave it you earlier.  Examples:

  • If somebody gives me a strong-sounding argument for X and several weak-sounding arguments for X, I'm usually less persuaded than if I just heard a strong-sounding argument for X.  The weak-sounding arguments are evidence that the person I'm talking to can't ev
... (read more)
Answer by TaranFeb 03, 202392

Strictly speaking asymptotic analysis is not very demanding: if you have a function  that you can bound above in the limit as a function of , you can do asymptotic analysis to it.  In practice I mostly see asymptotic analysis used to evaluate counterfactuals: you have some function or process that's well-behaved for  inputs, and you want to know if it will still be well-enough behaved if you had  inputs instead, without actually doing the experiment.  You're rendering ten characters on the screen in your... (read more)

Additionally, though this is small/circumstantial, I'm pretty sure your comment came up much faster than even a five-minute timer's worth of thought would have allowed, meaning that you spent less time trying to see the thing than it would have taken me to write out a comment that would have a good chance of making it clear to a five-year-old.

Another possibility is that he did some of his thinking before he read the post he was replying to, right?  On my priors that's even likely; I think that when people post disagreement on LW it's mostly after thin... (read more)

1[DEACTIVATED] Duncan Sabien1y
I agree that it wouldn't be valid as an absolute, or even as a strong claim. I'm not sure I agree that it is no evidence at all.

Yeah, I agree with all of this; see my own review.  My guess is that Alex_Altair is making the exact mistake you tried to warn against.  But, if I'm wrong, the examples would have been clarifying.

When you reach for this term, take a second to consider more specifically what you mean, and considering saying that more specific thing instead.

What considerations might lead you to not say the more specific thing?  Can you give a few examples of cases where it's better to say "outside view" than to say something more specific?  

3Daniel Kokotajlo1y
If "outside view" was a natural category that was useful to use, AND people generally had a shared understanding of what it meant, then it would slow things down unnecessarily to be more specific, at least often (sometimes even then you'd want to be more specific.) My original post cast doubt on both the naturalness/usefulness of the concept (not saying there's absolutely nothing tying the things in the Big Lists together, just saying that there isn't really any good evidence that whatever it is that ties them together is epistemically important) and the shared understanding (ho boy do different people seem to have different ideas of what it means and how it should be used and what evidential status it confers)

The amount of research and development coming from twitter in the 5 years before the acquisition was already pretty much negligible

That isn't true, but I'm making a point that's broader than just Twitter, here.  If you're a multi-billion dollar company, and you're paying a team 5 million a year to create 10 million a year in value, then you shouldn't fire them.  Then again, if you do fire them, probably no one outside your company will be able to tell that you made a mistake: you're only out 5 million dollars on net, and you have billions more wh... (read more)

If you fire your sales staff your company will chug along just fine, but won't take in new clients and will eventually decline through attrition of existing accounts.

If you fire your product developers your company will chug along just fine, but you won't be able to react to customer requests or competitors.

If you fire your legal department your company will chug along just fine, but you'll do illegal things and lose money in lawsuits.

If your fire your researchers your company will chug along just fine, but you won't be able to exploit any more research pr... (read more)

2clone of saturn1y
These statements seem awfully close to being unfalsifiable. The amount of research and development coming from twitter in the 5 years before the acquisition was already pretty much negligible, so there's no difference there. How long do we need to wait for lawsuits or loss of clients to cause observable consequences?
  1. This post is worthwhile and correct, with clear downstream impact.  It might be the only non-AI post of 2021 that I've heard cited in in-person conversation -- and the cite immediately improved the discussion.
  2. It's clearly written and laid out; unless you're already an excellent technical writer, you can probably learn something by ignoring its content and studying its structure.

Sure, I just don't expect that it did impact peoples' models very much*.  If I'm wrong, I hope this review or the other one will pull those people out of the woodwork to explain what they learned.

*Except about Leverage, maybe, but even there...did LW-as-a-community ever come to any kind of consensus on the Leverage questions?  If Geoff comes to me and asks for money to support a research project he's in charge of, is there a standard LW answer about whether or not I should give it to him?  My sense is that the discussion fizzled out unresolved, at least on LW.

A non-obvious thing here: last year, Lightcone put a fair amount of effort into doing lots of interviews, orienting on the Leverage situation, and attempting to publish a blogpost that offered a pretty clear and comprehensive set of information. We were specifically thinking of this from the standpoint of "it doesn't seem like there are very good community justice institutions or practices around, beyond random discussions", and thinking maybe we could contribute something useful. And then, well, a lot of stuff came up and we didn't get the piece over the finish-line of publishing.  So I'm coming at this from the perspective, partly "how valuable would it have been to get that across the finish line?". And I see both this piece and the Zoe piece as representing the collective situation.  I also do just agree some of the claims in this piece (implicit and explicit) that many of the cult-looking-behaviors of leverage are red herrings and are reasonable things I want to defend.

I liked this post, but I don't think it belongs in the review.  It's very long, it needs Zoe's also-very-long post for context, and almost everything you'll learn is about Leverage specifically, with few generalizable insights.  There are some exceptions ("What to do when society is wrong about something?" would work as a standalone post, for example), but they're mostly just interesting questions without any work toward a solution.  I think the relatively weak engagement that it got, relative to its length and quality, reflects that: Less W... (read more)

My take was "It'd be quite weird for this post to show up in the Best of LessWrong books, but I think part of the point of the review is to reflect on things that had some kind of impact on your worldmodels, even if the posts aren't directly optimized for that." 

AIUI it was a feature of early Tumblr culture, which lingered to various degrees in various subcommunities as the site grew more popular.  The porn ban in late 2018 also seemed to open things up a lot, even for people who weren't posting porn; I don't know why.

The way I understood the norm on Tumblr, signal-boosting within Tumblr was usually fine (unless the post specifically said "do not reblog" on it or something like that), but signal-boosting to other non-Tumblr communities was bad.  The idea was that Tumblr users had a shared vibe/culture/stigma that wasn't shared by the wider world, so it was important to keep things in the sin pit where normal people wouldn't encounter them and react badly.

Skimming the home invasion post it seems like the author feels similarly: Mastodon has a particular culture, created by the kind of people who'd seek it out, and they don't want to have to interact with people who haven't acclimated to that culture.

Interesting; that isn't something I knew about Tumblr. This is especially surprising given how often I see screenshots of Tumblr discussions shared on FB, like the post I responded to here. (I really don't like share-by-screenshot culture)

I'm a little curious what reference class you think the battle of Mariupol does belong to, which makes its destruction by its defenders plausible on priors.  But mostly it sounds like you agree that we can make inferences about hard questions even without a trustworthy authority to appeal to, and that's the point I was really interested in.

The reference class would be "wars to extend a country's territory permanently". As such there's an interest to have the value of the newly won territory as high as possible.  When waging war over a city, for both sides there are actions that can be taken to increase or decrease the amount of damage that the city takes.  In Ukraine, it seems that no party went out of its way to reduce the damage to cities. We know that from Amnesty trying to understand what happened and them finding that frequently Ukrainian army stationed itself inside the city and got shot at by the Russian army. This dynamic does explain that a part of the city is destroyed but it doesn't explain why 90% of Mariupol's residential buildings had been damaged or destroyed (Wikipedia numbers). The 90% sounds to me like this is more than just collateral damage but that someone made a conscious choice to destroy more of the city than they would need for purely military reasons.  One reason to do that might be propaganda reasons and to make the population fear you. Given that according to Russian propaganda Russia came to liberate the Russian minority, destroying a city with a large number of ethnic Russians makes little sense for that goal.  Often in war destruction is also done as a punishment. Russia punished the population in Chechnya for their local resistance in the Second Chechnyan War. It's unclear to me why the population of Mariupol would deserve to be punished from the Russian perspective.  On the other hand, at the time that Mariupol was taking Ukrainians might have thought that this war will end in a way where Donetsk and Luhansk would permanently be part of Russia. Under that assumption making Mariupol worth as little as possible seems to me like an understandable reason.  Reporting suggests that taking Crimea was expensive for Russia. Having Mariupol destroyed means Russia would have to invest more money into it to make it function again after the war.   

Usually that's just about denying strategic assets, though: blowing up railroads, collapsing mine shafts, that sort of thing.  Blowing up the museums and opera houses is pointless, because the enemy can't get any war benefit by capturing them.  All it does is waste your own explosives, which you'd rather use to blow up the enemy.  Scorched earth practiced by attackers, on the other hand, tends to be more indiscriminate: contrast the state of Novgorod post-WW2 with that of the towns west of it, or the treatment of rice fields by North Vietnamese vs. Americans during the Vietnam war.

But we have only very weak evidence of what goes on in the war zone unless both sides agree on some aspect.

I know we're in a hostile information space, but this takes epistemic learned helplessness way too far.  There are lots of ways to find things out other than being told about them, and when you don't have specific knowledge about something you don't have to adopt a uniform prior.

Taking Mariupol as an example, our two suspects are the Russians, who were attacking Mariupol and didn't have any assets there, and the Ukrainians, who were defending Mar... (read more)

The Wikipedia number for Mariupol's ethnic Russian population is 44%. Russia certainly had the intention to make Mariupol Russian territory. Making Mariupol Russian territory is worth more if it stays standing. Russia has to invest less money into rebuilding Mariupol if it's not destroyed.  The inability of the Russian army was not as apparent at the time Mariupol was taken as it's now, so it's quite plausible that Ukrainians didn't expect to be able to retake it at the time.  World War II was not a far fought to take over German territory, so it's not in the same reference class. That's especially true because of lessons from World War I, that the German population might have to see part of Germany being destroyed to really understand that they lost. 
3Teerth Aloke1y
Ever heard of scorched earth?

> Control-f "cold war"

> No results found

Asimov and the Apollo engineers grew up benefiting from progress; their children grew up doing duck-and-cover exercises, hiding from it under their desks.  Of course they relate to it differently!

This theory predicts that people who grew up after the cold war ended should be more prone to celebrate progress.  I think that's true: if you go to silicon valley, where the young inventors are, messianic excitement over the power of progress is easy to find.  Isaac Asimov wanted to put an RTG in your refrigerator, and Vitalik Buterin wants to put your mortgage on the blockchain; to me they have very similar energies.

There was lots of amyloid research in the Alzheimer's space before the fake 2006 paper, and in the hypothetical where it got caught right away we would probably still see a bunch of R&D built around beta-amyloid oligomers, including aducanumab.  You can tell because nobody was able to reproduce the work on the *56 oligomer, and they kept on working on other beta-amyloid oligomer ideas anyway.  It's bad, but "16 years of Alzheimer's research is based on fraud" is a wild overstatement.  See Derek Lowe's more detailed backgrounder for more ... (read more)

Dealing with human subjects, the standard is usually "informed consent": your subjects need to know what you plan to do to them, and freely agree to it, before you can experiment on them.  But I don't see how to apply that framework here, because it's so easy to elicit a "yes" from a language model even without explicitly leading wording.  Lemoine seems to attribute that to LaMDA's "hive mind" nature: best as I can tell, LaMDA is a sort of hive mind which is the aggregation of all of the different chatbots it is capable of creating. Some of

... (read more)

When I first read this I intuitively felt like this was a useful pattern (it reminds me of one of the useful bits of Illuminatus!), but I haven't been able to construct any hypotheticals where I'd use it.

I don't think it's a compelling account of your three scenarios. The response in scenario 1 avoids giving Alec any orders, but it also avoids demonstrating the community's value to him in solving the problem.  To a goal-driven Alec who's looking for resources rather superiors, it's still disappointing: "we don't have any agreed-upon research direction... (read more)

This seems right to me about most go clubs, but there’re a lot of other places that seem to me different on this axis. Distinguishing features of Go clubs from my POV: 1. A rapid and trustworthy feedback loop, where everyone wins and loses at non-rigged games of Go regularly.  (Opposite of schools proliferating without evidence.) 2. A lack of need to coordinate individuals.  (People win or lose Go games on their own, rather than by needing to organize other people into coordinating their play.) Some places where I expect “being in sync with the narrative” would diverge more from “just figuring out how to get stronger / how to do the object-level task in a general way”: 1. A hypothetical Go club that somehow twisted around to boost a famous player’s ego about how very useful his particular life-and-death problems were, or something, maybe so they could keep him around and brag about how they had him at their club, and so individual members could stay on his good side.  (Doesn’t seem very likely, but it’s a thought experiment.) 2. Many groups with an “ideological” slant, e.g. the Sierra Club or ACLU or a particular church  3. (?Maybe? not sure about this one) Many groups that are trying to coordinate their members to follow a particular person’s vision for coordinated action, e.g. Ikea's or most other big companies' interactions with their staff, or even a ~8-employee coffee shop that's trying to realize a particular person's vision
I’m curious what you think of the examples in the long comment I just made (which was partly in response to this, but which I wrote as its own thing because I also wish I’d added it to the post in general). I’m now thinking there’re really four concepts: 1. Narrative syncing.  (Example: “the sand is lava.”)   2. Narrative syncing that can easily be misunderstood as information sharing.  (Example: many of Fauci’s statements about covid, if this article about it is correct.)   3. Narrative syncing that sets up social pressure not to disagree, or not to weaken the apparent social norm about how we’ll talk about that.  (Example: “Gambi’s is a great restaurant and we are all agreed on going there,” when said in an irate tone of voice after a long and painful discussion about which restaurant to go to.”)   4. Narrative syncing that falls into categories #2 and #3 simultaneously.   (Example: “The 911 terrorists were cowards,” if used to establish a norm for how we’re going to speak around here rather than to share honest impressions and invite inquiry.) I am currently thinking that category #4 is my real nemesis — the actual thing I want to describe, and that I think is pretty common and leads to meaningfully worse epistemics than an alternate world where we skillfully get the good stuff without the social pressures against inquiry/speech. I also have a prediction that most (though not all) instances of #2 will also be instances of #3, which is part of why I think there's a "natural cluster worth forming a concept around" here.

Yeah, we're all really worked up right now but this was an utterly wild failure of judgment by the maintainer. Nothing debatable, no silver lining, just a miss on every possible level.

I don't know how to fix it at the package manager level though? You can force everyone to pin minor versions of everything for builds but then legitimate security updates go out a lot slower (and you have to allow wildcards in package dependencies or you'll get a bunch of spurious build failures). "actor earns trust through good actions and then defects" is going to be hard to handle in any distributed-trust scheme.

I'm not going to put this as an answer because you said you didn't want to hear it, but I don't think you're in any danger.  The problem is not very serious now, and has been more serious in the past without coming to anything.

To get a sense of where I'm coming from I'd encourage you to read up on the history of communist movements in the United States, especially in the 1920s (sometimes called the First Red Scare, and IMO the closest the US has ever come to communist overthrow).  The history of anarchism in the US is closely related, at least in... (read more)

It's definitely a fair suggestion that I should inform myself better on the actual history of communist ideology in the United States. If I'm scared by what I read, then I will have a better idea of what the problem is. If what I find out doesn't scare me, I'll be relieved, and leave with a better understanding of astronomy.

In this sort of situation I think it's important to sharply distinguish argument from evidence.  If you can think of a clever argument that would change your mind then you might as well update right away, but if you can think of evidence that would change your mind then you should only update insofar as you expect to see that evidence later, and definitely less than you would if someone actually showed it to you.  Eliezer is not precise about this in the linked thread: Engines of Creation contains lots of material other than clever arguments!

A re... (read more)

Maybe it's better to start with something we do understand, then, to make the contrast clear.  Can we study the "real" agency of a thermometer, and if we can, what would that research program look like?

My sense is that you can study the real agency of a thermometer, but that it's not helpful for understanding amoebas.  That is, there isn't much to study in "abstract" agency, independent of the substrate it's implemented on.  For the same reason I wouldn't study amoebas to understand humans; they're constructed too differently.

But it's possible that I don't understand what you're trying to do.

Yeah, that's the question, is agency substrate-independent or not, and if it is, does it help to pick a specific substrate, or would one make more progress by doing it more abstractly, or maybe both? 

Nah, we're on the same page about the conclusion; my point was more about how we should expect Yudkowsky's conclusion to generalize into lower-data domains like AI safety.  But now that I look at it that point is somewhat OT for your post, sorry.

My comment had an important typo, sorry: I meant to write that I hadn't noticed this through-line before!

I mostly agree with you re: Einstein, but I do think that removing the overstatement changes the conclusion in an important way.  Narrowing the search space from (say) thousands of candidate theories to just 4 is an great achievement, but you still need a method of choosing among them, not just to fulfill the persuasive social ritual of Science but because otherwise you have a 3 in 4 chance of being wrong.  Even someone who trusts you can't up... (read more)

I think we disagree on Yudkowsky's conclusion: his point IMO is that Einstein was able to reduce the search space a lot. He overemphasize for effect (and because it's more impressive to have someone who guesses right directly through these methods), but that doesn't change that Einstein reduced the state space a lot (which you seem to agree with). Many of the relevant posts I quoted talk about how the mechanism of Science are fundamentally incapable of doing that, because they don't specify any constraint on hypothesis except that they must be falsifiable. Your point seems to be that in the end, Einstein still used the sort of experimental data and methods underlying traditional Science, and I tend to agree. But the mere fact that he was able to get the right answer out of millions of possible formulations by checking a couple of numbers should tell you that there was a massive hypothesis-space reducing step before.

I think you've identified a real through-line in Yudkowsky's work, one I hadn't noticed before.  Thank you for that.

Even so, when you're trying to think about this sort of thing I think it's important to remember that this:

In our world, Einstein didn't even use the perihelion precession of Mercury, except for verification of his answer produced by other means.  Einstein sat down in his armchair, and thought about how he would have designed the universe, to look the way he thought a universe should look—for example, that you shouldn't ought to be

... (read more)
Thanks for the kind and thoughtful comment! That's a really good point. I didn't go into that debate in the post (because I tried to not criticize Yudkowky, and also because the post is already way too long), but my take on this is: Yudkowsky probably overstates the case, but that doesn't mean he's wrong about the relevance for Einstein's work of the constrains and armchair reasoning (even if the armchair reasoning was building on more empirical evidence that Yudkowsky originally pointed out). As you say, Einstein apparently did reduce the search space significantly: he just failed to find exactly what he wanted in the reduced space directly.

Fair enough!  My claim is that you zoomed out too far: the quadrilemma you quoted is neither good nor evil, and it occurs in both healthy threads and unhealthy ones.  

(Which means that, if you want to have a norm about calling out fucky dynamics, you also need a norm in which people can call each others' posts "bullshit" without getting too worked up or disrupting the overall social order.  I've been in communities that worked that way but it seemed to just be a founder effect, I'm not sure how you'd create that norm in a group with a strong existing culture).

It's often useful to have possibly false things pointed out to keep them in mind as hypotheses or even raw material for new hypotheses. When these things are confidently asserted as obviously correct, or given irredeemably faulty justifications, that doesn't diminish their value in this respect, it just creates a separate problem.

A healthy framing for this activity is to explain theories without claiming their truth or relevance. Here, judging what's true acts as a "solution" for the problem, while understanding available theories of what might plausibly b... (read more)

I want to reinforce the norm of pointing out fucky dynamics when they occur...

Calling this subthread part of a fucky dynamic is begging the question a bit, I think.

If I post something that's wrong, I'll get a lot of replies pushing back.  It'll be hard for me to write persuasive responses, since I'll have to work around the holes in my post and won't be able to engage the strongest counterarguments directly.  I'll face the exact quadrilemma you quoted, and if I don't admit my mistake, it'll be unpleasant for me!  But, there's nothing fucky h... (read more)

(My sense is that dxu is not referring to JenniferRM's post, so much as the broader dynamic of how disagreement and engagement unfold, and what incentives that creates.)

I expect that many of the people who are giving out party invites and job interviews are strongly influenced by LW.

The influence can't be too strong, or they'd be influenced by the zeitgeist's willingness to welcome pro-Leverage perspectives, right?  Or maybe you disagree with that characterization of LW-the-site?

5[DEACTIVATED] Duncan Sabien2y
Things get complicated in situations where e.g. 70% of the group is welcoming and 30% of the group is silently judging and will enact their disapproval later.  And the zeitgeist that is willing to welcome pro-Leverage perspectives might not be willing to actively pressure people to not discriminate against pro-Leverage folk.  Like, they might be fine with somebody being gay, but not motivated enough to step in if someone else is being homophobic in a grocery store parking lot, metaphorically speaking. (This may not describe the actual situation here, of course.  But again it's a fear I feel like I can't dismiss or rule out.)

When it comes to the real-life consequences I think we're on the same page: I think it's plausible that they'd face consequences for speaking up and I don't think they're crazy to weigh it in their decision-making (I do note, for example, that none of the people who put their names on their positive Leverage accounts seem to live in California, except for the ones who still work there).  I am not that attached to any of these beliefs since all my data is second- and third-hand, but within those limitations I agree.

But again, the things they're worried... (read more)

I expect that many of the people who are giving out party invites and job interviews are strongly influenced by LW. If that's the case, then we can prevent some of the things Duncan mentions by changing LW in the direction of being more supportive of good epistemics (regardless of which "side" that comes down on), with the hope of flow-through effects.

But it sure is damning that they feel that way, and that I can't exactly tell them that they're wrong.

You could have, though.  You could have shown them the many highly-upvoted personal accounts from former Leverage staff and other Leverage-adjacent people.   You could have pointed out that there aren't any positive personal Leverage accounts, any at all, that were downvoted on net.  0 and 1 are not probabilities, but the evidence here is extremely one-sided: the LW zeitgeist approves of positive personal accounts about Leverage.  It wo... (read more)

1[DEACTIVATED] Duncan Sabien2y
Thanks for gathering these.  They are genuinely helpful (several of them I missed). But yes, as you inferred, the people I've talked to are scared about real-life consequences such as losing funding or having trouble finding employment, which are problems they don't currently have but suspect they will if they speak up. I reiterate that this is a fact about them, as oppose to a fact about reality, but they're not crazy to have some weight on it.

Even if all you have is a bunch of stuff and learned heuristics, you should be able to make testable predictions with them.  Otherwise, how can you tell whether they're any good or not?

Whether the evidence that persuaded you is sharable or not doesn't affect this.  For example, you might have a prior that a new psychotherapy technique won't outperform a control because you've read like 30 different cases where a leading psychiatrist invented a new therapy technique, reported great results, and then couldn't train anyone else to get the same resul... (read more)

Certainly. But you might not be able to make testable predictions for which others will readily agree with your criteria for judgement. In the exchange, Geoff gives some "evidence", and in other places he gives additional "evidence". It's not really convincing to me, but it at least has the type signature of evidence. Eliezer responds: This is eliding that Geoff probably has significant skill in identifying more detail of how beliefs and goals interact, beyond just what someone would know if they heard about cognitive dissonance theory. Like basically I'm saying that if Eliezer sat with Geoff for a few hours through a few sessions of Geoff doing his thing with some third person, Eliezer would see Geoff behave in a way that suggests falsifiable understanding that Eliezer doesn't have. (Again, not saying he should have done that or anything.)

They're suggesting that you should have written "...this is an accurate how-level description of things like..."  It's a minor point but I guess I agree.

A related discussion from 5 years ago:

Republican Rome is the example I know best, sorta fits?

Rome fought a lot of wars, and they were usually pretty extractive: sometimes total wars in which the entire losing side was killed or enslaved, other times wars of conquest in which the losing states were basically left intact but made to give tribute (usually money and/or soldiers for the legions).  They definitely relied on  captured foreigners to work their farms, especially in Sicily where it was hard to escape, and they got so rich from tribute that they eliminated most taxes o... (read more)

The original startup analogy might be a useful intuition pump here.  Most attempts to displace entrenched incumbents fail, even when those incumbents aren't good and ultimately are displaced.  The challengers aren't random in the monkeys-using-keyboard sense, but if you sample the space of challengers you will probably pick a loser.  This is especially true of the challengers who don't have a concrete, specific thesis of what their competitors are doing wrong and how they'll improve on it -- without that, VCs mostly won't even talk to you. &... (read more)

Cummings seems to be making this same argument in the comments: the Pentagon is so unbelievably awful that its replacement doesn't have to be good, you can pick its successor at random and expect to come up with something better.  To believe this requires a lack of imagination, I think, an inability to appreciate how much scope for failure there really is.  But this is not really a question we can settle empirically -- we can only talk in vague terms about most of what the Pentagon does, and the counterfactuals are even less clear -- so I won't a... (read more)

One nice thing about startups is that they mostly fail if they aren't good.  When MySpace stagnated there wasn't one blessed successor, there were 100 different ones that had to fight it out.  The winner is, modulo the usual capitalist alignment failure, a better company than MySpace was.  Most of its competitors weren't.  From society's perspective this filter is great, maybe the best thing about the whole startup ecosystem.

Cummings doesn't seem to know this.  Replacing the Pentagon with a new organization ABC Inc. is not the hard... (read more)

Is that a serious blow to Cummings' thesis, though? If your idea is, say, "we need a ministry of eduction but the current one is irrecoverably broken", you don't need to invent Google or Amazon to replace it. Depending on how bad the status quo is (which is partly an empirical question), any random thing you come up with might be better than what's already there. In which case his use of the term "startup" would be misleading, but the overall thesis would stay relatively intact. That said, I am quite sympathetic to Chesterton's Fence in this argument. In particular, trying to abolish and replace the Pentagon on day 1 of a new administration (as Cummings suggests in his essay) is... optimistic in a world where other nations can hear you say that.
Exactly! Hes mistaken survival of the salient examples for some kind of intrinsic quality.

I think your hierarchy is useful, and it's helped clarify my own thinking.  I've never really liked the Poe/Shannon argument, because it seemed so historically contingent: if Shannon's culture had played go instead of chess, then his completely correct insight about the theoretical tractability of game trees wouldn't have helped us.  In your model, he'd have taken us from 0 to 3 instead of 0 to 4, which wouldn't have been enough to inspire a playable go program (pedantically, you need game trees before you can invent MCTS, but the connection from Shannon to AlphaGo is clearly much looser than from Shannon to Deep Blue).

Panspermia makes the Bayesian prior of aliens visiting us, even given that the universe can't have too much advanced life or we would see evidence of it, not all that low, perhaps 1/1,000.

Is this estimate written down in more detail anywhere, do you know?  Accidental panspermia always seemed really unlikely to me: if you figure the frequency of rock transfer between two bodies goes with the inverse square of the distance between them, then given what we know of rock transfer between Earth and Mars you shouldn't expect much interstellar transfer at all, even a billion years ago when everything was closer together.  But I have not thought about it in depth.

I am unaware if Hanson has written about this.  Panspermia could happen by the first replicators happening in space perhaps on comets and then spreading to planets.  As Hanson has pointed out, if life is extremely rare it is strange that life would originate on earth when there are almost certainly super-earths on which you would think life would be much more likely to develop.  A solution to this paradox is that life did develop on such an Eden and then spread to earth billions of years ago from a star system that is now far away.  Our sun might have been very close to the other star system when life spread, or indeed in the same system at the time.

A "compiler" is anything that translates a program from one representation to another.  Usually this translation is from a high-level language (like Java) to a lower-level language (like JVM bytecode), but you can also have e.g. a Python -> Javascript compiler that takes in Python code and produces Javascript.  A "natural language compiler", then, is one that takes in ordinary English-or-whatever sentences and emits something executable.  I think this is a pretty fair way to talk about Codex: it's the world's first natural language compil... (read more)

1Michaël Trazzi2y
Thanks for natural language stochastic compiler explanation, makes a lot of sense. I broadly get a sense of what you mean by "context window" since people have been mentioning that quite a lot when talking about GPT-3. As for whether it makes sense to write docstrings for trivial things, I think this is only pointing at the Codex demo examples where people write docstrings and get results, but for most of my use cases, and when it gets really interesting, is when it auto-completes 1) while I'm writing 2) when I'm done writing and it guesses the next line 3) when I start a line by "return " or "x = " and wait for his auto-completion. Here, I would have no idea how to formulate it in the docstring, I just generally trust its ability to follow the logic of the code that precedes it (and I find it useful most of the time).
Load More