LESSWRONG
LW

1500
habryka
50780Ω18022805786118
Message
Dialogue
Subscribe

Running Lightcone Infrastructure, which runs LessWrong and Lighthaven.space. You can reach me at habryka@lesswrong.com. 

(I have signed no contracts or agreements whose existence I cannot mention, which I am mentioning here as a canary)

Sequences

Posts

Sorted by New

Wikitag Contributions

Comments

Sorted by
Newest
56Habryka's Shortform Feed
Ω
7y
Ω
439
The Lightcone Principles
A Moderate Update to your Artificial Priors
A Moderate Update to your Organic Priors
Concepts in formal epistemology
Diagonalization: A (slightly) more rigorous model of paranoia
habryka3h20

Oh, "Yomi" is a very interesting suggestion, thank you! I've definitely encountered that term and can imagine it being a great replacement here. I'll read more about it and see how to adapt it to non-game contexts.

Reply
Diagonalization: A (slightly) more rigorous model of paranoia
habryka3h40

#5 Randomness feels least like a single coherent thing out of these 5. I'd break it into:

5a Maximin. Do things that work out best in the worst case scenario. This often involves a mixed strategy where you randomize across multiple possible actions (assuming you have a hidden source of randomness).

5b Erraticness. Thwart their expectations. Don't do the thing that they're expecting you to do, or do something that they wouldn't have expected.

Though #5b Erraticness seems like an actively bad idea if you have been fully diagonalized, since in case you won't actually succeed at thwarting their expectations and your erratic action will instead be just what they wanted you to do. It is instead a strategy for cat-and-mouse games where they can partially model you but you can still hope to outsmart them.

I found this section quite helpful and think splitting that into these two parts is probably the right call (including the caveat that this backfires if your opponent has actually diagonalized you). 

I am working on a post trying to find a set of more common-language abstractions for reasoning about this stuff, where I think the eraticness fits a bit better into. 

Reply
AI safety undervalues founders
habryka16h20

Hmm, I was referring here to "who I would want to hire at Lightcone" (and similarly, who I expect other mentors would be interested in hiring for their orgs) where I do think I would want to hire people who are on board with that organizational mission.

At the field level, I think we probably still have some disagreement about how valuable people caring about the AI X-risk case is, but I feel a lot less strongly about it, and think I could end up pretty excited about a MATS-like program that is more oriented around doing ambitious understanding of the nature of intelligence.

Reply
The Underexplored Prospects of Benevolent Superintelligences - PART 1: THE WISE, THE GOOD, THE POWERFUL
habryka16h20

Doesn't look like a missflag to me? I would still give the same warning. The whole thing seems very LLM-inspired to me and is the kind of thing I would probably just straight reject if I was making the call again. 

Reply
Simon Lermen's Shortform
habryka20h30

However, I think most people were in fact very concerned about AI existential risk.

Is your sense here "a large majority" or "a small majority"? Just curious about the rough data here. Like more like 55% or more like 80%?

Reply
Wei Dai's Shortform
habryka1d3-1

Sure! I was including "setting up a system that bans other people" in my definition here. I am not that familiar with how DSL works, but given that it bans people, and it was set up by Said, felt confident that thereby somehow Said chose to build a system that does ban people.

Though if Said opposes DSL banning people (and he thinks the moderators are making a mistake when doing so) then I would want to be corrected!

Reply
Diagonalization: A (slightly) more rigorous model of paranoia
habryka1d20

Has some things going for it, but probably too vulgar for most occasions, and also IMO implies too much malintent. I like "diagonalization" because it's kind of a thing you can imagine doing. 

I think my current favorite choice is "Leveling" and "starting a leveling war" from poker. Sentences like "man, I feel like this is too much trying to level them" and "this feels like it's starting a leveling war" are decent-ish pointers.

Reply
Wei Dai's Shortform
habryka1d20

I don't know about Scott. Him being personally active on the site was long before my tenure as admin, and I am not even fully sure how moderation or deletion at the time worked.

I don't think Said ever banned anyone, though he also wrote only a very small number of top-level posts, so there wasn't much opportunity. My guess is he wouldn't have even if he had been writing a lot of top-level posts.

Reply
Don't let people buy credit with borrowed funds
habryka1d40

I don't think the central-case valuable PhDs can be bought or sold so I'm not sure what you mean by market value here. If you can clarify, I'll have a better idea whether it's something I'd bet against you on.

I was thinking of the salary premium that having a PhD provides (i.e. how much more people with PhDs make compared to people without PhDs), which of course is measuring a mixture of real signaling value, and simply just measuring correlations in aptitude, but I feel like it would serve as a good enough proxy here at least directionally.

I would bet a fair amount at even odds that Stanford academics won't decline >1 sigma in collective publication impact score like h-index, Stanford funding won't decrease >1 sigma vs Ivy League + MIT + Chicago, Stanford new-PhD aggregate income won't decline >1 sigma vs overall aggregate PhD income, and overall aggregate US PhD income won't decline >1 sigma. I think 1 sigma is a reasonable threshold for signal vs noise.

What's the sigma here? Like, what population are we measuring the variance over? Top 20 universities? All universities? I certainly agree that Stanford won't lose one sigma of status/credibility/etc. as measured in all universities, that would require dropping Stanford completely from the list of top universities. I think losing 1 sigma of standing among top 20 universities, i.e. Stanford moving from something like "top 3" to "top 8" seems plausible to me, though my guess is a bit too intense. 

To be clear, my offered bet was more about you saying that academia at large is "too big to fail". I do think Stanford will experience costs from this, but at that scale I do think noise will drown out almost any signal. 

TBTF institutions usually don't collapse outside strong outside conquest or civilizational collapse, or Maoist Cultural Revolution levels of violence directed at such change, since they specialize in creating loyalty to the institution. So academia losing value would look more like the Mandarin exam losing value, than like Dell Computer losing value.

Hmm, I don't currently believe this, but it's plausible enough that I would want to engage with it in more detail. Do you have arguments for this? I currently expect more of a gradual devaluing of the importance of academic status in society, together with more competition about the relevant signifiers of status creating more noise, resulting in a relationship to academia somewhat more similar (though definitely not all the way there) as pre-WW2 society had to academia (which to my understanding was a much less central role in government and societal decision-making).

Reply
Wei Dai's Shortform
habryka1d12-1

I understand that you don't! But almost everyone else who I do think has those attributes does not have those criteria. Like, Scott Alexander routinely bans people from ACX, even Said bans people from datasecretslox. I am also confident that the only reason why you would not ban people here on LW, is because the moderators are toiling for like 2 hours a day to filter out the people obviously ill-suited for LessWrong.

Reply3
Load More
39Close open loops
9h
0
31Diagonalization: A (slightly) more rigorous model of paranoia
2d
13
107Put numbers on stuff, all the time, otherwise scope insensitivity will eat you
2d
3
80Increasing returns to effort are common
3d
6
92Don't let people buy credit with borrowed funds
4d
24
129Tell people as early as possible it's not going to work out
4d
11
273Paranoia: A Beginner's Guide
2d
55
70Two can keep a secret if one is dead. So please share everything with at least one person.
5d
1
127Do not hand off what you cannot pick up
6d
17
83Question the Requirements
7d
12
Load More
CS 2881r
2 months ago
(+204)
Roko's Basilisk
4 months ago
Roko's Basilisk
4 months ago
AI Psychology
a year ago
(+58/-28)