habryka

Coding day in and out on LessWrong 2.0. You can reach me at habryka@lesswrong.com

Sequences

Concepts in formal epistemology

Wiki Contributions

Comments

In the 7 months since I made that post, it's had < 5% of the comments engagement that this post has gotten in a day.

Popular and off-the-cuff presentations often get discussed because it is fun to talk about how the off-the-cuff presentation has various flaws. Most comments get generated by demon threads and scissor statements, sadly. We've done some things to combat that, and definitely not all threads with lots of comments are the result of people being slightly triggered and misunderstanding each other, but a quite substantial fraction are. 

Are there current, present day things going on in the EA and rationality community which you think warrant suspecting them of being incredibly net negative (causing worse worlds, conditioned on the current setup)? Things done in the last 6 months ?

I mean yes! Don't I mention a lot of them in the post above? 

I mean FTX happened in the last 6 months! That caused incredibly large harm for the world. 

OpenAI and Anthropic are two of the most central players in an extremely bad AI arms race that is causing enormous harm. I really feel like it doesn't take a lot of imagination to think about how our extensive involvement in those organizations could be bad for the world. And a huge component of the Lightcone Offices was causing people to work at those organizations, as well as support them in various other ways.

EAs are more about saying they care about numbers than actually caring about numbers, and didn't calculate downside risk enough in the past. The past events reveal this attitude and because it's not expected to have changed, we can expect it to still be affecting current EAs, who will continue causing great harm because of not actually caring for downside risk enough.

No, this does not characterize my opinion very well. I don't think "worrying about downside risk" is a good pointer to what I think will help, and I wouldn't characterize the problem that people have spent too little effort or too little time on worrying about downside risk. I think people do care about downside risk, I just also think there are consistent and predictable biases that cause people to be unable to stop, or be unable to properly notice certain types of downside risk, though that statement feels in my mind kind of vacuous and like it just doesn't capture the vast majority of the interesting detail of my model. 

I think this is an interesting idea!

In doing community building I have definitely used publicly available information about IQ distributions in a lot of different ways, and I do think things like SAT scores and top-university admissions and other things that are substantial proxies for IQ are things that I definitely use in both deciding who to respect, listen to and invest in. 

That said, while IQ tests do have some validity outside the 135+ regime, I do think it's a good bit weaker, and I put substantially less weight on the results of IQ tests being able to differentiate among the tails. I think it's definitely still an input, and I do think I kind of wish there were better ways to assess people's intelligence in various different contexts, but it doesn't feel like a hugely burning need for me, and most of my uncertainty about people usually isn't concentrated in being able to tell how smart someone is, but is more concentrated in how reliable they are, how ethical they are and how likely they are to get hijacked by weird societal forces and happy-death spirals and similar things, for which making standardized tests seem at least a good amount harder (especially ones that can't be gamed). 

Another huge problem with assigning status according to IQ test results is that IQ tests are not at all robust against training effects. If you want an official high IQ score, just spend a few dozen hours doing IQ tests, practicing the type of challenges you will be tested on. The SAT in-contrast is kind of useful anyways, because kind of everyone saturates on practicing for the SAT, at least in the US, but that test sadly also maxes out at an average that is below the average IQ test result in the LessWrong community, and so isn't super helpful in distinguishing between the remaining variance. If there was a well-tested IQ-test that was made to be training-resistant, that could potentially provide a bunch of value, but I don't know of one.

I kind of like this idea. I've been thinking for a long time about how to make LessWrong more of a place of active learning. My guess is there is still a lot of hand-curating to be done to create the right type of active engagement, but I do think the barrier to entry is a bunch lower now and it might be worth the time-investment.

I mean, I don't see the argument for more than that. Unless you have some argument for hardware progress stopping, my sense is that things would get cheap enough that someone is going to try the AI stuff that is happening today within a decade.

I missed the first chunk of your conversation with Dylan at the lurkshop about this, but at the time, it sounded like you suspected "quite large" wasn't 6-48 months, but maybe more than a decade.

A decade in-expectation seems quite extreme. 

To be clear, I don't think AGI happening soon is particularly overdetermined, so I do think this is a thing that does actually differ quite a bit depending on details, but I do think it's very unlikely that actions that people adjacent to rationality took that seriously sped up timelines by more than a decade. I would currently give that maybe 3% probability or something.

That seems right to me, but I interpreted the above for advice for one office, potentially a somewhat smaller one. Seems fine to me to have one hub for people who think more through the lens of agency.

I think I might change some things but it seems very unlikely to me I will substantially reduce investment in LessWrong. Funding is scarcer post-FTX, so some things might change a bit, but I do care a lot about LessWrong continuing to get supported, and I also think it's pretty plausible I will substantially ramp up my investment into LW again.

Yeah, in-particular between 11 and 13 things are quite non-logarithmic. I probably had some kind of reasoning for that when I wrote it, but I sure don't remember it.

I considered for a while to add a medium-like voting-system where you can just click n-times to give n votes until your maximum vote strength, but the UI ended up too complicated. Might be worth revisiting sometime again though.

Load More