Recent Discussion

Hi folks,

My supervisor and I co-authored a philosophy paper on the argument that AI represents an existential risk. That paper has just been published in Ratio. We figured LessWrong would be able to catch things in it which we might have missed and, either way, hope it might provoke a conversation. 

We reconstructed what we take to be the argument for how AI becomes an xrisk as follows: 

  1. The "Singularity" Claim: Artificial Superintelligence is possible and would be out of human control.
  2. The Orthogonality Thesis: More or less any less of intelligence is compatible with more or less any final goal. (as per Bostrom's 2014 definition)

From the conjuction of these two presmises, we can conclude that ASI is possible, it might have a goal, instrumental or final, which is at...

Here we get to a crucial issue, thanks! If we do assume that reflection on goals does occur, do we assume that the results have any resemblance with human reflection on morality? Perhaps there is an assumption about the nature of morality or moral reasoning in the 'standard argument' that we have not discussed?

1VCM3mWe do not say that there is no XRisk or no XRisk from AI.
1VCM4m... well, one might say we assume that if there is 'reflection on goals' that the results are not random.
1VCM6mapologies, I don't recognise the paper here :)

Personally, I can confirm that every yeast protein I work with that does not have a structure when fed through alphafold, produces absolute garbage with mean predicted errors on the order of ten or twenty angstroms and obvious nonsense in the structure.  

Granted I work with a lot of repetitive poorly structured proteins and someone has to get unlucky... but still.

Like you, I thought I saw the light at the end of the tunnel. It first appeared when vaccines started rolling out a few months back, and got brighter when my family got their vaccine cards signed. It got still brighter as I signed up for my first dose (#ModernaManiacs), got the first shot, waited four weeks, and got my second.

After about a week, a few days after the side effects faded, I stopped holding my breath—literally and figuratively—as people walked past. At two weeks post-second dose, I was “officially” “fully vaccinated.” The light had been reached.

My gym had just started allowing those vaccinated to unmask, and I took them up on the offer. It was science to the rescue, human ingenuity encapsulated in some messenger ribonucleic...

Came here to say exactly this, glad someone beat me to it.


Also, I can't quite tell if the OP is recommending wearing masks or mandating wearing masks

2Florin9hIf we should wear anything, it should be elastomeric respirators with P100 filters or DIY PAPRs [] (along with eye protection), not masks. While masks may have worked to control the transmission of the earlier, less contagious strains, they failed to prevent the massive winter wave caused by more contagious variants, and since Delta is even more contagious, mask are probably close to useless now.
2jimv6hAre you thinking about masks as PPE (personal protective equipment, which protects the wearer) or as source control (which protects those around the wearer)? My understanding was that most mask wearing is much more effective as source control, but that if you wear something that's well-fitting (so all the air you're inhaling is drawn through the mask rather than around the gaps at the side) and it's rated, say, N95 then it increasingly provides personal protection too. Of course, with source control the challenge is that you can't unilaterally get most of the benefits from it. It's about whether all the other people around you stay masked.
3waveman9h"Here’s what that looks like in the context of exponential growth:" True but actually it is worse than this. As places like Australia are finding, it is not just a matter of a different growth rate. Measures that stopped the pandemic in its tracks before fail completely in the face of delta. I would also point out that this is looking a bit like the Spanish Flu (which apparently actually started in the US midwest). Later variants were more infectious and attacked younger people more severely. I can attest from personal experience that you do not want to get long covid. There is a view that either you die or you are fine - nothing could be further from the truth. This is a long way from being over.


One of the greatest sources of joy in my life are my close friends. People who bring excitement and novelty into my life. Who expose me to new experiences, and ways of seeing the world. Who help me learn, point out my blind spots, and correct me when I am wrong. Who I can lean on when I need support, and who lean on me in turn. Friends who help me grow more into the kind of person I want to be.

I am especially grateful for this, because up until about 4 years ago, I didn’t have any close friends in my life. I had friends, but struggled to form real emotional connections. Moreover, it didn’t even occur to me that I could try to do this....

It takes about 200 hours of investment in the space of a few months to move a stranger into being a good friend.

My guess is that this number varies a lot between people? I can think of multiple friendships that have felt exciting and close within maybe 3-4 one-on-one encounters of 2-4 hours each.

This is a cross-post from my personal blog.

What is this post about?

While there are tons of guides out there that promise to increase your productivity 10x overnight most of them seem full of empty promises and don't even define what it is they actually mean by productivity. Due to this absence of a clear concept, many of these tips and tricks optimize a rather local and short-termist perspective of productivity that might be intuitive but is in my opinion not optimizing your true long-term goal.

In this post, I want to do three things: a) Present a definition of productivity to make it easier to evaluate what makes sense and what doesn't, b) provide a lot of framing, i.e. ways to think about productivity, and c) give a...

1deepthoughtlife14hOne addenda I would make: breaks also break your focus and it takes time to get back into it, so if you can make tasks that fit neatly in-between your breaks, then you only need to get back up to speed the same way you would have switching to the new task anyway.
3pjeby16hOverall, I found this difficult to read for layout reasons, mostly. The paragraphs and sections are huge, combining multiple ideas and making it hard to identify key points. Your definition of productivity, for example, appears in only one sentence, buried in a short paragraph in the middle of a large section that at first glance appears to be about what productivity isn't. More subheadings might be helpful, a lot more paragraph breaks definitely would. ...if you already understand and agree with them, sure. The goal of those books is to make an emotional, "near"/concrete-construal argument for their readers to actually implement the ideas proposed, or at the very least come to believe it desirable to implement the ideas. Simply listing off the ideas doesn't have the same effect, because it doesn't create an emotional experience or have any of the repetition. Most people remember stories and analogies much more easily, so it's a better way to prime intuition pumps than simple factual presentation. (This is why the Sequences are so bloody long, too. In principle, Bayes can be explained with an equation and a few paragraphs. But getting people to do something with it, or even to want to do something with it, is much harder.) Take, for example, this post. In reading it over I get an impression of myself as skimming through someone's notes from a semester-long course on productivity, that I might or might not want to take. But my emotional experience of reading it was a feeling of overwhelm at so many things to learn and do, and a desire to put off detailed reading for some imagined future when I have more time to read it and flesh out all the ideas that were not fully fleshed out. My intellectual reflection on that says, "Bruce Lee something something thousand kicks something something one kick a thousand times", and that I would rather read a book that spends 100 pages dealing with just one of the many, many ideas you presented here, rather than spend the time t

Thank you for the valuable feedback. I'll try to improve that more in future posts and add more paragraphs here. 

5Viliam18hThis was awesome! Thinking about productivity at job, I wonder whether there is a trade-off for the company, of a similar type like when you have a trade-off between short-tem productivity and long-term productivity. Maybe what is most productive (long-term) for the employee is not what is most productive (long-term) for the company. Like, maybe if you get too good, then the rational choice for you would be to leave (perhaps to start your own company). On the other hand, if you are perhaps less productive but more easy to replace, the company gets less value from you, but it can keep going if you leave for whatever reason, which also gives you less of a leverage in negotiation. I think about this because you write about power of specialization, but in the software development I think I see the opposite trend: specialists being replaced by "full-stack developers", who are later unified with system administrators into "DevOps", who later become "DevSecOps", and god knows where this will end, maybe one day we will see a "DevSecOpsManagerAccountantSalesmanJanitor" and I sincerely hope I will be retired by then. Yeah, I have seen books that had less content than this post. I suppose you have to produce a certain minimum amount of pages, otherwise people would not pay for the book (or would even read it in the book shop for free).

If it’s worth saying, but not worth its own post, here's a place to put it.

If you are new to LessWrong, here's the place to introduce yourself. Personal stories, anecdotes, or just general comments on how you found us and what you hope to get from the site and community are invited. This is also the place to discuss feature requests and other ideas you have for the site, if you don't want to write a full top-level post.

If you want to explore the community more, I recommend reading the Library, checking recent Curated posts, seeing if there are any meetups in your area, and checking out the Getting Started section of the LessWrong FAQ. If you want to orient to the content on the site, you can also check out the new Concepts section.

The Open Thread tag is here. The Open Thread sequence is here.

I don't myself work in AI risk so I'm not the ideal person to respond but I'm in the community for quite a while so given that nobody who actually works in the field answered I will try to give my answer:

80,000 hours has a general guide for AI risk: the also published a podcast.

One of the key features is that there's a pretty high bar to be payed to work in AI safety.

I don't want to apply to programs that aren't worth it (it's possible my qualifications are sufficient for some of the ones I'll apply to, but

... (read more)
2ChristianKl4hDeciding based on the two approaches based on which values they align with misunderstands the problem. A good strategy depends on what's actually possible. The idea that human/AI hybrids are competitive at requiring resources in an enviroment with strong AGIs is doubtful. That means that over time all the resources and power go to the AGIs.
2Yoav Ravid10hDoes anyone know an article that expands on the idea of separating teaching institutions and assessment? I wrote a short expansion (750~ words) myself, and will probably publish it soon, but I'd like to read articles by other people too if they exist.
4rodeo_flagellum18hI have being reading content from LW sporadically for the last several years; only recently, though, did I find myself visiting here several times per day, and have made an account given my heightened presence. From what I can tell, I am in a fairly similar position to Jozdien, and am also looking for some advice. I am graduating with a B.A. in Neuroscience and Mathematics this January. My current desire is to find remote work (this is important to me) that involves one or more of: [machine learning, mathematics, statistics, global priorities research]. In spirit of the post The topic is not the content [] , I would like to spend my time (the order is arbitrary) doing at least some of the following: discussing research with highly motivated individuals, conducting research on machine learning theory, specifically relating to NN efficiency and learnability, writing literature reviews on cause areas, developing computational models and creating web-scraped datasets to measure the extent of a problem or the efficacy of potential solution, and recommending courses-of-action (based on my assessments generated from the previous listed entity). Generally, my skill set and current desires lead me to believe that I will find advancing the capabilities of machine learning systems, quantifying and defining problem afflicting humans, and synthesizing research literature to inform action, all fulfilling, and that I will be effective in working on these things done as well. My first question: How should I proceed with satisfying my desires, i.e. what steps should I take to determine whether I enjoy machine learning research more than global priorities research, or vice versa? It is my plan to attend graduate school for one of [machine learning, optimization, computer science] at some point in life (my estimate is around the age of 27-30), but I would first like to experiment with working at an EA a

This is a cross-post from my personal blog.

What is this post about?

Over the summer of 2020, I slept pretty badly for a long period of time and it had pretty noticeable effects on my productivity, perceived energy levels, and well-being in general. This was mostly due to the fact that my room was under the southern roof of the house and it had around 28 degrees Celcius during "cool" summer nights which made it effectively impossible for me to get more than 6 hours of light sleep per night. In autumn, I moved to a new flat where I had to get a new bed, mattress, pillow, etc.

Since I had felt the negative consequences of bad sleep pretty recently, I figured I could conduct a bit of...

After going to an actual sleep doctor one of the surprising suggestions from the doctor was that sleeping on a flat surface can be supoptimal and having a mattress at an angle where the head is significantly higher then the feet can be helpful. Of course instead of changing the actual angle of the mattress cushions can produce similar effects.

The effect of applying the suggestion lead to me body relaxing in ways I didn't expect but I had the impression that after 1-2 months of adaption the effect wasn't there as strong anymore. 

Changing the angle of the mattress from time to time is likely useful and underrecommended. 

2ChristianKl2hThe CDC page and the one on noise machine seem to me to make claims about the maximum noise being the problem. Your above post seems to additionally make the claim that there's a recovery process that only happens when there's very little sounds which seems to me like an interesting separate claim from loud noise causes hearing damage.
2ChristianKl2hA practical alternative to white noise machines are nature sounds. Noise patterns like rain did exist in the natural enviroment and are likely more healthy then white noise. You likely still shouldn't make them too loud.
2ChristianKl3hThere's no sedative level and most melatonin products have doses that are too high to be clinically effective. What was the lowest dose you took?

Delta has taken over, and cases are rising rapidly, with a 58% rise this week after a 65% rise last week. There’s no reason to expect this to turn around in the near term. 

Three weeks ago, in One Last Scare, I ran the numbers and concluded that most places in America would ‘make it’ without a big scary surge from Delta. It’s time to look at what went wrong with that calculation, which I believe to be a failure to sufficiently integrate different parts of my model.  

Then there’s the question of what we are going to do about this, and whether we are going to destroy some combination of free speech and the ordinary day to day activities that constitute our lives and civilization, perhaps indefinitely, in...

3SDM4hYou don't really address the dosing controversy on vaccine efficacy, except mentioning that you think it's dumb to worry about, but there is something potentially concerning there. There's some evidence that the dosing interval is causing some differences between the vaccine efficacies reported in Israel and the UK. This discussion full of immunology science words I don't understand []says there is a measurable difference in response to delta for short vs long doses. The results from Israel, which used a 3-week dose interval, look bad [], and contrast with the UKs results [] , which were (mostly) on 8-12 week intervals. But there are apparently sampling biases []which mean the Israel data might not be fully reliable. See also here []. Overall, I'm not sure what to make of this - even if 3-week intervals is generating strong levels of neutralizing antibodies and won't become highly infectious or severely ill, it seems important to know if 2 Pfizer gives ~65% or ~90% protection. Unfortunately, I don't think we'll see studies on this for a while, so if anyone has better interpretations of this data it would be useful.
3Owain_Evans7hThe persistence of anosmia doesn't entail that other symptoms are caused by Covid. (IIRC the relevant cells in the nose take a while to regenerate). Though I agree this provides some evidence that Covid is the cause. This predicts that you'd find organ damage in these patients. Are there studies showing clear organ damage in people with mild cases 6 months later? I disagree. This does happen without an identifiable cause.

The persistence of anosmia doesn’t entail that other symptoms are caused by Covid

But it raises the probability.

3Zvi17hI agree that 55% would be an ambiguous evaluation, but 40% is substantially different than 65% and I've been using 40% in my recent models. Central mistake is the same in both cases. I don't remember what my update timeline was, I'd have to go back and look. Makes sense I would be too slow.

Pacific Gas & Electric is planning to spend $15-30b to bury power lines. I see why they're doing it: PG&E equipment sparked some of the worst fires in California history, including the 2018 Camp Fire which destroyed Paradise, but I'm not convinced that this is good for California overall.

Historically, the area used to burn periodically. We haven't allowed this for about a century, and flammable materials have been building up. It's all very likely to burn at some point, and burying power lines mostly just reduces the chance that it will be triggered by PG&E. Prescribed burns, spreading out the combustion and moving it to safer times of year, would reduce fire risk far more for the money. Even though when PG&E pays for something the money comes from their customers, CA residents, this isn't...

2Dagon14hWatchmen was pretty good on this front.Worm ( is LONG, but great.
1Gerald Monroe18hRight. The first step to a real solution would be to sector the forest into zones separated by a firebreak. Then schedule each zone for a burn every (time interval). However, cutting such barriers into a forest could island certain species and cause their extinction via genetic drift and diversity loss. (Essentially each isolated population has a high probability of losing genetic diversity each generation and so eventually the whole population will become fragile and will die out)

Then the solution could be to surround the entire forest by one firebreak, or perhaps for extra security make it two concentric ones with a narrow belt between them.

A creative (i.e. crazy, and probably not working in real life) solution could be to make the firebreak a spiral, so that technically the whole forest remains connected and the fire still spreads everywhere, but it spreads slowly, because it has to go in circles. People caught in the burning forest might even get a chance to outrun the fire. On a second thought, the same would be true for many a... (read more)

3Zolmeister20hI think it balances prescribed burns with other methods of fire-suppression (fire-breaks, thinning), and incentivizes local coordination among neighbors.