All of jsteinhardt's Comments + Replies

Let Us Do Our Work As Well

Thanks, really appreciate the references!

Economic AI Safety

If there was a feasible way to make the algorithm open, I think that would be good (of course FB would probably strongly oppose this). As you say, people wouldn't directly design / early adopt new algorithms, but once early adopters found an alternative algorithm that they really liked, word of mouth would lead many more people to adopt it. So I think you could eventually get widespread change this way.

Film Study for Research

Thanks for the feedback!

I haven't really digged into Gelman's blog, but the format you mention is a perfect example of the expertise of understanding some research. Very important skill, but not the same as actually conducting the research that goes into a paper.

Research consists of many skills put together. Understanding prior work and developing the taste to judge it is one of the more important individual skills in research (moreso than programming, at least in most fields). So I think the blog example is indeed a central one.

In research, especially in

... (read more)
2adamShimi2dSorry for taking so long to answer! I completely agree that it is a relevant and important skill, but there are many people with good understanding of prior work who are completely unable of producing interesting new research. Non-exhaustively, this includes being able to have new ideas, to develop them, to test them, to get feedback and adapt to the feedback. And given that understanding prior work emerges pretty naturally once you read a lot of papers, I'm personally more interested in training for these other skills. My argument was that blogs don't really help for that. Difference of opinion: for me, coming with ideas is incredibly cheap. I also have piles of promising ideas that I will never have the time to explore, and I keep having new ideas. I never needed any help in that, and so I am completely uninterested in any way to generate more ideas. The other skills of research require way more effort to me (not even sure how to disentangle them TBH), so I focus on those. And I have trouble finding any actual standard skills that translate directly between research field: even things like doing experiments have very different meaning and related skills depending on the field. Didn't want to imply that athletes never innovate. And that's an interesting example of the innovation from adjacent field. That's definitely how I get a lot of ideas. But that's still made incredibly more potent by being able to study and master the skills from your actual field. Which is really hard to do when there is no film study analogy for it.
Experimentally evaluating whether honesty generalizes

Actually, another issue is that unsupervised translation isn't "that hard" relative to supervised translation--I think that you can get pretty far with simple heuristics, such that I'd guess making the model 10x bigger matters more than making the objective more aligned with getting the answer right (and that this will be true for at least a couple more 10x-ing of model size, although at some point the objective will matter more).

This might not matter as much if you're actually outputting explanations and not just translating from one language to another. Although it is probably true that for tasks that are far away from the ceiling, "naive objective + 10x larger model" will outperform "correct objective".

2paulfchristiano2moI do expect "explanations of what's going on in this sentence" to be a lot weaker than translations. For that task, I expect that the model trained on coherence + similar tasks will outperform a 10x larger pre-trained model. If the larger pre-trained model gets context stuffing on similar tasks, but no coherence training, then it's less clear to me. But I guess the point is that the differences between various degrees of successful-generalization will be relatively small compared to model size effects. It doesn't matter so much how good the transfer model is relative to the pre-trained baseline, it matters how large the differences between the possible worlds that we are hoping to distinguish are. I guess my main hope there is to try to understand whether there is some setting where transfer works quite well, either getting very close to the model fine-tuned on distribution, or at least converging as the pre-trained model grows. Hopefully that will make it easier to notice the effects we are looking for, and it's OK if those effects are small relative to model doublings. (Also worth noting that "as good as increasing model size by 10%" is potentially quite economically relevant. So I'm mostly just thinking about the extent to which it can make effects hard to measure.)
Experimentally evaluating whether honesty generalizes

Thanks Paul, I generally like this idea.

Aside from the potential concerns you bring up, here is the most likely way I could see this experiment failing to be informative: rather than having checks and question marks in your tables above, really the model's ability to solve each task is a question of degree--each table entry will be a real number between 0 and 1. For, say, tone, GPT-3 probably doesn't have a perfect model of tone, and would get <100% performance on a sentiment classification task, especially if done few-shot.

The issue, then, is that the ... (read more)

2paulfchristiano2moPart of my hope is that "coherence" can do quite a lot of the "telling you what humans mean about tone." For example, you can basically force the model to talk (in English) about what things contribute to tone, and why it thinks the tone is like such and such (or even what the tone of English sentences is)---anything that a human who doesn't know French can evaluate. And taken together those things seem like enough to mostly pin down what we are talking about. I'd tentatively interpret that as a negative result, but I agree with your comments below that ultimately a lot of what we care about here is the scaling behavior and putting together a more holistic picture of what's going on, in particular: * As we introduce stronger coherence checks, what happens to the accuracy? Is it approaching the quality of correctness, or is it going to asymptote much lower? * Is the gap shrinking as model quality improves, or growing? Do we think that very large models would converge to a small gap or is it a constant? I'm also quite interested in the qualitative behavior. Probably most interesting are the cases where the initial model is incoherent, the coherence-tuned model is coherent-but-wrong, and the correctness-tuned model is correct. (Of course every example is also fuzzy because of noise from sampling and training, but the degree of fuzziness is smaller as we remove randomness.) In these cases, what is happening with the coherence-tuned model? Are we able to see cases where it cleanly feels like the "wrong" generalization, or is it a plausible ambiguity about what we were looking for? And so on. I'm interested in the related engineering question: in this setting, what can we do to improve the kind of generalization we get? Can we get some handle on the performance gap and possible approaches to closing it? And finally I'm interested in understanding how the phenomenon depends on the task: is it basically similar in different domains / for different kinds of q
2jsteinhardt2moActually, another issue is that unsupervised translation isn't "that hard" relative to supervised translation--I think that you can get pretty far with simple heuristics, such that I'd guess making the model 10x bigger matters more than making the objective more aligned with getting the answer right (and that this will be true for at least a couple more 10x-ing of model size, although at some point the objective will matter more). This might not matter as much if you're actually outputting explanations and not just translating from one language to another. Although it is probably true that for tasks that are far away from the ceiling, "naive objective + 10x larger model" will outperform "correct objective".
AI x-risk reduction: why I chose academia over industry

This doesn't seem so relevant to capybaralet's case, given that he was choosing whether to accept an academic offer that was already extended to him.

Covid 2/18: Vaccines Still Work

I think if you account for undertesting, then I'd guess 30% or more of the UK was infected during the previous peak, which should reduce R by more than 30% (the people most likely to be infected are also most likely to spread further), and that is already enough to explain the drop.

1TheMajor7moThis is a very good point, and in my eyes explains the observations pretty much completely. Thanks!
Making Vaccine

I wasn't sure what you meant by more dakka, but do you mean just increasing the dose? I don't see why that would necessarily work--e.g. if the peptide just isn't effective.

I'm confused because we seem to be getting pretty different numbers. I asked another bio friend (who is into DIY stuff) and they also seemed pretty skeptical, and Sarah Constantin seems to be as well:

Not disbelieving your account, just noting that we seem to be getting pretty different outputs from the expert-checking process... (read more)

2John_Maxwell7moFixed twitter link []
Making Vaccine

Have you run this by a trusted bio expert? When I did this test (picking a bio person who I know personally, who I think of as open-minded and fairly smart), they thought that this vaccine is pretty unlikely to be effective and that the risks in this article may be understated (e.g. food grade is lower-quality than lab grade, and it's not obvious that inhaling food is completely safe). I don't know enough biology to evaluate their argument, beyond my respect for them.

I'd be curious if the author, or others who are considering trying this, have applied this... (read more)

6johnswentworth7moI did not try this test. I had enough bio and physiology background to be confident in my own assessment, though I would not advise others to be similarly confident in my assessment - my background is not legible enough for that.

In my case, yes.  My bio expert indicated that it was likely to be effective (more than 50%, but less than 90%) and that the risks were effectively zero in terms of serious complications.

Regarding the food grade versus lab grade question, as well as inaccuracies or mistakes in construction of the vaccine, this was a question I spent a reasonable amount of time on.  The TL/DR is that the engineering tolerances are incredibly wide; the molecular weight of the chitosan isn't that important, the mixing rate isn't that important other than it be fast ... (read more)

Making Vaccine

I don't think I was debating the norms, but clarifying how they apply in this case. Most of my comment was a reaction to the "pretty important" and "timeless life lessons", which would apply to Raemon's comment whether or not he was a moderator.

4johnswentworth7moYeah, I don't mean to say your comment was bad as-written, just preemptively heading off a potential thread.
Making Vaccine

Often, e.g. Stanford profs claiming that COVID is less deadly than the flu for a recent and related example.

-1cistran7moFor children, it is as far as we know.
7waveBidder7moJohn Ioannidis [], of all people, who should know better.
Making Vaccine

Hmm, important as in "important to discuss", or "important to hear about"?

My best guess based on talking to a smart open-minded biologist is that this vaccine probably doesn't work, and that the author understates the risks involved. I'm interpreting the decision to frontpage as saying that you think I'm wrong with reasonably high confidence, but I'm not sure if I should interpret it that way.

You should make a top-level comment about this. Chance that the vaccine works and the associated risks are object-level questions well-worth discussing.

In general, frontpage decisions are not endorsements (though I don't know Raemon's thoughts in this particular case), and this comment section is not the place for a debate about frontpaging norms. This is definitely the place to talk about chance the vaccine works and associated risks, though.

Covid 12/24: We’re F***ed, It’s Over

That seems irrelevant to my claim that Zvi's favored policy is worse than the status quo.

Covid 12/24: We’re F***ed, It’s Over

This isn't based on personal anecdote, sudies that try to estimate this come up with 3x. See eg the MicroCovid page:

1rockthecasbah8moThat seems plausible right now, in January, at our current level of social distancing compliance. But why would the degree of distancing stay constant over vaccination? It hasn't even stayed constant the last 8 months when nobody has been vaccinated. So far we have a clear pattern. People voluntarily comply when the issue seems important because there are lots of infections, hospitalizations and deaths. During lulls the issue becomes less available and compliance drops. In the best case for essential worker vaccination, it produces a lull in February-March. But if you actually drop the reproduction rate then that 3x factor goes away immediately. Unless you have a reliable plan to get people to keep social distancing even when things seem over, vaccinating the vulnerable saves lives in expectation.
Covid 12/31: Meet the New Year

You may well be right. I guess we don't really know what the sampling bias is (it would have to be pretty strongly skewed towards incoming UK cases though to get to a majority, since the UK itself was near 50%).

1PPaul9moI might be missing something, but where in this link do you see the dominance? If it is the large proportion of sequencing showing B.1.1.7 (18/33 for Italy and 4/13 for Israel), isn't that due to increased surveillance, like testing positive people coming from the UK?
Covid 12/31: Meet the New Year

I don't think it's correct to say that it remains stable at 0.5-1% of samples in Denmark. There were 13 samples of the new variant last week, vs. only 3 two weeks ago, if I understood the data correctly. If it went from 0.5% to 1% in a week then you should be alarmed. (Although 3 and 13 are both small enough that it's hard to compute a growth rate, but it certainly seems consistent with the UK data to me.)

I think better evidence against non-infectiousness would be Italy and Israel, where the variant seems to be dominant but there isn't runaway growth. But:... (read more)

3dotchart9moAt the time of writing the weekly percentages were 0.3%, 0%, 0.2%, 0.5%, 0.9% which I did not perceive as weekly doubling. But I was likely fooled by the noise of the first weeks where numbers were too low to be meaningful. Yesterday latest weekly numbers came out and last week the percentage was 2.3%. So numbers are clearly worrying and in line with Zvi's post.
1lunis9moDo you have a source for B.1.1.7 being dominant in Italy/Israel? Assuming it’s already dominant there, that strongly suggests that it’s infectious enough to have rapidly outcompeted other strains, but that Italy/Israel were able to push down the higher R through some combination of behavioral change and vaccination. (Note: I can’t find any sources saying B.1.1.7 is dominant in Italy or Israel, and I’d be surprised if that were already the case.)
Covid 12/31: Meet the New Year

Zvi, I still think that your model of vaccination ordering is wrong, and that the best read of the data is that frontline essential workers should be very highly prioritized from a DALY / deaths averted perspective. I left this comment on the last thread that explains my reasoning in detail, looking at both of the published papers I've seen that model vaccine ordering: link. I'd be happy to elaborate on it but I haven't yet seen anyone provide any disagreement.

More minor, but regarding rehab facilities, from a bureaucratic perspective they are "congregate ... (read more)

Covid 12/24: We’re F***ed, It’s Over

Zvi, I agree with you that the CDC's reasoning was pretty sketchy, but I think their actual recommendation is correct while everyone else (e.g. the UK) is wrong. I think the order should be something like:

Nursing homes -> HCWs -> 80+ -> frontline essential workers -> ...

(Possibly switching the order of HCWs and 80+.)

The public analyses saying that we should start with the elderly are these two papers:

Notably, both p... (read more)

6rockthecasbah9moMany people on this website are hardcore social distancers, interacting only with essential workers. To them it seems natural that essential workers are the majority of the transmission and do not have immunity yet. But most people aren't social distancing very hard at all. In Nashville, were I currently am, the bars and restaurants are often full. My immune brother when to house parties and indoor concerts on New Years Eve. I doubt that essential workers constitute even a majority of current transmission. So we vaccinate 80 million people and reduce transmission by 50%, maybe. That would take months. Meanwhile, there are only 50 million Americans over 65, doing >90% of the dying, and we could vaccinate them in just two months. TLDR; The transmission argument for essential workers assumes people comply with social distancing. People aren't doing that anymore, so vaccinate the vulnerable.
4JesperO9moEven if this is right, it still seems incredibly dysfunctional for CDC (and other governing bodies) to not use age categories among healthcare workers, and other essential worker categories.
Why are young, healthy people eager to take the Covid-19 vaccine?

Mo Bamba (NBA) and Cody Garbrandt (UFC) are both pro athletes who are still out of commission months later. I found this looking for NBA information, and only about 50 NBA players have gotten Covid, so this suggests at least 2% chance of pretty bad long term symptoms.

Pain is not the unit of Effort

I think that the right amount level of effort leaves you tired but warm inside, like you look forward doing this again, rather than just feeling you HAVE to do this again.


This is probably true in a practical sense (otherwise you won't sustain it as a habit), but I'm not sure it describes a well-defined level of effort. For me an extreme effort could still lead to me looking forward to it, if I have a concrete sense of what that effort bought me (maybe I do some tedious and exhausting footwork drills, but I understand the sense in which this will carr... (read more)

Pain is not the unit of Effort

If most workouts are painful, then I agree you are probably overtraining. But if no workouts at all are painful, you're probably missing opportunities to improve. And many workouts should at least be uncomfortable for parts of it. E.g. when lifting, for the last couple deadlift sets I often feel incredibly gassed and don't feel like doing another one. But this can be true even when I'm far away from my limits (like, a month later I'll be lifting 30 pounds more and feel about as tired, rather than failing to do the lift).

My guess is that on average 1-2 work... (read more)

Why are young, healthy people eager to take the Covid-19 vaccine?

You could look at papers published on medrxiv rather than news articles, which would resolve the clickbait issue, though you'd still have to assess the study quality.

Why are young, healthy people eager to take the Covid-19 vaccine?

Have you tried googling yourself and were unable to find them? (Sorry that I'm too lazy to re-look them up myself, but given that LW is mostly leisure for me I don't feel like doing it, and I'd be somewhat surprised if you googled for stuff and didn't find it.)

2[anonymous]10moHa, I understand your laziness because I'm at least as lazy. Separating clickbait from quality information is too much work for my liking and so I'm crowdsourcing that classification here.
Why are young, healthy people eager to take the Covid-19 vaccine?

I also think you are probably overestimating vaccine risks (the main risk is that its effectiveness wanes, and that it interferes with future antibody responses from similar vaccines; not that you'll get horrible side effects) but that isn't necessary to explain why people want the vaccine now.

Why are young, healthy people eager to take the Covid-19 vaccine?

I think cutting the IFR by 25 on the basis of one study is a mistake, the chance of the study being fatally flawed is greater than 1 in 25. On the other hand 0.5% is overall CFR and would be lower for young people.

I think it's hard to cut risk of long term effects by more than a factor of 10 from published estimates. Note there is evidence of long term effects contrary to your claim, i.e. studies that do 6 week follow ups and find people still with some symptom. This isn't 6 months but is still surprisingly long and should shift our belief about 6 months a... (read more)

1[anonymous]10moGot any links?
5jsteinhardt10moI also think you are probably overestimating vaccine risks (the main risk is that its effectiveness wanes, and that it interferes with future antibody responses from similar vaccines; not that you'll get horrible side effects) but that isn't necessary to explain why people want the vaccine now.
Why Boston?

I noticed the prudishness, but "rudeness" to me parses as people actually telling you what's on their mind, rather than the passive-aggressive fake niceness that seems to dominate in the Bay Area. I'll personally take the rudeness :).

7maia1y... huh, is that the thing that makes it mysteriously easier for me to talk to people from the East Coast?
Why Boston?

On the other hand, the second-best place selects for people who don't care strongly about optimizing for legible signals, which is probably a plus. (An instance of this: In undergrad the dorm that, in my opinion, had the best culture was the run-down dorm that was far from campus.)

4jefftk1yThis was my experience at Swarthmore as well. But I think a lot of that came from this being a dorm that essentially, any student who wanted to live there would be able to get a room. The analogy would push toward choosing a place that has much cheaper housing costs!
Why Boston?

Many of the factors affecting number of deaths are beyond a place's control, such as how early on the pandemic spread to that place, and how densely populated the city is. I don't have a strong opinion about MA but measuring by deaths per capita isn't a good way of judging the response.

What's Wrong with Social Science and How to Fix It: Reflections After Reading 2578 Papers

That's not really what a p-value means though, right? The actual replication rate should depend on the prior and the power of the studies.

4rohinmshah1yReplied to John below
Covid-19 6/11: Bracing For a Second Wave

My prediction: infections will either go down or only slowly rise in most places, with the exception of one or two metropolitan areas. If I had to pick one it would be LA, not sure what the second one will be. The places where people are currently talking about spikes won't have much correlation with the places that look bad two weeks from now (i.e. people are mostly chasing noise).

I'm not highly confident in this, but it's been a pretty reliable prediction for the past month at least...

Estimating COVID-19 Mortality Rates

Here is a study that a colleague recommends: Tweet version:

Their point estimate is 0.64% but with likely heterogeneity across settings.

Quarantine Bubbles Require Directness, and Tolerance of Rudeness

I don't think bubble size is the right thing to measure; instead you should measure the amount of contract you have with people, weighted by time, distance, indoor/outdoor, mask-wearing, and how likely the other person is to be infected (I.e. how careful they are).

An important part of my mental model is that infection risk is roughly linear in contact time.

Quarantine Bubbles Require Directness, and Tolerance of Rudeness

As a background assumption, I'm focused on the societal costs of getting infected, rather than the personal costs, since in most places the latter seem negligible unless you have pre-existing health conditions. I think this is also the right lens through which to evaluate Alameda's policy, although I'll discuss the personal calculation at the end.

From a social perspective, I think it's quite clear that the average person is far from being effectively isolated, since R is around 0.9 and you can only get to around half of that via only household infection. S

... (read more)
6Vaniver1yBut, of course, any 12-person bubble that contains someone with a pre-existing health condition can't rest on 11 of the people thinking "oh, but I'm healthy!". I think 'the average person' is the wrong thing to think about here. When the infection is rare, R will be driven by the actions of the riskiest people, since they're the ones who predominantly have it, spread it, and catch it. If 50% of the population has an actual risk of 0, and there aren't any graph connections between them and the other 50% of the population, then the whole population R will be driven by the connected half (and will only have slowed by by whatever connections got severed to the hermit half). On the one hand, this is a message for hope ("you can probably relax to 'normal human' standards and only have an R of 1"), but also 'normal human' standards might be incompatible in other ways (someone who lives with 0 or 1 other person has much less to fear from a household secondary attack rate of 0.3 than someone who lives in a house of 12 people). Sure, 12 is a magic number, and actually weighing the tradeoffs should lead to different thresholds in different situations. But the overall thing you're trying to balance is "risk cost" against "socialization gains", and even if costs are linear, sublinear benefits scuttle these sorts of symmetry analyses. I think the bit of this that I'm having the hardest time wrapping my head around is something like "if you accept people that are as careful as you, then you are less careful than you used to be." Like, suppose you have a 12-person bubble, all of whom don't interact with the outside world. Then if you say "we are open to all bubbles with at most 12 people, all of whom don't interact with the outside world", you now potentially have a bubble whose size is measured in the hundreds, which is a pretty different situation than the one you started in.
Quarantine Bubbles Require Directness, and Tolerance of Rudeness

I think the biggest issue with the bubble rule is that the math doesn't work out. The secondary attack rate between house members is ~30% and probably much lower between other contacts. At that low of a rate, these games with the graph structure buy very little and may be harmful because they increase the fraction of contact occurring between similar people (which is bad because the social cost of a pair of people interacting is roughly the product of their infection risks).

4Raemon1yI'm somewhat confused what you mean by "the math doesn't work out." Compared to what? If you're well coordinated, it seems like the secondary household infection rate isn't too relevant, because you don't interact with anyone else and you don't get sick. If you're not well coordinated, in the absence of this rule you're probably off doing crazier things. Are you assuming something like "you're medium coordinated, which is enough to jump through the basic hoops for 12-person-bubble but not enough to avoid one person getting sick and then getting the bubble sick?" Seems plausible, but what number would you have picked? (bearing in mind that if you choose too low a number, after 6 months people are like 'screw it' and doing whatever)
Estimating COVID-19 Mortality Rates

I'm not trying to intimidate; I'm trying to point out that I think you're making errors that could be corrected by more research, which I hoped would be helpful. I've provided one link (which took me some time to dig up). If you don't find this useful that's fine, you're not obligated to believe me and I'm not obligated to turn a LW comment into a lit review.

6Benquo1yGiven that it apparently took you some time to dig up even as much as a tweet with a screen cap of some numbers that with quite a lot of additional investigation might be helpful, I hope you're now at least less "confused" about why I am "relying on this back of the envelope rather than the pretty extensive body of work on this question." If you want to see something better, show something better.
Estimating COVID-19 Mortality Rates

The CFR will shift substantially over time and location as testing changes. I'm not sure how you would reliably use this information. IFR should not change much and tells you how bad it is for you personally to get sick.

I wouldn't call the model Zvi links expert-promoted. Every expert I talked to thought it had problems, and the people behind it are economists not epidemiologists or statisticians.

For IFR you can start with seroprevalence data here and then work back from death rates:

R... (read more)

2Benquo1yThe director of NIAID publicly endorsed [] that model's bottom line.
2Douglas_Knight1yBecause of false positives, seroprevalence is massively overestimated everywhere that there hasn't been a massive outbreak. In those places the IFR is 1-2%. But can we extrapolate to normal outbreaks? If, as widely believed, an overrun medical system has worse mortality, then maybe the normal IFR really is only 0.5-1%. But if your meta-analysis directly measures that, it is not well-done.
4Benquo1yI clicked through to the tweet you mentioned, which contains a screencap of a chart purporting to show "An Approximate Percentage of the Population That Has COVID-19 Antibodies." No dates or other info about how these numbers might have been generated. Fortunately, Gottlieb's next tweet in the thread contains another screencap of the URLs of the studies mentioned in the chart. I hand-transcribed the Wuhan study [] URL, and found that while it was performed at a date that's probably helpful (April 20th) it's a study in a single hospital in Wuhan, and the abstract explicitly says it's not a good population estimate: I'd need to know more about e.g. hospitalization rates in Wuhan to interpret this. The New York numbers seem to come from a press release [] , with no clear info about how testing was conducted. All of these are point estimates, and to get ongoing infection rates, I'd need to fit a time series model with too many degrees of freedom. Not saying no one can do this, but definitely saying it's not clear to me how I can make use of these numbers without working on the problem full time for a few weeks. You've nonspecifically referred to experts and models a few times; that's not helpful and only serves to intimidate. What would be helpful would be if you could point to specific models by specific experts that make specific claims which you found helpful.
Estimating COVID-19 Mortality Rates

Ben, I think you're failing to account for under-testing. You're computing the case fatality rate when you want the infection fatality rate. Most experts, as well as the well-done meta analyses, place the IFR in the 0.5%-1% range. I'm a little bit confused why you're relying on this back of the envelope rather than the pretty extensive body of work on this question.

2Benquo1yIFR isn't that helpful when trying to use public case data to estimate a hazard rate. I'll add a note clarifying that in the post. Since what's reported are cases, case fatalities are the natural thing to multiply the rate of new cases by. Some apparently expert-promoted models have been total nonsense [] , and I prefer a back-of-the-envelope calculation whose flaws are obvious and easy for me to understand, to comparatively opaque sophisticated estimates which I can't interpret. Can you point me to a clear concise account that shows how to estimate IFR with available data and use it in a decision-relevant way?
Ben Hoffman's donor recommendations

I don't understand why this is evidence that "EA Funds (other than the global health and development one) currently funges heavily with GiveWell recommended charities", which was Howie's original question. It seems like evidence that donations to OpenPhil (which afaik cannot be made by individual donors) funge against donations to the long-term future EA fund.

4Benquo3yThe definitions of and boundaries between Open Phil, GiveWell, and Good Ventures, as financial or decisionmaking entities, are not clear.
RFC: Philosophical Conservatism in AI Alignment Research

I like the general thrust here, although I have a different version of this idea, which I would call "minimizing philosophical pre-commitments". For instance, there is a great deal of debate about whether Bayesian probability is a reasonable philosophical foundation for statistical reasoning. It seems that it would be better, all else equal, for approaches to AI alignment to not hinge on being on the right side of this debate.

I think there are some places where it is hard to avoid pre-commitments. For instance, while this isn't quite a philo... (read more)

1G Gordon Worley III3yI agree we must make some assumptions or pre-commitments and don't expect we can avoid them. In particular there are epistemological issues that force our hands and require we make assumptions because complete knowledge of the universe is beyond the capacity we have to know it. I've talked about this idea some [] and I plan to revisit it as part of this work.

FWIW I understood Zvi's comment, but feel like I might not have understood it if I hadn't played Magic: The Gathering in the past.

EDIT: Although I don't understand the link to Sir Arthur's green knight, unless it was a reference to the fact that M:tG doesn't actually have a green knight card.

6CronoDAS3yThe Arthurian Green Knight lets Gawain cut off his head, then picks it up and puts it back on. Trying to use force on the Green Knight is useless.
3LawChan4yI also think I wouldn't have understood his comments without MTG or at least having read Duncan's explanation to the MTG color wheel. (Nitpicking) Though I'd add that MTG doesn't have a literal Blue Knight card either, so I doubt it's that reference. (There are knights that are blue and green, but none with the exact names "Blue Knight" or "Green Knight".)
Takeoff Speed: Simple Asymptotics in a Toy Model.

Thanks for writing this Aaron! (And for engaging with some of the common arguments for/against AI safety work.)

I personally am very uncertain about whether to expect a singularity/fast take-off (I think it is plausible but far from certain). Some reasons that I am still very interested in AI safety are the following:

  • I think AI safety likely involves solving a number of difficult conceptual problems, such that it would take >5 years (I would guess something like 10-30 years, with very wide error bars) of research to have solutions that we are happy with.
... (read more)
2Aaron Roth4yGood points all; these are good reasons to work on AI safety (and of course as a theorist I'm very happy to think about interesting problems even if they don't have immediate impact :-) I'm definitely interested in the short-term issues, and have been spending a lot of my research time lately thinking about fairness/privacy in ML. Inverse-RL/revealed preferences learning is also quite interesting, and I'd love to see some more theory results in the agnostic case.
Takeoff Speed: Simple Asymptotics in a Toy Model.

Very minor nitpick, but just to add, FLI is as far as I know not formally affiliated with MIT. (FHI is in fact a formal institute at Oxford.)

Zeroing Out

Hi Zvi,

I enjoy reading your posts because they often consist of clear explanations of concepts I wish more people appreciated. But I think this is the first instance where I feel I got something that I actually hadn't thought about before at all, so I wanted to convey extra appreciation for writing it up.



Seek Fair Expectations of Others’ Models

I think the conflation is "decades out" and "far away".

Galfour was specifically asked to write his thought up in this thread:

It seems either this was posted to the wrong place, or there is some disagreement within the community (e.g. between Ben in that thread and the people downvoting).

1[comment deleted]4y
5gjm4yYou may well be right, but it's also possible that some readers think (1) Galfour did well to write up his thoughts but (2) now that we've seen them his thoughts are terrible. (Ridiculous over-the-top analogy: you ask a friend to tell you honestly and without filters what his political opinions are. He turns out to be an unreconstructed Nazi. You're glad he told you honestly, but now he's done it you don't want to be his friend any more.) Or some may think: (1) as above but (2) the comments above aren't actually writing up his thoughts about AGI, and aren't interesting. Or some may think: (1) as above but (2) Ben specifically suggested that personal thoughts on AGI should go on personal LW2 blogs rather than the front page, whereas here Galfour is saying that when he writes up his thoughts they will go on the front page. (Lest I be misunderstood: I have not downvoted this post; I don't think anything he wrote above was terrible; I also don't think it's terribly interesting, but since it's intended mostly as background I don't see any particular reason why it needs to be; I have no strong opinion on whether Galfour's AGI opinions, once written, will belong on the front page.)
Oxford Prioritisation Project Review

Points 1-5 at the beginning of the post are all primarily about community-building and personal development externalities of the project, and not about the donation itself.

Load More