This is a special post for quick takes by Noah Birnbaum. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
While I think LW’s epistemic culture is better than most, one thing that seems pretty bad is that occasionally mediocre/shitty posts get lots of upvotes simply because they’re written by [insert popular rationalist thinker].
Of course, if LW were truly meritocratic (which it should be), this shouldn’t matter — but in my experience, it descriptively does.
Without naming anyone (since that would be unproductive), I wanted to know if others notice this too? And aside from simply trying not to upvote something because it’s written by a popular author, anyone have good ideas for preventing this?
The effect seems natural and hard to prevent. Basically, certain authors get reputations for being high (quality * writing), and then it makes more sense for people to read their posts because both the floor and ceiling are higher in expectation. Then their worse posts get more readers (who vote) than posts of a similar quality by another author, who's floor and ceiling is probably lower.
I'm not sure the magnitude of the cost, or that one can realistically expect to ever prevent this effect. For instance, ~all Scott Alexander blogposts get more readership than the best post by many other authors who haven't built a reputation and readership, and this kind of just seems part of how the reading landscape works.
Of course, it can be frustrating as an author to sometimes see similar quality posts on LW get different karma. I think part of the answer here is to do more to celebrate the best posts by new authors. The main thing that comes to mind here is curation, where we celebrate and get more readership on the best posts. Perhaps I should also have a term here for "and this is a new author, so I want to bias toward curating them for the first time so that they're more invested in writing more good content".
4Yes, but you'd naively hope this wouldn't apply to shitty posts, just to mediocre posts. Like, maybe more people would read, but if the post is actually bad, people would downvote etc.
1That's right. One exception: sometimes I upvote posts/comments written to low standards in order to reward the discussion happening at all. As an example I initially upvoted Gary Marcus's first LW post in order to be welcoming to him participating in the dialogue, even though I think the post is very low quality for LW.
(150+ karma is high enough and I've since removed the vote. Or some chance I am misremembering and I never upvoted because it was already doing well, in which case this serves as a hypothetical that I endorse.)
1
One thing you could do is give users relatively more voting power if they vote without seeing the author of the post. I.e., you can enable a mode which hides post authors until you give a vote on the anonymized content. After that, you can still vote like normal.
Obviously there are ways author identity can leak through this, but it seems better than nothing.
You should probably link some posts, it's hard to discuss this so abstractly. And popular rationalist thinkers should be able to handle their posts being called mediocre (especially highly-upvoted ones).
5
A salient example to me: This post essentially consists of Paul briefly remarking on some mildly interesting distinctions about different kinds of x-risks, and listing his precise credences without any justification for them. It's well-written for what it aims to be (a quick take on personal views), but I don't understand why this post was so strongly celebrated.
5
This was also on my mind after seeing Jesse’s short form yesterday. Ryan’s “this is good” comment was above Louis’ thorough explanation of an alternative formal motivation for IFs. That would still be the case if I hadn’t heavy upvoted and weak downvoted.
I personally cast my comment up/downvotes as an expression of my preference ordering for visibility. I would encourage others to also do so. For instance, I suggest Ryan’s comment should’ve been agreement voted rather than upvoted by others. This stance has as a corollary to not vote if you haven’t read the other comments whose rankings you are affecting—or rather vote with any other marker of which LW has many.
This ‘upvotes as visibility preferences’ policy isn’t tractable for posts, so I suspect the solution there—if one is needed—would have to be done on the backend by normalization. Not sure whether this is worth attempting.
Link here since I don’t particularly want to call out Ryan, his comment was fine. https://www.lesswrong.com/posts/7X9BatdaevHEaHres/jesse-hoogland-s-shortform
9
idea: popular authors should occasionally write preregistered intentionally subtly bad posts as an epistemic health check
9
Duncan Sabien once ran the inverse experiment. He made a separate account to see how his posts would do without his reputation. The account only has one post still up, but iirc there used to be many more (tens). They performed similarly well to posts under his own name. Cool idea!
[plausibly I'm getting parts of the story wrong and someone who was around then will correct me]
1
i've thought big names should do this for conference papers to keep conferences honest (peer review is anonymous but as i understand it it's extremely obvious when a big name has written a paper)
4
That's not really true if people are popular rationalist thinkers because of skill at good rationalist writing. Meritocracy does not imply that people get judged based on individual pieces of their work, a meritocracy where people are primarily judged on their total output would still be a meritocracy.
I think the problem is more that post that make points that are popular and fit neatly into the world view of the reader are more likely to get upvotes than posts that challenge the world view of the reader and would require the reader to update their world view. Jimmy's introductionary post to his sequence is currently as I'm writing it at 12 karma.
While writing quality might be improved, it's a post that sets out to challenges the readers conception about how human reasoning works in practical contexts and that's why it's at low karma. I would love to see more posts like Jimmy that are about the core of how rationality works over posts that feel good to read and that get lot of upvotes without really changing minds much.
4
I think if someone is very well-known their making a particular statement can be informative in itself, which is probably part of the reason it is upvoted.
1
I had similar thoughts. And indeed I found funny that a forum of rationalists falls for this.
But this is human nature, and we have finite time and energy to focus on things.
You will find the same for the attention papers written by people at famous institutions get, even if low quality.
Making names, recently, I had a similar feeling when looking at "AI as Normal Technology". I did not know the authors, and I was skimming it because someone talked about it.
It is also a reason for which sometimes celebrities are called to do propaganda.
A strategy would be to make make posts anonymous for a limited amount of time (perhaps with some text randomization that would slightly change the style of the author/s, even though this might affect quality).
I was pretty unimpressed with Dario Amodei in the recent conversation with Demis Hassabis at the World Economic Forum about what comes after AGI.
I don’t know how much of this is a publicity thing, but it felt he wasn't really taking the original reasons for going into AI seriously (i.e. reducing x-risk). The overall message seemed to be “full speed ahead,” mostly justified by some kinda hand-wavy arguments about geopolitics, with the more doomy risks acknowledged only in a pretty hand-wavy way. Bummer.
My main takeaway of what Dario said in that talk is that Anthropic is very determined to kick off the RSI loop and willing to talk about it openly. Dario basically confirms that Claude Code is their straight shot at RSI to get to superintelligence as fast as possible (starting RSI in 2026-2027). Notably, many AI labs do not explicitly target this or at least don’t say this openly. While I think it is nice that Anthropic is doing alignment research and think that openly publishing their constitution is a good step, I think if they are successfully kicking off the RSI loop they have very low odds of succeeding.
@Zac Hatfield-Dodds @evhub @Dave Orr @Ethan Perez @Carson Denison @Drake Thomas @gasteigerjo @Aram Ebtekar Can you comment on this? Is that what they are planning to work on? Were you aware of this? Do you think that's a good thing to do?
8
(Epistemic status: third hand / rumor.) I heard that this is spoken about openly in Amodei's ~weekly all-hands, and this has been the case for a ~year.
2
Just listened to the talk and I'm confused about what people think Dario said. The only mention I noticed of RSI at the end was when the moderator said: "Last question. What will have changed in a year."
Dario said, paraphrasing, that the most important thing to watch this year is AIs building AIs, which could lead to wonders or a great emergency in front of us if it turns out that works and causes a meaningful speedup.
Is that what you're talking about? I'm pretty sure I agree that this is one of the most important things to pay attention to in the next year.
9
Would you please listen for 2 minutes to the beginning of the interview around the 1 minute mark, here:
https://www.youtube.com/watch?v=02YLwsCKUww&t=59s
Over the interview, it's clear that Dario's stated timelines are significantly more confidently short than Demis's; e.g. see here https://youtu.be/02YLwsCKUww?t=1030
Then, if you listen here for a couple minutes:
https://youtu.be/02YLwsCKUww?t=1326
it's clear that Dario's stated reason for NOT slowing down HIS research, seemingly refering to the short-timelines research, i.e. the RSI research, is something about China.
1
There's a relevant section near the beginning, when the moderator asks him about his timeline: https://youtu.be/9Zz2KrBDXUo?si=9kHfNC_ec-2yTk8e&t=61
2
Does Dario's new essay make you feel better, or worse?
1
Did you mean to ask me, rather than someone else in this thread? (I had a quick look at the new essay but haven't read it in full yet; what I did see didn't seem very surprising, so I don't really feel better or worse at this stage. Happy to say more once I've read it properly, but I think you might have meant to ask someone else, as I wasn't really taking a position in this thread -- just pointing out a relevant section of the talk that I thought you might have missed.)
2
I've seen this message, but per our confidentiality policy, no comment. (and don't take the below as indicative either way, glomar response etc)
I will note though that Dario did not actually talk about recursive self-improvement, nor about superintelligence; commentary on Lesswrong often assumes a shared ontology that just doesn't exist.
I also continue to endorse the confidentiality policy, and per my red lines I still trust Anthropic's leadership and think the company is on net good for the world.
2
Can you, or have you elsewhere, elaborate on the presumed shared ontology that doesn’t exist? I think I’d expect you to have a pretty good sense of the delta, and it could be important for learning how to talk to each other more effectively.
1
Why you think Anthropic has low odds of succeeding?
6
I think getting RSI and a shot at superintelligence right just appears very difficult to me. I appreciate their constitution and found the parts i read thoughtful. But I don't see them having found a way to reliably get the model to truly internalize it's soul document. I also assume if they were able there would be parts that break down once you get to really critical amounts of intelligence.
2
By "succeeding" you mean getting Safe ASI, as opposed to getting any ASI at all, right? At least that's how I read you, but at first I thought you meant "their RSI probably won't lead to ASI"
8
Their RSI very likely won't lead to safe ASI. That's what I meant, hope that clears it up. Whether it leads to ASI is a separate question.
1[comment deleted]
2
Whose original reasons?
Dario's. I am a bit confused what Dario was always planning to do, but early Anthropic definitely centrally recruited with the pitch of reducing AI x-risk.
2
Have you written up somewhere what you've taken away from the Dario episode regarding "our" credulity? Would be curious to hear. (I could try to generate more specific questions/prompts if you like.)
People: “Ah, yes. We should trust OpenAI with AGI.” OpenAI: https://www.nytimes.com/2024/07/04/technology/openai-hack.html “But the executives decided not to share the news publicly because no information about customers or partners had been stolen, the two people said. The executives did not consider the incident a threat to national security because they believed the hacker was a private individual with no known ties to a foreign government. The company did not inform the F.B.I. or anyone else in law enforcement.”
4
Non-paywall link: http://web.archive.org/web/20240709012837/https://www.nytimes.com/2024/07/04/technology/openai-hack.html
1
Good call. Thanks!
Confidence level: strongly held, mostly opinionated, based on observation of (imo) bad LW norms.
We should stop using the phrase “epistemic status” and start using “confidence level.” In principle, “epistemic status” is meant to convey richer meta-information than confidence alone (i.e. the kind of evidence or how seriously a claim should be taken). In practice, it almost never does—on LW it’s usually just a clunkier way of saying “x confidence.”
If we actually want to convey more with less, we should just say “confidence level” and briefly qualify it with the relevant epistemic details (I.e. “low confidence, based on analogy,” or “high confidence, but mostly theoretical”). That’s clearer, less in-group-y, and lower friction. I think this is a good way to save up some weirdness points.
(Alternatively, one can used “qualified confidence” - a bit more jargony but traded for a bit more accuracy, though I perosonally like confidence level most).
1
1
1Words have meanings. Confidence level is about my conclusion; epistemic status is about how I got there. These are different things, and the terms should be used accordingly.
I find epistemic status far more informative than confidence level. What can I do with someone’s “70%”? Nothing.
A bunch of examples of it being used:
Epistemic Status: My best guess (but, epistemic effort was "talked to like 2-3 people about it and it felt good to each of us")
—Source
Epistemic status: Exploratory, speculative, half-baked thought
—Source
Epistemic Status: I've really spent some time wrestling with this one. I am highly confident in most of what I say. However, this differs from section to section. I'll put more specific epistemic statuses at the end of each section.
—Source
Epistemic status - statistically verified.
—Source
And of course, my favorite of all time:
Epistemic Status: Eliezer Yudkowsky writing the sequences. They sentenced me to twenty years of boredom. Galileo. This army. Chris Christie to Marco Rubio at the debate. OF COURSE! A woman scorned. For great justice. The Fire of a Thousand Suns. Expelling the moneylenders from the Temple. My Name is Susan Ivanova and/or Inigo Montoyo. You killed my father. Prepare to die. Indeed. It’s a trap. Tomfidence. I swear on my honor. End this. I know Kung Fu. Buckle up, Rupert. May the Gods strike me down to Bayes Hell. Compass Rose. A Lannister paying his debts. The line must be drawn here. This far, no farther. They m...
Could someone point to an example of "epistemic status" used correctly, where you couldn't just substitute it with "confidence level"?
5
I generally don't care much about people's confidence levels. I don't Aumann agree that hard. But I do care how much effort someone has put in, how settled an idea is, whether is has been helpful or predictive. "Epistemic status: personal experience" is directly useful to me. I'll judge probability on merits however confident someone is (maybe not if I know their calibration curves, but I don't), but if I know what effort they did and didn't put in, I'll happily directly update on that. I don't think it's factually true that epistemic status 'almost never' conveys something other than a confidence level.
Epistemic status: did a few minutes informal searching to sanity check my claims, which were otherwise off the cuff.
5
Yeah but I don't think OP meant that by using "confidence level" you have to give a percentage. You can just swap out the phrase. Your two examples:
Confidence level: personal experience
Confidence level: did a few minutes informal searching to sanity check my claims, which were otherwise off the cuff.
I think these still work perfectly well, and now they are understandable to a much larger set of people.
1
The first does not work at all. “Personal experience” is not a level of confidence, and does not imply any particular level of confidence.
4
I've seen it used as "here's how rigorous my methodology was", and that seems meaningful. I can be very confident in something based on vibes alone[1], or I can run a series of intensive statistical tests and still be somewhat shaky about the apparent conclusion[2].
1. ^
"That guy in the black T-shirt looks mean, I bet he could win a fight against that guy in the leather jacket."
2. ^
"I've trained a random forest classifier on several thousand amateur boxing matches, and got 95 percent accuracy on my validation set after making sure there was no cross contamination. The resulting model says that the fellow with the big glasses is likely to win a bout against the guy in the tank top, but I still wouldn't put money on it."
2
Thinking about this more. How about "epistemic humility note" or some such thing. Why that: I think saying "humility" is good self-prompting and reader-signaling. Seems similar purpose to "epistemic status" but leaves less room for accidentally dodging that purpose, though of course some room remains. Also similar but maybe stronger: maybe gives you more of a pass to be wrong in the reader's mind (they don't think you think you're right). Also similar but maybe stronger: reminding people who are inclined to believe impressively collected results that those results can still be wrong, misleading, barking up the wrong tree, etc.
[...]
[...]
[...]
[...]
[...]
that last one reads as a warning sign to me, but seems easier to write than "Epistemic status: {same sentence}".
2
This post is very high epistemic status. Please show it the epistemic respect it epistemicologically deserves.
Lots of people in the Bay seem to be thinking about/preparing for/making funding decisions based on the idea that lots of philanthropy will be given to AIS/EA cause areas very soon (i.e. end of year-ish). I would love for someone to write the comprehensive steel man case against this, as I think it’s probably underrated (some reasons to think they won’t give the money/it won’t be as much as some assume. Happy to comment/ speak to whoever is interested in doing this.
2Marcus Abramovitch (who is active on EA Forum) often talks about the bearish case and might be a good candidate to speak with. I believe his position is sth like "I'll believe it when I see it, but from an outside view, people love saying they'll donate but rarely follow through".
Some other considerations for why money might be slower or lower:
- A lot of the money might go to funds (eg Longview or CG), which then don't deploy them quickly to projects, perhaps because lack of grantmaker capacity, or lack of trust, or general lack of incentive to deploy funds fast (in contrast to venture capital, where funders are competing to invest at lower valuations)
- The individual donors (esp Anthropic employees) might just be way too busy to evaluate where to give to
- The funds might not be spent on specific cause areas that one is expecting (eg maybe more GHD or AW than AIS. maybe not the field of AIS you were excited for)
On balance, I do think a lot of philanthropic giving will happen soon. But I think it would be great to have large liquid prediction markets or perp swaps on this kind of thing, so people and especially charities can hedge against lack of funding, or borrow against future funding.
1I agree. This is a major open question.
From experience, by default people say they donate but then mostly don't. Even if they donate the money to some foundation it often doesn't get deployed [ eg Future of Life institute has half a billion in assets since an Ethereum donation and has deployed almost nothing?].
Dustin & Cari, Jaan are the exception.
5
I disagree with "end of year-ish." I expect we'll really feel the effects of Anthropic founder/staff/investor philanthropy (but less in some ways than FTX) when those people are allowed to sell their shares.[1] But my median for that is mid-2027, and there might be delays to donating.
1. ^
I think about 8% of current Anthropic shares will ultimately be available for AI safety philanthropy. Anthropic is currently valued at $1T and will probably be valued at substantially more in a year. So that's present-value $80B and more in the future, whereas total AI safety philanthropy is <$2B/year.
1
8% is a lot higher than I was expecting. Do we have any good estimates of how much of Anthropic is owned by employees? NYT says employees and non-Microsoft investors own 47% of OpenAI (with MSFT owning 27% and the OAI Foundation owning 26%), but I don't know how Ant would compare.
1
I don't see why the funding situation will be any different at end of year. Everybody who works for e.g. Ant & OpenAI will have the same reasons to not sell their equity that they did at the beginning of the year. It's not like Ant researchers are having a hard time finding buyers.
5
No, OpenAI and Anthropic staff currently generally cannot sell their equity. This will change after they go public. Some staff would donate more—not to mention diversify—if they could.
1
An IPO realistically will unlock far more shares than any tender offer or secondary sale ever did.
US AISI will be 'gutted,' Axios reports: https://t.co/blQY9fGL1v. This should have been expected, I think, but it still seems worth sharing,
Someone I trust on this says:
AFAICT what's going on here is just that AISI and CHIPS are getting hit especially hard by the decision to fire probationary staff across USG, since they're new and therefore have lots of probationary staff - it's not an indication (yet) that either office is being targeted to be killed
2
To be frank though, that basically guts any chance of institutional capability, so it's probably going to be at best a hollow organization.
You do not randomly fire large swathes of people and expect most institutional knowledge to survive, because there's no reason for anyone to work there rather than be in the private sector.
7
True, but that's a different problem than them specifically targeting the AISI (which, based on Vance's comments, wouldn't be too surprising). Accidentally targeting the AISI means it's an easier decision to revert than if the government actively wanted to shut down AISI-like efforts.
I think people should know that this exists (Sam Harris arguing for misaligned AI being an x-risk concern on Big Think YouTube channel):
5
Thanks. For people that aren't likely to watch, I imagine it might also be worth saying he reports his view as being that we're in an arms race we can't opt out of (and that he's changed his mind regarding -- I think -- the overall appropriateness of such a race, though from what to what I'm not sure) due to insufficient political sanity and part of what constitutes sanity, he says, would be the US and China being able to create a climate whereby we don't fear each other, though it's not totally obvious whether he thinks a sufficient condition for the race to ASI being abandoned would be if people ended up no longer predicting their rivals might develop it first or whether he thinks some other fears need to be allayed as well (I get some sense that it's very much the latter, but it definitely wasn't clear).
0
Makes sense. Good clarification!
3
I'm no expert on Albanian politics, but I think it's pretty obvious this is just a gimmick with minimal broader significance.
This might feel obvious, but I think it's under-appreciated how much disagreement on AI progress just comes down to priors (in a pretty specific way) rather than object-level reasoning.
I was recently arguing the case for shorter timelines to a friend who leans longer. We kept disagreeing on a surprising number of object-level claims, which was weird because we usually agree more on the kinda stuff we were arguing about.
Then I basically realized what I think was going on: she had a pretty strong prior against what I was saying, and that prior is abstract en...
7
This'd be easier to engage with with real examples.
I think this is sort-of right, but I think "prior" is not a clear enough mechanistic description of what's going on. Like this is a fine abstraction but it doesn't give me many details to think through towards solutions.
Also, like, pretty much any human knowledge isn't intrinsically "prior", you believe things because of your life experience.
One example that you might classify as a prior but I think we can do better: some people have the intuitive sense that a significantly-more-intelligent person can stomp all over less intelligent people.
I think many of these people get the sense from having, say, played a bunch of competitive strategic games, and have a sense of how high skill-ceilings are.
Some people don't have that experience, instead their reason for the prior is just "it does sure looks like humans stomp all over chimpanzees", and generalize.
On the other side, some people have life experience where messy finnicky details got in the way of supposedly "intelligent" plans, and they have a stronger salient prior about intelligence being bogged now in finnicky details.
In all those cases I think you can do better than stop at "well their prior is different."
Or, rather: being enable to do that is a skill issue.
(See also: Propagating Facts into Aesthetics, which goes into one particular mechanism in more detail)
The Fallacy of Similar Magnitudes
A name I came up with for a pattern that comes up often enough to deserve one. Here’s a particular instance:
Suppose you're a longtermist who thinks the vast majority of the (net positive) moral weight is in the far future, that AI existential risk is above 1%, and that it's reasonably tractable to reduce. Some people who hold all three premises still say things like: "AI safety has maybe two orders of magnitude more resources than animal welfare, so the marginal dollar is better spent on animal welfare."
This is the fallacy...
Experts currently treat being persuaded as reasonably good evidence that something is true — their judgment is calibrated enough that when they find an argument convincing, that's correlated with the argument actually being correct. This allows them to update readily in light of new evidence, and is a big part of how intellectual progress happens: lots of innovation and advances in basically every subject come down to experts taking sometimes weird new ideas seriously.
One worry I have about superpersuasive AI is that it could erode this. If a superpersuasi...
2
This is one of the least possible concerns arising from superpersuasive AIs. It assumes that experts exposed to superpersuasive AIs still get to choose whether to believe what it says, and considers only higher-order epistemic harms instead of direct first-order harms like persuading people to kill each other and/or themselves.
Tyler Cowen often has really good takes (even some good stuff against AI as an x-risk!), but this was not one of them: https://marginalrevolution.com/marginalrevolution/2024/10/a-funny-feature-of-the-ai-doomster-argument.html
Title: A funny feature of the AI doomster argument
If you ask them whether they are short the market, many will say there is no way to short the apocalypse. But of course you can benefit from pending signs of deterioration in advance. At the very least, you can short some markets, or go long volatility, and then send those p...
3
Yes, that was a pretty terrible take. Markets quite clearly do not price externalities well, and never have done. So long as any given investor rates their specific investment as being unlikely to tip the balance into doom, they get the upside of directly financially benefiting from major economic growth due to AI, and essentially the same downside risk as if they didn't invest. Arguments like "short some markets, or go long volatility, and then send those profits to Somalia to mitigate suffering for a few years before the whole world ends" are obviously not even trying to seriously reflect the widespread investment decisions that affect real markets.
I saw this good talk on the Manifest youtube channel about using historical circumstances to calibrate predictions - this seems better for training than regular forecasting because you have faster feedback loop between the prediction and the resolution.
I wanted to know if anyone had recommendations on where to find some software or site where I can do more examples of this (I already know about the estimation game). I would do this myself, but it seems like it would be pretty difficult to do the research on the situation without learning the outcome. I would also appreciate people giving takes about why this might be a bad way to get better at forecasting.
A software intelligence explosion might be asymmetrically good for safety if you think safety research is absolute > relative:
Ajeya Cotra and others have argued that ML research being automated first could mean that safety work gets a boost during a software intelligence explosion. The typical worry is that this doesn't help if capabilities race ahead. But I think the picture is more nuanced depending on how you model safety progress — and the distinction matters not just for intelligence explosion dynamics but for how you allocate resources between sa...
This is a really good debate on AI doom -- I thought the optimistic side was a good model that I (and maybe others) should spend more time thinking about (mostly about the mechanistic explanation vs extrapolation of trends and induction vs empiricist framings), even though I think I disagreed with a lot of it on an object level:
https://marginalrevolution.com/marginalrevolution/2024/11/austrian-economics-and-ai-scaling.html
A good short post by Tyler Cowen on anti-AI Doomerism.
I recommend taking a minute to steelman the position before you decide to upvote or downvote this. Even if you disagree with the position object level, there is still value to knowing the models where you may be most mistaken.