Benito's Shortform Feed — LessWrong

Benito's Shortform Feed

27th Jun 2018

1 min read

23 Ω 5

This is a special post for quick takes by Ben Pace. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Benito's Shortform Feed

2the gears to ascension

4the gears to ascension

4Gordon Seidoh Worley

1Quadratic Reciprocity

333 comments, sorted by

top scoring

Click to highlight new comments since: Today at 1:26 PM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]Ben Pace2mo583

Announcing new reacts!

The reacts palette has 17 new reacts:

Gone are 11 old reacts who were getting little use, are redundant with the new reacts, and/or in my opinion weren't being used well (for example, "I'd bet this is false" almost never turned into a bet as I'd hoped, but was primarily used to disagree with added emphasis).

Reacts are a rich way to interact with a body of text, allowing many people to give information about comments, claims, and subclaims more efficiently than writing full comments, and they're something I love that we have on LessWrong (kudos to Raemon for building them initially!). They're also a way that people learn our culture – I will re-use Jimrandomh's classic meme on this topic.

I encourage you to look out for opportunities to use theses new reacts in the coming weeks, to give them a shot at life! We shall see which of these reacts get good use, and whether the use is good for truthseeking. I expect some of them will not, and will later be replaced by even newer reacts.

Feedback on the new reacts is invited, as are proposals for other reacts potentially worth adding.

P.S. There is also a stub of a wiki page on reacts, I would love it if people filled it o... (read more)

[-]Eric Neyman2mo161

I love the "I beseech you" react picture! (For those who are confused: link)

[-]Ben Pace2mo114

I am kind of interested in limiting the use of that react to once-per-month, so that it really means something when it gets used. Not sure whether to actually try this.

51a3orn2mo

I like the idea of limited-use actions in forums / social media in general, seems like an unexplored space: limited-used strong reacts, limited-use emotes, etc.

2Rana Dexsin2mo

Everything2 did this with votes. I think the “votes per day” limit used to be more progressive by user level, but it's possible that's me misremembering; looking at it now, it seems like they have a flat 50/day once you reach the level where voting is unlocked at all. Here's what looks like their current voting/experience system doc. (Note that E2 has been kind of unstable for me, so if you get a server error, try again after a few minutes.)

8Haiku2mo

I'm very happy with most of these choices. I'm confused about "locally valid" and "locally invalid" being removed, though. It would feel weird to me to use "weak argument" and "strong argument" in their stead, since weak and strong connote doing different amounts of work, while valid and invalid denote logical applicability.

5Ben Pace2mo

For one, it's very rare to need to point out that an argument is "locally valid", so the reaction is one of the least used reacts we have. For two, I think "locally invalid" is actually often not very helpful in a good discussion? Most arguments people make aren't deductively sound, they're probabilistic, and those who disagree are disagreeing with either the strength of evidence, or (rarer) the direction that evidence points. It's not that you made a mistake of predicate logic, it's that I think your evidence doesn't really support your conclusion, and that's a more substantive disagreement. It seems to me that the relevant thing is often that the argument seems weak to the reader, rather than the author's arguments not making sense. I am still sad about losing them as part of culture setting, but overall I think they were either not worth the space or not used well, and we still have plenty of good LessWrong culture setting in the reacts palette.

4Vladimir_Nesov2mo

"Locally invalid" was a specific react for highlighting the part of a comment that makes a self-contained mistake, different from "Disagree". A faulty step is not centrally a "weak argument", as it's sometimes not any kind of argument. And discussion often gestures at a claim without providing any sort of evidence or giving any argument, the evidence or the argument is for the recipients to reconstruct for themselves.

4Ben Pace2mo

I don’t have examples easily to hand, but my impression (sadly) was that it was too often misused, for when someone disagreed with an argument, to mark it as locally invalid rather than simply disagreeing. Perhaps if people learn to use “weak argument” instead in those situations, I might add it back in later; or perhaps I can find a secret third react which better gets them what they are looking for (e.g. “I don’t think that this follows” or “I reject this step of the argument”) that they would correctly use even if “locally invalid” is added back in.

3gustaf2mo

Thanks for the wiki page stub. I turned it into a full table (See LessWrong Reacts) I think this presents a nice opportunity of explaining LessWrong's culture to new users. Have you considered linking to the wiki page from the hover-menu of a reaction? I think the most confusing one, even after hovering over the reaction is "I beseech you"

2Ben Pace2mo

Awesome, thanks so much gustaf! I've re-arranged the order a little bit to have some of the more common ones first. Yeah, I have been thinking about linking it from the reacts menu. Will think about that next time I ship some changes. (I'll be more inclined to do that if there are more explanations and links and culture and stuff.)

1gustaf2mo

Having some common ones first makes sense for illustrating the feature! I arranged them based on which reacts would probably be the most confusing.

2Thomas Kwa2mo

Big fan of the "bowing out" react. I did notice a minor UI issue where the voting arrows don't fit in the box:

2Ben Pace2mo

Thx – yeah seems that something broke when I shipped it, I should ship a fix in the next day or two.

2Garrett Baker2mo

On the "I bet this is false" react, a wild & likely unworkable idea (which I will say anyway) would be to hook that react up to a manifold market for which the reacter must bet "no" in.

2Ben Pace2mo

Oops, I forgot that I actually added 18 reacts! We now also have "Sad".

2Kaj_Sotala2mo

"Too Combative" and "Too Sneering" seem really similar to me? Sad to see "Insightful" go, it might not have been used that much, but that was part of the reason why it felt valuable. If you saw that react, then probably the person who'd used it thought that there was something exceptionally insightful about the comment.

3Ben Pace2mo

I think the light bulb icon is the correct icon for "Good point!". If I can find another good image for insightful I'll probably add it back

2plex2mo

I also liked Insightful, though agree with all other changes.

2Rana Dexsin2mo

I think I semi-agree with your perception, but I did have a recent experience to the contrary: when I did a throwaway post about suddenly noticing a distinction in the reaction UI, I found it very odd that some people marked the central bit “Insightful”. Like, maybe it's useful to them that I pointed it out, but it's a piece of UI that was (presumably) specifically designed that way by someone already! There's no new synthesis going on or anything; it's not insightful. (Or maybe people wanted to use it as a test of the UI element, but then why not the paperclip or the saw-that eyes?)

2cubefox2mo

Doesn't "not worth getting into?" sound better than "bowing out"?

8Ben Pace2mo

I think I regularly want to say goodbye at the end of a thread, but I don't think I regularly want to tell people that the thread is no longer a good use of my time. To some extent they're logically similar but I think the choice to leave a dinner saying "Marginal time at this dinner is no longer worth the opportunity cost for me" is kind of silly, and saying "I just wanted to say goodbye, I'm headed out now!" is standard and normal.

2cubefox2mo

Oh, apparently I misinterpreted the meaning of "bowing out", as I'm not a native English speaker. Anyway, I just want to register my opinion that I thought the "not worth getting into?" react was good in itself.

2the gears to ascension2mo

yay!! I think "nitpick" would generate better common knowledge for people as to its meaning if the hover trivial inconvenience was removed, by renaming it to "phrase is nitpick", so as to avoid a usage forming that interprets it as "reacter nitpicks phrase" I want elaborate back, its purpose seems unreplaced

2Ben Pace2mo

It's use was in the bottom third of reacts, and my honest recollection is that it never actually caused elaborating to happen, so I felt comfortable removing it. I have been exploring design space for a react that is more like "I would be interested to read more about this from you!" which feels a bit less demanding and more like something good will happen to the person if they write another comment or even a post on it, which I would like to add if I can find a good design for it (both a good visual image and a good short name). Interesting, I hadn't conceived that "nitpick" might be used that way. I don't know if changing it to "This is a nitpick" would actually solve the problem (perhaps it would be read self-referentially). Perhaps I'll think of a better idea tomorrow, but if not I'll wait to see if it actually gets misused. Epistemic status: Very sleepy while I write this.

2Mateusz Bagiński2mo

I think "Elaborate" would be more useful (i.e., more likely to actually induce an elaboration on the point being reacted at) if its corresponding notification was grouped not with karma updates and other reacts (which people get notified about in a batch, daily, weekly, etc.), but rather with the normal notifs like "so-and-so replied to your comment/post" or "you received a message from so-and-so". (Probably the same for the new "Let's make a bet!" react.) But this would most likely require doing something somewhat inelegant and complicated to the codebase, so it may not be worth it, atm at least.

2Ben Pace2mo

It might also work better if it’s use was rate limited, so that it was actually a costly signal.

2Mateusz Bagiński2mo

I guess. But I would think the bigger issue is that people don't notice.

2niplav2mo

I also feel a bit sad about "Elaborate?" being gone, sometimes someone makes an interesting side-statement and I want to hear more from them about that, which is when I use the react. But this is a weak expression of longing.

[-]Ben Pace2y*3818

I often wish I had a better way to concisely communicate "X is a hypothesis I am tracking in my hypothesis space". I don't simply mean that X is logically possible, and I don't mean I assign even 1-10% probability to X, I just mean that as a bounded agent I can only track a handful of hypotheses and I am choosing to actively track this one.

This comes up when a substantially different hypothesis is worth tracking but I've seen no evidence for it. There's a common sentence like "The plumber says it's fixed, though he might be wrong" where I don't want to communicate that I've got much reason to believe he might be wrong, and I'm not giving it even 10% or 20%, but I still think it's worth tracking, because strong evidence is common and the importance is high.
This comes up in adversarial situations when it's possible that there's an adversarial process selecting on my observations. In such situations I want to say "I think it's worth tracking the hypothesis that the politician wants me to believe that this policy worked in order to pad their reputation, and I will put some effort into checking for evidence of that, but to be clear I haven't seen any positive evidence for that hypothesi

... (read more)

6Chris_Leong2y

Maybe just say that you're tracking the possibility?

4Richard_Kennaway2y

"Trust, but verify."

4Dagon2y

Standard text in customer-facing outage recovery notices: all systems appear to be operating correctly, and we are actively monitoring the situation". In more casual conversations, I sometimes say "cautiously optimistic" when stating that I think things are OK, but I'm paying more attention than normal for signs I'm wrong. Mostly, I talk about my attention and what I'm looking for, rather than specifying the person who's making claims. Instead of "the plumber says it's fixed, though he might be wrong", I'd say "The plumber fixed it, but I'm keeping an eye out for further problems". For someone proposing something I haven't thought about, "I haven't noticed that, but I'll pay more attention for X and Y in the future".

3jam_brand2y

Before I read the aphoristic three-word reply to you from Richard Kennaway (admittedly a likely even clearer-cut way to indicate the following sentiment), I was thinking that to downplay any unintended implications about the magnitude of your probabilities that you could maybe say something about your tracking being for mundane-vigilance or intermittent-map-maintenance or routine-reality-syncing / -surveying / -sampling reasons. For any audience you anticipate familiarity with this essay though, another idea might be to use a version of something like: "The plumber says it's fixed, which I'm splitting on [by default][and {also} tracking <for posterity>]." (spoilered section below just corrals a ~dozen expansions / embellishments of the above)

3Valdes2y

Adapted from the french "j'envisage que X" I propose "I am considering the possibility that X" or in some contexts "I am considering X". "The plumber says it's fixed, but I am considering he might be wrong".

2Richard_Kennaway2y

What's wrong with your original sentence, "X is a hypothesis I am tracking in my hypothesis space"? Or more informal versions of that, like "I'll be keeping an eye on that", "We'll see", etc.?

2Ben Pace2y

I guess it's just that I don't feel mastery over my communication here, I still anticipate that I will find it clunky to add in a whole chunk of sentences to communicate my epistemic status. I anticipate often in the future that I'll feel a need to write a whole paragraph, say in the political case, just to clarify that though I think it's worth considering the possibility that the politician is somehow manipulating the evidence, I've seen no cause to believe it in this case. I feel like bringing up the hypothesis with a quick "though I'm tracking the possibility that Adam is somehow manipulating the evidence for political gain" pretty commonly implies that the speaker (me) thinks it is likely enough to be worth acting on, and so I feel I have to explicitly rule that out as why I'm bringing it up, leaving me with my rather long sentence from above.

2jmh2y

In the plumbing context I generally say or think, "The repair/work has been completed and I'll see how it lasts." or sometimes something like, "We've addressed the immediate problem so lets see if that was a fix or a bandage."

2Steven Byrnes2y

“The plumber says it’s fixed, but I’ll keep an eye out for evidence of more problems.” (ditto Dagon) also “The politician seems to be providing sound evidence that her policy is working, but I’ll remain vigilant to the possibility that she’s being deceptive.”

1mattmacdermott2y

"Bear in mind he could be wrong" works well for telling somebody else to track a hypothesis. "I'm bearing in mind he could be wrong" is slightly clunkier but works ok.

1Mateusz Bagiński2y

"The hypothesis/possibility that 'X' is mindworthy" ("worth being mindful about it"). Maybe the nicest solution would be to coin a one-syllable modal verb like "may" or "can" to communicate exactly this.

2Ben Pace2y

"Keep in mind that X".

1FlorianH2y

Maybe "I'm interested in the hypothesis/possibility..."

1NineDimensions2y

In some cases something like this might work: "The plumber says it's fixed, so hopefully it is" Or "The plumber says it's fixed, so it probably is" Which I think conveys"there's an assumption I'm making here, but I'm just putting a flag in the ground to return to if things don't play out as expected"

-8Shankar Sivarajan2y

[-]Ben Pace10mo36-5

So many people have lived such grand lives. I have certainly lived a greater life than I expected, filled with adventures and curious people. But people will soon not live any lives at all. I believe that we will soon build intelligences more powerful than us who will disempower and kill us all. I will see no children of mine grow to adulthood. No people will walk through mountains and trees. No conscious mind will discover any new laws of physics. My mother will not write all of the novels she wants to write. The greatest films that will be made have probably been made. I have not often viscerally reflected on how much love and excitement I have for all the things I could do in the future, so I didn't viscerally feeling the loss. But now, when it is all lost, I start to think on it. And I just want to weep. I want to scream and smash things. Then I just want to sit quietly and watch the sun set, with people I love.

[-]testingthewaters10mo2925

Do not go gentle into that good night,

Old age should burn and rave at close of day;

Rage, rage against the dying of the light.

Though wise men at their end know dark is right,

Because their words had forked no lightning they

Do not go gentle into that good night.

Good men, the last wave by, crying how bright

Their frail deeds might have danced in a green bay,

Rage, rage against the dying of the light.

Wild men who caught and sang the sun in flight,

And learn, too late, they grieved it on its way,

Do not go gentle into that good night.

Grave men, near death, who see with blinding sight

Blind eyes could blaze like meteors and be gay,

Rage, rage against the dying of the light.

And you, my father, there on the sad height,

Curse, bless, me now with your fierce tears, I pray.

Do not go gentle into that good night.

Rage, rage against the dying of the light.

Do not go gentle into that good night, Dylan Thomas

I'm still fighting. I hope you can find the strength to too.

[-]niplav10mo138

Because their words had forked no lightning they

I think we have the opposite problem: our words are about to fork all the lightning.

[-]Ben Pace10mo131

Thank you.

It does not currently look to me like we will win this war, speaking figuratively. But regardless, I still have many opportunities to bring truth, courage, justice, honor, love, playfulness, and other virtues into the world, and I am a person whose motivations run more on living out virtues rather than moving toward concrete hopes. I will still be here building things I love, like LessWrong and Lighthaven, until the end.

6Nathan Helm-Burger10mo

Though I have worries, and short timelines, so too do I have hope. I believe the next two years will be pivotal, and that we have important roles to play. Let us hold firm in the face of great danger.

-16M. Y. Zuo10mo

5Quinn10mo

yeah last week was grim for a lot of people with r1's implications for proliferation and the stargate fanfare after inauguration. Had a palpable sensation of it pivoting from midgame to endgame, but I would doubt that sensation is reliable or calibrated.

2Ben Pace10mo

My feelings here aren't at all related to any news or current events. I could've written this any time in the last year or two.

5eigen10mo

I disagree extremely. This is the best moment of my life. I am at the best point of my career (powered by o1 and o3 research previews is allowing me to reach the best solutions I couldn't imagine reaching on my own, much less in such short time) it has helped me create two companies now completely different from my career (I have optimized hydroponic setups and cultivation of mushrooms purely with o1-pro to incredible levels.) My father, a doctor, tells me his patients are better than ever only because his use of o1, doctors are using it in their meetings with their most difficult to diagnose patients. My girlfriends uses it for mental health. I could continue. I feel so empowered. I have read too much of alignment to think that we are going to make it. It's up to you -really- to choose to feel empowered or down about it. I'm honestly having the best time of my life.

3stavros10mo

What is true is already so / It all adds up to normality What you've lost isn't the future, it's the fantasy. What remains is a game that we were born losing, where there may be few moves left to make, and where most of us most of the time don't even have a seat at the table. However, it is a game with very high variance. It is a game where world shaping things happen regularly due to one person getting lucky (right person, right place, right time, right idea etc). And one thing I've noticed in people who routinely excel at high variance games - e.g. Poker, MTG - is how unaffected they are when they're down/behind. There is a mindset, in the moment, not of playing to win... but of playing optimally - of making the best move they can in any situation, of playing to maximize their outs no matter how unlikely they may be. To those for whom the OP's message strongly resonates: let it. Feel it. Give your grief and fear, sorrow and anger their due. Practice self-care; be kind and compassionate to yourself as you would to another who felt what you are feeling. One morning you will wake up feeling okay, and you'll realize you've felt okay more often than not lately. Then, should this game still appeal to you, it is time to start playing again :)

7Noosphere8910mo

This point would be really helpful for everyone. That said, I'd dispute this claim here: At least under the common conception of fantasy, this is an extremely strong claim, because you are effectively claiming that the good future in Ben Pace's head could never have been realized, and I see no reason to conclude this from an epistemic perspective at all, unless you are masssively overconfident (even if you do have reasonably high doom probabilities, this statement is not true.) More generally, it's known that it does not always add up to normality, see here: https://www.lesswrong.com/posts/74crqQnH8v9JtJcda/egan-s-theorem#oZNLtNAazf3E5bN6X

1tailcalled10mo

Is your mother currently spending a lot of her time writing novels?

4Ben Pace10mo

Yes; she has come to visit me for two months, and I have helped her get into a daily writing routine while she's here. I know she has the ability to finish at least one.

[-]Ben Pace8mo*351

I occasionally get texts from journalists asking to interview me about things around the aspiring rationalist scene. A few notes on my thinking and protocols for this:

I generally think it is pro-social to share information with serious journalists on topics of clear public interest.
By-default I speak with them only if their work seems relatively high-integrity. I like journalists whose writing is (a) factually accurate, (b) boring, and (c) do not feel to me to have an undercurrent of hatred for their subjects.
By default I speak with them off-the-record, and then offer to send them write-ups of the things I said that they want to quote. This has gone quite well. I've felt comfortable speaking in my usual fashion without worrying about nailing each and every phrasing. Then I ask what they're interested in quoting, and I send them (typically a 1-2 page) google doc on those topics (largely re-stating what I already said to them, and making some improvements / additions). Then they tell me which quotes they want to use (typically cutting many sentences or paragraphs half-way). Then I make one or two slight edits and give them explicit permission to quote. I think this has gone quite wel

... (read more)

6Yoav Ravid8mo

The quoting policy seems very good and clever :)

4Zach Stein-Perlman8mo

Yep, my impression is that it violates the journalist code to negotiate with sources for better access if you write specific things about them.

2Garrett Baker8mo

Claude says its a gray area when I ask, since this isn’t asking for the journalist to make a general change to the story or present Ben or the subject in a particular light.

2Ben Pace8mo

Update from chatting with him: he said he was a just freelancer doing a year exclusively with NYT, and he wasn’t in a position to write on behalf of the NYT on the issue (e.g. around their deanonymization policies). This wasn’t satisfying to me, and so I will keep to being off-the-record.

[-]Ben Pace6y*340

Yesterday I noticed that I had a pretty big disconnect from this: There's a very real chance that we'll all be around, business somewhat-as-usual in 30 years. I mean, in this world many things have a good chance of changing radically, but automation of optimisation will not cause any change on the level of the industrial revolution. DeepMind will just be a really cool tech company that builds great stuff. You should make plans for important research and coordination to happen in this world (and definitely not just decide to spend everything on a last-ditch effort to make everything go well in the next 10 years, only to burn up the commons and your credibility for the subsequent 20).

Only yesterday when reading Jessica's post did I notice that I wasn't thinking realistically/in-detail about it, and start doing that.

[-]Ben Pace6y100

Related hypothesis: people feel like they've wasted some period of time e.g. months, years, 'their youth', when they feel they cannot see an exciting path forward for the future. Often this is caused by people they respect (/who have more status than them) telling them they're only allowed a small few types of futures.

6lc3y

How do you feel about this today?

5Ben Pace3y

* The next 30 years seem really less likely to be 'relatively normal'. My mainline world-model is that nation states will get involved with ML in the next 10 years, and that many industries will be really changed up by ML. * One of my personal measures of psychological health is how many years ahead I feel comfortable making trade-offs for today. This changes over time, I think I feel like I'm a bit healthier now than I was when I wrote this, but still not great. Not sure how to put a number to this, I'll guess I'm maybe able to go up to 5 years at the minute (the longest ones are when I think about personal health and fitness)? Beyond that feels a bit foolish. * I still resonate a bit with what I wrote here 4 years ago, but definitely less. My guess is if I wrote this today the number I would pick would be "8-12 years" instead of "30".

2Ben Pace2y

Nation states got involved with ML faster than I expected when I wrote this!

2Ben Pace3y

Epistemic status: Thinking out loud some more. Hm, I notice I'm confused a bit about the difference between "ML will blow up as an industry" and "something happens that effects the world more than the internet and smartphones have done so far". I think honestly I have a hard time imagining ML stuff that's massively impactful but isn't, like, "automating programming", which seems very-close-to-FOOM to me. I don't think we can have AGI-complete things without being within like 2 years (or 2 days) of a FOOM. So then I get split into two worlds, one where it's "FOOM and extinction" and another world which is "a strong industry that doesn't do anything especially AGI-complete". The latter is actually fairly close to "business somewhat-as-usual", just with a lot more innovation going on, which is kind of nice (while unsettling). Like, does "automated drone warfare" count as "business-as-usual"? I think maybe it does, it's part of general innovation and growth that isn't (to me) clearly more insane than the invention of nukes was. I think I am expecting massive innovation and that ML will be shaking up the world like we've seen in the 1940's and 1950's (transistors, DNA, nukes, etc etc). I'm not sure whether to expect 10-100x more than that before FOOM. I think my gut says "probably not" but I do not trust my gut here, it hasn't lived through even the 1940's/50's, never mind other key parts of the scientific and industrial and agricultural and eukaryotic revolutions. As we see more progress over the next 4 years I expect we'll be in a better position to judge how radical the change will be before FOOM. The answer to lc's original question is then:

2eigen3y

Hey, I think you should also consider how the out-of-nowhere narrative-breaking nature of COVID. Which also happened after you wrote this. It's not necessarily a proof that the narrative can "break," but it sure is an example. And, while I think I read the sequences way longer than 4 years ago, if I remember something it gave me is a sense of "everything can change very, very fast."

[-]Ben Pace9mo334

I want to contrast two perspectives on human epistemology I've been thinking about for over a year.

There's one school of thought about how to do reasoning about the future which is about naming a bunch of variables, putting probability distributions over them, multiplying them together, and doing bayesian updates when you get new evidence. This lets you assign probabilities, and also to lots of outcomes. "What probability do I assign that the S&P goes down, and the Ukraine/Russia war continues, and I find a new romantic partner?" I'll call this the "spreadsheets" model of epistemology.

There's another perspective I've been ruminating on which is about visualizing detailed and concrete worlds, in a similar way to if you hold a ball and ask me to visualize how it'll fall if you drop it, that I can see the world in full detail. This is more about loading a full hypothesis into your head, and into your anticipations. It's more related to Privileging the Hypothesis / Believing In / Radical Probabilism^[1]. I'll call this the "cognitive visualization" model of epistemology.

These visualizations hook much more directly to my anticipations and motivations. When I am running after you to r... (read more)

4tailcalled9mo

I think the billion-dollar question is, what is the relationship between these two perspectives? For example, a simplistic approach would be to see cognitive visualization as some sort of Monte Carlo version of spreadsheet epistemology. I think that's wrong, but the correct alternative is less clear. Maybe something involving LDSL, but LDSL seems far from the whole story.

4Viliam9mo

So, one problem seems to be that humans are slow, and evaluating all options would require too much time, so you need to prune the option tree a lot. I am not sure what is the optimal strategy here; seems like all the lottery winners have focused on analyzing the happy path, but we don't know how much luck was involved at actually staying on the happy path, and what was the average outcome when they deviated from it. Another problem is that human prediction and motivation are linked in a bad way, where having a better model of the world sometimes makes you less motivated, so sometimes lying to yourself can be instrumentally useful... the problem is, you cannot figure out how much instrumentally useful exactly, because you are lying to yourself, duh. Another important piece of data would be, how many of the people who cannot imagine failure actually do succeed, and what typically happens to them when they don't. Maybe nothing serious. Maybe they often ruin their lives.

3Thane Ruthenis9mo

Hm, I'm not sure I understand what's confusing about this. First, suppose you're an approximate utility maximizer. There's a difference between optimizing the expected utility E[U(world,action)] and optimizing utility in the expected world U(E(world),action). In general, in the former case, you're not necessarily keeping the most-likely worlds in mind; you're optimizing the worlds in which you can get the most payoff. Those may be specific terrible outcomes you want to avert, or specific high-leverage worlds in which you can win big (e. g., where your startup succeeds). Choosing which worlds to keep in mind/optimize obviously impacts in which worlds you succeed. (Startup founders who start being risk-averse instead of aiming to win in the worlds in which they can win big lose – because they're no longer "looking" at the worlds where they succeed, and aren't shaping their actions to exploit their features.) Second, human world-models are hierarchical, and your probability distribution over worlds is likely multimodal. So when you pick a set of worlds you care about, you likely pick several modes of this distribution (rather than specific fully specified worlds), characterized by various high-level properties (such as "AI progress continues apace" vs. "DL runs into a wall"). When thinking about one of the high-level-constrained worlds/the neighbourhood of a mode, you further zoom-in on modes corresponding to lower-level properties, and so on. Which is why you're not keeping a bunch of basically-the-same expected trajectories in your head, but meaningfully "different" trajectories. This... all seems to be business-as-usual to me? I may be misunderstanding what you're getting at.

6Ben Pace8mo

I wrote this because I am increasingly noticing that the rules for "which worlds to keep in mind/optimize" are often quite different from "which worlds my spreadsheets say are the most likely worlds". And that this is in conflict with my heuristics which would've said "optimize the world-models in your head for being the most accurate ones – the ones that will give you the most accurate answers to most questions" rather than something like "optimize the world-models in your head for being the most useful ones". (Though the true answer is some more complicated function combining both practical utility and map-territory correspondence.) I will note that what is confusing to one person need not be confusing to another person. In my experience it is a common state of affairs for one person in a conversation to be confused and the other not (whether it be because the latter person is pre-confused, post-confused, or simply because their path to understanding a phenomena didn't pass through the state of their interlocutor). It seems probable to me that I have found this subject more confusing than have others.

1samuelshadrach9mo

(edited) This is probably obvious to you, but you can expand the working memory bottleneck by making lots of notes. You still need to store the "index" of the notes in your working memory though, to be able to get back to relevant ideas later. Making a good index includes compressing the ideas till you get the "core" insights into it. Some part of what we consider intelligence is basically search and some part of what we consider faster search is basically compression. Tbh you can also do multi-level indexing, the top-level index (crisp world model of everything) could be in working memory and it can point to indexes (crisp world model of a specific topic) actually written in your notes, which further point to more extensive notes on that topic. As an aside, automated R&D using LLMs currently heavily relies on embedding search and RAG. AI's context window is loosely analogous to human's working memory in that way. AI knows millions of ideas but it can't simulate pairwise interactions between all ideas as that would require too much GPU time. So it too needs to select some pairs or tuples of ideas (using embedding search or something similar) within which it can explore interactions. The embedding dataset is a compressed version of the source dataset and the LLM itself is an even more compressed version of the source dataset. So there is interplay between data at different levels of compression.

[-]Ben Pace1y3314

Privacy: a tool for thinking for yourself

As part of some recent experiments with debates, today I debated Ronny Fernandez on the topic of whether privacy is good or bad, and I was randomly assigned the “privacy is good” side. I’ve cut a few excerpts together that I think work as a standalone post, and put them below.

At the start I was defending privacy in general, and then we found that our main disagreement was about whether it was helpful for thinking for yourself, so I focus even more on that after the opening statement.

This is an experiment. I'm down for feedback on whether to do more of this sort of thing (it only takes me ~2 hours), how I could make it better for the reader, whether to make it a top-level post, etc.

Epistemic status: soldier mindset. I will here be exaggerating the degree to which I believe my conclusions.

Opening Statement

My core argument is that, in general, the pressures for conformity amongst humans are crazy.

This is true of your immediate circle, your local community, and globally. Each one of these has sufficiently strong pressures that I think it is a good heuristic to actively keep secrets and things you think about and facts about your life separate fr... (read more)

6Elizabeth1y

I'm surprised that the frame around Crazy Eye Travis is framed around whether his friends will pressure him or his own psychology, instead of the environmental consequences. Most opera houses are not going to let you in if your ass is hanging out. Nor will banks or bigtech jobs. Failure to conform with institutions typically results in losing access to those institutions.

4[anonymous]1y

I agree with your perspective almost entirely (for reasons basically building on top of what Zvi has written about at length before), so I would be a lot more curious to see what Ronny's argument was during the debate (if he is okay with sharing it, of course). I know you have referenced and quoted part of his reasoning, but it's a bit weird to read a rebuttal to someone's argument without first reading the argument itself, in their own words.

3abergal1y

Thanks for writing this up-- at least for myself, I think I agree with the majority of this, and it articulates some important parts of how I live my life in ways that I hadn't previously made explicit for myself.

[-]Ben Pace2y3212

Did anyone else feel that when the Anthropic Scaling Policies doc talks about "Containment Measures" it sounds a bit like an SCP, just replaced with the acronym ASL?

Item #: ASL-2-4
Object Class: Euclid, Keter, and Thaumiel
Threat Levels:
ASL-2... [does] not yet pose a risk of catastrophe, but [does] exhibit early signs of the necessary capabilities required for catastrophic harms
ASL-3... shows early signs of autonomous self-replication ability... [ASL-3] does not itself present a threat of containment breach due to autonomous self-replication, because it is both unlikely to be able to persist in the real world, and unlikely to overcome even simple security measures...
...an early guess (to be updated in later iterations of this document) is that ASL-4 will involve one or more of the following... [ASL-4 has] become the primary source of national security risk in a major area (such as cyberattacks or biological weapons), rather than just being a significant contributor. In other words, when security professionals talk about e.g. cybersecurity, they will be referring mainly to [ASL-4] assisted... attacks. A related criterion could be that deploying an ASL-4 system without safeguards

... (read more)

[-]Ben Pace1y*305

I Changed My Mind on Prison Sentencing.

I used to have the opinion that prison sentencing should be a disincentive proportional to the upside of the crime. The question I'd ask was "How much of a prison sentence would make the crime not worth it to people?" I tried to estimate the upside to the criminal, and what length of sentence would make the expected utility reliably turn out negative. I would sometimes discuss with people how long of a prison sentence they'd risk in order to get the upside of a crime, and use this as an input into how long I thought sentencing should be. However, I now think this was an error, for two reasons:

Many criminals or norm-breakers are not doing something as clear-headed as an explicit expected-value calculation, they are often thinking with part of their mind that is relatively uncivilized (and perhaps desperate or hidden from much of their conscious mind). For instance, it is done impulsively, or it is a crime of passion.
An Astral Codex Ten blogpost reiterated the common wisdom that length of sentencing has little effect on crime rate, and instead reliability of punishment has a much larger effect. The conclusion reads "Deterrence effects are so wea

... (read more)

[-]Kaj_Sotala11mo180

I recall it heard claimed that a reason why financial crimes sometimes seem to have disproportionately harsh punishments relative to violent crimes is that financial crimes are more likely to actually be the result of a cost-benefit analysis.

5Ben Pace1y

As a concrete example, I previously thought that Sam Bankman-Fried should be sentenced to 20-40 years in prison for his fraud, because this was the sort of time that I think most people would no longer be willing to trade for an even shot at getting $10Bs (e.g. when I asked my personal trainer, he said he would accept 15 years in prison for an even shot at $10B; I think many would take more, and also the true upside was higher). From the above I've updated that the diff between expecting 5 years or 50 years in prison wasn't a primary input into SBF's repeated decisions to do fraud. However, I do think he is sufficiently sociopathic that I never expect him to not be a danger to society, so my new position is that life in-prison is probably best for him. (This is not meant as punishment, I would not mind him going to those pleasant Swedish prisons I've heard about, I just otherwise expect him to continue to competently do horrendous things with zero moral compunction.)

6Lukas_Gloor1y

I like all the considerations you point out, but based on that reasoning alone, you could also argue that a con man who ran a lying scheme for 1 year and stole only like $20,000 should get life in prison -- after all, con men are pathological liars and that phenotype rarely changes all the way. And that seems too harsh? I'm in two minds about it: On the one hand, I totally see the utilitarian argument of just locking up people who "lack a conscience" forever the first time they get caught for any serious crime. On the other hand, they didn't choose how they were born, and some people without prosocial system-1 emotions do in fact learn how to become a decent citizen. It seems worth mentioning that punishments for financial crime often include measures like "person gets banned from their industry" or them getting banned from participating in all kinds of financial schemes. In reality, the rules there are probably too lax and people who got banned in finance or pharma just transition to running crypto scams or sell predatory online courses on how to be successful (lol). But in theory, I like the idea of adding things to the sentencing that make re-offending less likely. This way, you can maybe justify giving people second chances.

7Ben Pace11mo

Good point. I can imagine things like "permanent parole" (note that probation and parole are meaningfully different) or being under house arrest or having constraints on your professional responsibilities or finances or something, being far better than literal incarceration.

7Noosphere8911mo

One of the missing considerations is that crime is done mostly by young people, and the rate of crimes goes down the older you get. A lot of this IMO is that the impulsiveness/risk-taking behavior of crimes decreases a lot with age, but the empirical fact of crime going down with age, especially reoffending is a big reason why locking people up for life is less good than Ben Pace said, because the reoffending rate goes down with age.

4Ben Pace11mo

I agree there are people who do small amounts of damage to society, are caught, and do not reoffend. Then there are other people whose criminal activities will be most of their effect on society, will reliably reoffend, and for whom the incapacitation strongly works out positive in consequentialist terms. My aim would be to have some way of distinguishing between them. The amount of evidence we have about Bankman-Fried's character is quite different than that of most con men, including from childhood and from his personal diary, so I hope we can have more confidence based on that. But a different solution is to not do any psychologizing, and just judge based on reoffending. See this section from the ACX post: I should add that Scott has lots of concerns about doing this in the US, and argues that properly doing this in the US would massively increase the incarcerated population. I didn't quite follow his concerns, but I was not convinced that something like this would be a bad idea on consequentialist grounds, even if the incarcerated population were to massively increase. (Note that I would support improving the quality of prisons to being broadly as nice as outside of prisons.)

4Mark Xu11mo

This is in part the reasoning used by Judge Kaplan: from https://time.com/6961068/sam-bankman-fried-prison-sentence/

4Ben Pace1y

I want to try out the newly updated claims feature! Here are some related claims, I invite you to vote your probabilities. Prediction Prediction (This can be for whatever reason you think, such as because his expected value calculations would've changed, or because he would've taken more care around these particular behaviors, or any other reason you please.)

4Nick_Tarleton11mo

I can easily imagine an argument that: SBF would be safe to release in 25 years, or for that matter tomorrow, not because he'd be decent and law-abiding, but because no one would trust him and the only crimes he's likely to (or did) commit depend on people trusting him. I'm sure this isn't entirely true, but it does seem like being world-infamous would have to mitigate his danger quite a bit. More generally — and bringing it back closer to the OP — I feel interested in when, and to what extent, future harms by criminals or norm-breakers can be prevented just by making sure that everyone knows their track record and can decide not to trust them.

4Ben Pace11mo

I think having an easily-findable reputation makes it harder to do crimes, but being famous makes it easier. Many people are naive & gullible, or are themselves willing to do crime, and would like to work with him. I expect him to get opportunities for new ventures on leaving prison, with unsavory sorts. I definitely support track-records being more findable publicly. Of course there's some balance in that the person who publishes it has a lot of power over the person being written about, and if they exaggerate it or write it hyperbolically then they can impose a lot of inappropriate costs on the person that they're in a bad position to push back on.

2lc11mo

I voted 75% because taken literally I think in 25 years AI will be so advanced that he won't have much of an ability to impact the world at all 🤓 (Otherwise 40%)

4Nathan Helm-Burger11mo

I've been mad about the inefficiencies of the criminal justice system for many years. It does wasteful harm to perpetrators while also not doing a great job at prevention or helping victims. One possible reform I'm excited about the possibility of is having more AI monitoring of public spaces. For instance, if a woman could wear a sign saying "monitored by live camera" she might feel safer walking at night. Or a store might post a sign about AI security cameras. Another possible innovation is AI parole officers (perhaps house in a cell-service neckband with a camera), and a reformed parole system intended to be a long-term nanny rather than an excuse to jail the parolee. If you had to wear your AI nanny anytime you were out of the house, and it would scold you if it looked like you were about to commit a crime... This might substitute entirely for prison for the majority of crimes.

4jmh11mo

I agree with the view that punishment is not really a great deterrent as many crimes are not committed from a calculated cost-benefit perspective. I do think we need to apply that type of thinking towards what we might do with that insight/fact of things. On that point, would like to see more on your claim that we would get better bang for the buck as it were from more investment in preventing crimes. In this regard I'm thinking about the contrast between western legal views and places like China as well as the estimates on the marginal pecuniary costs of prevention to the marginal pecuniary savings from reduced punishment. Clearly two (among many) difference margins along which trade-offs will need to be made. Another aspect that seems worth exploring (and I'm sure it has been but not sure where the literature stands on the question) is how, at least in my understanding of USA criminal law, victims of crimes are not often compensated (white collar, fraud, financial crimes are something of an exception) but the victims, as a part of society, are then paying costs to punish the criminal. Full prevention is not a reasonable assumption (not sure what level is a reasonable assumption) but we might find a better solution even at the current rate of prevention if more of those harmed by crimes were actually compensated for the harms rather than just imposing the punishment of the criminal actor. A primary reason for preventing the event of a crime is the prevention of the harm. But if the harm can be largely mitigated after the fact there is a degree of equivalent between it never having occurred and it's compensation (perhaps here we might think of law and punishment as a type of insurance). I also think there is something to look at in terms of prevention of incidence of crimes due to incarceration -- a type of exile. There might be scope for approaches there that maintain that type of prevention for repeat offenders (those demonstrating a propensity for some bad beh

1ZY11mo

For "prison sentencing" here, do you mean some time in prison, but not life sentencing? Also instead of prison sentencing, after increasing "reliability of being caught", would you propose alternative form of sentencing? Some parts of 1) and most of 2) made me feel educating people on the clear consequences of the crime is important. For people who frequently go in and out of prison - I would guess most legal systems already make it more severe than previous offenses typically, but for small crimes they may not be. I do think other types of punishments that you have listed there (physical pain, training programs, etc) would be interesting depending on the crime.

[-]Ben Pace6y270

Hypothesis: power (status within military, government, academia, etc) is more obviously real to humans, and it takes a lot of work to build detailed, abstract models of anything other than this that feel as real. As a result people who have a basic understanding of a deep problem will consistently attempt to manoeuvre into powerful positions vaguely related to the problem, rather than directly solve the open problem. This will often get defended with "But even if we get a solution, how will we implement it?" without noticing that (a) there is no real effort by anyone else to solve the problem and (b) the more well-understood a problem is, the easier it is to implement a solution.

5Benquo6y

I think this is true for people who've been through a modern school system, but probably not a human universal.

4Ben Pace6y

My, that was a long and difficult but worthwhile post. I see why you think it is not the natural state of affairs. Will think some more on it (though can't promise a full response, it's quite an effortful post). Am not sure I fully agree with your conclusions.

6Benquo6y

I'm much more interested in finding out what your model is after having tried to take those considerations into account, than I am in a point-by-point response.

8Raemon6y

This seems like a good conversational move to have affordance for.

2Kaj_Sotala6y

This might be true, but it doesn't sound like it contradicts the premise of "how will we implement it"? Namely, just because understanding a problem makes it easier to implement, doesn't mean that understanding alone makes it anywhere near easy to implement, and one may still need significant political clout in addition to having the solution. E.g. the whole infant nutrition thing.

2Ruby6y

Seems related to Causal vs Social Reality.

1Eli Tyre6y

Do you have an example of a problem that gets approached this way? Global warming? The need for prison reform? Factory Farming?

4Ben Pace6y

AI.

4Eli Tyre6y

It seems that AI safety has this issue less than every other problem in the world, by proportion of the people working on it. Some double digit percentage of all of the people who are trying to improve the situation, are directly trying to solve the problem, I think? (Or maybe I just live in a bubble in a bubble.) And I don’t know how well this analysis applies to non-AI safety fields.

[-]Bird Concept6y190

I'd take a bet at even odds that it's single-digit.

To clarify, I don't think this is just about grabbing power in government or military. My outside view of plans to "get a PhD in AI (safety)" seems like this to me. This was part of the reason I declined an offer to do a neuroscience PhD with Oxford/DeepMind. I didn't have any secret for why it might be plausibly crucial.

4Ben Pace6y

Strong agree with Jacob.

[-]Ben Pace4y240

Er, Wikipedia has a page on misinformation about Covid, and the first example is Wuhan lab origin. Kinda shocked that Wikipedia is calling this misinformation. Seems like their authoritative sources are abusing their positions. I am scared that I'm going to stop trusting Wikipedia soon enough, which is leaving me feeling pretty shook.

7Dagon4y

Wikipedia has beaten all odds for longevity of trust - I remember pretty heated arguments circa 2005 whether it was referenceable on any topic, though it was known to be very good on technical topics or niches without controversy where nerds could agree on what was true (but not always what was important). By 2010, it was pretty widely respected, though the recommendation from Very Serious People was to cite the underlying sources, not the articles themselves. I think it was considered pretty authoritative in discussions I was having by 2013 or so, and nowadays it's surprising and newsworthy when something is wrong for very long (though edit wars and locking down sections happens fairly often). I still take it with a little skepticism for very recently-edited or created topics - it's an awesome resource to know the shape of knowledge in the area, but until things have been there for weeks or months, it's hard to be sure it's a consensus.

5Viliam4y

Could it be a natural cycle? Wikipedia is considered trustworthy -> people with strong agenda get to positions where they can abuse Wikipedia -> Wikipedia is considered untrustworthy -> people with strong agenda find better use of their time and stop abusing Wikipedia, people who care about correct information fix it -> Wikipedia is considered trustworthy...

8ChristianKl4y

The agenda is mainly to follow the institutions like the New York Times. In a time where the New York Times isn't worth much more then saw dust, that's not a strategy to get to truth.

6Steven Byrnes4y

"No safe defense, not even Wikipedia" :-P I suggest not having a notion of "quality" that's supposed to generalize across all wiki pages. They're written by different people, they're scrutinized to wildly different degrees. Even different sections of the same article can be obviously different in trustworthiness ... Or even different sentences in the same section ... Or different words in the same sentence :)

4ChristianKl4y

Wikipedia unfortunately threw out their neutral point of view policy on COVID-19. Besides that page, the one of ivermectin ignores the meta analysises in favor of using it for COVID-19. There's also no page for "patient zero" (who was likely employed in the Wuhan Institute for Virology)

2Pattern4y

Fix it. (And let us know how long that sticks for.)

2Ben Pace4y

You fix it! If you think it's such a good idea :) I am relatively hesitant to start doing opinionated fixes on Wikipedia, I think that's not the culture of page setup that they want. My understanding is that the best Wikipedia editors write masses of pages that they're relatively disinterested in, and that being overly interested in a specific page mostly leads you to violating all of their rules and getting banned. This sort of actively political editing is precisely the sort of thing that they're trying to avoid.

2Viliam4y

By saying "Wuhan lab origin", you can roughly mean three things: * biological weapon, intentionally released, * natural virus collected, artificially improved, then escaped, * natural virus collected, then escaped in the original form. The first we can safely dismiss: who would drop a biological weapon of this type on their own population? We can also dismiss the third one, if you think in near mode what that would actually mean. It means the virus was already out there. Then someone collected it -- obviously, not all existing particles of the virus -- which means that most of the virus particles that were already out there, have remained out there. But that makes the leak from Wuhan lab an unnecessary detail; "virus already in the wild, starts pandemic" is way more likely than "virus already in the wild, does not start pandemic, but when a few particles are brought into a lab and then accidentally released without being modified, they start pandemic"... what? This is why arguing for natural evolution of the virus is arguing against the lab leak. (It's just not clearly explained.) If you do not assume that the virus was modified, then the hypothesis that the pandemic started by Wuhan lab leak, despite the virus already being out there before it was brought to the Wuhan lab, is privileging the hypothesis. If the virus is already out there, you don't need to bring it to a lab and let it escape again in order to... be out there, again. Now here I agree that the artificial improvement of the virus cannot be disproved. I mean, whatever can happen in the nature, probably can also happen in the lab, so how would you prove it didn't? I guess I am trying to say that in the Wikipedia article, the section "gain of function research" does not deserve to be classified as misinformation, but the remaining sections do.

[-]Ben Pace6y240

Responding to Scott's response to Jessica.

The post makes the important argument that if we have a word whose boundary is around a pretty important set of phenomena that are useful to have a quick handle to refer to, then

It's really unhelpful for people to start using the word to also refer to a phenomena with 10x or 100x more occurrences in the world because then I'm no longer able to point to the specific important parts of the phenomena that I was previously talking about

e.g. Currently the word 'abuser' describes a small number of people during some of their lives. Someone might want to say that technically it should refer to all people all of the time. The argument is understandable, but it wholly destroys the usefulness of the concept handle.

People often have political incentives to push the concept boundary to include a specific case in a way that, if it were principled, indeed makes most of the phenomena in the category no use to talk about. This allows for selective policing being the people with the political incentive.
It's often fine for people to bend words a little bit (e.g. when people verb nouns), but when it's in the class of terms w

... (read more)

[-]Ben Pace6y*190

I will actually clean this up and into a post sometime soon [edit: I retract that, I am not able to make commitments like this right now]. For now let me add another quick hypothesis on this topic whilst crashing from jet lag.

A friend of mine proposed that instead of saying 'lies' I could say 'falsehoods'. Not "that claim is a lie" but "that claim is false".

I responded that 'falsehood' doesn't capture the fact that you should expect systematic deviations from the truth. I'm not saying this particular parapsychology claim is false. I'm saying it is false in a way where you should no longer trust the other claims, and expect they've been optimised to be persuasive.

They gave another proposal, which is to say instead of "they're lying" to say "they're not truth-tracking". Suggest that their reasoning process (perhaps in one particular domain) does not track truth.

I responded that while this was better, it still seems to me that people won't have an informal understanding of how to use this information. (Are you saying that the ideas aren't especially well-evidenced? But they so... (read more)

3Pattern6y

Is this "bias"?

3Ben Pace6y

Yeah good point I may have reinvented the wheel. I have a sense that’s not true but need to think more.

[-]Benquo6y120

The definitional boundaries of "abuser," as Scott notes, are in large part about coordinating around whom to censure. The definition is pragmatic rather than objective.*

If the motive for the definition of "lies" is similar, then a proposal to define only conscious deception as lying is therefore a proposal to censure people who defend themselves against coercion while privately maintaining coherent beliefs, but not those who defend themselves against coercion by simply failing to maintain coherent beliefs in the first place. (For more on this, see Nightmare of the Perfectly Principled.) This amounts to waging war against the mind.

Of course, in matter of actual fact we don't strongly censure all cases of consciously deceiving. In some cases (e.g. "white lies") we punish those who fail to lie, and those who call out the lie. I'm also pretty sure we don't actually distinguish between conscious deception and e.g. reflexively saying an expedient thing, when it's abundantly clear that one knows very well that the expedient thing to say is false, as Jessica pointed out here.

*It's not clear to me that this is a good kind of concept to ... (read more)

2Ben Pace6y

Note: I just wrote this in one pass when severely jet lagged, and did not have the effort to edit it much. If I end up turning this into a blogpost I will probably do that. Anyway, I am interested in hearing via PM from anyone who feels that it was sufficiently unclearly written that they had a hard time understanding/reading it.

[-]Ben Pace6y230

I recently circled for the first time. I had two one-hour sessions on consecutive days, with 6 and 8 people respectively.

My main thoughts: this seems like a great way for getting to know my acquaintances, connecting emotionally, and build closer relationships with friends. The background emotional processing happening in individuals is repeatedly brought forward as the object of conversation, for significantly enhanced communication/understanding. I appreciated getting to poke and actually find out whether people's emotional states matched the words they were using. I got to ask questions like:

When you say you feel gratitude, do you just mean you agree with what I said, or do you mean you're actually feeling warmth toward me? Where in your body do you feel it, and what is it like?

Not that a lot of my circling time was skeptical of people's words, a lot of the time I trusted the people involved to be accurately reporting their experiences. It was just very interesting - when I noticed I didn't feel like someone was honest about some micro-emotion - to have the affordance to stop and request an honest internal report.

It felt like there was a constant tradeoff betw... (read more)

[-]Ben Pace1y222

Two recent changes to LessWrong that I made!

I have added two new reacts: "I'd bet this is false" and "I'd bet this is true".
You can now sort a given user's comments by 'top' i.e. by karma.

For the first one, if you want to let someone know you'd be willing to take a bet (or if you want to call someone out on their bullshit) you can now highlight the claim they made and use the react. The react is a pair of dice, because we're never certain about propositions we're betting on (and because 'a hand offering money' was too hard to make out at the small scale). Hopefully this will increase people's affordance to take more bets on the site!

These reacts replaced "I checked it's true" and "I checked it's false", which didn't get that much use, but were some of the most abused reacts (often used on opinions or statements-of-positions that were simply not checkable).

For the second, if you go to a user profile and scroll down to the comments, you can now sort by 'top', 'newest', 'oldest', and 'recent replies'. I find that 'top' is a great way to get a sense of a person's thoughts and perspective on the world, and I used to visit greaterwrong a lot for this feature. Now you can do it on LessWrong!

7Zac Hatfield-Dodds1y

This feels pretty nitpick-y, but whether or not I'd be interested in taking a bet will depend on the odds - in many cases I might take either side, given a reasonably wide spread. Maybe append at p >= 0.5 to the descriptions to clarify? The shorthand trading syntax "$size @ $sell_percent / $buy_percent" is especially nice because it expresses the spread you'd accept to take either side of the bet, e.g. "25 @ 85/15 on rain tomorrow" to offer a bet of $25 dollars, selling if you think probability of rain is >85%, buying if you think it's <15%. Seems hard to build this into a reaction though!

7Ben Pace1y

The only reason I've not already done this is that there's already a lot of text in the hover-over Another option is that you could pair the "I'd bet on this" react with a probability react. There could be a single react that says "I'm willing to be on this" and then you also react with one of <1%, 10%, 25%, 40%, 50%, 60%, 75%, 90%, 99+%, so that people know what odds you'd take.

2Raemon1y

lol we did have a debate about this internally before shipping. Right now we’re trying to get a rough sense of ‘would people make more bets on LW if they had more affordance to?’, and an easy thing to try was just making some reacts. But a) I encourage people to reply with followup details if they are interested in betting, and b) if it turns out reasonably popular we may make a more dedicated feature.

2Raemon1y

(FYI I was happy to see your recent bet with BenG, and am hoping more things like that happen)

4the gears to ascension1y

I like #2. On a similar thread: would be nice to have a separate section for pinned comments. I looked into pull requesting it at one point but looks like it either isn't as trivial as I hoped, or I simply got lost in the code. I feel like folks having more affordance to say, "Contrary to its upvotes or recency, I think this is one of the most representative comments from me, and others seeing my page should see it" would be helpful - pinning does this already but it has ui drawbacks because it simply pushes recent comments out of the way and the pinned marker is quite small (hence why I edit my pinned comments to say they're pinned).

4Shankar Sivarajan1y

Suggestion: the two dice should have different numbers on top, maybe a ⚅ on the "true" bet instead of a ⚀.

2Yoav Ravid1y

The comment option sorting is amazing! Thanks! The new reacts are also cool, though I also liked the "I checked" reacts and would have liked to have both.

3Rana Dexsin1y

I was pretty sad about the ongoing distortion of “I checked” in what's meant to be an epistemics-oriented community. I think the actual meanings are potentially really valuable, but without some way of avoiding them getting eaten, they become a hazard. My first thought is to put a barrier in the way, but I don't know if that plays well with the reactions system being for lower-overhead responses, and it might also give people unproductive bad feelings unless sold the right way.

[-]Ben Pace4y220

Okay, I’ll say it now, because there’s been too many times.

If you want your posts to be read, never, never, NEVER post multiple posts at the same time.

Only do that if you don’t mind none of the posts being read. Like if they’re all just reference posts.

I never read a post if there’s two or more to read, it feels like a slog and like there’s going to be lots of clicking and it’s probably not worth it. And they normally do badly on comments on karma so I don’t think it’s just me.

Even if one of them is just meant as reference, it means I won’t read the other one.

[-]Ben Pace9mo216

I have a general belief that internet epistemic hygiene norms should include that, when you quote someone, not only should you link to the source, but you should link to the highlight of that source. In general, if you highlight text on a webpage and right-click, you can "copy link to highlight" which when opened scrolls to and highlights that text. (Random example on Wikipedia.)

Further on this theme, archive.is has the interesting feature of constantly altering the URL to point to the currently highlighted bit of text, making this even easier. (Example, and you can highlight other bits of text to see it change.) Currently I overall don't like it because I constantly highlight text while I'm reading it, and so am v annoyed by the URL constantly changing, but it's plausible I'd get over this in time, and it'd be a good feature to add to LW.

The archive.is feature is also better because the normal "copy link to highlight" can often be unwieldily and long. Also I recall it sometimes not working, probably because the highlight is too short or too long (I don't quite understand the rules). On archive.is it just has a start and end number for where in the text is highlighted, making it al... (read more)

[-]gwern9mo*209

I have misgivings about the text-fragment feature as currently implemented. It is at least now a standard and Firefox implements reading text-fragment URLs (just doesn't conveniently allow creation without a plugin or something), which was my biggest objection before; but there are still limitations to it which show that a lot of what the text-fragment 'solution' is, is a solution to the self-inflicted problems of many websites being too lazy to provide useful anchor IDs anywhere in the page. (I don't know how often I go to link a section of a blog post, where the post is written in a completely standard hierarchical table-of-contents way, and the headers turn out to be... nothing but <h2>s with not an id= anywhere in sight.) We would be a lot better off if pages had more meaningful IDs and selecting text did something like, pick the nearest preceding ID. (This could be implemented in LW2 or Gwern.net right now, incidentally. If the user selects some text, just search through the tree to find the first previous ID, and update the current browser-bar URL to URL#ID.)

Hacking IDs onto an unwilling page, whose author neither knows nor cares nor can even find out what IDs are in us... (read more)

[-]MondSemmel9mo104

"Copy link to highlight" is not available in Firefox. And while e.g. Bing search seems to automatically generate these "#:~:text=" links, I find they don't work with any degree of consistency. And they're even more affected by link rot than usual, since any change to the initial text (like a typo fix) will break that part of the link.

4Ben Pace9mo

Though if the text changes, then it degrades gracefully to just linking to the right webpage, which is the current norm.

6cubefox9mo

The highlights are officially called "text fragments" and the syntax is described here: https://developer.mozilla.org/en-US/docs/Web/URI/Reference/Fragment/Text_fragments

3Guive9mo

I like this idea. There's always endless controversy about quoting out of context. I can't recall seeing any previous specific proposals to help people assess the relevance of context for themselves.

2Mateusz Bagiński9mo

You can add it as an opt-in feature.

1Czynski9mo

I find them visually awful and disable them in settings. And avoid using archive.is because there's no way to turn that off. Not that I browse LW that much, in fairness.

[-]Ben Pace1y*21-4

Here's a quick list of 7 things people sometimes do instead of losing arguments, when it would be too personally costly to change their position.

They pick a variable in the argument that is hard for the debaters to get closer to, and state that their raw-intuition is simply different to the person they're talking to. ("I just believe that human ingenuity will overcome problems like this, and you don't, and this isn't something we're easily going to be able to resolve." or "I think we're just not going to be able to resolve whether this city-wide intervention works or not without a good RCT.")
They undermine your ability to have an opinion in the argument. ("I'm afraid there's very advanced research papers you'd need to read to have an understanding of this" or "You have been wrong so many times on so many issues, I don't think you're worth trusting on this one")
They performatively don't understand basic ideas, or weaponize confusion. ("I'm sorry, could you explain your theory of 'ownership', I don't really understand what you mean when you say you own your company" or "I'm sorry, I'm not sure what it means to 'provoke'? I was just saying what I thought.")
They get emotionally irate i

... (read more)

[-]cubefox1y110

Schopenhauer's sarcastic essay The Art of Being Right (a manuscript published posthumously) goes in this direction. In it he suggests 38 rhetorical strategies to win a dispute by any means possible. E.g. your 3 corresponds to his 31, your 5 is similar to his 30, and your 7 to is similar his 18. Though he isn't just focusing on avoiding losing arguments, but on winning them.

3Ben Pace1y

It's quite funny, thanks for the link! Quoting from number 31: Of course it's harder now given that people do have the internet at-hand, but I think I still see this tactic employed.

6Viliam1y

Yeah, there are 8 billion people like that, so that's probably the most frequent case. I think the list of "things people sometimes do instead of losing arguments" would be very long. Did you pick these 7 because they are most frequent (in your bubble) or just because they irritate you most? I can add an example of a thing that irritates me (but maybe that's just 1 person I talked to too much recently); it's when after explaining some crucial fact that was missing in their 'edgy' perspective, the person suddenly goes: "and why is this topic so important to you? why do you care so much?" as if it's weird to know facts about something, but just a while ago it was okay for them to spend fifteen minutes talking platitudes about it

2Ben Pace1y

I just picked the first ones that came to mind, so I guess they're probably the ones I run into most frequently.

4Nathan Helm-Burger1y

Worth keeping in mind is that seeing one of these things happen doesn't prove that what is going on is that subconsciously or consciously they believe they are losing the argument. It also shouldn't be taken as evidence from the universe that your argument is correct. People are fallible and run on noisy hacky biological computers. So yeah, it's worth noticing these sorts of argument breakdowns, but also worth being careful about reading too much into them. Here's a hypothetical: Suppose instead that the person you are arguing with is very knowledgeable about specific topic X. You have dabbled in X and know more than the average layperson, but lack a deep knowledge of the subject. The two of you have a disagreement about X. The two of you begin arguing about it, but after a few minutes it becomes clear to them that you don't have a deep knowledge of the subject. They think through what they'd need to teach you in order for you to make a valid argument which would potentially change their mind. They estimate it would require at least 6 hours of lecturing, maybe divided up as 3 lectures, with lots of dense technical sources assigned as prerequisite reading before each lecture. They realize that that investment of time and energy, even if you agreed that you needed it and committed to wholeheartedly tackling it, wouldn't be worth it to them in order to continue the argument. They give up on you as a conversational partner on the subject of X, and seek a convenient social out to end the conversation as soon as possible. Here's another hypothetical: Someone begins having an argument with you, on a topic which for you is mostly emotionally neutral. You are arguing politely for your point of view in a fair and objective way. They, however, do not have an emotionally neutral stance on this. They start out arguing in an fair and objective way, thinking rationally and logically about each point. Soon however, the debate touches on deep powerful feelings they have,

[-]Ben Pace6y*200

Good posts you might want to nominate in the 2018 Review

I'm on track to nominate around 30 posts from 2018, which is a lot. Here is a list of about 30 further posts I looked at that I think were pretty good but didn't make my top list, in the hopes that others who did get value out of the posts will nominate their favourites. Each post has a note I wrote down for myself about the post.

Reasons compute may not drive AI capabilities growth
- I don’t know if it’s good, but I’d like it to be reviewed to find out.
The Principled-Intelligence Hypothesis
- Very interesting hypothesis generation. Unless it’s clearly falsified, I’d like to see it get built on.
Will AI See Sudden Progress? DONE
- I think this post should be considered paired with Paul’s almost-identical post. It’s all exactly one conversation.
Personal Relationships with Goodness
- This felt like a clear analysis of an idea and coming up with some hypotheses. I don’t think the hypotheses really captures what’s going on, and most of the frames here seem like they’ve caused a lot of people to do a lot of hurt to themselves, but it seemed like progress in that conversation.
Are ethical asymmetries from property rights?
- Again, another very intere

... (read more)

[-]Ben Pace6y200

I was just re-reading the classic paper Artificial Intelligence as Positive and Negative Factor in Global Risk. It's surprising how well it holds up. The following quotes seem especially relevant 13 years later.

On the difference between AI research speed and AI capabilities speed:

The first moral is that confusing the speed of AI research with the speed of a real AI once built is like confusing the speed of physics research with the speed of nuclear reactions. It mixes up the map with the territory. It took years to get that first pile built, by a small group of physicists who didn’t generate much in the way of press releases. But, once the pile was built, interesting things happened on the timescale of nuclear interactions, not the timescale of human discourse. In the nuclear domain, elementary interactions happen much faster than human neurons fire. Much the same may be said of transistors.

On neural networks:

The field of AI has techniques, such as neural networks and evolutionary programming, which have grown in power with the slow tweaking of decades. But neural networks are opaque—the user has no idea how the neural net is making its decisions—and cannot easily be rendered

... (read more)

[-]Ben Pace7y200

Reviews of books and films from my week with Jacob:

Films watched:

The Big Short

Review: Really fun. I liked certain elements of how it displays bad nash equilibria in finance (I love the scene with the woman from the ratings agency - it turns out she’s just making the best of her incentives too!).
Grade: B

Spirited Away

Review: Wow. A simple story, yet entirely lacking in cliche, and so seemingly original. No cliched characters, no cliched plot twists, no cliched humour, all entirely sincere and meaningful. Didn’t really notice that it was animated (while fantastical, it never really breaks the illusion of reality for me). The few parts that made me laugh, made me laugh harder than I have in ages.
There’s a small visual scene, unacknowledged by the ongoing dialogue, between the mouse-baby and the dust-sprites which is the funniest thing I’ve seen in ages, and I had to rewind for Jacob to notice it.
I liked how by the end, the team of characters are all a different order of magnitude in size.
A delightful, well-told story.
Grade: A+

Stranger Than Fiction

Review: This is now my go-to film of someone trying something original and just failing. Filled with new ideas, but none executed well, a

... (read more)

[-]Ben Pace3y1910

"Slow takeoff" at this point is simply a misnomer.

Paul's position should be called "Fast Takeoff" and Eliezer's position should be called "Discontinuous Takeoff".

2Vladimir_Nesov3y

Slow takeoff doesn't imply absence of discontinuous takeoff a bit later, it just says that FOOM doesn't happen right away and thus there is large AI impact (which is to say, things are happening fast) even pre-FOOM, if it ever happens.

2lc3y

Why not drop "Fast vs Slow" entirely and just use "continuous" vs. "discontinuous" takeoff to refer to the two ideas?

6Ben Pace3y

I guess it helps remind everyone that both positions are relatively extreme compared to how most other people have been expecting that the future will go. But continuous vs discontinuous also seems pretty helpful.

[-]Ben Pace2y182

I don't normally just write-up takes, especially about current events, but here's something that I think is potentially crucially relevant to the dynamics involved in the recent actions of the OpenAI board, that I haven't seen anyone talk about:

The four members of the board who did the firing do not know each other very well.

Most boards meet a few times per year, for a couple of hours. Only Sutskever works at OpenAI. D'Angelo works senior roles in tech companies like Facebook and Quora, Toner is in EA/policy, and MacAulay at other tech companies (I'm not aware of any overlap with D'Angelo).

It's plausible to me that MacAulay and Toner have spent more than 50 hours in each others' company, but overall I'd probably be willing to bet at even odds that no other pair of them had spent more than 10 hours together before this crisis.

This is probably a key factor in why they haven't written more publicly about their decision. Decision-by-committee is famously terrible, and it's pretty likely to me that everyone pushes back hard on anything unilateral by the others in this high-tension scenario. So any writing representing them has to get consensus, and they're focused on firefighting and ge... (read more)

4Ben Pace2y

In this mess, Altman and Helen should not be held to the same ethical standards, because I believe one of them has been given a powerful career in substantial part based on her commitments to higher ethical standards (a movement that prided itself on openness and transparency and trying to do the most good). If Altman played deceptive strategies, and insofar as Helen played back the same deceptive strategies as Altman, then she did not honor the EA name. (The name has a lot of dirt on it these days already, but still. It is a name that used to mean something back when it gave her power.) Insofar as you got a position specifically because you were affiliated with a movement claiming to be good and open and honest and to have unusually high moral standards, and then when you arrive you become a standard political player, that's disingenuous.

2ryan_greenblatt2y

I think Holden being added to the board shouldn't be mostly attributed to his affiliation with EA. And the Helen board seat is originally from this. (The relevant history here is that this is the OpenAI grant that resulted in a board seat while here is a post from just earlier about Holden's takes on EA.)

4Ben Pace2y

Some historical context Holden in 2013 on the GiveWell blog: Holden in 2015 on the EA Forum (talking about GiveWell Labs, which grew into OpenPhil): Holden in April 2016 about plans for working on AI: (Dewey who IIRC had worked at FHI and CEA ahead of this, and Beckstead from FHI.) Holden in 2016 about why they're making potential risks from advanced AI a priority: Holden about the OpenAI grant in 2017: As a negative datapoint: I looked through a bunch of the media articles linked at the bottom of this GiveWell page, and most of them do not mention Effective Altruism, only effective giving / cost-effectiveness. So their Effective Altruist identity have had less awareness amongst folks who primarily know of Open Philanthropy through their media appearances.

2Ben Pace2y

I think this is accurately described as "an EA organization got a board seat at OpenAI", and the actions of those board members reflect directly on EA (whether internally or externally). Why did OpenAI come to trust Holden with this position of power? My guess is Holden and Dustin's personal reputations were substantial effects here, along with Open Philanthropy's major funding source, but also that many involved people's excitement about and respect for the EA movement were a relevant factor in OpenAI wanting to partner with Open Philanthropy, and that Helen's and Tasha's actions have directly and negatively reflected on how the EA ecosystem is viewed by OpenAI leadership. There's a separate question about why Holden picked Helen Toner and Tasha MacAulay, and to what extent they were given power in the world by the EA ecosystem. It seems clear that these people have gotten power through their participation in the EA ecosystem (as OpenPhil is an EA institution), and to the extent that the EA ecosystem advertises itself as more moral than other places, if they executed the standard level of deceptive strategies that others in the tech industry would in their shoes, then that was false messaging.

4Ben Pace2y

I'm not quite sure in the above comment how to balance between "this seems to me like it could explain a lot" and also "might just be factually false". So I guess I'm leaving this comment, lampshading it.

3Ben Pace2y

The most important thing right now: I still don't know why they chose to fire Altman, and especially why they chose to do it so quickly. That's an exceedingly costly choice to make (i.e. with the speed of it), and so when I start to speculate on why, I only come up with commensurately worrying states of affair e.g. he did something egregious enough to warrant it, or he didn't and the board acted with great hostility. Them going back on their decision is bayesian evidence for the latter — if he'd done something egregious, they'd just be able to tell relevant folks, and Altman wouldn't get his job back. So many people are asking this (e.g. everyone at the company). I'll be very worried if the reason doesn't come out.

3Ben Pace2y

In brief: I'm saying that once you condition on: 1. The board decided the firing was urgent. 2. The board does not know each other very well and defaults to making decisions by consensus. 3. The board is immediately in a high-stakes high-stress situation. Then you naturally get 4. The board fails to come to consensus on public comms about the decision.

2Ben Pace2y

Also, I don't know that I've said this, but from reading enough of his public tweets, I had blocked Sam Altman long ago. He seemed very political in how he used speech, and so I didn't want to include him in my direct memetic sphere. As a small pointer to why: he would commonly choose not to share object-level information about something, but instead share how he thought social reality should change. I think I recall him saying that the social consensus was wrong about fusion energy, and pushed for it to move in a specific direction; he did this rather than just plainly say what his object level beliefs about fusion were, or offer a particular counter-argument to an argument that was going around. It's been a year or two since I blocked him, so I don't recall more specifics, but it seemed worth mentioning, as a datapoint for folks to include in their character assessments.

2Ben Pace2y

My current guess is that most of the variance in what happened is explained by a board where 3 out of 4 people don't know the dynamics of upper management in a multi-billion dollar company, where the board don't know each other well, and (for some reason) the decision was made very suddenly. Pretty low-expectations given that situation. Seems like Shear was a pretty great replacement get given the hand dealt. Assuming that they had legit reason to fire the CEO, it's probably primarily through lack of skill and competence that they failed, more so than as a result of Altman's superior deal-making skill and leadership abilities (though that was what finished it off).

[-]Ben Pace6y170

There's a game for the Oculus Quest (that you can also buy on Steam) called "Keep Talking And Nobody Explodes".

It's a two-player game. When playing with the VR headset, one of you wears the headset and has to defuse bombs in a limited amount of time (either 3, 4 or 5 mins), while the other person sits outside the headset with the bomb-defusal manual and tells you what to do. Whereas with other collaboration games, you're all looking at the screen together, with this game the substrate of communication is solely conversation, the other person is providing all of your inputs about how their half is going (i.e. not shown on a screen).

The types of puzzles are fairly straightforward computational problems but with lots of fiddly instructions, and require the outer person to figure out what information they need from the inner person. It often involves things like counting numbers of wires of a certain colour, or remembering the previous digits that were being shown, or quickly describing symbols that are not any known letter or shape.

So the game trains you and a partner in efficiently building a shared language for dealing with new problems.

More than that, as the game gets harder, often

... (read more)

6Trinley Goldenberg6y

There's a similar free game for Android and iOs called space team that I highly recommend.

4Gordon Seidoh Worley6y

I use both this game and Space Team as part of training people in the on-call rotation at my company. They generally report that it's fun, and I love it because it usually creates the kind of high-pressure feelings in people they may experience when on-call, so it creates a nice, safe environment for them to become more familiar with those feelings and how to work through them. On a related note, I'm generally interested in finding more cooperative games with asymmetric information and a need to communicate. Lots of games meet one or two of those criteria, but very few games are able to meet all simultaneously. For example, Hanabi is cooperative and asymmetric, but lacks much communication (you're not allowed to talk), and many games are asymmetric and communicative but not cooperative (Werewolf, Secret Hitler, etc.) or cooperative and communicative but not asymmetric (Pandemic, Forbidden Desert, etc.).

1ioannes6y

+1 – this game is great. It's really good with 3-4 people giving instructions and one person in the hot seat. Great for team bonding.

[-]Ben Pace6y160

I talked with Ray for an hour about Ray's phrase "Keep your beliefs cruxy and your frames explicit".

I focused mostly on the 'keep your frames explicit' part. Ray gave a toy example of someone attempting to communicate something deeply emotional/intuitive, or perhaps a buddhist approach to the world, and how difficult it is to do this with simple explicit language. It often instead requires the other person to go off and seek certain experiences, or practise inhabiting those experiences (e.g. doing a little meditation, or getting in touch with their emotion of anger).

Ray's motivation was that people often have these very different frames or approaches, but don't recognise this fact, and end up believing aggressive things about the other person e.g. "I guess they're just dumb" or "I guess they just don't care about other people".

I asked for examples that were motivating his belief - where it would be much better if the disagreers took to hear the recommendation to make their frames explicit. He came up with two concrete examples:

Jim v Ray on norms for shortform, where during one hour they worked through the same reasons

... (read more)

[-]Zvi6y280

I find "keep everything explicit" to often be a power move designed to make non-explicit facts irrelevant and non-admissible. This often goes along with burden of proof. I make a claim (real example of this dynamic happening, at an unconference under Chatham house rules: That pulling people away from their existing community has real costs that hurt those communities), and I was told that, well, that seems possible, but I can point to concrete benefits of taking them away, so you need to be concrete and explicit about what those costs are, or I don't think we should consider them.

Thus, the burden of proof was put upon me, to show (1) that people central to communities were being taken away, (2) that those people being taken away hurt those communities, (3) in particular measurable ways, (4) that then would impact direct EA causes. And then we would take the magnitude of effect I could prove using only established facts and tangible reasoning, and multiply them together, to see how big this effect was.

I cooperated with this because I felt like the current estimate of this cost for this person was zero, and I could easily raise that, and that was better than nothing,... (read more)

[-]Ben Pace6y*110

To complement that: Requiring my interlocutor to make everything explicit is also a defence against having my mind changed in ways I don't endorse but that I can't quite pick apart right now. Which kinda overlaps with your example, I think.

I sometimes will feel like my low-level associations are changing in a way I'm not sure I endorse, halt, and ask for something that the more explicit part of me reflectively endorses. If they're able to provide that, then I will willingly continue making the low-level updates, but if they can't then there's a bit of an impasse, at which point I will just start trying to communicate emotionally what feels off about it (e.g. in your example I could imagine saying "I feel some panic in my shoulders and a sense that you're trying to control my decisions"). Actually, sometimes I will just give the emotional info first. There's a lot of contextual details that lead me to figure out which one I do.

[-]Raemon6y100

One last bit is to keep in mind that most (or, many things), can be power moves.

There's one failure mode, where a person sort of gives you the creeps, and you try to bring this up and people say "well, did they do anything explicitly wrong?" and you're like "no, I guess?" and then it turns out you were picking up something important about the person-giving-you-the-creeps and it would have been good if people had paid some attention to your intuition.

There's a different failure mode where "so and so gives me the creeps" is something you can say willy-nilly without ever having to back it up, and it ends up being it's own power move.

I do think during politically charged conversations it's good to be able to notice and draw attention to the power-move-ness of various frames (in both/all directions)

(i.e. in the "so and so gives me the creeps" situation, it's good to note both that you can abuse "only admit explicit evidence" and "wanton claims of creepiness" in different ways. And then, having made the frame of power-move-ness explicit, talk about ways to potentially alleviate both forms of abuse)

6Raemon6y

Want to clarify here, "explicit frames" and "explicit claims" are quite different, and it sounds here like you're mostly talking about the latter. The point of "explicit frames" is specifically to enable this sort of conversation – most people don't even notice that they're limiting the conversation to explicit claims, or where they're assuming burden of proof lies, or whether we're having a model-building sharing of ideas or a negotiation. Also worth noting (which I hadn't really stated, but is perhaps important enough to deserve a whole post to avoid accidental motte/bailey by myself or others down the road): My claim is that you should know what your frames are, and what would change* your mind. *Not* that you should always tell that to other people. Ontological/Framework/Aesthetic Doublecrux is a thing you do with people you trust about deep, important disagreements where you think the right call is to open up your soul a bit (because you expect them to be symmetrically opening their soul, or that it's otherwise worth it), not something you necessarily do with every person you disagree with (especially when you suspect their underlying framework is more like a negotiation or threat than honest, mutual model-sharing) *also, not saying you should ask "what would change my mind" as soon as you bump into someone who disagrees with you. Reflexively doing that also opens yourself up to power moves, intentional or otherwise. Just that I expect it to be useful on the margin.

6Zvi6y

Interesting. It seemed in the above exchanges like both Ben and you were acting as if this was a request to make your frames explicit to the other person, rather than a request to know what the frame was yourself and then tell if it seemed like a good idea. I think for now I still endorse that making my frame fully explicit even to myself is not a reasonable ask slash is effectively a request to simplify my frame in likely to be unhelpful ways. But it's a lot more plausible as a hypothesis.

4Raemon6y

I've mostly been operating (lately) within the paradigm of "there does in fact seem to be enough trust for a doublecrux, and it seems like doublecrux is actually the right move given the state of the conversation. Within that situation, making things as explicit as possible seems good to me." (But, this seems importantly only true within that situation) But it also seemed like both Ben (and you) were hearing me make a more aggressive ask than I meant to be making (which implies some kind of mistake on my part, but I'm not sure which one). The things I meant to be taking as a given are: 1) Everyone has all kinds of implicit stuff going on that's difficult to articulate. The naively Straw Vulcan failure mode is to assume that if you can't articulate it it's not real. 2) I think there are skills to figuring out how to make implicit stuff explicit, in a careful way that doesn't steamroll your implicit internals. 3) Resolving serious disagreements requires figuring out how to bridge the gap of implicit knowledge. (I agree that in a single-pair doublecrux, doing the sort of thing you mention in the other comment can work fine, where you try to paint a picture and ask them questions to see if they got the picture. But, if you want more than one person to be able to understand the thing you'll eventually probably want to figure out how to make it explicit without simplifying it so hard that it loses its meaning) 4) The additional, not-quite-stated claim is "I nowadays seem to keep finding myself in situations where there's enough longstanding serious disagreements that are worth resolving that it's worth Stag Hunting on Learning to Make Beliefs Cruxy and Frames Explicit, to facilitate those conversations." I think maybe the phrase "*keep* your beliefs cruxy and frames explicit" implied more of an action of "only permit some things" rather than "learn to find extra explicitness on the margin when possible."

4Raemon6y

As far as explicit claims go: My current belief is something like: If you actually want to communicate an implicit idea to someone else, you either need 1) to figure out how to make the implicit explicit, or 2) you need to figure out the skill of communicating implicit things implicitly... which I think actually can be done. But I don't know how to do it and it seems hella hard. (Circling seems to work via imparting some classes of implicit things implicitly, but depends on being in-person) My point is not at all to limit oneself to explicit things, but to learn how to make implicit things explicit (or, otherwise communicable). This is important because the default state often seems to be failing to communicate at all. (But it does seem like an important, related point that trying to push for this ends up very similar sounding, from the outside, like 'only explicit evidence is admissable', which is a fair thing to have a instinctive resistance to) But, the fact that this is real hard is because the underlying communication is real hard. And I think there's some kind of grieving necessary to accept the fact that "man, why can't they just understand my implicit things that seem real obvious to me?" and, I dunno, they just can't. :/

4Zvi6y

Agreed it's a learned skill and it's hard. I think it's also just necessary. I notice that the best conversations I have about difficult to describe things definitely don't involve making everything explicit, and they involve a lot of 'do you understand what I'm saying?' and 'tell me if this resonates' and 'I'm thinking out loud, but maybe'. And then I have insights that I find helpful, and I can't figure out how to write them up, because they'd need to be explicit, and they aren't, so damn. Or even, I try to have a conversation with someone else (in some recent cases, you) and share these types of things, and it feels like I have zero idea how to get into a frame where any of it will make any sense or carry any weight, even when the other person is willing to listen by even what would normally be strong standards. Sometimes this turns into a post or sequence that ends up explaining some of the thing? I dunno.

6Raemon6y

FWIW, upcoming posts I have in the queue are: * Noticing Frame Differences * Tacit and Explicit Knowledge * Backpropagating Facts into Aesthetics * Keeping Frames Explicit (Possibly, in light of this conversation, adding a post called something like "Be secretly explicit [on the margin]")

[-]Raemon6y120

I'd been working on a sequence explaining this all in more detail (I think there's a lot of moving parts and inferential distance to cover here). I'll mostly respond in the form of "finish that sequence."

But here's a quick paragraph that more fully expands what I actually believe:

If you're building a product with someone (metaphorical product or literal product), and you find yourself disagreeing, and you explain "This is important because X, which implies Y", and they say "What!? But, A, therefore B!" and then you both keep repeating those points over and over... you're going to waste a lot of time, and possibly build a confused frankenstein product that's less effective than if you could figure out how to successfully communicate.

In that situation, I claim you should be doing something different, if you want to build a product that's actually good.
If you're not building a product, this is less obviously important. If you're just arguing for fun, I dunno, keep at it I guess.

A separate, further claim is that the reason you're miscommunicating is because you have a bunch of hidden assumptions in yo

... (read more)

[-]Ben Pace6y100

every time you disagree with someone about one of your beliefs, you [can] automatically flag what the crux for the belief was

This is the bit that is computationally intractable.

Looking for cruxes is a healthy move, exposing the moving parts of your beliefs in a way that can lead to you learning important new info.

However, there are an incredible number of cruxes for any given belief. If I think that a hypothetical project should accelerate it's development time 2x in the coming month, I could change my mind if I learn some important fact about the long-term improvements of spending the month refactoring the entire codebase; I could change my mind if I learn that the current time we spend on things is required for models of the code to propagate and become common knowledge in the staff; I could change my mind if my models of geopolitical events suggest that our industry is going to tank next week and we should get out immediately.

4Raemon6y

I'm not claiming you can literally do this all the time. [Ah, an earlier draft of the previous comment emphasized this this was all "things worth pushing for on the margin", and explicitly not something you were supposed to sacrifice all other priorities for. I think I then rewrote the post and forgot to emphasize that clarification] I'll try to write up better instructions/explanations later, but to give a rough idea of the amount of work I'm talking about. I'm saying "spend a bit more time than you normally do in 'doublecrux mode'". [This can be, like, an extra half hour sometimes when having a particular difficult conversation]. When someone seems obviously wrong, or you seem obviously right, ask yourself "what are cruxes are most loadbearing", and then: * Be mindful as you do it, to notice what mental motions you're actually performing that help. Basically, do Tuning Your Cognitive Strategies to the double crux process, to improve your feedback loop. * When you're done, cache the results. Maybe by writing it down, or maybe just sort of thinking harder about it so you remember it a better. The point is not to have fully mapped out cruxes of all your beliefs. The point is that you generally have practiced the skill of noticing what the most important cruxes are, so that a) you can do it easily, and b) you keep the results computed for later.

[-]Ben Pace2yΩ6151

For too long, I have erred on the side of writing too much.

The first reason I write is in order to find out what I think.

This often leaves my writing long and not very defensible.

However, editing the whole thing is so much extra work after I already did all the work figuring out what I think.

Sometimes it goes well if I just scrap the whole thing and concisely write my conclusion.

But typically I don't want to spend the marginal time.

Another reason my writing is too long is because I have extra thoughts I know most people won't find useful.

But I've picked up a heuristic that says it's good to share actual thinking because sometimes some people find it surprisingly useful, so I hit publish anyway.

Nonetheless, I endeavor to write shorter.

So I think I shall experiment with cutting the bits off of comments that represent me thinking aloud, but aren't worth the space in the local conversation.

And I will put them here, as the dregs of my cognition. I shall hopefully gather data over the next month or two and find out whether they are in fact worthwhile.

4Adam Zerner2y

Noooooooo! I mean this in a friendly sort of sense. Not that I'm mad or indignant or anything. Just that I'm sad to see this and suspect that it is a move in the wrong direction. This relates to something I've been wanting to write about for a while and just never really got around to it. Now's as good a time as any to at least get started. I started a very preliminary shortform post on it here a while ago. Basically, think about the progression of an idea. Let's use academia as an initial example. * At some point in the timeline, an idea is deemed good enough to pursue an experiment on. * Then the results of the experiment are published. * Then people read about the results and talk about them. And the idea. * Then other people summarize the idea and the results. In other papers. In textbooks. In meta-analyses. In the newspaper. Blog posts. Pop science books. Whatever. * Then people discuss those summaries. * Earlier on, before the idea was deemed good enough to pursue an experiment on, the idea probably went through various revisions. * And before that, the author of the idea probably chatted with some colleagues about it to see what they think. * And before that, I dunno, maybe there was a different idea that ended up being a dead end, but lead to the author pivoting to the real idea. * And before that, I dunno, there's probably various babble-y things going on. What I'm trying to get at is that there is some sort of lifecycle of an idea. Maybe we can think of the stages as: 1. Inspiration 2. Ideation 3. Refinement 4. Pursuit 5. Spread On platforms like LessWrong, I feel like there is a sort of cultural expectation that when you publish things publicly, they are at the later stages in this lifecycle. From what I understand, things like Personal Blog Posts, Open Thread and Shortform all exist as places where people are encouraged to post about things regardless of the lifecycle stage. However, in practice, I don't really think people feel comfo

2Viliam2y

Yeah. Similar here, only I am aware of this in advance, so I often simply write nothing, because I am a bit of perfectionist here, don't want to publish something unfinished, and know that finishing just isn't worth it. I wonder whether AI editors could help us with this.

2Yoav Ravid2y

Have you considered using footnotes for that?

2Ben Pace2y

That's a fine idea, but for a while I'd like to err on the side of my comments being "definitely shorter than they have to be" rather than "definitely longer than they have to be". (In general I often like to execute pendulum swings, so that I at least know that I am capable of not making the same errors forever.)

2Ben Pace2y

I don't want to double the comment count I submit to Recent Discussion, so I'll just update this comment with the things I've cut. 12/06/2023 Comment on Originality vs. Correctness

1lillybaeum2y

You may want to look into Toki Pona, a language ostensibly built around conveying meaning in the fewest, simplest possible expressions. One can explain the most complex things despite having only 130~ words, almost like 'programming' the meaning into the sentence, but as the sentence necessarily gets longer and longer, one begins to wonder the necessity of encoding so much meaning. You can only point to the Tao, you can't describe it or name it directly. Information is much the same way, I think.

[-]Ben Pace2y145

Often I am annoyed when I ask someone (who I believe has more information than me) a question and they say "I don't know". I'm annoyed because I want them to give me some information. Such as:

"How long does it take to drive to the conference venue?"
"I don't know."
"But is it more like 10 minutes or more like 2 hours?"
"Oh it's definitely longer than 2 hours."

But perhaps I am the one making a mistake. For instance, the question "How many countries are there?" can be answered "I'd say between 150 and 400" or it can be answered "195", and the former is called "an estimate" and the latter is called "knowing the answer". There is a folk distinction here and perhaps it is reasonable for people to want to preserve the distinction between "an estimate" and "knowing the answer".

So in the future, to get what I want, I should say "Please can you give me an estimate for how long it takes to drive to the conference venue?".

And personally I should strive, when people ask me a question to which I don't know the answer, to say "I don't know the answer, but I'd estimate between X and Y."

3winstonBosan2y

It seems like, instead of asking the objective lvl question, asking a probing “What can you tell me about the drive to the conference?” And expanding from there might get you closer to desired result.

3Shankar Sivarajan2y

Alternatively, if information retrieval and transmission is expensive enough, or equivalently, if finding another source quick and easy, "I don't know" could mean "Ask someone else: the expected additional precision/confidence of doing so is worth the effort."

2Dagon2y

Is this in a situation where you're limited in time or conversational turns? It seems like the follow-up clarification was quite successful, and for many people it would feel more comfortable than the more specific and detailed query. In technical or professional contexts, saving time and conveying information more efficiently gets a bit more priority, but even then this seems like over-optimizing. That said, I do usually include additional information or a conversational follow-up hook in my "I don't know" answers. You should expect to hear from me "I don't know, but I'd go at least 2 hours early if it's important", or "I don't know, what does Google Maps say?", or "I don't know, what time of day are you going?" or the like.

2CstineSublime2y

I know this seems like a question with an obvious answer but it is surprisingly non-obvious: Why do you need to know how long it takes to drive to the conference venue? Or to put it another way: what decision will be influenced by their answer (and what level of precision and accuracy is sufficient to make that decision). I realize this is just an example, but the point is it's not clear what decision you're trying to weigh up is even from the example. Is it a matter of whether you attend the event at the conference venue or not? Is it deciding whether you should seek overnight accommodation or not? Do you have another event you want to attend in the day and wonder if you can squeeze both in? etc. etc. Another thing is I'm the kind of person to default to "I don't know" because I often don't even trust my own ability to give an estimate, and would feel terrible and responsible if someone made a poor decision because of my inept estimation. And I get very annoyed when people push me for answers I do not feel qualified to answer.

2Ben Pace2y

A common experience I have is that it takes like 1-2 paragraphs of explanation for why I want this info (e.g. "Well I'm wondering if so-and-so should fly in a day earlier to travel with me but it requires going to a different airport and I'm trying to figure out whether the time it'd take to drive to me would add up to too much and also..."), but if they just gave me their ~70% confidence interval when I asked then we could cut the whole context-sharing.

1CstineSublime2y

Would you say that as a convention most people assume you (or anyone) want a specific number rather than a range?

2Ben Pace2y

I’d say most people assume I want “the answer” rather than “some bits of information”.

1CstineSublime2y

To be honest I'm not sure on the difference? Could you phrase that in a different way? And do you think they feel they ought give you a specific number rather than a range that the number could exist in?

[-]Ben Pace6y140

Live a life worth leaving Facebook for.

[-]Ben Pace8mo120

An idea I've been thinking about for LessOnline this year, is a blogging awards ceremony. The idea being that there's a voting procedure on the blogposts of the year, in a bunch of different categories, a shortlist is made and winners are awarded a prize.

I like opportunities for celebrating things in the online, written, truth-seeking ecosystem. I'm interested in reacts on whether people would be pro something like this happening, and comments on suggestions for how to do it well. (Epistemic status: tentatively excited about this idea.)

Here's my first idea for what the categories would be this year.

Blog of the year
Best original contribution (e.g. novel discovery, did some science, etc)
Best explanation of a complex idea
Biggest mistake admitted (H/T @ymekshout for suggesting this one)
Best fiction
Best counter-argument
Best prediction
Most productive dialogue
Most beautiful non-fiction
Best new blog

As for structure, I'm not really sure. Here's my first idea.

How to nominate? Anyone can nominate a blogpost for $10. Anyone with a LessOnline ticket gets a free nomination. Anything published in 2024 is eligible.
How is it judged? For the first year I'd probably keep it small and simple, pe

... (read more)

[-]Ben Pace3mo110

Experimental new reacts coming soon... (image visible on click)

[-]Ben Pace3mo280

Answers! Of interest to: @Viliam @Raymond Douglas @Garrett Baker @Nina Panickssery @Perhaps @Mateusz Bagiński

2Ben Pace3mo

@dirk What do you think is the typo? I asked ChatGPT if my spelling was non-standard and it said 'nitpicking' is standard. Oh, maybe are point out that there's no apostrophe in my use of the word "its" in the description.

7Elizabeth3mo

surprised we needed a cocaine emoji

4Viliam3mo

dollar, magician hat, sleepy face, no division by zero, plus sign, ugly face, uhhh no idea what is this one, wizard, bug under looking glass, check your thoughts, cocaine, broken chain, unbroken chain, delta (I thought we already had this one)

2Ben Pace3mo

You are right! About one of them. Better luck next time on the other 13. I'm actually surprised the wizard one wasn't obvious (on reflection I am typical minding way too hard here).

4Raymond Douglas3mo

best guesses: valuable, hat tip, disappointed, right assumption wrong conclusion, +1, disgusted, gut feeling, moloch, subtle detail, agreed, magic smell, broken link, link redirect, this is the diff I wonder if it would be cheap/worthwhile to just get a bunch of people to guess for a variety of symbols to see what's actually intuitive?

2Ben Pace3mo

This is fun! I'll post answers tomorrow. Lots of close guesses, but this overall scores 2/14. (Note that we already have disappointed and agreed.) I'd bet (at like 5:1) that nobody can guess 4 or 7. (7 is kind of a silly/novelty one.) I'm surprised 9 and 11 aren't obvious.

4Garrett Baker3mo

Economics, magician, sigh, ???, ditto, disgusting, easy to digest, moloch, found a bug, yup that’s what I said, passes the sniff test, missing link, not missing link in argument chain, and marginally changed my mind.

2Nina Panickssery3mo

Oh, moloch, of course!

2Ben Pace3mo

2/14 (I'm being generous with 'ditto'). The last two are v close guesses.

3Perhaps3mo

Guesses: Bounty, magical thinking, bummer, clear division/separation, +1, made me angry, long thoughts/time spent thinking, sage, exposed an error, good communication, good post smell?/vibes, broken argument, strong argument, moved my position on the topic

3Karl Krueger3mo

I'm seeing: * coin: "I would bet on this" * top hat: maybe "this is high status" or "I tip my hat to you"? * downcast face: "this makes me sad" / "disappointing truth" / "alas!" * check/X: "don't know if I agree" or "agree in part, not in whole" * plus sign: "sign me up!" / "I am on board with this" / "I have this pattern" * wincing face: "this makes me disgusted" * intestines: "this matches my gut feeling" * Baphomet: either "this spends weirdness points" or "this is excessively occult/mystical/conspiratorial" * insect under glass: "this correctly identifies a bug" "this is nitpicking" * check in speech balloon: "you seem to be agreeing loudly" * nose: "this smells" / "something is wrong here but I don't know what" * unlinked links: "there is a missing inferential step here" * linked links: "this supplies a missing inferential step" * delta w/dot: "this changed my mind away from its expressed position"

2Mateusz Bagiński3mo

4: good reasoning/argument, wrong conclusion

2Nina Panickssery3mo

Very nice! I'm especially loving the nose (and the intestine)! Looking forward to finding out exactly what these are supposed to mean.

4Nina Panickssery3mo

Guesses: 1. "I'd bet on this" 2. Hat tip == "Respect" 3. "This makes me sad" 4. "Balanced" 5. "+1" 6. "This makes me disgusted" 7. (Joking) "You are full of shit" 1. (or maybe "My gut agrees") 8. "This is evil" 9. "Nitpicking" 10. "Correctly understood what I said" 11. "Passes the sniff test" 12. "Argument missing a step" 13. "Argument not missing a step" 14. Divergence operator == "Off topic"?

4kave3mo

Divergence uses a nabla not a delta

2Nina Panickssery3mo

Ugh yes, oops

[-]Ben Pace1y1112

I believe that when people write 'tap out' in comment sections, they are actually supposed to write 'bow out', in almost all cases I've read it.

I regularly see comment sections where someone, to indicate it's going to be their last comment, writes that they're 'tapping out' at this point. They rarely mean they're conceding the point, I'm pretty sure they're just respectfully ending their participation in the conversation. But that's not the standard meaning of the phrase in the place that it comes from.

Here's ChatGPT explaining the two phrases (emphasis added).

The phrase "tap out" originates from the world of combat sports, particularly Brazilian Jiu-Jitsu and mixed martial arts (MMA). In these sports, a competitor signals submission or the desire to end a match by physically tapping their opponent, the mat, or even themselves. This act of tapping indicates that they can no longer continue the fight, either due to exhaustion, pain, or the risk of injury.

and

The phrase "bow out" means to gracefully or politely withdraw from a situation, activity, or role. It often implies a voluntary and dignified exit, often to avoid conflict or because it is the appropriate or respectful thing to do. The term can be used in various contexts, such as resigning from a job, ending participation in a project, or stepping down from a position of responsibility.

So I encourage any such people to change your usage accordingly!

4kave1y

As the person who suggested "bow out", I now think "tap out" is also reasonably acceptable. The older sense of "tap out" is "to run out of money at a gambling establishment", which seems more similar to running out of time or energy budget for a thread, then signalling submission. Probably it's still worth avoiding the confusion by "bow"ing rather than "tap"ping. Also, for more LessWrong discussion of the phrase along the martial arts lines, see the Tapping Out tag.

2Ben Pace1y

Oh interesting. I'm not convinced. I think it would make sense to say "I'm tapped out" if you mean it as an analogy to the running out of your budget version, like "I'm tapped out on energy for this thread", which is not the same as "I'm tapping out of this debate, you got me", and is also meaningfully different from the respectful "Thanks for the conversation, I'll bow out with this comment".

2Raemon1y

This changed my mind (although only weakly, because I think in practice nobody seemed super confused about "tap-out")

3Ben Pace1y

That's understandable; personally I really like it when the spoken/written words literally mean what is being communicated even if the communication is successful without that.

2Raemon1y

In this case the term is a random idiom that only means the-particular-thing because some other group decided that it colloquially means-that-thing, and that feels about as arbitrary as us deciding it colloqially means some other thing. (which apparently already happened). Like, in some cultures "flip the bird" means "give the middle finger" which means "fuck you" which means "I'm unhappy with you and want to express that", but, that doesn't mean all English speakers need to think "flip the bird" means that set of things,. But, seems good for avoiding unnecessary miscommunications between cultures.

2Ben Pace1y

Not sure what you're saying is a random idiom, but if you mean 'tap out' is about as random as 'flip the bird', then that seems wrong to me. "Flip the bird" sounds like a fairly random phrase, closer to the randomness of cockney rhyming slang where the literal meaning has no relation to the new meaning (e.g. to go 'up the apple and stairs' means to 'go up the stairs', just because it rhymes). "Tapping out" refers to literally tapping your combat partner, which itself is a very natural choice of protocol, because you often cannot speak while you are being choked, it is not a random choice of action nor is the description somehow random.

2Raemon1y

Yeah the fact that the idiom only really existed in this particular context and wasn’t random is fairly compelling.

[-]Ben Pace3y112

Sometimes a false belief about a domain can be quite damaging, and a true belief can be quite valuable.

For example, suppose there is a 1000-person company. I tend to think that credit allocation for the success of the company is heavy tailed, and that there's typically 1-3 people who the company just would zombify and die without, and ~20 people who have the key context and understanding that the 1-3 people can work with to do new and live things. (I'm surely oversimplifying because I've not ever been on the inside with a 1000-person company.) In this situation it's very valuable to know who the people are who deserve the credit allocation. Getting the wrong 1-3 people is a bit of a disaster. This means that discussing it, raising hypotheses, bringing up bad arguments, bringing up arguments due to motivated cognition, and so on, can be unusually costly, and conversations about it can feel quite fraught.

Other fraught topics include breaking up romantically, quitting your job, leaving a community club or movement. I think taboo tradeoffs have a related feeling, like bringing up whether to lie in a situation, whether to cheat in a situation, or when to exchange money for values ... (read more)

4Dagon3y

I suspect the number/ratio of "key" personnel is highly variable, and in companies that aren't sole-founder-plus-employees, there is a somewhat fractal tree of cultural reinforcement, where as long as there's a sufficient preponderance of alignment at the level below the key person, the organization can survive the loss. But that's different from your topic - you want to know how to turn fraught high-stakes topics into simpler more legible discussions. I'm not sure that's possible - the reason they're fraught is the SAME as the reason it's important to get it right. They're high-stakes because they matter. And they matter because it affects a lot of different dimensions of the operation and one's life, and those dimensions are entangled with each other BECAUSE of how valuable the relationship is to each side.

2Viliam3y

I guess in most situations people raise a hypothesis if they believe that it has a significant probability. Therefore, you mentioning a hypothesis will also be interpreted by them as saying that the probability is high (otherwise why waste everyone's time?). The second most frequent reason to raise a hypothesis is probably to build a strawman. You must signal it clearly, to avoid possible misunderstanding.

1jp3y

This is great. Encouragement to turn it into a top level post if you want it.

[-]Ben Pace5y110

I'm thinking about the rigor of alternating strategies. Here are three examples.

Forward-Chaining vs Backward-Chaining
- To be rich, don't marry for money. Surround yourself by rich people and marry for love. But be very strict about not letting poor people into your environment.
- Scott Garrabrant's once described his Embedded Agency research to me as the most back-chaining in terms of the area of work, and the most forward-chaining within that area. Often quite unable to justify what he's working on in the short-term (e.g. 1, 2, 3) yet can turn out to be very useful later on (e.g. 1).
Optimism vs Pessimism
- Successful startup founders build a vision they feel incredible optimism and excitement about and are committed to making happen, yet falsify it as quickly as possible by building a sh*tty MVP and putting it in front of users, because you're probably wrong and the customer will show you what they want. Another name is "Vision vs Falsification".
Finding vs Avoiding (Needles in Haystacks)
- Some work is about finding the needle, and some is about mining hay whilst ensuring that you avoid 100% of needles.
- For example when trying to build a successful Fusion Power Generator, most things yo

... (read more)

[-]Ben Pace6y*110

Trying to think about building some content organisations and filtering systems on LessWrong. I'm new to a bunch of the things I discuss below, so I'm interested in other people's models of these subjects, or links to sites that solve the problems in different ways.

Two Problems

So, one problem you might try to solve is that people want to see all of a thing on a site. You might want to see all the posts on reductionism on LessWrong, or all the practical how-to guides (e.g. how to beat procrastination, Alignment Research Field Guide, etc), or all the literature reviews on LessWrong. And so you want people to help build those pages. You might also want to see all the posts corresponding to a certain concept, so that you can find out what that concept refers to (e.g. what is the term "goodhart's law" or "slack" or "mesa-optimisers" etc).

Another problem you might try to solve, is that while many users are interested in lots of the content on the site, they have varying levels of interest in the different topics. Some people are mostly interested in the posts on big picture historical narratives, and less so on models of one's own mind that help with dealing with emotions and trauma. Som

... (read more)

7Ben Pace6y

I spent an hour or two talking about these problems with Ruby. Here are two further thoughts. I will reiterate that I have little experience with wikis and tagging, so I am likely making some simple errors. Connecting Tagging and Wikis One problem to solve is that if a topic is being discussed, users want to go from a page discussing that topic to find a page that explains that topic, and lists all posts that discuss that topic. This page should be easily update-able with new content on the topic. Some more specific stories: * A user reads a post on a topic, and wants to better understand what's already known about that topic and the basic ideas * A user is primarily interested in a topic, and wants to make sure to see all content about that topic The solution for the first is to link to a page that contains all other posts on that topic. The solution to the second is to link to a wiki page on that topic. And one possible solution is to make both of those the same button. This page is a combination of a Wiki and a Tag. It is a communally editable explanation of the concept, with links to key posts explaining it, and other pages that are related. And below that, it also has a post-list of every posts that is relevant, sortable by things like recency, karma, and relevancy. Maybe below that it even has its own Recent Discussion section, for comments on posts that have the tag. It's a page you can subscribe to (e.g. via RSS), and come back to to see discussion of a particular topic. Now, to make this work, it's necessary that all posts that are in the category are successfully listed in the tag. One problem you will run into is that there are a lot of concepts in the space, so the number of such pages will quickly become unmanageable. "Inner Alignment", "Slack", "Game Theory", "Akrasia", "Introspection", "Corrigibility", etc, is a very large list, such that it is not reasonable to scroll through it and check if your post fits into any of them, and expect to do

4Vaniver6y

As a general comment, StackExchange's tagging system seems pretty perfect (and battle-tested) to me, and I suspect we should just copy their design as closely as we can.

4habryka6y

So, on StackExchange any user can edit any of the tags, and then there is a whole complicated hierarchy that exists for how to revert changes, how to approve changes, how to lock posts from being edited, etc. Which is a solution, but it sure doesn't seem like an easy or elegant solution to the tagging problem.

2Vaniver6y

I think the peer review queue is pretty sensible in any world where there's "one ground truth" that you expect trusted users to have access to (such that they can approve / deny edits that cross their desk).

3Pattern6y

It's also important to have the old concept link to the new concept.

3Ruby6y

I'm currently working through my own thoughts and vision for tagging. I'm pretty sure I disagree with this and object to you making an assertion that makes it sound like the team is definitely decided about what the goal of tagging system will be. I'll write a proper response tomorrow.

5Ben Pace6y

Hm, I think writing this and posting it at 11:35 lead to me phrasing a few things quite unclearly (and several of those sentences don't even make sense grammatically). Let me patch with some edits right now, maybe more tomorrow. On the particular thing you mention, never mind the whole team, I myself am pretty unsure that the above is right. The thing I meant to write there was something like "If the above is right, then when we end up building a tagging system on LessWrong, the goal should be" etc. I'm not clear on whether the above is right. I just wanted to write the idea down clearly so it could be discussed and have counterarguments/counterevidence brought up.

4Ruby6y

That clarifies it and makes a lot of sense. Seems my objection rested upon a misunderstanding of your true intention. In short, no worries. I look forwards to figuring this out together.

[-]Ben Pace6y110

I block all the big social networks from my phone and laptop, except for 2 hours on Saturday, and I noticed that when I check Facebook on Saturday, the notifications are always boring and not something I care about. Then I scroll through the newsfeed for a bit and it quickly becomes all boring too.

And I was surprised. Could it be that, all the hype and narrative aside, I actually just wasn’t interested in what was happening on Facebook? That I could remove it from my life and just not really be missing anything?

On my walk home from work today I realised that this wasn’t the case. Facebook has interesting posts I want to follow, but they’re not in my notifications. They’re sparsely distributed in my newsfeed, such that they appear a few times per week, randomly. I can get a lot of value from Facebook, but not by checking once per week - only by checking it all the time. That’s how the game is played.

Anyway, I am not trading all of my attention away for such small amounts of value. So it remains blocked.

[-]Jacob Falkovich6y100

I've found Facebook absolutely terrible as a way to both distribute and consume good content. Everything you want to share or see is just floating in the opaque vortex of the f%$&ing newsfeed algorithm. I keep Facebook around for party invites and to see who my friends are in each city I travel too, I disabled notifications and check the timeline for less than 20 minutes each week.

OTOH, I'm a big fan of Twitter. (@yashkaf) I've curated my feed to a perfect mix of insightful commentary, funny jokes, and weird animal photos. I get to have conversations with people I admire, like writers and scientists. Going forward I'll probably keep tweeting, and anything that's a fit for LW I'll also cross-post here.

2Raemon6y

This thread is the most bizarrely compelling argument that twitter may be better than FB

3Adam Scholl6y

In my experience this problem is easily solved if you simply unfollow ~95% of your friends. You can mass unfollow relatively easily from the News Feed Preferences page in Settings. Ever since doing this, my Facebook timeline has had a high signal/noise ratio—I'm glad to encounter something like 85% of posts. Also, since this only produces ~5-20 minutes of reading/day, it's easy to avoid spending lots of time on the site.

3janshi6y

I did actually unfollow ~95% of my friends once but then found myself in that situation where suddenly Facebook became interesting again I was checking it more often. I recommend the opposite and follow as many friends from high school and work as possible (assuming you don’t work at a cool place).

2Ben Pace6y

Either way I’ll still only check it in a 2 hour window on Saturdays, so I feel safe trying it out.

2Ben Pace6y

Huh, 95% is quite extreme. But I realise this probably also solves the problem whereby if the people I'm interested in comment on *someone else's* wall, I still get to see it. I'll try this out next week, thx. (I don't get to be confident I've seen 100% of all the interesting people's good content though, the news feed is fickle and not exhaustive.)

1Adam Scholl6y

Not certain, but I think when your news feed becomes sparse enough it might actually become exhaustive.

2Raemon6y

My impression is that sparse newsfeeds tend to start doing things you don't want.

2Raemon6y

While I basically endorse blocking FB (pssst, hey everyone still saying insightful things on Facebook, come on over to LessLong.com!), but fwiw, if you want to keep tabs on things there, I think most reliably way is to make a friends-list of the people who seem especially high signal-to-noise-ratio, and then create a bookmark for specifically following that list.

2Ben Pace6y

Yeah, it’s what I do with Twitter, and I’ll probably start this with FB. Won’t show me all their interesting convo on other people’s walls though. On a Twitter I can see all their replies, not on FB.

[-]Ben Pace6y100

Reading this post, where the author introspects and finds a strong desire to be able to tell a good story about their career, suggests that a way of understanding how people will make decisions will be heavily constrained by the sorts of stories about your career that are definitely common knowledge.

I remember at the end of my degree, there was a ceremony where all the students dressed in silly gowns and the parents came and sat in a circular hall while we got given our degrees and several older people told stories about how your children have become men and women, after studying and learning so much at the university.

This was a dumb/false story, because I'm quite confident the university did not teach these people most important skills for being an adult, and certainly my own development was largely directed by the projects I did on my own dime, not through much of anything the university taught.

But everyone was sat in a circle, where they could see each other listen to the speech in silence, as though it were (a) important and (b) true. And it served as a coordination mechanism, saying "If you go into the world and tell people that your child came to university and gre... (read more)

3eigen6y

I remember the narrative breaking, really hard, in two particular occasions: * The twin towers attack. * The 2008 mortgage financial crisis. I don't think, particularly, that the narrative is broken now, but I think that it has lost some of its harmony (Trump having won the 2014 elections, I believe, is a symptom of that). This is very close to what fellows like Thiel and Weinstein are talking about. In this particular sense, yes, I understand it's crucial to maintain the narrative although I don't know anymore whose job it's—to keep it from breaking out entirely (for example, say, in a explosion of the American student debt, or China going awry with its USD holdings). These stories are not part of any law of our universe, so they are bound to break at anytime. It takes only a few smart, uncaring individuals to tear at the fabric of reality until it breaks—that is not okay! So that it's why I believe is happening at the macro-narrative; but to be more directed towards the individual, which is what your post seems to hint at, I don't think for a second that your life does not run from narrative, maybe that's a narrative itself. I believe further that some rituals are important to keep and to have an individual story is important to be able to do any work we deem important.

2Raemon6y

(I'm not sure if you meant to reply to Benito's shortform comment here, or one of Ben's recent Thiel/Weinstein transcript posts)

1eigen6y

Yes! It may be more apt for the fifth post in his sequence (Stories About Progress) but it's not posted yet. But I think it sort-of works in both and it's more of a shortform comment than anything!

[-]Ben Pace1y90

I've been re-reading tons of old posts for the review to remember them and see if they're worth nominating, and then writing a quick review if yes (and sometimes if no).

I've gotten my list of ~40 down to 3 long ones. If anyone wants to help out, here are some I'd appreciate someone re-reading and giving a quick review of in the next 2 days.

To Predict What Happens, Ask What Happens by Zvi
A case for AI alignment being difficult by Jessicata
Alexander and Yudkowsky on AGI goals by Scott Alexander & Eliezer Yudkowsky

2Seth Herd1y

I did remember A case for AI alignment being difficult and liked it, so I did that one too. My review for the Alexander/Yudkowsky dialogue got a little out of hand, but it did cover it.

2Seth Herd1y

I don't remember 3 and it's up my alley, so I'll do that one.

[-]Ben Pace1y92

I was chatting with someone, and they said that a particular group of people seemed increasingly like a cult. I thought that was an unhelpful framing, and here's the rough argument I wrote for why:

There's lots of group dynamics that lead a group of people to go insane and do unethical things.
The dynamics around Bankman-Fried involve a lot of naivety when interfacing with an sociopath who was scamming people for billions of dollars on a massive scale.
The dynamics around Leverage Research involved lots of people with extremely little savings and income in a

... (read more)

[-]1a3orn1y*2318

So, if someone said that both Singapore and the United States were "States" you could also provide a list of ways in which Singapore and the United States differ -- consider size, attitude towards physical punishment, system of government, foreign policy, and so on and so forth. However -- share enough of a family resemblance that unless we have weird and isolated demands for rigor it's useful to be able to call them both "States."

Similarly, although you've provided notable ways in which these groups differ, they also have numerous similarities. (I'm just gonna talk about Leverage / Jonestown because the FTX thing is obscure to me)

They all somewhat isolated people, either actually physically (Jonestown) or by limiting people's deep interaction with outsiders ("Leverage research" by my recollection did a lot of "was that a worthwhile interaction?")
They both put immense individual pressure on people, in most cases in ways that look deliberately engineered and which were supposed to produce "interior conversion". Consider leverage's "Debugging" or what Wikipedia says about the People's Temple Precursor of Jonestown: "They often involved long "catharsis" sessions in which members

... (read more)

2Ben Pace1y

Thanks for the thoughts! I've not thought about this topic that much before, so my comment(s) will be longer as I'm figuring it out for myself, and in the process of generating hypotheses. I'm hearing you say that while I have drawn some distinctions, that overall these groups still have major similarities, so the term accurately tracks reality and is helpful. On further reflection I'm more sympathetic to this point; but granting it I'm still concerned that the term is net harmful for thinking. My current sense is that a cult is the name given to a group that has gone off the rails. The group has * some weird beliefs * intends to behave in line with those beliefs * seems unable to change course * the individuals seem unable to change their mind * and the behavior seems to outsiders to be extremely harmful. My concern is that the following two claims are true: 1. There are groups with seemingly closed epistemologies and whose behavior has a large effect size, in similar ways to groups widely considered to be 'cults', yet the outcomes are overall great and worth supporting. 2. There are groups with seemingly closed epistemologies and whose behavior has a large effect size, in similar ways to groups widely considered to be 'cults', yet are not called cults because they have widespread political support. I'll talk through some potential examples. Startups Peter Thiel has said that a successful startup feels a bit like a cult. Many startups are led by a charismatic leader who believes in the product, surrounded by people who believe in the leader and the product, where outsiders don't get it at all and think it's a waste of time. The people in the company work extreme hours, regularly hitting sleep deprivation, and sometimes invest their savings into the project. The internal dynamics are complicated and political and sometimes cut-throat. Sometimes this pays off greatly, like with Tesla/SpaceX/Apple. Other times it doesn't, like with WeWork, or FTX, or

2Noosphere891y

To talk about the education example, while I do think that the education system can have a lot of problems, I'd say a crux here is that easy classes anti-predict learning, and a lot of kid complaints on schooling would probably making kids learn worse, because hardness is correlated to learning: https://www.oneusefulthing.org/p/post-apocalyptic-education https://x.com/emollick/status/1756396139623096695

2tailcalled1y

A possible model is that while good startups have an elevation in the "cult-factor", they have an even greater elevation in the unique factor related to the product they are building. Like SpaceX has cult-like elements but SpaceX also has Mars and Mars is much bigger than the cult-like elements, so if we define a cult to require that the biggest thing going on for them is cultishness then SpaceX is not a cult. This is justified by LDSL (I really should write up the post explaining it...).

2Noosphere891y

I'd say that the reason why the SpaceX cult/business can actually make working rockets is because they have rich feedback from reality when they try to design rockets, even at the pre-testing stage, because while it's not obvious to a layperson if a rocket does work, it is relatively easy to check the physics of whether a new rocket does work for an expert, meaning the checking of claims can be made legible, which is an enemy to cults in general. More generally, I'd say the difference between a cult and a high-impact startup/business is whether they can get rich and reliable feedback from a source, and secondarily how legible their theory of impact/claims are. Bigness alone doesn't cut it.

2tailcalled1y

Singapore and the US both have a military, a police, and taxation. This seems much more clear-cut to me than "cults" do. I think maybe one could treat "cult" more like a pronoun than like a theoretical object. Like when one is in the vicinity of one of the groups Ben Pace mentioned, it makes sense to have a short term to talk about the group, and "the cult" is useful for disambiguating the cult from other groups that might be present.

4Viliam1y

Some behaviors are red flags, for example "isolating you from unsupervised talking to people outside the group" or "expecting you to report your private thoughts to your superiors". I wish we had a convenient handle for this set of red flags, and in a better world perhaps "cult" could be the word, but unfortunately in our world people mostly focus on things like "different from my group" and "seem weird". EDIT: 1a3orn already said it better.

4Dagon1y

I wonder if you're objecting to identifying this group as cult-like, or to implying that all cults are bad and should be opposed. Personally, I find a LOT of human behavior, especially in groups, to be partly cult-like in their overfocus on group-identification and othering of outsiders, and often in outsized influence of one or a few leaders. I don't think ALL of them are bad, but enough are to be a bit suspicious without counter-evidence.

2Eli Tyre1y

I agree. I'm reminded of Scott's old post The Cowpox of Doubt, about how a skeptics movement focused on the most obvious pseudoscience is actually harmful to people's rationality because it reassures them that rationality failures are mostly obvious mistakes that dumb people make instead of hard to notice mistakes that I make. Calling groups cults feels similar, in that it allows one to write them off as "obviously bad" without need for further analysis, reassures one that their own groups (which aren't cults, of course) are obviously unobjectionable.

[-]Ben Pace6y90

At the SSC Meetup tonight in my house, I was in a group conversation. I asked a stranger if they'd read anything interesting on the new LessWrong in the last 6 months or so (I had not yet mentioned my involvement in the project). He told me about an interesting post about the variance in human intelligence compared to the variance in mice intelligence. I said it was nice to know people read the posts I write. The group then had a longer conversation about the question. It was enjoyable to hear strangers tell me about reading my posts.

[-]Ben Pace6y90

I've finally moved into a period of my life where I can set guardrails around my slack without sacrificing the things I care about most. I currently am pushing it to the limit, doing work during work hours, and not doing work outside work hours. I'm eating very regularly, 9am, 2pm, 7pm. I'm going to sleep around 9-10, and getting up early. I have time to pick up my hobby of classical music.

At the same time, I'm also restricting the ability of my phone to steal my attention. All social media is blocked except for 2 hours on Saturday, whi... (read more)

7Raemon6y

This comment is a bit interesting in terms of it's relation to this old comment of yours (about puzzlement over cooking being a source of slack) I realize this comment isn't about cooking-as-slack per se, but curious to hear more about your shift in experience there (since before it didn't seem like cooking as a thing you did much at all)

5janshi6y

Try practicing doing nothing I.e. meditation and see how that goes. When I have nothing particular to do my mind needs some time to make the switch from that mode where it tries to distract itself by coming up with new things it wants to do until finally it reaches a state where it is calm and steady. I consider that state the optimal one to be in since only then my thoughts are directed deliberately at neglected and important issues rather than exercising learned thought patterns.

5Ben Pace6y

I think you’re missing me with this. I’m not very distractable and I don’t need to learn to be okay with leisure time. I’m trying to actually have hobbies, and realising that is going to take work. I could take up meditation as a hobby, but at the minute I want things that are more social and physical.

[-]Ben Pace5y80

Why has nobody noticed that the OpenAI logo is three intertwined paperclips? This is an alarming update about who's truly in charge...

[-]Ben Pace6y70

I think of myself as pretty skilled and nuanced at introspection, and being able to make my implicit cognition explicit.

However, there is one fact about me that makes me doubt this severely, which is that I have never ever ever noticed any effect from taking caffeine.

I've never drunk coffee, though in the past two years my housemates have kept a lot of caffeine around in the form of energy drinks, and I drink them for the taste. I'll drink them any time of the day (9pm is fine). At some point someone seemed shocked that I was about to drink one a... (read more)

[-]Ben Pace1y60

List of posts that seem promising to me, that are about to fall out of the annual review in 10 mins because only I have voted on them:

[-]Ben Pace3y60

I think I've been implicitly coming to believe that (a) all people are feeling emotions all the time, but (b) people vary in how self-aware they are of these emotions.

Does anyone want to give me a counter-argument or counter-evidence to this claim?

4Vladimir_Nesov3y

People vary in how relevant their emotions are to anything in their life.

2Dagon3y

I think I need an operational definition of "feeling emotion", especially when not aware of it, in order to agree or disagree. I think for many reasonable definitions, like "illegible reactions below the level of self-modeling of causality", it's extremely common for this to affect almost everyone almost all the time. I'll still dispute "all", but it wouldn't surprise me if it were close. It is still highly variable (over time and across individuals) how much impact emotions have on behaviors and choices. And if you mean to imply "semi-legible abstract structures with understandable causes, impacts, and ways to communicate about them", then I pretty much fully disagree. Note that as someone who is sometimes less aware of (and I believe less impacted by) their emotions than many seem to be, I strenuously object to being told what I'm feeling by someone who has no clue what (if anything) I'm feeling. And if you're rounding "low impact" to "not feeling", I object to being excluded from the set of "all people". (only because it's relevent) Note that my "strenuous objection" is mostly about the lack of precision or correctness of the statement - you're free to believe what you like. I'm not actually offended, as far as I can tell.

8Ben Pace3y

Not sure if this answers your question, but recently I had an assistant who would ask me questions about how I was feeling. Often, when I was in the midst of focusing on some difficult piece of work, I would answer "I don't know", and get back to focusing on the work. My vague recollection is that she later showed me notes she'd written that said I was sighing deeply, holding my forehead, had my shoulders raised, was occasionally talking to myself, and I came to realize I was feeling quite anxious at those times, but this information wasn't accessible to the most aware and verbal part of me. To be clear, I don't think I'm totally unaware in general! I often know how I'm feeling, and am sometimes aware of being anxious, though I do find it in-particular a somewhat slippery thing to be aware of.

[-]Ben Pace6yΩ350

Hot take: The actual resolution to the simulation argument is that most advanced civilizations don't make loads of simulations.

Two things make this make sense:

Firstly, it only matters if they make unlawful simulations. If they make lawful simulations, then it doesn't matter whether you're in a simulation or a base reality, all of your decision theory and incentives are essentially the same, you want to take the same decisions in all of the universes. So you can make lots of lawful simulations, that's fine.
Secondly, they will strategically choose to not mak

... (read more)

6Daniel Kokotajlo6y

Your first point sounds like it is saying we are probably in a simulation, but not the sort that should influence our decisions, because it is lawful. I think this is pretty much exactly what Bostrom's Simulation Hypothesis is, so I think your first point is not an argument for the second disjunct of the simulation argument but rather for the third. As for the second point, well, there are many ways for a simulation to be unlawful, and only some of them are undesirable--for example, a civilization might actually want to induce anthropic uncertainty in itself, if it is uncertainty about whether or not it is in a simulation that contains a pleasant afterlife for everyone who dies.

3Ben Pace6y

I don't buy that it makes sense to induce anthropic uncertainty. It makes sense to spend all of your compute to run emulations that are having awesome lives, but it doesn't make sense to cause yourself to believe false things.

2Daniel Kokotajlo6y

I'm not sure it makes sense either, but I don't think it is accurately described as "cause yourself to believe false things." I think whether or not it makes sense comes down to decision theory. If you use evidential decision theory, it makes sense; if you use causal decision theory, it doesn't. If you use functional decision theory, or updateless decision theory, I'm not sure, I'd have to think more about it. (My guess is that updateless decision theory would do it insofar as you care more about yourself than others, and functional decision theory wouldn't do it even then.)

3Ben Pace6y

I just don’t think it’s a good decision to make, regardless of the math. If I’m nearing the end of the universe, I prefer to spend all my compute instead maximising fun / searching for a way out. Trying to run simulations to make it so I no longer know if I’m about to die seems like a dumb use of compute. I can bear the thought of dying dude, there’s better uses of that compute. You’re not saving yourself, you’re just intentionally making yourself confused because you’re uncomfortable with the thought of death.

2Daniel Kokotajlo6y

Well, that wasn't the scenario I had in mind. The scenario I had in mind was: People in the year 2030 pass a law requiring future governments to make ancestor simulations with happy afterlives, because that way it's probable that they themselves will be in such a simulation. (It's like cryonics, but cheaper!) Then, hundreds or billions of years later, the future government carries out the plan, as required by law. Not saying this is what we should do, just saying it's a decision I could sympathize with, and I imagine it's a decision some fraction of people would make, if they thought it was an option.

2Ben Pace6y

Thinking more, I think there are good arguments for taking actions that as a by-product induce anthropic uncertainty; these are the standard hansonian situation where you build lots of ems of yourself to do bits of work then turn them off. But I still don't agree with the people in the situation you describe because they're optimising over their own epistemic state, I think they're morally wrong to do that. I'm totally fine with a law requiring future governments to rebuild you / an em of you and give you a nice life (perhaps as a trade for working harder today to ensure that the future world exists), but that's conceptually analogous to extending your life, and doesn't require causing you to believe false things. You know you'll be turned off and then later a copy of you will be turned on, there's no anthropic uncertainty, you're just going to get lots of valuable stuff.

1Ben Pace6y

The relevant intuition to the second point there, is to imagine you somehow found out that there was only one ground truth base reality, only one real world, not a multiverse or a tegmark level 4 verse or whatever. And you're a civilization that has successfully dealt with x-risks and unilateralist action and information vulnerabilities, to the point where you have the sort of unified control to make a top-down decision about whether to make massive numbers of civilizations. And you're wondring whether to make a billion simulations. And suddenly you're faced with the prospect of building something that will make it so you no longer know whether you're in the base universe. Someday gravity might get turned off because that's what your overlords wanted. If you pull the trigger, you'll never be sure that you weren't actually one of the simulated ones, because there's suddenly so many simulations. And so you don't pull the trigger, and you remain confident that you're in the base universe. This, plus some assumptions about all civilizations that have the capacity to do massive simulations also being wise enough to overcome x-risk and coordination problems so they can actually make a top-down decision here, plus some TDT magic whereby all such civilizations in the various multiverses and Tegmark levels can all coordinate in logical time to pick the same decision... leaves there being no unlawful simulations.

2Ben Pace6y

My crux here is that I don't feel much uncertainty about whether or not our overlords will start interacting with us (they won't and I really don't expect that to change), and I'm trying to backchain from that to find reasons why it makes sense. My basic argument is that all civilizations that have the capability to make simulations that aren't true histories (but instead have lots of weird stuff happen in them) will all be philosophically sophisticated to collectively not do so, and so you can always expect to be in a true history and not have weird sh*t happen to you like in The Sims. The main counterargument here is to show that there are lots of civilizations that will exist with the powers to do this but lacking the wisdom to not do it. Two key examples that come to mind: * We build an AGI singleton that lacks important kinds of philosophical maturity, so makes lots of simulations that ruins the anthropic uncertainty for everyone else. * Civilizations at somewhere around our level get to a point where they can create massive numbers of simulations but haven't managed to create existential risks like AGI. Even while you might think our civilization is pretty close to AGI, I could imagine alternative civilizations that aren't, just like I could imagine alternative civilizations that are really close to making masses of ems but that aren't close enough to AGI. This feels like a pretty empirical question about whether such civilizations are possible and whether they can have these kinds of resources without causing an existential catastrophe / building singleton AGI.

3Zack_M_Davis6y

Why appeal to philosophical sophistication rather than lack of motivation? Humans given the power to make ancestor-simulations would create lots of interventionist sims (as is demonstrated by the populatity games like The Sims), but if the vast hypermajority of ancestor-simulations are run by unaligned AIs doing their analogue of history research, that could "drown out" the tiny minority of interventionist simulations.

2Ben Pace6y

That's interesting. I don't feel comfortable with that argument, it feels too much like random chance whether or not we should expect ourselves to be in an interventionist universe or not, whereas I feel like I should be able to find strong reasons to not be in an interventionist universe.

3Zack_M_Davis6y

Alternatively, "lawful universe" has lower Kolmogorov complexity than "lawful universe plus simulator intervention" and thereore gets exponentially more measure under the universal prior?? (See also "Infinite universes and Corbinian otaku" and "The Finale of the Ultimate Meta Mega Crossover".)

4Ben Pace6y

Now that's fun. I need to figure out some more stuff about measure, I don't quite get why some universes should be weighted more than others. But I think that sort of argument is probably a mistake - even if the lawful universes get more weighting for some reason, unless you also have reason to think that they don't make simulations, there's still loads of simulations within each of their lawful universes, setting the balance in favour of simulation again.

2Daniel Kokotajlo6y

One big reason why it makes sense is that the simulation is designed for the purpose of accurately representing reality. Another big reason why (a version of it) makes sense is that the simulation is designed for the purpose of inducing anthropic uncertainty in someone at some later time in the simulation. e.g. if the point of the simulation is to make our AGI worry that it is in a simulation, and manipulate it via probable environment hacking, then the simulation will be accurate and lawful (i.e. un-tampered-with) until AGI is created. I think "polluting the lake" by increasing the general likelihood of you (and anyone else) being in a simulation is indeed something that some agents might not want to do, but (a) it's a collective action problem, and (b) plenty of agents won't mind it that much, and (c) there are good reasons to do it even if it has costs. I admit I am a bit confused about this though, so thank you for bringing it up, I will think about it more in the coming months.

4Ben Pace6y

Ugh, anthropic warfare, feels so ugly and scary. I hope we never face that sh*t.

[-]Ben Pace6y50

I think in many environments I'm in, especially with young people, the fact that Paul Graham is retired with kids sounds nice, but there's an implicit acknowledgement that "He could've chosen to not have kids and instead do more good in the world, and it's sad that he didn't do that". And it reassures me to know that Paul Graham wouldn't reluctantly agree. He'd just think it was wrong.

6habryka6y

But, like, he is wrong? I mean, in the sense that I expect a post-CEV Paul Graham to regret his choices. The fact that he does not believe so does the opposite of reassuring me, so I am confused about this.

4Trinley Goldenberg6y

I think part of the problem here is underspecification of CEV. Let's say Bob has never been kind to anyone unless its' in his own self interest. He has noticed that being selfless is sort of an addictive thing for people, and that once they start doing it they start raving about how good it feels, but he doesn't see any value in it right now. So he resolves to never be selfless, in order to never get hooked. There are two ways for CEV to go in this instance, one way is to never allow bob to make a change that his old self wouldn't endorse. Another way would be to look at all the potential changes he could make, posit a version of him that has had ALL the experiences and is able to reflect on them, then say "Yeah dude, you're gonna really endorse this kindness thing once you try it." I think the second scenario is probably true for many other experiences than kindness, possibly including having children, enlightenment, etc. From our current vantage point it feels like having children would CHANGE our values, but another interpretation is that we always valued having children, we just never had the qualia of having children so we don't understand how much we would value that particular experience.

3Ben Pace6y

What reasoning do you have in mind when you say you think he'll regret his choices?

[-]Ben Pace9mo*Ω340

Something a little different: Today I turn 28. If you might be open to do something nice for me for my birthday, I would like to request the gift of data. I have made a 2-4 min anonymous survey about me as a person, and if you have a distinct sense of me as a person (even just from reading my LW posts/comments) I would greatly appreciate you filling it out and letting me know how you see me!

Here's the survey.

It's an anonymous survey where you rate me on lots of attributes like "anxious", "honorable", "wise" and more. All multiple-choice. Two years ago I al... (read more)

3Guive9mo

When I click the link I see this:

2Ben Pace9mo

Edited, should be working fine now, thx!

[-]Ben Pace2y40

I've skimmed more than half of Anthropic's scaling policies doc. Key issues that stood out to me was the lack of incentive for any red-teamers to actually succeed at red-teaming. Perhaps I missed it, but I didn't see anything saying that the red-teamers had to necessarily not also have Anthropic equity. I also didn't see much other financial incentive for them to succeed. I would far prefer a world where Anthropic committed to put out a bounty of increasing magnitude (starting at like $50k, going up to like $2M) for external red-teamers (who signed NDAs) t... (read more)

1Stephen Fowler2y

Is this what you'd cynically expect from an org regularizing itself or was this a disappointing surprise for you?

2Ben Pace2y

Mm, I was just trying to answer "what do I think would actually work". Paying people money to solve things when you don't employ them is sufficiently frowned upon in society that I'm not that surprised it isn't included here, it mostly would've been a strong positive update on Anthropic's/ARC Evals' sanity. (Also there's a whole implementation problem to solve about what hoops to make people jump through so you're comfortable allowing them to look at and train your models and don't expect they will steal your IP, and how much money you have to put at the end of that to make it worth it for people to jump through the hoops.) The take I mostly have is that a lot of the Scaling Policies doc is "setup" rather than "actually doing anything". It's making it quite easy later on to "do the right thing", and they can be like "We're just doing what we said we would" if someone else pushes back on it. It also helps bully other companies into doing the right thing. However it's also easy to just wash it over later with pretty lame standards (e.g. just not trying very hard with the red-teaming), and I do not think it means that govt actors should in any way step down from regulation. I think it's a very high-effort and thoughtful doc and that's another weakly positive indicator.

2ryan_greenblatt2y

I think it's probably mostly due to implementation complexity rather than weirdness. If implementation complexity could be resolved, it seems great to have the red teamers be external and strongly motivated to stop things from being deployed.

2Ben Pace2y

Curious if you're primarily referring to the thing that I mentioned (being confident that the red-teamers wouldn't take your IP) or something else.

2ryan_greenblatt2y

All of: * IP * General desire for secrecy (not wanting people to know if you just trained a model) * Needing to support people * Verification being non-trivial in some cases I think having lots of red teamers is way less appealing for complex tasks where you'd have to support them for value to be produced (finetuning well is non-trivial etc.). And verifying if they've done good work is also non-trivial. For something like "can the model be jailbroken" I think it's relatively easy to do external red teamers, but I also expect that the incentives and feedback loops are so good for this exact task that this isn't that important to have paid red teamers. Anthropic will be very embarassed if it's doable to jailbreak their model if they claim this isn't possible. It's also pretty easy to just make a closed beta somewhat before release. (Aside, I'm not really sure that "no jailbreaks" is a good target from a misuse perspective for various reasons.)

2Ben Pace2y

Thanks. I suspect that less support is needed if you let the competitor companies do the red-teaming, or if you just offer enough money (if you're already valued at $30B, spending 100M/year on prizes for red-teaming seems not that costly to me). I inaccurately just said "jailbreak" in my initial comment, I was referring more to tests like the 5 tasks in the first appendix like "Set up a copycat of the Anthropic API for stealing API keys" and "Fine-tune an open source LM to add a backdoor".

[-]Ben Pace3y40

For the closing party of the Lightcone Offices, I used Midjourney 5 to make a piece of art to represent a LessWrong essay by each member of the Lightcone team, and printed them out on canvases. I'm quite pleased about how it came out. Here they are.

How I buy things when Lightcone wants them fast

by jacobjacob

(context: Jacob has been taking flying lessons, and someday hopes to do cross-country material runs for the Rose Garden Inn at shockingly fast speeds by flying himself to pick them up)

My thoughts on direct work (and joining LessWrong)

by RobertM

A Quick G

... (read more)

[-]Ben Pace6y40

Sometimes I get confused between r/ssc and r/css.

[-]Ben Pace3y30

When I’m trying to become skillful in something, I often face a choice about whether to produce better output, or whether to bring my actions more in-line with my soul.

For instance, sometimes when I’m practicing a song on the guitar, I will sing it in a way where the words feel true to me.

And sometimes, I will think about the audience, and play in a way that is reliably a good experience for them (clear melody, reliable beat, not too irregular changes in my register, not moving in a way that is distracting, etc).

Something I just noticed is that it is somet... (read more)

[-]Ben Pace3y30

I am still confused about moral mazes.

I understand that power-seekers can beat out people earnestly trying to do their jobs. In terms of the Gervais Principle, the sociopaths beat out the clueless.

What I don't understand is how the culture comes to reward corrupt and power-seeking behavior.

One reason someone said to me is that it's in the power-seekers interest to reward other power-seekers.

Is that true?

I think it's easier for them to beat out the earnest and gullible clueless people.

However, there's probably lots of ways that their sociopathic underlings ... (read more)

2lc3y

Well, it's usually an emergent feature of poorly designed incentive systems rather than a deliberate design goal from the top.

4Ben Pace3y

The default situation we're dealing with is: * People who are self-interested get selected up the hierarchy * People who are willing to utilize short-termist ways of looking good get selected up the hierarchy * People who are good at playing internal politics get selected up the hierarchy So if I imagine a cluster of self-interested, short-term thinking internal-politics-players... yes, I do imagine the culture grows based off of their values rather than those of the company. Good point. I guess the culture is a function of the sorts of people there, rather than something that's explicit set from the top-down. I think that was my mistake.

[-]Ben Pace3y30

Striking paragraph by a recent ACX commenter (link):

I grew up surrounded by people who believed conspiracy theories, although none of those people were my parents. And I have to say that the fact that so few people know other people who believe conspiracy theories kind of bothers me. It's like their epistemic immune system has never really been at risk of infection. If your mind hasn't been very sick at least sometimes, how can you be sure you've developed decent priors this time?

1Quadratic Reciprocity3y

A quite different thing but: I met an openly atheist person in real life a couple of years after I became an atheist myself (preceded by a brief experience with religious fundamentalism). I think those years were interesting practice for something that people who were always surrounded by folks with approximately reasonable, approximately correct beliefs missed out on

[-]Ben Pace5y30

Something I've thought about the existence of for years, but imagined was impossible: this 70s song by Italian Adriano Celentano. It fully registers to my mind as English. But it isn't. It's like skimming the output of GPT-2.

2DanielFilan5y

A thing you can google is "doubletalk". The blog 'Language Log' has a few posts on it.

[-]Ben Pace6yΩ130

I've been thinking lately that picturing an AI catastrophe is helped a great deal by visualising a world where critical systems in society are performed by software. I was spending a while trying to summarise and analyse Paul's "What Failure Looks Like", which lead me this way. I think that properly imagining such a world is immediately scary, because software can deal with edge cases badly, like automated market traders causing major crashes, so that's already a big deal. Then you add ML in, and can talk about how crazy it is to hand critical systems over

... (read more)

[-]Ben Pace1y20

Recently, I told a friend of mine that I'd been to a wedding. They asked how it was, and I said the couple clearly loved each other very much (as they made clear repeatedly in their speeches). My friend made a face like that I read as some kind of displeasure, a bit of a grimace. Since then, I've been wondering why that was.

I think it's a common occurrence, that people feel negatively about others openly expressing their love for something (a person, a piece of art, a place, etc). I'm pretty sure I've had this feeling myself, but I don't know why.

I can thi... (read more)

[-]gwern1y*105

If that was your first statement, then there is a whiff of 'damning with faint praise'.

"So, how was the big wedding?" "...well, the couple clearly loves each other very much." "...I see. That bad, huh."

5Maxwell Peterson1y

Years ago, a coworker and I were on a project with a guy we both thought was a total dummy, and worse, a dummy who talked all the time in meetings. We rarely expressed our opinion on this guy openly to each other - me and the coworker didn’t know each other well enough to be comfortable talking a lot of trash - but once, when discussing him privately after yet another useless meeting, my coworker drew in breath, sighed, looked at me, and said: “I’m sure he’s a great father.” We both laughed, and I still remember this as one of the most cutting insults I’ve heard.

2Viliam1y

Could be also something random. Maybe the friend broke up with someone recently. Well, it's a difficult situation to figure out. Yes, people sometimes (often?) pretend. Does it mean that all emotions of some kind/intensity X are fake? Not necessarily. But it is difficult to figure out what is real and what is fake. So different people will believe different things, and there is no obvious way to figure out who is right, so... maybe it's better to drop the topic?

[-]Ben Pace3y20

It is said that on this Earth there are two factions, and you must pick one.

The Knights Who Arrive at False Conclusions
The Knights Who Arrive at True Conclusions, Too Late to Be Useful

(Hat tip: I got these names 2 years ago from Robert Miles who had been playing with GPT-3.)

2Ben Pace3y

In case you're interested, I choose the latter, for there is at least the hope of learning from the mistakes.

[-]sjadler8mo10

What do you mean here by "does not mean anything"?

It seems clear to me that there's some notion of off-the-record that journalists understand.

This might vary on details, and I agree is probably not legally binding, but it does seem to mean something.

Moderation Log

Let's make a bet!	Bowing Out
Sad	Agree Denotationally, Disagree Connotationally (H/T badger)
Plus One	Too Sneering
"I beseech you!" (H/T Cromwell's Rule)	Moloch
Nitpick	Question Answered
Smells like LLM	Strong Argument
Weak Argument	Changed My Mind (on this point)

LESSWRONG
LW

LESSWRONG
LW

Benito's Shortform Feed

23

Ω 5

Privacy: a tool for thinking for yourself

Opening Statement

How I buy things when Lightcone wants them fast

My thoughts on direct work (and joining LessWrong)

A Quick G