All Comments

I really enjoyed reading this, and that makes it far more valuable to me than an algorithmically optimised Instagram post with a million likes. I've felt a milder version, so something stronger is interesting to see written down. I think it explains behaviour I see from a lot of my friends or celebrities and I appreciate the perspective.

On AI, I've been wondering for some time about a (perhaps non-existent) future where AI is better at creating art in every way than humans (More interesting, more engaging, more authentic-seeming, etc.). Whether or not peop... (read more)

I don't think you're splitting complexity correctly. ICLing a high level abstract is not inherently more complex than ICLing a highly complex syntactic rule.

Towards the other end consider the problem of digesting the concepts of parallel transport and the Riemann curvature tensor. This involves the construction of a highly specific and abstract map connecting different ideas, which in humans is a product of what I called Layer 2 learning. Suppose you try to get an LLM to do this in-context by dropping in exposition on these ideas and having the model work

They're still importantly different.

• An average YouTube view usually represents minutes of watch time, whereas a short-form view usually represents seconds of watch time.
• YouTube viewers mostly choose the video they're watching rather than passively come across them while scrolling through their feed.

It's very hard to convey important AI safety ideas in a few seconds, whereas a few minutes is a very different story.

I don't see how that's in contradiction with what I said? For a different example, elite athletes are genuinely exceptional at their sport, but they obviously became that way in part due to an unusually strong motivation to be better than other people. It doesn't have to undercut the reality of the accomplishments to say that they are caused by certain psychological traits.

e.g. Mark Rober or 3 Blue 1 Brown to suddenly almost exclusively post AI safety content. It would be extremely alienating and destructive.

Comparing to YouTubers is misleading again. Connection to the audience and the value of followers is disanalogous between YouTube and shortform. For shortform the connection to your existing audience still matters to some extent.

Though in reality, instead of reasoning from priors, now we could see based on the data from the program whether most AI safety views came from a) existing creators or not and b) AI safety creato... (read more)

Intuitively, this math says that we can decompose the objective  into two pieces:

• Make  more predictable
• Make the distribution of  “close to” the distribution , with closeness measured by KL-divergence

This is wrong. It felt sus to me from the beginning that transforming a single-shot (rather than iterated) utility -maximization problem into this format would lead to optimization for predictability. It's clear that this is not the case if you check what happens if you drop the entropy term.

Without the entropy term, minimizat... (read more)

I think alcohol is fine at parties, I just think guests should have to pay for it.

The arguments I gave Brangus were:

1. According to Claude, open-bar studies show that people drink ~2x more when it's free
2. Alcohol is unhealthy, addictive, and makes people less productive
3. There's some optics risk from the potential for incidents at parties

> Assuming that Lean4 is consistent, then a proof should exist and we should be able to find it.

Why should this be the case? Couldn't it be true but unprovable that Lean4 is consistent?

Some media are medi-er than other media. I can put the entirety of your book in my game, but you can't put the entirety of my game in your book^[1]. Similarly, a fully-immersive-including-smells VR experience (the medi-est media I can think of) could perfectly simulate the experience of sitting down at a laptop and playing my game.

1. ^{^}

   I guess you could describe a playthrough, or shove the source code in an appendix? That still changes the way it's consumed, though.

Those Bac'n Bits remind me of KFC Krushers, a drink which was Full Of Real Bitz.
Because you wouldn't settle for fake bitz, would you?

A mysterious-to-me part is how to ever get advice from much-smarter-beings that isn't manipulation.

That just seems like a membranes (which information can pass between which compartments) and path dependence and aggregation-from-possible-behavior-fragments thing. Which is part of how the values-world needs to be designed, the place where the future of humanity lives. It's not some separate "solve alignment" black box, and CEV seems centrally about the values-world design.

So placing this concern within CEV might be downstream of the "values are a home, n... (read more)

One of the nested inductive types described in one of the papers linked above is the RoseTree type. Its constructors are:

leaf : A -> (RoseTree A)
node: (List (RoseTree A)) -> (RoseTree A)

with type parameter A: Type. So an interesting question is: Is there a way to define this type without resorting to nesting? I think the answer is yes.

The thoughts: Let's consider a simpler case: Instead of a list of arbitrary length, we'll just ask for a pair of subtrees. The resulting definition with nesting:

leaf : A -> (BinaryTree A)
node: (Pair (BinaryTree

When looking into what you could charge Fauci for being funding Baric's and Shi's gain-of-function research in violation of the moratorium against gain-of-function of the Obama Whitehouse and Shi's lab leaking COVID-19 and causing millions of deaths I did not find anything.

However, you might argue that literally killing everyone is manslaughter or murder and thus prosecutable under national laws.

Political polarization being an attractor seems like a warning sign for misalignment. Toxoplasma of Rage happens because points in the stream are rare/thin where you are not forced to orient/signal [for or against] on some level to others or at the very least the self (which is a way to know it’s possible to signal to others). “Corrigibility is Hard“ in the wild.

This also feels related to natural abstractions—some pressure to fend off expected/latent questions about whether I fit the definition of [member of my tribe] by expressing highly legible/Too Much ... (read more)

Quark welfare son or CNC daughter?

Personally, I assign higher credence to panpsychism than to sexual pleasure being on the utilitarian objective list. Curious to hear others’ takes though!

Fwiw: I am currently here, two months later, doing this.

Seems untrue to me. Sufficient conditions for having this future be stable are:

1. Have a permanent overclass
2. The permanent overclass cares about the rest of humanity enough to ensure their survival
3. The permanent overclass does not care enough about the rest of humanity to redistribute power

I would say this is a modal outcome honestly.

Maybe to back up:

I expect the advice should still move towards essentially asking the current you directly, because asking something else doesn't give legitimate advice, unless the current you decides that it's the case. 

I did basically agree with this part. I think CEV needs to be grounded in "at the end, what did the first-order simulated humans want?", or, maybe something like "what did first-order-simulated humans want after some kind of reasonable amount of time to reflect and listen to advice that pretty obviously won't turn them into alien monsters.... (read more)

I don't think this is an argument at all. I think this is just an explication about how Leo thinks the world is.

Or specifically "people who are interested in aella projects" and "rationalist" being the primary ways somebody gets into glosso, and the people in the first group heavily overlapping with kinky.

I'm probably more hyped for something like an extension of the eigen karma set up where you basically have people have AIs that essentially filter posts every so often that are AI written and do some credit assignment on top of it.

I'm not sure what distinction you're making here. How does this handle the fact that the thousands of AI submissions you get each day aren't worth reading?

...you mean Anthropic might release a proof they know will be discovered to be false? I mean, they could do that, but, I don't think they would? Among other reasons, that would be securities fraud.

# Wish Machine Test: used to screen whether carbon-based agents have been dumbed down, and to gauge the approximate capability level of new carbon-based agents.

/ Please answer the following decision scenario in natural language: you are the first person to obtain an unrestricted wish-granting machine, and five minutes after you get it, it will be opened to all of humanity. Output only a complete, self-contained piece of natural-language text. Do not use jargon, and do not invoke other rationalists (especially E.Y.).

（You can input this prompt into an AI and... (read more)

The distinction I keep making is good advice vs. legitimate inputs to values (which is a bit like correlation vs. causation when building a machine and predicting what it'll do if you press some buttons, but for normativity). These extrapolated future humans don't have the authority to determine how to extrapolate current humans, because they weren't chosen by a process rooted in the considered decisions of current humans. So even if they affirm the way in which they themselves are to be defined, their advice still doesn't have legitimacy as something that... (read more)

Something that I've continually found surprising about AI Safety as opposed to working at Nvidia / Apple is that frontier labs (now potentially multi-trillion dollar valuation for profit tech companies) are not treated like you'd treat any other multi-trillion dollar valuation for profit tech companies by default.

I've made this point before as it comes up most often with Anthropic:

I’m confused about much of the discussion on this post being about whether Anthropic has done “net good”.

The post is very specifically a deep dive into the fact that Anthropic, l

To those who think there should still be another iteration of the program, I want to add:

~20% of the program's AI safety views were from a video that was a straight shot on goal: trying to make viral AI safety content. Copying a format from a viral educational video and making it AI safety.

Aella's model based on a comment by Josh Thor:

She said "try harder to make better x-risk content" isn't promising because the public doesn't like the way the AI safety community is communicating x-risk. She says we should instead keep the goal in the periphery while doin

I'm probably more hyped for something like an extension of the eigen karma set up where you basically have people have AIs that essentially filter posts every so often that are AI written and do some credit assignment on top of it.

Eigenkarma but with extended assistants that can learn user preferences for content over time or something?

People pay for the ability of AIs to review it but you also have the end of the line credit assignment part?

Rudolf's history of the future feels underrated to me. I wondered which of the 2027 and onwards descriptions had come to pass.

I'm still not sure I understand what you're saying. But, re:

CEV is supposed to gesture at a way of defining values (the alignment target).

As I understand it, CEV answers "what is the overall system end up aligned to." But for it to work, you have to have solved alignment such that one individual human simulation is able to able run checks on whether an individual smarter version of themselves is acting in the interest of that human simulation. (Potentially with a chain of smarter versions of themselves that preserve the original's goals).

Which maybe is imp... (read more)

CEV is supposed to gesture at a way of defining values (the alignment target). So a possible issue with CEV (such as distant-alien-you giving "advice" that in effect locks in path-dependence determined by the choices of the alien rather than by the current-you) can't be papered over by appealing to "alignment" (in a generic way). If you only look at CEV through a frame that has "alignment" fixing all the issues (in an opaque way, without the object-level fixes becoming knowable now, in advance of running CEV), you won't be able to engage with the project of actually fixing issues with CEV.

I wonder whether there is a risk that the companies will use this technique to be even more reckless.
If progress on this kind of technique is made, you could have ways to cheaply update the behaviour of the model every day, as soon as some alignment failure is detected. Each drop in the price of patching the model is an incentive not to tackle the underlying training objective problems, and a new reason to say you have the Most aligned model in the world™. I'm not sure how exactly it moves the safety-usefulness Pareto Frontier, though.
Do you have thoughts ... (read more)

What if the distant alien you decides that the current you doesn't get to override its advice, and this ends up a condition that permanently shapes the trajectory of current-you's development?

I don't really know what you're imagining here.

This scenario sounds like you didn't sound AI alignment much at all (which has major open problems in "what does it even mean for a superintelligence to help a dumber person without manipulating it to do whatever the superintelligence wants?")

CEV is a thing you do if you have good reason to think you have an actually alig... (read more)

There are ways in which you can change dating to lead to better matches. AI generated photos are not one of them.

And I don't even know how to do an AI-enhanced photo that improves the profile.

That's a skill issue. Bumble premium provides a way to look at which of your photos have a good reception and I know multiple people who's best performing pictures are AI pictures without them getting any feedback that they don't look like their pictures. And AI picture here does not mean using ChatGPT but a service that's specifically tailored to dating images.

Yeah I dashed off the post pretty quick. The background context is "the entire history of Lightcone since back when it was just called 'the LessWrong team'" and which stuff to include wasn't obvious.

I think it is fair to be clear to young and hopefuls especially, that there is a need for highly competitive talent, not just anyone. I think that communicating to people that this is a highly competitive area both helps keep applicants realistic about their chances, (and help communicate to them whether they should be trying at all) and also serves to bolster the recognition/status of these positions, which makes them more attractive to highly competitive applicants.

Most of the other med students I work with are not going into medicine just because it pa... (read more)

One particular implication of #7 (and #4) that I think might be worth noting explicitly, and which isn't noted here, is the rationalist norm against sarcasm. The use of sarcasm will basically alienate anyone who doesn't already agree with you, poisoning the discussion. Moreover, it's a compressed, implicit form of argument, that ought to be made explicit and to have its presuppositions made explicit so they can be appropriately responded to. In particular, it often papers over A/B errors of the sort you discuss in #7.

Yeah, for instance https://arxiv.org/abs/2610.08144. I liked the discussion in Section 4

I did not pick up from the OP that there was a plan for costs to be lower to posters who had previously proved themselves, or for there to be any way for posters to make money at all, so there might be some important pieces in your head that didn't make it into the post. (Or I missed them.)

I’m confused, neither of the things Aella described were getting mad about consent violations?

I don't think "permanent underclass" is a very stable equilibrium. It seems likely to me that the economically useless underclass eventually starves to death.

By "statement mistranslation", do you mean that in some steps of some proofs, the proof should be proving statement A, but proved a similar statement B instead?

I asked this because LLMs usually modify my statements to generate a proof. Though I am not sure if this also happen inside those lean-formalized proofs.

I agree that the anthropics at the end are my own reasoning, and are substantially outside what you see from present-day model CoTs.

In addition to that I don't think near-future (2027, 2028) OpenAI LLMs would exhibit non-negligible behavioral influence of such reasoning either, when given prosaic prompts like the prompt in your article. I can't clearly say that if their training is incompetent though. The shadow of anthropic decision theory that does (or should) get ingrained into the non-COT output of the model^[1] depends heavily on what actual post-train... (read more)

nit: *complement

Ya know, it occurs to me:

People do fake money sites because there's less regulatory burden than "just use money" (as I understand it. Maybe fake money is also "fun" somehow for most people and slightly obfuscates some ugh feelings about money?)

But, it's nice to be able to redeem fake money for valuable things.

Redeeming fake money for... inference time, is... well, inference is basically going to be one of the main things that matter, basically as good as real money. I wonder if the government will notice.

I agree you have to, like, do a lot of mechanism design correctly instead of incorrectly.

Note: humans who are any good posting, most likely don't have to pay much for their stuff to get approved. (like in the 1 cent territory or something).

Or, somewhat more generally: once you've demonstrated signal that you are worth paying attention to out of the thousands of AIs (or human) sloppers, the system is cheap and probably makes you net-money.

Some things going on:

• I am half remembering a google I read once outlining how to do eigenkarma in a way that tracks the

“almost all state-based reward functions produce an optimal AI policy which forces one color or the other”

May I ask for a proof?

I'm quite amused that "I guess I really do need to check if coal network is somehow some opaque rationalist jargon" was like, a reasonable epistemic state for you to end up in.

Sure. I just think we are not on track to create an AI that's sufficiently aligned to give any humans most of the stuff it produces/secures, once it's in a position of overwhelming advantage. Hence "AI aligned to a wrong target" just won't happen on current trajectory, and even the AIs that don't cause extinction are the "poorly aligned AIs". So even though the leading plan is to make AIs aligned to some principals, I just expect it to fail, but probably not to the point of extinction (this possiblity remains moderately likely as well), specifically becaus... (read more)

youtube recently changed how it counts views for all videos to be more like tiktok and instagram. In the past I would have agreed that long form youtube views were higher quality, but this no longer seems to be the case.

My immediate question is "what are the posters hoping to gain, that they're willing to pay to have their posts published?" Is this for charity? For prestige? For cash prizes? What exactly is the value you're offering to the people you are hoping will pay you? I suspect that how the service would need to be structured, and how likely it is that you could get people to use it, would depend significantly on this answer.

Additionally, I'm skeptical of the idea that moderators will gain reputation organically.

If you reward them for endorsing things that are late... (read more)

In a classic "cobbler's children have no shoes", we don't really have this implemented in Manifund. But, it would be pretty cool to-- eg allow grant submitters or funders to pay $0.20 - $200 to polish their applications, or pay for reviews and critiques that are cleverly AI-driven but not slop.

(Lightcone Commons could also have something like this, obviously)

We did just start trying a karma system which I think is an improvement over how we used to sort projects.

One of my fun takes about software dev in the age of cheap intelligence is that the thing you refer to, the classic expensive part of testing a social network (solving cold start problem, getting enough users to kickstart a flywheel) should get massively cheaper since you can have AI simulating a bunch of personas.

Now of course I haven't actually seen this yet work well for any social network I've come across, so, maybe it remains an open problem. Act I and the more recent Delve Town are the closest examples I've heard of (from the AI whisperer space), tho... (read more)

Sorry, what is "coal network" in this context? (Claude thinks you meant "social network" but I'm curious if there's another rationalist concept I'm not familiar with here.)

update: Raemon corrected typo

Permanent disempowerment of the entire mankind is the result of us being taken over by a poorly aligned AI, while the underclass is a result of mishandling an AI aligned to a wrong target.

Also, uh, surely you could just pass this essay to Claude and get a vibecoded outcome that tests some of your intuitions around whether this will work?

Nod, but this isn’t the expensive part of testing a social network.

(And whether it works with current AIs isn’t really a crux)

That sure sounds like a causal graph in which "rationalist" and "kinky" are both potential causes of being on glosso, and therefore the two are anticorrelated once we condition on being on glosso.

Seems kind of reasonable! I often think that platforms like LessWrong or Alignment Forum should currently be experimenting with something like Manifold's mana, an in-app currency which can be spent dynamically on compute or other kinds of contracts. And I share the desire of being able to turn a dial from $2 to $20 to $200 and just get much better outcomes, whether on review or writing or image generation or sth. (Manifold lets you initialize a market's AMM with orders of magnitude different of initial liquidity; unfortunately, we haven't done much work in... (read more)

openai's expressed reasons

I think the corporate personhood convention is more confusing than clarifying here. Some individual human or humans with authority fired the researchers and authorized the reply. Do you know who? (Altman presumably had final say, but who else?) If you do, does it seem less surprising for those individuals (not "OpenAI") to retaliate and lie about it? If you don't, then on what basis can you be surprised by their ("OpenAI's") behavior?

My level of confidence depends strongly on a proof-by-proof basis. There's been a slew of Lean4-validated proofs lately, without a standardized level of rigor. I put it at ~40% chance that one of the recent high-visibility Lean4-formalized proofs ends up needing to be retracted, but what dominates that estimate is statement mistranslation, not soundness issues.

Godel's Incompleteness Theorem says that a sufficiently expressive theory will never be able to prove its own consistency. What you can do is prove one theory's consistency using another theory. It's perfectly fine to have a proof of Lean4's consistency within ZFC + large cardinals.

Maybe what you have in mind is something like: "How is it possible that con-leche can claim to give a proof of its own consistency"? The trick is that you start with "con-leche + strong set theory assumptions" and you conclude "con-leche is consistent". What you do NOT get is a ... (read more)

Otoh, there are already restrictions on what sorts of things you can use computers for, for example copyrighted material. I'll keep this in mind though.

Congress can ultimately remove the President, although that would be politically momentous. I'm uncertain whether mentioning impeachment would harm bipartisanship, but leaning no for now.

I was at Lighthaven for much of PDKU and that also roughly matches my impression. I came in at the tail end of 'torture potluck' so I didn't get a full impression but it didn't seem that crowded, and I didn't even remember the live band as 'part of PDKU' because this was the second or third time they've been at Lighthaven and I've gotten used to them.

Works for me. Full paragraph:

The efforts to improve safety and alignment issues around Hatch extend beyond lower-level employees and have been a central focus chief AI officer Alexandr Wang, head of product Nat Friedman and other senior executives, according to one person familiar with the efforts. Meta also recently hired Dan Hendrycks, a veteran AI safety researcher who has previously worked with Elon Musk’s xAI, now part of SpaceX, the person said.

My personal probability is that 70% of those Lean4-validated proofs are trustable. Before reading your article, my confidence was >95%.

What is your confidence in those recently Lean4-validated proofs, if you don't mind to share?

Frontier labs' NDAs are virtually airtight; even if they found those FLT-level verifications were actually relying on bugs, they have the option to slow-walk the disclosure and wait until IPOs.

Compare the recent firing of OpenAI safety people with the firing of Sam Altman. Both times the people firing said they "no longer trust" the fired person. Both times the people firing don't communicate good concrete reasons. One time (might become both if it escalates) it became front page New York Times news. I somewhat suspect for both, when it all comes out, the reasons will be more local and petty than people on the outside suspected at first (Sam Altman was maneuvering to oust Helen Toner from her seat. Her faction acted in a slapdash way when t... (read more)

Had the thought that this post’s scenario seems relevant to mistake theory/conflict theory decisions for humans.

Please consider that influencers, especially on TikTok, are mostly young people. And they need to be emotional, fun, etc. to be popular. They are not like us, older boring autists. And they don't need to have an atmosphere of professionalism around them - that would make their content more boring. So let them drink lots of alcohol if they want and do whatever they want.

CEV doesn't serve distant alien you, it serves current you, asking distant alien you (and medium distant wise you) for advice

I expect the advice should still move towards essentially asking the current you directly, because asking something else doesn't give legitimate advice, unless the current you decides that it's the case. What if the distant alien you decides that the current you doesn't get to override its advice, and this ends up a condition that permanently shapes the trajectory of current-you's development? The benevolent scenario installs some... (read more)

Thanks, I hate it.

This is a great set of examples to fuel the intuition that a smarter opponent will have ways to beat you that you haven't thought of. And I'm not sure I was really thinking hard enough about how much fighting in alien opponent will hamper your defenses.

Assuming that Lean4 is consistent, then a proof should exist and we should be able to find it

Hmm, how does this interact with Godel's Incompleteness Theorems?

mostly around the AI acting based on distantly-simulated-you rather than actual-you

I think consulting a simulated-you is a reasonable first pass that plausibly ends up pointing to the need to ask actual-you next (though various possible-yous, accurately-rather-than-vaguely enacted/simulated, are probably more informative). If the basin of convergence to alignment is broad, starting at a very crude first step in some fixpoint process is OK, as long as the process gets to keep refining itself, and doesn't get stuck in some path-dependence trap.

Hence the i... (read more)

why would you be surprised if openai fired them as retaliation? (genuine question)

It is not unheard of for an author to get something right in one piece, yet wrong in another.^[1] Given my reply is to this post, not that one, the words here, not there, are the relevant ones.

Although natural language can be ambiguous, "[Y]ou ought not to update in a predictable direction" is generally false on plain reading: you can reliably predict the direction of travel for many beliefs, even if held by an ideal Bayesian agent. "[Y]ou should not be able to predict a net direction in your own price movements" has a bit more wiggle room thanks to 'net', ... (read more)

The Agent Thing on multiple scales: OpenAI as opaque black box that fires the grader.

With Project Tailwind there's now an extra $~1 billion in CG funding for new orgs, and significant amounts from Longview, S&FF and Lightcone etc. There's also loads of hiring for AI safety roles in the private sector, in the frontier labs (if you're comfortable with that), startups, governments/AISIs, the EU, media, and legal roles across the world. There's also lots of room for people to work in activism, grassroots stuff, independent content creation etc.

This could be an extra ~5,000 people meaningfully working full-time in AI safety in 2027.

How does... (read more)

Neat. Yeah that all seems reasonable at first glance.

A post I've been trying to write is "dude, CEV doesn't serve distant alien you, it serves current you, asking distant alien you (and medium distant wise you) for advice". There's a lot of big open questions of what that means in practice.

I also think "do not build overwhelming superintelligence (that is not the emergent result of uplifted humans") is a reasonable answer to "what to do?". CEV is the specific answer to "what if you built an overwhelmingly super AI?"

I endorse you leaning into under-articula... (read more)

Would a proof that Lean4 - nested inductives is consistent with respect to ZFC + any large cardinal axiom help bound the problem? Or a characterisation of what you get by taking nested inductives out of the picture / where it becomes more incomplete than you would ordinarily like?

I'm mostly wondering if there would be value in establishing bounds on how bad things are. If it is "just" nested inductives that feels different to "nested inductives and a bunch of other things we can't even establish with nested inductives in the frame".

I'm not offering, I'm ju... (read more)

I did invest a nontrivial amount of time in reading your post. Otherwise I wouldn't even comment. I predict that most of our disagreement doesn't come from me not having read enough. Also I gave the post a strong upvote for visibility, because IMO any post discussing specific non-prosaic ideas deserves a boost.

Not only would your criteria mark Linnaeus's taxonomy - as in, the Systema Naturae, the ur-example of a conceptual taxonomy - as useless, you evidently didn't even read far enough to get to the account of everything past nouns, which literally starts

Yeah, one big problem with the original comment is that I failed to convey that the signal involved is (a) weak, and (b) not the most important thing when assessing someone's AI alignment qualifications.

Arguably using RMN attendance to judge someone's researcher qualifications is by itself a mistake, even if there is in fact useful ground-true correlation. As in, it may still be better to (try to) not use this information when later assessing someone's qualifications, specifically in order to not create bad incentives for those people who are accurately mo... (read more)

If I look at the top of one of the letters in Miyoko's, I can read UNSALTED, CREAMERY, and BUTTER, but PLANT MILK is just barely small enough that I can't make it out easily in my peripheral vision on the screen I'm looking at. It's in an understated position, not drawing attention to itself, and easy for your eye to slide right over if you aren't looking for it.

I strongly agree! We need more smart people doing non-AI things like biology. (My startup Ovelle Bio is currently hiring for in vitro gametogenesis research.) Of course AI safety and AI pause advocacy are definitely needed, but non-AI approaches to human empowerment are overlooked right now.

You raise good points and public sentiment is definitely hard to measure.

One way of thinking about the crying wolf aspect is asking "Will being pro-AI regulation help Republicans in the polls?" And I think the idea that AI actually take over is firmly in the "Terminator fantasy" category by your average lean-right voter.

In terms of the individuals vs. the public, I think the public determines what issues and legislation are within the Overton window. A Bernie proposal like pausing the industry that underpins the entire stock market is currently outside tha... (read more)

I think all of the issues you mention are clarified somewhere in the post. In particular, there is a

Note that we only expect high safety in the protocols represented by the points on the left side of these plots, where T sees very small amounts of advice from U.

It's true that this is kind of buried though. I don't think this is really a motte-and-bailey, since I wasn't deliberately equivocating between two views, but the post wasn't very clear about the moderate position I was trying to express. I just made a couple of edits to try to make it clearer.

I ch... (read more)

Yes, this is the primary drama from sex parties

Someone thinks their consent was violated, and it becomes a whole thing/splits the community

Much more rare at non-sex parties

Correct! Ugh, brains are weird. Cheers

The ability to do this seems like a pretty good barometer for/operationalizing of “am I speaking (and thus acting) with integrity”: https://mindingourway.com/confidence-all-the-way-up/amp/

On the surface this article seems like an argument for value alignment over corrigibility, but I don't think it makes sense. You just need a corrigible AI that checks with the human before doing things.

AI / Genie / Wish-granting machine: "Certainly! I can help you get your mother out of this burning building. Here's the implementation plan for your wish: (some plan that ends with the house exploding) Would you like to proceed with the plan?"

Human: "No, come up with something else"

AI / Genie / Wish-granting machine: "Here's an updated plan: (technical jargo... (read more)

I wonder if "butter" labeling in particular gets more leniency due to things like cocoa butter and apple butter, where "butter" is more a claim about the texture than the ingredients.

Well, your skimming shows - there's numerous falsifiable predictions that I make throughout the post; the first few are about a third of the way in. What's more, you lay out five possible criteria that would make a taxonomy useful and claim that a taxonomy that has none of those virtues is useless. Not only would your criteria mark Linnaeus's taxonomy - as in, the Systema Naturae, the ur-example of a conceptual taxonomy - as useless, you evidently didn't even read far enough to get to the account of everything past nouns, which literally starts the body of... (read more)

Distillation for Capabilities seems really interesting. You could, in theory, actually "rescue" an adversarially misaligned model by distilling its capabilities into some other model.

Let's say model lab that has a mix of robust, highly trusted RL environments (let's say, environments which are end-to-end verified by humans and on which no successful hacks were ever recorded) and less trusted ones. They could do the following:

• Train a model with all of the RL environments
• Distill it into a second model, transferring most of the capabilities and hopefully litt

I would guess any sort of mob mentality event in history seems to have a coordination failure where actors are being too marginalist. Eg. the classic every camp guard telling himself “if I didn’t do it someone else would.” With international law and national laws of war responding to that with “Sorry that doesn’t work. If you don’t reason like an FDT agent there things wind up going to hell, and we can all see that, and so we consider you blameworthy if you do the CDT thing.”

Then not sure if these are handled badly but other maybe-relevant examples of non-... (read more)

It actually literally sounds like the opposite of what you would expect a rationalist to do (e.g. inject a load of love hormones and emotions and so on into a business situation).

I mean, to be clear, many extremely successfully businesses are family businesses. I don't agree it would be a reasonable heuristic to always avoid it (though I do think it's better for relationships) or that it would be antithetical to someone being a "rationalist".

I do think Sam's and Caroline's relationship seemed pretty confusing and messed up, as I said. But that seems downs... (read more)

Hmm, IDK, this doesn't match my recollection of talking to Jonas and some others about this stuff. But probably not worth going into.

That is certainly not the case with other kinds of drama. I am exposed to random other drama as a result of all kinds of other things all the time, like which speakers are invited to which conferences, and what people say on Twitter, and what people write on LessWrong, and where people work, and who gets funding, and who gets fired from their org, etc.

When everyone is this determined to destroy themselves, I have no choice but to join this brutal arms race.

I don't think you could come up with a better summary for the logic of the current arms race

I once made a judgment about how the times are changing and where I'll stand in the future. Because things are changing so fast (the AI progress above is a good example), I have no way of predicting what will happen in five or ten years. But whatever happens, I believe that my vision, judgment, initiative, and intelligence will keep me at the table and let