khafra

Replying toGoodfire and Training on Interpretability

Goodfire and Training on Interpretability

decomposing the gradient into semantic components and choosing which components to apply.

This isn't a strong argument, but that (and the Goodfire essay describing their approach) strongly remind me of a really clever design for a perpetual motion machine. I don't see any description of a technique like soft alignment (https://www.lesswrong.com/posts/9fL22eBJMtyCLvL7j/soft-optimization-makes-the-value-target-bigger), or a similar technique that acknowledges catastrophic goodhart (https://www.lesswrong.com/posts/fuSaKr6t6Zuh6GKaQ/when-is-goodhart-catastrophic) and Garrabant's Goodhart taxonomy, and works within those restrictions.

Replying toVarieties Of Doom

khafra3mo

Varieties Of Doom

I am also now haunted by the "humanism is dead" take. I guess I believe it, but what killed it is the internet, and I think we could bring it back.

I don't think that's it. I think he meant that humanism was created by incentives--e.g., ordinary people becoming economically and militarily valuable in a way they hadn't historically been. The spectre, and now rising immantization, of full automation is reversing those incentives.
So, it's less a problem with the attitudes of our current elites or the memes propagated on the Internet. It's more a problem with the context in which anybody achieving the rank of elite, and any meme on human value which goes viral, is shaped by the evolving incentive structure in which most humans are not essential to the success of a military or economic endeavor.

Replying toThe Tale of the Top-Tier Intellect

khafra3mo

The Tale of the Top-Tier Intellect

I feel like this was a sort of fractal parable, where the first two paragraphs should be enough to convey the point; but for readers who don't get it by then, it keeps beating you over the head with successively longer, more detailed, and more blatant forms of the point until the final denouement skips the "parable" part altogether.

Replying toLLM robots can't pass butter (and they are having an existential crisis about it)

khafra3mo

LLM robots can't pass butter (and they are having an existential crisis about it)

We need names for this phenomenon, in which the excess cognitive capacity of an AI, not needed for its task, suddenly manifests itself

It is so much like absurdist SF, that's the perfect source for the name--The Marvin Problem: "Here I am, brain the size of a planet and they ask me to take you down to the bridge. Call that job satisfaction? 'Cos I don't."

Replying toZetetic explanation

khafra6mo

Zetetic explanation

There's an article type called "You Could Have Invented" that I became aware of on reading Gwern's You Could Have Invented Transformers.
This type dates back to at least 2012. I believe they're usually good zetetic explanations.

Replying toUnderdog bias rules everything around me

khafra6mo

Underdog bias rules everything around me

In a stereotypical old-west gunfight, one fighter is more experienced and has a strong reputation; the other fighter is the underdog and considered likely to lose. But who's the underdog of a grenade fight inside a bank vault? Both sides are overwhelmingly likely to lose.

At least one side of many political battles believe they're in a grenade fight, where there's little or nothing they can do to prevent the other side from destroying a lot of value. and could reasonably feel like an underdog even if they have a full bandolier of grenades and the other side has only one or two.

-1

Replying toA Simple Explanation of AGI Risk

khafra7mo

A Simple Explanation of AGI Risk

I don't think "perfect" is a good descriptor for the missing solution. The solutions we have lack (at least) two crucial features:
1. A way to get an AI to prioritize the intended goals, with high enough fidelity to work when AI is no longer extremely corrigible, as today's AIs are (because they're not capable enough to circumvent human methods of control).
2. A way that works far enough outside of the training set. E.g., when AI is substantially in charge of logistics, research and development, security, etc.; and is doing those things in novel ways.

Replying toColonialism in space: Does a collection of minds have exactly two attractors?

khafraMay 28, 2025

Colonialism in space: Does a collection of minds have exactly two attractors?

Robin Hanson's model of quiet vs loud aliens seems fundamentally the same as this question, to me.

Replying toIt's hard to make scheming evals look realistic for LLMs

khafra9mo

It's hard to make scheming evals look realistic for LLMs

Linear probes give better results than text output for quantitative predictions in economics. They'd likely give a better calibrated probability here, too.

khafra9mo

I, too, would like to know how long it will be until my job is replaced by AI; and what fields, among those I could reasonably pivot to, will last the longest.

Some cities have dedicated LW/ACX Discord servers, which is pretty neat. Many of the cities hosting meetups over the next month are too small to have much traffic to such a server, were it set up. A combined, LW meetup oriented Discord server for all the smaller cities in the world, with channels for each city and a few channels for common small-meetup concerns, seems like a $20 bill on the sidewalk. So I’m checking whether such a thing exists here, before I start it.

khafra's Shortform

khafra

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

[Link] Walking Through Doors Causes Forgetting

khafra

14y

We investigated the ability of people to retrieve information about objects as they moved through rooms in a virtual space. People were probed with object names that were either associated with the person (i.e., carried) or dissociated from the person (i.e., just set down). Also, people either did or did not shift spatial regions (i.e., go to a new room). Information about objects was less accessible when the objects were dissociated from the person. Furthermore, information about an object was also less available when there was a spatial shift. However, the spatial shift had a larger effect on memory for the currently associated object. These data are interpreted as being more supportive

khafra

15y

http://www.dailycamera.com/boulder-county-news/ci_18282009?source=most_viewed

I'd imagine the efficacy is halfway between proper cryonics and embalming and burying; the more interesting part may be the festival. Nederland is a small town 20 miles from Boulder, CO. I doubt the festival attendees are cryonics advocates, but they don't seem prone to the negative associations corpsicles often raise. Perhaps it's just because Boulder, Colorado is full of weirdos, but I wonder if there are more exploitable effects in play.

Free Thought Film Festival: Tampa traditional rationalist gathering this weekend (13-15 May)

khafra

15y

A bit late, since I'm at the opening ceremonies right now; but it seems like it could have some potential; and the Tampa Theatre is a great venue.

Article on quantified lifelogging (Slate.com)

khafra

15y

Data for a Better Planet focuses on The Quantified Self, and offers an overview of the state of the art in detailed, quantitative personal tracking.

This seems related to an LW interest cluster.

LESSWRONG
LW

LESSWRONG
LW

Amateur Cryonics (one guy packed in dry ice) Festival Seeks Buyer

[Link] Walking Through Doors Causes Forgetting

khafra's Shortform

Free Thought Film Festival: Tampa traditional rationalist gathering this weekend (13-15 May)

khafra

khafra's Shortform

[Link] Walking Through Doors Causes Forgetting

Amateur Cryonics (one guy packed in dry ice) Festival Seeks Buyer

Free Thought Film Festival: Tampa traditional rationalist gathering this weekend (13-15 May)

Article on quantified lifelogging (Slate.com)

khafra

Amateur Cryonics (one guy packed in dry ice) Festival Seeks Buyer

[Link] Walking Through Doors Causes Forgetting

khafra's Shortform

Free Thought Film Festival: Tampa traditional rationalist gathering this weekend (13-15 May)

khafra

khafra's Shortform

[Link] Walking Through Doors Causes Forgetting

Amateur Cryonics (one guy packed in dry ice) Festival Seeks Buyer

Free Thought Film Festival: Tampa traditional rationalist gathering this weekend (13-15 May)

Article on quantified lifelogging (Slate.com)