deep

In context, I guess your claim is: "if the 'compressor' is post-hoc trying a bunch of algorithms and picking the best one, the full complexity of that process should count against the compressor." Totally agree with that as far as epistemology is concerned! 

But I don't think the epistemological point carries over to the realm of rational-fic. 

• In part that's because I think of JKR-magic as in fact having a bunch of structure that makes it much easier to explain than it would be to explain a truly randomly-generated set of spells and effects (e.g. the pseudo-Latin stuff; the fact that wands are typically used). So I expect an retrofitted explanation wouldn't be crazy

It's like trying to compress a file that was generated by a random device —  
 

Gretta: You can't losslessly compress a truly random file. 

I don't think this is strictly true. You can't a priori build a compression scheme that will work for an arbitrary random file (No Free Lunch Theorem). But you can ex post identify the particular patterns in a particular random file, and pick a compression scheme that picks up on those patterns. You probably end up with a pretty ugly scheme that doesn't generalize, and so is unsatisfactory in some aesthetic sense. Especially if you're going for lossless compression, since there's probably a ton of noise that's just very hard to compress in... (read more)

Nice post! 

Your trader analogy made me think, you'd ideally want to have a training period with lots of honeypots and surveillance, so that the trader learns that crime doesn't pay.  

This suggests some level-3 ideas I didn't see in your post:

• In addition to improving the reward model, you could also alter the training environment by inserting honeypots.
• On the surveillance side, maybe it's more effective to concentrate your compute in order to catch sneaky behavior. So, say, after every $N$ times it performs some RL task, you use some weak filter to identify the $k$ most suspicious rollouts and then spend a bunch of compute on running an "auditing" process that checks for reward hacking.  

These... (read more)

OK, cool, I think I understand where you're coming from much better now. Seems like we basically agree and were just emphasizing different things in our original comments! 

I'm in violent agreement that there's a missing mood when people say "AIs will follow the law". I think there's something going on where people are like "but liberalism / decentralized competition have worked so well" and ignoring all the constraints on individual actors that make it so. Rule of law, external oversight, difficulty of conspiring with other humans, inefficiencies of gov't that limit its ability to abuse power, etc.

And those constraints might all fall away with the AGI transition. That's for a number of... (read more)

Apologies for the scrappiness of the below -- I wanted to respond but I have mostly a scattering of thoughts rather than solid takes.

I like the intelligence curse piece very much -- it's what I meant to reference when I linked the Turing Trap above, but I couldn't remember the title & Claude pointed me to that piece instead. I agree with everything you're saying directionally! But I feel some difference in emphasis or vibe that I'm curious about. 

-

One response I notice having to your points is: why the focus on value alignment? 

"We could use intent alignment / corrigibility to avoid AIs being problematic due to these factors. But all these issues... (read 448 more words →)

Thanks, I love the specificity here!

Prompt: if someone wanted to spend some $ and some expert-time to facilitate research on "inventing different types of guys", what would be especially useful to do? I'm not a technical person or a grantmaker myself, but I know a number of both types of people; I could imagine e.g. Longview or FLF or Open Phil being interested in this stuff.

Invoking Cunningham's law, I'll try to give a wrong answer for you or others to correct! ;)

Technical resources:

• A baseline Constitution, or Constitution-outline-type-thing
  • could start with Anthropic's if known, but ideally this gets iterated on a bunch?
  • nicely structured: organized by sections that describe different types of behavior or

You're allowed to care about things besides AI safety  

I worry that a lot of AI safety / x-risk people have imbibed a vibe of urgency, impossibility, and overwhelming-importance to solving alignment in particular; that this vibe distorts thinking; that the social sphere around AI x-risk makes it harder for people to update.

Yesterday I talked to an AI safety researcher who said he's pretty sure alignment will be solved by default. But whenever he talks to people about this, they just say "surely you don't think it's >99% likely? shouldn't you just keep working for the sake of that 1% chance?"

Obviously there's something real here - 1% of huge is huge. But... (read more)

Refurbishing the classic AI safety argument 

My initial exposure to AI safety arguments was via Eliezer posts. My mental model of his logic goes something like: 

"0) AI training will eventually yield high-quality agents;

1) high-quality agents will be utility maximizers; 

2) utility maximizers will monomaniacally optimize for some world-feature; 

3) therefore utility maximizers will seek Omohundro goals; 

4) they'll be smarter than us, so this will disempower us; 

5) value is fragile, so empowered AIs monomaniacally optimizing for their utility function fucks us over with very high probability"

VNM doesn't do what you want. As folks like @Rohin Shah and @nostalgebraist have pointed out, point 2 (and therefore 3 and 5) don't really follow. A utility function can have... (read 476 more words →)

What's up with incredibly successful geniuses having embarassing & confusing public meltdowns? What's up with them getting into naziism in particular?
 

Components of my model:

• Selecting for the tails of success selects for weird personalities; moderate success can come in lots of ways, but massive success in part requires just a massive amount of drive and self-confidence. Bipolar people have this. (But more than other personality types?)
• Endless energy & willingness to engage with stuff is an amazing trait that can go wrong if you have an endless pit of stupid internet stuff grabbing for your attention.
• If you're selected for overconfidence and end up successful, you assume you're amazing at everything. (And you are