So... the biggest savings here, by far, is the rent. At a guess it's bigger than everything else combined. If you don't have enough friends, or your friends all live in full houses, guess you're screwed here. Hope we're OK with the tyranny of structurelessness?

I'd like to see a breakdown by "years since doing MATS". What's the retention like, basically? Another breakdown I'd like to see, either for the displayed data or the years-since-MATS one - what's the split in AI safety between "(co)founded an org", "joined a new org", "joined an established org/(research/policy) group (technical vs governance)", and "on a grant, no real org", along with the existing "academic" (split by core academic/alt-academia?), "government" (which country?), and for-profit (maybe a breakdown of product type?). In any case, thanks for posting this!

First off, TFTP. I marked some stuff I thought was most relevant. This is helping remind me of some things I think about LLM confabulation and lack of binding/reasoning... I don't have my thoughts fully formed but there's something here about global inconsistency despite local compatibility, and how that cashes out in Problems. Something a little like an inability to define a sheaf, or homology detection, or something like that? I might say more better words later about it.

Thanks for writing this. I've noticed something in the same vein as well - for the last few months I've felt increasingly like a lot of the pieces and systems agent foundations types use have some kind of important commonality to them, though I'm not yet sure what form that could take. Condensation and natural latents and some of Francis Rhys Ward's work in II-MAIDS; ontology mismatch and Bayes nets and imprecise probability and category theory showing up repeatedly. There's something there to construct, but what?

I'm doing Budget Inkhaven: https://tiled-with-pentagons.blogspot.com/search/label/budget inkhaven

Similar to my earlier writing regimen, but shorter, faster, and slightly more daring. I'll crosspost anything I like especially well here.

Hm. So it's some larger problem of which the Curse of It From Bit is a subcluster, then? If I've understood you right, this is also something I've spent time thinking about by way of finding them with my face. Something like... some unfeeling machine/system/structure that fails to live up to its own spec, or which has a bad spec or no spec at all, and which makes that state of affairs your problem, you silly goose who wants to actually do things, you. There's the flavor of that thing from Zen and the Art of Motorcycle Maintenance where there's this tiny trivial-feeling problem that humiliates you with a combination of feeling trivially small while nonetheless demanding close attention and deep understanding by way of being a critical blocker to solving whatever problem, I think? I think that the yakshaving bit is downstream of the spec failure piece - it seems like a common failure mode of degraded systems/designs.

Oh! I have thoughts about this one!

On my model, it's a matter of basically anything that turns Bits into Its being kinda cursed, especially if whatever process lacks human actuation or even oversight and isn't done to truly exacting standards. Printing is an example of this; 3D printing and general CNC is another; graphic design (and the depths of what a color is) is a third. I'm thinking of the kind of thing you might find on r/shittyrobots as well; a minimal non-example might be polypeptide/oligonucleotide preparation, though I'm not totally clear on the workflow there or how much human labor is involved.

My gears-level model here is most starkly illustrated by 3d printing - something about the physical object creation might change, or be underspecified, or rely on shoddily-made connectors of some kind, and in any case, the actual machine that does the actuating isn't set up to check for failures during workflow or indeed do much more than the simple actuation and an initial calibration step. On top of that, the thing goes from being data which is everywhere and wherever you want it, to being a specific thing in a specific place; often you don't even get to have the affordance of knowing in advance which place it is. Possibly even the yakshaving you notice is a matter of some kind of associated ugh field.

Spoons are for doing ordinary things, doing things at all. [Blood] is for doing difficult things, doing them more or harder, doing them at significant internal cost.

Several years on from this post and @Raemon 's comment, where do we stand? From my own starkly limited ground-level perspective, it looks like pretty much everyone who's tried to figure out the secrets of training has smashed into the same flavor of mixed-success, shattered, and given up on it as a bad job. This seems sad and desperately misguided, but what do I know? I haven't exactly tried of my own accord, and I showed up too late to the party for the height of CFAR and the like. I know a few people are still trying to poke at questions like this one from various angles, but everyone I've talked to about the topic (5 or so of them?) seem extremely burned out on the whole enterprise, so maybe it's all for the best that I never went.

What would it look like, to actually try again in these latter days? If the model in this post is correct - and I think that it is, that training is vastly more valuable than selection - then there's gains to be had. It'd start - I think - by figuring out what went wrong previously and noticing and carefully attending to all the skulls without drowning in despair about them. The next step might be some careful trials - maybe randomized ones, maybe they need to be thoughtfully tailored to the student - and tests after the fact at a few time intervals. Something about the zone of proximal development is tickling the back of my mind here; so is something about crystallization and application of the skill, especially in a way that psychologically accords that you Have The Skill, which probably calls for some ordeal? Hard to say. I only have scattered thoughts about this. If anyone wants to try to elicit more of them from me I welcome the attempt.

I didn't realize it when I posted this, but the anvil problem points more sharply at what I want to argue about when I say that making the NAS blind to its own existence will make it give wrong answers; I don't think that the wrong answers would be limited to just such narrow questions, either.