(I forgot that more conversation might happen on a LW crosspost, and I again lament that the internet has yet to develop a unified routing system for same-content-different-edition discourse. Copied comment from a few days ago on substack:)

I really appreciate this (and other recent) transparency. This is much improved since AI 2027.

One area I get confused by (same with Davidson, with whom I've discussed this a bit) is 'research taste'. When you say things like 'better at research taste', and when I look at your model diagram, it seems you're thinking of taste as a generic competence. But what is taste? It's nothing but a partially-generalising learned heuristic model of experiment value-of-information. (Said another way, it's a heuristic value function for the 'achieve insight' objective of research).

How do you get such learned models? No other way than by experimental throughput and observation thereof (direct or indirect: can include textbooks or notes and discussions with existing experts)!

See my discussion of research and taste

As such, taste accumulates like a stock, on the basis of experimental throughput and sample efficiency (of the individual or the team) at extracting the relevant updates to VOI model. It 'depreciates' as you go, because the frontier of the known moves, which moves gradually outside the generalising region of the taste heuristic (eventually getting back to naive trial and error), most saliently here with data and model scale, but also in other ways.

This makes sample efficiency (of taste accumulation) and experimental throughput extremely important, central in my view. You might think that expert interviews and reading all the textbooks ever etc provide meaningful jumpstart to the taste stock. But they certainly don't help with the flow. So then you need to know how fast it depreciates over the relevant regime.

(Besides pure heuristic improvements, if you think faster, you can also reason your way to somewhat better experiment design, both by naively pumping your taste heuristics for best-of-k, or by combining and iterating on designs. I think this reasoning boost falls off quite sharply, but I'm unsure. See my question on this)

I'm interested to know how (if at all) you'd say the perspective you've just given deviates from something like this:

• pretrained LMs can simulate many personas, including interpolations and extrapolations of personas inferred from the training data
  • (aside: this presumably also includes various non-first-person non-'self'-representing 'personas' like wikipedia voice and stackoverflow voice)
• various forms of conditioning can elicit/filter personas
  • finetuning of all sorts
  • also prompting (e.g. playscript-style)
  • (this can include autoregressive 'self' prompting)
• depending on type and level of finetuning, weights can end up more or less cohering on a particular subset/subspace of personas
  • (this might be context-dependent, for example if certain prompt conditions, such as a user/assistant playscript, are consistently present in the finetuning, and might not generalise to other prompt conditions)

My current guess is you agree with some reasonable interpretation of all these points. And maybe also have some more nuance you think is important?

Given the picture I've suggested, the relevant questions are

• What are the returns to thinking longer for experiment design?
  • (I tentatively think they're quite harsh)
• What are the penalties for parallelised experiment design and/or for teamwork on experiment design?
  • (I've written even less on this, but I think it's quite difficult because the key is pooling research taste, either from best-of-k or some team iterative thing, but research taste is mostly acquired through experimental experience)
• (Perhaps also as you note, are there cognitive diversity penalties or bonuses at play?)

A complementary angle: we shouldn't be arguing over whether or not we're in for a rough ride, we should be figuring out how to not have that.

I suspect more people would be willing to (both empirically and theoretically) get behind 'ruthless consequentialist maximisers are one extreme of a spectrum which gets increasingly scary and dangerous; it would be bad if those got unleashed'.

Sure, skeptics can still argue that this just won't happen even if we sit back and relax. But I think then it's clearer that they're probably making a mistake (since origin stories for ruthless consequentialist maximisers are many and disjunctive). So the debate becomes 'which sources of supercompetent ruthless consequentialist maximisers are most likely and what options exist to curtail that?'.

"This short story perfectly depicts the motivations and psychological makeup of my milieu," I think wryly as I strong upvote. I'm going to need to discuss this at length with my therapist. Probably the author is one of those salty mid-performing engineers who didn't get the offer they wanted from Anthropic or whatever. That thought cheers me up a little.

Esther catches sight of the content on my screen over my shoulder. "I saw that too," she remarks, looking faintly worried in a way which reminds me of why I am hopelessly in love with what she represents. "Are we, like, the bad guys, or maybe deluding ourselves that we're the good guys in a bad situation? It seems like that author thinks so. It does seem like biding my time hasn't really got me any real influence yet."

I rack my brain for something virtuous to say. "Yeah, um, safety-washing is a real drag, right?" Her worry intensifies, so I know I'm pronouncing the right shibboleths. God, I am really spiritually emaciated right now. I need to cheer her up. "But think about it, we really are in the room, right? Who else in the world can say that? It's not like Vox or Krishna are going to wake up any time soon. That's a lot of counterfactual expected impact."

She relaxes. "You're right. Just need to keep vigilant for important opportunities to speak up. Thanks." We both get back to tuning RL environments and meta-ML pipelines.

The solution may be analogous: some form of paternalism, where human minds are massively protected by law from some types of interference. This may or may not work, but once it is the case, you basically can not start from classical liberal and libertarian assumptions.

I half believe this. I notice though that many modern societies have some paternalistic bits and pieces, often around addiction and other preference-hijacking activities... but are also often on the whole liberal+libertarian. It may be that more enclaves and 'protection' is needed at various layers, and then liberalism can be maintained within those boundaries.

Right, I think positional goods and the like are among several distortions of the basic premises of the welfare theorems (and indeed empirically many people are sad, lonely, etc. in our modern world of abundance) - I sometimes think those theorems imply a sort of normative 'well, just don't worry about other people's stuff!' (i.e. non-envy, which is, after all, a deadly sin). cf Paretotopia, which makes exactly this normative case in the AI futurism frame.

You're totally right, the representative agent actually was representative! So (unsurprisingly) this directs attention to how capital ownership might be distributed between (workers over different) sectors.

I noticed something funny about the optimal L: it's negative if wages are sufficiently dwarfed by capital interest. But this makes sense: if they could, these workers would buy more time in the day for leisure by sacrificing some income.

I'd guess (haven't mathed it through though) that general CES utility and production functions don't produce this independence of total labour on total capital, and that this is a special Cobb-Douglass case. But it's also not totally implausible. (After all, indeed humans have tended to work a lot on the whole throughout the last century or two of unprecedented capital accumulation, despite Keynes' predictions.)

This is great! A good concrete operationalisation.

I think this is tacitly assuming that capital ownership/income is already uniformly distributed across all workers, so the only exogenous variable is the capital accumulation. That's addressing part of the question quite nicely - though I was also wondering about the changing distribution (and prevalence) of capital ownership. I don't think this single representative agent is what cashes out if we imagine that distribution changing...? But it's a nice illustration of some kind of 'universal ownership' endpoint.

I'm also not sure of the Cobb-Douglasness of utility on consumption and leisure inputs! This seems to imply that, for a given leisure allowance, utility grows faster in consumption than I'm given to believe it does in practice. It's super interesting that things cancel out in this case, though. That said, it's an equivalent maximisation to treating utility as logarithmic in consumption plus logarithmic in leisure, which is not crazy.

This gives me some great inspiration if I want to come back to this and look at more modelling variations. Thanks,

I think there's a magic ^[1] that the military is somehow also fairly firmly aligned with the constitution and non-partisan, though nominally also under the president's command. I don't really get it, and I don't know how much this helps.

1. and it is magic, as in it's an inexplicable (to me) and presumed-contingent (social) technology ↩︎