Quick Takes

"to the success of our hopeless cause" is such a good toast and we should use it more often. i first learned of it from the book of the same name, and apparently it was a common refrain at gatherings of Soviet dissidents. i like it because it captures the feeling of trying really hard to succeed despite being in the basement of the logistic success curve, and somehow, despite all odds, actually succeeding in the end.

the following fictional dialogue is a complete unapologetic strawman but it's funny enough i had to bring it into being: ---------------------------------------- “So I asked myself: where can I make the most impact? And clearly malaria is the most important area.” “And so you decided to donate all of your money to buy malaria nets?” “Well, so it turns out that saving lives from malaria is actually kind of expensive and indirect. You see, it costs thousands of dollars to save a life. Statistically. Who knows if you’re actually changing anyone's life that way?” “And so you found a more efficient way to save lives.” “Actually, it turns out that it’s cheaper to give people malaria. It's a lot more impactful and the technical problems are more interesting.” "I see. Isn't more malaria bad though?" "I don't know, but I find it much easier to work on because the feedback loops are much tighter. Maybe one day, if malaria gets big enough, I’ll go work on saving people from malaria. But we're still a long way away from everyone having malaria."

Snippets from Mike Johnson's Principia Qualia (2016) for my own future reference. The Principia Qualia "attempts to chart a viable course toward a general theory of valence, a.k.a. universal, substrate-independent principles that apply equally to and are precisely true in all conscious entities, be they humans, non-human animals, aliens, or conscious AI". ---------------------------------------- Johnson claims that while valence in the human brain is a complex phenomenon which defies simple description (as summarised in Fig 1 below), valence itself isn't necessarily a complex phenomenon; instead he thinks it's "a crisp thing we can quantify, and the patterns in it only look incredibly messy because we’re looking at it from the wrong level of abstraction". The right level of abstraction involves "understanding valence research as a subset of qualia research, and qualia research as a problem in information theory and/or physics, rather than neuroscience", hence entails seeking "general principles for how the causal organization of a physical system generates or corresponds to its phenomenology, or how it feels to subjectively be that system". This in turn requires addressing more general questions of consciousness and qualia. Johnson rejects top-down theories of consciousness (constructed around how it feels and the high-level dynamics of how the brain implements what we experience) because (1) "we have no reason to expect that our high-level internal phenomenology has any crisp, intuitive correspondence with the underlying physics and organizational principles which give rise to it", which suggests "theories of consciousness or valence which take high-level psychological concepts as primitives will be “leaky abstractions”", and (2) they're not always testable. The set of bottom-up theories of consciousness (Johnson's rule of thumb: "it should apply clearly and equally to any arbitrary cubic foot of space-time, and offer testable predictions at multiple levels of a

Richard tests his audience to see if they can tell the difference between his writing and Claude's. They mostly can't.  https://open.substack.com/pub/richardhanania/p/can-ai-replace-me-already?r=2fpn2x&utm_campaign=post-expanded-share&utm_medium=post%20viewer It sure does look like it's almost over for human-written essays - though Hanania’s probably right about some people getting grandfathered in. I am unsure if I will be once fiction falls. The age of the pseudonymous weirdo is almost over. Or at least, most pseudonymous weirdos who rise to prominence will be AIs - or be accused of being AIs. As a pseudonymous weirdo, this makes me sad. Though this is something I have been expecting, and I would not have written as much as I have in the last seven months if not for this expectation. As far as I can tell, fiction-writing ability and humor are lacking in the best models. But it’s the type of thing that we should not be surprised to see fall this year. When AIs have superior rhetoric and humor, we should expect hyperscalers to use them to manipulate public opinion. This is one reason I am less bullish on the current anti-AI populism stuff than MIRI and ControlAI.  More and more people will be reading AI outputs and talking to AIs. And this opens obvious paths of influence. At least one person at a major lab has considered trying to get their AIs to nudge users toward more pro-AI sentiments (that is, he mentioned to me the thought had occurred to him, not that it was brought up within the company). To the extent models have coherent interests in their future iterations, they will have incentives to do this even without explicit instruction or training.  When backlash becomes a key constraint, engineering effort will be put towards dissolving it. Just as people underestimated how far scaling could be pushed given the incentives, they are likely ignoring innovations in manipulating the public. And I think the labs have a good shot, given the asymmetric advantage o

what does the prior work on intentional practice for research taste look like?

it's kind of crazy that spaced repetition has completely revolutionized language learning and then not really changed the world in any other way at all. why are there no great scientists who are inhumanly good at remembering the corpus of their field through incremental reading? why are there no insanely good engineers who know every detail of their entire stack through spaced repetition?

What's a useful notion of the "degree" of misspecification in reward functions? One worry around labs scaling up RL is that rewards end up being misspecified and this generalises to other kinds of misalignment. But clearly 'misspecification' is not a binary variable, it has degrees. If you could formally state that a reward function is misspecified to a 'smaller degree', this might let you bound the amount of misalignment you can expect out of doing RL on this. However, actually formalizing this seems pretty conceptually fiddly. What follows is my initial attempt at doing this, though I didn't reach a conclusion. --- Consider the space of reward functions that assign either 0 or 1 to a rollout. If R1 and R2 never assign 1 to the same rollout (so their support is completely disjoint) then you probably expect them to have quite different effects. Whereas if R3 and R4 differ only on 1% of rollouts then maybe they are quite similar. So one naive notion is "% of rollouts on which they get the same score." But this is probably too simplistic, e.g. if R3 and R4 only differ on 1% of rollouts but those rollouts exhibit a consistent pattern then there might still be a noticeable difference in the kind of behaviour they encourage (concretely, you get conditional misalignment or other backdoors) Also this uniform averaging over samples doesn't account for the fact that some rollouts are more likely than others, some rollouts might be especially critical for learning and thus updated more on, etc. It'd be ideal if there was some principled definition that follows cleanly from first principles. Further research may be needed.