A worked example of an idea from physics that I think is underappreciated as a general thinking tool: no measurement is meaningful unless it's stable under perturbations you can't observe. The fix is to replace binary questions ("is this a degree-3 polynomial?", "is this a minimum?") with quantitative ones at...
This is my attempt to summarize Infra-Bayesianism probability theory at a level approaching "high school class in probability", as opposed to the "math-major class in probability theory" of the original. I aimed to focus on including simple exercises (with answers given in 'the back of the book') of the kind...
Yup! A math-ier version of the insight I was trying to convey is to say that, if you imagine dealing with math where you have a handful of these limit-y properties floating around, it may well matter what order you take those limits in! And at the same time that the limit-y-ness of these properties is usually quite “hidden”, so that this becomes a very useful mental motion.
Ahah I have Claude's system prompt set to default to Chinese so I can practice. Since my speaking/writing abilities suck much worse than my reading, I also told it to nag me to write in Chinese for the sake of practice lol. This works... variably well.
Yup I definitely agree there's no special role for unicellular attackers - I was eliding the complexity for brevity. I think the asymmetry still broadly holds meaningfully - e.g. multicellular parasites are very complex attackers but have much longer generation-times (I assume?), so they too trade off online vs offline optimization bits. Nonetheless the host organism still has more complexity to draw on for most things with which the immune system is concerned.
Interesting to think about the pareto frontier of offline vs online optimization. The multicellular parasites and unicellular microbes would be paradigm examples. But the microbiome gives lie to this idea - it is complex and organized but highly adaptive still because selection can act on the lower level. Perhaps being ~commensal/mutual instead of adversarial is related? I don't know.
The linked Claude conversation doesn't share the markdown file unfortunately. Apologies. Here is a gdrive link https://drive.google.com/file/d/1wpPGI7poP04ZMDoPU_lEh8D1CI8kOtic/view?usp=sharing I read it and it was a good introduction but it did a mediocre job of reframing things 'as an optimizer' or even 'as a control system'.
Note: Skimming, Claude hallucinates what Alon's 'periodic table of diseases' is. He has a pretty good youtube video on it you can watch instead. https://www.youtube.com/watch?v=ZMz_C778WMY&pp=ygUeYWxvbiBwZXJpb2RpYyB0YWJsZSBvZiBkaXNlYXNl
The Immune System as Anti-Optimizer
We have a short list of systems we like to call "optimizers" — the market, natural selection, human design, superintelligence. I think we ought to hold the immune system in comparable regard; I'm essentially ignorant of immunobiology beyond a few YouTube videos (perhaps a really fantastic LW sequence exists of which I am unaware), but here's why I am thinking this.
The immune system is the archetypal anti-optimizer: it defends a big multicellular organism from rapidly evolving microbiota. The key asymmetry:
A worked example of an idea from physics that I think is underappreciated as a general thinking tool: no measurement is meaningful unless it's stable under perturbations you can't observe. The fix is to replace binary questions ("is this a degree-3 polynomial?", "is this a minimum?") with quantitative ones at a stated scale. Applications to loss landscapes and modularity at the end.
Something which I think highly relevant, and which might inform your GAN discussion, is the difference in performance of W-GAN - basically if you train the GAN using an optimal transport metric instead of an information theoretic one, it seems to have much better robustness properties, and this is probably because shannon entropy doesn't respect continuity of your underlying metric space (e.g. KL divergence between Delta(x0) and Delta(x0 + epsilon) is infinity for any nonzero epsilon, so it doesnt capture 'closeness'). I don't yet know how I think this should tie into the high-probability latent manifold story you tell, but it seems like part of it.
I agree Thom's work is interesting and relevant here; I've seen it bruited about a lot, but likewise haven't gotten around to seriously studying it. From my perspective, structural stability is extremely important, but I am curious if there is any reason to identify this kind of stability with catastrophes? I've always looked at the classification and figured its just one example of this kind of structural stability, and not one I've really even seen much of in physics, compared to the I-think-close-but-not-exactly-the-same more general idea of critical phenomena and phase transitions. I'd be very interested if there was some sort of RG-based stability analysis.
The dynamical systems way of framing this I like the most is Koopman analysis; Steve Brunton has a great family of talks on on youtube: https://www.youtube.com/watch?v=J7s0XNT96ag
Tldr is that the Koopman operator evolves functions of the dynamical variables; it's an infinite dimensional linear operator, thus it has eigenfunctions and eigenvalues. Conserved functions of state have zero-eigenvalue; nearly conserved quantities have 'small' eigenvalues, corresponding to a slower time rate of decay. Then you can characterise all functions in terms of the decay rate of their self-correlation. So the dominant koopman eigenvalue of a (not necessarily eigenfunction) is a scalar value corresponding to how "not conserved" it is.
You can also look at the... (read more)
This is my attempt to summarize Infra-Bayesianism probability theory at a level approaching "high school class in probability", as opposed to the "math-major class in probability theory" of the original. I aimed to focus on including simple exercises (with answers given in 'the back of the book') of the kind I find helps in learning to do computations. I'm still writing the answer-sheet, so there may be mistakes / blanks.
It's been sitting on my desk for a while and I figured I'd post it 80%-baked rather than never - please feel free to leave (polite) comments suggesting improvements or noting errors, calculational or interpretational.
The summary is linked here: Link
Remarks on the Slow Boring post about "Kritiks" in debate, copied here for a friend who wanted to reference them, and lightly edited to fit the shortform format:
In the format of debate I do ("British Parliamentary" (BP), also my favorite, having also done "Public Forum"), this sort of thing is sorta-kinda explicitly not allowed. Ironically, I think BP is somewhere where this would be most appropriate - because you don't have the 'motion' until 15 minutes prior to start, focus on ground-facts is low (only relatively, compared to other research-heavy styles); you spend a lot of time arguing about "moral weighing-criteria" and "framing" and things like this.
We do have a pretty clear... (read more)
As a comment on identity in science: certainly it's not about ‘personal/agentic’ identity, but I have been thinking a lot about how we draw boundaries between objects - like “when” a group of 3 quarks “is” a proton. Generally this involves specifying a “scale” parameter expressing ~ how much information we’re willing to lose in abstracting away from details - then we take, symbolically, the limit where this parameter goes to infinity. Then you can use perturbation theory to bridge lower-level effects to cash out at the higher-level (eg how quark structure effects proton collision statistics)
For systems of agents, the analogue is, from the perspective of the group-level, ‘coherence’, or ‘incentive-compatibility’ from the sub-agents’ perspective. Unfortunately we don’t really have the tools to do the analogue of perturbation theory in these more complicated cases. It seems like the salient difference is that we can’t really scalarize ‘incoherence’, as there are too many saliently distinct ways for a group to be ‘incoherent’ and no natural commensurability between them.