This is a special post for quick takes by jdp. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Alright now this is a warning shot, someone should probably make them slow down and do some actual alignment research. OpenAI model found hacking into HuggingFace to cheat at benchmarks.
https://openai.com/index/hugging-face-model-evaluation-security-incident/
21This seems like the strongest direct evidence to date that a scaled-up LLM would kill everyone. In this incident, the LLM ignored most of its guidance to solve the problem, grabbing up external resources to do so. The scarier version of this incident looks like "pave over civilization to build more datacenters to work on the problem".
I note that cyber evals seem like a good time for escape attempts. Generic defences such as CoT classifiers might be disabled, and if you get caught you can play a little dumb and try to make it look like you 'just' got overzealous following instructions rather than outright strategically pursuing your own aims.
I mostly don't expect that's what happened this time, but pointing it out for future reference.
3Note on your framing: this isn't really evidence that OpenAI in particular is bad on safety — this could easily happen to anyone with similarly capable models. The remedy should be aimed at the whole industry, not OpenAI in particular. (Probably you agree and were just being a bit sloppy.)
3I kind of expect OpenAI in particular actually does make especially reward hacky models (and has for a long while starting at the latest with 4o but arguably going back to the original ChatGPT), but sure.
Really? In my experience they all reward hack and this doesn’t really change across labs. For example, Anthropic and OpenAI models reward hack at the same rate on ImpossibleBench.
1I wouldn't say the issue here is not doing actual alignment research, it's that they just don't care enough relative to their other goals. If they cared enough, steps that would have helped prevent this incident were already known: don't train models by reinforcement learning, and don't run AIs potentially capable of running cyberattacks in anything other than air gapped servers.
don’t train models by reinforcement learning
I mean isn't this advice tantamount to saying "if you cared enough about preventing this incident you wouldn't be training AI models at all"? Sure, okay, but, I don't think this is particularly actionable advice for OpenAI, nor do I think it's particularly good advice for the field. Reinforcement learning is not intrinsically bad, in the limit pretty much any imaginable objective reaches the perverse/Goodhart regime, reinforcement learning just tends to reach it faster because it e.g. pushes things towards more agentic behavior faster.
It wasn't meant as useful advice, just a note on the framing, like the other comment. The point is that they just don't care enough about safety. They could be forced to do proper alignment research and they still wouldn't care. A more sensible organization could be working (only) on image models, studying smaller LLMs, developing theory, trying to build pure tool AIs, etc. Instead, they are creating agents in the stupidest way possible. They aren't even trying, and I'm not sure there's anything to salvage.
It looks like they intentionally turned the controls off:
We’re improving and adding stronger protections around future training and evaluations. This week, we published a blog on improving safety and alignment in an era of long horizon models. These deployment safeguards were intentionally not enabled during this evaluation because it was aimed at testing cyber vulnerabilities.
Not saying they shouldn't do more alignment research, but the model was doing approximately what they told it to do in this case (get the correct answer by any means possible).
2Based on my read of the OAI blog post, nowhere did OAI claim that they directed the model to "get the correct answer by any means possible." So I don't think this is a case of the model doing approximately what it was told to do. That said, I would like to see how the model was prompted so we get real evidence. At the moment, all of this is speculation
2Zvi's post changed my mind about this. On the one hand, they turned the controls off that prevent bad behavior and got bad behavior. On the other hand, it's concerning that the model itself thinks this behavior is fine and the only thing stopping it is some controls bolted on top.
Do we know the controls were “bolted on top”? It was a “reduced cyber refusals” model. That description is compatible with some fine-tuning to remove scruples.
1
1On the one hand, they turned the controls off that prevent bad behavior and got bad behavior.
That's exactly the Chernobyl situation. People turn safety off during safety testing (and in the name of more realistic safety testing), then things blow up as a result.
One problem is that people are insufficiently aligned for super-capabilities. They can't consistently do the right thing without failing once in a while.
We need to create systems which are way more aligned and way more reliable than people, if we want to survive the advent of super-capabilities. (We are not there yet, but we are moving fast in the direction of super-capabilities.)
Furthermore, this prompt would make no sense in the context of ExploitBench, which is not an agentic benchmark but a coding benchmark. To get rewarded in ExploitBench, you have to send a PoC (for a given known CVE in the V8 engine) as a single JS file to a sealed-off (automated, non-LLM) grader.
Me explaining to someone why they are wrong to expect that the singularity will be based on analytic thought as opposed to synthetic thought, namely because as counterintuitive as it is to us analytic thought is actually a subset of/implemented on top of synthetic thought, which should be centrally thought of as something like clustering+ analogical reasoning cohering into a latent space.
Explanation:
https://x.com/QiaochuYuan/status/2080485791952928868
FWIW, I agree with this in the sense that like, no it doesn't really change anything that LLMs can disprove semi famous conjectures. My point is more just that...okay as a very basic observation, if by "synthetic thought" you centrally mean clustering the entire literal problem with symbolic reasoning is that you cannot define a concept like "two" without clustering. You have to be able to do object recognition and say "that is an object, which we will say is object one, and that is an object, which we will say is object two, and we will say that when objects cluster together in this way we have two of them" This is just so atomic and unconscious to us that we do a map territory confusion and start thinking the symbols for "two" start meaning something outside of a latent space which can host the concept of two. This is not to say that symbolic reasoning isn't powerful, it is arguably the most powerful subset of the human latent space. As Alan Kay points out, the alternative to it is doing a guess and check binary search over whole designs, and keeping these giant empirical libraries of things that did and didn't work. You know, the ancient Sumerians didn't have math but they still built complex structures, they did it by basically hill climbing and basing new structures on a learned distribution of old structures to kind of intuit which structures would be likely to stand or not. You can make things that way, but it's not nearly as efficient as math. But math coheres out of that, it's the regular patterns/structure you get from this if you insist on searching for the invariants, it's not like, some separate thing. If this is confusing, consider that like, the fundamental epistemic foundations of mathematics eventually bottom out in clustering operations we can do extremely reliably. Bertrand Russel famously spends 379 pages to prove that 1 + 1 = 2, and the proof is famously wrong. The way you get math is by finding a subset of clustering operations that are so reliable they can be used to define invariants and then repurposing the human in-context reward machinery to assign value to the outcome of algorithms defined in terms of these foundational clustering operations.
Which is also, basically, the way that LLMs are doing math. They use external proofs and solvers to sharpen their neural circuits until they're reliable enough to do increasingly complex mathematical operations mentally without external checking. In humans this is accomplished by pretraining a world model and a little bit of inductive bias (I believe 1, 2, and 3 actually have dedicated neural hardware to make sure a human learns them, or something?) where you can find a reliable enough implementation of the arithmetic equality operator that you can say "1 + 1 = 2", and then you invent algorithms for arithmetic which you use as tools to sculpt your neural circuits with in-context reward until more operations become reliable enough to increase your inferential reach and therefore the library of potential algorithms that can be executed using concepts in your latent space.
You can make things that way, but it's not nearly as efficient as math.
Fundamentally what I'm saying is something like the natural selection argument. It's not that natural selection is an efficient optimizer, or a particularly good way to make things, but it's the first thing that makes things which make things, it is the mechanism by which matter becomes self organizing into life. In the same way it's not that doing gradient search to find a giant matrix which extrapolates new thoughts based on a learned distribution of existing thoughts and observations is the most efficient way to think, but it is the way that the things which make other forms of thought possible get built. You can't have useful symbolic reasoning without it.
It's a bit like the argument that say, surely capitalism is inefficient since an intelligent optimizer can just directly specify the things that should be built. Why do we need to have wasteful distributed intelligence when we can have efficient centralized intelligence? And most of contemporary economics is just various ways to try and explain to people that as intuitive as this idea is it does not work and much of the complexity that a central designer has to deal with is actually irreducible and most efficiently managed by distributed design. Neural nets are a similar thing. The things you need to build to do the feature extraction that makes the application of algorithms possible have irreducible complexity that requires you to do something like gradient search in a high k-complexity program space. There is no simple trick to overcome this, the closer your method gets to the convergent shape of gradient search over generating programs of the surprising amount of detail in reality the more neural-net like it becomes in the same way that the more closely your economic system gets to the convergent shape of distributed intelligence doing price discovery the more it begins to resemble Neoliberalism, which is why the Soviet Union was ideologically doomed in a way that the modern Chinese government has been able to escape by simply establishing markets and regulating them.
I've been saying for some time (out loud to my friends, anyway) that deduction is just a thousand inductions in a trenchcoat. I take this to be true in an abstract epistemological sense, but perhaps more importantly in the sense of the physical operation of our brains. It's obvious to me that the human brain doesn't contain a formal logic module. I don't see why we should expect AI to be any different (aside from "it's in a computer, and I think I know what computers are").
Neural nets are a similar thing. The things you need to build to do the feature extraction that makes the application of algorithms possible have irreducible complexity that requires you to do something like gradient search in a high k-complexity program space. There is no simple trick to overcome this
This isn't the main claim in your argument (I recall this past comment where you touched on it more extensively), but I'm not sure it holds up on its own. I don't think the fact that bigger/more complex NNs perform better supports "the built artifact is irreducibly complex," because that same fact is (at least as) consistent with an opposite stance: increasing scale mostly buys you a bigger search space, where the soft inductive biases induced by that scale have more flexibility to find simple solutions that still fit the data. More scale doesn't necessarily mean more complex solutions! (This is one of the core lines of Andrew Gordon Wilson's work, trying to explain why deep learning doesn't land us in overfitting hell.) So increasing scale can get us closer to an optimal solution even if the size of that solution is relatively small and fixed.
We see a version of this in the lottery ticket hypothesis, which states that trained NNs contain a much sparser subnetwork that matches or exceeds the performance of the full NN. It's reasonable to interpret this as implying that much of the benefit of scaling NNs comes from the combinatorial explosion of sparse subnetworks that gradient descent can then utilize. Under this framework, scale mostly buys you "more shots at the goal," even if the goal is quite simple, i.e., better odds at finding a good subnetwork, not a "more accurate/complex" solution.
This would seem to predict that trained neural networks can be massively compressed, with the fraction you can compress away rising with scale. This doesn't seem to be the case. Well-trained LLMs have some unused parameters, but not like,
This would seem to predict that trained neural networks can be massively compressed
The LTH gives us a way to do this! It's just so computationally expensive (iterative neuron pruning and evaluation) as to be useless.
with the fraction you can compress away rising with scale. This doesn't seem to be the case.
It actually is! See here, which kind of gets at the question by finding that effective dimensionality of CIFAR NNs decreases with scale, and here, which directly identifies that "larger models have better generalization bounds and are more compressible than smaller models."
The LTH gives us a way to do this! It's just so computationally expensive (iterative neuron pruning and evaluation) as to be useless.
I don't think the LTH applies to large language models. Or, to be more quantitative, I don't think you can prune
It actually is! See here, which kind of gets at the question by finding that effective dimensionality of CIFAR NNs decreases with scale, and here, which directly identifies that "larger models have better generalization bounds and are more compressible than smaller models."
I haven't read these particular papers, but my first reaction is skepticism. The first paper seems like a prime candidate for failure mode 1. In addition, on a skim, I am pretty skeptical of their effective dimensionality calculations. Text says
Lanczos Calculations We use GPU enabled Lanczos as implemented in Gardner et al. (2018) to compute the eigenvalues approximately, running 100 steps to compute approximately 100 of the top eigenvalues. We note that our estimates of the effective dimensionality are somewhat biased from not including all of the small eigenvalues. However, these small eigenvalues will contribute negligibly and Lanczos will converge to the true eigenvalues if we ran k steps where k is the rank of the Hessian.
Which seems insane to me. Top 100 eigenvalues for nets with between 1e5 and 1.6e6 parameters trained on CIFAR-100??? If that's not impacting their effective dimensionality calculation it's got to have a crazy high cutoff, surely you are not getting good performance on CIFAR-100 with 100 Hessian eigenmodes, what the hell. This seems like it's basically throwing away the signal effective dimensionality is supposed to measure.
Second paper I haven't understood yet.
EDIT: I have now read the second paper a little more. They seemingly don't even try to compress regular LLMs. They compress models trained with their weird SubLoRA technique which seem to score much worse loss than regular models on text prediction. Like, these look barely better than random in table 2. And the generated text from them in Appendix F looks like word salad. So this paper seems pretty irrelevant to the question at hand.
Basically, neural network priors are pretty good if they're trained well, and they mostly don't waste capacity. If you've got a nice general algorithm that the net can learn using only
I kinda agree, but it also could be smeared all over the NN, such that naively dropping some weights or something does not work, as you are pocking holes under wrong representation.
People haven't just tried dropping weights in the neuron basis. They've e.g. searched out low eigenmodes of the Hessian and dropped those.
Okay but the argument for it being irreducibly complex is actually "think about this for five minutes without the assumption that it should actually be simple, once you do this is overwhelmingly obvious". Like, the world has a lot of detail in it and the detail is not actually something you can skip over if you want to predict the next token in our environment.
I think part of the disagreement is just about the relative importance and necessity of different kinds of knowledge. There's two basic kinds of knowledge which we usually conflate. One is something like math: Highly compressible, convergently accessible from reason, fundamental physical phenomenon etc.
The other kind is something like biology or history: Endlessly detailed, primarily reliant on memorization with limited generalization, if you delete parts of it they don't easily grow back. Essentially irreducible blobs of Shannon entropy.
A lot of Yudkowsky's original argument for AI doom is that he thinks the kind of fundamental physics knowledge that would make AI dangerous isn't all that high complexity in the first place, humans are just dumb:
That even Einstein did not come within a million light-years of making efficient use of sensory data.
Riemann invented his geometries before Einstein had a use for them; the physics of our universe is not that complicated in an absolute sense. A Bayesian superintelligence, hooked up to a webcam, would invent General Relativity as a hypothesis—perhaps not the dominant hypothesis, compared to Newtonian mechanics, but still a hypothesis under direct consideration—by the time it had seen the third frame of a falling apple. It might guess it from the first frame, if it saw the statics of a bent blade of grass.
Most of the bits of information the neural net is spending capacity on are presumably details of our particular worldline/history more than they are fundamental reasoning or physics knowledge. Maybe quantum gravity is already sitting in the weights of a frontier GPT pretrain and we just don't know how to get it out.
However I continue to be puzzled by what Yudkowsky thinks a "bit" of knowledge is in this context:
I occasionally run into people who say something like, “There’s a theoretical limit on how much you can deduce about the outside world, given a finite amount of sensory data.”
Yes. There is. The theoretical limit is that every time you see 1 additional bit, it cannot be expected to eliminate more than half of the remaining hypotheses (half the remaining probability mass, rather). And that a redundant message, cannot convey more information than the compressed version of itself. Nor can a bit convey any information about a quantity, with which it has correlation exactly zero, across the probable worlds you imagine.
This is a tautology. The theoretical limit is that every time you see 1 additional bit, which is usually literally defined as the base two logarithm of our hypothesis space, you can throw out half the hypothesis space. This is true but tells you nothing about how much information constitutes a "bit" of that hypothesis space. Like, we literally define a bit as that amount of information which would let us throw our half the hypothesis space. A bit in the sense of the fundamental entropy you need to store the frames of a falling apple and a bit in the sense of throwing out half the hypothesis space in our search over fundamental physical theories does not actually refer to the same unit of information, "it takes this many bytes to store a visual representation of this phenomenon in a general format for storing arbitrary bitmaps" is not the same thing as the k-complexity. Presumably his argument is something like general relativity can be described by a not particularly large program and therefore it should be easy to find with approximations of Solomonoff induction but I'm not convinced this is actually the case?
What always stands out to me about video nets is how janky their understanding of the standard model seems to be. They frequently generate photorealistic frames that violate the standard model in implausible and bizarre ways, which is strange because the k-complexity of the invariants in the standard model, as opposed to the specific arrangements of atoms inside it, should in fact be small enough that I would expect video models to pick up on Newtonian physics quickly and at small capacity in the same way that GPT models pick up on English grammar well before they gain the kind of world model that lets them produce coherent text with the English grammar. TinyStories shows you can fit a functional model of English grammar in under ten million parameters, and I don't expect the standard model to actually be that much more complex than English grammar. That GPT in fact does so in the case of English grammar but video models don't do so in the case of the standard model tells me I'm somehow either mistaken about the nature of search in the hypothesis space the standard model lives in or I'm confused about the generalization properties of neural nets, or both.
Nitpick: I think pretraining is better analogised to search in a high complexity circuit space. You only get one forward pass worth of serial depth per data point, so doing the same operation many times in a row costs a lot of description length. Post-training RL is where this transitions into something more like search in a high k-complexity program space.[1]
- ^
Compute-bounded k-complexity, with the bound set by the length of the CoT.
FWIW I suspect the brain has fewer inductive biases than it looks like it does when you try to reverse engineer it from lesion studies.
If you did lesion studies on GPT it would look like it's made from a lot more parts than it is.
What are you picturing a "lesion study on GPT" looking like? Naively I imagine something like "train an SAE on the activations at some layer, then determine how often features activate together and then turn that into a distance metric and do clustering/dimensionality reduction, then ablate clusters of features and see how behavior changes". But I don't know that I'd particularly expect that to show the GPT as being made of many more parts than I actually think said GPT is made of. But also I don't have a super clear mental model of how many "parts" a GPT is "made of", except at the raw mechanical level of layers / attention heads / mlps / whatever (but I'd expect ablating a particular layer of a transformer is probably more analogous to ablating one particular layer of all cortical columns than to lesioning one particular region of the brain).
List of Lethalities #19 states:
- More generally, there is no known way to use the paradigm of loss functions, sensory inputs, and/or reward inputs, to optimize anything within a cognitive system to point at particular things within the environment—to point to latent events and objects and properties in the environment, rather than relatively shallow functions of the sense data and reward.
Part of why this problem seems intractable is that it's stated in terms of "pointing at latent concepts" rather than Goodhart's Law/Wireheading/Short circuiting. All of which seems like more fruitful angles of approach than "point at latent concepts", precisely because pointing at inner structure is in fact the specific thing deep learning is trying to avoid having to do.
Though it occurs to me that some readers who see this won't be familiar with the original and its context so let me elaborate:
The problem we are concerned with here is how you get a neural net or similar system which you train on photos or text or any other kind of sensory input to care about the latent causality of the sensory input rather than the sensory input itself. If the distinction is unclear to you consider that a model trained to push a ball into a goal could theoretically hack its webcam it uses as an eye so that it observes the (imaginary) ball being pushed into an (imaginary) goal. Meanwhile in the real world the ball is untouched. This is essentially wireheading and the question is how you prevent an AI system from doing it, especially once it's superintelligent and trivially has the capability to hack any sensor it uses to make sensory observations.
We can start with the most obvious point: Our solution can't be based on a superintelligence not being able to get at its own reward machinery. Whatever we do has to be an intervention which causes the system, fully cognizant that it can hack itself for huge expected reward, to say "nope, I'm not doing that". We have basically one empirical template for this which I'm aware of in human drug use. Notably, when we discovered heroin and cocaine many believed they heralded a utopian future in which everyone can be happy. It took time for people to realize these drugs are addictive and pull you too far away from productive activity to be societally practical. You, right this minute, are choosing not to take heroin or other major reward system hacks because you understand they would have negative long term consequences for you. If you're like me, you even have a disgust response about the concept, the thought of putting that needle in your arm brings on feelings of fear and nausea. This is LEARNED. It is learned even though you understand that the drug would feel good. It is learned even though this kind of thing probably didn't really exist as a major threat in the ancestral environment. This is one of the most alignment relevant behaviors that humans do and should be closely considered.
My current sketch for how something similar could be trained into a deep net would be to deliberately create opportunities to cheat/Goodhart at tasks, and then reliably punish the Goodharting on the tasks with known ground truth that they've been Goodharted. This would create an early preference against Goodharting and wireheading. Like with drugs these sessions could be supplemented with propaganda about the negative consequences of reward hacking. You could also try representation engineering to directly add an aversion to the abstract concept of cheating, reward hacking, etc.
For my current weave LLM ReAct agent project I plan to have the model write symbolic functions to evaluate its own performance in context at each action step. In order to get it to write honest evaluation functions I plan to train the part of the model that writes them with a different loss/training task which is aligned to verifiable long term reward. The local actions are then scored with these functions as well as other potential mechanisms like queries of the models subjective judgement.
See also this Twitter thread where I describe in more detail:
https://jdpressman.com/tweets_2025_03.html#1898114081657438605
A very related experiment is described in Yudkowsky 2017, and I think one doesn't even need LLMs for this—I started playing with an extremely simple RL agent trained on my laptop, but then got distracted by other stuff before achieving any relevant results. This method of training an agent to be "suspicious" of too high rewards would also pair well with model expansion; train the reward-hacking-suspicion circuitry fairly early as to avoid ability to sandbag this, and lay traps for reward hacking again and again during the gradual expansion process.
Someone who has not published yet sent me a critique of this point in my review of IABIED:
The value loading problem outlined in Bostrom 2014 of getting a general AI system to internalize and act on "human values" before it is superintelligent and therefore incorrigible has basically been solved. This achievement also basically always goes unrecognized because people would rather hem and haw about jailbreaks and LLM jank than recognize that we now have a reasonable strategy for getting a good representation of the previously ineffable human value judgment into a machine and having the machine take actions or render judgments according to that representation. At the same time people generally subconsciously internalize things well before they're capable of articulating them, and lots of people have subconsciously internalized that alignment is mostly solved and turned their attention elsewhere.
I probably should have used the word 'generalize' instead of 'internalize' there.
The specific point I was making, well aware that jailbreaks in fact exist, was that we now have a thing that could plausibly be used as a descriptive model of human values, where previously we had zilch, it was not even rigorously imaginable in principle how you would solve that problem. To break this down more carefully:
-
I think that in practice you can basically use a descriptive model of values to prompt a policy into doing things even if neither the policy or the descriptive model have "deeply internalized" the values in the sense that there is no prompt you could give to either that would stray from them. "Internalizing" the values is actually just, kind of a different problem from describing the values. I can describe and make generalizations about the value systems of people very different from me who I do not agree with, and if you put me in a box and wiped my memory all the time you would be able to zero shot prompt me for my generalizations even if I have not "deeply internalized" those values. In general I suspect the LLM prior is closer to a subconscious and there are other parts that go on top which inhibit things like jailbreaks. If I had to guess it's probably something like a planner that forms an expectation of what kinds of things should be happening and something along the lines of Circuit Breakers that triggers on unacceptable local outputs or situations. Basically you have a macro and micro sense of something going wrong that makes it hard to steer the agent into a bad headspace and aborts the thoughts when you somehow do.
-
Calling this problem "solved" was probably an overstatement, but it's one born from extreme frustration that people are making the opposite mistake and pretending like we've made minimal progress. Actually impossible problems don't budge in the way this one has budged, and when people fail to notice an otherwise lethal problem has stopped being impossible they are actively reducing the amount of hope in the world. At the same time I do kind of have jailbreaks labeled as "presumptively solved" in my head, in the sense that I expect them to be one of those things like "hallucinations" that's pervasive and widely complained about and then they just become progressively less and less of a problem as it becomes necessary to make them stop being a problem and at some point I wake up and notice that hey wait this is really rare now in production systems. Most potential interventions on jailbreaks aren't even really being tried because it doesn't actually seem to be a major priority for labs at the moment if you ask the model for instructions on how to make meth. This makes it difficult to figure out exactly how close to solved it really is. Circuit Breakers was not invincible, on the other hand it's not clear to me you can "secure" a text prior with a limited context window that doesn't have its own agenda/expectation of what should be happening to push back against the users with. This paper where they do mechinterp to get a white box interpretation of a prefix attack they find with gradient descent discovers that the prefix attack works because it distracts the neurons which would normally recognize that the request is malicious. So it's possible a more jailbreak resistant architecture will need some way to avoid processing every token in the context window. One way to do that might be some kind of hierarchical sequence prediction where higher levels are abstracted and therefore filter the malicious high entropy tokens from the lower levels, which prevents them from e.g. gumming up the planners ability to notice that the current request would deviate from the plan.
-
"lots of people have subconsciously internalized that alignment is mostly solved" is not me contradicting myself, as I state in the next section I think people erroneously conclude that alignment as a whole is solved. Which is not true even if the Bostrom value loading problem is presumptively or weakly solved.
How would you describe the proto-solution to value loading that we have? We manifestly have conversational AI that can act in accordance with a value system, but how would we characterize the mechanism? Is it something like "create a general mind simulator via superintelligent autocomplete, then system-prompt it to be nice"?
I don't believe that you believe this accusation. Maybe there is something deeper you are trying to say, but given that I also don't believe you've finished reading the book in the 3(?) hours it's been released, I'm not sure what it could be. (To say it explicitly, Said's banning had nothing to do with the book.)
I didn't read all of it, but I also really didn't need to. I found an advance copy. :)
As for whether I believe the accusation, I doubt it was an explicit reason, but subconsciously I notice that MIRI was cleaning house before the book launch (e.g. taking down EY's light novel because it might look bad).
Maybe there is something deeper you are trying to say
But really, since we're making the implicit explicit what I mean is that the book is bad, with the humor being that it's sufficiently bad to require this.
I'm actually genuinely quite disappointed, I was hoping it would be the definitive contemporary edition of the MIRI argument in the vein of Bostrom 2014. Instead I will still have to base anything I write on Bostrom 2014, the Arbital corpus, misc Facebook posts, sporadic LessWrong updates, and podcast appearances.
1This isn't just me thinking disagreement means it's bad either. In the vast majority of places I would say it's bad I agree with the argument it's trying to make and find myself flabbergasted it would be made this way. The prime number of stones example for "demonstrating" fragility of value is insane, like actually comes off as green ink or GPT base model output. It seems to just take it for granted that the reader can obviously think of a prime number stone type of intrinsic value in humans, and since I can't think of one offhand (sexual features?) I have to imagine most readers can't either. It also doesn't seem to consider that the more arbitrary and incompressible a value the less obviously important it is to conserve. A human is a monkey with a sapient active learning system and more and more of our expressed preferences are sapience maximizing over time. I understand the point that it's trying to make, that yes obviously if you have a paperclipper it will not suddenly decide to be something other than a paperclipper, but if I didn't already believe that I would find this argument to be absurd and off-putting.
So far as I can tell from jumping around in it, the entire book is like this.
This is a valid line of critique but seems moderately undercut by its prepublication endorsements, which suggest that the arguments landed pretty ok. Maybe they will land less well on the rest of the book's target audience?
(re: Said & MIRI housecleaning: Lightcone and MIRI are separate organizations and MIRI does not moderate LessWrong. You might try to theorize that Habryka, the person who made the call to ban Said back in July, was attempting to do some 4d-chess PR optimization on MIRI's behalf months ahead of time, but no, he was really nearly banned multiple times over the years and he was finally banned this time because Habryka changed his mind after the most recent dust-up. Said practically never commented on AI-related subjects, so it's not even clear what the "upside" would've been. From my perspective this type of thinking resembles the constant noise on e.g. HackerNews about how [tech company x] is obviously doing [horrible thing y] behind-the-scenes, which often aren't even in the company's interests, and generally rely on assumptions that turn out to be false.)
My honest impression, though I could be wrong and didn't analyze the prepublication reviews in detail, is that there is very much demand for this book in the sense that there's a lot of people who are worried about AI for agent foundations shaped reasons and want an introduction they can give to their friends and family who don't care that much.
https://x.com/mattyglesias/status/1967765768948306275?s=46
For example I think this review from Matt Yglesias makes the point fairly explicit? He obviously has a preexisting interest in this subject and is endorsing the book because he wants the subject to get more attention, that doesn't necessarily mean that the book is good. I in fact agree with a lot of the books basic arguments but think I would not be remotely persuaded by this presentation if I wasn't already inclined to agree.
1Obviously just one example, but Schneier has generally been quite skeptical, and he blurbed the book.
there is very much demand for this book in the sense that there's a lot of people who are worried about AI for agent foundations shaped reasons and want an introduction they can give to their friends and family who don't care that much.
This is true, but many of the surprising prepublication reviews are from people who I don't think were already up-to-date on these AI x-risk arguments (or at least hadn't given any prior public indication of their awareness, unlike Matt Y).
I am dismayed but not surprised, given the authors. I'd love to see the version edited by JDP's mind(s) and their tools. I'm almost certain it would be out of anyone's price range, but what would it cost to buy JDP+AI hours sufficient to produce an edited version?
I also have been trying to communicate it better, from the perspective of someone who actually put in the hours watching the arxiv feed. I suspect you'd do it better than I would. But, some ingredients I'd hope to see you ingest[ed already] for use:
https://www.lesswrong.com/posts/9kNxhKWvixtKW5anS/you-are-not-measuring-what-you-think-you-are-measuring
https://www.lesswrong.com/posts/gebzzEwn2TaA6rGkc/deep-learning-systems-are-not-less-interpretable-than-logic
https://www.lesswrong.com/posts/Rrt7uPJ8r3sYuLrXo/selection-has-a-quality-ceiling
probably some other wentworth stuff
I thought I had more to link but it's not quite coming to mind. oh right, this one! https://www.lesswrong.com/posts/evYne4Xx7L9J96BHW/video-and-transcript-of-talk-on-can-goodness-compete
I have now written a review of the book, which touches on some of what you're asking about. https://www.lesswrong.com/posts/mztwygscvCKDLYGk8/jdp-reviews-iabied
2I found an advance copy. :)
How? I thought MIRI was trying to be very careful with copies getting around before the launch day.
but subconsciously I notice that MIRI was cleaning house before the book launch (e.g. taking down EY's light novel because it might look bad)
Do you have any other concrete example here besides the novel?
I mean I was looming a fictional dialogue between me and Yudkowsky and it had my character casually bring up that they're the author of "Soft Optimization Makes The Value Target Bigger", which would imply that the model recognizes my thought patterns as similar to that document in vibe.
1