Huh. Well, this is definitely not what I expected. Should update further against LLMs generalizing in predictable ways.
Also, the Celeste post specifically is not about fellowships only. It is about doing SOMETHING, including things like persuading everyone you can and handing flyers on the street.
I don't really place Sequence into the category of technical scientific iterature. It always read like an amazing science popularization series to me. Actual technical literature rarely forms communities.
Hey, you may not be giving it enough credit. There is at least one other point about mystification. I agree with you about the thought periment being fine.
Surfacing failures
This does not sound like a good scientific practice to me. Understanding why methods work and understanding rationalizing why they should not are different activities and a goal of "show that X is bad" seems to be the latter. Sometimes alignment approaches really do generalize better than we expected(CoT or even linear probes).
Scott Alexander has a post - "shameless guesses, not hallucinations". I think your take is similar to it.
Project Lawfull is another Yudkowsky's work that imho much better fits the description of "grimdark power-scaling quasi-isekai fantasy" than HPMOR.
The way that models switch preferences is still very chaotic. Sometimes, when given a choice between A and B, the order in which options are presented controls the answer(e.g. for Opus 5.5 "Which number do you like more - 195(194) or 194(195)?"). However the example above is inconsistent. It also could be that models just produce volatile answers, when they don't have any particular belief.
"Here are 10 reasons against a near-term AI slow-down"
You meant 9 reasons , right?
The problem with neutral information about LLMs is that for every safety researcher trying to use it for alignment, there may be 4 more capability researchers trying to squeeze more capabilities out of it. That is, I would expect capabilities side of humanity to be collectively smarter than alignment part due to sheer scale and funding.
But it is not yet possible to fuse NNs together into a superagent either! Fusing NNs via training would be a much bigger leap out of the distribution as opposed to making them better at coordination.
This first sentence was somewhat misleading to me. Normally, I would expect "crazy" to mean literal mental disorder, not "posting controversial stuff online". Not being aware of the apparent intensity of the political sitution, I was expecting to read about Musk being hospitalized after an incident of X.
Yeah, neither can they make coffee. I would actually expect teleoperation to be possible in mostly manual situations, where the only complex part is the expert knowledge(e.g. running a given biology/chemistry experiment in a lab may require basic lab skills and very advanced chemistry knowledge).
In fact, I woud predict that a large portion of articles in natural sciences ~5 years from now, will have most of its evidence created by AI operating a bunch of bachelours in a lab.
I don't think this is misalignment. Llama is very stupid, so simple math problem may be genuinely hard for it and expert opinion may be decisive. The fact that frontier models don't do that suggests that the reason is the relative complexity of the problem. If I got a solution to a large integral and then von Neyman looked at it and said I am wrong, I would also fold fast.
Wow. I can't wait to hear more about your research!
Edited. Originally contained a stupid question
This is probably not a useful comment, but I really want someone to reassure me that I am wrong, because I don't like the view I am presenting here.
On another note, serious physical activity is something people might want to avoid while thinking hard. I remember reading that cognitive performance is weakened when brain is concerned with coordinating your movement and maintaining balance. This probably applies only to harder physical activities like lifting.
I guess we would eventually go there. Machines are going to become better friends/partners at some point, because humans are flawed. Not simply "flawed like anything real", but too flawed. They have limited time and emotional energy and so on. If the majority of humanity will spent their time with AI that is capable of making them happier/more motivated/more productive than humans would won't benefits overshadow the downsides?
P.S. There is a creepy feeling coming from the world of people disconnected from each other, but I think it comes from the fear of the unknown(or known, but really weird like dating a rock). But I may be simply too biased.
Actually, I find this ending pretty nice. This AI does not seem to transform the universe into paperclips, it shows progress and values intelligence and as a bonus humans are still alive. A tyrant AI still seems to be human enough for me.
What does this predict? That RL would not typically breach patterns completely absent from pre-training? I really like the explanation, but I am not sure if it is testable with pre-training absorbing most of the human knowledge base...