fwiw, this is why cosleeping and night nursing is convenient if you can manage it. you don't have to get fully conscious when the baby fusses at 2:14 AM, you just roll over, feed him, and fall more or less asleep while being gnawed. of course this requires good genetic luck and a little bit of willingness to tolerate actuarial risks, but it's so much easier than chronic sleep deprivation, and boy does the baby like it.
https://www.daniellitt.com/blog/2026/9/13/a-beginning-for-mathematics/ Daniel Litt's take on mathematics and AI. his main point is that AI is not going away, so we need to change the way we educate mathematicians so you can't just "vibecode" a PhD thesis you don't understand, or get hired as faculty on the basis of your ability to spend money on tokens.
a PhD should focus on a high-quality defense that demonstrates your ability to communicate deep expertise.
talks and sustained mathematical discussion should count for more, as those are unfakeable.
we also need to reward coming up with good conjectures and research programs.
ultimately it's not realistic to "reserve" certain math problems for humans-only. if it's possible to push a button and solve a problem, someone will, and someone will read the output. and this is not necessarily bad! if this is done right, we can learn more via the enhanced "reach" that automation brings!
if we just push buttons to get theorems we don't even read, eventually nobody will care enough about math to even push the buttons!
he thinks this is mostly a symptom of problems that existed in math before AI could do math -- treating papers as the be-all-and-end-all of academic status, a flood of low quality & incorrect papers, insufficient rewards for training and mentorship, superficial engagement with the material.
he argues (i think fairly convincingly) that "behavioral rewards" are inherently going to be gamed most of the time. the model will much more often learn tricks to get the reward without "doing what we mean", than it will learn to "do what we mean."
Byrnes' arguments do not depend on any "anthropomorphizing" moves like ascribing beliefs and goals to the model. if tricks for getting a reward without "doing what we mean" are common, and they tend to result in getting the reward more reliably than "doing what we mean", or are easier, simpler patterns to learn than the full nuanced picture of "what we mean", then reward hacking will predominate!
Byrnes' suggested solution looks like non-consequentialist rewards inspired by human social emotions; for instance, "model what a human would think of your thoughts and actions right now, and get reward if you predict a human would approve."
he claims that this is what "neurotypical" humans do all the time -- they are frequently experiencing "wow, my peers and idols would really approve of me right now" and this is what the emotion of "pride" is.
that seems huge if true. I don't have much ability to feel pride, and I also never have a pleasant sense that people would approve of me (when they aren't right there currently approving of me.) usually when I imagine what other people would think of me, I imagine their disapproval. it seems wild to me if other people are really going around thinking "good on me, I bet people approve of me"!
it also seems maybe questionable to give bots social anxiety and social status motivations. We often disapprove ethically of the behavior of people who are too obsessed with what others think of them!
one might also worry that this is also gameable. isn't training on "inner" states, like chain of thought or residual vectors, considered a great way to lose the monitorability signal that allows us to understand AI processes/"motivations" in the first place?