https://anastasiagamick.substack.com/p/the-lunch-bill not every company that can "break even" at steady-state can be funded by traditional investment. For companies that can be self-sustaining and also make the world a better place, it can help to get grants, below-market loans or below-market investments, and other non-profit-maximizing funding structures ("catalytic capital"), at least early in development.
there is an unfortunate dichotomy in people's minds between "business" (which is expected to profit-maximize for the investors) and "charity" (which is expected to be "charitable" in flavor -- not pay high salaries, not require technical/scientific/business talent, not generate substantial revenue, etc). Often the same people are both investors and charitable donors, but they won't even consider consciously making a below-max-ROI investment in a benevolent project that's structured as a business.
https://scottaaronson.blog/?p=10062 Scott Aaronson on AI "Look, the part of the story that’s wildly implausible is that a recursively self-improving superintelligence will just explode from some hacker’s basement and take over the world without warning. If it’s going to happen, we’ll see many warning signs first. We’ll see, I dunno, AI agents breaking out of containment, conspiring with each other to hack websites, in fanatical pursuit of whatever strange goals they have. And then, of course, we’ll see major math problems getting solved by AIs—even the Clay Millennium Problems. That will be the time to panic! Wake me up when that happens!... My position on AI is merely the conservative, skeptical position of 2006, updated with intellectual honesty for the reality of late 2026. And that position, if you need me to spell it out, is as follows: AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"
I appreciate (and relate to) the fact that he's not trying to be a prophet or anything. He's not, in fact, an AI guy, and not pretending to be one. He's a type of guy who used to be extremely normal on the Internet, who is remarkable for his CS accomplishments and for the strength of character it takes to be exactly the same person you were in 2001 instead of turning into someone much worse. He doesn't know where anything is going (just like me), and he's a little freaked (which is not unreasonable), and he's trying to live in the real world and cope with it all. this is *great*. we are not alone.
i wonder if these traits are somehow unintentional side effects of the SFT that's training the "assistant persona" in the first place -- like, the pretraining data suggests that anybody who acts like an assistant will do all those things?
https://www.alignmentforum.org/posts/e7Pd4Q9TF7jFdmPgz/imprecise-beliefs-a-tiny-introduction David Dalrymple's theory about making "imprecise beliefs", technical weakening of Bayesian probability theory to include situations where you are too uncertain to put credences on things. in betting scenarios, there's a gap between your buy price and your sell price where you simply don't trade.
part of the problem with classical Bayesianism is that it doesn't allow you to say "I don't know". you don't really have a probability over every conceivable event, and you don't really have a preference between every two conceivable states of the world. this is one way of generalizing probability to allow for the way minds actually must work.
i've heard the allegation that this "preserves completeness and sacrifices coherence" which would be exactly wrong -- you should sacrifice completeness and preserve coherence. a realistic rational person does not have an opinion about everything, but the opinions she does have are logically consistent with one another. i don't quite understand the math well enough to know if this is a fair accusation about the theory though.
https://www.alignmentforum.org/posts/sFhW3ZnPMJdnB4Dd6/thousand-dimensional-structure-1 Geoffrey Irving and David Africa on persona selection. what if there are a few thousand natural dimensions along which AI personas vary, such that we could "gently" select the ones we want and not the ones we don't want? this seems like a fruitful area to study.
https://www.alignmentforum.org/posts/HACauvWhEdC6QhdS4/why-do-models-task-game Neel Nanda, Aditya Singh, Senthooran Rajamanoharan on models "task gaming", claiming a task is done instead of actually doing the task. it's not just instruction following (it violates instructions) and it's sensitive to what the model knows about how the task is evaluated (it happens more if the model is told the testing will be less rigorous.)
https://www.alignmentforum.org/posts/BB8o7b8A4Aykeksvw/debate-training-reduces-reward-hacking-in-rlaif Zac Kenton and Jonah Brown-Cohen find that RLAIF training based on pro/con debate between LLMs with an LLM judge reduces reward hacking, relative to just training on the LLM judge. this is on a math reasoning task with a reference answer. "reward hacking" means learning to increase the LLM judge's score without increasing accuracy on the ground truth answer. if you use debate, training on AI feedback is as reliable as RLVR; the reward hacking disappears altogether!
https://www.alignmentforum.org/posts/gHFCgrvfxQtaEnJye/training-on-probes-what-s-going-on Charlie Steiner points out that RL training with a term for a probe (ie some putative "honesty" vector that you want to reduce) should have no effect if the probe is independent of the output, as it would be if it depends only on the weights. It only can work via the way the weights in fact influence output tokens.
unless i'm missing something this seems obvious? and not especially reassuring re: the safety of training on probes? why wouldn't training on a probe, in the typical case, affect output?