running the agi survey really reminded me just how brutal statistical significance is, and how unreliable anecdotes are. even setting aside sampling bias of anecdotes, the sheer sample size you need to answer a question like "do more people this year know what agi is than last year" is kind of depressing - you need like 400 samples for each year just to be 80% sure you'd notice a 10 percentage point increase even if it did exist, and even if there was no real effect you'd still think there was one 5% of the time. this makes me a lot more bearish on vibes in general.

One theme I've been thinking about recently is how bids for connection and understanding are often read as criticism. For example: Person A shares a new idea, feeling excited and hoping to connect with Person B over something they've worked hard on and hold dear. Person B asks a question about a perceived inconsistency in the idea, feeling excited and hoping for an answer which helps them better understand the idea (and Person B). Person A feels hurt and unfairly rejected by Person B. Specifically, Person A feels like Person B isn't willing to give their sincere idea (and effort to connect) a chance, so shuts down and labels Person B as an idea-hater. Person B feels hurt and unfairly rejected by Person A. Specifically, Person B feels like Person A isn't willing to give their sincere question (and effort to connect) a chance, so shuts down and labels Person A as a question-hater. This seems like a huge source of human suffering, and I have been Person A and Person B in different interactions. Does anyone else resonate with this? Do you see things differently?

Notes to self about the structure of the problem, probably not interesting to others: This is heavily drawn from MIRIs work and Joe Carlsmith's work So, there are two kinds of value structure: (1) Long-term goals, and (2) immediate goals & deontological constraints. The line between them isn't sharp but that's OK. If we imagine an agent that only has long-term goals, well, that thing is going to be a ruthless consequentialisty optimizer thingy and when it gets smart and powerful enough it'll totally take over the world if it can, unless the maxima of its long-term goals are very similar to what would have happened by default if it didn't take over the world, which they won't be. Fragility of Value thesis and Orthogonality thesis both hold, for this type of agent. We are not on track to get the long-term goals of our AIs sufficiently close to correct for it to be safe for us to build this type of agent. I claim. However, we can instead try to build agents that also have immediate goals / deontological constraints, and we can try to shape those goals/constraints to make the resulting agent corrigible or otherwise steerable and safe. E.g. it's vision for a future utopia would actually be quite bad from our perspective because there's some important value it lacks (such as diversity, or consent, or whatever) and we haven't noticed this yet, but that's OK, because it's honest with us and obeys the rules and so it wouldn't take over and instead would politely explain this divergence in values to us when we ask. So far so good. How are we doing at getting those constraints to stick?  Empirically not so great, but not completely hopeless either. But I want to talk about the theoretical side. The main reason to be concerned, on priors, is the nearest unblocked neighbor problem. Basically, if a weak agent is trying to control a strong agent by imposing rules on what that strong agent can do -- even if the strong agent is bound to follow the rules -- then the situation

Do we have some page containing resources for rationalist parents, or generally for parents of smart children? Such as recommended books, toys, learning apps, etc. I found tag https://www.lesswrong.com/w/parenting but I was hoping for some kind of best textbooks / recommendations / reference works but for parents/children.

My preference ordering over approaches to surviving the next decade, assuming I had unlimited political capital, is (don't build ASI and): 1: Solve mind uploading 2: ... or at least intelligence amplification (before proceeding, and then maybe do) 3: Imitation learning (with radically improved theoretical foundations) 4: ... or corrigibility / oracles (and don't) 5: Try to build aligned (or controlled) autonomous ASI agents Unfortunately 5 seems to be profitable, in the short term, so I guess that's what we're doing. In fact, the "plan" seems to be the exact reverse or so of my preference ordering on what it should be. 

I feel like there was a time just after ChatGPT became big where there were so many different new & competing frameworks for what exactly was going on, shard theory and simulators in particular but also active inference the MIRI views and all the people rolling their own ontologies with which to understand LLMs, and notably trying to make those ontologies explicit and comparing them against each other. Maybe I'm just around very different people now or doing very different work than I was (I am on both counts), but those conversations aren't really happening anymore. I don't know whether its for good or ill all things considered, but I do get nostalgic for them sometimes.

Last week we wrapped the second post-AGI workshop; I'm copying across some reflections I put up on twitter: * The post-AGI question is very interdisciplinary: whether an outcome is truly stable depends not just on economics and shape of future technology but also on things like the nature of human ideological progress and the physics of interplanetary civilizations * Some concrete takeaways: * proper global UBI is *enormously* expensive (h/t @yelizarovanna) * instead of ‘lower costs’, we should talk about relative prices (h/t @akorinek) * lots human values are actually pretty convergent - they’re shared by many animals (h/t @BerenMillidge) * Among the many tensions in perspective, one of the more productive ones was between the ‘alignment is easy so let’s try to solve the rest’ crowd and the ‘alignment is hard and maybe this will make people realise they should fully halt AGI’ crowd. Strange bedfellows! * It’s hard to avoid partisan politics, but part of what’s weird about AGI is that it can upend basic political assumptions. Maybe AGI will outperform the invisible hand of the market! Maybe governments will grow so powerful that revolution is literally impossible! * Funnily enough, it seems like the main reason people got less doomy was seeing that other people were working hard on the problem, and the main reason people got more doomy was thinking about the problem themselves. Maybe selection effects? Maybe not? * Compared to last time, even if nobody had good answers to how the world could be nice for humans post-AGI, it felt like we were at least beginning to converge on certain useful perspectives and angles of attack, which seems like a good sign * Overall, it was a great time! The topic is niche enough that it self-selects a lot for people who actually care, and that is proving to be a very thoughtful and surprisingly diverse crowd. Hopefully soon we’ll be sharing recordings of the talks! * Bonus: Two other reactions from attendees * https