I really dislike the phrase "human values". I think it's confusing because: * It obscures a distinction between human preferences and normative values, i.e. what the author thinks is good in a moral sense * Insofar as the author thinks that fulfilling human preferences is good, they often leave this...
Some months ago I read the book "Talent" by Tyler Cowen and Daniel Gross. Published in 2022, it discusses how to spot talent using one's individual judgment (e.g. assessing a founder as VC). Importantly, it's not a book about how to create a standardized hiring process at a big company...
Some months ago I read the classic management book High Output Management and made a note of quotes that rang particularly true to me. I normally dislike this genre (management books), and disagree with some popular ones (I sympathize with this review of Scaling People, for example), but found High...
Excerpt from Dwarkesh Patel's interview with Andrej Karpathy that I think is valuable for LessWrong-ers to read. I think he's basically correct. Emphasis in bold is mine. > Andrej Karpathy 00:29:53 > > I guess I built the repository over a period of a bit more than a month. I...
Newcomb’s problem is a famous paradox in decision theory. The simple version is as follows: > Two boxes are designated A and B. The player is given a choice between taking only box B or taking both boxes A and B. The player knows the following: > > * Box...
AI risk arguments often gesture at smarter AIs being "more rational"/"closer to a perfect utility maximizer" (and hence being more dangerous) but what does this mean, concretely? Almost anything can be modeled as a maximizer of some utility function. The only way I can see to salvage this line of...
A few days before “If Anyone Builds It, Everyone Dies” came out I wrote a review of Scott’s review of the book. Now I’ve actually read the book and can review it for real. I won’t go into the authors’ stylistic choices like their decision to start every chapter with...
I think there is a tension with saying that society should lower the demands on parents, so that people have more kids
I think the only possible tension here is re. embryo selection. And it's not a real tension. The claim is something like "if what's giving you pause is the high demand on parents, just wing it and have kids anyway and anyhow" + "if you already know you want to have a kid and want to optimize their genes/happiness here are some ways to do it". I think most Rationalists would agree that the life of an additional non-embryo-selected, ordinary-parented child is still worth creating. Or in other words, one set of claims is about the floor of how much effort you can put in per child and it still be a good idea to have the child. The other set of claims is about effective ways to put more effort in if you want to (mainly what's discussed is embryo selection for health/intelligence).
I assume by "health-optimizing genetic manipulation" you mean embryo selection (seeing as gene editing is not possible yet). Indeed, Rationalists are more likely to be interested in embryo selection. And indeed, it is costly. But I'd say this is different from costly parenting - it's a one-time upfront cost to improve your child's genetics.
I ~never hear the 2nd thing among rationalists ("improve your kid's life outcomes by doing a lot of research and going through complicated procedures!").
Homeschooling is often preferred not because it substantially improves life outcomes but because it's nicer for the children and often parents. School involves a lot of wasted time/effort, and is frustrating and boring for many children. And so by homeschooling you can make their childhood nicer irrespective of life outcomes.
I was actually thinking to make a follow-up post like this. I basically agree.
Let's talk about two kinds of choice:
- choice in the moment
- choice of what kind of agent to be
I think this is the main insight - depending on what you consider the goal of decision theory, you're thinking about either (1) or (2) and they lead to conflicting conclusions. My implicit claim in the linked post is that when describing thought experiments like Newcomb's Problem, or discussing decision theory in general, people appear to be referring to (1), at least in classical decision theory circles. But on LessWrong people often switch to discussing (2) in a confusing way.
the core problem in decision theory is reconciling these various cases and finding a theory which works generally
I don't think this is a core problem because in this case it doesn't make sense to look for a single theory that does best at two different goals.
I think those other types of startups also benefit from expertise and deep understanding of the relevant topics (for example, for advocacy, what are you advocating for and why, how well do you understand the surrounding arguments and thinking...). You don't want someone who doesn't understand the "field" working on "field-building".
My bad, I read you as disagreeing with Neel's point that it's good to gain experience in the field or otherwise become very competent at the type of thing your org is tackling before founding an AI safety org.
That is, I read "I think that founding, like research, is best learned by doing" as "go straight into founding and learn as you go along".
I naively expect the process of startup ideation and experimentation, aided by VC money
It's very difficult to come with AI safety startup ideas that are VC-fundable. This seems like a recipe for coming up with nice-sounding but ultimately useless ideas, or wasting a lot of effort on stuff that looks good to VCs but doesn't advance AI safety in any way.
I disagree with this frame. Founders should deeply understand the area they are founding an organization to deal with. It's not enough to be "good at founding".
This makes sense as a strategic choice, and thank you for explaining it clearly, but I think it’s bad for discussion norms because readers won’t automatically understand your intent as you’ve explained it here. Would it work to substitute the term “alignment target” or “developer’s goal”?
I really dislike the phrase "human values". I think it's confusing because:
Some months ago I read the book "Talent" by Tyler Cowen and Daniel Gross. Published in 2022, it discusses how to spot talent using one's individual judgment (e.g. assessing a founder as VC). Importantly, it's not a book about how to create a standardized hiring process at a big company ("please recall that this is a book about talent search, not just a book about hiring"). For instance, the book advises tailoring an interview to the interviewee and having unstructured conversations, which is the opposite of what managers are instructed to do when interviewing candidates in a standard corporate environment (to avoid bias and coordinate on consistent standards).
The book is interesting largely... (read 4222 more words →)
Some months ago I read the classic management book High Output Management and made a note of quotes that rang particularly true to me. I normally dislike this genre (management books), and disagree with some popular ones (I sympathize with this review of Scaling People, for example), but found High Output Management pretty reasonable. It's also interesting to see the extent to which its recommendations continue to be followed in successful organizations to this date (the book was published in 1983, but is still popular and recommend amongst tech managers). This post is a list of my copied quotes (headings mine).
... (read 1340 more words →)Given a choice, should you delegate
Excerpt from Dwarkesh Patel's interview with Andrej Karpathy that I think is valuable for LessWrong-ers to read. I think he's basically correct. Emphasis in bold is mine.
... (read 1282 more words →)Andrej Karpathy 00:29:53
I guess I built the repository over a period of a bit more than a month. I would say there are three major classes of how people interact with code right now. Some people completely reject all of LLMs and they are just writing by scratch. This is probably not the right thing to do anymore.
The intermediate part, which is where I am, is you still write a lot of things from scratch, but you use the autocomplete that’s available now from these models.
The risk of incorrectly believing in moral realism
(Status: not fully fleshed out, philosophically unrigorous)
A common talking point is that if you have even some credence in moral realism being correct, you should act as if it's correct. The idea is something like: if moral realism is true and you act is if it's false, you're making a genuine mistake (i.e. by doing something bad), whereas if it's false and you act as if it's true, it doesn't matter (i.e. because nothing is good or bad in this case).
I think this way of thinking is flawed, and in fact, the opposite argument can be made (albeit less strongly): if there's some credence in... (read more)
Newcomb’s problem is a famous paradox in decision theory. The simple version is as follows:
... (read 2027 more words →)Two boxes are designated A and B. The player is given a choice between taking only box B or taking both boxes A and B. The player knows the following:
- Box A is transparent and always contains a visible $1,000.
- Box B is opaque and its content has already been set by the predictor:
- If the predictor has predicted that the player will take both boxes A and B, then box B contains nothing.
- If the predictor has predicted that the player will take only box B, then box B contains $1,000,000.
The player does not know what the predictor predicted or what
Whenever I read yet another paper or discussion of activation steering to modify model behavior, my instinctive reaction is to slightly cringe at the naiveté of the idea. Training a model to do some task only to then manually tweak some of the activations or weights using a heuristic-guided process seems quite un-bitter-lesson-pilled. Why not just directly train for the final behavior you want—find better data, tweak the reward function, etc.?
But actually there may be a good reason to continue working on model-internals control (i.e. ways of influencing model behavior outside of modifying the text input or training process, by directly changing internal state). For some applications, you may want to express... (read more)
Criticism quality-valence bias
Something I've noticed from posting more of my thoughts online:
People who disagree with your conclusion to begin with are more likely to carefully read and point out errors in your reasoning/argumentation, or instances where you've made incorrect factual claims. Whereas people who agree with your conclusion before reading are more likely to consciously or subconsciously gloss over any flaws in your writing because they are onboard with the "broad strokes".
So your best criticism ends up coming with a negative valence, i.e. from people who disagree with your conclusion to begin with.
(LessWrong has much less of this bias than other places, though I still see some of it.)
AI risk arguments often gesture at smarter AIs being "more rational"/"closer to a perfect utility maximizer" (and hence being more dangerous) but what does this mean, concretely? Almost anything can be modeled as a maximizer of some utility function.
The only way I can see to salvage this line of reasoning is to restrict the class of utility functions one can have such that the agent's best-fit utility function cannot be maximized until it gets very capable. The restriction may be justified on the basis of which kind of agents are unstable under real-world conditions/will get outcompeted by other agents.
Think clearly about the current AI training approach trajectory
If you start by discussing what you expect to be the outcome of pretraining + light RLHF then you're not talking about AGI or superintelligence or even the current frontier of how AI models are trained. Powerful, general AI requires serious RL on a diverse range of realistic environments, and the era of this has just begun. Many startups are working on building increasingly complex, diverse, and realistic training environments.
It's kind of funny that so much LessWrong arguing has been around why a base model might start trying to take over the world. When that's beyond the point. Of course we will eventually start RL'ing models on hard, real-world goals.
Example post / comment to illustrate what I mean.
A few days before “If Anyone Builds It, Everyone Dies” came out I wrote a review of Scott’s review of the book.
Now I’ve actually read the book and can review it for real. I won’t go into the authors’ stylistic choices like their decision to start every chapter with a parable or their specific choice of language. I am no prose stylist, and tastes vary. Instead I will focus on their actual claims.
The main flaw of the book is asserting that various things are possible in theory, and then implying that this means they will definitely happen. I share the authors’ general concern that building superintelligence carries a significant risk, but I... (read 3126 more words →)
Could HGH supplementation in children improve IQ?
I think there's some weak evidence that yes. In some studies where they give HGH for other reasons (a variety of developmental disorders, as well as cases when the child is unusually small or short), an IQ increase or other improved cognitive outcomes are observed. The fact that this occurs in a wide variety of situations indicates that it could be a general effect that could apply to healthy children.
Examples of studies (caveat: produced with the help of ChatGPT, I'm including null results also). Left column bolded when there's a clear cognitive outcome improvement.
| Treatment group | Observed cognitive / IQ effects of HGH | Study link |
|---|---|---|
| Children with isolated growth hormone |
What, concretely, is being analogized when we compare AI training to evolution?
People (myself included) often handwave what is being analogized when it comes to comparing evolution to modern ML. Here's my attempt to make it concrete:
One implication of this is that we should not talk about whether one or another species tries to survive and increase in number ("are humans aligned with evolution's goals?") but rather whether genetic material/individual genes are doing so.
Midjourney is best at producing a diverse and aesthetically pleasing range of styles and doesn’t refuse “in the style of…” requests. However, it is worst at text-in-images, avoiding uncanny AI artifacts (like extra fingers or unrealistic postures), and precise instruction-following (it messes up the specifics). Another major downside is that they don’t offer an API.
GPT-5 produces less artistic outputs but is better at following precise instructions on text and composition details.
Gemini “Nano Banana” is somewhere in the middle where it is ok-ish at everything—better at style than GPT-5 but worse than Midjourney, better at instruction-following than Midjourney but worse than GPT-5.
Socioeconomic status, parental education, and parental intelligence have strong effects on child IQ and are themselves correlated with breastfeeding practices. When studies ignore these confounders, they often find significant IQ increases in breastfed groups (5-8 points).
As more confounders are accounted for, the gap typically shrinks to 2-3 points. Some relevant studies:
However, the gold standard are sibling... (read more)
View this post on my blog for higher resolution images.
There’s a UK smoothie brand called “Innocent Drinks”. Back in 2012, when I was twelve years old, a friend from school invited me and another girl to her place. After hanging out in her house, we all decided to go on an adventure to the local Nando’s. On the way, we walked past Innocent’s London HQ (which they call “Fruit Towers”). I peeked inside through the glass and was delighted to see something I had harped on about often as a kid (and heard adults laugh at)—indoor fake grass flooring.
Finally, someone shared my idea of what a... (read 1078 more words →)
The motte and bailey of transhumanism
Most people on LW, and even most people in the US, are in favor of disease eradication, radical life extension, reduction of pain and suffering. A significant proportion (although likely a minority) are in favor of embryo selection or gene editing to increase intelligence and other desirable traits. I am also in favor of all these things. However, endorsing this form of generally popular transhumanism does not imply that one should endorse humanity’s succession by non-biological entities. Human “uploads” are much riskier than any of the aforementioned interventions—how do we know if we’ve gotten the upload right, how do we make the environment good enough without having... (read more)
On people's arguments against embryo selection
A recent NYT article about Orchid's embryo selection program triggered a surprising to me backlash on X where people expressed disgust and moral disapproval at the idea of embryo selection. The arguments generally fell into two categories:
(1) "The murder argument" Embryo selection is bad because it involves creating and then discarding embryos, which is like murdering whole humans. This argument also implies regular IVF, without selection, is also bad. Most proponents of this argument believe that the point of fertilization marks a key point when the entity starts to have moral value, i.e. they don't ascribe the same value to sperm and eggs.
(2) "The egalitarian argument" Embryo... (read 588 more words →)
On optimizing for intelligibility to humans (copied from substack)
... (read more)One risk of “vibe-coding” a piece of software with an LLM is that it gets you 90% of the way there, but then you’re stuck—the last 10% of bug fixes, performance improvements, or additional features is really hard to figure out because the AI has written messy, verbose code that both of you struggle to work with. Nevertheless, to delegate software engineering to AI tools is more tempting than ever. Frontier models can spit out almost-perfect complex React apps in just a minute, something that would have taken you hours in the past. And despite the risks, it’s often the right decision to prioritize
People talk about meditation/mindfulness practices making them more aware of physical sensations. In general, having "heightened awareness" is often associated with processing more raw sense data but in a simple way. I'd like to propose an alternative version of "heightened awareness" that results from consciously knowing more information. The idea is that the more you know, the more you notice. You spot more patterns, make more connections, see more detail and structure in the world.
Compare two guys walking through the forest: one is a classically "mindful" type, he is very aware of the smells and sounds and sensations, but the awareness is raw, it doesn't come with a great deal of conscious... (read more)
Great article, thanks for writing about this