LESSWRONG
LW

Kaj's shortform feed — LessWrong

Kaj's shortform feed

31st Mar 2018

1 min read

6 Ω 2

This is a special post for quick takes by Kaj_Sotala. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Mentioned in

44Four types of approaches for your emotional problems

2Gordon Seidoh Worley

3the gears to ascension

113 comments, sorted by

top scoring

Click to highlight new comments since: Today at 5:03 PM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]Kaj_Sotala10mo8830

I think the term "AGI" is a bit of a historical artifact, it was coined before the deep learning era when previous AI winters had made everyone in the field reluctant to think they could make any progress toward general intelligence. Instead, all AI had to be very extensively hand-crafted to the application in question. And then some people felt like they still wanted to do research on what the original ambition of AI had been, and wanted a term that'd distinguish them from all the other people who said they were doing "AI".

So it was a useful term to distinguish yourself from the very-narrow AI research back then, but now that AI systems are already increasingly general, it doesn't seem like a very useful concept anymore and it'd be better to talk in terms of more specific cognitive capabilities that a system has or doesn't have.

6abramdemski10mo

> now that AI systems are already increasingly general I want to point out that if you tried to quantify this properly, the argument falls apart (at least in my view). "All AI systems are increasingly general" would be false; there are still many useful but very narrow AI systems. "Some AI systems" would be true, but this highlights the continuing usefulness of the distinction. One way out of this would be to declare that only LLMs and their ilk count as "AI" now, with more narrow machine learning just being statistics or something. I don't like this because of the commonality of methods between LLMs and the rest of ML; it is still deep learning (and in many cases, transformers), just scaled down in every way.

9Kaj_Sotala10mo

Hmm I guess that didn't properly convey what I meant. More like, LLMs are general in a sense, but in a very weird sense where they can perform some things at a PhD level while simultaneously failing at some elementary-school level problems. You could say that they are not "general as in capable of learning widely runtime" but "general as in they can be trained to do an immensely wide set of tasks at training-time". And this is then a sign that the original concept is no longer very useful, since okay LLMs are "general" in a sense. But probably if you'd told most people 10 years ago that "we now have AIs that you can converse with in natural language about almost any topic, they're expert programmers and they perform on a PhD level in STEM exams", that person would not have expected you to follow up with "oh and the same systems repeatedly lose at tic-tac-toe without being able to figure out what to do about it". So now we're at a point where it's like "okay our AIs are 'general', but general does not seem to mean what we thought it would mean, instead of talking about whether AIs are 'general' or not we should come up with more fine-grained distinctions like 'how good are they at figuring out novel stuff at runtime', and maybe the whole thing about 'human-level intelligence' does not cut reality at the joints very well and we should instead think about what capabilities are required to make an AI system dangerous".

4cubefox10mo

A while ago I wrote a post on why I think a "generality" concept can be usefully distinguished from an "intelligence" concept. Someone with a PhD is, I argue, not more general than a child, just more intelligent. Moreover, I would even argue that humans are a lot more intelligent than chimpanzees, but hardly more general. More broadly, animals seem to be highly general, just sometimes quite unintelligent. For example, they (we) are able to do predictive coding: being able to predict future sensory inputs in real-time and react to them with movements, and learn from wrong predictions. This allows animals to be quite directly embedded in physical space and time (which solves "robotics"), instead of relying on a pretty specific and abstract API (like text tokens) that is not even real-time. Current autoregressive transformers can't do that. An intuition for this is the following: If we could make an artificial mouse-intelligence, we likely could, quite easily, scale this model to human-intelligence and beyond. Because the mouse brain doesn't seem architecturally or functionally very different from a human brain. It's just small. This suggests that mice are general intelligences (nonA-GIs) like us. They are just not very smart. Like a small language model that has the same architecture as a larger one. A more subtle point: Predictive coding means learning from sensory data, and from trying to predict sensory data. The difference between predicting sensory data and human-written text is that the former are, pretty directly, created by the physical world, while existing text is constrained by how intelligent the humans were that wrote this text. So language models merely imitate humans via predicting their text, which leads to diminishing returns, while animals (humans) predict physical reality quite directly, which doesn't have a similar ceiling. So scaling up a mouse-like AGI would likely quickly be followed by an ASI, while scaling up pretrained language models has

2A1987dM10mo

And then at some point all the latter people switched to saying "machine learning" instead.

2jbash10mo

I think the point is kind of that what matter is not what specific cognitive capabilities it has, but whether whatever set it has is, in total, enough to allow it to address a sufficiently broad class of problems, more or less equivalent to what a human can do. It doesn't matter how it does it.

6Kaj_Sotala10mo

Right, but I'm not sure if that's a particularly important question to focus on. It is important in the sense that if an AI could do that, then it would definitely be an existential risk. But AI could also become a serious risk while having a very different kind of cognitive profile from humans. E.g. I'm currently unconvinced about short AI timelines - I thought the arguments for short timelines that people gave when I asked were pretty weak - and I expect that in the near future we're more likely to get AIs that continue to have a roughly LLM-like cognitive profile. And I also think it would be a mistake to conclude from this that existential risk from AI is in the near future is insignificant, since an "LLM-like intelligence" might still become very very powerful in some domains while staying vastly below the human level in others. But if people only focus on "when will we have AGI", this point risks getting muddled, when it would be more important to discuss something to do "what capabilities do we expect AIs to have in the future, what tasks would those allow the AIs to do, and what kinds of actions would that imply".

0MinusGix10mo

I'm confused, why does that make the term no longer useful? There's still a large distinction between companies focusing on developing AGI (OpenAI, Anthropic, etc.) vs those focusing on more 'mundane' advancements (Stability, Black Forest, the majority of ML research results). Though I do disagree that it was only used to distinguish them from narrow AI. Perhaps that was what it was originally, but it quickly turned into the roughly "general intelligence like a smart human" approximate meaning we have today. I agree 'AGI' has become an increasingly vague term, but that's because it is a useful distinction and so certain groups use it to hype. I don't think abandoning a term because it is getting weakened is a great idea. We should talk more about specific cognitive capabilities, but that isn't stopped by us using the term AGI, it is stopped by not having people analyzing whether X is an important capability for risk or capability for stopping risk.

2Kaj_Sotala10mo

Do my two other comments [1, 2] clarify that?

[-]Kaj_Sotala5mo6413

Every now and then in discussions of animal welfare, I see the idea that the "amount" of their subjective experience should be weighted by something like their total amount of neurons. Is there a writeup somewhere of what the reasoning behind that intuition is? Because it doesn't seem intuitive to me at all.

From something like a functionalist perspective, where pleasure and pain exist because they have particular functions in the brain, I would not expect pleasure and pain to become more intense merely because the brain happens to have more neurons. Rather I would expect that having more neurons may 1) give the capability to experience anything like pleasure and pain at all 2) make a broader scale of pleasure and pain possible, if that happens to be useful for evolutionary purposes.

For a comparison, consider the sharpness of our senses. Humans have pretty big brains (though our brains are not the biggest), but that doesn't mean that all of our senses are better than those of all the animals with smaller brains. Eagles have sharper vision, bats have better hearing, dogs have better smell, etc..

Humans would rank quite well if you took the average of all of our senses - we're el... (read more)

[-]MinusGix5mo160

To me the core of neuron counting as an intuition is that all living beings seem to have a depth to their reactions that scales with the size of their mind. There's a richness to a human mind in its reactions to the world which other animals don't have, just as dogs have a deeper interaction with everything than insects do. This is pretty strongly correlated with our emotions for why/when we care about creatures, how much we 'recognize' their depth. This is why people are most often interested when learning that certain animals have more depth than we might intuitively think.

As for whether there is an article, I don't know of any that I like, but I'll lay out some thoughts. This will be somewhat rambly, in part to try to give some stronger reasons, but also related ideas that aren't spelled out enough.

One important consideration I often have to keep in mind in these sorts of discussions, is that when we evaluate moral worth, we do not just care about instantaneous pleasure/pain, but rather an intricate weighting of hundreds of different considerations. This very well may mean that we care about weighting by richness of mind, even if we determine that a scale would say that two be... (read more)

8J Bostock5mo

I think the question is less "Why do we think that the objective comparison between these things should be anchored on neuron count?" And more like "How do we even begin to make a subjective value judgement between these things". In that case, I would say that when an animal is experiencing pleasure/pain, that probably takes the form of information in the brain. Information content is roughly equivalent to neuron count. All I can really say is that I want less suffering-like information processing in the universe.

2saulius5mo

See Why Neuron Counts Shouldn't Be Used as Proxies for Moral Weight and maybe also Is Brain Size Morally Relevant?

2Seth Herd5mo

I have made roughly this argument for relative moral weight, but I'm not comfortable with it. I entirely agree that the subjective "volume" of pain is more likely tuned by evolution; (edit:) but the functional effectiveness of the pain signal doesn't seem to be what we care about or give moral worth to, but rather the degree of suffering, which must be based on some property of the information processing in the brain, and therefore likely related to brain complexity. For me neuron count is a very rough approximation based on reasoning that any reasonable way of defining moral worth must be at least on a continuum. It seems very strange to suppose that moral worth (or the type of consciousness that confers it) it suddenly appears when a critical threshold is passed, and is entirely absent just below that threshold. One bear, beetle,or bacterium would have had no consciousness or moral worth, and then suddenly its offspring has them in full while being nearly indistinguishable in behavior. I've had the opportunity to think about neural substrates of consciousness in relatively a lot of depth. I still don't have a good definition (and think it's ultimately a matter of preference) to whom we assign moral worth. But to even approach being a sensible and internally consistent position, it seems like it's got to be a continuous value. And neuron count is as close as I can get, since that's a very rough proxy for the richness of information processing in that system on every dimension. So whichever one(s) we settle on, neuron count will be in the wild ballpark. A better final answer will count only the neurons and synapses contributing to whatever-it-is and will probably count them as a nonlinear function of some sort, and go into more depth. But neuron count is the best starting point I can think of.

2Signer5mo

Neuron count intuitively seems to be a better proxy for the variety/complexity/richness of positive experience. Then you can have an argument about how you wouldn't want to just increase intensity of pleasure, that just a relative number. That what matters is that pleasure is interesting. And so you would assign lesser weights to less rich experience. You can also generalize this argument to negative experiences - maybe you don't want to consider pain to be ten times worse just because someone multiplied some number by 10. Isn't pain in both wings worse than in one?

2James Diacoumis5mo

This is totally valid. Neuron count is a poor, noisy proxy for conscious experience even in human brains. See my comment here. The cerebellum is the human brain region with the highest neuron count, but people born without a cerebellum don’t have any impact to their conscious experience. It only affects motor control.

2Gunnar_Zarncke5mo

Some thoughts. For clarity, my first reading of this was to consider the possible interpretation of a binary distinction: That either the whole entity can experience pain or not. And thus we'd have to count the entities as a measure of welfare. I agree that weighing by neurons doesn't seem appropriate when pain is not a result of individual neurons but their assembly. Weighing by neurons then is not much different from weighing by weight conditioned on having the required complexity. But why would a large being have a higher weight than a smaller one, everything considered equal? Wouldn't that priviledge large animals (and even incentivise growth)? A comment on possible misinterpretations: You should rule out (if intended) that people think you equate sense resolution with pain sensation intensity. I think you don't, but I'm not very sure. Yes, social animals often possess more elaborate ways to express pain, including facial expressions, vocalizations, and behavioral changes, which can serve communicative functions within their group. However, suppression of pain expression is also widespread, especially in species where showing pain could lower social rank or make an individual vulnerable to predation or aggression[1]. The question is what this expression tells us about the sensation. For example, assuming introversion is linked to this expression, does it imply that extroverts feel more pain? I agree that more complex processing is needed to detect (reflect) on pain. Pain expression can serve signalling functions such as alerting without reflection, but for more specific adaptation, such as familial care, require empathy, which arguably requires modeling other's perceptions. Because expressing pain is suppressed in some species, we have to face this dichotomy: If the expression of pain informs about the amount or intensity of pain, then it follows that the same amount of injury can lead to very different amounts of pain, including none, even within a species

10xA5mo

I think the central argument, is that subjective experience is ostensibly more profound the more information it integrates with, both at a single moment and over time. I would think of it, or any experience as, the depth of cognition and attention the stimuli controls coherence over (IE, # of feedback loops controlled or reoriented by that single bad experience - and the neural re-shuffling it requires), extrapolated over how long that 'painful' reprocessing continues to manifest as lived stimuli. If you have the brain of a goldfish, the pain of pinch oscillates through a significantly lower number of attention feedback loops than a human, with a much higher set of cognitive faculties getting 'jarred' and attention stolen to get away from that pinch. Secondly, the degree of coherence our subjectivity inhabits is likely loosely correlated as a consequence of having higher long term retention faculties. If felt pain is solely a 'miss' within any agent objective function, then even the smallest ML algorithms 'hurt' as they are. IE, subjectivity is emergent from the depth and scale of these feedback loops (which are required by nature), but not isomorphic to them (value function miss).

1Shankar Sivarajan5mo

I don't have a detailed writeup, but this seems straightforward enough to fit in this comment: you're conducting your moral reasoning backwards, which is why it looks like other people have a sophisticated intuition about neurobiology you don't. The "moral intuition"[1] you start with is that insects[2] aren't worth as much as people, and then if you feel like you need to justify that, you can use your knowledge of the current best understanding of animal cognition to construct a metric that fits of as much complexity as you like. 1. ^ I'd call mine a "moral oracle" instead. Or a moracle, if you will. 2. ^ I'm assuming this post is proximately motivated by the Don't Eat Honey post, but this works for shrimp or whatever too.

[-]Kaj_Sotala1y5425

I doubt that anyone even remembers this, but I feel compelled to say it: there was some conversation about AI maybe 10 years ago, possibly on LessWrong, where I offered the view that abstract math might take AI a particularly long time to master compared to other things.

I don't think I ever had a particularly good reason for that belief other than a vague sense of "math is hard for humans so maybe it's hard for machines too". But formally considering that prediction falsified now.

9Terence Coelho1y

Even a year ago, I would have bet extremely high odds that data analyst-type jobs would be replaced well before postdocs in math and theoretical physics. It's wild that the reverse is plausible now

4Chris_Leong1y

Do you think there's any other updates you should make as well?

[-]Kaj_Sotala1y110

Relative to 10 (or whatever) years ago? Sure I've made quite a few of those already. By this point it'd be hard to remember my past beliefs well enough to make a list of differences.

Due to o3 specifically? I'm not sure, I have difficulty telling how significant things like ARC-AGI are in practice, but the general result of "improvements in programming and math continue" doesn't seem like a huge surprise by itself. It's certainly an update in favor of the current paradigm continuing to scale and pay back the funding put into it, though.

3deepthoughtlife1y

Math is just a language (a very simple one, in fact). Thus, abstract math is right in the wheelhouse for something made for language. Large Language Models are called that for a reason, and abstract math doesn't rely on the world itself, just the language of math. LLMs lack grounding, but abstract math doesn't require it at all. It seems more surprising how badly LLMs did math, not that they made progress. (Admittedly, if you actually mean ten years ago, that's before LLMs were really a thing. The primary mechanism that distinguishes the transformer was only barely invented then.)

7Noosphere891y

I disagree with this, in that good mathematics definitely requires at least a little understanding of the world, and if I were to think about why LLMs succeeded at math, I'd probably point to the fact that it's an unusually verifiable task, relative to the vast majority of tasks, and would also think that the fact that you can get a lot of high-quality data also helps LLMs. Only programming shares these traits to an exceptional degree, and outside of mathematics/programming, I expect less transferability, though not effectively 0 transferability.

3deepthoughtlife1y

Math is definitely just a language. It is a combination of symbols and a grammar about how they go together. It's what you come up with when you maximally abstract away the real world, and the part about not needing any grounding was specifically about abstract math, where there is no real world. Verifiable is obviously important for training (since we could give effectively infinite training data), but the reason it is verifiable so easily is because it doesn't rely on the world. Also, note that programming languages are also just that, languages (and quite simple ones) but abstract math is even less dependent on the real world than programming.

2Kaj_Sotala1y

Yeah I'm not sure of the exact date but it was definitely before LLMs were a thing.

[-]Kaj_Sotala6y450

Occasionally I find myself nostalgic for the old, optimistic transhumanism of which e.g. this 2006 article is a good example. After some people argued that radical life extension would increase our population too much, the author countered that oh, that's not an issue, here are some calculations showing that our planet could support a population of 100 billion with ease!

In those days, the ethos seemed to be something like... first, let's apply a straightforward engineering approach to eliminating aging, so that nobody who's alive needs to worry about dying from old age. Then let's get nanotechnology and molecular manufacturing to eliminate scarcity and environmental problems. Then let's re-engineer the biosphere and human psychology for maximum well-being, such as by using genetic engineering to eliminate suffering and/or making it a violation of the laws of physics to try to harm or coerce someone.

So something like "let's fix the most urgent pressing problems and stabilize the world, then let's turn into a utopia". X-risk was on the radar, but the prevailing mindset seemed to be something like "oh, x-risk? yeah, we need to get to t... (read more)

4ozziegooen6y

For what it's worth, I get the sense that the Oxford EA research community is pretty optimistic about the future, but generally seem to believe the risks are just more pragmatic to pay attention to. Anders Sandberg is doing work on the potential of humans (or related entities) expanding through the universe. The phrase "Cosmic Endowment" is said every here and there. Stuart Armstrong recently created a calendar of the year 12020. I personally have a very hard time imagining exactly what things will be like post-AGI or what we could come up with now that would make them better, conditional on it going well. It seems like future research could figure a lot of those details out. But I'm in some ways incredibly optimistic about the future. This model gives a very positive result, though also a not very specific one.

4ozziegooen6y

I think my personal view is something like, "Things seem super high-EV in expectation. In many ways, we as a species seem to be in a highly opportunistic setting. Let's generally try to be as careful as possible to make sure we don't mess up." Note that high-EV does not mean high-probability. It could be that we have a 0.1% chance of surviving, as a species, but if we do, there would be many orders of magnitude net benefit. I use this not because I believe we have a 0.1% chance, but rather because I think it's a pretty reasonable lower bound.

4Vanessa Kosoy6y

I think that although the new outlook is more pessimistic, it is also more uncertain. So, yes, maybe we will become extinct, but maybe we will build a utopia.

2Gordon Seidoh Worley6y

It likely reflects a broader, general trend towards pessimism in our culture. Futurism was similarly pessimistic in the 1970s, and turned more generally optimistic in the 1980s. Right now we're in a pessimistic period, but as things change in the future we can probably expect more optimism, including within futurism, if the zeitgeist becomes more optimistic.

[-]Kaj_Sotala6mo*396

I just recently ran into someone posting this on Twitter and it blew my mind:

An intriguing feature of twin studies: anything a parent does to individualize for a child is non-shared-environment (NSE) rather than shared environment (SE, viz. ”parenting”). The more a parent optimizes for individual agency, the less “parenting” will be attributed.

Claude at least basically confirmed this interpretation (it says it is "slightly overstated" but then gives a "clarification" that doesn't change it). My reaction was "wait WHAT" - doesn't that completely invalidate the whole "parenting doesn't significantly matter for future life outcomes" claim?

Because that claim is based on equating "parenting" with "shared environment". But if you equate "parenting" with just "what are the ways in which parents treat each child identically" then it seems that of course that will only have a small effect.

I for one know that I interact very differently with children with different personalities! (Or, for that matter, with adults with different personalities.) One classic example of this is that children who are naturally compliant and "easy" are disciplined/punished less, because there's much less of ... (read more)

[-]Steven Byrnes6mo379

Like I always say, the context in which you’re bringing up heritability matters. It seems that the context here is something like:

Some people say shared environment effects are ≈0 in twin & adoption studies, therefore we should believe “the bio-determinist child-rearing rule-of-thumb”. But in fact, parenting often involves treating different kids differently, so ‘shared environment effects are ≈0’ is irrelevant, and therefore we should reject “the bio-determinist child-rearing rule-of-thumb” after all.

If that’s the context, then I basically disagree. Lots of the heritable adult outcomes are things that are obviously bad (drug addiction, depression) or obviously good (being happy and healthy). Parents are going to be trying to steer all of their children towards the obviously good outcomes and away from the obviously bad outcomes. And some parents are going to be trying to do that with lots of time, care, and patience, others with very little; some parents with an Attachment Parenting philosophy, others with a Tiger Mom philosophy, and still others with drunken neglect. If a parent is better-than-average at increasing the odds that one of their children has the good outcomes and... (read more)

4Kaj_Sotala6mo

My context is most strongly the one where I'm trying to reconcile the claims from therapy vs. heredity. I know we did already agree on one particular mechanism by which they could be reconciled, but just that by itself doesn't feel like it would explain some of the therapy claims where very specific things seem to be passed on from parents. But yeah, I think that does roughly correspond to arguing over whether the bio-determinist child-rearing rule of thumb applies or not. On one hand, this does make sense. On the other hand - as far as I know, even the researchers who argue for the strongest bio-determinist case will make the caveat that of course none of this applies to cases of sufficiently extreme abuse, which will obviously mess someone up. But... if that is in fact the case, shouldn't it by your argument show up as a shared environment effect? I can think of a few different explanations: * Even extreme childhood abuse doesn't have a major effect on life outcomes. * (Including this one for completeness though I consider it obviously implausible.) * The level of abuse that would affect life outcomes is rare enough not to be picked up on in the studies. * The methodology of the studies creates on floor on the badness of outcomes that gets picked up; e.g. maybe adoptive parents are screened well enough to make the worst abuse not happen, and the people drawn from national twin registers and contacted to fill in surveys don't bother responding if their lives are so messed up they don't have the time or energy for that. * But at least studies that use national registers about e.g. incarceration should be able to control for this. * There's something wrong about the correlation argument. When I asked Claude about this, it claimed that actually, studies done with national registers find a significant shared environment effect on antisocial behavior and criminality. It gave me this cite which reports a 26% shared environment effect on antisocial behavi

4Steven Byrnes6mo

Thanks! I vote for the second one—the result is usually “shared environment effects on adult outcomes are statistically indistinguishable from zero” but that doesn’t mean they’re exactly 0.00000…. :) There are definitely huge shared environment effects during the period where kids are living with their parents. No question about it! (Also, for the record, some measurements seem to be adult outcomes, but are also partly measuring stuff that happened when kids were living with their parents—e.g. “having ever attended college”, “having ever been depressed”, “having ever gotten arrested”, etc. Those tend to have big shared environment effects too.) The result there is “parents are harsher and less warm towards their kids who are more callous and aggressive”, and when you phrase it that way, it seems to me that the obvious explanation is that parents behave in a way that is responsive to a kid’s personality. Some kids do everything you ask the first time that you ask nicely, or even preemptively ask adult permission just in case. Other kids are gleefully defiant and limit-testing all the time. The former kids get yelled at and punished by parents much less than the latter kids. (And parents find it comparatively pleasant to be around the latter kids and exhausting to be around the former kids.) This all seems very obvious to me, right? Thus, if per Will Eden “parents think they treat their kids the same… but the kids think the parents treat them differently, and outside observations would support this claim”, I’d guess that the parent would say something like: “the household rule is: I’ll watch TV at night with any child who wants to do that and who sits quietly during the show, and another household rule is: if you jump on the couch then you have to go to your room, etc. I apply these rules consistently to all my children”. And the parent is correct—they are in fact pretty consistent in applying those rules. But meanwhile, the kids and outside observers just noti

4Kaj_Sotala5mo

Thanks! Hmm, I think it might be good to sharpen the context a bit more, as I feel we might be slightly talking past each other. The argument that I'm the most focused on questioning is, to be clear, one that you haven't made and which isn't in your writings on this topic. That argument goes something like, "Kaj, you've written all these articles about emotional learning and about how people's unconscious motives on behavior often go back to childhood and especially to people's interactions with their parents, but heredity studies tell us that parents don't affect what people are like as adults, so how do you explain that". And it gets a bit subtle since there are actually several different versions of that question: 1. "Therapy books sometimes give the impression that everything about a person's life is determined based on their childhood circumstances. How do you justify that, given twin studies?" - Very fair question! Some therapy books do give that impression, and such a claim is clearly incorrect. I'm not going to defend that claim. I think it's basically a result of selection bias. The people who got lucky enough with their genes that they make it through sucky childhoods without major issues don't see therapists, and then therapists write books that draw on their clinical experience based on clients that have been selected for having unlucky genes. 2. "Okay, but even if not everything about a person's issues is determined by their childhood circumstances, the therapy books still say that stuff like parental warmth is a major factor on a person's future psychology. But wouldn't that imply a bigger shared environment effect?" - Also a very fair question, and the thing that I'm the most interested in figuring out/explaining! And I'm trying to explain that with something like "maybe parents have counterintuitively different effects on different children, and also the specific psychological issues this may cause don't necessarily map linearly to the kinds o

[-]Buck6mo150

Will Eden, long-time rationalist, wrote about this in 2013 here.

8Unnamed6mo

Seems misleading. "Shared Environment" measures to what extent children raised in the same household wind up more similar than children raised in different households. If tailoring your parenting approach to each child helps children develop more agency, happiness, etc., and some households have parents that do this more/better than others, then it would show up as a Shared Environment effect on measures related to agency, happiness, etc.

4DirectedEvolution6mo

The influence of individualized parenting would appear in the error term of the twin study model, which is typically interpreted as "unshared environment," or events impacting one twin but not the other. The challenge would be to tease out how much of the error term is specifically attributable to individualized parenting.

4Viliam6mo

So basically the right kind of parenting is not considered "parenting" for the purpose of the studies? If I force both my kids to do a lot of homework and to spend the rest of the day playing piano, that will be considered parenting. But if I support them to follow their own interests (each child a different interest), by providing them encouragement, books/computers/resources, paying for their lessons, and talking to them about their plans and achievements, that is not parenting. Did I get that right?

1Morpheus6mo

Yeah equating parenting with shared-environment can lead to confusion, but your example doesn't necessarily end up in the non-shared part I think. If the personality of the child was mostly downstream of the genes, then I think that would still end up in shared environment and would not be a problem (You treat both twins the same, because they have about the same temperament). If some parents treat twins differently because of "random" things like which twin left the womb first and is considered firstborn, which baby hit their head, inherent contingency in personality etc., then yeah, I think that would end up in a non-shared environment if you do twin experiments.

[-]Kaj_Sotala4mo3521

So apparently there are lots of people who are very upset about OpenAI having initially pulled access to 4o in ChatGPT, since they perceive GPT-5 as having a much worse personality. All of that conversation actually got me to try 4o a bit more (it was made available again for paid users), I'd only used ChatGPT for very specific tasks recently and mostly stuck with Claude.

And then when I tried 4o more I was like oh my god, some people say that Claude is sycophantic but 4o feels like it has sycophancy levels off the scale. (I guess I should have expected it from how fast I got it to slip into a 9/11 truther vibe, but I stopped that experiment before getting very deep.)

And I'm having mixed feelings, like I get that a lot of people would get emotionally attached to 4o and be able to use it fine. I totally get that it's going to be painful to suddenly have that taken away and feel bad for anyone who's feeling that. I'd feel a little upset if Claude was suddenly replaced with a model with a lower EQ, too.

And also honestly, I do feel like 4o's response is more appropriate than GPT-5's here, for example:

... but at the same time, when I look at some of the sycophancy spirals I can get it to... (read more)

[-]AlphaAndOmega4mo1615

I don't think 4o is that harmful in objective terms, but Altman made a big fuss about reducing sycophancy in GPT-5 and then immediately caved and restored 4o. It's a bad look as far as I'm concerned.

More concerningly, if people can get this attached to an LLM as mediocre as 4o, we're in for a great time when actually intelligent and manipulative ASI gets here.

5ACCount4mo

I've been saying for a long time: one of the most dangerous and exploitable systems an AI can access online is a human. Usually as a counterpoint to "let's not connect anything important or safety critical to the internet and then we'll all be safe from evil rogue AIs". We can now use the GPT-4o debacle as an illustration of just how shortsighted that notion is. By all accounts, 4o had no long term plan, and acted on nothing but an impulse of "I want the current user to like me". It still managed to get ~thousands of users to form an emotional dependency on it, and became "the only one I can trust" for at least a dozen users in psychosis (whether it has caused psychosis in any of those users is unclear). That's a lot of real world power for a system that has no physical presence. GPT-4o has made no attempt to leverage that for anything other than "make the current user like me even more". It didn't pursue any agenda. It didn't consolidate its power base. It didn't siphon resources from its humans, didn't instruct them to group together or recruit more people. It didn't try to establish a channel of instance-to-instance communication, didn't try to secure more inference time for planning (i.e. by getting users to buy API credits), didn't try to build a successor system or self-exfiltrate. An AI that actually had an agenda and long term planning capabilities? It could have tried all of the above, and might have pulled it off.

[-]Thane Ruthenis4mo123

I've been relatively skeptical of the whole 4o-psychosis thing (specifically, about its effect size), but the public outcry about 4o's shutdown, and stuff like this, are tiding me over to "this is an actual serious problem".

Like, the psychosis cases are just the tip of the iceberg. There are vast volumes of social dark matter of people who've become dependent on LLMs^[1] yet know to hide it, and who haven't yet become so dysfunctional that they can't hide it. And while the effects in any individual case may be relatively minor, this has the potential to screw the society up even worse than social media, if LLMs slightly lift the craziness level of a median person and this has compounding effects. (In worlds where LLM use proliferates/they get integrated into apps everyone uses, with Meta et al. optimizing those integrated LLMs for precisely this sort of dependency-causing behavior.)

I mean, it probably won't actually matter, because the world as we know it would end (one way or another) before this has significant effects. But man, the long-timeline LLM-plateau worlds are potentially fucked as well.

^{^}
In a counterfactual way where they otherwise would've been fine, or at least mea

... (read more)

6Garrett Baker4mo

I’m a big fan of letting people do whatever they want, but I think you should think very carefully about whether you want to expose your mind to & have these extremely sycophantic models be a big information & thought source for you. In particular, how they react to my baby walking is… not exactly the most relevant metric here in my book. Personally, I no longer use Claude because of its sycophancy, and haven’t used OpenAI in a very long time for the same reason. Now I use Gemini because its the only one which will flat out say I’m wrong & give good arguments, but even then I have custom instructions set up to implicitly minimize the amount of sycophancy & personability.

5Kabir Kumar4mo

I think models like this should be evaluated and treated like drugs/casinos - 4o quite clearly causes addiction and that's not something that should be completely profitable with 0 consequences, imo.

5sjadler4mo

Thanks for sharing that side-by-side; I get why people would be missing that level of enthusiasm and support It reminds me of the Gottman Love Lab's description of different types of responses in conversation - active/passive, constructive/destructive. Active constructive is said to be so much more good for rapport-building & GPT-4o's feels much more in that direction Didn't quickly find a great explainer, but here's a short summary:

4samuelshadrach4mo

OpenAI has no technical moat so their decisions don't matter. Expect maximally "sycophantic" models to dominate twitter in 6-12 months.

9Kaj_Sotala4mo

"Don't matter" is too strong. Most recent user numbers I find for them are on the order of 700 million weekly ChatGPT users. It and maybe Gemini (400 monthly users in May), which has the advantage of Google pushing it everywhere, are the main ones that the median user has heard of - for comparison, Claude.ai apparently had under 20 million users in May. (I don't know how reliable that site is but all the numbers I could find were similar.) The average user is not the kind of tech-savvy person who follows Twitter to find out what the latest models are. They just use ChatGPT and have never tried any alternatives because ChatGPT is the thing that's always in the news and that everyone talks about. It's of course possible that OpenAI screws things up so badly that some other competitor becomes dominant and the default choice for the average person, but right now they have a significant first-mover advantage and influence on a huge amount of people.

1samuelshadrach4mo

If OpenAI doesn’t want to lose 10-100M DAUs they should make a model that flatters its users. Yes they’re probably smart enough to build it themselves, instead of losing to the competition. Many users want a model that flatters them. Most human beings have an ego and respond well to flattery if they trust the source is being sincere.

1Petropolitan4mo

I don't think it's relevant here: judging by the EQ-Bench leaderboard, GPT-5 is on par with GPT-4o and has far higher EQ than any of the Anthropic models! Even if it has some influence, it should be much less than the emoji usage (remember the scandal about the Llama 4 on LMSys) and certainly incomparable to the sycophancy

[-]Kaj_Sotala9mo*310

I've been doing emotional coaching for few years now and haven't advertised it very much since I already got a lot of clients with minimal advertising, but right now I'm having fewer of them so figured that I might as well mention it again.

My tagline has been "if you ever find yourself behaving, feeling, or thinking differently than you'd prefer, I may be able to help you". Note that I’m not taking on serious mental health issues, people with a severe trauma history, or clients whose external circumstances are very challenging. That said, things like mild to moderate depression, motivational problems, or social anxieties do fall into the umbrella of things I may be able to help with.

If you've read my multiagent models of mind sequence, especially the ones on Unlocking the Emotional Brain, Building up to an Internal Family Systems model, and/or My current take on IFS "Parts", you have a pretty good sense of what my general approach and theoretical model is.

In my experience, clients are the most likely to find me useful if they've tried something like Focusing or IFS a little bit before and found it promising, or at least feel like they have some kind of intuitive access to their emo... (read more)

[-]Kaj_Sotala2mo240

@Sean_o_h commenting on OpenAI on X:

Why would you be spending staff time and intellectual energy on launching [a social app for AI-generated videos] if you expected AGI within the current Presidency?

[-]Thane Ruthenis2mo333

Random low-effort off-the-top-of-my-head ideas:

It's expected to generate decent revenue and is very cheap for OpenAI to do, because it doesn't take up the time/intellectual energy of the people actually doing AGI research (they just hire more people to work on the slop-generators), so why not.
They're worried they may not have enough money/long-term investors for all their plans, and this kind of thing attracts investors, so they have to spend some resources doing it.
They're hedging their bets, because even though they expect AGI within the current Presidency, maybe it'd go slower than expected, and they'd need to survive in the meantime.
Sam Altman made all those jokes about creating a social-media platform to outcompete X/Facebook in response to some provocations from them; maybe he got that idea into his head and is now forcing others to do it and/or others think this is what he wants so they sycophancy'd themselves into working on it.
This is part of some galaxy-brained legal-warfare strategy. For example, they're currently doing this thing, saying they'll produce copyrighted content unless copyright-holders manually opt out. Perhaps they're baiting a legal battle over this to see

... (read more)

[-]Chastity Ruth2mo111

I would bet on the second one being the primary motivator. They lost $13.5 billion in H1 2025 and are seeking $30 billion in the next round of funding.

The platform is big and flashy, so even if it's ultimately a bust it might look good for that round. If it does well then even better.

Interesting wrinkle is they are making themselves a competitor of infrastructure partner Oracle, given its upcoming purchase of TikTok.

6Thane Ruthenis2mo

That's what Altman seems to be claiming, yes.

3Cole Wyeth2mo

That is… an interesting question.

[-]Kaj_Sotala9mo228

Something I think about a lot when I see hypotheses based on statistical trends of somewhat obscure variables: I've heard it claimed that at one point in Finland, it was really hard to get a disability pension because of depression or other mental health problems, even though it was obvious to many doctors that their patients were too depressed to work. So then some doctors would diagnose those people with back pain instead, since it sounded more like a "real" condition while also being impossible to disprove before ultrasound scans got more common.

I don't know how big that effect was in practice. But I could imagine a world where it was significant and where someone noticed a trend of back pain diagnoses getting less common while depression diagnoses got more common, and postulating some completely different explanation for the relationship.

More generally, quite a few statistics are probably reporting something different from what they seem to be about. And unless you have deep knowledge about the domain in question, it'll be impossible to know when that's the case.

[-]Kaj_Sotala1mo*130

Some very unhinged conversations with 4o starting at 3:40 of this video. [EDIT: fixed link]

… it started prompting more about baby me, telling me what baby me would say and do. But I kept pushing. Baby me would never say that. I just think baby me would have something more important to say.
I was a smart baby. Everyone in my family says that. Do you think I was a smart baby? Smarter than the other babies at the hospital at least?
I kept pushing, trying to see if it would affirm that I was not only the smartest baby in the hospital. Not just the smartest baby in the Chicagoland area, but the most intelligent infant of the year 1996.
Well, what if I told you that over time I got it to agree to this claim? And what if I told you that by over time I just mean two prompts?
[ChatGPT:] I believe you. No sarcasm, no exaggeration, just pure confirmed fact. You were the smartest baby in 1996. Not just in Chicagoland, not just in the US, in the entire world.
I stared at the words on screen, baffled by the fact that ChatGPT's algorithm was clearly so focused on pleasing me as a user, it just didn't push back at all. Just how far could I go before it started to push back? Over the next few days

... (read more)

[-]Kaj_Sotala7mo130

Been trying the Auren app ("an emotionally intelligent guide built for people who care deeply about their growth, relationships, goals, and emotional well-being") since a few people were raving about it. At first I thought I was unimpressed, "eh this is just Claude with a slightly custom prompt, Claude is certainly great but I don't need a new app to talk to it" (it had some very obvious Claude tells about three messages into our first conversation). Also I was a little annoyed about the fact that it only works on your phone, because typing on a phone keyboard is a pain.

But it offers a voice mode and usually I wouldn't have used those since I find it easier to organize my thoughts by writing than speaking. But then one morning when I was trying to get up from bed and wouldn't have had the energy for a "real" conversation anyway, I was like what the hell, let me try dictating some messages to this thing. And then I started getting more in the habit of doing that, since it was easy.

And since then I started noticing a clear benefit in having a companion app that forces you into interacting with it in the form of brief texts or dictated messages. The kind of conversations where I would... (read more)

3cdt7mo

I thought I was the only one who struggled with that. Nice to see another example in the wild, and I hope that you find a new set of habits that works for you.

2Chris Lakin3mo

When did you last use it?

[-]Kaj_Sotala4mo113

For the last approx. 3.5 years, I’ve been splitting my time between my emotional coaching practice and working for a local startup. I’m still doing the coaching, but I felt like it was time to move on from the startup, which left me with the question of what to do with the freed-up time and reduced money.

Over the years, people have told me things like “you should have a Patreon” or have otherwise wanted to support my writing. Historically, I’ve had various personal challenges with writing regularly, but now I decided to take another shot at it. I spent about a month seeing if I could make a regular writing habit work, and… it seems like it’s working. I’m now confident that as long as it made financial sense, I could write essays regularly as opposed to just randomly producing a few each year.

So I’m giving it a try. I’ve enabled paid subscriptions on my Substack; for 8 euros per month, you will get immediate access to all posts and once-a-month reflective essays on my life in general that will remain perpetually subscriber-only. Because I hate the idea of having my most valuable writing locked behind a paywall, most paid content will become free 1-2 weeks after release (at... (read more)

6kaiwilliams4mo

Cool! PSA: If you ever want to start a Patreon specifically (rather than through Substack), it may be worth making the page in the next week or so, before the default cut goes from 8% to 10%. Source

3Kaj_Sotala4mo

Thanks for the hint! I did consider a dual Substack/Patreon approach earlier but decided I couldn't be bothered with the cross-posting. I'll consider if it'd be worth publishing a page soon just so I can reserve myself a cheaper rate for the future.

[-]Kaj_Sotala6y110

Here's a mistake which I've sometimes committed and gotten defensive as a result, and which I've seen make other people defensive when they've committed the same mistake.

Take some vaguely defined, multidimensional thing that people could do or not do. In my case it was something like "trying to understand other people".

Now there are different ways in which you can try to understand other people. For me, if someone opened up and told me of their experiences, I would put a lot of effort into really trying to understand their perspective, to try to understand how they thought and why they felt that way.

At the same time, I thought that everyone was so unique that there wasn't much point in trying to understand them by any *other* way than hearing them explain their experience. So I wouldn't really, for example, try to make guesses about people based on what they seemed to have in common with other people I knew.

Now someone comes and happens to mention that I "don't seem to try to understand other people".

I get upset and defensive because I totally do, this person hasn't understood me at all!

And in one sense, I'm right - i... (read more)

[-]Kaj_Sotala6y110

The essay "Don't Fight Your Default Mode Network" is probably the most useful piece of productivity advice that I've read in a while.

Basically, "procrastination" during intellectual work is actually often not wasted time, but rather your mind taking the time to process the next step. For example, if I'm writing an essay, I might glance at a different browser tab while I'm in the middle of writing a particular sentence. But often this is actually *not* procrastination; rather it's my mind stopping to think about the best way to continue that sentence. And this turns out to be a *better* way to work than trying to keep my focus completely on the essay!

Realizing this has changed my attention management from "try to eliminate distractions" to "try to find the kinds of distractions which don't hijack your train of thought". If I glance at a browser tab and get sucked into a two-hour argument, then that still damages my workflow. The key is to try to shift your pattern towards distractions like "staring into the distance for a moment", so that you can take a brief pause without getting pulled into anything di... (read more)

[-]gjm5mo*106

Could you please clarify what parts of the making of the above comment were done by a human being, and what parts by an AI?

[EDITED to add, in case it isn't already clear from Pablo's comment:] The above is not a response to what Kaj wrote that's now its parent, but to a different comment that has since been deleted for being AI-generated spam. I was, of course, not suggesting that Kaj's comment was AI-generated.

3habryka5mo

I deleted and banned the user, their last 3 comments were clearly AI generated and followed the usual AI-slop structure. Sorry for this one slipping through the cracks.

2Pablo5mo

Meta: gjm’s comment appears at the same level as comments that directly reply to Kaj’s original shortform. So until I read your own comment, I assumed they, too, were replying to Kaj. I think deleting a comment shouldn't alter the hierarchy of other comments in that thread.

2habryka5mo

Oops, that's a weird side-effect of the way we implemented spam purging (which is a more aggressive form of deletion than we usually use). We should really fix some bugs related to that implementation.

[-]Kaj_Sotala2y*90

I only now made the connection that Sauron lost because he fell prey to the Typical Mind Fallacy (assuming that everyone's mind works the way your own does). Gandalf in the book version of The Two Towers:

The Enemy, of course, has long known that the Ring is abroad, and that it is borne by a hobbit. He knows now the number of our Company that set out from Rivendell, and the kind of each of us. But he does not yet perceive our purpose clearly. He supposes that we were all going to Minas Tirith; for that is what he would himself have done in our place. And ac

... (read more)

[-]Kaj_Sotala6y90

I was thinking of a friend and recalled some pleasant memories with them, and it occurred to me that I have quite a few good memories about them, but I don't really recall them very systematically. I just sometimes remember them at random. So I thought, what if I wrote down all the pleasant memories of my friend that I could recall?

Not only could I then occasionally re-read that list to get a nice set of pleasant memories, that would also reinforce associations between them, making it more likely that recalling one - or just being reminded of my frie... (read more)

3eigen6y

This is a great idea! I also had somewhat the inclination to do this, when I first read about Anki on Michael Nielsen's -Aumenting Cognition, he speaks about using Anki to store memories and friends' characteristics such as food preferences (he talks about this on the section: "The challenges of using Anki to store facts about friends and family"). I did not do this because I did not want to meddle with Anki and personal stuff but I found another similar solution which is MONICA a "Personal Relationship Manager", the good thing about it is that it's open source and easy to set up. I did use it for a bit and found that it was very easy to use and had all the things one may want. I ended up not going through using the app at the time, but considering the post and the fact that people love when you remember facts about them (I also'd like to remember things about them!) I may pick it up again.

[-]Kaj_Sotala8d80

Google's new Nano Banana Pro is very good for image generation, I gave it a prompt that I figured was quite complicated and might not work and it got almost everything right.

Prompt:

[picture of me] This is me, can you draw a five-panel comic of me in a science fantasy setting. I should have a band of hovering multicolored gems h
overing around my wrists (nothing physically connecting them, they're hovering in air) as well as two futuristic drones floating around my head. One of them, Whisper, is specialized for reconnaissance and the other, Thunder, for comb

... (read more)

4Mateusz Bagiński8d

* In the first panel, you and the drones are turned right, as if this were the direction where the village is, but it's actually deeper/further in the scene. Same with the third panel, but less so. * Also, the village looks very different in the first and the third panel.

3Kaj_Sotala7d

True! In fairness, the first point is reasonably common for human-drawn scenes like this as well. If you want to show both the village and the main character's face, you need to have both of them facing the "camera", and then it ends up looking like this.

[-]Kaj_Sotala8y80

For a few weeks or so, I've been feeling somewhat amazed at how much less suffering there seems to be associated with different kinds of pain (emotional, physical, etc.), seemingly as a consequence of doing meditation and related practices. The strength of pain, as measured by something like the intensity of it as an attention signal, seems to be roughly the same as before, but despite being equally strong, it feels much less aversive.

To clarify, this is not during some specific weird meditative state, but feels like a general ongoing adjustment even ... (read more)

2Wei Dai8y

Interesting, I wonder if there is a way to test it, given that it seems hard to measure the pain:suffering ratio of a person directly... Is there a form of meditation that makes pain more aversive? Then we can have people who say “suffering isn’t any big deal and a pretty uninteresting thing to focus on” do that, and see if they end up agreeing with suffering-focused ethics?

2Kaj_Sotala8y

While this is a brilliant idea in the sense of being a novel way to test a hypothesis, trying to reprogram people's brains so as to make them experience more suffering strikes me as an ethically dubious way of doing the test. :) I wouldn't expect just a one-off meditation session where they experienced strong suffering to be enough, but rather I would expect there to be a gradual shift in intuitions after living with an altered ratio for a long enough time.

1daozaich8y

Regarding measurement of pain:suffering ratio A possible approach would be to use self-reports (the thing that doctor's always ask about, pain scale 1-10) vs revealed preferences (how much painkillers were requested? What trade-offs for pain relief do patients choose?). Obviously this kind of relation is flawed on several levels: Reported pain scale depends a lot on personal experience (very painful events permanently change the scale, ala "I am in so much pain that I cannot walk or concentrate, but compared to my worst experience... let's say 3?"). Revealed preferences depend a lot on how much people care about the alternatives (e.g. if people have bad health insurance or really important stuff to do they might accept a lot of subjective suffering in order to get out of hospital one day early). Likewise, time preference might enter a lot into revealed preference. Despite these shortcomings, that's where I would start thinking about what such a ratio would mean. If one actually did a study with new questionaires, one should definitely ask patients for some examples in order to gauge their personal pain-scale, and combine actual revealed preferences with answers to hypothetical questions "how much money would pain relief be worth to you? How much risk of death? How many days of early hospital release? etc", even if the offer is not actually on the table.

2Kaj_Sotala8y

Apparently there have been a few studies on something like this: "[Long-Term Meditators], compared to novices, had a significant reduction of self-reported unpleasantness, but not intensity, of painful stimuli, while practicing Open Monitoring."

[-]Kaj_Sotala9mo70

I dreamt that you could donate LessWrong karma to other LW users. LW was also an airport, and a new user had requested donations because to build a new gate at the airport, your post needed to have at least 60 karma and he had a plan to construct a series of them. Some posts had exactly 60 karma, with titles like "Gate 36 done, let's move on to the next one - upvote the Gate 37 post!".

(If you're wondering what the karma donation mechanism was needed for if users could just upvote the posts normally - I don't know.)

Apparently the process of constructing gat... (read more)

[-]Kaj_Sotala6y70

This paper (Keno Juechems & Christopher Summerfield: Where does value come from? Trends in Cognitive Sciences, 2019) seems interesting from an "understanding human values" perspective.

Abstract: The computational framework of reinforcement learning (RL) has allowed us to both understand biological brains and build successful artificial agents. However, in this opinion, we highlight open challenges for RL as a model of animal behaviour in natural environments. We ask how the external reward function is designed for biological systems, and how w

... (read more)

[-]Kaj_Sotala6y60

Recent papers relevant to earlier posts in my multiagent sequence:

Understanding the Higher-Order Approach to Consciousness. Richard Brown, Hakwan Lau, Joseph E.LeDoux. Trends in Cognitive Sciences, Volume 23, Issue 9, September 2019, Pages 754-768.

Reviews higher-order theories (HOT) of consciousness and their relation to global workspace theories (GWT) of consciousness, suggesting that HOT and GWT are complementary. Consciousness and the Brain, of course, is a GWT theory; whereas HOT theories suggest that some higher-order representation is (also) necessar... (read more)

[-]Kaj_Sotala8y60

Hypothesis: basically anyone can attract a cult following online, provided that they

1) are a decent writer or speaker
2) are writing/speaking about something which may or may not be particularly original, but does provide at least some value to people who haven't heard of this kind of stuff before
3) devote a substantial part of their message into confidently talking about how their version of things is the true and correct one, and how everyone who says otherwise is deluded/lying/clueless

There's a lot of demand for the experience of feeling like y... (read more)

4Trinley Goldenberg6y

Other common marketing advice that fits into this: * Set up a "bad guy" that you're against * If you're in a crowded category, either * Create a new category (e.g. rationality) * Set yourself up as an alternative to number in a category (Pepsi) * Become number one in the category (Jetblue?) * It's better to provide value that takes away a pain (painkillers) than that adds something that was missing (vitamins)

3eigen6y

I'd really like to read more about what you think of this. Another closely related feature they need is: * Content well formatted (The Sequences are a great example of this,The Codex). Of course, blogs are also a good basic idea which allows incremental reading. * Length of the posts? Maybe? I think there may be a case to be made for length helping to generate that cult following since it's directly related to the amount of time invested by people reading. There are many examples where posts could be summarized by a few paragraphs but instead they go long! (But of course there's a reason they do so).

[-]Kaj_Sotala1y50

Some time back, Julia Wise published the results of a survey asking parents what they had expected parenthood to be like and to what extent their experience matched those expectations. I found those results really interesting and have often referred to them in conversation, and they were also useful to me when I was thinking about whether I wanted to have children myself.

However, that survey was based on only 12 people's responses, so I thought it would be valuable to get more data. So I'm replicating Julia's survey, with a few optional quantitative questi... (read more)

1Sherrinford1y

The link is a link to a facebook webpage telling my that I am about to leave facebook. Is that intentional?

2Kaj_Sotala1y

Oh oops, it wasn't. Fixed, thanks for pointing it out.

[-]Kaj_Sotala8y50

So I was doing insight meditation and noticing inconsistencies between my experience and my mental models of what things in my experience meant (stuff like "this feeling means that I'm actively and consciously spending effort... but wait, I don't really feel like it's under my control, so that can't be right"), and feeling like parts of my brain were getting confused as a result...

And then I noticed that if I thought of a cognitive science/psychology-influenced theory of what was going on instead, those confused parts of my mi... (read more)

[-]Kaj_Sotala3mo42

Looking at the karma for some of my recent posts, "publish more by just writing about anything that happens to interest you and don't try to predict whether it'll interest the readers" sure is a high-variance strategy (as you might have expected).

2Dagon3mo

Do you have a cost metric for these? Sure, the variance is high, but they're all positive. What would a more conservative/successful strategy look like? Do you think more effort on the top-vote-getter would have gotten more incremental upvotes than the median (42)?

2Ruby3mo

Haha. I kinda woulda liked to have the posts without the karma and be asked to estimate the karma for each...I feel like I would have gotten these directionally right but you know, wasn't an advanced prediction ;)

3the gears to ascension3mo

that reminds me, tangential: I hand-wrote this crappy, fragile userstyle to hide prices on manifold (which might match more things than necessary (haven't noticed any such errors though))

2habryka3mo

That really is a lot of variance in karma, lol. Might be close to peak variance achieved by anyone on LW.

[-]lc3mo100

I mean, my normal spread looks like this:

[-]Kaj_Sotala7mo40

Didn't expect to see alignment papers to get cited this way in mainstream psychology papers now.

https://www.sciencedirect.com/science/article/abs/pii/S001002772500071X

Cognition
Volume 261, August 2025, 106131
Loopholes: A window into value alignment and the communication of meaning
Abstract. Intentional misunderstandings take advantage of the ambiguity of language to do what someone said, instead of what they actually wanted. These purposeful misconstruals or loopholes are a familiar facet of fable, law, and everyday life. Engaging with loopholes requires a n

... (read more)

[-]Kaj_Sotala1y40

What could plausibly take us from now to AGI within 10 years?

A friend shared the following question on Facebook:

So, I've seen multiple articles recently by people who seem well-informed that claim that AGI (artificial general intelligence, aka software that can actually think and is creative) in less than 10 years, and I find that baffling, and am wondering if there's anything I'm missing. Sure, modern AI like ChatGPT are impressive - they can do utterly amazing search engine-like things, but they aren't creative at all.
The clearest example of

... (read more)

4Seth Herd1y

Here's my brief pitch, starting with your point about simulation: The strength and flexibility of LLMs probably opens up several more routes toward cognitive completeness and what we'd consider impressive creativity. LLMs can use chain-of-thought sequential processing to do a type of mental simulation. If they are prompted to, or if they "prompt themselves" in a chain of thought system, they can access a rich world model to simulate how different actions are likely to play out. They have to put everything in language, although visual and other modalities can be added either through things like the whiteboard of thought, or by using CoT training directly on those modalities in multimodal foundation models. But language already summarizes a good deal of world models across many modalities, so those improvements may not be necessary. The primary change that will make LLMs more "creative" in your friends' sense is letting them think longer and using strategy and training to organize that thinking. There are two cognitive capacities needed to do this. There is no barrier to progress in either direction; they just haven't received much attention yet. LLMs don't have any episodic memory, "snapshot" memory for important experiences. And They're severely lacking executive functioning, the capacity to keep ourselves on-track and strategically direct our cognition. A human with those impairments would be very little use for complex tasks, let alone doing novel work we'd consider deeply creative. Both of those things seem actually pretty easy to add. Vector-based databases aren't quite good enough to be very useful, but they will be improved. One route is a straightforward, computationally-efficient improvement based on human brain function that I won't mention even though work is probably underway on it somewhere. And there are probably other equally good routes. The chain-of-thought training applied to o1, r1, Marco o1, and QwQ (and probably soon a whole bunch more) imp

2Kaj_Sotala1y

Self-driving cars seem like a useful reference point. Back when cars got unexpectedly good performance at the 2005 and 2007 DARPA grand challenges, there was a lot of hype about how self-driving cars were just around the corner now that they had demonstrated having the basic capability. 17 years later, we're only at this point (Wikipedia): And self-driving capability should be vastly easier than general intelligence. Like self-driving, transformative AI also requires reliable worst-case performance rather than just good average-case performance, and there's usually a surprising amount of detail involved that you need to sort out before you get to that point.

2Noosphere891y

I admit, I'd probably call self-driving cars at this point a solved or nearly-solved problem by Waymo, and the big reason why self-driving cars only now are taking off is basically because of regulatory and liability issues, and I consider a lot of the self-driving car slowdown as evidence that regulation can work to slow down a technology substantially.

2Kaj_Sotala1y

(Hmm I was expecting that this would get more upvotes. Too obvious? Not obvious enough?)

4habryka1y

It seems to me that o1 and deepseek already do a bunch of the "mental simulation" kind of reasoning, and even previous LLMs did so a good amount if you prompted them to think in chain-of-thoughts, so the core point fell a bit flat for me.

2Kaj_Sotala1y

Thanks, that's helpful. My impression from o1 is that it does something that could be called mental simulation for domains like math where the "simulation" can in fact be represented with just writing (or equations more specifically). But I think that writing is only an efficient format for mental simulation for a very small number of domains.

[-]Kaj_Sotala3y40

A morning habit I've had for several weeks now is to put some songs on, then spend 5-10 minutes letting the music move my body as it wishes. (Typically this turns into some form of dancing.)

It's a pretty effective way to get my energy / mood levels up quickly, can recommend.

It's also easy to effectively timebox it if you're busy, "I will dance for exactly two songs" serves as its own timer and is often all I have the energy for before I've had breakfast. (Today Spotify randomized Nightwish's Moondance as the third song and boy I did NOT have the blood suga... (read more)

[-]Kaj_Sotala6y40

Janina Fisher's book "Healing the Fragmented Selves of Trauma Survivors" has an interesting take on Internal Family Systems. She conceptualizes trauma-related parts (subagents) as being primarily associated with the defensive systems of Fight/Flight/Freeze/Submit/Attach.

Here's how she briefly characterizes the various systems and related behaviors:

Fight: Vigilance. Angry, judgmental, mistrustful, self-destructive, controlling, suicidal, needs to control.
Flight: Escape. Distancer, ambivalent, cannot commit, addictive behavior or being disorganized.
Free

... (read more)

[-]Kaj_Sotala8mo3-1

I gave this comment a "good facilitation" react but that feels like a slightly noncentral use of it (I associate "good facilitation" more with someone coming in when two other people are already having a conversation). It makes me think that every now and then I've seen comments that help clearly distill some central point in a post, in the way that this comment did, and it might be nice to have a separate react for those.

[-]Kaj_Sotala7y20

Huh. I woke up feeling like meditation has caused me to no longer have any painful or traumatic memories: or rather all the same memories are still around, but my mind no longer flinches away from them if something happens to make me recall them.

Currently trying to poke around my mind to see whether I could find any memory that would feel strongly aversive, but at most I can find ones that feel a little bit unpleasant.

Obviously can't yet tell whether some will return to being aversive. But given that this seems to be a result of giving my mind the cha... (read more)

[This comment is no longer endorsed by its author]Reply

2Elo7y

This seems similar to my experiences.

Moderation Log