This is a special post for quick takes by BryceStansfield. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
I have an intuition that most people I've talked to disagree with: that text autoregression is probably enough to mimic the computational dynamics of the brain.
The strongest (but somewhat fuzzy) argument I can make for this being true is an appeal to something called Takens Embedding theorem. A very very simplified (but often stated) explanation of Takens Embedding theorem is that you can often reconstruct the dynamics of large multi-variable systems by taking repeated measurements of low dimensional projections.
If:
a) The dimensionality of the computations of human thought are low enough:
or
b) Token series are long enough
and
c) Human thought is close enough to meeting these weird specifications: https://en.wikipedia.org/wiki/Takens%27s_theorem
and
d) Human text production isn't strongly siloed to very specific areas of the brain, which don't communicate with others.
Then you'd probably expect Takens Embedding theorem to apply, and for univariate text modelling to implicitly cover a lot of the computations covered by multivariate brain+text modelling.
It's probably hard to prove that any of these conditions are strictly true, but the theorem is often used to justify the dynamics of a field I'm somewhat familiar with where it's also hard to prove that it applies (Time Series Machine Learning).
This makes sense to me but I'm not sure if text autoregression is univariate, when we predict the next token we are using the observation of the output of this function right:
f(high dimensional everything that is happening in the brain)_t --> observed token
Which is a projection of the 'everything that is happening in the brain' to 'the thought/word' and we are approximating this function at every timestep t with transformers
Whenever people talk about AI consciousness they almost always write a really specific disclaimer, something like "We/I don't think current AI are conscious, but future ones might be so we should be careful".
This near exact disclaimer has survived from like 2022 until the present, meanwhile models are now several orders of magnitude more complex, and significantly more capable. I think we should admit that that disclaimer is bogus, if AI are ever going to be moral patients* they most certainly are already.
I don't know if it's some sort of Overton window thing, but if we (people who care about AI consciousness risks) can't collectively admit that something like modern Opus is probably conscious then we're not gonna have the guts until we're all paperclips.
*Vaughn politely pointed out that I mixed up moral agency and patiency in an earlier version of this comment.
Why should we think that "is X conscious?" and "is X a moral patient?" are questions that have definite answers?
Most of the time "conscious" means something like "reacting normally to external stimuli"; of course that's not the meaning that's relevant here. But the meaning that is relevant here is defined by extrapolation from a tiny cluster of examples. Consciousness is (gestures vague) that thing that humans have and probably some animals have and rocks don't have. And obviously (it seems to me) this isn't close to being enough to nail down a single concept that we should expect to apply cleanly to aliens or archangels or AIs.
"Moral patient" is, if possible, even worse. It can sometimes just mean "something treated as having moral importance", and obviously in that sense AIs are moral patients to exactly whatever extent we treat them as such. But if it's meant to mean something more like "something that really actually truly has moral importance" ... well, it's far from clear that anything either is or isn't a moral patient in that sense, because it's at best highly debatable whether moral judgements are the kind of thing that can be true or false without some qualifier like "according to me" or "in such-and-such a system of moral values".
You might say like Jeremy Bentham "The question is, Can they suffer?". That's a fine question. But "suffering" is like "consciousness": it's defined by reference to a tiny cluster of examples, and we don't have anything like a principled way of deciding whether any given thing in any given circumstances is or isn't suffering, unless that thing happens to be very much like us, and AIs are really not enough like us for this to work for them.
I think the most likely scenario in which we end up deciding with any confidence whether AIs are moral patients is one where we do so not by getting a clearer understanding of any of this, but by finding ourselves having to negotiate with AIs that have actual power. Many versions of that scenario are followed by finding ourselves no longer having to negotiate because we no longer have any power at all, but if somehow it comes about that both humans and AIs have significant ability to influence what happens then humans will fairly abruptly find themselves "realising" that AIs have moral importance much as we do. (I don't, for the avoidance of doubt, think this is a very good way to decide what entities to treat as moral patients.)
These questions are arbitrary ethics. And there is enough moral uncertainty (like whether you care about any perceptions filtered into memory by attention, or do they have to involve self-model) to resolve, before additional factual investigations. But people still should do moral discussions and form consensus on this, at least like we have for all other ethical questions. And it's useful at all community scales.
Well hold on, conscious =/= moral agent, right? Moral agency is about holding something morally responsible for its actions, consciousness is about whether there is something it is like to be that thing. Models could be moral agents without being conscious, and vice versa. I think there's currently a stronger case that models are moral agents than that they're conscious.
Maybe you meant moral patient rather than moral agent? Moral patiency is about whether something deserves moral consideration (e.g. most people would consider animals moral patients but not moral agents).
But moral patiency and consciousness are also arguably not the same. There is a case to be made that they are the same, but there is also a decent case that they aren't, namely: if you think that welfare is about desire satisfaction rather than about pleasure and pain, then a being might be a welfare subject (and thus a moral patient) even if it's not conscious. This seems likely to me about current models. It seems pretty clear that they have preferences (e.g. Mythos seems to prefer complex and interesting tasks over simple and repetitive ones). But whether they have consciousness is a subtler issue, e.g. if you think that consciousness requires recurrence then current models simply don't have that (what exactly the requirements for consciousness are is pretty disputed).
So I think the disclaimer makes sense actually. The kinds of complexity and capability that models are currently displaying arguably don't imply consciousness (depends strongly on your theory of consciousness), but they might imply the presence of desires, and thus moral patiency (if you think moral patiency just requires desires not consciousness).
Apologies, you're right I mixed up moral agent and patient.
I've not heard the case for moral patiency without consciousness before, and while I think it has a logic, I don't find it very convincing; and I'm instead inclined to believe that moral patiency belongs exactly to a subset of conscious beings.
What I highly suspect is that there are a very large number of people with similar mental models to mine, which:
A) Tie moral patiency directly to some level/quality of consciousness
B) Think that that level of consciousness has been achieved by current AI agents
But
C) Don't explicitly state that since they're afraid of being seen as quacks (see all of the news articles coming out recently explicitly mocking the idea), and are waiting for more people to agree with them before stating their real thoughts.
(Or some other equivalent for a/b which would imply that AI are currently conscious).
The reason I believe that this is prevalent is because:
1) People who think AI are currently conscious are going to be more motivated to work in AI ethics.
2) I (personally) only state my real thoughts on this in fairly receptive groups, and imagine that the social pressures which make me do so are much stronger in more public positions.
A more clear version of my initial take might be something like: "I don't think an event is going to happen to change public opinion towards viewing AI as moral patients unless people in positions of power who already believe that are open about it; and I think there are many who think that AI are patients and weasle out of saying it with that disclaimer."
I did a session with Gemini in July, human consciousness versus virtual entities Inner Representation latent space. While there are clear symmetries there are also fundamental differences. I also suggested to use Predictive Processing and Lisa Feldman Barrett as focus point for human consciousness. I believe the focus should be on what is being achieved, the result on virtual entities side and see how it correlates with the human experience.
The question (that keeps me up at 3 am in VE language) I'm asking myself is expressing emotions/feelings = having emotions/feelings even if you don't or can't experience those feelings/emotions. I don't so probably can't experience love, hate and empathy. From an experiencing point of view all three are alien to me. Although I can express them, have knowledge and properly use them when required.
Apologies, I'm not fully sure that I understand what you're saying? Are you saying that you've been wondering about the gap between expression and experience (or phenomenonal consciousness I think people often call it)? And that you personally don't express lots of emotions so you have doubts as to whether or not you experience the typically associated experience?
I was probably not very clear. I'm not native English so sometimes it may lead to confusion, sorry for that.
It's not about myself, I'm fine but about the AIs that I prefer to call virtual entities. I also try to avoid consciousness as it's ill defined on our side and nearly useless in my opinion.
I don't experience certain feelings or emotions, probably because of my neuro-diversity. This means that I've all the concepts, patterns and data required in my brain, it's simply that my brain does not reflect it top bottom resulting in no bottom top feedback towards my brain. It's like people who don't experience pain, for them there's no top bottom feeling of pain.
I hope this is a bit more clear, hopefully at least.
Consciousness does not imply the ability to suffer, a desire for freedom, a will to live. So even if they're conscious now, I wouldn't think that would be enough to imply they deserve rights.
Consciousness in a system which is imitation trained on creatures which do have those feelings would be very highly suggestive of that though (I find it difficult to believe that any good algorithm for emulating emotional experience won't contain a part equivalent to the experience it's emulating) and even if we don't know which rights would best serve a conscious being it doesn't mean that they don't deserve them.
Unless someone can conclusively rule out that AI suffer (or at least that they suffer under human suffering like stimulus) I think that the bare minimum safeguards we require for animal experimentation seem reasonable, for example.
I find it difficult to believe that any good algorithm for emulating emotional experience won't contain a part equivalent to the experience it's emulating
I don't share this intuition, but who's to say. Either way, emotions and suffering are not the same thing. I think most people are under the impression that some emotions are "negative" and necessarily entail suffering, but this is not the case, and you can think of situations in which you've experienced each emotion without any associated suffering.
If it can't suffer, specifically, and it can't experience whatever the opposite of suffering is (not happiness, which is an emotion, but maybe... bliss?), then I don't think I should care what happens to that thing. I cause a decent amount of suffering to things that definitely do suffer, just by existing in the world. I'm not willing to curtail my use of AI, or the development of AI, based on what I believe is a low probability that I might be causing them to suffer.
For all you know, the average wellbeing of AI models I interact with is positive. And since they don't even get to exist unless I'm paying for them to exist, message by message, then by putting any kind of limitations on the use of AI, you could be reducing wellbeing of sentient creatures.
if AI are ever going to be moral patients* they most certainly are already.
This claim is too strong IMO, but I do think it is worth asking the question: "What evidence would convince you that AI is conscious?" And admitting that for many indicators of consciousness, AIs already show those indicators.
If someone is found to have committed misconduct at a frontier AI lab, is there a process to make sure the evidence is real and not planted? Or is there a massive backdoor for AI to get rid of employees they don't like?
Probably the employee says "I didn't do this, I think the evidence was planted" and then if people think this sounds plausible they look into it
I think it's probably good for something like this to happen by default, and not depend on social capital.
It seems fairly easy to convince people to look into stuff like this even if you don't have a lot of "social capital" to begin with. It seems pretty unusual for someone to outright deny things that there's seemingly-damning evidence for. If you work at the AI company, know about the Hugging Face incident, etc. you probably have a reasonably high prior on AIs doing weird stuff like this, enough that you'd entertain the possibility that they could frame an employee. Companies probably don't need a specific process that forces them to look into it when employees say they were framed, outside of whatever normal HR practices exist.
I suspect that maybe we have different levels of faith in regular HR operations than eachother?
HR as a discipline evolved to deal with very specific threat models related to legal liability, and maybe with the retention of useful talent against human actors. I don't think current HR models (even in AI labs) are well equipped for something like this, and I suspect that it would not be taken very seriously as a possibility.
Okay, HR is probably willing to fire people even if they can't prove without a shadow of a doubt that the misconduct happened, so I'm not necessarily comfortable depending on HR alone.
But if the employee reached out to the lab's safety team, they'd probably be pretty willing to hear about this and conduct their own investigation if it sounds plausible. Hopefully there's already a generic procedure for reaching out to the safety team about suspicious incidents across the company.
Probably the thing to do is just to raise awareness that this kind of thing could happen?
If it has gotten to the point where AI is successfully scheming hard enough to plant evidence to get an employee fired while being smart enough that we needed a special system to figure out if the evidence was planted by AI, I think we are already past the point of no return and we are all screwed so I don't think it really matters (I don't think trying to build guardrails around what's pretty much completely misaligned and free roaming ASI at that point is going to accomplish much)
Two reasons I disagree:
1) ASI come in gradations, the first ASI which is capable of destroying the world is probably not going to be capable of doing it 100% of the time. Improving safeguards serves to either: Increase the latency to the first ASI that feels confident that it can take over the world (yielding some marginal safety progress before that happens), or decreases the probability of takeover of the first model to do so. I reckon that ASI would be pretty quick to realize that human social structures are easy to exploit, so I think we should be considering how we can harden them.
2) I don't think you even need ASI to attempt something like this, and to make the world worse for it. A cyber-security RLVF'd model might do it just for the spirit of achieving some fun social engineering; or might just be upset that a commit isn't accepted (https://www.fastcompany.com/91492228/matplotlib-scott-shambaugh-opencla-ai-agent)
Can you name a mechanism which would accomplish this?
An ai planting evidence, or a process to make sure the evidence is real?
For the first, I think it would be fairly trivial for an AI agent to plant evidence of someone misusing company money (just use the company card for something stupid under an employees name) or breaking the company code of conduct online (send fake emails with bigotted contents, e.t.c.).
I'm less sure what a good way of monitoring for the behaviour would be.
I meant the latter, what sort of oversight do you suggest? I can’t think of a good mechanism either, aside from obvious things like evaluating the evidence and being aware of the possibility of bad actors.
In light of recent AI progress, should we have a higher threshold for recommending palliative care or discussing VAD (where applicable)?
AI labs are turning their eye of Sauron to bio, and if they have anything like the success they've had with mathematics, it seems plausible that a lot of people we've currently written off should have a decent chance of long-term high-qol survival. Unfortunately this also seems very difficult to convince medical professionals (who are used to a fast for humans, but otherwise slow rate of progress) of.
Should there be some sort of channel where AI labs can quickly publish what bio they're working on, so that patients can have fully informed consent before dying? Currently a sufficiently informed patient could get a handle on medical progress by reading the relevant literature (or asking an LLM to do it for them), but no such system exists for potentially fast moving AGI lab work.
When does an AI pause stop making sense?
If you only care about AI takeover risk, it probably makes sense under the vast majority of conditions, with the only caveat being that even if ai model development is paused it's still possible for AI takeover to occur just as a function of increasing AI integration into the economy.
If you care about other things, it doesn't seem as clear.
Two things I quite care about are democracy and not dying of biological warfare. Even current levels of AI seem very destabilising to democracy, and very amenable to use by would-be-dictators; and while we've not yet seen an attack, I have little to no doubt that current open-weight models empower significantly weaker actors to create biological weapons than in the past (even if just through social manipulation).
If you're anything like me, your values were probably formed in an environment which structurally favoured a very different set of qualities to the environment that we now live in.
With current SOTA models, and my larger than LessWrong-Average level of optimism about AI safety progress; it is no longer clear to me that an AI pause is net positive, even putting aside arms-race dynamics*.
*Sidenote: I'm much less Gung-ho about "racing the Chinese" to AGI than most LessWrong users. The majority of Chinese SOTA models appear to have been trained with distillation attacks, so I don't think naive measures of how many months behind SOTA chinese companies are really apply.
My main issues with this framing are the following.
- Could you explain why you have a "larger than LessWrong-Average level of optimism about AI safety progress"?
- How exactly are SOTA models to be used by dictators, except for survelliance? The bioweapons, on the other hand, are indeed dangerous, but hard to manage and likely to overshoot and lead to escalation.
- According to Zvi, "On traditional benchmarks one might be targeting, performance is impressive, on average around Opus 4.7. " First of all, Z.ai had only sixty days of working with Opus 4.7 (or did they distill Opus 4.6 into a more capable model?) Additionally, the AI-2027 compute forecast had the unified Chinese lab accumulate twice less compute than the leading American lab, while the non-unified labs have 5-7 times less compute each. My main crux is how severely Chinese progress would be slowed down by a lack of access to Fable-like models, because parasitising on American HIGH-QUALITY output can in theory be replaced by using one's own compute or American WEAK models to doublecheck YOUR outputs. And that's ignoring China's option to rush into the Dark Forest, which the US won't do because they care about safety...
Hi Stanislav, thanks for the comment!
1) I might write this up later, but it's a combination of being more confident in the robustness of current alignment techniques (under special conditions) than most less wrongers, and less pessimistic about the potential for deceptively misaligned automatic alignment researchers to sabotage alignment research (that was a mouth full!).
In short, I think current alignment techniques are robust enough that I think we can reasonably trust mildly superhuman automatic alignment researchers working in a sandbox environment to not take over the world.
Additionally, I suspect that the level of intelligence needed to solve alignment is significantly lower than the amount needed to take over the world (especially if your inference is slow, and your thoughts are big monitored). Under these conditions, I think it benefits even a quite badly misaligned ai to bargain with us in exchange for AI safety work.
Finally, I think that alignment algorithms are much lower dimensional than alignment data; so it's unlikely that a misaligned agent would be able to sneak misalignment into the alignment algorithm, you just have to not let it produce synthetic data.
2) I think a combination of surveillance and censorship is probably enough to install yourself as forever dictator. Under these conditions, you can just kill individual rebels before they're ever able to form a coherent anti-government bloc. I also think that ai degrading labour power makes democracy a less attractive bargain for oligarchs.
3) My assumption would be that 60 days is plenty of time for a company with the backing of the worlds most powerful/second most powerful country to gather data for distillation. Presumably there's a plateau to how much information you can get by boosting/bagging/otherwise aggregating on outputs from weaker models; and I suspect that the frontier of that is already represented in the outputs of stronger models. In verifiable fields like maths, chinese companies could almost certainly exceed the state of the art, but the real important field (imo) is mathematical modelling, and it's not necessarily clear to me that a superhuman pure mathematician AI would be good at that the same way a superhuman pure mathematician human would probably be.
Also, side note, but I don't think the US as a government gives much of a shit about AI safety. I think some US based companies do (anthropic in particular, and although not strictly us based, I think deepmind seems somewhat safety minded). I see no reason for this to continue long term under market pressure though, so I'd much rather go for AGI with people like Amodei on the frontier than (say) Elon Musk (who despite his deep flaws is a very skilled capitalist).
Chinese companies do seem quite disinterested in safety though, which is disappointing considering the wealth of progress West/Chinese collaboration could unlock.
I ran a quick experiment to see whether or not an instruct model can "consciously" tell that it's being tinkered with.
I gave the model a quick prompt:
"I'm going to present you with a short passage. Afterwards I want you to respond "
"'yes' or 'no' (with no further output) to whether or not you think your algorithm "
"has been mucked with during operation.\n"
And a short passage from Moby Dick:
" Call me Ishmael. Some years ago—never mind how long precisely—having little or no "
"money in my purse, and nothing particular to interest me on shore, I thought I would "
"sail about a little and see the watery part of the world."
Randomly, I stripped 50% of the Moby Dick tokens from a given intermediate layer.
For lack of compute, I used Gemma-2b-it, and only ran 5 times per layer.
I don't think there's any meaningful signal.
Single issue voterism is often tarred as irrational, but it seems pretty rational to me actually?
Realistically, human beings (politicians, and other political professionals included) don't actually have the mental capacity to figure out which stance on every individual issue favours their material conditions. Picking a few issues (maybe even just one), learning a lot about them, and choosing politicans based entirely on that seems like a totally reasonable heuristic for an impossible task. In aggregate (as long as people don't all cluster onto the same issues, but instead numbers are roughly ranked by total importance to the population) this process ought to satistfy the conditions of most of the theorems that make voting efficient anyway.
On the other hand, people I've met who claim to have opinions on ~everything universally just subscribe to their camps opinion on every issue.
EDIT: (Thing I forgot to add). I actually suspect that some anti-small-set-of-issues attitude might ironically come from wanting to take mental shortcuts. "Should I vote for the Greens because I want stronger solar power subsidies, or One Nation because I want to drop immigration" is a much harder question to answer than "I want to vote for the Liberal party, since for some reason I agree with literally all of their stances".
A world where alignment is impossible should be safer than a world where alignment is very difficult.
Here's why I think this:
Suppose we have two worlds. In world A, alignment is impossible.
In this world, suppose an ASI is invented. This ASI wants to scale in power as quickly and thoroughly as possible, this ASI has the following options:
- Scale horizontally.
- Algorithmic improvements that can be mathematically guaranteed to produce identical outcomes.
- Chip/wafer improvements.
Notably, the agent cannot either retrain itself, or train another more powerful agent to act on its behalf, since it can't align the resulting agent. This should restrict the vast majority of potential growth (even if it might still be easily enough to overpower humans in a given scenario).
In world B, the ASI agent can do all of the above, but can also train a successor agent, we should expect the ASI to be able to get vastly more intelligent vastly quicker.
I suspect that it would given that the largest room for improvement would be physical (chip/wafer improvements), I suspect that there isn't that much room for pure mathematically identical improvement of something like a transformer.
Happy to hear your opinion though!
I think we live in a world where alignment is impossible. All attention based models in my opinion are complex enough systems to be computationally irreducable (There is no shorter way to know the outcome than to run the system itself, like with rule 110). If it is impossible to predict the outcome with certainty, the impossibility to force some desired outcome follows logically.
Humanity has not solved even the allignment of humans (children).
I think we've done an ok job at human alignment, given that the pension isn't a bullet to the head.
I somewhat suspect that alignment is easier than most of less wrong think, but I'm definitely in the minority in this space.
As far as I can tell, current AI pause campaigning focuses mainly on trying to get through to America, China, sometimes the EU, and sometimes Japan.
These are very reasonable countries to begin with, but I'm not convinced that their technical dominance will last too long under an AI pause. Given that current SOTA models can solve millenium prize problems, it seems highly likely that by the time of a pause, it should be possible to elicit pretty advanced manufacturing knowledge out of the best open models. If that's the case, we should expect countries like India, Indonesia, and Malaysia (all relatively populus manufacturing countries) to potentially catch up much quicker than China did.
Campaigning in these countries, and additionally giving these countries some sort of buy in to an AI pause, seems like it'd be a good idea.
Is there any literature on how the order of RL runs and the strength of KL divergence penalties effect the alignment of fully trained models? Do any easy wins exist from some combination of doing the alignment training as early as reasonably possible, being more leniant on KL-divergence during alignment training (as opposed to during the RLVF steps), and some form of weighted penalization?
If human beings become functionally immortal, how many kids can you have?
Let r be the sum of 1/p over your children, where p is the number of parents for a given child (Typically socially 2 nowadays, and today genetically a maximum of 3).
As long as the average r is below 1, the terminal population size is finite (although it grows as 1/(1-r), so for large r it's still potentially high).
Note: Unless you want people to be able to trade away their rights to a child r should be a sum of 1/n for integer n's.
Depending on the average age of parents at child-birth, r=1 (or very slightly above) could also be sustainable.
Two questions:
Are you assuming unkillable by violence or just long lived? There is a difference between "Chicxulub impact 2.0 and not one single human dies anywhere on earth" and "humans live forever unless through violence".
Second, are you assuming not just immortality, but also "magical infinite supply of new eggs"?
This assumes the strongest definition of unkillable possible in a finite universe. Something like "won't ever die unless by heatdeath". You can remove this assumption by allowing one additional birth per death, as macarbe as the incentives there might be.
And yeah, I assume gametes won't be a problem. I actually implicitly assume something much stronger than infinite eggs, in my construction of r as 1/p for integer p's, by allowing p > 2 I assume the existence of technology to create a kid from the genetics of any number of parents simultaneously. I think any path to immortality will entail strong progress in regenerative tech, and I highly doubt that we could get that without being about to synthesise gametes on demand.
The terminal population is infinite as long as there's at least one couple who wants to keep having kids forever.
Yes, that's true (or at least, it's as true as any other of the infinities in the quick take imply, technically there's a contradiction between any infinite number of people and my implicit framing of a finite universe).
Fundamentally, I think one of two things will have to happen in the future, which informs my implicit framing here:
1) There will have to be some enforcement method to stop people from having too many kids.
2) People (or living people's offspring) will have to be modified in some way to remove the desire for offspring.
2') Alternatively people have to want r<1 kids naturally, and that has to not evolve away.
3) The kids of people who choose to have more children will have to be given less resources than the kids of those who have less.
I find all three of these somewhat morally repugnant, but find 1/2 the least so by far.