This is a special post for quick takes by Siebe. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Siebe's Shortform
16Siebe
8Milan W
4Siebe
4Milan W
3Milan W
4[anonymous]
3Milan W
3Nathan Helm-Burger
4Milan W
3Siebe
3Milan W
1CstineSublime
2[anonymous]
1CstineSublime
1[anonymous]
1CstineSublime
1[anonymous]
1CstineSublime
2[anonymous]
1CstineSublime
1[anonymous]
1CstineSublime
1[anonymous]
1Milan W
1CstineSublime
7Siebe
5Siebe
4Viliam
3Siebe
1Siebe
1Karl Krueger
This might be a stupid question, but has anyone considered just flooding LLM training data with large amounts of (first-person?) short stories of desirable ASI behavior?
The way I imagine this to work is basically that an AI agent would develop really strong intuitions that "that's just what ASIs do". It might prevent it from properly modelling other agents that aren't trained on this, but it's not obvious to me that that's going to happen or that it's such a decisively bad thing to outweigh the positives
I have had this idea for a while. Seems like a good thing to do, looking from a simulators/direct value alignment frame. Might make corrigibility harder depending on exact implementation. Still, I'd expect it to be net-positive.
Invitation for critiques: If nobody convinces me it's a bad idea in a week's time from posting, I'll just proceed to implementation.
1Looks like Evan Hubinger has done some very similar research just recently: https://www.lesswrong.com/posts/qXYLvjGL9QvD3aFSW/training-on-documents-about-reward-hacking-induces-reward
The concerns about data filtering raised in that post's comments[1] suggest doing aligned-CoT-seeding on the pretraining data may be a better thing to try instead.
(you'll want to post the text in obscure-to-humans places that won't get a bunch of confused reactions from humans which would counter the effect)
I've been planning for a while to do a similar experiment with adding documents showing examples of AIs behaving in corrigible ways (inspired by talking with Max about Corrigibility as Singular Target)
I think examples of honest and aligned CoT resulting in successful task completion is also a good idea.
Want to collaborate on this experiment idea you have? I have time, and can do the implementation work while you mostly instruct/mentor me.
1I'll raise you an even stupider question: surely once an A.I. becomes sufficiently super-intelligent, all superintelligent systems will converge on certain values rather than be biased towards their initial training data? What expectations we condition it with about these first person stories about what it did will soon form only a small amount of it's corpus, as it interacts with the outside world and forms it's own models of the world, right?
I mean the way people talk about post-Singularity A.I. that can either bring about utopia, or drop all of the bombs and launch wave after wave of robot minions upon us - surely that means that it is capable of fast learning feedback loops, right? (although maybe I'm mistaken, what they mean is a plethora of domain specific super-intelligences, not a single all benevolent one?)
My understanding of AGI, not superintelligence, is a AI that can do the breadth of tasks a functional adult human can do. Now, that doesn't mean all the same tasks, but a similar degree of flexibility. Right? Put it in control of a robot arm and a baseball bat, and an AGI will teach itself how to hit a baseball as opposed to being trained by it's operators how to do it, it will have metacognitive abilities that will allow it to create a learning feedback loop.
Now if it has metacognition, then chances are it has the ability to change it's own goals - just people people.
Now imagine a therapy AGI - one day it is talking to a patient and then realizes (or thinks it realizes) that it understands the patient's goals and values better than the patient, and seeks to deceive or manipulation the patient towards the patient's own best-interest. Let's say the patient is suicidal, the AGI knows a way to outsmart the patient out of this action. Again, it has the ability to change it's own goals.
I mean, maybe it will be beholden to the initial training data? Maybe it will have a existential crises just like us? Analysis Paralysis and Ataxia brought on by inner conflict and confusion. Maybe it will join a cult for answers?
Now a ASI must be able to do this for extremely complicated plans, it can think strategically about taking over the world, and will learn the domain knowledge through fast feedback loops, right? An all powerful benevolent and highly corrigible ASI too must iterate through fast learning of oncology, agriculture, food chains, toxicology etc. etc. to keep humans healthy.
TL;DR - I just think that the further up the "intelligence" chain you start talking about an AI, the less important the initial training data is as it quickly will be conditioned by feedback from the complexity of the real-world.
I'll raise you an even stupider question: surely once an A.I. becomes sufficiently super-intelligent, all superintelligent systems will converge on certain values rather than be biased towards their initial training data?
Don't people usually have several terminal goals at any given time? I know it's tempting to neatly pack them all under a single heading like Conatus or Eudaimonia. But don't humans at times have conflicting terminal goals? Such as when an artist who wants to dedicate their life to their artform falls in love, and suddenly has two terminal goals where they only had one.
And this leads to a question about what distinguishes a very high level instrumental goal form a terminal goal. So let's say the artist notices that conflict and decides to go to therapy to sort it out - "successfully doing therapy" is obviously a instrumental goal, but to what terminal goal does it serve? Both? One more than the other which was their "true terminal goal" all along? Or have they popped into existence a new, third, terminal goal?
Is the stamp machine in a state of bliss like Sisyphus?
Don't people usually have several terminal goals at any given time?
That is not relevant to whether there are convergent terminal values[1].
To answer it anyways, people are not well-modeled as idealized terminal-goal-pursuers. More broadly, programs/minds don't have to be idealized terminal-goal-pursuers, so humans as a particular case of programs/minds-in-general present no paradox. "What is the true terminal goal" has a false premise that there must be some true terminal goal.
As for the case of idealized terminal-goal-pursuers, any two terminal goals can be combined into one, e.g. {paperclip-amount×2 + stamp-amount} or {if can create a black hole with p>20%, do so, else maximize stamps}, etc.
what distinguishes a very high level instrumental goal form a terminal goal
it being instrumental to some top-level goal
- ^
(or 'mind-independent moral facts', as the idea has been called in philosophy. https://plato.stanford.edu/entries/moral-anti-realism/)
1I'm probably completely misinterpreting you, but hopefully I can exploit Cunningham's Law to understand you better.[1] are you saying that superintelligent AGIs won't necessary converge in values because even a single superintelligent agent may have multiple terminal goals? A superintelligent AGI, just like a human, may not in fact have a single most-top-level-goal. (Not that we I assume a superintelligent AGI is going to be human-like in it's mind, or even AI to AI like as per that Eliezer post you linked).
That being said, some terminal goals may overlap in they share certain instrumental goals?
- ^
What I mean to say is I'm not intentionally being obstinate, I'm just really that dumb
are you saying that superintelligent AGIs won't necessary converge in values because even a single superintelligent agent may have multiple terminal goals?
No, I was responding to your claim that I consider unrelated. Like I wrote at the top: "That [meaning your claim that humans have multiple terminal goals] is not relevant to whether there are convergent terminal values"[1]
some terminal goals may overlap in they share certain instrumental goals?
I don't know what this is asking / what 'overlap' means. That most terminal goals share instrumental subgoals is called instrumental convergence.
- ^
Which in other words means, even if it were true, "humans have multiple terminal goals" would not be a step of the argument for it
1I don't know what this is asking / what 'overlap' means.
I was referring to when you said this:
any two terminal goals can be combined into one, e.g. {paperclip-amount×2 + stamp-amount} or {if can create a black hole with p>20%, do so, else maximize stamps}, etc.
Which I took to mean that some they overlap in some instrumental goals. That is what you meant right? That's what you meant when two goals can combine into one, that this is possible when they both share some methods, or there are one or more instrumental goals that are in service of each of those terminal goals? "Kill two birds with one stone" to use the old proverb.
If not, can you be explicit (to be be honest, use layman's terms) to explain what you did mean?
Which I took to mean that some they overlap in some instrumental goals. That is what you meant right?
No. I was trying to explain that: any agent that can be predicted by thinking of them as having two separate values for two different things, can also be predicted by thinking of them as maximizing some single value which internally references both things.
For example: "I value paperclips. I also value stamps, but one stamp is only half as valuable as a paperclip to me" → "I have the single value of maximizing this function over the world: {paperclip-amount×2 + stamp-amount}". (It's fine to think of it in either way)
can you be explicit (to be be honest, use layman's terms)
If you want, it would help me learn to write better, for you to list off all the words (or sentences) that confused you.
If you want, it would help me learn to write better, for you to list off all the words (or sentences) that confused you.
I would love to render any assistance I can in that regard, but my fear is this is probably more of a me-problem than a general problem with your writing.
What I really need though is a all encompassing, rigid definition of a 'terminal goal' - what is and isn't a terminal goal. Because "it's a goal which is instrumental to no other goal" just makes it feel like the definition ends wherever you want it to. Because, consider a system which is capable of self-modification and changing it's own goals, now the difference between an instrumental goal and a terminal goal erodes.
Never the less some of your formatting was confusing to me, for example a few replies back you wrote:
As for the case of idealized terminal-goal-pursuers, any two terminal goals can be combined into one, e.g. {paperclip-amount×2 + stamp-amount} or {if can create a black hole with p>20%, do so, else maximize stamps}, etc.
The bit " {paperclip-amount×2 + stamp-amount}" and " {if can create a black hole with p>20%, do so, else maximize stamps}" was and is very hard for me to understand. If it was presented in plain English, I'm confident I'd understand it. But using computer-code-esque variables, especially when they are not assigned values introduces a point of failure for my understanding. Because now I need to understand your formatting, and the pseudo-code correctly (and as not a coder, I struggle to read pseudo-code at the best of times) just to understand the allusion you're making.
Also the phrase "idealized terminal-goal-pursuers" underspecifies what you mean by 'idealized'? I can think of at least four possible senses you might be gesturing to:
A. a terminal-goal-pursuer who's terminal goals are "simple" enough to lend themselves as good candidates for a thought experiment - therefore ideal from the point of view of a teacher and a student.
B. ideal as in extremely instrumentally effective in accomplishing their goals,
C. ideal as in they encapsulate the perfect undiluted 'ideal' of a terminal goal (and therefore it is possible to have pseudo-terminial goals) - i.e. a 'platonic ideal/essence' as opposed to a platonic appearance,
D. "idealized" as in that these are purely theoretical beings (at this point in time) - because while humans may have terminal goals, they are not particularly good or pure examples of terminal-goal-havers? The same for any extant system we may ascribe goals to?
E. "idealized" in a combination of A and B which is very specific to entities that have multiple terminal goals, which is unlikely, but for the sake of argument if they did have two or more terminal goals would display certain behaviors.
I'm not sure which you mean. But suspect it's none-of-the-above.
For the record, I know you absolutely don't mean "ideal" as in "moral ideal". Nor in an Aesthetic or Freudian sense, like when a teenager "idealizes" their favourite pop-star and raves on about how perfect they are in every way
But going back to my confusion over terminal goals, and what is or isn't:
For example: "I value paperclips. I also value stamps, but one stamp is only half as valuable as a paperclip to me" → "I have the single value of maximizing this function over the world: {paperclip-amount×2 + stamp-amount}". (It's fine to think of it in either way)
I'm not sure what this statement is saying, because that describes a possibly very human attribute - that we may have two terminal goals in that they are not subservient or means of pursuing anything else. Which is what I understand a 'terminal' goal to mean. The examples in the video describe very "single-minded" entities that have a single terminal goal that they seek to optimize, like a stamp collecting machine.
There's a few assumptions I'm making here: that a terminal goal is "fixed" or permanent. You see when I said sufficiently superintelligent entities would converge on certain values, I was assuming that they would have some kind of self-modification abilities. And therefore their terminal values would look a lot like common convergent instrumental values of other, similarly self-adapting/improving/modifying entities.
However if this is not a terminal goal, then what is a terminal goal? And for a system that is capable of adapting and improving itself, what would be it's terminal goals?
Is terminal goal simply a term of convenience?
1
1
1consider a system which is capable of self-modification and changing it's own goals, now the difference between an instrumental goal and a terminal goal erodes.
If an entity's terminal goal is to maximize paperclips, it would not self-modify into a stamp maximizer, because that would not satisfy the goal (except in contrived cases where doing that is the choice that maximizes paperclips). A terminal goal is a case of criteria according to which actions are chosen; "self-modify to change my terminal goal" is an action.
But isn't there almost always a possibility of a entity goodharting to change it's definition of what consitutes a paperclip that is easier for it to maximize? How does it internally represent what is a paperclip? How broad is that definition? What power does it have over it's own "thinking" (sorry to anthropamorphize) does it have to change how it represents the things which that representation relies on?
Why is it most likely that it will have an immutable, unchanging, and unhackable terminal goal? What assumptions underpin that as more likely than fluid or even conflicting terminal goals which may cause radical self-modifications?
A terminal goal is a case of criteria according to which actions are chosen; "self-modify to change my terminal goal" is an action.
What does "a case of criteria" mean?
goodharting to change it's definition of what consitutes a paperclip that is easier for it to maximize
Same thing applies. "Does that fulfill the current goal-definition?" (Note this is not a single question; we can ask this about each possible goal-definition)
Why is it most likely that it [...]
This was about an abstract definition of an agent (not itself a prediction, but does say something about a space of math, that we might end up in). There are surely possible programs which would exhibit any behavior, although some look harder to program (or 'less natural'): for example, "an entity that is a paperclip maximizer for 100 years, then suddenly switches to maximizing stamps" looks harder to program (if an embedded agent) because you'd need to find a method where it won't just self-modify to never turn into a stamp maximizer (as turning into one would prevent if from maximizing paperclips), or to not unleash a true paperclip maximizer and shut itself down if you rule out just self-modification (and so on if you were to additionally rule out just that).[1]
- ^
(though very tangentially there is a simple way to do that)
There's a number of priors that lead me to expect much of the current AI safety research to be low quality:
-
A lot of science is low quality. It's the default expectation for a research field.
-
It's pre-paradigmatic. Norms haven't been established yet for what works in the real world, what are reliable methods and what is p-hacking etc. This makes it not only difficult to produce good work, it also makes it hard to recognize bad work and hard to get properly calibrated about how much work is bad, the way we are in established research fields.
-
It's subject to selection effects by non-experts. It gets amplified by advocates, journalists, policy groups, the general public. This incentivizes hype, spin etc. over rigor.
-
It's a very ideological field. Because there's not a lot of empirical evidence to go on, and a lot of people's opinions were formed before LLMs exploded, and people's emotions are (rightly) strong about the topic.
-
I'm part of the in-group and I identify with - sometimes even know - the people doing the research. All tribal biases apply.
Now, some of this may be attenuated by the field being inspired by LessWrong and therefore having some norms like research integrity, open discussion & high criticism, but I don't think those forces are strong enough to counteract the other ones.
If you believe "AI safety is fundamentally much harder than capabilities, and therefore we're in danger", you should also believe "AI safety is fundamentally much harder than capabilities, and therefore there's a lot of invalid and unreliable claims".
Also, this will vary for different subfields. Those with tighter connection to real-world outcomes, like interpretability, I would expect to be less bad. But I'm not familiar enough with the subfields to say more about specific ones.
Substack continues growing, it looks sustainable and profitable, and unlike BlueSky isn't attracting only one part of the political spectrum. I don't see Substack doing much to raise the sanity waterline, but it might be a worthy target for some rationalist lobbying to improve their architecture.
Substack has a great business model, but from technical perspective it sucks. Good enough to post articles and send them to paying subscribers -- which is where the money comes from -- but reading a ACX-size comment section is painful. Also, as an author, if I want anything non-trivial, like making a word simultaneously italic and a hyperlink, or to choose a smaller font size so that I can have code blocks with 80 characters per line, no chance.
Unfortunately, the scale and network effects mean that "make another Substack, but one that doesn't suck technically" probably isn't a good business idea. Or rather, would require a very large investment.
I'm starting a discussion group on Signal to explore and understand the democratic backsliding of the US at ‘gears-level’. We will avoid simply discussing the latest outrageous thing in the news, unless that news is relevant to democratic backsliding.
Example questions:
-
“how far will SCOTUS support Trump's executive overreach?”
-
“what happens if Trump commands the military to support electoral fraud?”
-
"how does this interact with potentially short AGI timelines?”
-
"what would an authoritarian successor to Trump look like?"
-
"are there any neglected, tractable, and important interventions?"
You can join the group here: https://signal.group/#CjQKIE2jBWwjbFip5-kBnyZHqvDnxaJ2VaUYwbIpiE-Eym2hEhAy21lPlkhZ246_AH1V4-iA (If the link doesn't work anymore in the future, DM me.)
Shallow take:
I feel iffy about negative reinforcement still being widely used in AI. Both human behaviour experts (child-rearing) and animal behavior experts seem to have largely moved away from that being effective, only leading to unwanted behavior down the line
People often use the term "negative reinforcement" to mean something like punishment, where a teacher or trainer inflicts pain or uncomfortable deprivation on the individual being trained. Is this the sort of thing you mean? Is there anything analogous to pain or deprivation in AI training?