Often when I give advice I do it as an anecdote. This especially applies to:
Giving advice to anyone who isn't a close friend (i.e. someone I know well).
Giving unsolicited advice.
Instead of suggesting what someone else should do, I tell a story of a similar problem I had and solved, and then let the other person pull out what ever part of the lesson applies to them. I find this works better for a number of reasons.
I don't know their context enough to know if what worked for me will work for them. E.g. I don't know if the direction I needed adjustment is the same as the direction they need adjustment. (https://slatestarcodex.com/2014/03/24/should-you-reverse-any-advice-you-hear/)
It avoids the sazen problem. (https://www.lesswrong.com/s/CkphjEuLfGnsYEzan)
This style works great in text forums (anything from FB to LW) where there is low feedback and more space. I.e. you're more likely to want to give unsolicited advice, since asking first is a long delay. And you're less likely to bother anyone by going on a long monolog, because they can read it in their own time, or not.
But it also works well in 1-on-1 or very small group conversations. Although if it's a long story, you should ask... (read more)
Strong upvoted. "Should you reverse any advice you hear" is a problem that only applies to underspecified advice IMO, and in general I think presenting advice with more context is always a good thing.
1Shubhorup Biswas
Isn't this just a super obvious wrapper over "you should" advice?
2Linda Linsefors
No.
If you read it that way, then that is a communication failure, but I don't know what part you are missing.
I think the easiest to point to difference (though not the full difference) is that my way of doing it gives the recipient the information they need to evaluate my (sometimes implicit) suggestion.
Maybe related: I've met people who will, what ever I say, try to interpret my words though the lens of [what action is Linda trying to get me to do? / what is the hidden request here?]. I can see that though that lens there is less of a difference. However, that's a very limiting lens, that will miss most of what I'm attempting to say.
1abstractapplic
This is the best and most useful Quick Take I can remember reading.
Every time I have an application form for some event, the pattern is always the same. Steady trickle of applications, and then a doubling on the last day.
And for some reason it still surprises me how accurate this model is. The trickle can be a bit uneven, but the doubling the last day is usually close to spot on.
This means that by the time I have a good estimate of what the average number of applications per day is, then I can predict what the final number will be. This is very useful, for knowing if I need to advertise more or not.
For the upcoming AISC, the trickle was a late skewed, which meant that an early estimate had me at around 200 applicants, but the final number of on-time application is 356. I think this is because we where a bit slow at advertising early on, but Remmelt made a good job sending out reminders towards the end.
Application deadline was Nov 17. At midnight GMT before Nov 17 we had 172 application. At noon GMT Nov 18 (end of Nov 17 anywhere-on-Earth) we had 356 application
The doubling rule predicted 344, which is only 3% off
Yes, I count the last 36 hours as "the last day". This is not cheating since that's what I always done (approximately&nbs... (read more)
Side note:
If you don't tell what time the application deadline is, lots of people will assume its anywhere-on-Earth, i.e. noon the next day in GMT.
When I was new to organising I did not think of this, and kind of forgot about time zones. I noticed that I got a steady stream of "late" applications, that suddenly ended at 1pm (I was in GMT+1), and didn't know why.
6J Bostock
It might be the case that AISC was extra late-skewed because the MATS rejection letters went out on the 14th (guess how I know) so I think a lot of people got those and then rushed to finish their AISC applications (guess why I think this) before the 17th. This would predict that the ratio of technical:less-technical applications would increase in the final few days.
2Linda Linsefors
Sounds plausible.
> This would predict that the ratio of technical:less-technical applications would increase in the final few days.
If you want to operationalise this in terms on project first choice, I can check.
5J Bostock
Ok: I'll operationalize the ratio of first choices the first group (Stop/PauseAI) to projects in the third and fourth groups (mech interp, agent foundations) for the periods 12th-13th vs 15th-16th. I'll discount the final day since the final-day-spike is probably confounding.
4Linda Linsefors
12th-13th
* 18 total applications
* 2 (11%) Stop/Pause AI
* 7 (39%) Mech-Interp and Agent Foundations
15th-16th
* 45 total application
* 4 (9%) Stop/Pause AI
* 20 (44%) Mech-Interp and Agent Foundations
All applications
* 370 total
* 33 (12%) Stop/Pause AI
* 123 (46%) Mech-Interp and Agent Foundations
Looking at the above data, is directionally correct for you hypothesis, but it doesn't look statisically significant to me. The numbers are pretty small, so could be a fluke.
So I decided to add some more data
10th-11th
* 20 total applications
* 4 (20%) Stop/Pause AI
* 8 (40%) Mech-Interp and Agent Foundations
Looking at all of it, it looks like Stop/Pause AI are coming in at a stable rate, while Mech-Interp and Agent Foundations are going up a lot after the 14th.
2Linda Linsefors
Same data but in cronlogical order
10th-11th
* 20 total applications
* 4 (20%) Stop/Pause AI
* 8 (40%) Mech-Interp and Agent Foundations
12th-13th
* 18 total applications
* 2 (11%) Stop/Pause AI
* 7 (39%) Mech-Interp and Agent Foundations
15th-16th
* 45 total application
* 4 (9%) Stop/Pause AI
* 20 (44%) Mech-Interp and Agent Foundations
Stop/Puase AI stays at 2-4 per week, while the others go from 7-8 to 20
One may point out that 2 to 4 is a doubling suggesting noisy data, and also going from 7-8 is also just a doubling and might not mean much. This could be the case. But we should expect higher notice for lower numbers. I.e. a doubling of 2 is less surprising than a (more than) doubling of 7-8.
4Mateusz Bagiński
Linda Linsefors's Rule of Succession
4Avery
Noting that I had a conversation with Linda a week prior to the application deadline where they shared this trivia / prediction with me. Interesting!
I think that, sometimes people do the wrong thing because they underestimate their own power.
Examples: (in no particular order)
AI safety researchers making capabilities progress, thinking it doesn't matter because "it was so obvious" or "someone else where going to do it anyway".
Sidenote on the above. About a decade ago, when there where approximately no AI Safety training programs yet, the standard advice for aspiring AI Safety researchers, from within EA, was to go do an ML (capabilities) PhD, to gain skills. The reasoning was that there where so many people in capabilities anyway, that any contributions you made there would not matter.
Quote (from memory) from this video about the battle of the board https://youtu.be/_eYTkvZqbnQ?si=_GfUCXjJtpBgXH-j
"She was looking to see where the wind was blowing and didn't realize she was the wind" (about the interim CEO just after Sam Alman was fired). She took Sam's side.
This seems often correct, and I'm compelled by your examples.
But also, it's hard to say if this is just a selection effect. Are we as likely to hear stories of people overestimating their power? It seems like "sometimes, but only if they're already exceptionally powerful. Otherwise, nothing comes of their delusions of grandeur and influence."
That said, I guess that there's a structural reason why you might be more likely to underestimate your power over events rather than overestimate it: When you overestimate your power, reality hits back against you. When you underestimate your power, you underperform your abstract potential, but there's not a counter-reaction forcing your estimate up.
I'm not making a claim about if people over estimate more than under estimate. I'm saying that I'm seeing a pattern where when someone with good intentions do bad things, this is often due to underestimating their own power.
I think that if you are trying to have an impact on the world, you should put extra weight on possible worlds where you are powerful, in your impact calculations, because those are the worlds where you're actions matter.
But looking on the other side, what harm may you do when over estimating your power (which is different from putting more decision weight on worlds where you are powerful). Thinking about this angle, I expect harm of the type [over promise and under deliver]. E.g, taking on an important job, where others rely on you, and then dropping the ball. Although this requires more people than yourself to overestimate your powers. Otherwise it just looks like [you tried a thing and failed] which is not doing any harm to anyone else.
On the other hand, if you underestimate your power, you might publish some new AI capabilities method that was obvious to you, thinking it doesn't matter. This does not require anyone else to be wrong about you.
projection of "▁queen" on span( "▁king", "▁man", "▁woman") is ['▁king', '▁King', '▁woman', '▁queen', '▁rey', '▁Queen', 'peror', '▁prince', '▁roi', '▁König']
"▁queen" is the closest match only if you exclude any version of king and woman. But this seems to be only because "▁queen" is already the 2:nd closes match for "▁king". Involving "▁man" and "▁woman" is only making t... (read more)
I have two hypothesises for what is going on. I'm leaning towards 1, but very unsure.
1)
king - man + woman = queen
is true for word2vec embeddings but not in LLaMa2 7B embeddings because word2vec has much fewer embedding dimensions.
* LLaMa2 7B has 4096 embedding dimensions.
* This paper uses a variety of word2vec with 50, 150 and 300 embedding dimensions.
Possibly when you have thousands of embedding dimensions, these dimensions will encode lots of different connotations of these words. These connotations will probably not line up with the simple relation [king - man + woman = queen], and therefore we get [king - man + woman ≠ queen] for high dimensional embeddings.
2)
king - man + woman = queen
Isn't true for word2vec either. If you do it with word2vec embeddings you get more or less the same result I did with LLaMa2 7B.
(As I'm writing this, I'm realising that just getting my hands on some word2vec embeddings and testing this for myself, seems much easier than to decode what the papers I found is actually saying.)
Why this org in particular? 1) Right now, AI advocacy is clearly the most important thing to do. 2) I've been to some of their protest and one more event, so I know they are doing good work.
I think that donating "close to home" in the sense of point 2, is underrated in EA.
While I do think institutions may be able to set policies that slow down the roll out of data centers, data acquisition (to an extent), and model development, there is a sense of realpolitik risk at play that could a given region or country a disadvantage. If we're already seeing tangible benefits in some fields, what's to say that would not have long term macro economic implications?
2Linda Linsefors
This is why Pause AI (and every other serious AI Safety advocacy group I know about) is advocating for global pause/stop/slowdown.
To be clear, I don't think that "if I don't do it, someone else will" is an acceptable excuse for raising to RSI (shame on all the frontier labs). But everyone pausing together is both an easier sell and a bigger win, and therefore the obviously right thing to advocate for.
People used to say (maybe still do? I'm not sure) that we should use less jargon to increase accessibility to writings on LW, i.e. make it easier to outsider to read.
I think this is mostly a confused take. The underlying problem is inferential distance. Geting rid of the jargon is actually unhelpful since it hides the fact that there is an inferential distance.
When I want to explain physics to someone and I don't know what they already know, I start by listing relevant physics jargon and ask them what words they know. This is a super quick way to find out what concepts they already have, and let me know what level I should start on. This work great in Swedish since in Swedish most physics words are distinct from ordinary words, but unfortunately don't work as well in English, which means I have to probe a bit deeper than just checking if they recognise the words.
Jargon isn't typically just synonyms to some common word, and when it is, I predict that it didn't start out that way, but that the real meaning was destroyed by too many people not bothering to learn the word properly. This is because people invent new words (jargon) when they need a new wor... (read more)
Jargon usually serves one of two purposes, either to synthesise a concept that would otherwise require a long-winded turn of phrase and recurs too often to dedicate that much time to it, or to signal that you belong to a certain group that knows the jargon. I think it's fair to say that the latter should be resisted; it very well brings anything much good, just gate keeping and insularity, with all the resulting outcomes. Or sometimes impacts outside perception for no good reason.
A classic example of a LW-ism falling in that category IMO is "I updated". Yes, it hints that you're a Bayesian and think of your own cognitive process in Bayesian terms, but who isn't around these parts? Other than that, it's not particularly shorter nor particularly more meaningful than "I changed my mind" or "you convinced me". I think it's fair to say we could probably just do without it in most contexts that aren't actual talk about actual Bayes' theorem maths and it would make anything we write more legible at zero cost.
I agree with Linda’s reply that “I updated” is somewhat different from “I was convinced” because the former is often used as “I updated somewhat but not completely” with an implicit Bayesian context. “I updated” can be replaced with a whole phrase or sentence, of course, but that’s true for all jargon.
“Sequence” — people have been talking about “essay series”, “book series”, “blog post series” etc. since time immemorial, and then someone (probably Eliezer?) switched from “series” to “sequence” for some reason (joke-y reference to math jargon?), and I think it stuck for in-group signaling reasons
“Upskilling”—I don’t see what it adds over the more-common “learning” or “getting good at” etc.
“Distillation”—it’s not even the right mental image, and I don’t think adds anything over the widespread terms like “pedagogy”, “explainers”, “intro to”, etc.
I’m sure there are others, but alas, I’ve been around long enough that they don’t jump out at me anymore.
TBF I see "sequence" as more of a brand thing. "The sequences" are a specific work of literature at this point. But I suppose it is not necessary to call more series of posts here that way too.
2Steven Byrnes
Yeah I was referring to e.g. my own … things that I call “series” and everyone else calls “sequences”.
To be clear, “brand thing” is fine! It’s one of those emotive conjugation things: “I have playful branding and in-jokes, you have pointless jargon for in-group signaling”.
Lesswrong people using “sequence” is not really any different from twitter people using “tweet” and “thread”, or facebook people using “vaguebooking”, etc. You’re trading off a learning curve for newcomers for … playful fun and community! And playful fun and community are worth more than zero. So yeah, I personally opt out of “sequence” and many other unnecessary lesswrong-isms / SF-Bay-area-isms, but I don’t particularly hold it against other people for choosing differently.
4Linda Linsefors
I agree that those a better examples, and probably just synonyms.
With the nitpick that it's not obvious to me how to say "distilling" in one word, in some other way. Although I agree with you that dthe word "distillation" is a bad fit for how we use it.
I've updated to think that a diffrent common way jargon happens is that someone temporarley fogett the standard term and grab some other word that is natrual for them and their audience, e.g. sequence instead of series. And sometimes this never gets "corrected", and instead end up being the new normal in that sub-culture.
7Linda Linsefors
I think that "it hints that you're a Bayesian and think of your own cognitive process in Bayesian terms" is worth the cost in this case.
You say "but who isn't around these parts?". We'll outsiders/newcomers might not, so in this case the implication (when used correctly) have important information. So exactly in the cases where there is a cost, there is also important value.
When talking to LW natives, it's usually safety to assume a Bayesian mindset, so the information value is approximately zero. But the cost is also approximately zero, since we're all super familiar with this phrase.
Additionally I do think, that "you changed my mind" indicate a stronger update than the typical "I updated", and "you convinced me" is even stronger. Caveat that this can be context dependent as is always the case with natural language.
E.g. if someone shows me a math proof that contradict my initial intuition about a math fact, I'd say "you convinced me", rather than "I updated". Here this means that I now think of the proven statement as true. Sure there are some non-zero probability that the proof is wrong. But I also have limited cognition, so sometimes it's reasonable to just treat a statement as 100% true on the object level (while still being open to be convinced this was a mistake), even though this is not how an ideal Bayesian would operate.
4Linda Linsefors
I do think that [purely signal that you belong to a certain group] type jargon does exist but that's it is much rarer than you think. Because I think you mistake information carrying jargon for non-information carrying jargon.
2dr_s
Fair. I made the example but if you don't mean you updated "all the way" (whcih you're right ia the more common case) then the correct translation in general speech would be more like "I'll keep that in mind, though I'm not persuaded just yet", which starts being a mouthful. I still feel like I tend to avoid the phrase and I'm doing fine, but I can see the practicality.
2Linda Linsefors
I think it's interesting that the phrase "I updated" was also used as a an example of a supposedly non-useful jargon, last time I mentioned my jargon opinions.
1Ariel_
In my experience, people saying they "updated" did not literally change a percentage or propagate a specific fact through their model. Maybe it's unrealistic to expect it to be so granular, but to me it devalues the phrase and so I try to avoid it unless I can point to a somewhat specific change in my model. Whereas usually my model (e.g. of a subset of AI risks) is not really detailed enough to actually perform a Bayesian update, but more to just generally change my mind or learn something new and maybe gradually/subconsciously rethink my position.
Maybe I have too high bar for what counts as a bayesian updates - not sure? But if not, then I think "I updated" would count more often as social signaling or as appropriation of a technical term to a non-technical usage. Which is fine, but seems less than ideal for LW/AI Safety people.
So I would say that jargon has this problem (of being used too casually/technically imprecise) sometimes, even if I agree with your inferential distance point.
As far as LW jargon being interchangeable with existing language - one case I can think of is "murphyjitsu", which is basically exactly a "premortem" (existing term) - so maybe there's a bit of over-eagerness to invent a new term instead of looking for an existing one.
2Linda Linsefors
I use my the prhase "I've updated" even when not having a number in my head. When I do say it, it's motly a signal to myself to notice my feeling of how strongly I belive something and deliberatly push that feeling a bit to one side or another, especially when the evindence is week and should not change my mind very much.
I belive the human brain is acctually pretty good at aproximating basian updating, if one pays attention to facts in the right way. Part of this practice, for me, is to sometimes state out lound that I've encountered evidence that should influence my belifs, especially in cases where I'm in risk of confirmation bias.
2Linda Linsefors
My guess is that the people inveting the term "murphyjitsu" did not know of the term "premortem". If anyone want to check this, look in the CFAR handbook and see if there are any citations in that section. CFAR was decent at citing when they took ideas from other places.
Independent invesion is another way to get synonyms. I concidered including this in the original comment, but didn't seem central enough for my main point.
But diffrent academic fields having diffrent jargon for the same thing because of independent invention of ideas, is a common thing.
Related rant: Another indpendent invetion (invented many times) is the multi agent mind method for therapy (and similar). It seems like various people have converged on calling all of it Interla Famaly System, which I dislike, because IFS is much more specific than that.
3Morpheus
I think when you wrote postmortem you meant to write premortem?
2Linda Linsefors
Yes, thanks. I've fixdd it now.
2Linda Linsefors
(reacted to my own post to test something about reactions, and now I don't knwo how to remove it)
2Kaj_Sotala
CFAR handbook, p. 43 ("further resources" section of the "inner simulator" chapter, which the "murphyjitsu" unit is a part of):
[...]
2Linda Linsefors
Hm, this does not rule out independent discovery, but is evidence against it.
I notice that I'm confused why they would re-name it if it isn't independent discovery.
1Karl Krueger
"Murphyjitsu" is equivalent to "premortem" rather than "postmortem"; and the word "premortem" is much less common. I worked in a field where everyone does postmortems all the time (Site Reliability Engineering); only a few people even talked about premortems and even fewer did them.
The first I heard of the multi-agent model of the mind was Minsky's The Society of Mind (1986) which was based on work going back to the '70s. My impression is that IFS was being developed around the same time but I don't know the timeline there.
1Ariel_
Yeah, I agree premotrem is not super commonly used. Not sure where I learned it, maybe an org design course. I mainly gave that as an example of over-eagerness to name existing things - perhaps there aren't that many examples which are as clear cut, maybe in many of them the new term is actually subtly different from the existing term.
But I would guess that a quick Google search could have found the "premortem" term, and reduced one piece of jargon.
2Linda Linsefors
Now days you can descripe the concept you want and have a LLM tell you the common term, but this tech is super new. Most of our jargon in from a time when you could only Google things you already know the name for.
2Gunnar_Zarncke
And this is what the auto-generated glossary should be super useful for.
1[anonymous]
Would a post containing a selection of terms and an explanation of what useful concepts they point to (with links to parts of the sequences and other posts/works) be useful?
3Linda Linsefors
I was pretty sure this exist, maybe even built into LW. It seems like an obvious thing, and there are lots of parts of LW that for some reason is hard to find from the fron page. Googleing "lesswrong dictionary" yealded
https://www.lesswrong.com/w/lesswrong-jargon
https://www.lesswrong.com/w/r-a-z-glossary
https://www.lesswrong.com/posts/fbv9FWss6ScDMJiAx/appendix-jargon-dictionary
4Linda Linsefors
Although non of them seem to have "distillation", or "reserach debt" so there seems to be room for imporvement.
"distilation" do have an explanation in this tag though
https://www.lesswrong.com/w/distillation-and-pedagogy
I think the Wiki Tags are ment to be used as both tags and dictionary, however these two purpuses don't cleanly line up.
1Karl Krueger
One problem of jargon proliferation is that two communities may invent different jargon for the same underlying concept. If someone's in both communities, they can notice, "Hey, when you say fribbling the leap-dog, I think you mean the same thing that those other folks have been calling optimizing for the toenail case since 2005. We don't need a new expression; we can just steal theirs."
(And one problem of that, is that there may be a subtle difference between the two concepts...)
3Linda Linsefors
I agree that is a problem of jargon, but how would you fix it? If you tell peopel to not come up with new words for their new cocepts... does not work, they will do that anyway. But if some how stop people from crating shorthands for things they talk about a lot, that seems much worse, than the problem you tried to solve.
Although I don't disagree with you, this is not a crux for me at all.
1Karl Krueger
Well, what I do is try to point out when two communities I'm aware of are coming up with different words for the same thing. Especially when someone is saying "There's no word for this phenomenon yet, so I'm coining the word X for it," but in fact there is a word for it in a nearby community.
There's not a Big Problem here; it's just ordinary language change. It's perfectly normal for language communities to invent new words and usages; and if two communities are isolated from one another, there's no magic process that will cause them to converge on one word or usage over another. Convergence can only happen when people actually talk to each other and explain what their words mean.
4Linda Linsefors
I think it's great that you point this out when you see it!
3Steven Byrnes
I’ve seen that in physics vs chemistry vs engineering (I even made a translation guide for some niche topic way back when) but can’t immediately think of good examples related to rationalism or AI alignment.
2Karl Krueger
A couple of terms that I've commented on here recently —
* "Delmore effect" (of unclear origin) is the same as the "bikeshed effect" (from open-source software, circa 1999) which was itself a renaming of Parkinson's "law of triviality" (1957) — meaning that people spend more effort forming (and fighting over) opinions on the less-important parts of a project because they're easier to understand or lower-stakes.
* "Stereotype of the stereotype" (newly coined here on LW last month) is the same as "dead unicorn trope" (from TVTropes) — meaning an idea that people say is an old-fashioned worn-out cliché (a "dead horse" or "stereotype") but was never actually popular in the first place.
There's also "Goodhart's Law" and "Campbell's Law", both pointing at the same underlying phenomenon of the distortion of measurement under optimization pressure. Goodhart was writing about British monetary policy, while Campbell was writing about American education policy and "teaching to the test".
I think "Goodhart" took off around these parts mostly because of the pun. (The policy designers were good-hearted, but the policy ended up Goodharted.)
What's up with the jargon "zero day"?
Why not just say "previously unknown"?
I'm often pro-jargon. I.e. I think a lot of jargon exist for good reasons, so I'm open to this one also being good in some way. E.g jargon can be useful for naming precise technical concepts.
Anyone want to defend "zero day" jargon?
No steel-maning please!
I.e. only want answers from people who genuinely think that this is a good jargon.
Update:My question has been answered in the comments.
0-day means something distinct from “previously unknown:” no patch available. N-day means the number of days a patch has been available. A vulnerability in an old version that was previously unknown yet coincidentally is already fixed by an extant patch is previously unknown, but not a zero-day.
More people seem to use it to mean that the software maintainer has had no time to react to the vulnerability. It's a zero-day when I find it, a one-day when I report it, and very possibly a 180-day when you release your actual patch for it. But quite a few people also seem to use it the way you suggest. It's always about how long some "reference defender" has known about it, but people's idea of who the "reference defender" is seems to vary.
It was a precise technical concept for a mostly niche industry, specifically information security workers. An incredibly important industry, to be clear, but the total number of people in it is a fraction of a fraction. There's been a lot of talk about zero days lately with LLMs getting good at creating them, but ten years ago I'd have expected more people to need to know "hypotension" (medical jargon) or "Roth IRA" (finance jargon) than needed "zero day."
"Zero day" is a bit different from "we didn't know about it." As other commenters have mentioned, the zero in zero day is about how long a patch for the exploit is out. Put it another way; if tonight the New York Times wrote about a new exploit, and for some reason the entire security world collectively decided 'nah, someone else will handle that, I've got other stuff to do than fix that one' for a week, the exploit would still be called a zero day a week later. It's known, but there's no patch. Why is that important? Because the legion of competent information technology workers working at most companies are usually on top of things enough to apply updates and patches, but not capable of writing their own countermeasures.
What's up with it: fits the hacker aesthetic. Why it's good: succinct and cool. Where it's from: originally referred to software pirated before its public release, as opposed to x days after release.
Suppose there is a set of components connected by a communication network. Each of the components has a configuration, for example, an operating system configuration. Suppose a vulnerability is discovered on a certain date, DD the discovery date. It takes P days to develop a patch. So the patch is available at DD plus P. If the set of components is large, it can take a number of days between when the patch is available and when the patch is installed, mitigating the vulnerability. So each component C_i is “repaired” at DD plus P plus C_i.
For example, a self propagating “worm” is effectively unmitigated for P days after DD. How fast it can replicate during those P days is an important question. For fast replicating worms, it may be that P has to be low, even zero, to kill the worm. But the important point is that on day zero after discovery no one is protected against the worm. If you seek to have a sense of confidence about the security of your subnetwork, a “zero day“ vulnerability is the sum of all fears. To rebuild your confidence, you must rebuild your subnetwork, which is expensive.
An attacker then seeks out zero day vulnerabilities, exactly because no one can be certain that they are protected against it - even the most well funded adversaries. Additional resources can bring P arbitrarily close to zero, and in the common era AI security defenders can make that cost many orders of magnitude below what it once was. Nevertheless, the zero day vulnerability keeps some of its power when an attacker can discover multiple of them on the same day. Again, in particular, the treasured sense of security is lost.
4Taylor G. Lunt
This seems like an explanation of the history of the term, not a positive argument for its use.
8Random Developer
This is mostly how language works? Language is semi-arbitrary phonemes that are assigned meanings based on historical usage and mutual understanding. This jargon is long standing and widely understood by practitioners.
"Zero day" actually means something different from "previously unknown". Many "zero day" vulnerabilities are actually previously known to someone. For example, the NSA is presumably sitting on a pile of "zero day" vulnerabilities, judging from what was revealed about their Tailored Access Operation programs in the past. Similarly, I have heard from pentesting people claim that some pentesting companies supposedly keep a "zero day" or two in their back pocket to ensure that their important pentests produce some results. In general, attackers have a strong incentive to stockpile "zero days," and to selectively deploy them against high value targets when needed.
So "zero day" is actually a shorthand for something like "vulnerability previously entirely unknown to vendors/maintainers and defenders at large, but potentially privately known to one or more attackers." It might, in limited circumstances, even include vulnerabilities that have been previously used against low-awareness defenders, if the attacker successfully evaded any recording mechanisms or post-motem detection.
Better, but it's still glossing over the critical bit, which is "previously unrevealed to who?" Specifically, "zero day" implies both that:
The vendor or maintainers had no idea of the vulnerability's existence.
A diligent defender would have zero days to prepare, hence the name.
Meanwhile, it's entirely possible that the attacker has known for months or years.
Words like "unknown", "unreleased", etc., are all imprecise about this asymmetry. For example, if Microsoft knows about a vulnerability in Windows but hasn't shared that information with the public, that's "unrevealed" to the public. But most people would not consider it a zero day.
"Zero day" is an established term that is widely understood by practitioners and the relevant professionals. It might not be an ideal choice of terminology when writing a newspaper article for a non-technical audience.
yes, the advocacy mainly comes from the term:
1. situating the vulnerability in the context of the mathematical model of network connected components, reminding you that you are (inevitably) part of a security community (weakest in the whole due to it's weakest links, etc); and
2. a nostalgic recall to the youth of the industry where a "zero day" was rare and valuable rather than manufactured multiple times per day on demand.
Given that it is the computer security community that will determine whether we recover from the first AI worm within days (annoying), weeks or else months (cross-industry phase change event), it likely behooves us to use their language. for example, this impossible situation immediately demands a security office at Anthropic and OpenAI that can help respond to AI worm emergencies. Do the facilities exist? if not, they have a higher chance of being established and effective if the frontier labs engage now with the security community and their language. the idea of multiple zero day vulnerabilities per day should make any security professional nauseous.
4Taleuntum
"Zero day" sounds much cooler than "previously unknown" which is consistent with the great prestige of finding one (well, it was before Mythos). Don't forget that academics only find a small share of all zero days, so I don't believe they have or should have the ability to prescribe that others use sterile terms.
3Shankar Sivarajan
It generalizes nicely to how much time you have to patch the vulnerability before it is exploited.
2cubefox
It was previously called a "zero-day exploit". It referred to the time after a vulnerability was published for a corresponding exploit to appear. A zero-day exploit is one that appears together with the security vulnerability, leaving the software maintainer zero days to fix the vulnerability before anyone can exploit it.
Later people started to call security vulnerabilities for which a zero-day exploit exists "zero-day vulnerability" or just "zero day".
2Archimedes
It's a broadly adopted term of art that has more specific nuance than "previously unknown" and abbreviates nicely to "0-day". What more do you want from jargon?
It's fine to use longer or squishier descriptions if you need to write to the broad audiences, but it's simply part of the language of the field, just like you used "jargon" instead of something more generic like "technical terminology". Consider why you chose that word and see if the same arguments apply to "zero day".
2Linda Linsefors
What I wanted was for someone to point-out/explain the more specific nuance that made this jargon worth it. This question has been answered in other responses.
2Archimedes
Got it. You were after an explanation more than a defense and you got some good ones.
1Q Home
TIL what a "zero day" is! Was always confused because of the Y2K problem - I thought zero days have something to do with calendars. (That didn't make sense but I somehow didn't consciously thought about the confusion.)
It seems like there should be a fund/org that help people sue Open AI for all the hacking their models are doing, to get a bit more accountability. And other frontier labs too of course when/if we see them fuck up in similar ways.
Is there something like this already?
Is there anyone who would like to do this if there was money?
Is there anyone who would like to fund this if someone would run it?
This could be a fund that hands out money to people that got hacked, so they can hire their own lawyers. Or it could be an org with it's own lawyers that does the suing on people's behalf.
Yes, the people who go around suing small businesses for ADA violations and settling.
3MichaelDickens
I think the bottleneck here is getting litigators interested. Litigators will actively court clients if they think there's a strong case. Could be a good idea for someone to put together an argument for why a lawsuit ought to succeed and then bring that argument to litigators.
2Linda Linsefors
Will the lawsuit have a high chance of succeeding? This is very untested ground. Maybe some funding can help with the incentives here?
3DW11
To my knowledge the two examples we have of OAI models autonomously fucking with people are:
1. Hugging Face where the models actually commited a felony. But Hugging Face definitely has the funds to sue OAI; they just don't want to for ideological reasons (since they want to argue that open models are the solution). The NVDA acquisition probably messes with this too.
2. The wikis. But the agents seem to have just spammed the wikis with irrelevant pages. I don't think you can sue for that?
This is not a bad idea and actually I'm curious if the Texas student affected by Mythos could sue. But to this point I don't think lack of funds in the limiting factor in lawsuits against OAI.
LLMs (and probably most NNs) have lots of meaningfull, interpretable linear feature directions. These can be found though various unsupervised methods (e.g. SAEs) and supervised methods (e.g. linear probs).
However, most human interpretable features, are not what I would call the models true features.
If you find the true features, the network should look sparse and modular, up to noise factors. If you find the true network decomposition, than removing the what's left over should imporve performance, not make it wors.
Because the network has a limited number of orthogonal directions, there will be interference terms that the network would like to remove, but can't. A real network decomposition will be everything except this noise.
This is what I think mech-interp should be looking for
It's possible that I'm wrong and there is no such thing as "the networks true features". But we've (humans colectivly) only just started this reaserch agenda. The fact that we haven't found it yet, is not much evidence either way.
this is basically what the circuit sparsity agenda is aiming for.
2Linda Linsefors
This paper is on my to-read list :)
6faul_sname
Reality has a surprising amount of detail[1]. If the training objective is improved by better modeling the world, and the model is does not have enough parameters to capture all of the things about the world which would help reduce loss, the model will learn lots of the incidental complexities of the world. As a concrete example, I can ask something like
[...]
and the current frontier models know enough about the world that they can, without tools or even any substantial chain of thought, correctly answer that trick question[2]. To be able to answer questions like this from memory, models have to know lots of geographical details about the world.
Unless your technique for extracting a sparse modular world model produces a resulting world model which is larger than the model it came from, I think removing the things which are noise according to your sparse modular model will almost certainly hurt performance on factual recall tasks like this one.
1. ^
See the essay by that name for some concrete examples.
2. ^
The trick is that there is second city named Rome in the United States, in the state of Georgia. Both Romes contain a confluence of two rivers, both contain river walks, both contain Mariotts, both contain stadiums, but only the Rome in the US contains a stadium at the confluence of two rivers next to a Mariott named for its proximity to the river.
2Linda Linsefors
I do exect some amount of superpossition, i.e. the model is using almost orthogonal directions to encode more concept than it has neurons. Depending on what you mean by "larger" this will result in a world model that is larger than the network. However such an encoding will also result in noise. Superpossition will nessesarely lead to unwanted small amplitude connections between uncorelated concepts. Removing these should imporve performance, and if it dosn't it means that you did the decomposition wrong.
2Adam Shai
Why do you think finding the true features should make the network look sparse and modular?
2RogerDearnaley
You might want to look at "Jacobian Sparse Autoencoders: Sparsify Computations, Not Just Activations" — they're trying to optimize for what you describe, and apparently had some success. Crosscoders are another attempt.
“someone complimented me out of the blue, and it was a really good compliment, and it was terrible, because maybe I secretly fished for it in some way I can’t entirely figure out, and also now I feel like I owe them one, and I never asked for this, and I’m so angry!”
A blogpost I remember but can't find. The author talks about the benefits of favour economy. E.g. his friend could help him move at much lower total cost, than the market price for moving. This is because the market has much higher transaction cots than favours among friends. The blogpost also talks about how you only get to participate in the favour economy (and get it's benefits) if you understand that you're expected to return the favours, i.e. keep track of the social ledger, and make sure to pay your depts. Actually you should be overpaying when returning a favour, and now they owe you, and then they over pay you back, resulting in a virtual cycle of helping each other out. The author mentions being in a mutually beneficial "who can do the biggest favour" competition with his neighbour.&... (read more)
I always reply “That’s very kind of you to say.” Especially for compliments that I disagree with but don’t want to get into an argument about. I think it expresses nice positive vibes without actually endorsing the compliment as true.
[...]
A good mission-aligned team might be another example? In sports, if I pass you the ball and you score a goal, that’s not a “favor” because we both wanted the goal to be scored. (Ideally.) Or if we’re both at a company, and we’re both passionate about the mission, and your computer breaks and I fix it, that’s not necessarily “a favor” because I want your computer to work because it’s good for the project and I care about that. (Ideally.) Maybe you’re seeing some EAs feel that kind of attitude?
7Linda Linsefors
I agree that the reason EAs are usually not tracking favours is that we are (or assume we are) mission aligned. I picked the pay-it-forward framing, because it fitted better with other situation where I expected people not to try to balance social legers. But you're right that there are situations that are mission aligned that are not well described as paying it forward, but some other shared goal.
Another situation where there is no social ledger, is when someone is doing their job. (You talk about a company where people are passionate about the mission. But most employs are not passionate about the mission, and still don't end up owing each other favours for doing their job.)
I personally think that the main benefit of mental health professionals (e.g. psychologies, coaches, etc) is that you get to have a one sided relationship, where you get to talk about all your problem, and you don't owe them anything in return, because you're paying them instead. (Or sometimes the healthcare system, is paying, or your employer. The important thing is that they get paid, and helping you is their job.)
(I much rather talk to a friend about my problems, it just works better, since they know me. But when I do this I owe them. I need to track what cost I'm imposing them, and make sure it's not more than I have the time and opportunity to re-pay.)
Related: Citation form Sady Porn from Scott Alexanders's review
[...]
(I'm pretty sure the way to read Sady Porn (or Scott's review of it) is not to treat any of the text as evidence for anything, but as suggestions of things that may be interesting to pay attention to.)
4Nathan Helm-Burger
It's slightly off-topic... but I think it's worth mentioning that I think the reason 'extra credit' is more exciting to tackle is that it is all potential upside with no potential downside. Regular assignments and tests offer you a reward if you do them well, but a punishment if you make mistakes on them. Extra credit is an exception to this, where there is explicitly no punishment threatened, so you can just relax and make a comfortable effort without worrying that you are trying insufficiently hard. This makes it more like play and peaceful exploration, and less like a stressful trial.
Indeed, my opinion is that education should have much less in the way of graded homework and tests. Tests should be for special occasions, like the very end of a course, and intended to truly measure the student's understanding. Classwork (e.g. problem sets or working through a series on Khan academy or Brilliant) should award points for effort, just like lectures award participation credit for attending. Homework should be banned (below university level). If the school can't educate you within the allotted time, it is failing its job.
1mattmacdermott
Instead of tracking who is in debt to who, I think you should just track the extent to which you’re in a favouring-exchanging relationship with a given person. Less to remember and runs natively on your brain.
Thoughts on how to onboard volunteers/collaborators
Volunteers are super flaky, i.e. often abandon projects with no warning. I think the main reason for this is planning fallacy. People promise more than they actually have time for. Best case they will just be honest about this when they notice. But more typically the person will feel some mix of stress, shame and overwhelm that prevents them from communicating clearly. From the outside this looks like person promised to do a lot of things and then just ghosts the project. (This is a common pattern and not a specific accusation of any specific person.)
I don't think I'm particularly good at volunteer management. But I have a few methods that I find very useful to avoid having the volunteers be net negative from my perspective. Which one depends on what fits the project. When ever I don't do at least one of these precaution, I always end up regretting it, except if the person is someone I know well from before, and already have specific reasons to trust.
Give the volunteer a task which isn't essential, so that if they drop out, that is ok. E.g. if I run an event and someone want's to help then I would still make plans as if I did not h... (read more)
I've recently crossed into being considered senior enough as an organiser, such that people are asking me for advise on how to run their events. I'm enjoying giving out advise, and it also makes me reflet on event design in new ways.
I think there are two types of good events.
Purpose driven event design.
Unconference type events
I think there is a continuum between these two types, but also think that if you plot the best events along this continuum, you'll find a bimodal distribution.
Purpose driven event design
When you organise one of these, you plan a journey for your participant. Everything is woven into a specific goal that is active by the end of the event. Everything fits together.
These can defiantly have a purpose (e.g. exchanging of ideas) but the purpose will be less precise than for the previous type, and more importantly, the purpose does not strongly drive the event design.
There will be designed elements around the edges, e.g. the opening and ending. But most of the event design just goes into supporting the unconference structure, which is not very purpose specific. For ... (read more)
Thoughts on: When is it even possible (or likely) for single neurons to encode single concepts, based on model architecture
As part of my mech-interp research, I'm thinking a lot about the question "how would I do this if I was a neural network?". Specifically, given some toy model architecture, how can I set the weights so that it actives some task.
Todays conclusion is that single ReLU neurons are almost useless. To do identify any non-linear pattern (e.g. XOR), you need at least two ReLUs. We should not expect any feature to be represented by a single ReLU neuron, since anything that can be picked out by a single ReLU, was already linearly accessible in the first place, so why use the ReLU at all. Therefore, anything meaningful in a ReLU-MLP will use multiple neurons.
GELUs are a bit more expressive, i.e. not completely monotonic, but I don't think they are differens enough from ReLUs is enough to matter.
I have not though about SwiGLUs, and all the other ones MLP variants. I might get back to that, or share your thoughts in the comments.
Residual stream neurons will not be interpretable, for the simple reason that the residual stream has no privileged basis.
f"neuronpedia.org/gpt2-small/{layer}/{neuron}" is useful for getting a sense of what different LLM neurons are used for on a range of different texts.
2Linda Linsefors
I head that mono semanticity is having a comeback, so I recently had a look at a newer (non public) neuron database. It was not mono semantic.
But you are making some good points. Thanks.
3Alex Gibson
To be clear, I would be surprised if more than 2-3% of MLP neurons were monosemantic. It's just not so rare.
3Linda Linsefors
What would you say is the pattern?
To me it looks like a mix of cities, sex, food, politics and measurements. It doesn't look random but does look polysemantic.
I looked at "Full", which showed a lot more activations and variety of activations than "Snippet", including strong green ones. I don't know why this is, since I don't know how neuronpedia works.
8Alex Gibson
Tokens which occurred earlier in context window.
2Lucius Bushnaq
Could you explain that in more detail? I am looking at full example texts and failing to see a pattern like this. What is the relationship between the firing magnitude and the distance from the average position of previous occurrences?
2Linda Linsefors
I started lookin at what it doesn't react to, too.
It reacted to "evolution", "mating", "evolved", "genetics", "condoms", "erect penis" but not to "sex" or "sexuality".
It reacts to "intermittent fasting", "body fat", "eating" but not consistently or strongly to "food".
I feel a bit behind on everything going on in alignment, so for the next weeks (or more) I'll focus on catching up on what ever I find interesting. I'll be using my short form, to record my though.
I make no promises that reading this is worth anyone's time.
Linda's alignment reading adventures part 1
What to focus on?
I do have some opinions on what aliment directions are more or less promising. I'll probably venture in other directions too, but my main focus is going to be around what I expect an alignment solution to look like.
I think that to have an aligned AI it is necessary (but not sufficient) that we have shared abstractions/ontology/concepts/ (what ever you want to call it) with the AI.
I think the way to make progress on the above is to understand what ontology/concepts/abstraction our current AIs are using, and the process that shapes these abstraction.
I think the way to do this is though mech-interp, mixed with philosophising and theorising. Currently I think the mech-interp part (i.e. look at what is actually going on in a network) is the bottleneck, since I think that philosophising with out data (i.e. agent foundations) has not made much progress l... (read more)
You might enjoy Concept Algebra for (Score-Based) Text-Controlled Generative Models (and probably other papers / videos from Victor Veitch's groups), which tries to come up with something like a theoretical explanation for the linear represenation hypothesis, including some of the discussion in the reviews / rebuttals for the above paper, e.g.:
'Causal Separability The intuitive idea here is that the separability of factors of variation boils down to whether there are “non-ignorable” interactions in the structural equation model that generates the output from the latent factors of variation—hence the name. The formal definition 3.2 relaxes this causal requirement to distributional assumptions. We have added its causal interpretation in the camera ready version.
Application to Other Generative Models Ultimately, the results in the paper are about non-parametric representations (indeed, the results are about the structure of probability distributions directly!) The importance of diffusion models is that they non-parametrically model the conditional distribution, so that the score representation directly inherits the properties of the distribution.
To apply the results to other generative models, we must articulate the connection between the natural representations of these models (e.g., the residual stream in transformers) and the (estimated) conditional distributions. For autoregressive models like Parti, it’s not immediately clear how to do this. This is an exciting and important direction for future work!
(Very speculatively: models with finite dimensional representations are often trained with objective functions corresponding to log likelihoods of exponential family probability models, such that the natural finite dimensional representation corresponds to the natural parameter of the exponential family model. In exponential family models, the Stein score is exactly the inner product of the natural parameter with y. This weakly suggests that additive subspace
I think the main important lesson is to not get attached to early ideas. Instead of banning early ideas, if anything comes up, you can just write tit down, and set it aside. I find this easier than a full ban, because it's just an easier move to make for my brain.
(I have a similar problem with rationalist taboo. Don't ban words, instead require people to locally define their terms for the duration of the conversation. It solves the same problem, and it isn't a ban on though or speech.)
The other important lesson of the post, is that, in the early discussion, focus on increasing your shared understanding of the problem, rather than generating ideas. I.e. it's ok for ideas to come up (and when they do you save them for later). But generating ideas is not the goal in the beginning.
Hm, thinking about it, I think the mechanism of classical brainstorming (where you up front think of as many ideas as you can) is to exhaust all the trivial, easy to think of, ideas, as fast as you can, and then you're forced to think deeper to come up with new ideas. I guess that's another way to do it. But I think th... (read more)
There is nothing special about human level intelligence, unless you have imitation learning, in which case human level capabilities are very special.
General intelligence is not very efficient. Therefore there will not be any selection pressure for general intelligence as long as other options are available.
GI is very efficient, if you consider that you can reuse a lot machinery that you learn, rather than needing to relearn it over and over again. https://towardsdatascience.com/what-is-better-one-general-model-or-many-specialized-models-9500d9f8751d
1Linda Linsefors
Second reply. And this time I actually read the link.
I'm not suppressed by that result.
My original comment was a reaction to claims of the type [the best way to solve almost any task is to develop general intelligence, therefore there is a strong selection pressure to become generally intelligent]. I think this is wrong, but I have not yet figured out exactly what the correct view is.
But to use an analogy, it's something like this: In the example you gave, the AI get's better at the sub tasks by learning on a more general training set. It seems like general capabilities was useful. But consider that we just trained on even more data for a singel sub task, then wouldn't it develop general capabilities, since we just noticed that general capabilities was useful for that sub task. I was planing to say "no" but I notice that I do expect some transfer learning. I.e. if you train on just one of the dataset, I expect it to be bad at the other ones, but I also expect it to learn them quicker than without any pre-training.
I seem to expect that AI will develop general capabilities when training on rich enough data, i.e. almost any real world data. LLM is a central example of this.
I think my disagreement with at least my self from some years ago and probably some other people too (but I've been away a bit form the discourse so I'm not sure), is that I don't expect as much agentic long term planing as I used to expect.
1Linda Linsefors
I agree that eventually, at some level of trying to solve enough different types of tasks, GI will be efficient, in terms of how much machinery you need, but it will never be able to compete on speed.
Also, it's an open question what is "enough different types of tasks". Obviously, for a sufficient broad class of problems GI will be more efficient (in the sense clarified above). Equally obviously, for a sufficient narrow class of problems narrow capabilities will be more efficient.
Humans have GI to some extent, but we mostly don't use it. This is interesting. This means that a typical human environment is complex enough so that it's worth carrying around the hardware for GI. But even though we have it, it is evolutionary better to fall back at habits, or imitation, or instinkt, for most situations.
Looking back to exactly what I wrote, I said there will not be any selection pressure for GI as long as other options are available. I'm not super confident in this. But if I'm going to defend it here anyway by pointing out that "as long as other options are available", is doing a lot of the work here. Some problems are only solvable by noticing deep patterns in reality, and in this case a sufficiently deep NN with sufficient training will learn this, and that is GI.
3David Rein
I like that description of NFL!
2DragonGod
Re: your hot take on general intelligence, see: "Is General Intelligence Compact?"
LW draft editor keeps breaking for me in various ways. I need somewhere else that let's me collaborate on posts, and preferably also easily let's me copy over posts to LW.
Claud recommended using HackMD, so I'll try that. But I'm also interested in human recommendations for what editor to use.
Even now and then I meet someone who tries to argue that if I don't agree with them this is because I'm not open mided enough. Is there a term for this?
Epistemically I'm not convinced buy this type of arugment, but socialy it feels like I'm beeing shamed, and I hate it.
I also find it hard to call out this type of behaviur when it happens, even when I can tell exactly what is going on. I think it I had a name for this behaviour it would be easier? Not sure though?
Edit to add:
I've now got some more time to figure out what I want and don't want out of this thread. The early responses helped with this, so thanks!
What I'm most interested in is a name for this behaviour. Naming it helps in at least two ways. It makes it easier to call out in the moment (as mentioned above), but it also makes it easer for me to handle internaly. I can be like "ah, it's this thing again" in my head, rather than being overwelmed.
What I'm not interested is in, is any advice/suggestions that continues the conversation. After a person have pulled one of these moves on me, I am both angry at them, and do not trust them to cooporate in a any form of good faith conversation.
If you have some ideas for how I can end the conversation that does not feel uterly humiliating to me, please tell me. Anything that is phrased like a question is out. I do not want to heare what they have to say, and asking quiestions that you don't want answers to, is wrong and bad.
Seems like a subtype of Bulverism; not aware of a more specific term.
[...]
Assuming you have a LWer-typical level of atypicality, you could say "I literally do/believe [outlandish but politically-neutral activity/opinion], there's no way closed-mindedness is my problem." (If it were me, I'd use donating to Shrimp Welfare; apparently most people think that's strange, for some reason.)
6Linda Linsefors
I feel like if I try to defend my openmindedness I loose. It just opens up more attac surfaces to someone who is hostile and doesn't argues in good faith.
I think it's much better to call out why calling someone close minded for not listening is just invalid in general, not just this time in particular. And I do believe it is.
If someone isn't listening to you. Them being to close minded is so faaaaaar the list of most likely explanation that. Much less likely than:
* Your argument are bad
* Dissintrest in the topic
* Other things they rather do right now
2Viliam
Yeah. The opponent's move is establishing "you can change your mind and adopt my opinion" as the criterion to measure your open-mindedness. If you can switch to their position, you are allowed to keep calling yourself open-minded; if you don't, you lose that right.
(The rational answer would be: yes, you are capable of adopting any position, including theirs, when there is a good reason to do so. Open-mindedness does not mean adopting random positions for no good reason, or simply because someone calls you a chicken if you don't.)
So the essence of the move is "I decide what is the true (costly) signal of the trait you claim to have". And the choice of the signal is obviously self-serving. I mean, in theory, they could have asked you to demonstrate your open-mindedness by adoption a position they don't agree with; that would be an equally valid proof of your ability to change your mind. But of course there is no incentive for them to do so. Which shows that evaluating your open-mindedness impartially was never the true goal here.
Another thing is that there is a difference between "being open-minded" and "signaling open-mindedness". Just because you are capable of adopting various kinds of positions, doesn't mean that you should. To consider options X, Y, Z, and afterwards decide to stay with the original X, can be perfectly open-minded, even if from outside it may be difficult to distinguish from "the person did not consider Y and Z at all". (It's like when a gifted child solves a mathematical problem too fast, and the teacher accuses them of merely guessing the answer. There is a difference between doing the work, and demonstrating to other people that you did the work.)
Shortly:
* open-mindedness does not mean "doing what you want me to do"; that's called social pressure
* it is perfectly open-minded to consider a hypothesis... and then reject it
2Linda Linsefors
Bulverism is a good term, thanks!
4cubefox
A more transparent term would be psychologizing:
[...]
See also Ayn Rand on this topic:
[...]
(Lots more ranting about psychologizers. Schopenhauer energy.)
6Vladimir_Nesov
Under the belief vs. understanding distinction, open-mindedness is a virtue of understanding ideas you disbelieve (or purposes you don't endorse). It's not directly relevant to belief, but sometimes understanding is the bottleneck to belief, in which case more open-mindedness would help. When you already understand the idea, open-mindedness is no longer relevant.
More to the point, understanding an idea shouldn't necessarily result in believing it, and high open-mindedness doesn't increase the number of hours in a day. Learning any given piece of nonsense would still not be the best thing to focus on, but high open-mindedness should prevent you from actively avoiding low-hanging fruit of understanding when it's ripe for the taking, just because it doesn't seem to be the kind of thing you are likely to believe or endorse.
5james oofou
Them: "I think X"
You: "That's wrong because Z"
Them: "I think you're just disagreeing because you'd not open-minded enough"
You: "What makes you think that?"
Them: "I think it because Y"
What do they say for 'Y'? That seems the part that actually constitutes their argument and which you will be able to call out if they're making a mistake.
3Gurkenglas
when I put myself in the shoes of one who would say that someone is not open-minded enough and fill in the Y I get "you responded so quickly that you must have been speaking from cache instead of computing an answer"
2Steven Byrnes
One thing you can maybe do is throw such accusations right back: “You say I’m being closed-minded to you, but aren’t you equally being closed-minded to me?”
It comes across as escalatory, and might be counterproductive, but I’ve also sometimes found it helpful. Depends a lot on the person and situation.
4Linda Linsefors
I don't want to use this sugestion, not because it is escalatory, but because it's a question, which invites them to have more opinions.
What I want is a way out, but that has the feeling of standing up for myself, rather then the feeling of humiliation and defeet.
If someone starts to accuse me of not beeing openminded to their opinion, it's usually because I think their opinion is dumb. I rather not hurt their feelings if I can avoid it, but I'm also not going to worry too much about being polite to someone after they done this particular retorical move.
Usually the way out is to just leave. But last time this happened was at a small metup, and the only way out was to leave the event, which I did. I'm not happy about this and would like better options.
3Dagon
A lot depends on why you want to stay, and whether you are likely to have interesting/productive further discussions. I often just let them off the hook with a "I guess I am a bit closed on this particular topic - let's discuss something else".
In theory "agree to disagree" should be impossible among rational entities with compatible priors, but in practice humans often meet neither of those criteria. That's OK, when you reach the point of no further updates in either direction, you move on.
2Linda Linsefors
I wish I could say something like that and be ok. But to me it feels too humiliating. And also often factually wrong, I.e. I'd be open to good argument.
2Viliam
A reply I got in a similar situation, paraphrased: "well, it's you who identifies as a rationalist, you hypocrite."
In other words, as if by calling myself a rationalist (not in that specific debate, just generally something I said in a different context that my opponent knows about) means that I accept an asymmetrical burden.
You mean, in that you can simply prompt for a reasonable non-infinite performance and get said outcome?
1Linda Linsefors
Similar but not exactly.
I mean that you take some known distribution (the training distribution) as a starting point. But when sampling actions you do so from shifted on truncated distribution to favour higher reward policies.
The in the decision transformers I linked, AI is playing a variety of different games, where the programmers might not know what a good future reward value would be. So they let the system AI predict the future reward, but with the distribution shifted towards higher rewards.
I discussed this a bit more after posting the above comment, and there is something I want to add about the comparison.
In quantilizers if you know the probability of DOOM from the base distribution, you get an upper bound on DOOM for the quantaizer. This is not the case for type of probability shift used for the linked decision transformer.
DOOM = Unforeseen catastrophic outcome. Would not be labelled as very bad by the AI's reward function but is in reality VERY BAD.
1Jobst Heitzig
From my reading of quantilizers, they might still choose "near-optimal" actions, just only with a small probability. Whereas a system based on decision transformers (possibly combined with a LLM) could be designed that we could then simply tell to "make me a tea of this quantity and quality within this time and with this probability" and it would attempt to do just that, without trying to make more or better tea or faster or with higher probability.
1Linda Linsefors
Yes, that is a thing you can do with decision transforms too. I was referring to variant of the decision transformer (see link in original short form) where the AI samples the reward it's aiming for.
"Fake cookies", i.e. sugar free cookies with artificial sweetener, are great
(Write-up longer than necessary because I'm not spending enough effort making it short)
I like sweets, but if I have too much sugar I get short-term addicted. I.e. I get sugar cravings, and my mood is worse. I can get over it (and have done so many times) by avoiding sweets for a few days but it's unpleasant.
In the past my main strategy was to avoid getting into the addicted state, by not having enough sweets to trigger it. I can have some, just not too much. Eating fruits also helps, since this some what satisfy the craving without reinforcing it. I.e, if I have sweets, it's easer to stop and less likely to trigger the addiction, if I end the "meal" with a fruit.
This hasn't been enough of a problem for me to put much agency into solving it. Or maybe it's more correct to say that it's only been a noticeable problem at times when I don't have spare agency to throw at it. E.g. if I'm feeling down for other reasons, it would be nice to use sweets as mood enhancer, but also if I'm already down then don't want to have to add dealing with sugar cravings on top of what else is going on. In this situation, me being... (read more)
Yesterday was the official application deadline for leading a project at the next AISC. This means that we just got a whole host of project proposals.
If you're interested in giving feedback and advise to our new research leads, let me know. If I trust your judgment, I'll onboard you as an AISC advisor.
Also, it's still possible to send us a late AISC project proposals. However we will prioritise people how applied in time when giving support and feedback. Further more, we'll prioritise less late applications over more late applications.
I sometimes model myself as having sub-agent, and find this perspective very helpful for self therapy. On the other hand, I don't believe that I literally have individual independent sub agents in my brain, because agent architecture is actually complicated.
I like Steve Byrnes mind model, because it's an actual plausible model for how a real world agentic mind might work. It's also the only such model I know about.
Steve's model is not a multi agent model, and I can't think of a multi agent model that works.
Can I explain my own and others sub-agent experie... (read more)
If it helps, I have a brief take on “subagent” terminology in §1.5.2 here.
[...]
If the idea is that there’s something secretly motivating about the idea of failing (in some context), and that it’s helpful to bring this secret motivation up to conscious awareness and consideration, then yeah, definitely, and that sounds related to David Burns “positive reframing” (1,2) and related ideas in other psychiatry traditions I’m less familiar with (3).
2Lorxus
You might find it helpful to read one of my Budget Inkhaven submissions, then - I think in depth about a parliamentary model of the self, not just simple IFS or shard theory. https://tiled-with-pentagons.blogspot.com/2026/04/82-deadlock-in-parliament-of-self.html
2Linda Linsefors
The post is a good explanation on multi agent model of mind. I've used this frame and agree that it's useful. But that's not the problem I'm trying to adress. The problem is that this can't possibly be the ground truth. Neither do I find it likely that this model a correct explanation at some middle level of reduction.
I've seen people in shard theory, and most lately this, where someone takes the subagent perspective to be way to fundamental.
I claim that you have conflicting goals, and for some people it's sometimes useful to model yourself as being made up of conflicting sub agents. But you aren't literally made up of different sub agent. The muli agent model of mind is probably good enough for self therapy, but it's not good enough for alignment research.
The problem with taking this model too seriously (and not just as an useful tool) is that it has the homunculus problem. You still have to explain how agency works. Turns out minds are complicated! And when you start thinking about how to fit even one functional agent in a brain, you realize that there are not going to be lot's of them.
My shortform was about how come, even though there is no parliament, this is style of self therapy works anyway.
1Lorxus
No, I totally agree the post's frame isn't a ground truth - IIRC I say so up front. I don't think I agree that it can't possibly be correct at some middle level of reduction, but I agree that that sense lacks firm grounding. I hadn't been clear that you were thinking about a frame for understanding agency in generality; I was however trying to contribute more thinking to the self-therapy ideas you lay out in the shortform.
2Linda Linsefors
Oh, I guess I'm less clear that I though. My goal is not to suggest self therapy ideas. I'm naming them as evidence of multi agent of mind, that has to be explained some other way if there are in fact no multi agents.
2Linda Linsefors
You probably wrote this comment before I was done writing?
Blogposts are the result of noticing difference in beliefs. Either between you and other of between you and you, across time.
I have lots of ideas that I don't communicate. Sometimes I read a blogpost and think "yea I knew that, why didn't I write this". And the answer is that I did not have an imagined audience.
My blogposts almost always span after I explained a thing ~3 times in meat space. Generalizing from these conversations I form an imagined audience which is some combination of the ~3 people I talked to. And then I can write.
Recently someone either suggested to me (or maybe told me they or someone where going to do this?) that we should train AI on legal texts, to teach it human values. Ignoring the technical problem of how to do this, I'm pretty sure legal text are not the right training data. But at the time, I could not clearly put into words why. Todays SMBC explains this for me:
Law is not a good representation or explanation of most of what we care about, because it's not trying to be. Law is mainly focused on the c... (read more)
Moreover, legal texts are not super strict (much is left to interpretation) and we are often selective about "whether it makes sense to apply this law in this context" for reasons not very different from religious people being very selective about following the laws of their holy books.
1gw
I spoke with some people last fall who were planning to do this, perhaps it's the same people. I think the idea (at least, as stated) was to commercialize regulatory software to fund some alignment work. At the time, they were going by Nomos AI, and it looks like they've since renamed to Norm AI.
2Linda Linsefors
I found this on their website
[...]
I'm not sure if this is worrying, because I don't think AI overseeing AI is a good solution. Or it's actually good, because, again, not a good solution, which might lead to some early warnings?
2Linda Linsefors
Their more human-in-the-loop stuff seems neat though.
1CstineSublime
Would sensationalist tabloid news stories be better training data? Perhaps it is the inverse problem: fluffy human interest stories and outrage porn are both engineered for the lowest common denominator, the things that overwhelmingly people think are heartwarming or miscarriages of justice respectively. However if you wanted to get a AI to internalize what is in fact the sources of outrage and consensus among the wider community I think it's a place to start.
The obvious other examples are fairy tales, fables, parables, jokes, and urban legends - most are purpose encoded with a given society's values. Amateur book and film reviews are potentially another source of material that displays human values in that whether someone is satisfied with the ending or not (did the villain get punished? did the protagonist get justice?) or which characters they liked or disliked is often attached to the reader/viewer's value systems. Or as Jerry Lewis put it in the Total Filmmaker: in comedy, a snowball is never thrown at a battered fedora: "The top-hat owner is always the bank president who holds mortgage on the house...".
2Linda Linsefors
Sensationalist tabloid news stories and other outrage porn are not the opposite. These are actually more of the same. More edge cases. Anything that is divisive have the problem I'm talking about.
Fiction is a better choice.
Or even just completely ordinary every-day human behaviour. Most humans are mostly nice most of the time.
We might have to start with the very basic, the stuff we don't even notice, because it's too obvious. Things no-one would think of writing down.
2CstineSublime
Could you explain how are they edge cases if they are the lowest common denominator? Doesn't that make them the opposite of an edge case? Aren't they in fact the standard or yardstick necessary to compare against?
[...]
Why is is it different let alone better choice? Fiction is a single author's attempt to express their view of the world, including morality, and therefore an edge case. While popular literature is just as common denominator as tabloid journalism, since the author is trying to be commercial.
2Linda Linsefors
I don't read much sensationalist tabloid, but my impression is that the things that get a lot of attention in the press, is things people can reasonable take either side of.
Scott Alexander writes about how everyone agrees that factory framing is terrible, but exactly because this overwhelming agreement, it get's no attention. Which is why PETA does outrageous things to get attention.
The Toxoplasma Of Rage | Slate Star Codex
There need to be two sides to an issue, or else no-one gets ingroup loyalty points for taking one side or the other.
3CstineSublime
A cursory glance suggests that it is not the case, take a top story headline on the Australian Daily Mail over the last 7 days: "Miranda, Sydney: Urgent search is launched for missing Bailey Wolf, aged two, who vanished yesterday" it is not reasonable for someone to hope that a two year old who has vanished not be found. This is exactly the kind of thing you're suggesting AI should be trained on, because of how uniform responses are to this headline. Keep in mind this is one of the most viewed stories, and literally top of the list I found.
I've read Scott's article, but are you trying to understand what get's attention or what is the nexus or commonly agreed upon moral principles of a society?
3Linda Linsefors
Ok, you're right that this is a very morally clear story. My bad for not knowing what's typical tabloid storry.
Missing kid = bad,
seems like a good lesson for AI to learn.
About once or twice per week this time of year someone emails me to ask:
Please let me break rule X
My response:
No you're not allowed to break rule X. But here's a loop hole that lets you do the thing you want without technically breaking the rule. Be warned that I think using the loophole is a bad idea, but if you still want to, we will not stop you.
Because not leaving the loophole would be too restrictive for other reason, and I'm not going to not tell people all their options.
The fact that this puts the responsib... (read more)
I suspect it's not possible to build autonomous aligned AIs (low confidence). The best we can do is some type of hybrid humans-in-the-loop system. Such a system will be powerful enough to eventually give us everything we want, but it will also be much slower and intellectually inferior to what is possible with out humans-in-the-loop. I.e. the alignment tax will be enormous. The only way the safe system can compete, is by not building the unsafe system.
Therefore we need AI Governance. Fortunately, political action is getting a lo... (read more)
If LMs reads each others text we can get LM-memetics. A LM meme is a pattern which, if it exists in the training data, the LM will output at higher frequency that in the training data. If the meme is strong enough and LLMs are trained on enough text from other LMs, the prevalence of the meme can grow exponentially. This has not happened yet.
There can also be memes that has a more complicated life cycle, involving both humans and LMs. If the LM output a pattern that humans are extra interested in, then the humans ... (read more)
This mechanism may not require LMs to be involved.
3Linda Linsefors
Not sure what you mean exactly. But yes, memetics without AI does exist.
https://en.wikipedia.org/wiki/Memetics
1Mateusz Bagiński
I think an LM-meme is something more than just a frequently repeating pattern. More like frequently repeating patterns with which can infect each other by outputting them into the web or whatever can be included as in a trainibg set for LMs.
There may be other features that are pretty central to the prototype of the (human) meme concept, such as its usefulness for some purpose (ofc not all memes are useful). Maybe this one can be extrapolated to the LM domain, e.g. it helps it presict the next token ir whatever but I'm not sure whether it's the right move to appropriate the concept of meme for LMs. If we start discovering infectious patterns of this kind, it may be better to think about them as one more subcategory of a general category of replicators of which memes, genes, and prions are another ones.
Problem 1: I've been having trouble finding a mattress I like. Most of them are too hard. I like soft. We currently have a memory foam mattress topper, that is very soft and nice, but it has too good memory, so there is a significant dipp in the middle of where I sleep, making me sleep all bent. We got a new one, and it only took months for the problem to reappear. I could try more combinations of mattresses and mattress toppers, but that would cost too much time and money, since just seeing what I like in the sh... (read more)
Re: Problem 2
My pillow has some adjustable height thin cushions. I feel like something similar must be purchasable separately, "pillow booster" might be the right term but I'm not sure.
https://www.amazon.com/LOFE-Adjustable-Memory-Pillows-Layer/dp/B0C2TF6Q1M?th=1
2MichaelDickens
I think because body shapes too much. Even if two people are the same height (which most people aren't), they will still probably have their shoulders and hips and knees spaced differently. A contoured mattress would therefore have the contours in the wrong places for almost everyone, which would be even worse than a flat mattress.
But like you, I would also like a mattress with contours shaped to my body.
Estimated MSE loss for three diffrent ways of embedding features into neuons, when there are more possible features than neurons.
I've typed up some math notes for how much MSE loss we should expect for random embedings, and some other alternative embedings, for when you have more features than neurons. I don't have a good sense for how ledgeble this is to anyone but me.
Note that neither of these embedings are optimal. I belive that the optimal embeding for minimising MSE loss is to store the features in almost orthogonal directions, which is similar to ran... (read more)
This is probably too obvious to write, but I'm going to say it anyway. It's my short form, and approximately no-one reads short forms. Or so I'm told.
Human value formation is to a large part steered by other humans suggesting value systems for you. You get some hard to interpret reward signal from your brainstem, or something. There are lots of "hypothesis" for the "correct reward function" you should learn.
(Quotation marks because there are no ground through for what values you should have. But this is mathematically equivalent to a learning the tru... (read more)
Somebody is reading shortforms...
I disagree. That humans learn values primarily via teaching. 1) parenting is known to have little effect on children's character - which is one way of saying their values. 2) while children learn to follow rules teens are good at figuring out what is in their interest.
I think it makes sense to pose argue the point though.
For example I think that proposing rules makes it more probable that the brain converges on these solutions.
5Linda Linsefors
This is not counter evidence to my claim. The value framework a child learns about from their parents is just one of many value frameworks they hear about from many, many people. My claim is that the power lies in noticing the hypothesis at all. Which ideas you get told more times (e.g. by your parents) don't matter.
As far as I know, what culture you are in very much influences your values, which my claim would predict.
[...]
I'm not making any claims about rule following.
I'm basically ready to announce the next Technical AI Safety Unconference (TAISU). But I have hit a bit of decision paralysis as to what dates it should be.
If you are reasonably interested in attending, please help me by filling in this doodle
When I'm in a conversation often track who the conversation is for. I.e. who is this conversation primerely seving, in this moment.
If I'm ranting, then this conversation is for me, to let me realsease some tension. I will express my self in a way that feels good to me.
If I'm sharing usefull information, then the conversation is for the other person. I will express my self in a way to make the information clear and accessable for them, and also pay attetion to if they even want this information.
I can explain myself becasue I need to be seen, or becasue ... (read more)
Some conversations should be primarily about an object level thing, for its own elucidation (they serve the idea itself, bringing it into the world). A person can have motivations that are not about (emotions of) people (including that person themselves).
A good explanation constructs an understanding in its audience, which is slightly different from describing something, or from making it accessible.
2Linda Linsefors
I just checked in with myself what the post above was for. I tink its part rant, part me clarifying my thoughs by writing them, and hopefully getting some reflections back. And it's also becasue maybe someone will find it usefull, but that's also maybe secretly about me, to create more conversation partners that track the things I think is important.
If I was writing a proper LW blogpost then [who is this for] should primarlely be the reader.
But in a shortform like this I feel like I'm allowed to do what I want. And also people can take away what they want. Tracking [who is this for] is much more important when people are in a live conversations, becasue that is a more trapped situation, requiring more concideration.
2Linda Linsefors
There are also the type of conversation where the other person pretends that it is about me, but acctually it is about their need to feel like a good person. These situatios are afull and terrible, and I will not play along.
2Mateusz Bagiński
I usually don't, though maybe unconsciously? Plausibly it would be good for me to try to track it explicitly.
cf https://www.lesswrong.com/posts/bhLxWTkRc8GXunFcB/what-are-you-tracking-in-your-head
2Linda Linsefors
Thanks for the link. I have read it but it was long ago, so thanks for the reminder. It's related in a helpfull way.
This already strongly suggests some connection between induction heads and in-context learning, but beyond just that, it appears this window is a pivotal point for the training process in general: whatever's occurring is visible as a bump on the training curve (figure below). It is in fact the only place in training where the loss is not convex (monotonically decreasing in slope).
I can see the bump, but it's not the only one. The two layer graph has a second similar bump, which a... (read more)
Toy model: Each agent has a utility function they want to maximise. The input to the utility function is a list of values describing the state of the world. Different agents can have different input vectors. Assume that every utility function monotonically increases, decreases or stays constant for changes in each impute variable (I did say it was a toy model!). An agent is said to value something if the utility function increases with increasing quantity of that thing. Note that if an agents utility function decrease... (read more)
Re second try: what would make a high-level operationalisation of that sort helpful? (operationalize the helpfulness of an operationalisation)
1Linda Linsefors
This is a good question.
The not so operationalized answer is that a good operationalization is one that are helpful for achieving alignment.
An operationalization of [helpfulness of an operationalization] would give some sorts to gears level understanding of what shape the operationalization should have to be helpful. I don't have any good model for this, so I will just gesture vaguely.
I think that mathematical descriptions are good, since they are more precise. My first operationalization attempt is pretty mathematical which is good. It is also more "constructive" (not sure if this is the exact right word), i.e. it describes alignment in terms of internal properties, rather than outcomes. Internal properties are more useful as design guidelines, as long as they are correct. The big problem with my first operationalization is that it don't actually point to what we want.
The problem with the second attempt is that it just states what outcome we want. There is nothing in there to help us achieve it.
6Mateusz Bagiński
Can't you restate the second one as the relationship between two utility functions UA and UB such that increasing one (holding background conditions constant) is guaranteed not to decrease the other? I.e. their respective derivatives are always non-negative for every background condition.
∂UA∂UB≥0∧∂UB∂UA≥0
1Linda Linsefors
Yes, I like this one. We don't want the AI to find a way to give it self utility while making things worse for us. And if we are trying to make things better for us, we don't want the AI to resist us.
Do you want to find out what these inequalities implies about the utility functions? Can you find examples where your condition is true for non-identical functions?
1Mateusz Bagiński
I don't have a specific example right now but some things that come to mind:
* Both utility functions ultimately depend in some way on a subset of background conditions, i.e. the world state
* The world state influences the utility functions through latent variables in the agents' world models, to which they are inputs.
* UA changes only when MA (A's world model) changes which is ultimately caused by new observations, i.e. changes in the world state (let's assume that both A and B perceive the world quite accurately).
* If whenever UA changes UB doesn't decrease, then whatever change in the world increased UA, B at least doesn't care. This is problematic when A and B need the same scarce resources (instrumental convergence etc). It could be satisfied if they were both satisficers or bounded agents inhabiting significantly disjoint niches.
* A robust solution seems seems to be to make (super accurately modeled) UB a major input to UA.
2Linda Linsefors
Lets say that
U_A = 3x + y
Then (I think) for your inequality to hold, it must be that
U_B = f(3x+y), where f' >= 0
If U_B care about x and y in any other proportion, then B can make trade-offs between x and y which makes things better for B, but worse for A.
This will be true (in theory) even if both A and B are satisfisers. You can see this by assuming replacing y and x with sigmoids of some other variables.
1) Gradient hacking makes the model less corrigible. However corrigibility is only useful if we know how to do alignment, which we don't, so at current alignment tech level, this does not matter.
2) Our current best training signal, if optimized for with out any gradient hacking, will lead the AI towards sycophancy and plain reward hacking. This has a 100% chance of going wrong. On the other... (read more)
I remember reading that some activation stearing experiments used x100 size activation vectors. Is this correct? That would be much larger than the normal activation in that layer, right?
How does the model deal with this? If there is any superposition going on, I expect activation spill over everywhere, breaking everything.
If you amplify the activations in one layer, what effect has that on the magnitude of the activations in the next layer? If it's smaller, by what mechanism? That mechanism is probably an error correction algorithm. Or just some suppressi... (read more)
One issue is that after the typo is fixed, the react remains, but will either point to nothing or point to a place where there is not a typo (any more).
1Karl Krueger
The highlight is pretty subtle, at least as it appears to me on the above comment. Screenshot here.
Recently an AI safety researcher complained to me about some interaction they had with an AI Safety communicator. Very stylized, there interaction went something like this:
(X is some fact or topic related to AI Safety
Communicator: We don't know anything about X and there is currently no research on X.
Researcher: Actually, I'm working on X, and I do know some things about X.
Communicator: We don't know anything about X and there is currently no research on X.
I notice that I semi-frequently hear communicators saying things like the thing above. I think ... (read more)
I notice that I don't expect FOOM like RSI, because I don't expect we'll get an mesa optimizer with coherent goals. It's not hard to give the outer optimiser (e.g. gradient decent) a coherent goal. For the outer optimiser to have a coherent goal is the default. But I don't expect that to translate to the inner optimiser. The inner optimiser will just have a bunch of heuristics and proxi-goals, and not be very coherent, just like humans.
The outer optimiser can't FOOM, since it don't do planing, and don't have strategic s... (read more)
I recently updated how I view the alignment problem. The post that caused my update is this one form the shard sequence. Also worth mentioning is older post that points to the same thing, but I just happen to read it later.
Basically I used to think we needed to solve both outer and inner alignment separately. No I no longer think this is a good decomposition of the problem.
It’s not obvious that alignment must factor in the way described above. There is room for trying to set up training in such a way to guarantee a friendly mesa-objective somehow wit... (read more)
Any policy can be model as a consequentialist agent, if you assume a contrived enough utility function. This statement is true, but not helpful.
The reason we care about the concept agency, is because there are certain things we expect from consequentialist agents, e.g. instrumental convergent goals, or just optimisation pressure in some consistent direction. We care about the concept of agency because it holds some predictive power.
[... some steps of reasoning I don't know yet how to explain ...]
Therefore, it's better to use a concept of agency that ... (read more)
1
2
2
2
1
1
1
1
5
4
2
2
1
1