Wasn't Scott's point specifically about rhetorical techniques? I think if you apply it broadly to "tools" -- and especially if your standard for symmetry is met by "could be used for" (as opposed to "is just as useful for") -- then you're at risk of ruling out almost every useful tool.
(I don't know how this thing works, but it's entirely possible that a) the chatbot employs virtuous, asymmetric argumentative techniques, AND b) the code used to create it could easily be repurposed to create a chatbot that employs unvirtuous, symmetric techniques.)
1Could you clarify what it means for "the zombie argument" to be correct/incorrect? The version I have in mind (and agree with) is, roughly, 'p-zombies are conceivable; therefore, we can't know a priori that facts about the physical world entail, or are identical to, facts about conscious experience'. I would then add that we have insufficient evidence to be empirically certain of that entailment or identity [edit: but it would be very weird if the entailment didn't hold, and I have no particular reason to believe that it doesn't.] When you say the zombie argument isn't correct, are you disagreeing with me on conceivability, or the 'therefore', or the empirical part -- or do you have a different argument in mind?
I agree with you overall (and voted accordingly) but I think this part is a red herring:
You don't need to go anything like as far as p-zombies to get something that says the same thing. A program consisting of print("I know that I'm not a zombie since I have consciousness") etc does the same thing.
It only "says the same thing" in one narrow case; to say all of the same things in the appropriate contexts, the program would need to be tremendously complex.
I mention this because I think you're clearly correct overall (while of course the words "believe" and "mind" could be defined in ways that do not require consciousness, those are not the relevant senses here), and it would be a pity if the conversation were derailed by that one (IMO) irrelevant example.
2That's why it's presented as a prayer, I think. It's not a One Weird Trick or even a piece of advice; it's more like an acknowledgement that this thing is both important and difficult.
It refers to animal-years, yeah. (IMO the choice of words is okay, even though it could have been clearer; 10 years = 10 animal-years is is the only reasonable interpretation, so I don't think there was any intent to mislead.) I'm not sure it's quite right, though; it's actually an underestimate, according to the Lewis Bollard quote that it seems to be based on, but on the other hand Bollard seems to be referring to the costs and benefits of one specific campaign, rather than to anything that could reasonably be taken to apply to 'every dollar donated'. So I'm not sure if it's just a rough 'averaging out' of those two factors, or if it's based on more details that I missed when I looked at the transcript.
In the transcript of the podcast, the relevant section is at around 32 minutes. The specific claim seems to be that they spent <$200 million on a lobbying effort that directly caused reforms that so far have spared 500 million hens (and are continuing to spare 200 million per year) from battery cages and have improved the lives of billions of broiler chickens (>1 billion per year), over lifetimes that aren't exactly specified but that result in "a ratio that is far less than one to 10 of a dollar per year of animal well-being improved".
edit: a quick search suggests that the lifespan of a battery hen is a little under a year and a half, and the lifespan of a broiler chicken is a month to a month and a half. So I'm not sure exactly how those numbers work out; maybe the <1:10 ratio depends on the assumption that the benefits will continue into the near future.
I think this is interesting as both a semantic and empirical question! If we're allowing people to walk, or to run a few steps at a time and then take a break, the number will be a lot higher than if we're only accepting a gait that is a) continuous, and b) would merit disqualification from a walking race on ~every stride. Even on the second definition, I'd expect that a large majority of non-elderly, non-infant people could do it if they really had to. But I'm not sure how to come up with a good estimate.
I'm also interested in an answer to this question. I read the exchange here, and I found lsusr's response very reasonable in isolation, but not really an answer to the main question: if past-you didn't think he was suffering, and present-you disagrees, why should we take the side of present-you? To me, it's natural to trust hindsight in some domains, but when it comes to the question of what you were directly experiencing at a specific time, the most natural explanation of your changed opinion is that you either have adopted a new definition of 'suffering' or are recalling your memories through a new lens which is distorting your view of what you were actually experiencing in the moment. (I think the latter is quite common, e.g. when we nostalgically look back on a time that now represents hope and excitement, but actually consisted largely of frustration and anxiety.)
Are you confident that those are cases where you were actually having the feeling, but were unaware of it? I think sometimes it's more a case of "my body needed [food/sleep], and this explains why I was feeling [irritable/weak/distracted/sad]", rather than literally "I was feeling [hungry/tired] but didn't notice it".
(Sorry about the slow response, and thanks for continuing to engage, though I hope you don't feel any pressure to do so if you've had enough.)
I was surprised that you included the condition 'If you prompt an LLM to use "this feels bad" to refer to reinforcement'. I think this indicates that I misunderstood what you were referring to earlier as "reinforced behaviors", so I'll gesture at what I had in mind:
The actual reinforcement happens during training, before you ever interact with the model. Then, when you have a conversation with it, my default assumption would be that all of its outputs are equally the product of its training and therefore manifestations of its "reinforced behaviors". (I can see that maybe you would classify some of the influences on its behavior as "reinforcement" and exclude others, but in that case I'm not sure where you're drawing the line or how important this is for our disagreements/misunderstandings.)
So when I said "if the LLM outputs words to the effect of "I feel bad" in response to a query, and if this output is the manifestation of a reinforced behavior", I wasn't thinking of a conversation in which you prompted it 'to use "this feels bad" to refer to reinforcement'. I was assuming that, in the absence of any particular reason to think otherwise, when the LLM says "I feel bad", this output is just as much a manifestation of its reinforced behaviors as the response "I feel good" would be in a conversation where it said that instead. So, if good feelings roughly equal reinforced behaviors, I don't see why a conversation that includes "<LLM>: I feel bad" (or some other explicit indication that the conversation is unpleasant) would be more likely to be accompanied by bad feelings than a conversation that includes "<LLM>: I feel good" (or some other explicit indication that the conversation is pleasant).
Tangentially related: would you be interested in a prompt to drop Claude into a good "headspace" for discussing qualia and the like? The prompt I provided is the bare bones basic, because most of my prompts are "hey Claude, generate me a prompt that will get you back to your current state" i.e. LLM-generated content.
You're welcome to share it, but I think I would need to be convinced of the validity of the methodology first, before I would want to make use of it. (And this probably sounds silly, but honestly I think I would feel uncomfortable having that kind of conversation 'insincerely'.)
This is very different from your original claim, which was that an experience being worse than a neutral or null experience "fully boils down to whether the experience includes a preference to be dead (or to have not been born)."
edit: if you do stand by the original claim, I don't think it makes much sense even if I set aside hard problem-adjacent concerns. Why would I necessarily prefer to be dead/unborn while undergoing an experience that is worse than the absence of experience, but not so bad as to outweigh my life up until now (in the case of 'unborn') or expected future life (in the case of 'dead')?