DanielW's Shortform

DanielW

DanielW's Shortform — LessWrong

DanielW's Shortform

14th Mar 2026

1 min read

1

This is a special post for quick takes by DanielW. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

84 comments, sorted by

top scoring

Click to highlight new comments since: Today at 1:01 PM

[-]DanielW1mo190

It is brought up quite often, but I have not (yet!) heard a satisfactory answer to the question: "why are so many leading people in advocating AI risk also investing in developing LLMs and AI-dependent ecosystem?"

It seems to me, the position (in simple form) holds:

Divesting from a technology discourages it's development/advancement n aggregate (investing in a technology encourages it)
Therefore, if the development/advancement of a technology is, in aggregate, undesirable, one should be divest from it and minimize any investment (not withstanding benefits that are greater than the risks of development)

This has been the approach to things like fossil fuels, which i believe is correct. Yes, individual use has minimal impact, and what matters is the aggregate but individual divestment impacts the aggregate and makes development less profitable and alternatives more desirable. Of course, sometimes fossil fuel usage is the best viable option for some technologies and the goal isn't to eliminate it completely but rather to drive up the costs and decrease the profits to the industry so their use will be restricted to areas where the benefits are far in excess than the risk and development ... (read more)

[-]Neel Nanda1mo1617

The effect size of divestment just seems way too small to matter to me - AI is the current biggest frenzy among investors, data centres are holding up US GDP, it's making trillion dollar companies, some of the fastest growing consumer products ever, etc. And there's substantial productivity advantages to people using AI. I don't feel like a plan involving divestment and boycotting will work and I would be sad to see safety orgs following it rather than things I thought could meaningfully reduce x risk.

3DanielW1mo

They are not either-or, rather obviously. It is much easier to get regulation passed and get firms to sign on to agreements when something is unprofitable and unpopular. It is very hard to get them to sign on when it is profitable. I am not even saying they should be focusing on divestment and boycotting campaigns, I am saying the engaging in investment and promotion campaigns seems rather non-sensical. Like, I think it is fine that a lot of climate activists don't do the absolute utmost to eliminate their fossil fuel usage and exposure to it, but if they went around promoting contests for like 'best steak made on a coal-fired grill' and various uses of coal power, I would look at them like wtf are you doing.

3Vivek S1mo

About the investment question specifically: The point is that it seems very hard to make much progress to making AI unprofitable from just not investing. I haven't done a BOTEC yet so I'm open to be convinced by a BOTEC. There are various advantages for someone who cares about safety to invest in major AI companies, even if they are fully altruistic. The biggest is that it can enable you to buy compute right before and during the Intelligence Explosion To me, the steelman best reason not invest is not like "making firms unprofitable" but rather psychological reasons to not secretly hope for your portfolio to go up. You can mostly mitigate this by having investments to be secure against an AI crash.

1DanielW1mo

It is a gradient. More profitable and popular the industry is, the more difficult it is to regulate. The less profitable and popular it is, the easier it is to regulate. Also, on an adjacent point, PR wise for advocates against something, being seen to publicly promote the thing you are saying could kill all humans doesn't look great. I agree, spending a few thousand less in AI contests is unlikely to move them from the black to the red (most are technically in the red atm). [...] There is more to it. If I understand what you are saying, betting on the death of everyone is dumb even if you are right, by any sensible consideration of utility. Having $10 and being alive is infinitely preferable to having 1,000,000,000 in an account and everyone being dead. "Everyone dies" is not a scenerio it makes sense to expose yourself yourself to or hedge against regardless of how likely.

1Vivek S1mo

I agree its a gradient, but the point I was making was that I think the movement along the gradient from not investing is minimal relative to other factors. [...] I was a bit unclear in my comment above, but the potential counterargument was that "If you invest too much on AI and especially if you're exposed to an AI crash, you may end up doing motivated reasoning against a pause because you want your portfolio to go up". To me its worth engaging with this argument, but as I said, there are ways to circumvent this like not being overly exposed to crash. Just wrote this to clarify a few points because I was a bit unclear.

1DanielW1mo

I agree it is relatively small, but when if you think the risk in question is one of a substantial chance of everyone on earth dying, even extremly tiny pushes towards the end times seem rather unappealing. Also, I think it is an especially meaningful calculation for the people who are relatively 'high profile' around the AI risks, since there are knock-on effects with their public actions. Any promotion, purchase or investment to some extent promotes more buy-in by making the industry look a little better in public or the bottom line, but when it is publicly signaled by someone with some perceived authority there are additional knock-on effects. [...] Got it, yea, I wasn't discussing the motivated reasoning side just the actual effects, which when it comes to potentially catastrophic risks like AI and climate change, the risks are present in the aggregate so anything that has a promotes a drivers of the risk (whether CO2 emissions or AI development) is inseperatable from the larger system-wide movement on those drivers. If one existed in a world where the larger, global system-wide movements of the risk drivers could be reasonably controlled, then use can be managed on a system wide level so the individual drivers never rise to the level of having catastrophic risks. However, in our present world there is no centralized global control over carbon emissions or AI development and as such imagining whether or not in such a world your use would be within tolerances isn't meaningful.

[-]Brendan Long1mo108

This has been the approach to things like fossil fuels, which i believe is correct. Yes, individual use has minimal impact, and what matters is the aggregate but individual divestment impacts the aggregate and makes development less profitable and alternatives more desirable.

I think this is all completely pointless, although I'd probably feel differently about being an early-stage investor in a new oil company. While you can make an industry too expensive by preventing investment, the only cases of that actually happening involve legal risk and restrictions, not investors spontaneously ignoring the dollar bills on the sidewalk.

1DanielW1mo

Divestment from fossil fuels does lessen development of fossil fuels and increases the development of alternative energy sources. [...] In the real world, we see portfolios excluding fossil fuels, individual retail investors refusing to put money into fossil fuels, divestment from public institutions, etc.

5Noosphere891mo

My current take is that to a large extent, the reason this can happen is because there are substitute technologies like solar + batteries that became cheap enough in the 2010s and are still getting cheaper such that you can actually substitute it for fossil fuels. AI has for all intents and purposes zero substitute. To put it another way, there's ~zero counterfactual impact from the things you list, and ~all of the impact if you wanted to end fossil fuel usage is to pursue policies that make gas more expensive and making alternative low-carbon stuff cheaper. It's not a coincidence that the commitments you mentioned were made about as soon as it was practical to make them without suffering economic losses, and by the late 2010s are even profitable, so in essence this was a thing where the profitable thing also correlated to what we want. Unfortunately AI lacks this, so divestment/exclusion (without government intervention) doesn't work.

1DanielW1mo

Keep in mind, for the most part, AI right now is barely profitable. AI adopting firms are only doing marginally better than other firms and that had only been in the past year that it has really been making a measurable difference. The attitude that AI profitability is unstoppable and omnipresent doesn't match the current reality. The more investment there is, the sooner it is likely for AI firms to accelerate and develop larger profit margins, but if there is a hump of profitability with no turning back, we are just at the foot of it. The less appealing and profitable it looks and the lest investment there is, the longer it will take. And the more likely regulation is. The more appealing it is and the more heagily it is invested in, the more the development will accelerate and the less likely regulation is. [...] As I described before, it is a feedback loop. Divestment and moving sentiment makes other methods more profitable, leads to more less investment in fossil fuels and more in alternatives, and that combined with public media efforts makes regulation much more feasible. [...] No public divestment and disintentives to use fossil fuels, investments don't move into developing alternative energy, outside of niche applications, solar and wind are seen as unprofitable and not researched, more research is put into fossil fuels, making them cheaper making the bar to entry higher for alternatives, policy makers are less willing to impose regulations on fossil fuels since the costs of doing so are greater (since alternatives are more expensive) further slowing development. You can look at places in the US where attitudes towards fossil fuel are different and investment is seen as more acceptable and you see exactly that. [...] This is just not true. AI is being used as a substitute. Sometimes for more specialized SaaS solutions, sometimes for reducing the number of HR staff, and sometimes replacing interns assigned to doc review, etc. You can argue it has a qual

1Brendan Long1mo

I'm not convinced that this divestment meaningfully reduces fossil fuel extraction or usage. https://ourworldindata.org/grapher/energy-mix?source=fossil_fuels&metric=by_source

1DanielW1mo

Comparing to 1800 is quite obviously misleading... I didn't say it decreased it in absolute terms. Obviously energy and transportation demands have radically expanded, especially if you are doing it on total, not per-capita basis. Individual divestment is not the long term final solution, but it is part of reducing the risk and moving the needle towards a solution. If you look at more recent years, development of fossil fuels has been far more stagnant and largely driven by advancements in the developing world.

4Nisan1mo

I think it comes down to estimating (1) the altruistic impact of investment/divestment, (2) the personal benefit of investment, and (3) the altruistic impact of having more wealth. You could make a priori arguments that (1) outweights (2) and (3) and put the burden of proof on AI investors; but they could make a priori counterarguments that put the burden of proof back on you. Eventually you have to disagree on the numbers.

1DanielW1mo

See the example I gave. I doubt gwern had his wealth substantially enhanced by advertising a contest to produce AI-generated fiction. If you believe that the development of AI's pose a substantial, catastrophic risk to humanity, it seems that encouraging or investing in the development of AI is something one should view as inherently negative and only acceptable when the benefits clearly outweigh the harms. I can see an argument that in some cases they do just that, which is part of why I was explicitly not talking about AI startups like his (since that comes down to what you think the value add is and how much you think it will contribute to expanding the aggregate AI industry, which is very subjective lacking strong good evidence).

2Nisan1mo

No, I don't think so. I think it's acceptable to: * indirectly encourage the development of AI when the benefits outweigh the harms; or * directly encourage the development of AI when the benefits clearly outweigh the harms. I think the Unslop AI fiction contest only indirectly encouraged the development of AI, so it passes the first option. Arguably the benefits clearly outweigh the harms, in which case it passes the second option as well. Do you really think it's unacceptable to indirectly encourage AI development when the benefits probably outweigh the harms? Does a similar principle govern other bad activities?

1DanielW1mo

To reuse an example, I think it is fine that a lot of climate activists don't do the absolute utmost to eliminate their fossil fuel usage and exposure to it even though that is directly funding the AI industry. If, however, they went around promoting contests for 'best steak made on a coal-fired grill' and boasting the various uses of coal power, I would look at them like wtf are you doing. [...] I don't see how the indirectly v. directly is meaningful here. What matters is more the degree of promotion and the degree of benefit. One should be somewhat risk adverse about the calculations as well, but it is just weighing the positive and negative. It seems pretty straightforward that in general if you think development of X (in present legal and social environments) has a substantial catastrophic risk, something that promotes X should be difficult to justify. That doesn't mean it is never okay to use X, but in most cases you should need to see a pretty large benefit for the expected harms to outweigh the expected benefit.

2Nisan1mo

But if something only promotes X a little bit, then you only need a little bit of benefit to outweigh the harm, right? I think that's what's going on with the Unslop AI writing contest. They caused maybe a few thousand dollars to be spent on AI tokens. After subtracting the cost of producing tokens, that gave the AI labs maybe a few hundred dollars to spend on training a bigger model, which might have been the model that destroys the world. That's a tiny piece of a big catastrophe. In exchange, we might have gotten some enjoyable AI fiction. (In which case people would spend more money and get more fiction.)

[-]DanielW1mo100

Let me put it in terms of an analogy I used elsewhere: if a climate activist held a contest for the "best veggie burger cooked using solar power," and talked about how great and useful solar power was that would seem normal and unremarkable. If, however, a climate activist held a contest for the "best coal-fire steak" and expounded on the remarkable properties of coal and utility of coal power, I think that would rightly raise a few eyebrows.

Global warming and AI risks are driven by the aggregate of factors. While it is true that "the impact of promoting coal-fire steaks" is pretty minimal, it is still counter productive. There is no amount that doesn't contribute. I wouldn't really fault an critique of carbon emissions from enjoy a coal-fire steak in their home, if they happened to greatly prefer it, but if they are going around promoting the greatness of coal foster that undermines their message and contributes to positive-feedback cycles of increasing dependence on and investment in coal-fire systems which raise the risks further in excess of their direct contributions to carbon emissions.

But if something only promotes X a little bit, then you only need a little bit of benefit

... (read more)

2Nisan1mo

It depends on how much, right? Like, if you put on your last clean shirt which happens to have the logo of an AI company on it in order to go out to the laundry room to do laundry in the middle of the night, there's a small chance someone will see you, and notice your shirt, and that will make the AI company a tiny bit more appealing to that person. But it's worth it to wear a clean shirt for that short trip. I think a lot of things are like that.

1DanielW1mo

Yes, it is still scaling. But it is something that one should default to avoiding as reasonable (like with fossil fuel) and it is especially odd when leading figures seem to promote AI when the benefits are at best marginal (e.g. it is not clear to me what benefit there particularly is to having slightly better LLM fiction when there is already more published human written fiction than anyone could reasonably consume).

2Nisan1mo

Ok, so you endorse the principle that "what matters is more the degree of promotion and the degree of benefit", that it's just a matter of "weighing the positive and negative", and for a fixed benefit and catastrophic harm there is a small enough degree of promotion that is outweighed by the benefit. If you want to apply this principle, then you need to estimate the benefits and harms of the Unslop contest and be prepared to concede that it's okay if the numbers turn out that way. On the other hand, you said that some activities are "inherently negative", that one should "default to avoiding" them unless the "benefits clearly outweigh the harms". If you want to apply this principle, the burden of proof is on the Unslop contest organizers to justify their actions. But I think you'd be wrong to apply this principle in this case. I think it would be kind of absurd, like criticizing someone for wearing their AI t-shirt to the laundry room. (I haven't engaged with your coal-fired steak example because I don't share your ethical intuitions about activism.)

2DanielW1mo

There is a pretty massive difference between actively endorsing and encouraging the use of AI and inadvertently wearing a shirt with a logo because you're lazy. [...] I am not saying it is more harmful in my personal estimation, even. I am saying the more seriously you view AI risk the more likely you should be to find it more harmful and yet the people doing these public promotions seem to be those who are vocally the most critical of the risks. The benefit, I would say, is pretty much zero in this case. Slightly improved AI fiction doesn't have much of any marginal utility. There is more published fiction produced by humans every day than anyone could reasonably consume. Humans enjoy producing fiction, having it published and read. Those are real utilities which having a normal contest for fiction writing contributes to. On terms of the htklity from producing fiction, a normal contest seems to have greater utility than an AI contest even before considering the risk of contributing to AI industry. If you view the AI industry as seriously posing a catastrophic risk, that should also increasingly overwhelm any benefit.

3Nisan1mo

Ok, so I think applying the deontological principle to the Unslop contest is absurd, and you don't think it's absurd. I don't think the particular factors you mentioned support your case, and I can elaborate on why if you like. But I think it would be more productive to talk about the benefit. The Unslop contest was organized in part by Gwern, Alexander Wales, and Jamie Wahls, who are all writers. (I don't know about Roon.) I have read that writers have a compulsion to write. They can't not write. Writing is like breathing to them. How could they not be curious if current AI models can write good fiction? How could they not wonder if these new creatures we've created, who breathe words the way we breathe air, can write excellently? Both Jamie Wahls and Gwern have used fiction to express new ideas that are important to them. (I haven't read much Alexander Wales — a mistake I'd like to correct soon.) They clearly believe we need more fiction, expressing ideas that haven't been expressed yet. (But see....) It would be more enjoyable to read new fiction that is excellent, even if it's written by an AI. Think of something that's just as meaningful and beautiful and good to you. What if someone told you the benefit was objectively zero, and in your pursuit you were actively promoting harm, so you should stop? Would you be more inclined to tally up the harms and benefits? Would it be harder to be convinced that pursuing good is inherently bad?

2DanielW1mo

As I said, for some humans writing gives them a real utility. This is a benefit of human writing so encouraging it has some utility. [...] So your argument is that the utility is that it will provide some satisfaction to the curiosity of Gwern, Alexander Wales, and Jamie Wahls? Increasing catastrophic risk for your own curiosity is an exceptionally self-interested calculation, perhaps one you can justify but it is an antisocial one. It is the same self-interested reasoning I would consider fallacious when talking about climate change--the risk is in aggregate so your self-interest should be considered in a social context. One person doing it once to satisfy their curiosity has minimal impact, sure, but because the catastrophic risk is in the aggregate you cannot disentangle the effects from induvial contributions. If everyone thinks "my part has little effect, so I might as well maximize my own utility" then that has a pretty outsized negative effect. I think we as a society should frown on behavior that has outsized negative externalities, whether that is burning fossil fuels, polluting or--if you believe in a catastrophic AI risk--contributing to the AI industry. [...] I agree with much of what Gwern says here, and he disagrees with you. He explicitly views subsidizing additional fiction production as, at best, difficult to justify. More new SF and fiction of good quality in any given genre on any given is already being produced in excess of what one can reasonably consume, though most of it is not the 'best' and it is easier to identify the 'best' among older fiction than new. The gain in utility from increasing the production of good quality fiction (assuming we grant that AI is doing that) is negligible. Gwern explicitly articulates: "people write too much new fiction" and "they also read too much." I wouldn't necessarily agree with the first statement, if you get utility from writing there is pretty much no negative externalities--maybe you could get uti

2Nisan1mo

The difference is that you're applying a deontological principle to existential risk in a way that imho doesn't make sense. I was hoping you'd accept my invitation to try out the principle in a domain where your intuitions are reversed — where the benefit feels real and central, where the harm feels theoretical, trivial, unserious. That would be a better test of your principles. It is an uncomfortable thing to do, and you don't have to. You've been a good sport. Thanks for the conversation!

2DanielW1mo

Must have missed this, but i do this every day. E.g., i think compassion is an average good, and default to being less skeptical of the utility of any given act of compassion. [...] I think there is still some misunderstanding of the other on one or both our parts.. My claims were entirely based in utilitarian heuristics, I am not sure what deontological principle you are referring to. Let me put it another way, just on the case of promotion in general (not even including the additional negative utility of weakening the message of activists) 1. If in the aggregate of X has substantial enough risk to result in a negative expected utility, then the average case of X has a negative expected utility. 2. If the average case has a negative utility, then any given case should be viewed skeptically (and the greater the average negative utility the more critical one should be). 3. If you view the aggregate of the current tragetory of AI as having enough harm/risk to be of a negative expexted utility (e.g., if you believe there is substantial risk of current trends leading to develops that cause the extinction of all humans), then under that view the average contribution to AI has a negative expected utility. 4. If the average contribution to AI has a negative expected utility, then any given contribution should be viewed skeptically (and the greater the risk, the more one should default to opposition) 5. Much "pop" criticism of AI views it as a net negative not because of existential risk but rather job loss and amorphous negative views of AI products, among other factors. 6. These "pop" critics default to viewing AI developments critically (but will generally view AI as at least acceptable when it has what they view as an outsized positive effect). 7. These "pop" critics, thus, appear consistent with (4). 8. Some of the people proclaiming to view the aggregate of the current tragetory of AI as the most negative (i.e., they place the most credence in estimations

2Nisan26d

To be clear, I don't think there's anything wrong with having deontological principles. I try to follow some deontological principles myself! Here are some things you've said that make me think you're not using entirely utilitarian reasoning (bolding mine): [...] Utilitarianism would ask: Do the benefits outweigh the harms? But a deontology also cares about the bolded terms: Is the act inherently negative? Should it be avoided by default? Is there an unacceptable harm that cannot be balanced by any amount of benefit? Is the act intentional or inadvertent? Is there an extenuating motivation? In fact my first comment in this thread recommended that you do the utilitarian thing and estimate the benefits and harms. Instead you responded with arguments that, to my eye, would let you stick to your conclusion even if the benefits turned out to technically outweigh the harms. What's a concrete example of an ordinary, everyday act of compassion you'd wholeheartedly endorse?

1DanielW26d

Yes, and I am saying you can (under the described beliefs) use carbon emissions or promoting AI as heuristics for the harms outweighing the benefits. If the only information I know about an action is that it increases carbon emissions or it promotes AI then if I believe those things are--in aggregate--bad then my expectation for any given action should be that it is bad. If I know the net expected utility of set is negative (i.e. where ). Without additional information about expected utility of any given element , I can say . We never have perfect knowledge of utility expectations. [...] The actual utility calculation is unclear and depends on one's personal values which I wasn't working on. But going by what you gave me for gwern's expected utility, I should be able to say: * * *

2Nisan26d

I disagree with that. So, that's my utilitarian counterargument. What do you think of the (generally reasonable) deontological language I highlighted in the quotes above? And do you want to choose a concrete example of an ordinary, everyday act of compassion you wholeheartedly endorse?

1DanielW26d

I disagree with it as well, but it is the estimated utility in the post from gwern you directed me to. [...] Hugging a loved one, holding the door for a stranger, giving a dollar to charity, etc.

4Nisan24d

That's great. So now I'll construct a homunculus of an argument for a conclusion that neither of us agrees with, but which follows the structure, form, and abstract principles you have used. Notice how the principles feel less sound this time around. "You shouldn't hug your loved ones", says the homunculus, "because it encourages close contact between people, which leads to the spread of infectious disease, including catastrophic pandemics. You should instead practice social distancing, which discourages close contact. "Pandemics pose substantial, catastrophic risks, including loss of life. So encouraging or promoting close physical contact is something one should view as inherently negative. One should default to not hugging, and only hug when the benefits clearly outweigh the harms. "In this case, the benefit is pretty much zero. I doubt you had your wealth substantially enhanced by this hug. The benefit is, what, some momentary comfort for you and your loved one? Increasing pandemic risk for your own comfort is antisocial behavior. If everyone did that, the effects would add up. It has negative externalities, and you shouldn't do it. "For someone who supposedly supports the idea of public health, you're remarkably enthusiastic about supporting such disease-promoting behavior. Eyebrow-raising, to say the least." ---------------------------------------- Obviously we don't agree with the homunculus' conclusion, so something is wrong with its argument. Maybe one or more of its general principles is flawed, or has unaddressed limitations?

1DanielW24d

I do not believe the net effect of human contact is negative. Human-human contact does not promote aggregate risk in the case of catastrophic disease in the same way in our current environment. If I believed that the net expected utility of human-human contact was negative, then yes I would be skeptical of human-human contact. I do not believe that. [...] Yes, when the net utility from human-human physical contact is negative, I do consider increasing that risk as anti-social behavior and yes the risk do add up. In normal circumstances, the net utility is not negative in some special circumstances (i.e., during a pandemic) the net utility can be negative, in which case yes you should seek to avoid human-human physical contact. Not doing so for your own comfort (e.g., throwing parties or social events involving close physical contact during a pandemic) is antisocial behavior.

4Nisan24d

"It does promote pandemic risk", says the homunculus. "You're making physical contact look more appealing. It shifts the public sentiment. You're going around promoting the greatness of physical contact, undermining your public-health messaging and contributing to a positive-feedback cycle of increasing physical contact and positive sentiment for the same. There's no amount that doesn't contribute. "The expected harm of promoting a deadly pandemic even a little bit is pretty substantial. Anything that increases that risk should be difficult to justify. You would need to see a pretty large benefit to outweigh the risk, and as I said the benefit in this case is pretty much zero."

1DanielW24d

I don't know what case you think you are responding to, but it isn't one I made. When I think the net expected utility of (marginal) actions within a set of actions is negative (in the current world), then I am skeptical of actions in that set. I do not think the net expected utility of all human contact is negative. If you believe the expected outcome of AI is development is negative, I cannot imagine you do not also think the net expected utility of all additional contributions to AI is also negative. If you think that, then you should be skeptical towards any contribution. That is exactly the attitude I take currently with carbon emissions, for example. I think the net expected utility of all (marginal) anthropogenic carbon emissions are negative. Therefore, I try to avoid actions that increase carbon emissions and default to viewing carbon emitting activities as negative. As I have repeated, that isn't to say actions that increase carbon emissions are always bad, sometimes they may be justified by greater social utility and sometimes we may be willing to do them for our personal benefits--we are only human after all. In certain periods, I have taken this attidude with outside-human contact as well. When I expect the net expected utility of all additional contact between people is negative (e.g., during a major pandemic), then I am skeptical towards participating in any activity that would promote or encourage that kind of contact.

2Nisan24d

Ah, I hadn't appreciated how central that principle was in your thinking. Attempting to say it in my own words: "If there's a set of actions such that taking all the actions would be worse than refraining from all the actions, because of the net harm, then we should presume that taking any particular action in that set is worse than not taking it, absent a really good reason." I'll note this principle is quite sensitive to how you group actions into families. Take the set of all carbon emissions starting now. If we suddenly refrained from all of them, we'd face famine pretty soon, which would be worse than global warming. So the principle doesn't apply to marginal carbon emissions (in the sense of emissions we're about to make but haven't made yet). If instead you take the set of future carbon emissions minus the ones we were "going to do anyways", in the course of our daily lives, then you probably think it's better to refrain from all these "extra" emissions, so by your principle one should by default refrain from any particular marginal emission. ("Marginal" here means going out of one's way, doing something non-ordinary.) (But what if one was planning on going into a career in oil drilling? Is that an "extra" action?) The homunculus says, "We already had SARS, MERS, and COVID. We'll surely get a fourth coronavirus epidemic soon, and the next one could be deadly enough to outweigh the benefit of the preceding decade of human contact. Better to refrain from all but medically necessary human contact until we develop broad-spectrum vaccines against all coronaviruses. Therefore any particular hug should be avoided by default." I'd be happy to give up the entire deep learning revolution to avoid existential risk from AI, so your principle would have me refrain from praising Google Translate in 2017.

1DanielW24d

I would drop the 'absent a really good reason.' It isn't necessarily a strong recommendation--it is approximately a default assumption. I previously formalized it as: If I know the net expected utility of set is negative (i.e. where ). Without additional information about expected utility of any given element , I can say . Absent further information and evaluation of an action, the default expectation should be it is negative (in proportion to how negative you view the aggregate and a weighting of the factor as part of the whole). [...] I think you are misunderstanding what I the expected utility function would work out to, the counter-factual isn't "stop all carbon emissions," it is a probability distribution of what we expect carbon emissions to be. It would be: . Where j represents each possible amount of future emissions, represents the probability of j emissions, represents the expected utility of the consequences of j level of emissions. We can also say: (that is, the expected utility resulting from j level emissions is the sum-product of the probability of each outcome at j level of emissions times the utility of each of those outcomes). Because the probability distribution of carbon emissions is well beyond a level that is safe and the bottom-end of the distribution hardly touches on a level where we would expect more harm from famine, the expected utility of future carbon emissions works out to be negative. Our actions/inactions nudge the probability curve slightly one way or another, they do not flip it on and off. IF we got to the point where the probability distribution was such that it was well below catastrophic risk, then the EU would become positive. [...] If you believe that risk is so substantial and human contact is part of that risk, then yes I would say you should try to minimize human contact. I do not believe the risk is that substantial, I believe that in net marginal human-human contact is a social neutral-to-positive. There co

2Nisan22d

Thanks for clarifying. Here's my main criticism of your principle. [...] I basically agree with this. If all you know about an action is that it belongs to a set of actions with presumed independent effects that add up to negative value, then the expected value of that action is negative. However, you usually have additional information about the action, so the argument doesn't apply. So your principle is very weak. I think this is the answer to your question at the top of this thread: Why do people concerned about catastrophic risks from AI sometimes do things in the direction of supporting AI? Because they know the details of the particular actions they're taking, and based on those details they judge those actions to have positive expected value. (Ideally! Many people are nincompoops who do things for bad reasons. But I think the Unslop contest was fine.) That's why I said "you have to disagree on the numbers". I think you tend to use a stronger version of the principle which is something like, "Come on, there's no way you have enough information to know that it's positive expected value, something funny is going on." To which I would say, "No really, let's talk about the numbers, it checks out." I'm running out of energy to spend on this thread, though.

1DanielW22d

Also, I should say, that when you plug in the numbers you get big negatives. Let's say i estimate at the current probilibility distribution of AI development has a net expected utility of say -1,000,000,000,000 ( say, my p(doom) is ~1/8, i think 8 billion humans will die and i value each human at 1,000). If i think my actions (including positive feedback!) are contributing 1/100,000,000th of AI development, then my expectated utility from the AI development I contribute should be ~ -10,000 (about the same as the value I would place on 10 human lives). So unless I think i am saving 10 lives (or equivelent) in contributing 1 hundredth of a millionth of AI development, I should find my actions negative. Current total investment in AI is under 2 trillion. Being conservative (a dollar now means more than a dollar later!) you should value contributing $2,000 to AI development as being roughly as negative as killing a stranger (under a simplified framework, that just looks at dollars).

2Nisan22d

Now we're getting somewhere. You're assuming that providing one billionth of this year's investment in AI means decreasing the expected utility from human extinction by one billionth. But that's not right. If you invest in AI, that means the AI companies will choose to raise less money from other investors. The net effect is that AI companies will raise a little more money than if you hadn't invested (but less than the amount you invest), get a better interest rate, and spend less effort on fundraising this year. And this will have some effect on the probability of an extinction-level AI catastrophe this generation, but the effect isn't linear in the amount you invest. In a parallel thread, your theory of impact was that divestment has an effect on regulation. That's also nonlinear. Investing in AI is something I'm likely to do. (Arguably >10% of the S&P 500 by market cap use marginal investment to try to build frontier models.) Here's how I justify it: The amount of stock I can buy won't move the stock price perceptibly. If I estimated the price movement it would be tiny. That would translate into AI companies expending a tiny bit less effort on selling stock this year to fund datacenters, which accelerates timelines a little bit, which increases existential risk a very tiny amount. On the other hand, if I invest my savings in a broad-market index fund that includes AI, that'll increase my savings and I'll have more time to spend doing AI safety research, calling my Congressperson, etc., which are more directly impactful. Plus, I have lots of personal uses for more money. I'm basically satisfied with where this thread ended up. (I just wanted to convey that it's in general reasonable to do things that seem individually worth it, even if they seem to be promoting AI in some way.) I want to flag that I won't have the energy to find a crux in our disagreement over the Unslop contest or whether I should buy index funds. Though if you have a good argument about why n

1DanielW22d

You seem to be confusing some things, probably because I was not as precise with definitions of terms as I should have been. [...] Divestment was an example of a common behavior activists engage in (which I would broadly speaking endorse). The impact from divestment on a industry is different from the negative impact of funding, advertising them, and paying an industry. They not direct inversions of each other, divesting is a very different activity. [...] I said "contributing to AI development" here, not investing. What is meant by "investing" is more nuanced (e.g. buying stocks second-hand on the open market is not actually 'investing' in the economic sense). Here, I am including something as simple as buying AI products (a substantial portion of money sent will contribute to AI development). [...] This is (generally) the exact opposite of how investment works. Me investing a dollar, encourages Steve to invest a dollar which provides collateral to borrow another 2 dollars. Me not investing a dollar encourages Steve not to invest a dollar. Me spending a dollar a product boosts that company's profits, encouraging others to invest more dollars and moving outward the company's expected demand, encouraging them to invest internally in expanding that product. Investments follow trends, both of other investments and of consumer behavior, moving investment towards one thing moves more investment towards it. [...] A very tiny increase in an extreme risk is very negative. And remember, these are aggregate risks that cover some probability distribution of outcomes, not just flat +/- x risk. It is accurate, as such, to estimate the EU as a portion of the whole, not just the absolute movement in risk. [...] Assuming you aren't a private equity firm or buying directly in IPOs, your personal finances have minimal direct impact on the economic investment which is the investment one cares about. Divesting from exposed firms is a signaling behavior and really most usefu

1DanielW22d

Yes, it is a weak default case, but it generalizes. If rational, the people who find the aggregate risk the most serious should also have the highest threshold for engaging in any behavior increasing that risk. This largely holds when it comes to the public actions of e.g. climate activists and even what i called "pop" AI criticis, but the opposite is true for some of the people who proclaim to be the most critical of AI. [...] Then give the numbers. The only concrete numbers you gave me was gwern thinking there was negative expected utility to the production of new fiction, which runs directly counter to your claimed rationale.

4Random Developer1mo

I do not work for a frontier lab, I hold no stock in AI or related companies (except via passive index funds), and I have downgraded my personal subscription plans so that I have a single $20/month plan I use to check capabilities. I made the decision not to buy stock in 2022 although I correctly expected buying it to be profitable. However, I am a software developer, and I would like to eat between now and the time that the Terminators roll off the fully robotic assembly line. So I do use AI at work (while advocating for self-hosting cheaper open models, actually, that do not directly fund frontier research, though that advocacy is for cost reduction, not larger risks), write a few small tools to help with AI agent sandboxing, and run smaller models on my own hardware. I think we have a significant chance of developing superintelligence within 3-20 years. If this occurs, I think we have a nearly 100% chance of losing control, and a high chance of actual human extinction or being reduced to a marginalized species living in situations analogous to chimpanzees (ecologically margin habitat, some preserves, and zoos).

3Vaniver1mo

An important fact is that many of us are not, or have divested from investments made in rosier times.

1DanielW1mo

Definitely, I should have been clearer in my OP: it is not a universal observation but an observation of the patterns of some relatively high-profile people in area. Many certainly do seem to take the approach (which I agree with) of favoring against any investment in AI industry, using it only where one believes there is a very large utility gain in excess of whatever investment one is giving to AI developments (which is the approach I would advocate, whether for AI or using fossil fuels). But some seem to take the attitude of it not mattering (since it is only at the aggregate level of the industry that poses the most serious risks they identify) and actually take positions that actively, publicly, promote AI developments, usage and dependance. If one is an altruist (if you are purely self-interested there are different calculations of the risk), this seems to me to be fallacious and counter productive in the case of fossil fuels as well as AI.

5Zack_M_Davis1mo

Does your altruism take into account benefits from economic growth? I think the people whose behavior you're puzzled by have an attitude more like "AI and fossil fuels are great except for the potential for catastrophic risk, which demands targeted mitigations" (as contrasted to "AI and fossil fuels are Bad in general; only use if truly unavoidable").

1DanielW1mo

Yes, I explicitly described both in the case of fossil fuels and AI. Because the results of expanded development are catastrophic, the default leans heavily in one direction. If the rent is large enough to be in excess of whatever the investment is, then it is reasonably justified. However, the example I gave was, e.g. using AI to produce more fiction. It seems pretty clear to me that the utility gained by having AIs produce fiction instead is far less than any reasonable consideration of the harm of investing in AI if you believe there is a sizable risk of AI killing all humans.

2Zack_M_Davis1mo

I think the idea is that we shouldn't have to indiscriminately divest from the whole industry when we could "just" build fiction-writing AIs and not catastrophically risky AIs (as you suggested about Gwern's company).

4DanielW1mo

The problem with these catastrophic risk is it manifests in the aggregate. Any use of fossil fuels increases the aggregate risk directly. With AI the increase is less direct but no less real (more money in expands the industry, and makes development look more appealing, encouraging further buy-in). If you had global centralized control of all fossil fuel and the entire AI industry, then yes, targeted mitigations would be sufficient since you could allocate everything so the risk was never catastrophic. But the world we live in isn't that, any use of fossil fuels contributes to the risk and any divestment/decrease lessons the risk. In some cases, divesting or boycotting would be so costly as to not be worth it anyways, but the more seriously you view the risk the marginal you should consider the benefits. Given we do not have centralized control, increasing investment and use (whether fossil fuels or AI) increases the risk. [...] I wasn't suggesting gwern's company was bad or not, I explicitly said I wasn't commenting on his business--I don't know enough to exactly do an analysis and, in any case, it would feel inappropriate even if I had a real position. My comment was particularly on the promotion of AI in general use, encouraging participation in AI-generated fiction and other use cases where I cannot see how someone who believes in a substantial catastrophic risk risk from the AI industry's development would view that benefit as outweighing the risk.

[-]DanielW3mo30

I was thinking of doing more research and a write up, but someone may have covered it or knows of a good reference (I have seen there is some related discussions here on uncertainty and Bayesian updating).

Priors are not all equal, there is a big (rather obvious) difference between a prior that is based on empirical data and quantitatively robust models, and priors that are based on personal guesses and intuitions. Personally, with my own empirical and skeptical training as background, I find conflating these somewhat unhelpful and giving the appearance of ... (read more)

[-]DanielW6mo3-1

If prediction markets are efficient, no one should use them (how they are currently being used). This is something that has bothered me for a while about the claims with prediction markets, particularly when compared to equity markets. Prediction markets are fundamentally a negative sum game (since you lose fees/interest taken by the platforms). If they were efficient (i.e., the price reflects all available information accurately), then you should in general always expect to lose in the long run if you don't have any private information. Equity markets hav... (read more)

2papetoast6mo

Isn't a natural conclusion then that in reality prediction markets are not completely efficient? There are subsidy, irrational traders/gamblers, and risk hedging (yes I know you mentioned it). Subsidy can come in another form, a player can spend money trying to elicit information so that they can use the information somewhere else. Also efficiency can be relative to a trader even if everyone only trades on public information because in reality no one has the time to discover and aggregate all the public information. This is just wrong

2DanielW6mo

I'll be blunt and start here, as it is an important mistake I see made a lot. I think a lot of people misunderstand what market efficiency refers to. It has nothing to do with individual trader behavior. You could be seeking out the worst investments, that wouldn't matter to the question of how efficient the market is. Market efficiency is just about the price. To the extent the price accurately reflects available information (or the underlying fundamental value) the market is efficient. An individual traders behavior is entirely irrelevant. Also, efficiency doesn't equate to "good" so it can be a bit misleading. You could say roulette is 95% efficient, since the price is just off by ~5% of the underlying value (which can be ascertained by available information). But there isn't any way for you to reliably capture that 5%, it is off because of value captured by the house. And, like prediction markets, roulette is a negative sum game. [...] Sure, hence the "if" in my original comment. If you are doing it because you like to gamble, that's fine. Risk hedging also theoretically makes sense, but isn't currently how the markets are being used in any meaningful aense (and as mentioned runs dirextly contrary to how proponents like Hanson often suggest they should be used). If you think the average traders are predictably irrational it should be possible to take their money and enrich the platform in the process. That seems like a bad thing and a good reason to advocate against prediction markets. Negative sum games are not exactly great for the economy, and anyone with altruistic views should view negative sum games as definitionally net negatives, I would argue. Proponents of prediction markets generally argue they are efficient markets. If they aren't, the utility generally claimed to exist isn't there. [...] Spending money doesn't gain information. As I mentioned in my other comment. There is no additional price discovery function you can access to by placing a b

1papetoast6mo

I think I am implicitly just saying that prediction markets are not perfectly efficient. I think that is normal and still have great utility, I don't think efficient markets are something to be worried about because they will only exist mathematically. All these behaviors that only make sense when the market is not efficient makes sense exactly because the market is not efficient. But, there are different degrees of inefficiency. It is a scale, and I think prediction markets are pretty accurate already even if it is not mathematically efficient. Have you seen the brier score on different prediction markets? They are obviously not perfect, but it is pretty close. Close enough that it sometimes makes sense to act as if market probability = true probability. Also the prediction markets are not negative sum for society as a whole, the fee doesn't just evaporate. It simply goes to the platform. Taking money from dumb people in a zero-sum game is fine. [...] Again I am assuming an imperfect world. There are a lot of public information but you don't want to spend time aggregating them, instead you create a market and put $10k in the automatic market maker to incentivize people to make their best guess.

2DanielW6mo

I wouldn't say anyone is really discussing perfect efficiency. You could reasonably say roulette is a fairly efficient market, the price accurately reflects 95% of the value. There isn't really value you can reliably, fairly make off them (otherwise someone like Jane Street would probably have scooped it up). You may be able to make money in some, but odds are more likely than not you are wrong and the juice isn't worth the squeeze. [...] Yes, and if that is the case, your chances are greater than not that no matter how intelligent you are if that is true using a prediction market will lose you money. [...] What, though? The main valid arguments for utility i see--risk management and price discovery--are not how the platforms are being used or not good reasons to trade (on current platforms) respectively. To be clear, I think there are valid arguments for utility in potential, but what is currently being promoted by proponents in extant markets doesn't match the potential utility. [...] In theory, maybe. In reality, doesn't happen on any current platforms. To the extent they adopt automatic market makers, those market makers act more as bookies, providing liquidity only at prices in excess of reasonable fair odds. Some play money platforms use automatic market makers designed to encourage price discovery (which Hanson has advocated for) but I am not aware of any large prediction markets that do so. [...] They have costs and there still is time value issues (to be truly net-zero, the total value of any position on average would have to appreciate by the risk-free rate + a risk premium), so it is still a net negative sum in that sense. But I was talking from the perspective of a person trading, not society as a whole. [...] I don't necessarily think so. Like, a lot of scams are zero sum, but I would say scamming people is still wrong (even in cases where it might not legally be fraud). Taking advantage of people being gullible or gamblers is generally some

1DanielW6mo

I didn't mention it, but of course another benefit a lot of proponents reference is price discovery. This is certainly a real benefit (though, I am a bit skeptical about some of the claims of the extent of utility), but as an individual not a good reason to trade. I have seen some, like Scott Alexander, argue that platforms will pay traders to use them. This seems facially implausible to me and not clear how it would be structured. But in any case, it isn't how the platforms actually work (some do pay interest on positions, but they pay less than the risk-free rate). Also, some have argued for actually encouraging insider trading, since allowing insiders to trade will make the prices even more accurate predictions. Even if we grant this argument, this makes it even worse reason for a typical trader to enter the market. It would mean paying a premium to play an unfair game, paying to get worse odds seems inherently non-sensical.

[-]DanielW15d10

Has @Eliezer Yudkowsky or others talking about decision theory here contended with/incorporated the insights from ideas of 'ratifiablility' in CDT? There seem to be some mistakes/misaprehensions (e.g., this thread with Yud) about not understanding how CDT deals with some problems (like rock-paper-scissors).

Joyce has talked about this at length.

See a short 2020 slide presentation here: https://websites.umich.edu/~jjoyce/papers/IU.pdf

And a more robust discussion of ratifiable problems in CDT on page 474 here: https://eprints.ukh.ac.id/id/eprint/240/1/2016_B... (read more)

3Eliezer Yudkowsky14d

I treat ratifiable mixed strategy CDT as standard CDT. My objection to CDT is "the standard sophisticated variant ratifies bad bets" not "no known variant of CDT ever makes a choice".

1DanielW14d

Thank you for the explanation! Though, in the games you are talking about in that thread it seems there isn't a ratifiable solution or the solution CDTers would argue is ratifiable requires a slight restructuring which leads to the same solution as under FDT, or am I missing something? Also, Joyce argues you should pretty much do backwards induction of the expectations for future choices and only consider the causality of current choices. "When considering an act whose performance you can ensure right now, you should treat it as an instrument for changing the world and evaluate it by its causal expected utility; when considering an act you cannot perform now, its prospective performance should be treated as you would any event not under your control, i.e., as an item of news." It leads to some weird implications which Joyce describes. E.g., choosing between A and (B or C) and then choosing between (B or C) is meaningfully different then choosing between A, B or C.

1DanielW15d

Probably should have tagged @Caspar Oesterheld since Joyce is directly responding to his case. Edit: to be clear Oesterheld does address ratifiablility in his article. But as Joyce describes, CDT necessities a choosing mechanism to have a solution (if it doesn't, the CDTer should never be able to pick either box). Under such a mechanism, as i noted in an older thread, the problem as Caspar set it up is impossible with true beliefs if there isn't some additional preference.

[-]DanielW6mo*1-2

Does Logical Decision Theories actually give meaningfully better recommendations on real world problems, particularly voting, frequently referenced?

One of the main reasons given for preferring logical decision theories (LDT), or particularly functional decision theory (FDT) is that agents do better in real world problems. Indeed, the article here on logical decision theory opens by discussing voting. I recently posted a discussion of a hypothetical where FDT agents perform worse, but I think when applying it in practice to the real world case of voting whi... (read more)

1[comment deleted]5mo

[-]DanielW6mo*0-9

I have been thinking about this for a while, and will probably formalize it at some point, but would like to get some of your thoughts in case there is some obvious case/backgroun I am missing.

In some more realistic formulations of dilemmas, agents that make decisions under Functional Decision Theory may have generally inferior outcomes to rational agents acting under alternative decision theories (in this case, I am just going to consider causal decision theory), which creates a seeming paradox that I am sure many readers will already expect (though this... (read more)

2Firmament6mo

My knowledge of decision theory comes exclusively from reading relevant LessWrong posts when the mood takes me, but it seems to me that FDT-Will would instead act like this: 1. Assume Derek uses CDT and Will uses FDT (I do not know what would happen if both Derek and Will used FDT, but what happens would probably be the same as what happens when two FDT agents play the Ultimatum Game, since the situations are similar). 2. Imagine Derek demands $100,000,199.99 from Will. 3. When Will gets back to town, he reasons like so: 4. 1. "If I implement the algorithm that always pays Derek money when he requests less than $100,000,200, then the copy of the algorithm living in Derek's mind when he was setting the price will also pay, meaning Derek will demand $100,000,199.99 from me, leaving me one cent better off." 2. "If I implement the algorithm that does not pay no matter what, then the copy of the algorithm living in Derek's mind when he was setting the price will also not pay no matter what, meaning Derek will not demand any money at all, since he predicts that I will not pay any sum of money he asks for. Derek will then drive me to town free of charge (since he wants me to live), leaving me $1,000,200 better off." 5. Will determines that the algorithm that doesn't pay money no matter what gives better outcomes, so he doesn't pay. Derek terminates the simulation and, in real life, gives Will a ride free of charge. 6. (Will then terminates that simulation and, in realer-life, decides to become an FDT agent) Thus, FDT outperforms CDT, since CDT pays $199.99 and FDT pays nothing at all.

2DanielW6mo

You are modeling a slightly different scenario that assumes Will has full knowledge of Derek's price-setting behavior. In that scenario, you are correct, but the scenario is explicitly assuming asymmetry in that regard. Derek knows Will's expected value when setting the scenario, but Will only has information to determine the expected value after the scenario has been set. From Will's perspective, if he was an agent that accepted the deal, he would get taken back. If he was an agent that rejected the deal, he wouldn't. Derek has knowledge of will's decision making in this scenario, the reverse is not true. If the question was "what should Will's behavior be?" Will would win under FDT, since that would indeed imply he had knowledge and as such should adopt a 'never pay'. But if we calculate his EV under the perspective of Derek price setting, without assuming Will has full knowledge, Will's EV for paying back is positive for accepting the deal and negative if he wouldn't pay back (since as far as Will is concerned, if he wouldn't pay Derek, he would have been left to die), so as a rational agent should pay Derek. Edit: to be clear, your scenario also implicitly assumes a degree of asymmetry, just on the side of Will's decision making. If we assume symmetry or allow negotiation, we would expect Will to adopt a strategy somewhere between -1 and 1,000,200 if we assume FDT agents (since those are the scenarios where they can both accept a deal as a net positive). If they are CDT agents, we expect them to negotiate somewhere between -1 and 200 (since those are the scenarios where they can both accept a deal as a net positive).

2Firmament6mo

Ah, I see! So the scenario favors CDT only because Will lacks full information on the problem. Will thinks he's playing Parfit's Hitchhiker, but in reality he's playing Ultimatum. I dunno, it doesn't really seem like a fair problem. You could construct a problem that unfairly favors CDT instead, like "Newcomb's Problem but, unbeknownst to you, if you one-box, you die". When choosing his decision theory, maybe Will should decide to run FDT and then conservatively make CDT-like actions when he doesn't have full information, if he's expecting to encounter a lot of situations like this.

2DanielW6mo

Exactly, it is unfair to Will under FDT (which is part of why I framed it from Derek's perspective), but I would argue it is a lot closer to what we see in the real world. Usually there is some asymmetry, people have nuanced utility functions and when there is some net positive utility actors desire to capture as much of the net benefit for themselves as they can. While dishonest agents can lead to worse outcomes (e.g., in the traditional Parfit's dilemma, someone who are under CDT is simply left to die, with no choice in the matter), unmitigated honesty can lead to an actor being a patsy, and taken advantage of. Realistic utility functions generally model and moderate this, I would argue, better than FDT. One of the main reasons Yudowsky and Soares give for preferring FDT is that in the real world they view it as generally leading to better outcomes. I am not convinced that is the case, where these sorts of unfair situations arise. [...] Yes, but that isn't very realistic. The purpose of the construction is that more realistic assumptions can lend to FDT being worse off. A more direct comparison to this would be Newcomb's Problem with Transparent Boxes where the Predictor doesn't actually care if you one-box. But Newcomb's problem doesn't seem very realistic on its face. This framing, I would argue, more closely resembles how we actuslly handle asymmetric problems. Pricing decisions are based on willingness to pay, contracts are devised with incentives around what we anticipate will make the cost of cheating greater than the cost of being honest, etc. These observed behaviors align better with an understanding of expected value under causal decision theory. If we assumed an agent that always has at least symetric information and predictive ability, I think it is fair to say that for such an agent, FDT would win out. But in reality is that is rarely the case. Amazon knows more about your purchase histories, price decisions and whatnot than you know about Amazo

1Firmament6mo

Hm, I guess it's an empirical question then, of whether these situations happen in real life often enough to warrant using CDT or FDT-acting-like-CDT. I think FDT still wins out in the end, because FDT will emulate CDT if it realizes it lives in an FDT-hostile world (thus taking CDT-like actions out of an abundance of caution), while CDT has a harder time emulating FDT (CDT would need to use precommitment, while FDT does not). I think that, in real life, humans are already near-optimal on this. People in low-trust FDT-hostile communities will grow up running CDT, while people in high-trust communities grow up running FDT. Issues occur when a person's community changes or if they move to a new community, but that problem is perhaps outside of the scope of decision theory. So the moral of the story is "run FDT, but emulate CDT if you live near a lot of information-asymmetric FDT-hostile problems" And, of course, the point of LessWrong decision theory is to choose which one to give to a superintelligence (I think), since humans can't actually choose their own decision theory (for the most part). And superintelligences probably won't end up on the short side of an information asymmetry very often. [...] Oops, meant "unfairly favors FDT instead; if you two-box, you die". It is indeed not very realistic, but it is possible to construct an isomorphic question in a more realistic setting, much like how Parfit's Hitchhiker is a more realistic yet isomorphic framing of Newcomb's Problem. So, again, it's an empirical question of which decision theory is favored in situations that occur in real life. Claude Opus 4.6 Extended weighs in with what I believe to be insightful and correct critique (oneshot/no selection; told to "focus on logical errors"; I'm "Poster 2" and you're "Poster 1") There are a few errors and confusions here that neither poster fully identifies: The honesty parameter is doing most of the work, and it's doing different work for each DT. Under CD

2DanielW6mo

I sort of agree, but I don't think it is one we can strictly answer. I gave some reasons we might think (in real world scenarios) that CDT tends to better explain behavior (e.g., Braess's paradox), though I do not believe it is one we can have enough data to answer for all time. [...] I disagree. I think the difference in high and low trust societies can largely be attributed to differences in utility functions and signaling effects. This appears empirically true (high-trust societies tend to have more information on each others activities, making signaling more meaningful, tend to poll as valuing trust, etc). Also, the problem isn't "FDT-hostile" per-say. Derek is just trying to maximize his utility, he only cares what value he thinks he can get Will to honestly pay him. FDT does worse because it recommends taking the deal in Parfit's hitchhiker and doesn't have a posterior restraint on honest signaling. If Derek was a misanthrope, Will could lose under CDT. If Derek valued saving Will's life at -195 (instead of +6), Derek would leave CDT Will to die and still save FDT Will. This is why I predicated that Derek has to be a decent person, he prefers a scenario where everyone wins to one where he wins and Will dies. [...] Minor nitpick but Parfit's hitchiker is isomorphic under CDT and to Newcomb's Problem with transparent boxes under both CDT and EDT but not Newcomb's problem in general. CDT doesn't care about evidentiary probability, but if you don't know what is in the boxes EDT says you should act probabilistically. [...] I mean, I guess that is not unfair, but it seems bad. It would imply honest signals are good for CDT agents and dishonest signals are good for FDT agents. Also, Claude is wrong or missing the point. [...] Yes, this is the point. In the real world, for some value people are unwilling to cheat and for some value people are willing to cheat. The number of people, on seeing someone drop a $5 bill, that would return it to the person is

2Firmament6mo

Indeed, whatever humans do seems to be closer to CDT than other decision theories, although humans use various concepts like justice, trust, and honor to approximate FDT occasionally. [...] The differences in utility functions are the humans' way of implementing FDT (since FDT is too hard to reason about for evolution to instill it directly), and the signaling effects/mutual knowledge are what makes FDT worth it. Really, in this scenario, the fact that Will values honesty and promise-keeping means that CDT-Will is implementing a decision theory somewhere between CDT and FDT. FDT-Will effectively values honesty at infinity, while CDT-Will values it at $200. Your argument seems to be that, due to uncertainties in real life, the optimal value to place on honesty is somewhere between zero and infinity, but not at either extreme. Which is true. [...] I guess FDT-hostile is too strong a way to put it, since it implies the problem is an unfair problem. But as @papetoast said, there are some problems that FDT does better on, and some that CDT does better on (like the variant where Derek is misanthropic), and this one is one that CDT does better on. [...] Oops, I wasn't aware that there was a distinction between the transparent-box version and the opaque-box version. Thank you for the correction. [...] I am confused as to what you mean. CDT and FDT-emulating-CDT act the same, so they're equally honest and get equally as much benefit from honesty. Is this about this specific problem? Or all similar problems? But this doesn't seem to be load-bearing, so, whatever. [...] True, the $200 honesty-value seems to be there just to make CDT act more-FDT-like. [...] Dang, you're right, I really should have noticed that. [...] Wait, does the $200 honesty-value actually matter here? It doesn't seem like it changes the results of the hypothetical if you remove it, and removing it would make it easier to reason about. [...] It seems that if Derek didn't value honesty so hi

2DanielW6mo

I don't think it is approximating FDT. I think it is just different values. Laws and policies may make CDT agents approximate what FDT agents would do without those laws, but that is not what I mean. Real humans have complex sets of desires/utility functions. [...] I think this may be a central confusion. You are misunderstanding the hypothetical somewhat. FDT-Will values honesty at $200. He and CDT-Will would both be willing to be dishonest if it got him a >$200 payoff. To take my prior example, if someone dropped a $100 bill, he would return it. But if they dropped >$200, he would pocket it. The reason he is willing to pay up to the value of his life +$200 is because his assessment of the value is not based on how he values honesty, it is only based on how he expect agents like himself to be treated. [...] Out of order, but i think it is more relevant here. Absent the $200, Will is monetarily better off (since Derek would drive them back anyway). It is to show in real scenarios what factors the other party might use to determine how much to demand of the hitchhiker. In the original, why the driver asks for what he does is ignored. Realistically, people don't set prices at random. The value of the $200 is meant to show price setting behavior in a more realistic CDT environment. It is not relevant to CDT winning. CDT wins in the given because Derek is a Decent Driver (hence the name). If Derek wasn't a decent guy, CDT would still win if (and only if) they valued the signaling + honesty greater than Derek thought driving back was costly. Derek loses every dollar he values honesty more than Derek sees saving him as costly (though it doesn't affect his actual expected value payoff, just amount that is actual cash money). But I agree on the complexity. I guess it wpuld have been better to first present Will as a simply agent with no external values and then show how he would behave under CDT with more realistic values. But the more realistic values are what I'd a

2Firmament6mo

I mean, my intuition says that the right utility function can turn any CDT agent into an FDT agent, and any FDT agent can be described in terms of a CDT agent with a certain utility function. Like, CDT will one-box if it intrinsically values one-boxing in Newcomblike problems. So, a human with weird desires for justice and a human running FDT act the same if you squint. [...] I'm making a specific claim about this specific scenario. I agree that both CDT-Will and FDT-Will will pick up $300 on the ground and keep it. But in this scenario, back in the desert, all parties involved hash out exactly what they're going to do in the future. So if both Derek and Will were running CDT, but they were to value honesty at infinity dollars, then they would act exactly the same as if they both ran FDT but valued honesty at zero dollars. So the honesty parameter acts as a way to interpolate between CDT and FDT. [...] This seems like the crux of the issue. I think that the whole reason Will is in this mess is because Derek places a value of $1,000,000 on trust for some reason, making him act exactly like an FDT agent. If the tables were turned, and we got rid of the "no negotiation allowed" rule, and Derek was a $0 honesty CDT agent and Will was an FDT agent (or alternatively a CDT agent with a high value on honesty), then Will could say "I precommit to not letting you drive me to to town unless you pay me $0.99 right now" (we assume Derek has money on him or that Will is capable of somehow paying Derek a negative amount in town) and then Derek would have no choice but to comply. And if instead both agents ran dishonest CDT, then words mean nothing and Derek would silently drive Will to town for the $1 altruism utility. So the moral of the story is that whoever runs FDT wins, with ties broken by whoever has the information advantage. The magnitude of "winning" is very different, because FDT-Derek-FDT-Will ends up netting Derek a million dollars while FDT-Derek-CDT-Will only g

2DanielW6mo

I don't think that follows. The CDT and FDT agents have the same utility functions and behave differently. Of course, if you gave them different tailored utility functions you could get them to behave the same in any given case, but that doesn't seem very sensible, imo. [...] In that case, Derek could demand infinite dollars from Will and Will would pay it. [...] If you remove Derek valuing honesty, his optimal decisions work out identically, as I daid in the OP. I made him value it trivially highly so I didn't have to include a discussion of those scenarios to show they are suboptimal for Derek, but you can calculate his EV yourself, they will always be less than thr scenarios I described in the OP. [...] Derek's honesty value doesn't affect those scenarios. In a negotiation, the turn order, information asymmetry etc determine who wins. [...] PEr the original problem, Derek's optimal move is identical under CDT and FDT. How much he can get from Will is the only variable which depends on Will's utility calculations. This isn't a tiebreaker, it is what value they ascribe to different scenarios. Since CDT-Will's posterior calculation is limited to his causal effects, the value that can be extracted from him is much lower.

1Firmament6mo

What I mean is that you can think of "CDT agent with certain utility function" and "FDT agent" as exactly the same. They're the same concept. So when you say "I don't think it is approximating FDT. I think it is just different values." I reply that "different values" and "approximating FDT" are the exact same thing, at least in the case where the mentioned "different values" are "justice, trust, and honor", in my opinion. [...] Well, Will only values his life at a million dollars, so he would rather die than pay more than that. I admit that when I wrote that bit I was mentally conflating "one million" and "infinity" to simplify reasoning. Hopefully all the other shortcuts I'm using don't break anything. Intuition says the "infinity" here comes from Derek and Will's infinitely accurate predictions. As in, if the predictions were less than infinitely accurate, then you would need less than infinity dollars of honest-value to make CDT act like FDT. Dunno if that's true and it doesn't matter if it does, so, whatever. [...] I should have clarified more, oops. I was talking about a minor variation of the scenario where the "negotiation is not possible" restriction is lifted (while still keeping the information asymmetry somehow). In this case, with no other changes, the problem is basically the same, since Derek just says "btw I swear on God almighty that I am not negotiating at all, since this way I get the best outcomes" and then the rest of the scenario plays out the same (as long as we posit that Will's memory of this exchange is magically erased and so FDT-Will doesn't consider changing his behavior to get a better deal) And meanwhile if Derek's $1,000,000 value on honesty is set to $0 BUT he uses FDT then the exact same thing happens absent any weird commitment-race dynamics with FDT-Will Meanwhile if Derek has $0 honesty and is CDT and he says "btw no negotiation" then FDT-Will can say "no, screw you, we're negotiating or I swear I will bury my head in the s

1DanielW6mo

Also, putting this in another post since I think it is a major point, if we assume some cost to bargaining, for Derek it approximates something like a dove-hawk game, where Derek gets the first move. Will's game is more complex as he is operating under information assymetry, so depends on the odds he assigns some probailities If we consider the value Will pays as X (negative if Derek pays Will), if we assume some cost (C) of both negotiating the outcome, the payoffs works out to (I don't know how/if you can put tables into comments so I just have to write them out): Payoffs given FDT-Will with Negotiation: (1) Will accepts the initial offer (for FDT-Will, X = 1,000,199.99): * Derek: 1 + 1,000,199.99 * Will: 0 * -1,000,000 + 1 * -999,999.99 (2) Will Contests and Derek Accepts (say X = -0.99[1]): * Derek: 1 - 0.99 * Will: P(Derek rejects)*-1,000,0000 + P(Derek accepts)*(200.99) + P(Derek contests)*( (3) Will) (3) Will and Derek contest over X. X is unspecified under the assumptions, any number where X > (C - 1) and X < (1,000,200 - C) is feasible: * Derek: 1 + X - C * Will: P(Derek rejects)[2]*-1,000,0000 + P(Derek accepts)*(200- X) - C Counterfactual: Derek doesn't offer an amount and Will doesn't contest (X = 0) * Derek: 1 * Will: 1,000,200 Payoffs given CDT-Will with Negotiation: (1) Will accepts the initial offer (for CDT-Will, X = 199.99, since anything greater wouldn't be paid): * Derek: 1 + 199.99 * Will: 0 * -1,000,000 + 1 * 0.01 (2) Will Contests and Derek Accepts (X = -0.99): * Derek: 1 - 0.99 * Will: P(Derek rejects)*-1,000,0000 + P(Derek accepts)*(200.99) + P(Derek contests)*( (3).Will) (3) Will and Derek contest over X. X is unspecified under the assumptions, any number where X > (C - 1) and X < (200 - C) is feasible: * Derek: 1 + X - C * Will: P(Derek rejects)*-1,000,0000 + P(Derek accepts)*(200- X) - C Counterfactual: Derek doesn't offer an amount and Will doesn't contest (X = 0) * Derek: 1 * Will: 1,000,200 While

1DanielW6mo

They are not. A CDT agent is fundamentally doing a different expected value calculation than a FDT agent. This is why they can lead to radically different outcomes. [...] Okay, play out the scenario. He offers to take CDT will back for $199.99, what does will say? Will's expected values are: 1. "Ok" - payoff = $0.01 2. "No" - payoff = -$1,000,000 (this is assuming an honest/total no, the other case is under 3) 3. "No take me for X amount." - payoff = ???[1] (he doesn't know whether Derek will accept or not, note that this includes the case where X is zero or the cases where X is negative). Now the question becomes "how does Will estimate the payoffs for 3?" What is his expectation for Derek to negotiate? Etc. If we assume sufficient risk aversion (which I would argue is the most probable outcome) 1 is still preferable. Let's imagine Derek offers to take FDT will back for $1,000,199.99. Will's expected values are: 1. Be an agent that would say "OK" - payoff = -$999,999.99 2. Be an agent that would say "No" - payoff = -$1,000,000 3. Be an agent that would say "no take me for X amount" - payoff = ??? 4. Edit: Be an agent that would say "Ok" but not actually pay Derek - payoff = -$1,000,000 (I realize I forgot to include this one, as in Parfit's Hitchiker, the agents' expected outcome if they wouldn't pay Derek honestly is that they would be left to die)[2] FDT-Will has the same problem as CDT-Will. Though, for FDT-Will, unlike CDT-Will, I would argue under most reasonable assumptions there would be some preferable value for X under 3 that FDT-Will would estimate has a better expected value. Given that, he would try to negotiate a value somewhere between -$1 and $1,000,199.99. Where he would negotiate that value depends on risk aversions and how he estimates the responses from Derek. [...] I am not convinced this is true. I don't see why FDT-Derek would behave differently. If we assume information symmetry, then you get the same commitment race with the

1Firmament6mo

I don't have the time to give this full consideration, but on the whole I think you are correct if Will has the information asymmetry in both the negotiation phase and the payment phase, whereas I was implicitly assuming Will having full information in negotiation and suddenly gaining an information asymmetry in the payment phase (which doesn't make much sense). So, yeah, I think I agree.

1papetoast6mo

I feel like in general when unbeknownst to you you have a hostile telepath inspecting you, you are just fucked in arbitrary ways that are decision theory-agnostic. Completely speaking from intuition, this is very close to (but definitely not identical to) the no free lunch theorem where any DT benefits you in some universes and hurts you in others, in a roughly but probably not exactly symmertric way.

2DanielW6mo

I don't think that is quite true. And as I said elsewhere, when we assume non-telepaths we get FDT losing by amounts dependent on the degree of information asymmetry. In this case, the driver, Derek, is able to capture "as much money as the agent will be willing, upon arriving in town, to pay them to prevent the scenario from happening." For CDT, lacking retro-causality, they will only be willing to pay up to whatever their honesty value and signaling value is (i.e. less than the $200 for Will). For the FDT agent, they will be willing to pay up to whatever they value the totality of the outcomes (live and pay vs. die and don't). CDT being willing to be dishonest in retrospect means there is less value to capture from them. In the real world, CDT agents are what we act like. If we want people to be more honest, we try to increase the value of honesty and signaling. To prevent people like Derek capturing this, we put limits on it. One could imagine instead Derek demanding them to sign an contract before saving them. CDT will would now also be willing to pay up to $1,000,200 if that contract would be enforceable. But in the real world, if Derek demanded that much, under U.S. law the contract would likely be thrown out as unconscionable if he demanded in excess of a million dollars for a short car ride. He could probably get away with demanding a contract of a few hundred or even thousand dollars, but if he was charging thousands of times the fair market value for a car ride, any court would likely throw that out. I think that is right, there should be limits on what you can enforce on another party in such situations.

1papetoast6mo

I am again speaking from intuition only and don't want to put more time thinking about this for now. I may not even endorse what I say if I put 5 minutes into thinking. [...] This seems like a good thing [...] This means CDT-Will will die if Derek' has a different utility function and is only willing to drive them home for $201+? This is the "other" universes I'm talking about. In an even more realistic scenario, Will should have a prior for the minimum amount Derek is willing to get to drive them home. I expect this would make FDT-Will get some better calculations.

1DanielW6mo

Why? Take my example with the contracts, I don't think that is actually a good outcome to be able to impose any contract on a disadvantaged party. Having the world of deals you can impose on someone you find at your mercy, so to say, restricted by what is socially permissible and enforceable seems like a preferable state of affairs. Absent legal/social frameworks, enforceability being limited by agent values and willingness to be beholden to deals seems like a preferable state of affairs to such limits not being in place. [...] Yes, if we assume Derek is a misanthrope he will kill Will if WIll is not willing to pay him some amount greater than his misanthropy. But I do not think that is a realistic state of affairs and I think on the flip side you can get asymmetric information causing FDT agents to behave sub optimally when presented with misanthropic actors.[1] [...] In the real world, we are often price takers or price setters and rarely negotiating as equal parties. Will may have the prior in my scenario for what he thinks will would be willing to accept. What his prior is, however, is irrelevant, he is not offered that price and doesn't get to proposition Derek. His only choices are "do I accept Derek's offer?" and when they get to town having accepted the offer he gets to decide "do I honor the offer?" If he wouldn't honor the offer, Derek wouldn't pick him up so he dies. 1. ^ E.g., as the first example that comes to mind, let's say your child has been kidnapped. Your kidnapper just happened to capture your child, by pure chance not intentionally, but you have no way to know that. You think that paying off blackmailers makes it more likely you will be blackmailed. The blackmailer demands a payment (lets say there is an escrow and they cannot cheat), but you, as an FDT agent, decline to negotiate. So the blackmailer kills your kid and disappears. A CDT agent pays the blackmailer, not considering the odds their decision may have on them being bla

2papetoast6mo

I think you are intuiting the question of "which DT is better" using the real world too heavily in a sort of "I think a world where people all do this is better" -> "this DT is better" way. You can't just hope things work out this way. [...] Yes, thats why you use laws / precommitments to prevent it. I guess I used "good" and that misled you a bit, I think it is game theoretically good, not morally ideal. [...] As I said, this is very close to the no free lunch theorem where any DT benefits you in some universes and hurts you in others. I fully expect you can construct a situation including a hostile telepath where DT A outperforms DT B for any A/B. [...] We are assuming Derek knows everything about Will right? So if Will changes his strategy based on his prior then Derek knows that too.

1DanielW3mo

Might make it longer at some point, as there are a lot more nuances, but in case you are interested, I made a post discussing the use of decision theory and my view about how FDT fails to provide the same use, even utilizing the real world cases that Yudkowsky tends to cite for preferring FDT: https://www.lesswrong.com/posts/ALvpivH4Ys6Jp7CKK/the-disutility-of-fdt-on-utility-functions-and-voting

1DanielW6mo

Mostly fair, as i think you said elsewhere, i think I misunderstood you as making a value claim when you meant better in some other terms. But one of the main reasons Yud and Soares give for preferring FDT over CDT is a belief that FDT leads to better outcomes. That is what I find unconvincing. It seems to me that more realistic assumptions better model observations under CDT (e.g. Braess's paradox, to use an exampl I did elsewhere) and can lead to better outcomes. That was my central thesis. I do agree, that it is usually trivial to conceive of scenarios where any given theory loses to another in some sense. [...] Yes, but I would argue it is good to have mediating forces outside of laws. Derek can get either kf them to sign a contract before hand for a $1,000,199, but only FDT would say that they should honor that contract absent any mechanism to enforce it. While I don't think it can be proven, it seems sensible before considering enforcement mechanisms we should consider honoring contracts based on how much we value honesty, associated signals and other such considerations. It seems less sensible to say we should honor them based solely on value estimates of the entire scenario they fall under. It alao seems sensible, if we include enforcement mechanism, that such mechanism be aet up to prevent people not following contracts that are generally deemed not unreasonable and preventing unconscionable conditions from being imposed even on agents that rationally consented to them (as would be the case with the agents consenting to a 1,000,200 contract). [...] You mean Derek knows it, right? But it doesn't change Will's value calculation, so it shouldn't change his strategy a priori even if he had a prior for what he thinks Derek would accept. He would change his decision if we assumed he knew how Derek was likely to price set and adapted his strategy on that, though.

1DanielW6mo

While I didn't explore the general case, which would be easier to if/when I formalize the dilemma, my intuition from the specific thought experiment is that generally when faced with an adversary (that is less than optimally adversarial, as is more common in the real world) that has an asymmetric advantage over an agent (e.g., as in the above case the adversary is a price-setter with an ability to predict agent's decisions) if the agent has some non-zero social values, CDT agents do better than FDT agents. From a policy perspective, it also seems more reasonable to imagine agents under CDT; policies aimed at aligning agent's causal expectations with optimal social outcomes seem more effective at addressing e.g. freeriders. Of course, decision theory looks at problems from the agents perspective, not how we should assume agents are likely to act from a policy perspective, but part of the theoretical utility is in developing models for behavior which can serve a policy perspective. And from there, CDT which is commonly implicit in agent based models, seems to work out better in practice. Braess's paradox which is derived from real world observations is easily explained assuming agents make decisions under CDT, but if we assumed agents acted under FDT, it wouldn't occur. Edit: to be clear, Braess' paradox not occurring would be a good thing, if people made driving decisions in a way that optimized overall traffick that would be better. But in this world we live in it does occur. Also, it is noteworthy that if we imagine individual FDT agents, their utility would likely be unchanged and Braess's paradox would still occur, since they would make their decisions based on the empirically observed behavior of others who don't operate under FDT.

Moderation Log