I welcome thoughts, criticisms, etc.
What audience are you writing for? Your current writing style has the advantage that it is relatively clear and gives you an enviable air of objectivity, but the disadvantage that it's somewhat emotionless. Compared to, say, Eliezer's writing, it's hard to get a sense for how much you care about the points you're making. This is an issue if you intend to reach a broad-ish audience but less of an issue if your intended audience is people approximately as good at taking ideas seriously as you are.
I don't really understand your argument about progress. Right now it reads to me like "assuming that net progress is approximately zero, we conclude that net progress is approximately zero."
In particular, for a broad range of values, the first thing to do is to establish a stable, technologically sophisticated civilization at a large scale, which can then direct its action on the basis of careful argument and reflection.
This is unclear to me unless you say something about how coordination problems are solved in such a civilization. Increasing the scale of civilization increases the scale of the coordination problems you need to solve, which is not obviously a good thing. Plausibly a civilization which is too large may face coordination problems too intractable to make certain kinds of changes which may end up being extremely valuable. (Plausibly we already live in such a civilization.)
What audience are you writing for?
People who expect that careful thought is useful for world-bettering, and are specifically looking for it (to guide decisions, validate a community, or seed higher quality discussions). I am not willing to trade objectivity for emotion at the usual exchange rate, though I think it's worth keeping your point in mind---right now I'm probably passing up very cheap opportunities to be more engaging.
I don't really understand your argument about progress. Right now it reads to me like "assuming that net progress is approximately zero, we conclude that net progress is approximately zero."
My argument seems to be poorly articulated and perpetually confusing. I'll take another pass at clarifying it.
But I don't see how it assumes that net progress is approximately zero---all of the assumptions concern the intrinsic value of the events of the next year, which seems orthogonal to "progress." But I agree that the argument does somehow feel like question-begging (though I think it does no such thing).
ETA: I updated that post, if you want to take another look at it.
Increasing the scale of civilization increases the scale of the coordination problems you need to solve, which is not obviously a good thing.
Yes. I explicitly disavowed this view (that progress for progress' sake is good) in the earlier post you mentioned.
This line probably has an extra comma---"which can then direct its action..." is an extra quality, and I'm packing a lot into that + "stable."
With respect to sculpting social values, I mention this as a pressing question. Other than that, do you disagree with the substance of what I've written? It was intended to be a concise statement of the basic principle (building a stable civilization that reflects our values is good, whatever those values turn out to entail), though I'll make it more careful.
(My confusion with the post is that I'm not sure what do you mean by "value". Today and tomorrow are not alternatives in any straightforward decision problem ("waiting a day" doesn't make them into alternatives either, it doesn't even seem to be a relevant decision, more of a personal identity word game; if you wait for tomorrow, today still happens). One possibility is that you might compare their potential as optimization targets, so that the decision in question is where to focus your optimization effort, but the wording of the post doesn't seem to naturally fit this interpretation.)
Consider the choice: cause the world fo today to exist, or the world of tomorrow.
We often talk about whether X or Y is better when they are not alternatives in a straightforward decision problem, for example is it better to receive an A- or a B+ on a test, and how much? We don't choose between X or Y directly, but considering their comparison helps provide intuition about more complicated comparisons between possible outcomes. We don't choose between the world of today or the world of tomorrow, but we also don't choose between performing better or worse on a test.
Consider the choice: cause the world of today to exist, or the world of tomorrow.
What actual choice is informed by considering this decision problem, what motivates posing it? (I don't know, which is why I originally ruled that out as a possible interpretation, and suggested a less straightforward interpretation that involved an actual choice.)
We don't choose between the world of today or the world of tomorrow, but we also don't choose between performing better or worse on a test.
(I'm not appealing to the impossibility-in-practice of choosing today vs. tomorrow, so the distinction between solvable and unsolvable decision problems doesn't seem relevant. The difficulty is that I'm not seeing the motivation for the problem.)
Edit: New hypothesis (which seems to fit with your reply to Qiaochu): You are talking about "progress" in the post, so the implicit choice in question might be between keeping the world closer to the past and pushing it faster closer to the future, so you are asking which is better and how much better. This makes sense.
What actual choice is informed by considering this decision problem, what motivates posing it? (I don't know, which is why I originally ruled that out as a possible interpretation, and suggested a less straightforward interpretation that involved an actual choice.)
Consider economic development, population growth, technological development, basic science progress. My intuition is that all of these things are good, but that seems at odds with this analysis. If the new analysis is right, that would cause me to reconsider those valuations, which are relevant to things like "how useful is the science I do" or "how useful is improving the pace of scientific progress in general" or etc.
I think your interpretation at the end is correct.
I think your interpretation at the end is correct.
OK.
Your conclusion still leaves me confused. In any given choice considered by itself, the scale of value also doesn't make sense, you need to compare with something outside of that choice to say that the difference between the available options is insignificant. In a decision problem, you zoom in, not give up. So what is the difference between today and tomorrow insignificant in comparison with? Any personal-level change is much smaller, likely also much smaller than predictable differences in the world if I had bothered to find them. So to make that judgment, it seems necessary to involve something like the (low) expected impact of your actions on progress, and at that point I lose track of the (hypothetical) argument.
The conclusion is that the effects of progress are small compared with anything that has an appreciable effect on the future (for someone with aggregative, time-insensitive values). If we break down an action as the sum of two changes---one parallel to progress, one orthogonal to it---the effects of the orthogonal part are typically going to be much larger than the effects of the parallel part.
Originally I ended with a discussion of x-risk reduction, but it became unwieldy and I didn't want to put time in. Perhaps I should end with a link to some discussion of future-shaping elsewhere.
Maybe it would help if you cited Nick Bostrom's differential technological development and Luke Muehlhauser and Anna Salamon's differential intellectual progress explained how your idea is related to them?
It seems like you're drawing the same conclusion as them but perhaps through a new argument, but it's confusing that you don't cite them. It's also unclear to me whether the division of changes into V1, V2, V3 in that post is a useful one. I think the classification you gave in a later post makes a lot more sense (aside from the missing "philosophical progress" which you've since added).
ETA: I updated that post, if you want to take another look at it.
I'm still confused about this post. First, I am no longer sure what you mean by "progress." Second, I am no longer sure what you mean by "the value of the world of today" or "the value of the world of tomorrow."
This line probably has an extra comma---"which can then direct its action..." is an extra quality, and I'm packing a lot into that + "stable."
Got it.
By "progress" I mean the usual thing, e.g. "forward or onward movement toward a destination." I never use it in the body of the post though, so hopefully it's not too bad if it is unclear.
By "the value of the world of today" - "the value of the world of tomorrow" I mean: the value of the change between the world of today and the world of tomorrow. How much would you pay to cause that change to happen, right now?
It's not literally an option on the table, but that's nearly always the case when you ask "how valuable is X?" It's just one part of a larger set of consequences of a decision. For example, I could say "what is the value of increasing the world's population by 1%?" and that would be meaningful (or not meaningful) in exactly the same way. Any change that increases the population by 1% will also have other impacts, but it is helpful to break the change down as a sequence of changes (e.g. boosting the population, changing demographics, etc.)
I seem to have some chronic difficulty explaining this :) There are similar problems when I try in person. I assume it's something odd or wrong about how I think about this issue in particular.
I think Vladimir and I have similar confusions but he seems to be describing them more clearly so I'll let him handle this conversation.
Second, I am no longer sure what you mean by "the value of the world of today" or "the value of the world of tomorrow."
(Tomorrow and the rest of the future are not alternatives in any straightforward decision problem, so I'm guessing the point of comparing them is about the question of which is the richer optimization target, probably in the sense where you don't compare the difficulty in having an impact.)
Over the last few months I've started blogging about effective altruism more broadly, rather than focusing on AI risk. I'm still focusing on abstract considerations and methodological issues, but I hope it is of interest to others here. Going forward I intend to cross-post more often to LW, but I thought I would post the backlog here anyway. With luck, I'll also have the opportunity to post more than bi-weekly.
I welcome thoughts, criticisms, etc.