I'm bumping into walls but hey now I know what the maze looks like.
Yeah I think I'm wrong about this. Thanks to all of you commenters for feedback. I'm updating.
Got it. Thank you.
What specifically? I don't need a long explanation (you can get on with your life), just a pointer.
"Process" and "wants to" are in the map, not the territory. I don't think anyone needs any justification for pointing that discrepancy out. Even if "process" and "wants to" are useful heuristics, I would not be miffed if LW posts resurfaced from time to time to remind everyone that we are not living in the territory here. I explain this in more detail in my response to Razied's comment.
Fair enough. I certainly didn't try to mince words. My goal was to violently shave off any idea of "agency" my friend was giving to evolution. He was walking around satisfied with his explanation that evolution selects for the fittest and is therefore optimizing for the fittest. The point of the dialogue format was to point out that you can call it an optimization process, but when you taboo that word you figure out it's hard to pinpoint exactly what is being optimized for. If you're going to call something an optimization process, you'd better tell me exactly what is being optimized for. If you can't, you are probably using that word as a curiosity stopper or something.
I think we'll be able to pinpoint what evolution optimizes for, someday.  Gravity as a force optimizes for the creation of stars: enough so that loose clouds of hydrogen are pretty much guaranteed to form stars. You could say "gravity optimizes for the creation of stars from hydrogen clouds" and anticipate experience with seamless accuracy. Evolution is like this except it's so much more complex that in order to explain it as an optimization process you'll have to resort to the dreaded word "emergence".
I think there's also something to be said about reminding people from time to time that "optimization pressure" and "emergence" and are in the map, not the territory; the territory is a different beast. I think you could reasonably take on the "true" way of seeing things for an hour or two after reading this post, and then go back to your business believing in the heuristic that evolution is an optimization process (once you've finished with your partial transfiguration).
Note the verb "optimized", which implies that something active is going on.
In fact, most of the work has probably been done by Dawkins and others and there's a mountain of math out there that explains exactly what evolution is optimizing for. If that's the case, I definitely want to understand it someday, and find all of this very exciting. But neither I nor my friend are in a position to explain what evolution is optimizing toward, at least in a way that would let us accurately anticipate experience.
There are versions of the thought experiment where if Omega predicts you will choose to use a randomizer, it won't put the money in box B. But in just the default experiment, this seems like an entertaining outcome!
Haha, well I changed the title and it has 10 more karma than it had yesterday, so there was something up with the title. Thank you again!
Yeah I was wondering what made this post fail and an unclear name might be part of it. Thanks for the feedback!
These are all sub-types of procrastination. In my experience, thinking about this as "procrastination" is less helpful than ignoring that word entirely and finding the specific reason why I'm procrastinating instead. I'm not trying to redefine procrastination, only saying that you may want to taboo it.
Have you read Children of Time, by Adrian Tchaikovsky? It's a beautiful and relatively recent science fiction book and winner of the Arthur C. Clarke Award. It approaches the theme of other beings, artificial consciousness, emergent consciousness, empathy, and quite a few other things. That doesn't entirely cut it, but to me it seems like it is speaking directly to your post.
Without spoiling too much, it follows an event in which engineered retroviruses designed to make apes intelligent, hurled into a newly-terraformed planet by human colonists, accidentally makes the portia genus of jumping spider more intelligent instead. The book launches into imaginative and precise evolutionary worldbuilding, tracing the rise of an entire civilisation swarming with what are to us alien minds. (Portia are more on the level of octopi than bears as far as otherness is concerned.) Portia are a type of jumping spider native to rainforests in the East Indies. Despite having only 200,000 neurons or so, they are considered some of the most intelligent critters at their scale. They sustain themselves entirely off eating other types of spiders, by spending hours calculating the best angle of attack and silently crawling around the enemy web, before attacking at once. They seem to be capable of thinking up plans, and are masters of spatial reasoning (they non-coincidentally have particularly good eyes for their size). The word "arachnid" might send a shiver down your spine right now, but by the end of reading this book, I swear, your arachnophobia will be cured (perhaps not at the instinctual level, but at the intellectual do-I-like-spiders-and-think-they-are-cool level). What would you expect a civilisation of intelligent portia to be like? What threats do you think they would face? Well, jot down your predictions and go find out! https://www.amazon.com/Children-Time-Adrian-Tchaikovsky/dp/1447273303