Nice project! Good luck!
As I currently understand it, your plan is to write an essay arguing that humanity should devote lots of resources to figuring out how to steer the far future? In other words, "We ought to steer the near future into a place where we have a better idea of where to steer the far future, and then steer the far future in that direction."
Your argument would have to center around the lemma that we currently don't know what we want, or more poetically that our power far outstrips our wisdom, so that we ought to trade off the one for the other. I very much agree with this, though I would be interested to see how you argue for it. Keep us posted!
As for your questions, I'm afraid I don't have much help to offer. If you haven't already you should take a look at surveys of population ethics, meta-ethics, and normative ethics literature. And I don't understand your second question; could you elaborate?
If you haven't already you should take a look at surveys of population ethics, meta-ethics, and normative ethics literature.
Could I get some links, please? I'm a CS grad student by trade, so academic literature on ethics is an Outside Context thing to me.
And I don't understand your second question; could you elaborate?
Imagine that I know what I want very well, but do not possess a "blank" AGI agent (or "blank genie" as such things were previously described) into which my True Wishes can be inserted to make them come true. What other modes of implementation might be open to me for implementing my True Wishes?
As I currently understand it, your plan is to write an essay arguing that humanity should devote lots of resources to figuring out how to steer the far future? In other words, "We ought to steer the near future into a place where we have a better idea of where to steer the far future, and then steer the far future in that direction."
Certainly this is how I want to describe our steering towards the near-future. "Far future" depends on the definition: without some kind of Blank Optimization Process like an AI, it's hard to claim that we can steer our future 100 million years ahead of time, say, rather than being randomly wiped out by an asteroid one day 200,000 years from now, or reverting to such normal, predictable patterns of human behavior that history degrades into non-progressing cycles, etc.
Ah, but that's a thesis in itself! "If we don't truly know what we want, even the mid-term future, let alone the far future, risks becoming mere chaos! Progress itself is at risk!"
Sorry, no links--I'm an outsider to current meta-ethics too. But I'm sure people have thought about e.g. the question "How am I supposed to go about figuring out what is good / what I value?]" Rawls, for example, famously introduced the notion of "Reflective Equilibrium," which is a good first pass at the problem I'd say.
Imagine that I know what I want very well, but do not possess a "blank" AGI agent (or "blank genie" as such things were previously described) into which my True Wishes can be inserted to make them come true. What other modes of implementation might be open to me for implementing my True Wishes?
It sounds like you are asking how we could get our goals accomplished without AI. The answer to that is "The same way people have accomplished their goals since the dawn of time--through hard work, smart economic decisions, political maneuvering, and warfare." If you are asking what the most effective ways for us to implement our True Wishes are, after AGI, then... it depends on your Wishes and on your capabilities as a person, but I imagine it would have to do with influencing the course of society at large, perhaps simply in your home country or perhaps all around the world. (If you don't care so much what other people do, so long as you have a nice life, then the problem is much simpler. I'm assuming you have grand ideas about the rest of the world.)
Making lots of money is a really good first step, given capitalism and stable property rights. Thanks to increasing democracy and decreasing warfare in the world, getting many people to listen to your ideas is important. In fact, if you know what your True Wishes are, then you are probably going to be pretty good at convincing other people to follow you, since most people aren't even close to that level of self-awareness, and since people's True Wishes probably overlap a lot.
Edit: P.S. If you haven't read Nick Bostrom's stuff, you definitely should. He has said quite a bit on how we should be steering the far future. Since you are on LW it is highly likely that you have already done this, but I might as well say it just in case.
In fact, if you know what your True Wishes are, then you are probably going to be pretty good at convincing other people to follow you, since most people aren't even close to that level of self-awareness, and since people's True Wishes probably overlap a lot.
I don't believe the Fascist Dictator route is ethically desirable. This is not to accuse you of encouraging me to become a fascist dictator, but to point out that mind-killed, group-thinking, affective-death-spiralling cults driven by the rhetoric and charisma of even one Truly Clever, Truly Inspired Leader are a sociological attractor we must proactively avoid. Otherwise, they happen on their own the instant some poor sod with more intelligence, charisma, and inspiration than wisdom starts preaching his ideas to the masses.
True, we ought to proactively avoid such cults. My suggestion was merely that, in the event that you figure out what your True Wishes are, persuading other people to follow them is an effective way to achieve them. It is hard to get something accomplished by yourself. Since your True Wishes involve not being a fascist dictator, you will have to find a balance between not telling anyone what you think and starting a death cult.
Perhaps I shouldn't have said "convincing other people to follow you" but rather "convincing other people to work towards your True Wishes."
Sir Karl Popper's book 'In Search of a Better World' (among many others of his work) contains clear and convincing arguments against utopias with their unintended consequences as well as a possible means to limit the harm caused by political action. I would feed an AI with Popper's ethics as a start toward making it not unfriendly.
That recommendation is either a very serious compliment for Popper, or an incredible folly regarding AI. Or both.
Anyway, I'll pick it up. I always did like the antitotalitarians (was raised on them, sort of), even if I now consider them a bit pessimistic. That is, observing that human values are complex, multifaceted, and require reflection doesn't really mean they're impossible to fulfil, merely that we must be very careful not to walk into a self-laid, self-sprung trap.
Maybe check out Koheleth/Qoheleth/Ecclesiastes. It considers and rejects various conceptions of the good life before settling on something vaguely Stoic. The bits about God read like later emendations.
Large parts of the so-called Wisdom in Qohelet actually come from earlier Mesopotamian and Canaanite texts the Jewish sages didn't want to admit to having read. The earlier versions are actually considerably more fun: the Epic of Gilgamesh advises that the lot of mortals is, basically, to eat, drink, and be merry -- not that bad if you add some puzzles and an exercise routine!
Qohelet, on the other hand, shows an immense cognitive bias that we're really going to have to work around if we ever want a genuinely useful ethics: depression. However supposedly Stoic the ancients may have preached about being, starve them to death or cut off their heads and they'd complain. Clearly there was something they actually cared about, but because they were, like most of us, all too human, they could only really tell you what it was in the negative. They didn't want to die (at least, not with much suffering), they didn't want to be enslaved, they didn't want to be in debt, they mostly didn't want the lifestyle they were accustomed to disturbed in any way.
But if you had asked them if they wanted chocolate or vanilla ice-cream, they'd have had a hard time making decisions.
Tell them God commands a war on the other hand, and aha, now the world is making moral demands on them! Now they can stop mucking about having a good time and really get something done!
And this is why I am reluctant to cite Qohelet. Anyway, Nisan, how goes the model-theoretic semantics work?
The bits about God read like later emendations.
The wisdom literature is pretty heterodox, yeah.
So someone I follow on Facebook linked to this essay contest on the subject of: How Should Humanity Steer the Future? My urge to make a joke of it immediately kicked in. The impulse to joke turned into an impulse to actually submit an essay when the words "steering the future" set off a minor "ding!" in my head.
At least regarding AI, many papers and articles have already been published on what that problem is: even well-intentioned people could accidentally create a Completely or Partially Alien Mind-Design that has no care and no sympathy for us or our values. Should such a thing grow more powerful than us, it would proceed to kill us all. We would be in the way, as a bacterium is to us, and dealt with identically.
Blah blah blah blah.
To me personally, by sheer madness of personal preferences, that is not the interesting part. Danger, even existential danger, seems to me quite passe these days. In only the time I've been alive, we've been under threat of nuclear apocalypse via war between nation-states or nuclear-weapons usage by terrorist groups, global warming and other environmental damages are slowly destroying the only planetary-level habitat we humans have, and in the past five or so years we've been dealing with continental-level economic collapses and stagnation as well (I personally subscribe to the theory that capitalism itself is a human-Unfriendly optimization process, which is partially apropos here). Those are just the apocalypses: then we have to add in all the daily pains, indignities and deaths suffered by the vast majority of the world's people, many of whom are so inured to their suffering that they consider it a normal or even morally appropriate part of their lives. Only at the end of that vast, astronomical summation can we say we have totalled humanity's problems.
All that, only in the time I've been alive, for about 25 years now, even when, statistically speaking, I'm living in a steadily improving golden age (or at least a gilded age that's fighting to become a golden age). No previous generation of humanity was ever so rich, so educated, so able to dream! Why did I constantly feel myself to be well-past the Despair and Cynicism Event Horizons?
Which left the younger me asking: what's the point, then? How can it be that "everything's amazing and nobody's happy"? What have we failed to do or achieve? Are we failing to find meaning in our lives because we don't serve God? But God is silent! Do our social structures pervert our daily lives away from the older, deeper "red in tooth and claw" human nature? But the utility increases are clear and demonstrable (besides which, those ideologies sicken me, personally)!
Enter the project to solve meta-ethics, normally considered the key to building an AI that will not go obviously and dramatically human!wrong in its operation. Or, as I would put it, the project to fully and really describe the set of all wishes we actually want to make, by indirection if necessary.
Which is what's so much more interesting to me than mere AI safety engineering. Not that I want Clippy to win, but I would certainly be happier with an outcome in which Clippy wins but I know what I was shooting for and what I did wrong, versus an outcome in which Clippy wins but I die ignorant of some of What is Good and Right. Dying in battle for a good cause is better than dying in battle for a bad cause, and actually dying at home as a helpless victim because I couldn't find a good cause is preferable to damaging others for a bad cause!
After all, how can my younger self be anti-utopian, anti-totalitarian and pro-revolutionary? He considers the existing social conditions in need of overthrow and radical improvement, and yet cannot define the end goal of his movement? Goodness, light, and warmth are like pornography: we know them when we see them (and still be told that instinctive warm-fuzzy recognition doesn't even resemble actual goodness?)? How then, might I today recognize these things when I see them, and do so smartly?
Hence my desired essay topic, and hence my asking for advice here. It's easy to recapitulate the Oncoming AI-Induced Omnicide Problem, but I don't want to go at it from that angle. Inherently, we should wish to know what we wish for, especially since we already possess powerful optimization processes (like our economies, as I said above) well below the superintelligent level. If someone wants to ask about steering the future, by now, my instinctive first response is, "We must know where we actually want the future to go, such that if we really set out for there, we will look forward to our arrival, and such that once we get there, we will be glad we did, and such that we make sure to value our present selves while we're on the journey. OR ELSE." I want to use the essay to argue this view, with reference not only to AI but to any other potential application for a partial or complete descriptive theory of meta-ethics.
Questions (or rather, Gondor calling for aid):
Thanks!