I observe that many people are driven to have as much impact on the world as possible. This often leads to a sense of urgency especially around AI timelines. To take a random example, here’s jefftk today:
AI is so quickly approaching (or exceeding) expert human abilities across so many areas that most people should be planning for 1-3 more years in which they can productively contribute. Use the time well!
Many people around here are committed to some kind of utilitarian view, where the moral or ethical worth of an action is a function of its consequences. From this point of view, it’s understandable that impact is important; maybe we have a duty to take the actions which produce the best consequences. As a result, your duty could be particularly binding in a critical window: an analogy might be that maybe it's generally virtuous to be willing to fight for your country, but it’s especially important to do so if your country is actually in danger.
If you are working on something directly related to whether AI goes well or poorly, I think this urgency may make sense. But does it actually make sense otherwise?
Say for example, that you are working on some problem in (I don’t know) medicine or mathematics, and you think that in some ~small number of years it will be solved anyway by AI. Why exactly is it important that you rush to make a contribution? After all, if solving problem X is intrinsically valuable, and you are a consequentialist, then we are considering scenario A, with two options
You solve problem X ~now, because you work on it very hard.
AI solves problem X in ~small number of years.
Now, solving X may be very valuable. Maybe solving X is curing some disease that kills a bunch of people a year, so you get a ~small number of person years of utility from X being available a few years early.
However, when we compare that to scenario B where the options are:
You solve problem X ~now, because you work on it very hard.
X remains unsolved for an unknown, but presumably long, amount of time.
then it seems like the expected utility of working very hard on X is actually much greater in scenario B than scenario A. So, from a consequentialist point of view, maybe if you have updated to believing you are in scenario A rather than scenario B, you should instead chill out and relax, rather than feeling a greater sense of urgency?
Now an obvious difference between scenario A and scenario B is personal achievement. That is, even though in scenario A, the actual contribution to global welfare was smaller in scenario B, by solving problem X you still did something morally valuable. The danger seems to be that, once AI has solved X, there doesn’t remain anything valuable for you to do; what you do doesn’t matter.
However, if you are a consistent viewpoint-indifferent consequentialist, why should you care about this so much? Especially if your consequentialism is expressed purely in terms of happiness and suffering? As the only change in happiness or suffering between the two scenarios we've been considering is the personal sense of achievement of the hero.
One obvious answer is that people intrinsically value what they do mattering; they want to be a “speaker of words and a doer of deeds”; to be seen and recognised as having contributed to a worthy problem. In the End of History and the Last Man, Fukuyama calls this urge ‘thymos’, and notes that liberal thought, of which utilitarianism is certainly a descendant, has generally neglected its importance. I think this is what I find curious about this: although utilitarianism doctrinally doesn’t place any particular intrinsic value on heroism, utilitarians typically have a revealed preference for it: they don’t just want the greatest happiness to be attained, but to personally help attain it: not just for lots of happiness to be about, but to be personally responsible for causing it. In the words of Gavin Belson from Silicon Valley, "I don't want to live in a world where someone makes the world a better place, better than we do."
Now, personally, I do suspect that thymos is important. But one problem with assigning actual value to thymos, is that it makes a post-instrumental AI future look extremely bad even if it’s ‘successful’ ; it will significantly reduce the opportunities for people to perform acts of genuine moral worth. If heroism is important in its own terms and not just because of its consequences, then a world stripped of opportunities for human heroes might be bad, even if these 'last men' live lives of considerable material comfort. I think a lot of people feel a horror of this post-instrumental condition subliminally; hence the urgency to ‘grab’ some heroic deed while there is still time. But then, is a utopia where opportunities for heroism are all artificial really such a good thing?
There would presumably still be be opportunities for 'artificial' heroism, such as being really good at sports or games, or succeeding at personal challenges like running or learning a musical instrument, in a post-instrumental future. This is often proposed (not always entirely convincingly) as a solution to this problem by those who have considered this, like Bostrom in Deep Utopia, or Dario Amodei in machines of loving grace. I think this might work, but I notice that relatively few people are responding to an impending collapse of opportunities for heroism by brushing up on their chess skills, or cultivating a culture that recognises purely non-functional achievements. Maybe this would be a more productive response than a rush to devour the last scraps of heroism? Unless one suspects that the real thing tastes much better than the artificial substitutes.
I observe that many people are driven to have as much impact on the world as possible. This often leads to a sense of urgency especially around AI timelines. To take a random example, here’s jefftk today:
Many people around here are committed to some kind of utilitarian view, where the moral or ethical worth of an action is a function of its consequences. From this point of view, it’s understandable that impact is important; maybe we have a duty to take the actions which produce the best consequences. As a result, your duty could be particularly binding in a critical window: an analogy might be that maybe it's generally virtuous to be willing to fight for your country, but it’s especially important to do so if your country is actually in danger.
If you are working on something directly related to whether AI goes well or poorly, I think this urgency may make sense. But does it actually make sense otherwise?
Say for example, that you are working on some problem in (I don’t know) medicine or mathematics, and you think that in some ~small number of years it will be solved anyway by AI. Why exactly is it important that you rush to make a contribution? After all, if solving problem X is intrinsically valuable, and you are a consequentialist, then we are considering scenario A, with two options
Now, solving X may be very valuable. Maybe solving X is curing some disease that kills a bunch of people a year, so you get a ~small number of person years of utility from X being available a few years early.
However, when we compare that to scenario B where the options are:
then it seems like the expected utility of working very hard on X is actually much greater in scenario B than scenario A. So, from a consequentialist point of view, maybe if you have updated to believing you are in scenario A rather than scenario B, you should instead chill out and relax, rather than feeling a greater sense of urgency?
Now an obvious difference between scenario A and scenario B is personal achievement. That is, even though in scenario A, the actual contribution to global welfare was smaller in scenario B, by solving problem X you still did something morally valuable. The danger seems to be that, once AI has solved X, there doesn’t remain anything valuable for you to do; what you do doesn’t matter.
However, if you are a consistent viewpoint-indifferent consequentialist, why should you care about this so much? Especially if your consequentialism is expressed purely in terms of happiness and suffering? As the only change in happiness or suffering between the two scenarios we've been considering is the personal sense of achievement of the hero.
One obvious answer is that people intrinsically value what they do mattering; they want to be a “speaker of words and a doer of deeds”; to be seen and recognised as having contributed to a worthy problem. In the End of History and the Last Man, Fukuyama calls this urge ‘thymos’, and notes that liberal thought, of which utilitarianism is certainly a descendant, has generally neglected its importance. I think this is what I find curious about this: although utilitarianism doctrinally doesn’t place any particular intrinsic value on heroism, utilitarians typically have a revealed preference for it: they don’t just want the greatest happiness to be attained, but to personally help attain it: not just for lots of happiness to be about, but to be personally responsible for causing it. In the words of Gavin Belson from Silicon Valley, "I don't want to live in a world where someone makes the world a better place, better than we do."
Now, personally, I do suspect that thymos is important. But one problem with assigning actual value to thymos, is that it makes a post-instrumental AI future look extremely bad even if it’s ‘successful’ ; it will significantly reduce the opportunities for people to perform acts of genuine moral worth. If heroism is important in its own terms and not just because of its consequences, then a world stripped of opportunities for human heroes might be bad, even if these 'last men' live lives of considerable material comfort. I think a lot of people feel a horror of this post-instrumental condition subliminally; hence the urgency to ‘grab’ some heroic deed while there is still time. But then, is a utopia where opportunities for heroism are all artificial really such a good thing?
There would presumably still be be opportunities for 'artificial' heroism, such as being really good at sports or games, or succeeding at personal challenges like running or learning a musical instrument, in a post-instrumental future. This is often proposed (not always entirely convincingly) as a solution to this problem by those who have considered this, like Bostrom in Deep Utopia, or Dario Amodei in machines of loving grace. I think this might work, but I notice that relatively few people are responding to an impending collapse of opportunities for heroism by brushing up on their chess skills, or cultivating a culture that recognises purely non-functional achievements. Maybe this would be a more productive response than a rush to devour the last scraps of heroism? Unless one suspects that the real thing tastes much better than the artificial substitutes.