All of cousin_it's Comments + Replies
Like I said, people who have been outcompeted won't keep owning a lot of property for long. Even if that property is equity in AI companies, something or other will happen to make them lose it. (A very convincing AI-written offer of stock buyback, for example.)
Even if you have long term preferences, bold of you to assume that these preferences will stay stable in a world with AIs. I expect an AI, being smarter than a human, can just talk you into signing away the stuff you care about. It'll be like money-naive people vs loan sharks, times 1000.
Maybe an even better analogy is non-Euclidean geometry. Agent foundations is studying a strange alternate world where agents know the source code to themselves and the universe, where perfect predictors exist and so on. It's not an abstraction of our world, but something quite different. But surprisingly it turns out that many aspects of decision-making in our world have counterparts in the alternate world, and in doing so we shed a strange light on what decision-making in our world actually means.
I'm not even sure these investigations should be tied to AI...
They think (correctly) that AI will take away many jobs, and that AI companies care only about money and aren't doing anything to prevent or mitigate job loss.
You think AIs won't be able to offer humans some deals that are appealing in the short term but lead to AIs owning everything in the long term? Humans offer such deals to other humans all the time and libertarianism doesn't object much.
OpenAI currently creates a massive amount of value for humanity and by default should be defended tooth and nail.
Interesting how perspectives differ on this. From what I see around me, if tomorrow a lightning from God destroyed all AI technology, there'd be singing and dancing in the streets.
People who have been outcompeted won't keep owning a lot of property for long. Something or other will happen to make them lose it. Maybe some individuals will find ways to stay afloat, but as a class, no.
1
1A happy path: merge with the AIs?
I think if you're serious about preserving human value into the future, you shouldn't start with something like "let's everyone install AI-brain connectors and then we'll start preserving human value, pinky promise". Instead you should start by looking at human lives and values as they exist now, and adopt a standard of "first do no harm".
I think AI risk insurance might be incompatible with libertarianism. Consider the "rising tide" scenario, where AIs gradually get better than humans at everything, gradually take all jobs and outbid us for all resources to use in their economy, leaving us with nothing. According to libertarianism this is ok, you just got outcompeted, mate. And if you band together with other losers and try to extract payment from superior workers who are about to outcompete you, well, then you're clearly a villain according to libertarianism. Even if these "superior workers" are machines that will build a future devoid of anything good or human. It makes complete sense to fight against that, but it requires a better theory than libertarianism.
This seems quite plausible actually. Even without the objective morality angle, a morally nice AI could imagine a morally nice world that can only be achieved by having humans not exist. (For example, a world of beautiful and smart butterflies that are immune to game theory, but their existence requires game-theory-abiding agents like us to not exist, because our long-range vibrations threaten the tranquility of the matrix or something.) And maybe the argument is genuinely so right that most humans upon hearing it would agree to not exist, something like collectively sacrificing ourselves for our collective children. I have no idea how to deal with this possibility.
I think these are all still pretty bad. For example if there are human uploads but no stronger AI, that will lead to horrors ("data can't defend itself" - Sam Hughes). If there are biological superbrains, same. Look at what we humans did with our intelligence, we've introduced a horror (factory farms and fish farms) that surpasses anything in nature. Going forward we must take two steps in morality for every step in capability, otherwise horrors will increase proportionally.
The physical world with all its problems is kind of a marvel, in that it allows a d...
This is like saying "I imagine there will be countries that renounce firearms". There aren't such countries. They got eaten by countries that use firearms. The social order of the whole world is now kept by firearms.
The same will happen with AI, if it's as much a game changer as firearms.
1Yeah. It also worries me that a lot of value change has happened in the past, and much of it has been caused by selfish powerful actors for their ends rather than ours. The most obvious example is jingoism, which is often fanned up by power-hungry leaders. A more subtle example is valuing career or professional success. The part of it that isn't explained by money or the joy of the work itself, seems to be an irrational desire installed into us by employers.
Then it seems to me that judging the agent's purity of intentions is also a deep problem. At least for humans it is. For example, a revolutionary may only want to overthrow the unjust hierarchy, but then succeed and end up in power. So they didn't consciously try to gain power, but maybe evolution gave them some behaviors that happen to gain power, without the agent explicitly encoding "the desire for power" at any level.
I'm no longer sure the problem makes sense. Imagine an AI whose on-goal is to make money for you, and whose off-goal is to do nothing in particular. Imagine you turn it on, and it influences the government to pay a monthly stipend to people running money-making AIs, including you. By that action, is the AI making money for you in a legitimate way? Or is it bribing you to keep it running and avoid pressing the shutdown button? How do you even answer a question like that?
The thesis seems wrong to me. Horror games meet the same kind of need as horror movies do, and horror movies aren't a power fantasy. Puzzle games meet the same kind of need that puzzle toys do, and I never heard of anyone calling them power fantasies. Esport games meet the same kind of need as sports do, and sports doesn't work as a power fantasy for most people. Adventure games meet the same kind of need as listening to a good campfire story, which isn't always a power fantasy either. And then there are driving simulators where you just drive across the U...
Hah. And even if the king had a computer that could simulate how the golem would react to the suicide order, even that wouldn't help the king, if the golem followed updateless decision theory.
I don't fully understand Vanessa's approach yet.
About caring about other TDT agents, it feels to me like the kind of thing that should follow from the right decision theory. Here's one idea. Imagine you're a TDT agent that has just been started / woken up. You haven't yet observed anything about the world, and haven't yet observed your utility function either - it's written in a sealed envelope in front of you. Well, you have a choice: take a peek at your utility function and at the world, or use this moment of ignorance to precommit to cooperate with ever...
I think you can make it more symmetrical by imagining two groups that can both coordinate within themselves (like TDT), but each group cares only about its own welfare and not the other group's. And then the larger group will choose to cooperate and the smaller one will choose to defect. Both groups are doing as well as they can for themselves, the game just favors those whose values extend to a smaller group.
About 2TDT-1CDT. If two groups are mixed into a PD tournament, and each group can decide on a strategy beforehand that maximizes that group's average score, and one group is much smaller than the other, then that smaller group will get a higher average score. So you could say that members of the larger group are "handicapped" by caring about the larger group, not by having a particular decision theory. And it doesn't show reflective inconsistency either: for an individual member of a larger group, switching to selfishness would make the larger group worse ...
Administered by the state, of course. Open air prison where you can choose where to live, when to go to bed and wake up, what to eat, who to work with and so on, would feel a lot less constraining to the spirit than the prisons we have now.
I think that's the key factor to me. It's a bit hard to define. A punishment should punish, but not constrain the spirit. For example, a physical ball and chain (though it looks old-fashioned and barbaric) seems like an okay punishment to me, because it's very clear that it only limits the body. The spirit stays free, yo...
Just my opinions.
-
How an anarchist society can work without police. To me the example of Makhno's movement shows that it can work if most people are armed and willing to keep order, without delegating that task to anyone. (In this case they were armed because they were coming out of a world war.) Once people start saying "eh, I'm peaceful, I'll delegate the task of keeping order to someone else", you eventually end up with police.
-
Is police inherently bad. I think no, it depends mostly on what kind of laws it's enforcing and how fairly. Traffic laws, a
Exile can either be administered by the state, in which case it's an open air prison, or by emergent criminal gangs. Forced labor emerges either way I think.
Maybe one example is the idea of Dutch book. It comes originally from real world situations (sport betting and so on) and then we apply it to rationality in the abstract.
Or another example, much older, is how Socrates used analogy. It was one of his favorite tools I think. When talking about some confusing thing, he'd draw an analogy with something closer to experience. For example, "Is the nature of virtue different for men and for women?" - "Well, the nature of strength isn't that much different between men and women, likewise the nature of health, so maybe virtue works the same way." Obviously this way of reasoning can easily go wrong, but I think it's also pretty indicative of how people do philosophy.
I don't say it's not risky. The question is more, what's the difference between doing philosophy and other intellectual tasks.
Here's one way to look at it that just occurred to me. In domains with feedback, like science or just doing real world stuff in general, we learn some heuristics. Then we try to apply these heuristics to the stuff of our mind, and sometimes it works but more often it fails. And then doing good philosophy means having a good set of heuristics from outside of philosophy, and good instincts when to apply them or not. And some luck, in ...
I'm pretty much with you on this. But it's hard to find a workable attack on the problem.
One question though, do you think philosophical reasoning is very different from other intelligence tasks? If we keep stumbling into LLM type things which are competent at a surprisingly wide range of tasks, do you expect that they'll be worse at philosophy than at other tasks?
This could even be inverted. I've seen many people claim they were more romantically successful when they were poor, jobless, ill, psychologically unstable, on drugs and so on. Have experienced something like that myself as well. My best explanation is that such things make you come across as more real and exciting in some way. Because most people at most times are boring as hell.
That suggests the possibility to get into some kind of hardship on purpose, to gain more "reality". But I'm not sure you can push yourself on purpose into as much genuine panic and desperation as it takes. You'd stop yourself earlier.
Some years ago I made a version of it that works on formulas in provability logic. That logic is decidable, so you can go ahead and code it, and it'll solve any decision problem converted into such formulas. The same approach can deal with observations and probability (but can't deal with other agents or any kind of logical probability). You could say it's a bit tautological though: once you've agreed to convert decision problems into such formulas, you've gone most of the way, and FDT is the only answer that works at all.
Whether or not fish suffer, it seems people are mostly ok with the existence of "suffering farms" as long as they're out of sight. It's just a nasty fact about people. And once you allow yourself to notice suffering, you start noticing that it's everywhere. There's a good book "Pilgrim at Tinker Creek" where the author lives in a cabin in the woods to be close to nature, and then gradually starts to notice how insects are killing each other in horrible ways all the time, under every leaf, for millions of years.
And so, whether or not fish suffer, maybe I sh...
I think the biggest problem with this idea is that, when you summarize a historical situation leading up to a certain event, that information has already been filtered and colored by historians after the event (for example if they were historians for the winning side in a war). It may be very different from what most contemporaries knew or felt at the time.
I think even a relatively strong AI will choose to takeover quickly and accept large chance of failure. Because the moment the AI appears is, ipso facto, the moment other AIs can also appear somewhere else on the internet. So waiting will likely lead to another AI taking over. Acting early with a 15% chance of getting caught (say) might be preferable to that.
Huh? It seems to me that in the deductive version the student will still, every day, find proofs that the exam is on all days.
Wait, but you can't just talk about compensating content creators without looking on the other side of the picture. Imagine a business that sells some not-very-good product at too-high price. They pay Google for clever ad targeting, and find some willing buyers (who end up dissatisfied). So the existence of such businesses is a net negative to the world, and is enabled by ad targeting. And this might not be an edge case: depending on who you ask, most online ads might be for stuff you'd regret buying.
If the AI can rewrite its own code, it can replace itself with a no-op program, right? Or even if it can't, maybe it can choose/commit to do nothing. So this approach hinges on what counts as "shutdown" to the AI.
I don't know if we have enough expertise in psychology to give such advice correctly, or if such expertise even exists today. But for me personally, it was important to realize that anger is a sign of weakness. I should have a lot of strength and courage, but minimize signs of anger or any kind of wild lashing out. It feels like the best way to carry myself, both in friendly arguments, and in actual conflicts.
Yeah, it would have to be at least 3 individuals mating. And there would be some weird dynamics: the individual that feels less fit than the partners would have a weaker incentive to mate, because its genes would be less likely to continue. Then the other partners would have to offer some bribe, maybe take on more parental investment. Then maybe some individuals would pretend to be less fit, to receive the bribe. It's tricky to think about, maybe it's already researched somewhere?
Cochran had a post saying if you take a bunch of different genomes and make a new one by choosing the majority allele at each locus, you might end up creating a person smarter/healthier/etc than anyone who ever lived, because most of the bad alleles would be gone. But to me it seems a bit weird, because if the algorithm is so simple and the benefit is so huge, why hasn't nature found it?
But to me it seems a bit weird, because if the algorithm is so simple and the benefit is so huge, why hasn't nature found it?
How is nature supposed to gather statistical data about the population to determine what the majority allele is?
Coming back to this idea again after a long time, I recently heard a funny argument against morality-based vegetarianism: no animal ever showed the slightest moral scruple against eating humans, so why is it wrong for us to eat animals? I go back and forth on whether this "Stirnerian view" makes sense or not.
Here's a debate protocol that I'd like to try. Both participants independently write statements of up to 10K words and send them to each other at the same time. (This can be done through an intermediary, to make sure both statements are sent before either is received.) Then they take a day to revise their statements, fixing the uncovered weak points and preemptively attacking the other's weak points, and send them to each other again. This continues for multiple rounds, until both participants feel they have expressed their position well and don't need to ...
I think ideas like Nash equilibrium get their importance from predictive power: do they correctly predict what will happen in the real world situation which is modeled by the game. For example, the biological situations that settle on game-theoretic equilibria even though the "players" aren't thinking at all.
In your particular game, saying "Nash equilibrium" doesn't really narrow down what will happen, as there are equilibria for all temperatures from 30 to 99.3. The 99 equilibrium in particular seems pretty brittle: if Alice breaks it unilaterally on roun...
I don't see any group of people on LW running around criticizing every new idea. Most criticism on LW is civil, and most of it is helpful at least in part. And the small proportion that isn't helpful at all, is still useful to me as a test: can I stop myself from overreacting to it?
Civility >>> incivility, but it is insufficient to make criticism useful and net positive.
There is a LOT wrong with the below; please no one mistake this for unnuanced endorsement of the comic or its message; I'm willing to be more specific on request about which parts I think are good versus which are bad or reinforcing various confusions. But I find this is useful for gesturing in the direction of a dynamic that feels very familiar on LW:
I think orthogonality and instrumental convergence are mostly arguments for why the singleton scenario is scary. And in my experience, the singleton scenario is the biggest sticking point when talking with people who are skeptical of AI risk. One alternative is to talk about the rising tide scenario: no single AI taking over everything, but AIs just grow in economic and military importance across the board while still sharing some human values and participating in the human economy. That leads to a world of basically AI corporations which are too strong fo...
If AI-induced change leads to enough concentration of economic and military power that most people become economically and militarily irrelevant, I don't expect democracy to last long. One way or another, the distribution of political power will shift toward the actual distribution of economic and military power.
That's one way to look at it, though I wouldn't put the blame on capitalists only. Workers will also prefer to buy goods and services produced with the help of AI, because it's cheaper. If workers could get over their self-interest and buy only certified AI-free goods and services, the whole problem would stop tomorrow, with all AI companies going out of business. Well, workers won't get over their self-interest; and neither will capitalists.
I think there's no need for secrecy. If AI can develop a datacenter maintained by robots or other similar tech, human companies will be happy to buy and sell it, and help with the parts the AI can't yet do. Think of it as a "rising tide" scenario, where the robot sector of the economy outgrows the human sector. Money translates to power, as the robot sector becomes the highest bidder for security services, media influence, lobbying etc. When there comes a need to displace humans from some land and resources, it might look to humans less like a war and more like a powerful landlord pushing them out, with few ways to organize and push back. Similar to enclosures in early-modern England.
I think if it happens, it'll help shift policy because it'll be a strong argument in policy discussions. "Look, many researchers aren't just making worried noises about safety but taking this major action."
Hm, pushing a bus full of kids towards a 10% chance of precipice is also pretty harsh. Though I agree we should applaud those who decline to do it.
Yeah, it's not the kind of strike whose purpose is to get concessions from employers. Though I guess the thing in Atlas Shrugged was also called a "strike" and it seems similar in spirit to this.
If nothing else, I expect mildly-superhuman sales and advertising will be enough to ensure that the human share of the universe will decrease over time. And I expect the laws will continue being at least mildly influenced by deep pockets, to keep at least some such avenues possible. If you imagine a hard lock on these and other such things, well that seems unrealistic to me.