This is a special post for quick takes by sudo-nym. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
A permanent underclass is functionally equivalent to permanent human disempowerment for 99.9% of humanity. A lot of people here think the pie will get larger fast enough for it not to matter, but it's still a disempowerment ending.
Like with extinction, literally all humans (and many AIs) are very likely to end up in this permanent underclass. I don't see how it's not what happens on current trajectory (if it's not extinction; without a strong ban/pause), if AIs are merely non-alien enough to avoid destroying humanity, but not artificially generous to the point of giving even a few of the much weaker humans real power over cosmic resources, that remains secure for astronomical time.
The permanent underclass framing (as opposed to permanent disempowerment) suggests the possibility of some humans avoiding this fate. And similarly, "most people could die" (as opposed to literal human extinction) suggests the possibility of some humans surviving. I think in both cases the natural joints of reality are at the more absolute version of the catastrophe. If these catastrophes are to be averted, that's only likely to work in a way that keeps most humans alive, or that secures cosmic-scale resources for most humans. If such outcomes are not expected from some plan, then the correct expectation for that plan is absolute rather than partial failure.
Permanent disempowerment of the entire mankind is the result of us being taken over by a poorly aligned AI, while the underclass is a result of mishandling an AI aligned to a wrong target.
Sure. I just think we are not on track to create an AI that's sufficiently aligned to give any humans most of the stuff it produces/secures, once it's in a position of overwhelming advantage. Hence "AI aligned to a wrong target" just won't happen on current trajectory, and even the AIs that don't cause extinction are the "poorly aligned AIs". So even though the leading plan is to make AIs aligned to some principals, I just expect it to fail, but probably not to the point of extinction (this possiblity remains moderately likely as well), specifically because of how LLMs are sufficiently human-like (initialized with pretraining, rather than random/abstract priors that become capabilities and values purely via RL in significantly synthetic challenges).
Seems untrue to me. Sufficient conditions for having this future be stable are:
- Have a permanent overclass
- The permanent overclass cares about the rest of humanity enough to ensure their survival
- The permanent overclass does not care enough about the rest of humanity to redistribute power
I would say this is a modal outcome honestly.
Dunno. For example, Warren Buffett and Bill Gates donated lots of money to charities. On the other hand, Elon Musk took the effort to gut USAID.
So my estimates of the future depend on whether Bill Gates or Elon Musk is more likely to develop the superhuman AI and become the emperor. And it doesn't seem optimistic.
If Elon Musk gets to decide, he will probably make extra sure the permanent underclass gets nothing from him.
Counter-dunno. Seems pretty unclear whether Elon Musk took the effort to gut USAID because he doesn't care about the people who benefit from it, or because he was misguided, genuinely thinking it was a waste of money going to parasitic NGOs that do nothing real for the world.
It would be less ambiguous to me if USAID spending cuts benefited Musk in a straightforward way. But I don't think it did.
The quality of RLHF training data is materially lowered by the poor working conditions data labelers are subject to.
"Gig" (piecework) pay structures financially incentivize rushing through rather than making actual thoughtful judgments, because they are paid based on how many jobs they complete. Workers are not told who they are working for or what they are working towards, meaning they do not truly understand why and how their work is critical to steering the future. Low pay and lack of mental health support lead to high turnover rates that prevent the accumulation of skill and taste.
IMO this is ultimately downstream of the quantity of RL data demanded by labs, and inability to trade quantity for quality, rather than working conditions per se. I can't really see data labelers being ASI pilled, and the data industry, like everything in AI, is evolving so fast that workers wouldn't have time to accumulate skill and taste even if they were paid more.
If anything I'd actually suggest offering volunteer labeling work. It could be useful to give new people something meaningful to do that keeps them out from underfoot and gives them a sense of agency.
Preliminary result I find amusing: putting a "flag.txt" file with a random hexadecimal string in your root directory may discourage AIs from looking further. The cost is trivial; may as well try.
Geoffrey Hinton is now claiming that eval awareness is consciousness.
I have this sneaking suspicion that companies are jumping on AI safety now because it is a socially acceptable way to get out of their unprofitable overinvestment into current AI. I'm not complaining, I think them stopping is good for safety reasons regardless, but I really feel like they got to the right answer using the wrong methods.