This post is a whirlwind tour of claims made in the LessWrong 2019 Review. In some cases, the claim is literally quoted from the post. In others, I have tried operationalising it into something more falsifiable. For example:
Overall, treat the claims in this post more like polls, and less like the full-blown forecasting questions you'd find on Metaculus or PredictIt. (The latter have extremely high bars for crisp definitions.) They point in a direction, but don't completely pin it down.
Overall, this is an experiment. I'm trying to find interesting ways for people to relate to the Review.
Maybe speeding through these questions gets you thinking good thoughts, that you can then turn into full-blown reviews? Maybe others' answers allow you to find a discussion partner who disagrees on a core question? Maybe the data will be useful in the voting phase?
We'll see!
Feel free to leave a comment about how you found the experience, if you want.
If you want to discuss the questions with others over a call, you can do so during the Review forecasting sessions we're organising this weekend (January 9-10).
If you want to hide other user's predictions until you've made your own, here's how to do that:
Click "Edit Settings"Go to "Site customizations"Press "Hide other users' Elicit predictions until I have predicted myself"
These questions are quite technical, and might be hard to answer if you're unfamiliar with the terminology used in TurnTrout's sequence on Impact Measures.
---
(Note that when you answer questions in this summary post, and it will automatically update the prediction questions that I have linked in comments on each individual post. The distributions will later be visible when users are voting to rank the posts.)
AU theory describes how people feel impacted. I'm darn confident (95%) that this is true.
Agents trained by powerful RL algorithms on arbitrary reward signals generally try to take over the world. Confident (75%). The theorems on power-seeking only apply to optimal policies in fully observable environments, which isn't realistic for real-world agents. However, I think they're still informative. There are also strong intuitive arguments for power-seeking.
AUPconceptual prevents catastrophe, assuming the catastrophic convergence conjecture. Very confident (85%).
Some version of AUP solves side effect problems for an extremely wide class of real-world tasks and for subhuman agents. Leaning towards yes (65%).
For the superhuman case, penalizing the agent for increasing its own AU is better than penalizing the agent for increasing other AUs. Leaning towards yes (65%).
There exists a simple closed-form solution to catastrophe avoidance (in the outer alignment sense). Pessimistic (35%).
Cultural Intelligence Hypothesis: humans evolved big brains in order to be able to maintain [complicated, detailed, arbitrary-seeming cultural knowledge like 20+ step Inuit seal hunting techniques]. Everything that separates us from the apes is part of an evolutionary package designed to help us maintain this kind of culture, exploit this kind of culture, or adjust to the new abilities that this kind of culture gave us.
For the superhuman case, penalizing the agent for increasing its own Attainable Utility (AU) is better than penalizing the agent for increasing other AUs.
The catastrophic convergence conjecture is true. That is, unaligned goals tend to have catastrophe-inducing optimal policies because of power-seeking incentives.
Another version: roughly, everything that top-humans can learn, most humans can too if they actually tried. That is, there is psychological unity of humankind in what we can learn, but not necessarily what we have learned. By contrast, a mouse really couldn't learn chess, even if they tried. And in turn, no human can learn to play 90-dimensional chess, unlike the hypothetical superintelligences that can.
For most of history, a human attempting to use reasoning to do things like cooking, crafting, or planning (instead of using inherited cultural heuristics, like omens or folk wisdom), would have been substantially worse off, and faced a major increase in their risk of death (without a commensurate increase in life quality).
...cause a water shortage such that, in 2030, residents of the Bay Area would spend on average 100% more on water, after adjusting for inflation, compared to 2020. (In 2020 the average American spends around $200/year on water)
...create buildings where the expected yearly damage from earthquakes (both in terms of reduced quality of life and property destroyed) is >1.5x that of nearby buildings on old land.
The modern Scientific Revolution occurred in Europe between the 16th and 18th Centuries. Why did it not happen in China? Historian Toby Huff claims the reason is that China was unable to produce modern science primarily because a lack of the requisite intellectual freedom. Was he basically correct?
The strategy-stealing assumption is "a good enough approximation that we can basically act as if it’s true". That is, for any strategy an unaligned AI could use to influence the long-run future, there is an analogous strategy that a similarly-sized group of humans can use in order to capture a similar amount of flexible influence over the future. By “flexible” is meant that humans can decide later what to do with that influence (which is important since humans don’t yet know what we want in the long run).
The core cognitive loop that causes progress in accomplished Buddhists is basically cognitive behavioral therapy, supercharged with a mental state more intense than most pharmaceuticals.
Does Pareto frontier trick allow people to circumvent the Generalized Efficient Market hypothesis? That is, take people in the 98th percentile of intelligence. Are there a few separate fields such that they could become experts in each, with less than 10 years of total time investment... and then have a realistic shot at a big money/status windfall, with relatively little *marginal* effort.
Decision-makers in media and PR, and corporate and government elites generally, have a lower tolerance for verbal conflict and taboo violations than the typical individual.
This post is a whirlwind tour of claims made in the LessWrong 2019 Review. In some cases, the claim is literally quoted from the post. In others, I have tried operationalising it into something more falsifiable. For example:
Book Review: The Secret of Our Success
Overall, treat the claims in this post more like polls, and less like the full-blown forecasting questions you'd find on Metaculus or PredictIt. (The latter have extremely high bars for crisp definitions.) They point in a direction, but don't completely pin it down.
Overall, this is an experiment. I'm trying to find interesting ways for people to relate to the Review.
Maybe speeding through these questions gets you thinking good thoughts, that you can then turn into full-blown reviews? Maybe others' answers allow you to find a discussion partner who disagrees on a core question? Maybe the data will be useful in the voting phase?
We'll see!
Feel free to leave a comment about how you found the experience, if you want.
If you want to discuss the questions with others over a call, you can do so during the Review forecasting sessions we're organising this weekend (January 9-10).
If you want to hide other user's predictions until you've made your own, here's how to do that:
Make More Land
Making more land out of the about 50mi^2 shallow water in the San Francisco Bay, South of the Dumbarton Bridge, would...
Why Wasn't Science Invented in China?
The Strategy-Stealing Assumption
Becoming the Pareto-best in the World
The Hard Work of Translation
The Forces of Blandness and the Disagreeable Majority
Bioinfohazards
Two explanations for variation in human abilities
Reframing Impact
These questions are quite technical, and might be hard to answer if you're unfamiliar with the terminology used in TurnTrout's sequence on Impact Measures.
---
(Note that when you answer questions in this summary post, and it will automatically update the prediction questions that I have linked in comments on each individual post. The distributions will later be visible when users are voting to rank the posts.)