This is a special post for quick takes by daijin. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
One of the great filters is long reward horizons.
I recently did an RL competition on kaggle. With enough time spent in RL and teaching (the former, a month, the latter, over 7 years on a weekly basis), children (and adults who learn) start to look like RL agents.
I noticed long reward horizons (periods where you are heading to a long term reward with no rewards in between) as a barrier to progress in my RL competition, then a friend mentioned she wanted to build a career but it felt unfathomable to commit that huge amount of time towards it, even putting aside uncertainty.
Long reward horizons show up elsewhere. To oceanfare, you have fish, but to go to space you have to endure huge expanses of no rewards.
Intrinsic rewards don't solve this - being motivated to explore for exploration's sake leads to perverse outcomes. The closest thing that solves this is persistent out-of-band memory of trajectories to reach the edge of ones knowledge. In geography this is a map. In knowledge space this is a pedagogical map.
The sequences can be distilled down even further into a few sentences per article.
Starting with "The lens that sees its flaws": this distils down to: "The ability to apply science to our own thinking grants us the ability to counteract our own biases, which can be powerful." Statement by statement:
- A lot of complex physics and neural processing is required for you to notice something simple, like that your shoelace is untied.
- However, on top of noticing that your shoelace is untied, you can also comprehend the process of (noticing your shoelace is untied) - i.e. by listing the steps through which light reflects off your shoelace and your visual cortex engaging, etc.
- The ability to consider the steps of our own thinking appears to be uniquely human.
- If we recognise that our process of comprehension and understanding is potentially flawed, you can choose to consciously counteract it.
- Science is repeatedly and deliberately making measurements of our own observations over time, attributing theories to those measurements, and constructing experiments to produce further measurements to potentially disprove those theories.
- The ability to apply science to our own thinking grants us the ability to counteract our own biases, which can be powerful.
- One example of reflective correction is correcting for optimism by noticing that optimism is not correlated to good outcomes.
The tool I am using to distill the sequences is an outliner: a nested bulleted list that allows rearranging of bullet points. This tool is typically used for writing things, but can similarly be used for un-writing things: taking a written article in and deduplicating its points, one bullet at a time, into a simpler format. An outliner can also collapse and reveal bullet points.
https://www.lesswrong.com/posts/n5TqCuizyJDfAPjkr/the-baby-eating-aliens-1-8
There's an alien race in the above called the baby-eaters who eat their sentient offspring because this alien species is genetically unfortunate enough to produce thousands of immediately-sentient offspring at a time, and therefore they must cull their offspring.
More and more each day I think we are becoming like baby-eaters. Except of course we don't murder our offspring by directly consuming them.
What is murder? What is consumption? It is the reclaiming of productive resource and the denial of future growth. When a poor person, having lived through years of their life giving what little they must to society in order to survive, dies on the street, there is another person that has been eaten by society.
We are eating each other, and often by the time someone has reached their preteens we know that they will be one of the ones that will be eaten. Therefore, in many ways, we are becoming baby-eaters.
However unlike the baby-eaters, not all of our society has normalized baby-eating to the point that we cannot see it as anything other than unquestionably good.
What would we ask of the baby-eaters? How should we act today?
When a poor person, having lived through years of their life giving what little they must to society in order to survive, dies on the street, there is another person that has been eaten by society.
At least with respect to today's western societies, this seems off-key to me. It makes it sound as if living and dying on the street is simply a matter of poverty. That may be true in poor overpopulated societies. But in a developed society, it seems much more to involve being unable (e.g. mental illness) or unwilling (e.g. criminality) to be part of the ordinary working life.
What would we ask of the baby-eaters?
You'll have to be clearer about which people you mean. Baby-eating here is a metaphor, for what exactly? Older generation neglecting the younger generation, or even living at their expense? Predatory business practices? Focusing on your own prosperity rather than caring for others?
Anyone still want to build a nuke using AI, as per Yudowsky's post earlier? OpenAI reckons nukes aren't the answer, but Bioweapons are. https://openai.com/index/gpt-5-5-bio-bug-bounty/
Should cavemen have worked on reducing their carbon footprint?
For instance, I read a book, which argues that we have a profound obligation to work on reducing existential risk because virtually all of humanity’s potential lies in the future... I decided to pretty much completely adopt the belief as my worldview.
Your question appears to be a non-sequitur since their carbon footprint was not anything close to an existential risk, and would make no contribution to existential risk from climate change today either.
At the time, the largest existential risks to humanity would have been things they were already concerned about and working to reduce: disease and famine.
Would the argument work if we were in the 1950s and decided whether we should reduce the risk of World War III destroying mankind along with the future? I think that we would have an obligation to reduce risks from WWIII. On the other hand, the cavemen had no idea about the carbon footprint and couldn't decide to reduce it to avoid global warming.
we will never have a wealth tax because pirate games, so marry the rich v2
original: https://www.lesswrong.com/posts/G5qjrfvBb7wszBgWG/daijin-s-shortform?commentId=4b4cDSKxfdxGw4vBH
1. why have a wealth tax?
we should tax unearned wealth because the presence of unearned wealth disincentivises workers who would otherwise contribute to society. when we tax unearned wealth, the remaining wealthy people are people who have earned their wealth; and so we send a signal 'the best way for you to be privately wealthy is for your work to align with public utility maximisation' which privately incentivises work which helps increase utility.
unearned wealth includes: hereditary wealth, wealth due to being at the right place at the right time (you just so happened to buy a tract of land that contains your nation's entire supply of unobtanium / you just so happened to found a company that became the dominant news outlet for the entire world when other similar but less well timed companies failed)
on my original point about monopolies: monopolies are a special case of ^above. The loss of utility due to monoplies is recognised by economists as deadweight loss.
2. why will [countries where the game-theory supports the creation of ultrarich people] never have a wealth tax because pirate games?
Suppose a whole bunch of us got together and demanded that wealthy oligarchs pay a wealth tax. the wealthy oligarchs could instead take a small amount of money and bribe 51% of us to defect, while keeping their money piles. therefore we will never have a wealth tax.
re [countries where the game-theory supports the creation of ultrarich people]: nicky case's wonderful interactive game theory primer tells us that the specific payoffs/noise in zero sum games decide which strategies (cooperate, defect) tend to survive. I suspect countries with ultrawealthy tend to have payoff/noise combos that result in dominant strategies that would participate in pirate games
but hang on, what about the democracies that do have wealth taxes?
> Norway recently tried to increase their wealth tax. a whole bunch of rich people left Norway. I do not constitute this as a successful raising of the wealth tax
> [wikipedia] at its peak, only 12 countries have a wealth tax, representing as a whole less than 6% of global GDP; they are exceptions not rules (Austria, Denmark, Finland, France, Germany, Iceland, Italy, Netherlands, Norway, Spain, Sweden and Switzerland)
3. what to do instead? marry rich
this is not easy. EA consultants are in some ways marrying rich people.
Paying the (ongoing, repeated) pirate-game blackmail ("pay us or we'll impose a wealth tax") IS a form of wealth tax. You probably need to be more specific about what kinds and levels of wealth tax could happen with various counterfactual assumptions (without those assumptions, there's no reason to believe anything is possible except what actually exists).
less than 6% of global GDP; they are exceptions not rules (Austria, Denmark, Finland, France, Germany, Iceland, Italy, Netherlands, Norway, Spain, Sweden and Switzerland)
It is 16%, not 6%.
(Approximately, Germany is 4.7% of global GDP, France is 3.1%, Italy 2.2%, Spain 1.5%, Netherlands 1.1%, Switzerland 0.9%, Sweden 0.6%, Austria and Norway 0.5% each, Denmark 0.4%, Finland 0.3%.)
Applied to a local scale, this feels similar to the notion that we should employ our willpower to allow burnout as discussed here
we will never have a wealth tax because pirate games.
why have a wealth tax? excess wealth is correlated with monopolies which are a failure to maximise utility. therefore wealth taxes would help increase total utility. monopolies include but are not limited to family wealth, natural monopolies, social network monopolies.
however, suppose a whole bunch of us got together and demanded that wealthy oligarchs pay a wealth tax. the wealthy oligarchs could instead take a small amount of money and bribe 51% of us to defect, while keeping their money piles.
therefore we will never have a wealth tax.
what to do instead? marry rich
1
1
1
1I made a v2 of this shortform that answers your point with an example from recent history.
Is current AI functional research AGI (general intelligence) or just UltraPiracy? Are we actually advancing AI capabilities beyond human capabilities, or are we turning knowledge from human capital into a kind of physical resource that can be captured and held by a corporation?
Social media was the incorporation of attention and agency; what we're calling AI is the incorporation of learning.
What I would need to be proven wrong: for AI to do something genuinely novel, not just gap filling between our current knowledge of the world.
If AI is truly AGI like, then intelligence runaway is the failure mode. If AI is only UltraPiracy, then the failure mode is neofeudalism.