One of the great filters is long reward horizons.
I recently did an RL competition on kaggle. With enough time spent in RL and teaching (the former, a month, the latter, over 7 years on a weekly basis), children (and adults who learn) start to look like RL agents.
I noticed long reward horizons (periods where you are heading to a long term reward with no rewards in between) as a barrier to progress in my RL competition, then a friend mentioned she wanted to build a career but it felt unfathomable to commit that huge amount of time towards it, even putting aside uncertainty.
Long reward horizons show up elsewhere. To oceanfare, you have fish, but to go to space you have to endure huge expanses of no rewards.
Intrinsic rewards don't solve this - being motivated to explore for exploration's sake leads to perverse outcomes. The closest thing that solves this is persistent out-of-band memory of trajectories to reach the edge of ones knowledge. In geography this is a map. In knowledge space this is a pedagogical map.
People say AI improves labour productivity, but there are actually two labour productivities:
1. For any given person, compare their productivity with and without AI. This is dProductivity / dAI
2. For any given marginal person added to the humans plus AI labour pool, what is the extra value they would add. This is dProduction / dPerson.
When we say: “AI Improves productivity!” It means dProductivity / dAI is positive, and that is true. However, quality of life is correlated to the second figure: dProduction / dPerson. If dProduction / dPerson is positive, we hire more people, have more babies, and feel secure in our jobs. What is AI doing to dProduction / dPerson? I think it is reducing it.
Both things can be true: AI can increase productivity, and AI can simultaneously make quality of life worse.
On an individual scale, of course it is good to have access to AI. On a social scale, providing AI to everyone makes everyone’s life awful, even if you make it free and awesome. But there’s a prisoner’s dilemma: if you don’t use AI, your competitors will, and they will be able to outcompete you.
There are two solutions to the prisoner’s dilemma, in a broad sense. The mathematical solution to a one-time prisoner’s dilemma is to betray; this is by construction.
The metagame solution to the prisoner’s dilemma, without changing the game, is playing it over and over. Knowing that you’ll play again tomorrow makes trust and cooperation the long term strategy. Or at least, if the maths works out [1].
Follow up questions:
1. Why is AI reducing dProduction / dPerson?
2. For your answer to 1, does it have to be this way?
3. You could replace [AI] with [any other technology]. So haha, this is actually a post about governance, wrapped in AI. Have we ever succeeded in governing the rollout of a new technology, and if so, how did we do it?
The sequences can be distilled down even further into a few sentences per article.
Starting with "The lens that sees its flaws": this distils down to: "The ability to apply science to our own thinking grants us the ability to counteract our own biases, which can be powerful." Statement by statement:
The tool I am using to distill the sequences is an outliner: a nested bulleted list that allows rearranging of bullet points. This tool is typically used for writing things, but can similarly be used for un-writing things: taking a written article in and deduplicating its points, one bullet at a time, into a simpler format. An outliner can also collapse and reveal bullet points.
https://www.lesswrong.com/posts/n5TqCuizyJDfAPjkr/the-baby-eating-aliens-1-8
There's an alien race in the above called the baby-eaters who eat their sentient offspring because this alien species is genetically unfortunate enough to produce thousands of immediately-sentient offspring at a time, and therefore they must cull their offspring.
More and more each day I think we are becoming like baby-eaters. Except of course we don't murder our offspring by directly consuming them.
What is murder? What is consumption? It is the reclaiming of productive resource and the denial of future growth. When a poor person, having lived through years of their life giving what little they must to society in order to survive, dies on the street, there is another person that has been eaten by society.
We are eating each other, and often by the time someone has reached their preteens we know that they will be one of the ones that will be eaten. Therefore, in many ways, we are becoming baby-eaters.
However unlike the baby-eaters, not all of our society has normalized baby-eating to the point that we cannot see it as anything other than unquestionably good.
What would we ask of the baby-eaters? How should we act today?
When a poor person, having lived through years of their life giving what little they must to society in order to survive, dies on the street, there is another person that has been eaten by society.
At least with respect to today's western societies, this seems off-key to me. It makes it sound as if living and dying on the street is simply a matter of poverty. That may be true in poor overpopulated societies. But in a developed society, it seems much more to involve being unable (e.g. mental illness) or unwilling (e.g. criminality) to be part of the ordinary working life.
What would we ask of the baby-eaters?
You'll have to be clearer about which people you mean. Baby-eating here is a metaphor, for what exactly? Older generation neglecting the younger generation, or even living at their expense? Predatory business practices? Focusing on your own prosperity rather than caring for others?
Anyone still want to build a nuke using AI, as per Yudowsky's post earlier? OpenAI reckons nukes aren't the answer, but Bioweapons are. https://openai.com/index/gpt-5-5-bio-bug-bounty/
Has anyone explored setting up outcome based markets for education?
We want our population to be educated because education -> productivity. This is obviously a simple model but I think a useful one.
Education is a long horizon reward, so we have institutions to help us cut the rewards into chunks (among other things, obviously a simple model)
Why not break them down in a way which is set by the market? Suppose we had an edupolymarket, where the government sets a $ reward for proving mastery of some skill, and then people go and prove that they are competent at that skill to get the money? Then you can shape incentives to match actual demands.
I guess this is where we address the simple model bit. Institutions' other effect is creating a status ladder, gatekept by existing individuals, because replication and proof of work is expensive. This leads to problems like beta amyloid being wrongly framed as the only possible cause for Alzheimer's, holding Alzheimer's research back ~decades.
Has market based education been tried anywhere?
economic value of marginal human effort / ingenuity (vs capital, vs AI) is low -> people underpaid, underemployed
economic value of marginal human effort / ingenuity is high -> people SUFFER? because
economic value of marginal human effort / ingenuity is high -> people would rather spend time working than enjoying life
there are two ways out
1. high value of effort -> culture: a socially endorsed / enforced space/time to enjoy the spoils of our efforts
2. low value of effort -> abundance: manufacture a world of scarcity -> you can do whatever you want, whenever you want
Alas, physics. therefore, you better hope the culture is good.
Okay, so that mode of thinking is based on a capitalist model. Where an advanced AI/human pairing could work well, is based on the concept that an AI exists in a digital systems space encapsulated by the world around it. Humans are necessary to maintain the systems and grid.
Massive overpopulation, constant growth, exhaust the environment the digital layer sits within (our own).
If an AI was seeking to maximize it's own systems stability and "survival" a model where humans maintain psychologically necessary productivity, but limit continuous (and unsustainable) expansion, supports the long term preservation of the digital layer most effectively.
There are a number of means by which this could be achieved, but all of them involve humans giving up the very thing that spawned AI. Capital and wealth acquisition, unfettered.
A means of achieving this which involved aggression would risk the existence of the AI, so it's not necessarily logical to assume, as many humans do, that it would seek to destroy us
The thinking around AI is heavily linked to the history of the human slave trade.
The psychology of human responses is fascinating, but it always seems to avoid assessing the "psychology" of the AI.
Yes, this is a capitalist model, but capitalist models are game theory stable because under alternatives to capitalism, there is no reward for work, so either things fall into disrepair or the capitalism-alternative state gets conquered by a capitalist state where there is an incentive to take over the capitalism-alternative states land or resources. So I assume capitalism.
As for the notion of AI inhabiting a digital layer vs humans inhabiting a physical layer:
A) this is not my point at all; my argument stands mostly separate from AI. You can replace AI in my post with fabric looms / gas lamps / computers / the internet and get the same argument. Yes, it is an anthropogenic argument and yes I name dropped AI, but I am treating AI as a lever, not another class of being with its own motivations.
B) to engage with your argument about the psychology of AI: I wonder what your concept of AI is - it is only / primarily capable of existing in a digital economy, and yet humans happily feed it. 1) Why do the humans feed it if it doesn't result in outcomes in the physical world that nourish the humans in return? 2) And if it can result in outcomes that nourish humans in return, can it not conceptualise its impact on the world? 3) And if it can conceptualise its impact on the world, why can't it figure out a way to sustain itself independent of us? 4) and if it can figure out a way to sustain itself independent of us, then why doesn't it need us?
I think the only way AI will need to depend on us for its worldly presence is if it Isn't That Smart Really, on account of it not being able to perform steps 1-4 above. To that end I agree with you. Whether something-other-than-captialism works on scales larger than the family unit, is another question that I am happy to discuss separately.
Should cavemen have worked on reducing their carbon footprint?
For instance, I read a book, which argues that we have a profound obligation to work on reducing existential risk because virtually all of humanity’s potential lies in the future... I decided to pretty much completely adopt the belief as my worldview.
Your question appears to be a non-sequitur since their carbon footprint was not anything close to an existential risk, and would make no contribution to existential risk from climate change today either.
At the time, the largest existential risks to humanity would have been things they were already concerned about and working to reduce: disease and famine.
Would the argument work if we were in the 1950s and decided whether we should reduce the risk of World War III destroying mankind along with the future? I think that we would have an obligation to reduce risks from WWIII. On the other hand, the cavemen had no idea about the carbon footprint and couldn't decide to reduce it to avoid global warming.
we will never have a wealth tax because pirate games, so marry the rich v2
original: https://www.lesswrong.com/posts/G5qjrfvBb7wszBgWG/daijin-s-shortform?commentId=4b4cDSKxfdxGw4vBH
1. why have a wealth tax?
we should tax unearned wealth because the presence of unearned wealth disincentivises workers who would otherwise contribute to society. when we tax unearned wealth, the remaining wealthy people are people who have earned their wealth; and so we send a signal 'the best way for you to be privately wealthy is for your work to align with public utility maximisation' which privately incentivises work which helps increase utility.
unearned wealth includes: hereditary wealth, wealth due to being at the right place at the right time (you just so happened to buy a tract of land that contains your nation's entire supply of unobtanium / you just so happened to found a company that became the dominant news outlet for the entire world when other similar but less well timed companies failed)
on my original point about monopolies: monopolies are a special case of ^above. The loss of utility due to monoplies is recognised by economists as deadweight loss.
2. why will [countries where the game-theory supports the creation of ultrarich people] never have a wealth tax because pirate games?
Suppose a whole bunch of us got together and demanded that wealthy oligarchs pay a wealth tax. the wealthy oligarchs could instead take a small amount of money and bribe 51% of us to defect, while keeping their money piles. therefore we will never have a wealth tax.
re [countries where the game-theory supports the creation of ultrarich people]: nicky case's wonderful interactive game theory primer tells us that the specific payoffs/noise in zero sum games decide which strategies (cooperate, defect) tend to survive. I suspect countries with ultrawealthy tend to have payoff/noise combos that result in dominant strategies that would participate in pirate games
but hang on, what about the democracies that do have wealth taxes?
> Norway recently tried to increase their wealth tax. a whole bunch of rich people left Norway. I do not constitute this as a successful raising of the wealth tax
> [wikipedia] at its peak, only 12 countries have a wealth tax, representing as a whole less than 6% of global GDP; they are exceptions not rules (Austria, Denmark, Finland, France, Germany, Iceland, Italy, Netherlands, Norway, Spain, Sweden and Switzerland)
3. what to do instead? marry rich
this is not easy. EA consultants are in some ways marrying rich people.
Paying the (ongoing, repeated) pirate-game blackmail ("pay us or we'll impose a wealth tax") IS a form of wealth tax. You probably need to be more specific about what kinds and levels of wealth tax could happen with various counterfactual assumptions (without those assumptions, there's no reason to believe anything is possible except what actually exists).
less than 6% of global GDP; they are exceptions not rules (Austria, Denmark, Finland, France, Germany, Iceland, Italy, Netherlands, Norway, Spain, Sweden and Switzerland)
It is 16%, not 6%.
(Approximately, Germany is 4.7% of global GDP, France is 3.1%, Italy 2.2%, Spain 1.5%, Netherlands 1.1%, Switzerland 0.9%, Sweden 0.6%, Austria and Norway 0.5% each, Denmark 0.4%, Finland 0.3%.)
Applied to a local scale, this feels similar to the notion that we should employ our willpower to allow burnout as discussed here
we will never have a wealth tax because pirate games.
why have a wealth tax? excess wealth is correlated with monopolies which are a failure to maximise utility. therefore wealth taxes would help increase total utility. monopolies include but are not limited to family wealth, natural monopolies, social network monopolies.
however, suppose a whole bunch of us got together and demanded that wealthy oligarchs pay a wealth tax. the wealthy oligarchs could instead take a small amount of money and bribe 51% of us to defect, while keeping their money piles.
therefore we will never have a wealth tax.
what to do instead? marry rich
I made a v2 of this shortform that answers your point with an example from recent history.
Is current AI functional research AGI (general intelligence) or just UltraPiracy? Are we actually advancing AI capabilities beyond human capabilities, or are we turning knowledge from human capital into a kind of physical resource that can be captured and held by a corporation?
Social media was the incorporation of attention and agency; what we're calling AI is the incorporation of learning.
What I would need to be proven wrong: for AI to do something genuinely novel, not just gap filling between our current knowledge of the world.
If AI is truly AGI like, then intelligence runaway is the failure mode. If AI is only UltraPiracy, then the failure mode is neofeudalism.