PashaKamyshev

Humans aren't fitness maximizers

I think this post suffers pretty badly from Typical Mind Fallacy. This thinking isn't alien to me. I used to think exactly like this 8 years ago, but since marriage and kid I now disagree with basically every point.

One claim that is hopefully uncontroversial: Humans are not literally optimizing for IGF,

I think this is controverisial because it's basically wrong :)

First, its not actually obvious what "definition" of IGF you are using. If you talk about animals, the definition that might fit is "number of genes in the next generation". However if you talk about humans, we care about both "number of genes in the next generation" and "resources given to the children".... (read 780 more words →)

-2

-4

Replying toUploads are Impossible

Robin's whole Age of Em is basically pronouncing "biology is over" in a cheerful way.

Some posts from Nate:

here

I want our values to be able to mature! I want us to figure out how to build sentient minds in silicon, who have different types of wants and desires and joys

I don't want that, instead i want a tool intelligence that augments me by looking at my words and actions. Digital minds (not including “uploads”) are certainly possible and highly undesirable for most people simply due to competition for resources and higher potential for conflict. I don’t buy lack-of-resource-scarcity for a second.

here

uploading minds; copying humans; interstellar probes that aren't slowed down by needing

... (read more)

-10

-9

Replying toUploads are Impossible

I generally don't think LLMs today are conscious, as far as i can tell neither does Sam Altman, but there is some disagreement. They could acquire some characteristics that could be considered conscious as scale increases. However merely having "qualia" and being conscious is not the same thing as being functionally equivalent a new human, let alone a specific human. The term "upload" as commonly understood is a creation of a software construct functionally and qualia-equivalent to a specific human.

a human brain in a vat wouldn't be so far from the experience of language models.

Please don't try to generalize over all human minds based on your experience. Human experience is more than... (read more)

Replying toUploads are Impossible

Thanks for the first part of the comment.

As mentioned in my above comment, the reason for mixing "can" and "should" problems is that they form a "stack" of sorts, where attempting to approximately solve the bottom problems makes the above problems harder and verification is important. How many people would care about the vision if one could never be certain the process succeeds?

Replying toUploads are Impossible

Fixed the link formatting and added a couple more sources, thanks for the heads up. The temperature claim does not seem unusual to me in the slightest. I have personally tried to do a relatively cold bath and noticed my "perception" alter pretty significantly.

The organ claim does seem more unusual, but I have heard various forms of it from many sources at this point. It does not however seem in any way implausbile. Even if you maintain that the brain is the "sole" source of cognition, the brain is still an organ and is heavily affected by the operation of other organs.

-1

Replying toUploads are Impossible

There is a lot to unpack. I have definitely heard from leaders of the community claims to the tune of "biology is over," without further explanation of what exactly that means or what specific steps are expected to happen when the majority of people disagree with this. The lack of clarity here makes it hard to find a specific claim of "I will forcefully do stuff to people they don't like," but me simply saying "I and others want to actually have what we think of as "humans" keep on living" is met with some pushback.

You seem to be saying that the "I" or "Self" of people is somehow static through large... (read 376 more words →)

-3

-5

Value Learning – Towards Resolving Confusion

Epistemic status: pretty confident. Probability this does well on LessWrong: pretty low

This is part 2 of my value learning sequence in which I talk about what humans actually value and how to get AGI to do it.

In this post, I wish to push back against certain visions of the future which seem both dystopian and unachievable. My personal good visions have a simple notion: humanity must seek to correct its problems, but in doing so not get overzealous in its hatred of the human form. Retro sci-fi had somewhat of the right idea (my writing for emphasis).

Many other visions expressed online from both sides of the AI safety debate seem to want to force... (read 2176 more words →)

-5

Part 1 - Keep calm and use revealed preferences. Originally posted on substack

In this post, I want to talk about the problem of value learning and specifically highlight a large amount of confusion around the problem and the term “values” generally. I hope to offer a paradigm that somewhat helps mitigate the confusion. This post is extremely difficult primarily because it requires a certain amount of detachment from many cultural convictions, including and especially “rationalist” convictions, to understand that certain cultures (current or future) may feel pretty different about issues X Y or Z.

Value learning is a subproblem of AGI alignment. It is not the whole problem, there are many other subproblems,... (read 5362 more words →)

Replying toEvolution provides no evidence for the sharp left turn

Evolution provides no evidence for the sharp left turn

The general vibe of the first two parts seems correct to me. Also, an additional point is that evolution's utility function of inclusive genetic fitness didn't completely disappear and is likely still a sub-portion of the human utility function. I suspect there is going to be disagreement on this, but it would also be interesting to do a poll on this question and break it down by people who do or do not have kids.

-1

Replying toAn AI Realist Manifesto: Neither Doomer nor Foomer, but a third more reasonable thing

An AI Realist Manifesto: Neither Doomer nor Foomer, but a third more reasonable thing

Yes I think we understand each other. One thing to keep in mind is that different stakeholders in AI are NOT utilitarians, they have local incentives they individually care about. Given the fact that COVID didn't stop gain-of-function research, this means that getting EVERYONE to care would require a death toll larger than COVID. However, getting someone like CEO of google to care would "only" require a half - a - trillion dollar lawsuit against Microsoft for some issue relating to their AIs.

And I generally expect those - types of warning shots to be pretty likely given how gun-ho the current approach is.

Replying toAn AI Realist Manifesto: Neither Doomer nor Foomer, but a third more reasonable thing

An AI Realist Manifesto: Neither Doomer nor Foomer, but a third more reasonable thing

I am mostly agreeing with you here, so I am not sure you understood my original point. Yes Reality is giving us things that for a set of reasonable people such as you and me should be warning shots.

Since a lot of other people don't react to them, you might become pessimistic and extrapolate that NO warning shot is going to be good enough. However I posit that SOME warning shots are going to be good enough. An AI - driven bank run followed by an economic collapse is one example, but there could be others. Generally I expect that when warning shots reach "nation-level" socio-economic problems, people will pay attention.

However, this will happen before doom.

An AI Realist Manifesto: Neither Doomer nor Foomer, but a third more reasonable thing

What does it mean for an LLM such as GPT to be aligned / good / positive impact?

Cross-posted from substack

AI has been a hot topic in recent Twitter discourse, with two opposing camps dominating the conversation: the Doomers and the AI builders. The Doomers, led by Eliezer Yudkowsky and other rationalists, advocate for caution and restraint in the development of AI, fearing that it could pose an existential threat to humanity. Prominent figures in this camp include Elon Musk, who has expressed concerns about the potential dangers of AI while also founding AI-focused companies like OpenAI and up-and-coming “BasedAI.” On the other side of the debate are the AI builders, including Yann LeCunn and Sam Altman, who are eager to push the boundaries of AI development and explore its... (read 2315 more words →)

AI as a Civilizational Risk Part 6/6: What can be done

(cross post from my sub-stack)

What does it mean for a language model to be good / aligned / positive impact?

In this essay, I will consider the question of what properties that an LLM such as GPT-X needs to have in order to be a net positive for the world. This is both meant to be a classification of OpenAIs and commentator's opinions on the matter as well as my input of what the criteria should be. Note, I am NOT considering the question of whether the creation of GPT-4 and previous models have negatively contributed to AI risk by speeding up AI timelines. Rather these are pointers towards criteria that one might... (read 2944 more words →)

AI as a Civilizational Risk Part 5/6: Relationship between C-risk and X-risk

Fix or destroy social media

One of the critical positive developments is the potential of Elon Musk to buy Twitter. The acquisition happened as the essay was being finalized. Positive effects that can happen are cracking down on bots and re-vamping moderation to avoid bad AI-driven mischaracterization of public opinion. However, the main benefit would be a potential implementation of a non-optimization feed ranking algorithm in the vein of TrustRank. Proper feed ranking would promote socially cohesive ideas instead of wedge issues.

Aside from Elon's specific actions around Twitter, most social media needs to be destroyed or drastically reformed. We need to be careful around ranking algorithms. Any algorithm with an "optimization... (read 1189 more words →)

AI as a Civilizational Risk Part 4/6: Bioweapons and Philosophy of Modification

The general model presented in the previous parts suggests that the US and potentially other parts of the world will suffer some form of civilizational collapse, potentially as early as 10 years from now. In the absence of decisive action to limit the spread of narrow behavioral modification AIs, such collapse is inevitable. Social media optimizational pressure points only towards the destruction of social cohesion. The collapse of civilization is a big deal, however, this isn't the same as an even more serious subject of existential risks.

I want to discuss the relationship between civilization and existential risk, specifically the P(x-risk | unaligned AGI) by 2070. I would argue that this probability is... (read 2074 more words →)

AI as a Civilizational Risk Part 3/6: Anti-economy and Signal Pollution

Bio-Chemical Weapons made by AI

One of the core risks involved with both narrow AIs and general AIs is the creation of biological and chemical weapons. While the probability of creating stronger and stronger behavior modification in AIs is close to 100%, the probability of creating biological weapons using AI is smaller. However, the potential downsides are more significant. A larger probability is that AIs create many chemical quasi-weapons compounds that can end up in manufacturing due to a misunderstood safety profile.

Analogizing signal pollution to manufacturing pollution is worth considering, especially because we have yet to solve manufacturing pollution. There are many credible hypotheses that many chemicals have polluted the land of... (read 2223 more words →)

AI as a Civilizational Risk Part 2/6: Behavioral Modification

Any social problem in your society can and will be made worse by adding AI. Just because we tolerate a problem without society collapsing does not mean we can tolerate an AI-enhanced version of it.

Spam as an example of Anti-Economy

We can talk about the problems of behavioral modification in economic terms. Propaganda and engagement AIs are centralizing benefits and distributing costs, a well-known economic issue of externalities. The tools we have created do not price in those externalities. The externalities may be so significant that large companies are potentially net negative for humanity if they use too many behavior modification AIs. I like to call this the anti-economy. A new term... (read 3922 more words →)

AI as a Civilizational Risk Part 1/6: Historical Priors

Altering people for the worse using AI is easier than altering them for the better.

Redefining what “Capital” means

Narrow AIs and optimization algorithms of social media news feeds have already caused many adverse outcomes without being AGIs. When people talk about AI on Twitter or Less Wrong, they use manufacturing or job-replacement analogies as the core function of AI. This analogy imagines AI as a powerful extension of industrial capital, defined by its "ownership of the means of production." This 19/20th-century thought is becoming increasingly removed from the reality of capital today.

The core business of the largest tech companies, whether social media giants, advertising behemoths like Google, or middle-layer companies like... (read 2973 more words →)