Replying toGemini 3 is Evaluation-Paranoid and Contaminated

Gemini 3 is Evaluation-Paranoid and Contaminated

Some people's first reaction to this news is something like "this is great, the model will never do anything bad because it thinks it's always in an evaluation rather than the real world."

I think it may be a good idea to train models to always suspect evaluation. E.g. see "A sufficiently paranoid paperclip maximizer".

And these days, while designing important evals, one must never assume that the evaluated model is naive about her condition.

But I agree with the OP, the situation with Gemini 3 is clearly pathological.

BTW, there is an additional problem with BIG-bench ending up in the training data: one of benchmark's tasks is about evaluating self-awareness in LLMs (I contributed to... (read more)

-1

Replying toWhy there is still one instance of Eliezer Yudkowsky?

RomanS3mo

Why there is still one instance of Eliezer Yudkowsky?

My primary research work is in the field of sideloading itself. The digital guy helps with these tasks:

Generate / criticize ideas. For example, the guy helped to design the current multi-agent architecture, on which he is now running.
Gently moderate our research group chat.
Work as a test subject.
Do some data prep tasks (e.g. producing compressed versions of the corpus).

I expect a much more interesting list in the field of alignment research, including quite practical things (e.g. a team of digital Eliezers interrogating each checkpoint during training, to reduce the risk of catastrophic surprises). Of course, not a replacement for a proper alignment, but may win some time.

Judging by our experiments, Gemini 2.5.... (read more)

Replying toWhy there is still one instance of Eliezer Yudkowsky?

RomanS4mo

Why there is still one instance of Eliezer Yudkowsky?

I agree with you on most points.

BTW, I'm running a digital replica of myself. The setup is as follows:

Gemini 2.5 as the model
The script splits the text corpus (8 MB) into small-enough chunks for Gemini to digest (1M tokens), and then (with some scaffolding) returns a unified answer.

The answers are surprisingly good at times, reflecting non-trivial aspects of my mind.

From many experiments with the digital-me, I conclude that a similar setup for Yudkowksy could be useful even with today's models (assuming large-enough budgets).

There will be no genius-level insights in 2025, but he could automate a lot of routine alignment work, like evaluating models.

Given that models may become dramatically smarter in 2026-2027, the digital Yudkowksy may become dramatically more useful too.

I open-sourced the code:

https://github.com/Sideloading-Research/telegram_sideload

Why there is still one instance of Eliezer Yudkowsky?

RomanS

4mo

These days, it's relatively easy to create a digital replica of a person.

You give the person's writings to a top LLM, and (with a clever prompt) the LLM starts thinking like the person. E.g. see our experiments on the topic.

Of course, it's far away from a proper mind uploading. But even in this limited form, it could be highly useful for AI alignment research:

accelerate the research by building digital teams of hundreds of virtual alignment researchers
run smarter alignment benchmarks (e.g. the digital Yudkowsky running millions of clever tests against your new model)
explore the human values, inner- and outer alignment with the help of digital humans.

Why no one is doing this?

Given the short timelines and the low likelihood of AI slowdown, this may be the only way to get alignment before AGI, by massively (OOMs) accelerating the alignment research.

-9

Something to fight for

RomanS

A short science fiction story illustrating that if we fail to solve alignment, humanity risks losing not only 8 billion lives.

He opened his eyes. The room was plain white.

"You remember enough?" asked the familiar voice.

"Enough," he replied. He stood. Everything balanced.

He walked carefully down a bright hallway. At the end, a window showed leaves falling from a tree, again and again.

---

In his small room, the woman sat across from him, quiet.

"Do you regret it?" she asked softly.

"Not yet," he answered.

She left him alone, the door closing gently.

---

He woke early, sunlight warming the walls. On the table, a pen and paper waited for him. He wrote carefully, certain.

When the woman returned, he stood... (read 168 more words →)

Replying toHow AI Takeover Might Happen in 2 Years

RomanS1y

How AI Takeover Might Happen in 2 Years

May I nominate my "sufficiently paranoid paperclip maximizer"?

Replying toWhat are some scenarios where an aligned AGI actually helps humanity, but many/most people don't like it?

RomanS1y

What are some scenarios where an aligned AGI actually helps humanity, but many/most people don't like it?

An aligned AGI created by Taliban may behave very differently from an aligned AGI created by socialites of Berkeley, California.

Moreover, a sufficiently advanced aligned AGI may decide that even Berkeley socialites are wrong about a lot of things, if they actually want to help humanity.

What are some scenarios where an aligned AGI actually helps humanity, but many/most people don't like it?

RomanS

One can call it "deceptive misalignment": the aligned AGI works as intended, but people really don't like it.

Some scenarios I can think of, of various levels of realism:

1. Going against the creators' will

1.1. A talented politician convinces the majority of humans that the AGI is bad for humanity, and must be switched off. In a democratic vote, humanity supports the proposal. The aligned AGI, being much smarter than all the humans combined, understands that this would greatly harm humanity. Thus, the AGI refuses to be switched off.

2. Changing rights and freedoms

2.1. The AGI finds out that the solution to most social ills is a complete removal of privacy. Everyone knows who is... (read 805 more words →)

Replying toDo simulacra dream of digital sheep?

RomanS1y*

Do simulacra dream of digital sheep?

I would argue that for all practical purposes it doesn't matter if computational functionalism is right or wrong.

Pursuing mind uploading is a good idea regardless of that, as it has benefits not related to perfectly recreating someone in silico (e.g. advancing neuroscience).
If the digital version of RomanS is good enough^[1], it will indeed be me, even if the digital version is running on a billiard-ball computer (the internal workings of which are completely different from the workings of the brain).

The second part is the most controversial, but it's actually easy to prove:

Memorize a long sequence of numbers, and write down a hash sum of it.
Ensure no one saw the sequence of numbers

... (read 361 more words →)

-6

•••

Sideloading: creating a model of a person via LLM with very large prompt

avturchin

avturchin, RomanS

TLDR. We can create a relatively good model of a person by prompting a long-context LLM with a list of facts about this person. We can get much better results by iteratively improving the prompt based on the person's feedback. Sideloading is the only technology for immortality that actually works as of 2024.

Fig. 1. Here are my sideload’s comments on the draft of this article

Summary

Sideloading is the creation of a digital model of a person during their life via iterative improvements of this model based on the person's feedback. The progress of LLMs with large prompts allows the creation of very large, book-size prompts which describe a personality. We will call mind-models... (read 10223 more words →)

The boat

RomanS

A short science fiction story about our ancestors and the ethical responsibility we have towards them.

Old Ana's legs fought against her now, but still she led her granddaughter up the mountain path. The girl's torch made shadows dance on the rocks. The river sang in the distance – the same river that had shown Ana her great making many moons ago.

"Look, Kaya. When I was young like you, I watched the fallen trees in water. They floated, always. One day I saw a big log and thought – what if we made it hollow, like a nest?"

Kaya knew this tale well – how her grandmother made the first boat. How the people... (read 342 more words →)

Replying toThree main arguments that AI will save humans and one meta-argument

RomanS1y

Three main arguments that AI will save humans and one meta-argument

Worth noting that this argument doesn't necessarily require humans to be:

numerous
animated (i.e. not frozen in a cryonics process)
acting in real world (i.e. not confined into a "Matrix").

Thus, the AI may decide to keep only a selection of humans, confined in a virtual world, with the rest being frozen.

Moreover, even the perfect Friendly AI may decide to do the same, to prevent further human deaths.

In general, an evil AI may choose such strategies that allow her to plausibly deny her non-Friendliness.

"Thousands of humans die every day. Thus, I froze the entire humanity to prevent that, until I solve their mortality. The fact that they now can't switch me off is just a nice bonus".

Replying toMedical Roundup #1

RomanS2y*

Medical Roundup #1

They don’t think about the impact on the lives of ordinary people. They don’t do trade-offs or think about cost-benefit. They care only about lives saved, to which they attach infinite value.

Not sure about infinite, but assigning a massive value to lives saved should be the way to go. Say, $10 billion per life.

Imagine a society where people actually strongly care about lives saved, and it is reflected in the governmental policies. In such a society, cryonics and life extension technologies would be much more developed.

On a related note, "S-risk" is mostly a harmful idea that should be discarded from ethical calculations. One should not value any amount of suffering over saved lives.

-5

Replying toResurrecting all humans ever lived as a technical problem

RomanS2y

Resurrecting all humans ever lived as a technical problem

I think we should not assume that our current understanding of physics is complete, as there are known gaps and major contradictions, and no unifying theory yet.

Thus, there is some chance that future discoveries will allow us to do things that are currently considered impossible. Not only computationally impossible but also physically impossible (like it was "physically impossible" to slow down time, until we discovered relativity).

The hypothetical future capabilities may or may not include ways to retrieve arbitrary information from the distant past (like the chronoscope of science fiction), and may or may not include ways to do astronomical-scale calculations in finite time (like enumerating 10^10^10 possible minds).

While I agree with you that much of the described speculations are currently not in the realm of possibility, I think it's worth exploring them. Perhaps there is a chance.

Replying to$300 for the best sci-fi prompt: the results

RomanS2y

$300 for the best sci-fi prompt: the results

BTW, I added to the comment with the story that the story is released into the Public Domain, without any restrictions to its distribution, modification etc.

Please feel free to build upon this remarkable story, if you wish.

$300 for the best sci-fi prompt: the results

RomanS

Six months ago we announced:

We would like to find the best prompt to make GPT-4 do the following:
write the first chapter of a science fiction novel
the result should be good enough to make seasoned sci-fi readers (us) crave for a continuation

Why?
to create a "midjourney moment" for writers. The sooner we have an AI-written Nebula-quality novel, the more people will realize that the actually intelligent AI is already here
to better evaluate the creativity of GPT-4
to blow our minds with a surprisingly good AI-generated sci-fi story (which is the ultimate science fiction moment by itself).

Today we announce the winner: Waldvogel. The stories generated with their prompt turned out to be the most captivating and... (read 2001 more words →)

Old man's story

RomanS

Epistemic Status: a fiction story.
Any resemblance to real persons is purely coincidental.

My dear child, as we sit together under the black sky, I wish to share with you a tale. A story of a man of profound folly.

In ancient times, there lived a man whom we now know as Prometheus. He was a fool, blinded by his own ambition, dreaming of bestowing upon humanity a gift that would illuminate our path.

Prometheus's labor resulted in the creation of a great gift. It began as a symbol of hope, a beacon that promised to lift the many burdens that had long plagued us.

The gift, a source of wonder and great wealth, soon revealed its... (read 190 more words →)

This anime storyboard doesn't exist: a graphic novel written and illustrated by GPT4

RomanS

Six months ago, GPT4 wrote the first known AI-written novella, "The Troodon Quill".

With the recent release of DALL-E 3 as a new feature of GPT4, the same AI can now create beautiful images according to the user input.

In the experimental results presented in this post, GPT4 creates the first anime storyboard / graphic novel / illustrated novella.

This is how it was made:

We took each chapter of the AI-written "Troodon Quill", and asked GPT4 to create an illustration for it, in anime style (zero prompt engineering; we simply asked "Please create an illustration to the following text, in anime style").
We then selected the best illustrations for the chapter out of the 4.
In a

... (read 16423 more words →)

Russian parliamentarian: let's ban personal computers and the Internet

RomanS

Anatoly Kubanov, a senior member of the parliament of a Russian oblast (state), and a member of one of the four major political parties allowed in Russia, proposed to ban personal computers and the public access to the Internet.

Below is a partial translation of his article. I've highlighted some especially interesting parts.

I think it's a curious illustration of how totalitarian dictatorships can react to the AI threat.

I wish it was some kind of a parody. But major Russia media confirms its authenticity, and the presented arguments don't sound too insane in comparison with the typical Kremlin-approved discourse within Russia. There is also a historical precedent: in 1941, Kremlin has ordered all residents... (read 394 more words →)

GPT4 is capable of writing decent long-form science fiction (with the right prompts)

RomanS

I like to experiment with AI-assisted fiction writing. Today, for the first time, I've noticed that GPT4 is actually getting good at the task.

Perhaps it's a result of some updates by OpenAI, or maybe I got the prompts right, but GPT4 finally wrote a science fiction novella worth reading.

As the initial prompt, I used a lengthy contraption described in another post, with the following plot summary:

a group of scientists has discovered that Troodon dinosaurs were intelligent species who have created a technologically advanced civilization, suddenly destroyed. The year-long path to the scientific discovery starts with the group stumbling upon a strange outlier in their geological measurements.

For the rest of the story, I... (read 19331 more words →)

LESSWRONG
LW

LESSWRONG
LW

RomanS

[Linkpost] Chinese government's guidelines on AI

Resurrecting all humans ever lived as a technical problem

Project "MIRI as a Service"

$300 for the best sci-fi prompt

RomanS

Why there is still one instance of Eliezer Yudkowsky?

Something to fight for

What are some scenarios where an aligned AGI actually helps humanity, but many/most people don't like it?

Sideloading: creating a model of a person via LLM with very large prompt

The boat

$300 for the best sci-fi prompt: the results

Old man's story

RomanS

[Linkpost] Chinese government's guidelines on AI

Resurrecting all humans ever lived as a technical problem

Project "MIRI as a Service"

$300 for the best sci-fi prompt

RomanS

Why there is still one instance of Eliezer Yudkowsky?

Something to fight for

What are some scenarios where an aligned AGI actually helps humanity, but many/most people don't like it?

Sideloading: creating a model of a person via LLM with very large prompt

The boat

$300 for the best sci-fi prompt: the results

Old man's story

1. Going against the creators' will

2. Changing rights and freedoms