cdkg

LESSWRONG
LW

cdkg — LessWrong

7mo

Tom Davidson from Forethought Research and I have a new paper responding to some recent skeptical takes on the Singularity Hypothesis (e.g. this one). Roughly half the paper is philosophical and the other half is empirical. Both halves argue that we should take the Singularity Hypothesis more seriously than many people have been taking it of late. I'm sharing this with the community here because (i) I think it will be of general interest, and (ii) this is a project I'm still working on, and I would appreciate feedback from the community.

Here's the abstract:

The singularity hypothesis posits a period of rapid technological progress following the point at which AI systems become able

... (read 178 more words →)

Does natural selection favor AIs over humans?

cdkg

I wanted to share a new paper from the special issue on AI safety that I'm editing, which takes up the influential idea that evolutionary theory gives us some reason to think that the project of value alignment is bound to fail and (in my opinion) shows that it has serious problems.

If you don't have institutional access to the article, I'm also hosting it on my personal website here: https://www.cd.kg/wp-content/uploads/2024/10/selfish_machine.pdf

[Linkpost] A Case for AI Consciousness

cdkg

cdkg, Simon Goldstein

Just wanted to share a new paper on AI consciousness with Simon Goldstein that members of this community might be interested in. Here's the abstract:

It is generally assumed that existing artificial systems are not phenomenally conscious, and that the construction of phenomenally conscious artificial systems would require significant technological progress if it is possible at all. We challenge this assumption by arguing that if Global Workspace Theory (GWT) — a leading scientific theory of phenomenal consciousness — is correct, then instances of one widely implemented AI architecture, the artificial language agent, might easily be made phenomenally conscious if they are not already. Along the way, we articulate an explicit methodology for thinking about how to apply scientific theories of consciousness to artificial systems and employ this methodology to arrive at a set of necessary and sufficient conditions for phenomenal consciousness according to GWT.

[Linkpost] Will AI avoid exploitation?

cdkg

As some of you may know, I'm editing a special issue of the journal Philosophical Studies on AI safety (along with @Dan H ). I thought I'd share the first paper from the issue, which deals with some issues in AI safety theory that have been frequently discussed on LessWrong.

Here's the abstract:

A simple argument suggests that we can fruitfully model advanced AI systems using expected utility theory. According to this argument, an agent will need to act as if maximising expected utility if they’re to avoid exploitation. Insofar as we should expect advanced AI to avoid exploitation, it follows that we should expect advanced AI to act as if maximising expected utility. I spell out this argument more carefully and demonstrate that it fails, but show that the manner of its failure is instructive: in exploring the argument, we gain insight into how to model advanced AI systems.

You can find the paper here: https://link.springer.com/article/10.1007/s11098-023-02023-4

Replying toShutdown-Seeking AI

cdkg3y*

Shutdown-Seeking AI

My intention is not to criticize you in particular!

Let me describe my own thought process with respect to the originality of work. If I get an academic paper to referee and I suspect that it's derivative, I treat it as my job to demonstrate this by locating a specific published work that has already proposed the same theory. If I can't do this, I don't criticize it for being derivative. The epistemic rationale for this is as follows: if the experts working in an area are not aware of a source that has already published the idea, then even if the idea has already been published somewhere obscure, it is useful for... (read more)

Replying toShutdown-Seeking AI

cdkg3y

Shutdown-Seeking AI

I should clarify that I think some of Gwern's other points are valuable — I was just quite put off by the beginning of the post.

Replying toShutdown-Seeking AI

cdkg3y

Shutdown-Seeking AI

I'm referring to this exchange:

Christopher King: I believe this has been proposed before (I'm not sure what the first time was).
Gwern: This has been proposed before (as their citations indicate), and this particular proposal does not seem to introduce any particularly novel (or good) solutions.
Simon Goldstein: Is there other work you can point us to that proposes positively shutdown-seeking agents?
Gwern: No, I haven't bothered to track the idea because it's not useful.

I find it odd that so many people on the forum feel certain that the proposal in the post has already been made, but none are able to produce any evidence that this is so. Might the present proposal perhaps be... (read more)

Replying toShutdown-Seeking AI

cdkg3y

Shutdown-Seeking AI

"LessWrong is an online forum and community dedicated to improving human reasoning and decision-making. We seek to hold true beliefs and to be effective at accomplishing our goals. Each day, we aim to be less wrong about the world than the day before."

As an academic interested in AI safety and and a relative outsider to LessWrong, I've been somewhat surprised at the collective epistemic behavior on the forum. With all due respect to Gwern, repeating claims that work has already been done and then refusing to substantiate them is an epistemic train wreck. Comments that do this should be strongly downvoted, and posters that do this should be strongly discouraged. Also, it... (read more)

-2

Replying toThe Crux List

cdkg3y

The Crux List

If everything is a crux, is anything a crux?

Replying toLanguage Agents Reduce the Risk of Existential Catastrophe

cdkg3y

Language Agents Reduce the Risk of Existential Catastrophe

Thanks for this. It sounds like we actually agree on most points (in light of your last paragraph).

We discuss concerns very similar to your A. and B. in section 6. It would be helpful for us if you could identify the parts of our discussion there that you don't agree with.

You write:

It will derive from the prompts, and from it’s long-term memory (which will be likely be given to it), what kind of dumb outer loop is running on the outside. If it has internal misaligned desires, it will manipulate the outer loop according to them, potentially generating the expected visible outputs for deception.

Imagine you're an LLM like GPT-4. Hundreds of thousands... (read more)

Replying toMinimum Viable Exterminator

cdkg3y

Minimum Viable Exterminator

If we're worried about all possible paths from AI progress to human extinction, I think these conditions are too strong. The system doesn't need to be an agent. As avturchin points out, even if it is an agent, it doesn't need to be able to act in the real world. It doesn't even need to be able to accept human extermination as a goal. All that is required is that a human user be able to use it to form a plan that will lead to human extinction. Also, I think this is something that many people working on AI safety have realized — though you are correct that most research attention has been devoted (rather inexplicably, in my opinion) to worries about superintelligent systems.

Replying toLanguage Agents Reduce the Risk of Existential Catastrophe

cdkg3y

Language Agents Reduce the Risk of Existential Catastrophe

Thanks for the feedback! I agree that language agents are relatively new, and so our claims about their safety properties will need to be empirically verified.

You write:

One example: "The functional roles of these beliefs and desires are enforced by the architecture of the language agent."
I think this is an extremely strong claim. It also cannot be true for every possible architecture of language agents. As a pathological example, wrap the "task queue" submodule of BabyAGI with a function that stores the opposite task it has been given, but returns the opposite task to what it stored. The plain english interpetation of the data is no longer accurate.
The mistake is to assume that

cdkg

cdkg, Simon Goldstein

This post was written by Simon Goldstein, associate professor at the Dianoia Institute of Philosophy at ACU, and Cameron Domenico Kirk-Giannini, assistant professor at Rutgers University, for submission to the Open Philanthropy AI Worldviews Contest. Both authors are currently Philosophy Fellows at the Center for AI Safety.

Abstract: Recent advances in natural language processing have given rise to a new kind of AI architecture: the language agent. By repeatedly calling an LLM to perform a variety of cognitive tasks, language agents are able to function autonomously to pursue goals specified in natural language and stored in a human-readable format. Because of their architecture, language agents exhibit behavior that is predictable according to the laws... (read 7574 more words →)

The Polarity Problem [Draft]

Dan H

Dan H, cdkg, Simon Goldstein

This is a draft written by Cameron Domenico Kirk-Giannini, assistant professor at Rutgers University, and Simon Goldstein, associate professor at the Dianoia Institute of Philosophy at ACU, as part of a series of papers for the Center for AI Safety Philosophy Fellowship's midpoint. Dan helped post to the Alignment Forum. This draft is meant to solicit feedback. The authors would especially welcome pointers to work related to the topic which they have not cited.

Abstract:

If it is possible to construct artificial superintelligences, it is likely that they will be extremely powerful. A natural question is how many such superintelligences to expect. Will the future be shaped by a single superintelligence (a unipolar outcome),... (read 13004 more words →)

Replying toUnderspecification of Oracle AI

cdkg3y

Underspecification of Oracle AI

Hello,

The idea would be to consider a scenario in which it is something like a law of nature that the predictions of oracles can never be read, in just the same way that the authors are considering a scenario in which it is something like a law of nature that oracles do not and cannot exist.

Replying toUnderspecification of Oracle AI

cdkg3y*

Underspecification of Oracle AI

Instead of having counterfactual oracles operate as though they do not and cannot exist, why not have them operate as though the predictions of oracles can never be read? Designing them in this way would also allow us to escape the following worry from your post:

"...consider a single oracle predicting under the counterfactual that it does not exist. When it is approached with an important question, it has strong evidence that people want to ask that question to an oracle, and since it does not exist it predicts that a new counterfactual oracle will be built to be asked the question. This process is repeated recursively, with answers propagating back down the... (read more)

LESSWRONG
LW

LESSWRONG
LW

Language Agents Reduce the Risk of Existential Catastrophe

The Polarity Problem [Draft]

[Linkpost] A Case for AI Consciousness

[Linkpost] Will AI avoid exploitation?

cdkg

cdkg

Rebooting the Singularity

Does natural selection favor AIs over humans?

[Linkpost] A Case for AI Consciousness

[Linkpost] Will AI avoid exploitation?

Language Agents Reduce the Risk of Existential Catastrophe

The Polarity Problem [Draft]

cdkg

Language Agents Reduce the Risk of Existential Catastrophe

The Polarity Problem [Draft]

[Linkpost] A Case for AI Consciousness

[Linkpost] Will AI avoid exploitation?

cdkg

cdkg

Rebooting the Singularity

Does natural selection favor AIs over humans?

[Linkpost] A Case for AI Consciousness

[Linkpost] Will AI avoid exploitation?

Language Agents Reduce the Risk of Existential Catastrophe

The Polarity Problem [Draft]