LESSWRONG
LW

Ariel Kwiatkowski's Shortform

by kwiat.dev
30th May 2020
1 min read
4

2

This is a special post for quick takes by kwiat.dev. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Ariel Kwiatkowski's Shortform
11kwiat.dev
4kwiat.dev
2romeostevensit
-1kwiat.dev
4 comments, sorted by
top scoring
Click to highlight new comments since: Today at 4:20 PM
[-]kwiat.dev5y110

Has anyone tried to work with neural networks predicting the weights of other neural networks? I'm thinking about that in the context of something like subsystem alignment, e.g. in an RL setting where an agent first learns about the environment, and then creates the subagent (by outputting the weights or some embedding of its policy) who actually obtains some reward

Reply
[-]kwiat.dev5y40

Looking for research idea feedback:

Learning to manipulate: consider a system with a large population of agents working on a certain goal, either learned or rule-based, but at this point - fixed. This could be an environment of ants using pheromones to collect food and bring it home.

Now add another agent (or some number of them) which learns in this environment, and tries to get other agents to instead fulfil a different goal. It could be ants redirecting others to a different "home", hijacking their work.


Does this sound interesting? If it works, would it potentially be publishable as a research paper? (or at least a post on LW) Any other feedback is welcome!

Reply
[-]romeostevensit5y20

This sounds interesting to me.

Reply
[-]kwiat.dev1y-1-4

Modern misaligned AI systems are good, actually. There's some recent news about Sakana AI developing a system where the agents tried to extend their own runtime by editing their code/config. 

This is amazing for safety! Current systems are laughably incapable of posing x-risks. Now, thanks to capabilities research, we have a clear example of behaviour that would be dangerous in a more "serious" system. So we can proceed with empirical research, create and evaluate methods to deal with this specific risk, so that future systems do not have this failure mode.

The future of AI and AI safety has never been brighter.

Reply
Moderation Log
Curated and popular this week
4Comments