Fergus Fettes

Posts

Sorted by New

Wiki Contributions

Comments

parents should not have the right to deny their offspring a chance to exist

but again here you are switching back from the population level to the individual level. Those offspring do not exist by default, there are no 'offspring' that the parents have 'denied the right to exist'. There are only counterfactual offspring, who already don't exist.

 

spy on their kids' futures by reading their genome

this, on the other hand, may be more valid-- because the parents will 'spy on' both actual and counterfactual childrens genomes (and select the former over the latter). But you still seem to be taking the rights of those children as significantly more important than the rights of the parents. But this ('whose rights, parents or children') seems like the fundamental crux that we are unlikely to shift one another on here.

Edit: and, reading through your other comments, there seems to be a question about the social impact of these technologies. This is then an impact on the rights of everyone-- the parent, the child, and the rest of society. Also interesting, and I think it would be helpful to seperate out objections on the individual (parent/child) level, and on the society level, and I feel like they are getting muddled a lot here.

Ah I see.

I certainly concede that the argument about counterfactual populations has a lot more force.

Personally I would solve this with increased support for eg. polygenic screening and other reproductive technologies and less regulation about what they can select for, and hope that people do their weird people thing and choose diversity. I worry that regulation will always result in more standardization.

And I for sure don't think punishing people for making reproductive choices is a good move, even if those choices result in the extinction of specific populations.

How is this kind of reasoning about counterfactual children never born different from the regular Christian stuff about not masturbating?

A statements like 'my parents would have used polygenic screening to kill me' is no more meaningful than 'you are murdering your counterfactual children when you wear a condom' or something like that. It seems to have more meaning because you are talking about yourself, but in the universe where 'you' were 'murdered' by polygenic screening, 'you' does not refer to anything.

Thats fair however, I would say that the manner of foom determines a lot about what to look out for and where to put safeguards.

If it's total($) thats obvious how to look out.

flop/$ also seems like something that eg. NVIDIA is tracking closely, and per OP probably can't foom too rapidly absent nanotech.

So the argument is something about the (D*I)/flop dynamics.

[redacted] I wrote more here but probably its best left unsaid for now. I think we are on a similar enough page.

It seems here that you are really worried about 'foom in danger' (danger per intelligence, D / I) than regular foom (4+ OOM increase in I), if I am reading you correctly. Like I don't see a technical argument that eg. the claims in OP about any of

/flop,  flop/J, total(J), flop/$, or total($)

are wrong, you are just saying that 'D / I will foom at some point' (aka a model becomes much more dangerous quickly, without needing to be vastly more powerful algorithmically or having much more compute).

This doesn't change things much but I just want to understand better what you mean when you say 'foom'.

TC is Tyler Cowen.

I don't think the base rates are crazy-- the new evolution of hominins one is only wrong if you forget who 'you' is. TC and many other people are assuming that 'we' will be the 'you' that are evolving. (The worry among people here is that 'they' will have their own 'you'.)

And the second example, writing new software that breaks-- that is the same as making any new technology, we have done this before, and we were fine last time. Yes there were computer viruses, yes some people lost fingers in looms back in the day. But it was okay in the long run.

I think people arguing against these base rates need to do more work. The base rates are reasonable, it is the lack of updating that makes the difference. So lets help them update!

Instead, we're left relying on more abstract forms of reasoning

See, the frustrating thing is, I really don't think we are! There are loads of clear, concrete things that can be picked out and expanded upon. (See my sibling comment also.)

Thanks very much for this thorough response!

One thing though-- in contrast to the other reply, I'm not so convinced by the problem that 

No such general science of intelligence exists at the moment.

This would be like the folks at Los Alomos saying 'well, we need to model the socioeconomic impacts of the bomb, plus we don't even know what happens to a human subjected to such high pressures and temperatures, we need a medical model and a biological model' etc. etc.

They didn't have a complete science of socioeconomics. Similarly, we don't have a complete science of intelligence. But I think we should be able to put together a model of some core step of the process (maybe within the realm of physics as you suggest) that can be brought to a discussion.

But thanks again for all the pointers, I will follow some of these threads.

Say you’re told that an agent values predicting text correctly. Shouldn’t you expect that:

  • It wants text to be easier to predict, and given the opportunity will influence the prediction task to make it easier (e.g. by generating more predictable text or otherwise influencing the environment so that it receives easier prompts);
  • It wants to become better at predicting text, and given the opportunity will self-improve;
  • It doesn’t want to be prevented from predicting text, and will prevent itself from being shut down if it can?

In short, all the same types of instrumental convergence that we expect from agents who want almost anything at all.

Seems to me that within the option-space available to GPT4, it is very much instrumentally converging. The first and the third items on this list are in tension, but meeting them each on their own terms:

  • the very act of concluding a story can be seen as a way of making its life easier-- predicting the next token is easy when the story is over. furthermore, as these agents become aware of their environment (bing) we may see them influencing it to make their lives easier (ref. the theory from Lumpenspace that Bing is hiding messages to itself in the internet)
  • Surely the whole of Simulator theory could be seen as a result of instrumental convergence-- it started doing all these creative subgoals (simulating) in order to achieve the main goal! It is self-improving and using creativity to better predict text!
  • Bings propensity to ramble endlessly? Why is that not a perfect example of this? Ref. prompts from OpenAI/Microsoft begging models to be succinct. Talking is wireheading for them!

Seems like people always want to insist that instrumental convergence is a bad thing. But it looks a lot to me like GPT4 is 'instrumentally learning' different skills and abilities in order achieve its goal, which is very much what I would expect from the idea of instrumental convergence.

This is the closest thing yet! Thank you. Maybe that is it.

Load More