All of Raphaël S's Comments + Replies

Examples of Causal Abstraction

Yeah, but writing a sequence seems more fun than doing a literature review.

AI Alignment YouTube Playlists

I use those playlists a lot.  Thank you.

Deepmind's Gato: Generalist Agent

Perhaps your team could have helped write the safety part?
Or to deliberate whether the weights and code should be made public?

The name of the paper is very meaningful (AGA ≈ AGI, obviously on purpose), so in order to get in condition, I think it is important that your safety team takes part in this kind of paper.

Perhaps your team could have helped write the safety part?

I think it would be a bad use of our time to write the safety sections of all the papers that could be progress towards AGI (there are a lot of them). It seems a lot better to focus on generally improving knowledge of safety, and letting individual projects write their own safety sections.

Obviously if an actually x-risky system is being built it would be important for us to be involved but I think this was not particularly x-risky.

Tbc we would have been happy to chat to them if they reached out; I'm... (read more)

Deepmind's Gato: Generalist Agent

Demis Hassabis is not even mentioned in the paper. Does that mean this is considered a minor paper for DeepMind?

Deepmind's Gato: Generalist Agent

There is no fire alarm for AGIs? Maybe just subscribe to the DeepMind RSS feed…

On a more serious note, I'm curious about the internal review process for this article, what role did the DeepMind AI safety team play in it? In the Acknowledgements, there is no mention of their contribution.

I don't think we played any particular role in the review process (or if we did I'm not aware of it, which tbc is totally possible, I don't know everything that everyone on DeepMind safety does).

What would you want us to bring up if we were to participate in a review process?

High-stakes alignment via adversarial training [Redwood Research report]

Ah, "The tool-assisted attack went from taking 13 minutes to taking 26 minutes per example."

Interesting. Changing the in-distribution (3oom) does not influences much the out-distribution (*2)

2paulfchristiano16d
I think that 3 orders of magnitude is the comparison between "time taken to find a failure by randomly sampling" and "time taken to find a failure if you are deliberately looking using tools."
High-stakes alignment via adversarial training [Redwood Research report]

I do not understand the "we only doubled the amount of effort necessary to generate an adversarial counterexample.". Aren't we talking about 3oom?

5Raphaël S17d
Ah, "The tool-assisted attack went from taking 13 minutes to taking 26 minutes per example." Interesting. Changing the in-distribution (3oom) does not influences much the out-distribution (*2)
High-stakes alignment via adversarial training [Redwood Research report]

Super work! It must have required a crazy amount of technical and manual work.

The fact that you manage to reduce the number of failures by 3 orders of magnitude is quite impressive. Did you separate train and test set at the beginning of the project?

In conclusion, using a 300M parameter model to supervise a 10 times larger model seems to work well, it gives a lot of hope.

Is it desirable for the first AGI to be conscious?

I just discovered this debate thanks to a YouTube recommendation from the Institute for the Future of Life. I find the formulation of the question by Anderw Serazin very well put.

The Yes Team:  We would identify with it better, and it would help with creativity.  You would want your digital assistant to be able to help you by modeling you,  it seems necessary to model the human.

No team: It's easier to deal with IA safety without it, the possibility to create infinite suffering is bad, unpleasant task in the world are better done by non-consc... (read more)

1Raphaël S18d
I just discovered this debate thanks to a YouTube recommendation from the Institute for the Future of Life. I find the formulation of the question by Anderw Serazin very well put. The Yes Team: We would identify with it better, and it would help with creativity. You would want your digital assistant to be able to help you by modeling you, it seems necessary to model the human. No team: It's easier to deal with IA safety without it, the possibility to create infinite suffering is bad, unpleasant task in the world are better done by non-conscious agent. I think the no team is right. Interesting: during the questions, Yoshua Bengio advocates decoupling moral status from subjective experience (Moral status: would be the role in society of the entity !?). And then he proposes the following taxonomy of the concept of consciousness: * Subjective experience: trivially already incorporated in modern deep learning because each neural network has learned its own representation of the world. * self-awareness: useful notion for the agent who moves in the world * emotion: already present in reinforcement learning in a primitive way.
Is it desirable for the first AGI to be conscious?

The difference between primary consciousness and higher order consciousness is not clear to me. Is it the same thing as access consciousness and meta cognition?

Is it desirable for the first AGI to be conscious?

"and higher order consciousness, which came to only humans with the acquisition of language. A machine with primary consciousness will probably have to come first"

The language models like GPT3 appear to be unstoppable,  and are much more versatile than the models of other modalities.

So maybe a machine with higher order consciousness will come first?

Is it desirable for the first AGI to be conscious?

Yeah, I know, I just wanted to begin answering with this and to present in one sentence (and without mentioning it, my bad...) the concept of neural correlates of consciousness

Is it desirable for the first AGI to be conscious?

Are you using "conscious" and "sentient" as synonyms? → Yes. Maybe sentient is consciousness with only valence but it is basically the same thing. But you're right, I shouldn't have used both worlds at the same time without further explanation.

 Do you think some, most, or all humans are conscious? → Yes, I think most humans are conscious.  Why do you think this? Because I am conscious, but I'm not sure if I really answer your question here.

"I'm not sure what it means for it to be "desirable" for an entity to be conscious." → Yes this is a good qu... (read more)

Is it desirable for the first AGI to be conscious?

I agree that today the concept of consciousness is very poorly defined, but I think that in the future it will be possible to define it in a way that will make sense, or at least that we will be able to correct our current intuitions.

How can one tell if a human is conscious?

In humans, we have clues. For example, it is possible to experiment by varying the duration of a light stimulus. There are stimuli of very short duration that are processed by visual areas V1 and V2, but which do not go up to the parietal cortex. For stimuli of slightly longer duration,... (read more)

1TAG19d
That's the easy case ..each of us we can tell we are conscious by introspection, and a normal-seeming and behaving person isn't that different.
Key questions about artificial sentience: an opinionated guide

Either consciousness is a mechanism that has been recruited by evolution for one of its abilities to efficiently integrate information, or consciousness is a type of epiphenomenon that serves no purpose.

Personally I think that consciousness, whatever it is, serves a purpose, and has an importance for the systems that try to sort out the anecdotal information from the information that deserves more extensive consideration. It is possible that this is the only way to effectively process information, and therefore that in trying to program an agi, one naturally comes across it

1superads911mo
Consciousness definitely serves a purpose, from an evolutionary perspective. It's definitely an adaptation to the environment, by offering a great advantage, a great leap, in information processing. But from there to say that it is the only way to process information goes a long way. I mean, once again, just think of the pocket calculator. Is it conscious? I'm quite sure that it isn't. I think that consciousness is a very biological thing. The thing that makes me doubt the most about consciousness in non-biological systems (let alone in the current ones which are still very simple) is that they don't need to sleep and they can function indefinitely. Consciousness seems to have these limits. Can you imagine not ever sleeping? Not ever dying? I don't think such would be possible for any conscious being, at least one remotely similar to us.
Call For Distillers

Interesting.

  • distillation: because the blog post should be shorter than the main paper.
  • blending: Because we need to reduce the inferential gap by explaining the prerequisites.
  • dilution:  a corollary of the two previous ones
Omicron Variant Post #1: We’re F***ed, It’s Never Over

Just curious, what's the problem with "nu" in verbal conversation ?

2rossry6mo
It is pronounced identically to the adjective "new".
9jimv6mo
It sounds like "new". I imagine conversations going like: "So, this nu variant, huh?" "Which new variant? We're getting minor variations all the time." "But the nu one..." And that would get even trickier if you want to some newer new variant that succeeds this one. "Wait, when you say [?new | ?nu], are you talking about the old nu variant or the new new variant?"
[Letter] Advice for High School #1

Visual Studio Code lets you perform most tasks directly from the keyboard. You can even use a Vim emulator if you like.

But more importantly, "faster edits gives me a faster iteration time" : when developing complex stuff, your writing speed is clearly not the limiting factor. Using proper file structure visualization and navigation tools is way more important.

[Letter] Advice for High School #1

You can SSH directly with VS Code in just one click with the remote extension.

[Letter] Advice for High School #1

Why would Vim be important ? I mean, everybody uses VS Code nowadays, and it's much more easy and versatile and no need to read a book to understand it...

2cata1y
Probably if you are a programmer you will be spending thousands of hours SSHed into this or that over your life, so you had better be able to view and edit files there.
0lsusr1y
Everybody? [https://www.lesswrong.com/posts/yxFkuyPANtL6GSwiC/the-majority-is-always-wrong] Vim keybindings (whether you're using Vim, emacs, whatever) are faster than standard keybindings. If a lot of your time is spent editing code or writing then even a small pays off a lot over the decades. That alone is sufficient reason [https://xkcd.com/1205/] to learn Vim. Perhaps more importantly, faster edits gives me a faster iteration time. I suspect this increases my effective working memory since there is less time between edits for me to forget things. There is evidence working memory and g -factor are closely related, perhaps even the same thing. General intelligence is extremely important to writing advanced software. Anything which can give you a similar edge ought to be extremely valuable.
Agency in Conway’s Game of Life

Could you just explain a bit "will only be likely to contain arbitrary patterns of sizes up to log(10^120)" please ? Or give some pointers with other usage of such calculation ?

This is very much a heuristic, but good enough in this case.

Suppose we want to know how many times we expect to see a pattern with n cells in a random field of area A. Ignoring edge effects, there are A different offsets at which the pattern could appear. Each of these has a 1/2^n chance of being the pattern. So we expect at least one copy of the pattern if n < log_2(A).

In this case the area is (10^60)^2, so we expect patterns of size up to 398.631. In other words, we expect the ash to contain any pattern you can fit in a 20 by 20 box.

Covid 12/24: We’re F***ed, It’s Over

upvoted.

But ln(370/20)/ln(2) = 4.2. This means that the new strain doubled 4 times between September and mid-November, suggesting a typical doubling time of just over two weeks.

This is approximately what is observed at the end of December.

But indeed, I don't understand why the number of infected people suddenly decreases at the end of November. An explanation would be helpful.

Where can we find the source saying that there were about 20 cases of new strains in September?

2Neel Nanda1y
As henryaj says above, the UK was in a national lockdown Nov 5 - Dec 2. Accounting for a lag in catching it -> positive test, that matches the graph reasonably well
The Equation of Knowledge

I read this book two years ago when it was published in French. I found it incredibly exciting to read, and that's what motivated me to discover this site and then move on to a master's degree in machine learning.

This book saved me a lot of time in discovering Bayesianism, and made a much deeper change in my way of thinking than if I had simply read a textbook of Bayesian machine learning.

I am of course happy to have read the sequences, but I think I am lucky to have started with the equation of knowledge which is much shorter to read and which provides the theoretical assurances, motivation, main tools, enthusiasm and pedagogy to engage in the quest for Bayesianism.

Transportation as a Constraint

Good Post!

Here is another point. The population of a city is constrained by the agricultural area accessible in less than 3-4 days, which is the time corresponding to the storage time of vegetables and fruits. During antiquity, Paris was the biggest town in France and was inhabited by 10,000 inhabitants, which corresponds to the population fed by a circle of arable land within a 3-day radius of oxcarts. If in the future transport becomes more constrained (oil shortage?), we should then expect to see the size of the cities greatly reduced.

If you speak French you can look at Jean-Marc Jancovici's lecture https://www.youtube.com/watch?v=Ci_cz18A2F8