Wiki Contributions


Apply for Productivity Coaching and AI Alignment Mentorship

Are you offering productivity/performance coaching for new alignment researchers, or coaching + research mentorship for alignment researchers? 

If you're offering research mentorship, it might be helpful to give some examples of the type of research you do

Framings of Deceptive Alignment

Yeah, I totally agree. My motivation for writing the first section was that people use the word 'deception' to refer to both things, and then make what seem like incorrect inferences. For example, current ML systems do the 'Goodhart deception' thing, but then I've heard people use this to imply that it might be doing 'consequentialist deception'. 

These two things seem close to unrelated, except for the fact that 'Goodhart deception' shows us that AI systems are capable of 'tricking' humans. 

This phrase is mainly used and known by the euthanasia advocacy movement. This might also be bad, but not as bad as the association with Jim Jones.

Thoughts on Dangerous Learned Optimization

My initial thought is that I don't see why this powerful optimizer would attempt to optimize things in the world, rather than just do some search thing internally. 

I agree with your framing of "how did this thing get made, since we're not allowed to just postulate it into existence?". I can imagine a language model which manages to output words which causes strokes in whoever reads its outputs, but I think you'd need a pretty strong case for why this would be made in practice by the training process. 

Say you have some powerful optimizer language model which answers questions. If you ask a question which is off its training distribution, I would expect it to either answer the question really well (e.g. it genealises properly), or it kinda breaks and answers the question badly. I don't expect it to break in such a way where it suddenly decides to optimize for things in the real world. This would seem like a very strange jump to make, from 'answer questions well' to 'attempt to change the state of the world according to some goal'. 

But I think if we trained the LM on 'have a good on-going conversation with a human', such that the model was trained with reward over time, and its behaviour would effect its inputs (because it's a conversation), then I think it might do dangerous optimization, because it was already performing optimization to affect the state of the world. And so a distributional shift could cause this goal optimization to be 'pointed in the wrong direction', or uncover places where the human and AI goals become unaligned (even though they were aligned on the training distribution). 

Guidelines for cold messaging people

This list of email scripts from 80,000 Hours also seems useful here

peterbarnett's Shortform

My Favourite Slate Star Codex Posts

This is what I send people when they tell me they haven't read Slate Star Codex and don't know where to start.

Here are a bunch of lists of top ssc posts:

These lists are vaguely ranked in the order of how confident I am that they are good (if interested in psychology almost all the stuff here is good


I think that there are probably podcast episodes of all of these posts listed below. The headings are not ranked by which heading I think is important, but within each heading they are roughly ranked. These are missing any new great ACX posts. I have also left off all posts about drugs, but these are also extremely interesting if you like just reading about drugs, and were what I first read of SSC. 

If you are struggling to pick where to start I recommend either Epistemic Learned Helplessness or Nobody is Perfect, Everything is Commensurable

The ‘Thinking good’ posts I think have definitely improved how I reason, form beliefs, and think about things in general. The ‘World is broke’ posts have had a large effect on how I see the world working, and what worlds we should be aiming for. The ‘Fiction’ posts are just really really good fiction short stories. The ‘Feeling ok about yourself’ have been extremely helpful for developing some self-compassion, and also for being compassionate and non-judgemental about others; I think these posts specifically have made me a better person.


Posts I love:

Thinking good


World is broke and how to deal with it

Fiction (read after moloch stuff I think)


Feeling ok about yourself


Other?? (this is actual crazy, read last)

Hypothesis: gradient descent prefers general circuits

Hmm, this might come down to how independent the parameters are. I think in large networks there will generally be enough independence between the parameters for local minima to be rare (although possible). 

As a toy example of moving from one algorithm to another, if the network is large enough we can just have the output being a linear combination of the two algorithms and up-regulate one, and down regulate the other:

And  is changed from 1 to 0. 

The network needs to be large enough so the algorithms don't share parameters, so changing one doesn't affect the performance of the other.  I do think this is just a toy example, and it seems pretty unlikely to spontaneously have two multiplication algorithms develop simultaneously. 

Hypothesis: gradient descent prefers general circuits

I think there is likely to be a path from a model with shallow circuits to a model with deeper circuits which doesn't need any 'activation energy' (it's not stuck in a local minimum). For a model with many parameters, there are unlikely to be many places where all the deriviatives of the loss wrt all the parameters are zero. There will almost always be at least one direction to move in which decreases the shallow circuit while increasing the general one, and hence doesn't really hurt the loss.

Book notes: "The Origins of You: How Childhood Shapes Later Life"

I'm from Dunedin and went to highschool there (in the 2010s), so I guess I can speak to this a bit.

Co-ed schools were generally lower decile (=lower socio-economic backgrounds) than the single sex schools (here is data taken from wikipedia on this). The selection based on 'ease of walking to school' is still a (small) factor, but I expect this would have been a larger factor in the 70s when there was worse public transport. In some parts of NZ school zoning is a huge deal, with people buying houses specifically to get into a good zone (especially in Auckland) but this isn't that much the case in Dunedin.

Based on (~2010s) stereotypes about the schools, rates of drug use seemed pretty similar between co-ed and all-boys schools, and less in all-girls. And rates of violence were higher in all-boys schools, and less in co-ed and all-girls. But this is just my impression, and likely decades too late. 

Load More