I don't understand why Remmelt going "off the deep end" should affect AI safety camp's funding. That seems reasonable for speculative bets, but not when there's a strong track-record available. 

It is, we’ve been limiting ourselves to readings from the sequence highlights. I’ll ask around to see if other organizers would like to broaden our horizons.

I mean, one of them’s math built bombs and computers & directly influenced pretty much every part of applied math today, and the other one’s math built math. Not saying he wasn’t smart, but no question are bombs & computers more flashy. 

Fixed!

The paper you're thinking of is probably The Developmental Landscape of In-Context Learning.

@abramdemski I think I'm the biggest agree vote for alexander (without me alexander would have -2 agree), and I do see this because I follow both of you on my subscribe tab. 

I basically endorse Alexander's elaboration. 

On the "prep for the model that is coming tomorrow not the model of today" front, I will say that LLMs are not always going to be as dumb as they are today. Even if you can't get them to understand or help with your work now, their rate of learning still makes them in some sense your most promising mentee, and that means trying to get as much of the tacit knowledge you have into their training data as possible (if you want them to be able to more easily & sooner build on your work). Or (if you don't want to do that for whatever reason) just generally not being caught flat-footed once they are smart enough to help you, as all your ideas are in videos or otherwise in high context understandable-only-to-abram notes.

In the words of gwern, 

Should you write text online now in places that can be scraped? You are exposing yourself to 'truesight' and also to stylometric deanonymization or other analysis, and you may simply have some sort of moral objection to LLM training on your text.

This seems like a bad move to me on net: you are erasing yourself (facts, values, preferences, goals, identity) from the future, by which I mean, LLMs. Much of the value of writing done recently or now is simply to get stuff into LLMs. I would, in fact, pay money to ensure Gwern.net is in training corpuses, and I upload source code to Github, heavy with documentation, rationale, and examples, in order to make LLMs more customized to my use-cases. For the trifling cost of some writing, all the worlds' LLM providers are competing to make their LLMs ever more like, and useful to, me.

in some sense that’s just hiring you for any other job, and of course if an AGI lab wants you, you end up with greater negotiating leverage at your old place, and could get a raise (depending on how tight capital constraints are, which, to be clear, in AI alignment are tight).

I think its this

Over the past few days I've been doing a lit review of the different types of attention heads people have found and/or the metrics one can use to detect the presence of those types of heads. 

Here is a rough list from my notes, sorry for the poor formatting, but I did say its rough!

• Bigram entropy
• positional embedding ablation
• prev token attention
• prefix token attention
• ICL score
• comp scores
• multigram analysis
• duplicate token score
• induction head score
• succession score
• copy surpression heads
• long vs short prefix induction head differentiation
• induction head specializations
  • literal copying head
  • translation
  • pattern matching
• copying score
• anti-induction heads
• S-inhibition heads
• Name mover heads
• Negative name mover heads
• Backup name mover heads
• (I don't entirely trust this paper) Letter mover heads
• (possibly too specific to be useful) year identification heads
  • also MLPs which id which years are greater than the selected year
• (I don't entirely trust this paper) queried rule locating head
• (I don't entirely trust this paper) queried rule mover head
• (I don't entirely trust this paper) "fact processing" head
• (I don't entirely trust this paper) "decision" head
• (possibly too specific) subject heads
• (possibly too specific) relation heads
• (possibly too specific) mixed subject and relation heads

And yes, I do think that interp work today should mostly focus on image nets for the same reasons we focus on image nets. The field’s current focus on LLMs is a mistake

A note that word on the street in mech-interp land is that often you get more signal & a greater number of techniques work on bigger & smarter language models over smaller & dumber possibly-not-language-models. Presumably due to smarter & complex models having more structured representations.