(...) the term technical is a red flag for me, as it is many times used not for the routine business of implementing ideas but for the parts, ideas and all, which are just hard to understand and many times contain the main novelties.
- Saharon Shelah
For my most of my writing see my short-forms (new shortform, old shortform)
Twitter: @FellowHominid
Personal website: https://sites.google.com/view/afdago/home
Yeah this seems pretty sensible. It's probably closely connected to causal scrubbing and John Wentworth's notion of natural abstraction as variables invariant under resampling.
AI x-risk is going mainstream. With that I've seen a lot of ill-informed drive-by takes. Bravo on actually immersing yourself in the arguments. This looks excellent - I love the four perspectives. Next time a family member asks I will point them towards this post.
The relevant literature here is Jacob Cannell's analysis. If he's right about the math than indeed human brains are already within one OOM of pareto-optimality. Of course this doesn't mean that Superintelligence is impossible or that AGI won't be completely wild (and Cannell certainly doesn't say so!). Just that the argument 'human brains are already optimal' is actually technically true.
https://www.lesswrong.com/posts/xwBuoE9p8GE7RAuhd/brain-efficiency-much-more-than-you-wanted-to-know
From these vague terms it's a little hard to say what you have in mind. They sound pretty deep to me however.
It seems your true rejection is not really about deep ideas per se, more so the particular flavor of ideas popular on this website.
Perhaps it would be an idea to write a post on why you are bullish on these research directions?
Yes this seems clearly true.
Tho I would think that Nate's a too subtle thinker to think that AI assistance is literally useless - just that most of the 'hardest work' is not easily automatable which seems pretty valid.
i.e. in my reading most of the hard work of alignment is finding good formalizations of informal intuitions. I'm pretty bullish on future AI assistants helping, especially proof assistants, but this doesn't seem to be a case where you can simply prompt gpt-3 scaled x1000 or something silly like that. I understand Nate thinks that if it could do that it is secretly doing dangerous things.
Correlated equilibria are more general, more realistic equilibrium concept than Nash equilibria. They can arise in situations where by making use of a public signal the players can improve on the Nash equilibria. Providing that public signal is a huge value vacuum begging to be Pareto-optimised
I think this is exactly illustrating John's point no?
[fwiw I think John's overstating things a little bit. Certainly, one can be a good CEO without being able to do some of the highly specialized engineering needed in your product]
I like this classification. Clear, crisp, easy to use. I'll see if I will utilize it in my daily thinking.
Great stuff Jeremy!
Two basic comments:
1. Classical Learning Theory is flawed and predicts that neural networks should overfit when they don't.
The correct way to understand this is through the lens of singular learning theory.
2. Quantilizing agents can actually be reflectively stable. There's work by Diffractor (Alex Appel) on this topic that should become public soon.