I think that a large and significant chunk of the goal-intelligence plane would be ruled out if moral truths are self-motivating, contrary to what Bostrom claims in his presentation of the orthogonality thesis.
In the seminal paper The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents, Nick Bostrom introduces his Orthogonality Thesis, proposing the independence of goal-content and intelligence level:
The Orthogonality Thesis
Intelligence and final goals are orthogonal axes along which possible agents can freely vary. In other words, more or less any level of intelligence could in principle be combined with more or less any final goal.
Bostrom then goes on to address various objections that might be raised and provides counter... (read 1829 more words →)
Great stuff, and perfect timing as I just read the Toy Models paper yesterday! My immediate intuition for why the least sparse features are dropped in the equal importance case is that they are dropped because they are the most expensive to represent. Being the least sparse, they require more of a dimension than any others, so the tradeoff of number of features versus quality of representation stacks up against them. In other words, the choice for what to do with your last few slots might be to represent say 2 relatively dense features, and squeeze all the others up quite a bit, or to represent say 5 sparser features, requiring less... (read more)