Right, you're interested in syntactic measures of information, more than a physical one My bad.
the initial conditions of the universe are simpler than the initial conditions of Earth.
This seems to violate a conservation of information principle in quantum mechanics.
On #4, which I agree is important, there seems to be some explanation left implicit or left out.
#4: Middle management performance is inherently difficult to assess. Maze behaviors systematically compound this problem.
But middle managers who are good at producing actual results will therefore want to decrease mazedom, in order that their competence be recognized. Is it, then, that incompetent people will be disproportionately attracted to - and capable of crowding others out from - middle management? That they will be attracted is a no-brainer, but that they will crowd others out seems to depend on further conditions not specified. For example, if an organization lets people advance in two ways, one through middle management, another through technical fields, then it naturally diverts the competent away from middle management. But short of some such mechanism, it seems that mazedom in middle management is up for grabs.
When I read
To be clear, if GNW is "consciousness" (as Dehaene describes it), then the attention schema is "how we think about consciousness". So this seems to be at the wrong level! [...] But it turns out, he wants to be one level up!
I thought, thank goodness, Graziano (and steve2152) gets it. But in the moral implications section, you immediately start talking about attention schemas rather than simply attention. Attention schemas aren't necessary for consciousness or sentience; they're necessary for meta-consciousness. I don't mean to deny that meta-consciousness is also morally important, but it strikes me as a bad move to skip right over simple consciousness.
This may make little difference to your main points. I agree that "There are (presumably) computations that arguably involve something like an 'attention schema' but with radically alien properties." And I doubt that I could see any value in an attention schema with sufficiently alien properties, nor would I expect it to see value in my attentional system.
how to quote
Paste text into your comment and then select/highlight it. Formatting options will appear, including a quote button.
People often try to solve the problem of counterfactuals by suggesting that there will always be some uncertainty. An AI may know its source code perfectly, but it can't perfectly know the hardware it is running on.
How could Emmy, an embedded agent, know its source code perfectly, or even be certain that it is a computing device under the Church-Turing definition? Such certainty would seem dogmatic. Without such certainty, the choice of 10 rather than 5 cannot be firmly classified as an error. (The classification as an error seemed to play an important role in your discussion.) So Emmy has a motivation to keep looking and find that U(10)=10.
Thanks for making point 2. Moral oughts need not motivate sociopaths, who sometimes admit (when there is no cost of doing so) that they've done wrong and just don't give a damn. The "is-ought" gap is better relabeled the "thought-motivation" gap. "Ought"s are thoughts; motives are something else.
Technicalities: Under Possible Precisifications, 1 and 5 are not obviously different. I can interpret them differently, but I think you should clarify them. 2 is to 3 as 4 is to 1, so I suggest listing them in that order, and maybe adding an option that is to 3 as 5 is to 1.
Substance: I think you're passing over a bigger target for criticism, the notion of "outcomes". In general, agents can and do have preferences over decision processes themselves, as contrasted with the standard "outcomes" of most literature like winning or losing money or objects. For example, I can be "money pumped" in the following manner. Sell me a used luxury sedan on Monday for $10k. Trade me a Harley Davidson on Tuesday for the sedan plus my $5. Trade me a sports car on Wednesday for the Harley plus $5. Buy the sports car from me on Thursday for $9995. Oh no, I lost $15 on the total deal! Except: I got to drive, or even just admire, these different vehicles in the meantime.
If all processes and activities are fair game for rational preferences, then agents can have preferences over the riskiness of decisions, the complexity of the decision algorithm, and a host of other features that make it much more individually variable which approach is "best".
If there were no Real Moral System That You Actually Use, wouldn't you have a "meh, OK" reaction to either Pronatal Total Utilitarianism or Antinatalist Utilitarianism - perhaps whichever you happened to think of first? How would this error signal - disgust with those conclusions - be generated?
Shouldn't a particular method of inductive reasoning be specified in order to give the question substance?