I honestly don't understand what that thing is, actually.
This was also my first response when reading the article, but on second glance I don't think that is entirely fair. The argument I want to convey with "Everything is chemicals!" is something along the lines of "The concept that you use the word chemicals for is ill-defined and possibly incoherent and I suspect that the negative connotations you associate with it are largely undeserved.", but that is not what I'm actually communicating.
Suppose I successfully convince people that everything is, in fact, chemicals, people start using the word chemicals in a strictly technical sense and use the word blorps for what is currently the common sense definition of chemicals. In this situation "Everything is chemicals!" stops being a valid counterargument, but blorps is still just as ill-defined and incoherent a concept as it was before. People correctly addressed the concern I raised, but not the concern I had, which suggest that I did not properly communicate my concern in the first place.
There isn't an obvious question that, if we could just ask an Oracle AI, the world would be saved.
"How do I create a safe AGI?"
Edit: Or, more likely, "this is my design for an AGI, (how) will running this AGI result in situations that I would be horrified by if they occure?"
I don't think it is realistic to aim for no relevant knowledge getting lost even if your company loses half of its employees in one day. A bus factor of five is already shockingly competent when compared to any company I have ever worked for, going for a bus factor of 658 is just madness.
One criticism, why bring up Republicans, I'm not even a Republican and I sort of recoiled at that part.
Agreed. Also not a Republican (or American, for that matter), but that was a bit off putting. To quote Eliezer himself:
In Artificial Intelligence, and particularly in the domain of nonmonotonic reasoning, there's a standard problem: "All Quakers are pacifists. All Republicans are not pacifists. Nixon is a Quaker and a Republican. Is Nixon a pacifist?"
What on Earth was the point of choosing this as an example? To rouse the political emotions of the readers and distract them from the main question? To make Republicans feel unwelcome in courses on Artificial Intelligence and discourage them from entering the field?
Funding this Journal of High Standards wouldn't be a cheap project
So where is the money going to come from? You're talking about seeing this as a type of grant, but the amount of money available for grants and XPrize type organizations is finite and heavily competed for. How are you going to convince people that this is a better way of making scientific progress than the countless other options available?
> If you only get points for beating consensus predictions, then matching them will get you a 0.
Important note on this: Matching them guarantees a 0, implementing your own strategy and doing poorer than the consensus could easily get you negative marks.
Also teaching quality will be much worse if teachers are different people than those actually doing the work, a teacher who works with what he is teaching gets hours of feedback everyday on what works and what does not, a teacher who only teaches has no similar mechanism, so he will provide much less value to his students.
No objectsion to the rest of your post, but I'm with Elizer on this. Teaching is a skill that is entirely separate from whatever subject you are teaching and this skill also strongly influences the amount of value a teacher can provide to their students. If you combine the tasks you end up selecting/training for two separate skillsets, which means you get people that are ill optimized for at least one of their tasks.
Maybe we can have the healer-doctors oversee the curriculum taught by the teacher-doctors?
I read the source before reading the quote and was expecting a quote from The Flash.
Correct, but it is a kind of fraud that is hard to detect and easy to justify to oneself as being "for the greater good" so the scammer is hoping that you won't care.
Rationality isn't just about being skeptical, though, and there is something to be said for giving people the benefit of the doubt and engaging with them if they are willing to do so in an open manner. There are obviously limits to the extend to which you want to do so, but so far this thread has been an interesting read so I wouldn't worry to much about us wasting our time.