Link to long comments that I want to pin, but are too long to be pinned:

https://www.lesswrong.com/posts/Zzar6BWML555xSt6Z/?commentId=aDuYa3DL48TTLPsdJ

https://www.lesswrong.com/posts/uMQ3cqWDPHhjtiesc/?commentId=Gcigdmuje4EacwirD

This post is essentially talking about an issue that arises even without AI alignment, and that is relevant for capitalism, and the big issue is that AI will by default give people the ability to replace humans, making those humans essentially superfluous, and often it's not good when a human has no leverage.

This post is a +4 for me, if only because this is the first good argument against something like a capitalist economic order surviving AGI for me, and importantly doesn't ideologize like so many critiques of capitalism do.

Third reason “patterns not holding” is less central an issue than it might seem: the Generalized Correspondence Principle. When quantum mechanics or general relativity came along, they still had to agree with classical mechanics in all the (many) places where classical mechanics worked. More generally: if some pattern in fact holds, then it will still be true that the pattern held under the original context even if later data departs from the pattern, and typically the pattern will generalize in some way to the new data. Prototypical example: maybe in the blegg/rube example, some totally new type of item is introduced, a gold donut (“gonut”). And then we’d have a whole new cluster, but the two old clusters are still there; the old pattern is still present in the environment.

While a trivial version of something like this holds true, the Correspondence principle doesn't apply everywhere, and while there are 2 positive results on a correspondence theorem holding, there is a negative result stating that the correspondence principle is false in the general case of physical laws/rules whose only requirement is that they be Turing-computable, which means that there's no way to make theories all add up to normality in all cases.

More here:

https://www.lesswrong.com/posts/XMGWdfTC7XjgTz3X7/a-correspondence-theorem-in-the-maximum-entropy-framework

https://www.lesswrong.com/posts/FWuByzM9T5qq2PF2n/a-correspondence-theorem

https://www.lesswrong.com/posts/74crqQnH8v9JtJcda/egan-s-theorem#oZNLtNAazf3E5bN6X

https://www.lesswrong.com/posts/74crqQnH8v9JtJcda/egan-s-theorem#M6MfCwDbtuPuvoe59

https://www.lesswrong.com/posts/74crqQnH8v9JtJcda/egan-s-theorem#XQDrXyHSJzQjkRDZc

I enjoyed reading this post; thank you for writing it. LessWrong has an allergy to basically every category Marx is a member of - "armchair" philosophers, socialist theorists, pop humanities idols - in my view, all entirely unjustified.

To be fair here, Marx was kind of way overoptimistic about what could be achieved with central economic planning in the 20th century, because it way overestimated how far machines/robots could go, and also this part where he says communist countries don't need a plan because the natural laws would favor communism, which was bullshit.

More here:

In his review of Peter Singer's commentary on Marx, Scott Alexander writes:

[...] Marx was philosophically opposed, as a matter of principle, to any planning about the structure of communist governments or economies. He would come out and say it was irresponsible to talk about how communist governments and economies will work. He believed it was a scientific law, analogous to the laws of physics, that once capitalism was removed, a perfect communist government would form of its own accord. There might be some very light planning, a couple of discussions, but these would just be epiphenomena of the governing historical laws working themselves out.

The Coherent Extrapolated Volition of a human Individual (CEVI) is a completely different type of thing, than the Coherent Extrapolated Volition of Humanity (CEVH). Both are mappings to an entity of the type that can be said to want things. But only CEVI is a mapping from an entity of the type that can be said to want things (the original human). CEVH does not map from such an entity. CEVH only maps to such an entity. A group of billions of human individuals can only be seen as such an entity, if one already has a specific way of resolving disagreements, amongst individuals that disagree on how to resolve disagreements. Such a disagreement resolution rule is one necessary part of the definition of any CEVH mapping.

I like to state this as the issue that all versions of CEV/group alignment that want to aggregate thousands of people's or more values requires implicitly resolving disagreements in values, which in turn require value-laden choices, and at that point, you are essentially doing value-alignment to what you think is good, and the nominal society is just a society of you.

I basically agree with Seth Herd here, in that instruction following is both the most likely and the best alignment target for purposes of AI safety (at least assuming offense-defense balance issues aren't too severe).

My guess is that this is reasonably plausible assuming the short timelines are in fact going to happen, but it's going to be up against a backdrop of a shock to government competence such that the people who could do a national project completely fail to even get started, let alone complete a herculean effort, since all the possible choices for the role are selected based on loyalty, not competence.

I expect the new administration to break the presidential government and agencies competence by extreme amounts, such that I wouldn't be totally surprised if by the end of the administration, there would be a complete inability to have a national AI project/nationalize the business at all.

The main point here is that it can no longer be just our uncertainty in our map, something else must be added, which was the point.

Another way to say it is that probability can't just be in the mind, so while the probabilities encode our ignorance, it can't be all of the story (according to Wigner functions).

It was way down in the last comment, so maybe you should go to the end of the comment I linked here for more information.

Also, a difference here that doesn't matter for this discussion, but might matter for the general approach, might ultimately be that I disagree with this statement "since functions are a part of the map", because I think the map-territory distinction can often be blurry or fully dissolved in some cases, and also functions can have results when you evaluate them using an algorithm, making them part of the territory (for that specific function).

• Noise: the world is noisy and infinitely detailed. The training data for all but the simplest toy models have some amount of noise in inputs and labels. Your picture of a cat will not be a platonically perfect cat: it will have imperfections due to pixellation, due to atmospheric phenomena and camera artefacts interacting with the integrity of the image; the cat's fur will be affected by accidents of dirt and discoloration. Labels may be garbled or imprecise. Etc. Similarly, text (though it is usually thought of as discrete, and thus seemingly less susceptible to noise than pictures) suffers from external noise: the writer may be affected by distractions in the environment, by texts read recently, and so on. While it's possible to capture some amount of this (e.g. mood) in a predictive speech generation process, there will always be some amount of sufficiently fine-grained random context (that mosquito bite behind your left shoulder that makes you remember a hiking trip with your grandpa and causes your writing to be more wistful) that ultimately must be abstracted out as noise by state-of-the-art ML systems. 

The big reason for this is quantum physics, at a high level, because the uncertainty principles don't allow you to remove all noise from a system, or even arbitrarily much noise, meaning that there can only be finite accuracy to labels and inputs from basically any source:
 

https://en.wikipedia.org/wiki/Quantum_noise

I must admit, I think the "probability is in the mind, not the territory" either vacuously true or possibly false if you think that maps aren't the same things as low-resolution territories.

One example of where probability is in the mind, not the territory is false for non-trivial definitions of maps is Wigner functions, which turn out to be basically equivalent to a wavefunction, where it behaves basically like classical Bayesian probability theory, but with the caveat that for quantum physics, negative probabilities are allowed, and the most important impact here is you can't treat the uncertainty as just ignorance anymore (though they can include our ignorance).

More here:

https://www.lesswrong.com/posts/Y6LhXdGfwsAStMuhr/ackshually-many-worlds-is-wrong#nsebEbJbxqkekTbsK