Hm, I disagree! I think I share some of your skepticism, in that the fact that a very large fraction of survey users with long-term personality changes report that those changes were positive doesn't cause me to be confident that I'd believe they were positive, or that a smarter, wiser version of me and others would believe that they're positive, or that the average member of society would believe they're positive, etc. 

However, "almost useless" seems too strong to me; for me at least, it was still a meaningful update to know that people believed there were long-term changes and that they said those changes were positive and not negative. I'd have been much more concerned if people said the changes were negative (I think false positives of good changes are ore common than false negatives on bad changes) and psychedelics would have seemed lower stakes if there'd been fewer reports of long-term changes.

Also I think a bunch of the other questions have fewer issues related to self-evaluation, since a bunch of the questions are either more objective ("how many times did you trip?") or explicitly subjective ("did you experience the trip as a positive one?"). 

But yeah, I agree that evaluation by others would be really valuable. 

>in the case of Krebs and Johansen (2013, 2015), it is ~13% reporting lifetime psychedelic use, while in the ACS readers survey it is ~100%.

Just to be clear to casual readers, this wasn't the whole ACX Readers Survey, I just only looked at the subset that filled out my psychedelic survey and seemed to have actually done psychedelics (i.e. the conclusion "all ACX readers do psychedelics" would be very incorrect). I don't know what fraction of ACX readers have done psychedelics. 

I have those for the people that put them in, but didn't use them. If someone else was keen to do specifical analyses and explained why they'd be interesting, I'd definitely consider asking Scott for permission to share the data or trying to do the analysis myself.