[repost from amor et licentia]

The psycho- and social technologies I’m most excited about are usually not the thoroughly evidence-backed ones that my friends encounter in their sessions with licensed therapists. This seems insane to some people, so I want to explain myself.

Let me start with two stories.

  1. I once knew a person who applied to a special needs school as a provider for horse-assisted therapy. The job interview went great, until she was asked whether she is certified in the leading method of horse therapy. Unfortunately, she had to confess that she could not provide any such certification. Because she had designed the method and the certification process herself.
  2. Once, I lead a workshop for processing AI x-risk anxiety that was based on the Work That Reconnects and adapted to rationalist culture. A participant asked: "...but is this backed by scientific studies?", to which I responded: "No. Just by 50 years of R&D among a bunch of hippies." The asker stared at me in dismay and disbelief, and I went on with my workshop.

I think it's good when people like my acquaintance’s potential employer or my aspiring rationalist have an appropriate amount of suspicion towards snake oil salespeople like us. Mind that survivorship bias is a thing and most things people try don’t actually work.

At the same time, I think both my acquaintance and I are entirely justified in our unlicensed loiterings. Because believe it or not - any therapeutic intervention that is now standardized and deployed on mass-scale has once not been backed by scientific evidence. And, the innovators who developed them have usually not been certified by the institutions that have been built to protect and preserve their heritage.

Freud was considered a crackpot when he first suggested that actually, how peoples' childhoods play out might have an influence on how they behave and misbehave as adults. When the Stoics laid the foundations of cognitive behavioral therapy, the scientific method wasn't even around. When Carl Rogers developed client-centered therapy, it was considered blasphemous that a psychologist dare do something that borders on psychotherapy, which back then was reserved to medical professionals. When the various schools of humanistic psychology emerged, that might have been impossible without the intellectual influence of outcasts and acidheads like Alan Watts or Aldous Huxley. Similarly, the current mindfulness-based third wave of psychotherapy would be unthinkable if some bums in India several millennia ago hadn’t decided to see what happens when you just sit very, very still for a while. Without any double blind experiments to reassure them while their minds disintegrated and went down all kinds of scary avenues.

In other words: The leading edge of cultural innovation never happens in health insurance-paid sessions with licensed therapists.

My young rationalist may be right to mistrust people like me. Meanwhile, I feel like I’m also right to trust that the past decade of studying human minds theoretically, on the meditation cushion, and in relationships, prepared me for the real world.

And thus, I skip the psychological papers and immerse myself in the healing circles of the world directly. Sometimes that’s just good fun. Sometimes, I find a tool that works so well that it seems worth spreading.

Then, I go forth and teach it to everyone who wants to listen and doesn’t bother asking for my license.

6

New Comment
21 comments, sorted by Click to highlight new comments since: Today at 1:20 PM

… any therapeutic intervention that is now standardized and deployed on mass-scale has once not been backed by scientific evidence.

And, often, is still not backed by scientific evidence, even after it’s been deployed on a mass scale.

Freud was considered a crackpot when he first suggested that actually, how peoples’ childhoods play out might have an influence on how they behave and misbehave as adults.

As far as we know, therapy based on Freudian ideas is nothing more than pseudoscience, and most of the claims about the effectiveness of such things are baseless. This stuff was considered nonsense; then—in the manner of fashion trends and passing fads—were believed to be effective; and now, once again and increasingly, is understood to be nonsense after all.

Given the replication crisis, and recent developments in various social-science fields, it seems likely (and other evidence points this way) that the same is true of most or all of the other forms of therapy that you mention. (See also the dodo bird verdict.)

Similarly, the current mindfulness-based third wave of psychotherapy would be unthinkable if some bums in India several millennia ago hadn’t decided to see what happens when you just sit very, very still for a while. Without any double blind experiments to reassure them while their minds disintegrated and went down all kinds of scary avenues.

It seems like what happens if you just sit very, very still for a while is that your mind disintegrates and goes down all kinds of scary avenues. This is a bad thing! Going crazy is bad, and not good.

It really seems like the rationalists who mistrusted you were right to mistrust you, but that you, on the other hand, are wrong to “trust that the past decade of studying human minds theoretically, on the meditation cushion, and in relationships, prepared me for the real world”.

In other words: The leading edge of cultural innovation never happens in health insurance-paid sessions with licensed therapists.

Of course it doesn’t—who would have thought otherwise? But just because something happens in some other place does not, in fact, mean that said thing is anything but nonsense.

So, in other words - I am wrong, hippies are wrong, and most if not all therapies that look so far like they are backed by evidence are likely wrong, too.

Who or what do you suggest we turn to for fixing our stuff?

Who said there has to be any solution? The universe makes us no such guarantee. The answer to your question could very well be “nobody can fix your stuff; suffer”.

(Well, until the singularity, when godlike friendly AIs can rewrite our whole brains to eliminate flaws, or some such speculative thing. But that can be said of anything, and is irrelevant now; I mention it for completeness only—yes, there may not be any physical law that prevents any given problem from being solved, but that doesn’t mean that we can actually solve it.)

Agreed. But sitting around and sulking is a bummer, so I rather keep learning, exploring, and sometimes finding things that work for me.

Sure, but the conclusion that any given approach you’ve found actually works must be arrived at in the usual way—by updating on evidence—and the prior probability is low. And such a favorable conclusion about the approaches you list in the OP is unwarranted.

(Edited to add note)

(Note: this is my third comment on this topic, and as such, any further comments will be delayed by the rate limit.) If you would like to see further responses, I believe there is some sort of setting which you can use to enable further comments from me.)

I did not recommend any particular intervention in my post. I just tried to explain some part of my understanding of how new psycho- and social technologies are generated, and what conclusions I draw from that.

If you expect most if not all established therapeutic interventions to not survive the replication crisis - what would you consider sufficient evidence for using or suggesting a certain intervention?

For example, a friend of mine felt blue today and I sent them a video of an animated dancing seal without extensively googling for meta-analyses on the effect of cute seal videos on peoples' moods beforehand. Would you say I had sufficient evidence to assume that doing so is better than not doing so? Or did I commit epistemic sin in making that decision? This is an honest question, because I don't yet get your point.

I just tried to explain some part of my understanding of how new psycho- and social technologies are generated, and what conclusions I draw from that.

I think the problem is that some techniques that are created that way work while others don't. 

When using techniques we care about whether those we use work. I personally do think that updating on evidence is important and if your goal is technique creation then it matters. 

I personally think that making something a "licensed technique" is often a way to create an environment where updating on evidence on how the technique can be improved gets harder but I completely agree with Said that updating on evidence is crucial.

When it comes to sending your friend videos of animated seals, I would expect that for that to work well it's important that you understand your friend well enough to know that they appreciate getting videos of seals. Likely, you got positive feedback for it. 

I don't think one should generalize from that technique to send everyone who feels blue seal videos. Before doing that it would be good to build a better model of when people are happy to get seal videos and when it annoys them. I would expect that the relationship to the seal video sender also matters.

I don't think sending them such a video counts as therapy, even if you're literally doing it to make someone feel better., because it's short term and minor, and you've probably tried a lot of short term, minor interventions in your life, enough to get an idea of what might work. And if you're wrong, the consequences would be minor.

If someone had depression and you claimed you could cure the depression long term by sending them a cute video, I would indeed say you don't have enough evidence.

Yea, but I don't remember claiming anywhere that I can cure anybody's depression, and don't really intend to ever do that...?

It's an illustrative example. Even if you don't believe that therapy can cure depression specifically, it's supposed to be able to cure things like it.

The problem is your comparison of theraputic interventions to sending someone a cute video. The cute video is there to cure a short term, minor, issue. The therapy is there to cure a long term major, issue. These are different.

Some questions we should distinguish here:

  1. How confident are you that the thing you're looking into here is going to pay off? (Whatever that means - give you something that lets you reliably improve people's lives more than existing methods?) I get the sense that you're reasonably confident, but I don't think you explicitly say. Gordon's comment talks about "high EV and low probability", so I guess Gordon got a different sense from your writing than I did.

  2. How confident should you be that it's going to pay off? (It is well known that people are sometimes poorly calibrated.)

  3. How confident should other people be that it's going to pay off? You might have illegible evidence that it's a fruitful field of study, in which case it seems correct for you to be more confident than others. (Analogy: if you're selling a used car that you know is not a lemon, it seems correct for you to be more confident of this than your buyer, because distinguishing sellers of lemons from non-lemons is difficult.) But also, I think a lot of people think they have illegible evidence for something when what they actually have is wishful thinking or something. I feel like that's especially the case in this field of study.

    In any case, I think most people should have low confidence that what you're looking into should pay off; and I think you should probably agree with me on this, even if you think you personally should have high confidence.

  4. Should you be looking into this thing? My instinctive reaction is "why not, seems harmless, worst case scenario is you waste your time and your time is yours to waste". But actually I agree with what I think Said is saying, that a common result of people looking into this sort of thing is to go crazy, and this is not actually harmless in the same way that doing drugs is not harmless. So I dunno. Not endorsed, but maybe my thinking here is: "if you are not worried about going crazy and taking precautions to avoid, you should not be looking into this; but I'm not a cop, this is not community consensus, so I might judge you for doing things I think are dangerous but I'm not going to additionally judge you for doing things that are widely known to be dangerous."

  5. Should you be teaching this thing? Under what conditions? I think my feeling is that the conditions should probably not rule out teaching it entirely, but same caveat as with (4), you should be cautious about yourself and your students going crazy. Also I think you should expect skepticism from your students per (2), and I think you should explicitly endorse it. Like, your reaction to "is this backed by scientific studies?" gives me a vibe of "I don't care and I don't think you should care", and I think that's a bad vibe. I want vibe closer to "your skepticism is great here, totally warranted; but I'm explicitly asking you to set it aside for a bit".

In other words: The leading edge of cultural innovation never happens in health insurance-paid sessions with licensed therapists.

ART actually grew out of EMDR, which was by then licensed.

EMDR has been licensed, but it was not a result of "insurance-paid sessions". 

It was made up by Francine Shapiro. Shapiro got diagnosed with breast cancer and then sought out NLP. She overcame her cancer and gave media interviews about how great NLP. She explained the world the ideas about how NLP people figured out meaning of eye accessing cues. 

Because of the stereotypes against NLP, she made up an origin story that doesn't mention NLP and back then there was no internet that allowed people to google her to learn about her NLP background. 

See https://link.springer.com/article/10.1007/s10879-023-09582-x

As far as ART goes, it seems to be developed by Laney Rosenzweig. On her website she writes:

A Licensed Marriage and Family Therapist for over 3 decades in the mental health field.  In addition to her private practice, Laney trains licensed mental health professionals in the use of ART.  “The most difficult aspect of discussing ART is the fact that it sounds too good to be true.”

While she should have more than one decade of practice before she developed ART in 2008, she avoids speaking about in what she trained before that point on her website. 

While EMDR seems to be part of what inspired her, she chose to avoid making it easy to find out her background, maybe for similar reasons that Shapiro obfuscated hers.

Wow, that article has some delicious allegations.

Upon careful consideration, Shapiro’s accounting for the origins of EMDR is questionable. This is because saccades during everyday functioning are physiologically invisible (Moses & Hart, 1987). Rosen (1995) addressed this concern by asking six individuals if they could experience eye movements while walking around and thinking of positive and negative thoughts. None were successful.

After publication of Rosen’s challenge to Shapiro’s origin story she alerted members of an EMDR listserv (traumatic-stress@freud.apa.org, September 12, 1996) that a responsive critique would be published by a “world renowned perceptual psychology researcher.“ Shapiro was referring to Robert Welch [...]

Welch’s praise of Shapiro’s sensitivity and diligence, following as it did Shapiro’s praise of his expertise, occurred without either party disclosing a likely conflict of interest: they had a relationship and married (Carey, 2019). Remarkably, a similar failure to disclose involved Shapiro’s earlier marriage in 1969 to Gerald Puk (retrieved March 1, 2021 from https://www.nycmarriageindex.com/) when both were students in Brooklyn, New York. [...]

Licensed in New York State and without academic credentials (PsycInfo, retrieved on March 1, 2021) Puk was not on the faculty at the Professional School of Psychological Studies in California: yet somehow he became a member of Shapiro’s dissertation committee (Shapiro, 1988). As with Welch, Shapiro’s relationship history with Puk remained undisclosed to relevant parties (Anne Hanley, dissertation committee member, personal communication March 10, 2021).

...

It was in 1985 that Shapiro published an article in Holistic Life Magazine and discussed Neuro-Linguistic Programming (NLP) theories on various topics including the importance of eye movement patterns (Shapiro, 1985, pp. 41–43):

Neuro-Linguistic Programming is a technique developed over eight years ago. . .. It has been dubbed the “Super-Achievers” technology because the research team studied the most successful people they could find in law, medicine, business and psychology to see what made them so successful. .. In NLP, the key is that since people share the same neurological system, responses are predictable, verifiable, and repeatable. In other words, Neuro-Linguistic Programming is scientifically rather than merely theoretically based.

One of the findings of the Neuro-Linguistic Programming research is that all people cross-culturally (with the exception of the Basque nationality) show how they are thinking by the way their eyes move. . . Even without their saying a word, if you watch their eyes carefully, you can determine whether they are seeing a picture, hearing, or feeling something. As a further refinement, you can tell if they are remembering something or constructing it. Thousands have learned to walk on red-hot coals without injury, using Neuro-Linguistic Programming.. . Using Neuro-Linguistic Programming, people are shown how to tap into their own unlimited source of personal power, get rid of even the basic fear of fire and change their physiology to walk across the coals. The major dilemma that people are confronted with in Neuro-Linguistic Programming is the question of manipulation and free will. Since the powerful technology allows you to practically “read minds” and have people respond automatically in any way you choose, there is a distinct ethical issue.

(For those who aren't familiar: Wiki on firewalking)

Of course, it is possible that a person who appears to be generally dishonest, and over-credulous (and/or consciously dishonest) about the magic powers of NLP, might have stumbled upon a genuinely correct technique.  But it would seem prudent to, at the very least, discount any evidence that came from that person and anyone connected to her.

For anyone who wants more juicy reading, there are articles from a few NLP people listed on https://www.nlp.ch/pdfdocs/Historie_EMDR_Wingwave.pdf . I myself would go with Connirae Andreas perspective. (That's the wife of Steve Andreas who among other things is responsible for Transform Your Self which was previously discussed on LessWrong) 

Well, there goes that bit of overconfidence. Thanks.

Be wary of survivorship bias! Of course, everything that is licensed and proven to work was not at some point. But so is everything that was tried out and ineffective / harmful, and (hopefully) still not licensed.

But things that work, accumulate evidence that they do, and good chances get licensed. That's the way science works.

I don't blame anyone for trying things out, in fact we need people who do to figure out what actually works. But it's also perfectly reasonable not to want to do that.

Yep, added a reference to survivorship bias to the text. Thanks.

This is an excellent point that I think is under-appreciated, especially by would-be and new rationalists.

It's really tempting to dismiss stuff that looks like it shouldn't work. And to some extent that's fair, but only because most stuff doesn't work, including the stuff that looks like it should work. Things have to be tried, and even then the result of our best attempts at controlled experiments sometimes return inconclusive results. Determining causal relationships is hard, and when you find something that seems to work sometimes you just have to go with it whether it makes sense or not since reality is going to be how it is whether or not it fits within your model.

Meanwhile we've got to get on with the project of living our best lives whether the things we do seem like they should lead to winning. If you want to win, you've got to sometimes be willing to take the status hit, get out there, and do something weird that people will think is nuts to try because it sometimes works. It doesn't mean throwing out everything you know, but it does mean bothering to go live in the real world where things are messy and you can't always figure out what's up.

Time will tell. If you keep doing crazy stuff after it becomes clear it doesn't work, sure, that's a mistake. But it's also a mistake not to check. If you never verified that you don't have psychic powers, can't teleport, that healing crystals don't work, etc. then you're also going to miss out on things like weird therapy modalities that do work for some people for unclear reasons and idiosyncratic dietary changes that dramatically improve your life but would make someone else's life worse.

As you point out, the leading edge is not respectable. Being on the leading edge has costs, but also rewards. It's a test of one's strength of a rationalist to be able to go all in on something that has high EV and low probability to see if it might work. The only real failure is the failure to update once the evidence comes in.

Peer review and its consequences have been a disaster for the human race.

You're making valid points: Not everything that is not backed up by scientific studies is useless or harmful. Also, for innovation to happen, we need to try things, because new approaches are just not necessarily backed by scientific evidence. 


Yet, it is important to also not trust what is *not* backed by evidence. Lobotomy was a thing. And believing that people with mental instability were possessed by ghosts, and so on. 


Some useful handles are: has it worked for the past centuries and stood the test of time? Is it an area where intuition can easily be deployed? Can the underlying mechanisms be explained by some well established science? What are the risks when it goes wrong? 
 

Finally: Sometimes it does make sense to ask a specialist. A friend tried to self cure with MDMA and went into a big depression (yep this is anecdotal). Some major mental illnesses benefit most from medication. Hippie culture doesn't focus on curing specific symptom clusters, but focusses on general human/communal well-being, and thus generates generalists by default.

New to LessWrong?