No standard metric for CFAR workshops?

ioannes

11 No standard metric for CFAR workshops?

6th Sep 2018

3 min read

11

Update: CFAR used standard metrics in its 2015 study, which I didn't know about when drafting this post. It doesn't appear that they tracked these metrics in their most recent impact report.

My outstanding questions are in this comment.

Update #2: CFAR replies to outstanding questions here.

It seems strange that CFAR doesn't use a standardized metric to track the impact of its workshops over time.

From CFAR's mission statement:

CFAR exists to try to make headway in this domain – the domain of understanding how human cognition already works, in practice, such that we can then start the process of making useful changes, such that we will be better positioned to solve the problems that really matter.

A couple of frameworks from psychology could serve as useful metrics for assessing progress towards this mission:

Was there a change in a participant's big 5 personality traits after workshop attendance?
Was there a change in a participant's Raven's Progressive Matrices performance after workshop attendance?

As far as I can tell, CFAR's thesis is that cognitive changes will drive changes in behavior (and correspondingly, impact on the world).

I'd expect big cognitive changes to result in changes in big 5
personality traits. Specifically, I'd expect improved cognition to result in decreased neuroticism & increased conscientiousness.

I'd also expect big cognitive changes to result in improved performance on the Raven's Matrices.

In other words, if CFAR workshops drive big changes in cognition, I'd expect these changes to reflect on well-validated psychological measures. If there's not a before-workshop/after-workshop change in these measures, that would be evidence that CFAR workshops are not causing big cognitive changes in workshop participants.

As far as I know, workshop participants aren't being assessed on measures like this, so it's hard to know what impact the workshops are actually having.

CFAR current metric is "increase in expected impact" or IEI:

In May 2016, we set out to count the number of alumni who have had an increase in expected impact due to their involvement with CFAR by sending out a survey to our alumni...

For each person’s responses, we manually coded whether it seemed like 1) their current path was high-impact, 2) their current path was substantially better than their old path, and 3) CFAR played a significant role in this change. We counted someone as having an “increase in expected impact” (IEI) if they met all three criteria.

18% of workshop participants surveyed had an IEI.

A metric like IEI is better than no metric at all, but it suffers from limitations:

IEI relies on retrospective self-report (i.e. at the time of survey, respondent thinks back to what they were doing before their workshop, what the workshop was like, and what they did afterwards. They then synthesize all this into a story about what change the workshop had.)

In contrast, a big 5 trait survey relies on immediate self-report (i.e. at time of survey, respondent answers about how things are for them right then).

This strikes me as more reliable than retrospective self-report.

A Raven's Matrices test relies on cognitive performance at time of survey, which seems even higher signal than a big 5 survey.

IEI was developed in-house, so it is hard to compare the CFAR workshop to other interventions on the basis of IEI.

In contrast, many interventions measure change in big 5 traits & Raven's Matrices performance.

I'm bringing this up because I found the lack of a standardized, well-validated metric surprising when I thought about it.

It seems plausible that CFAR is already tracking metrics like this privately. If that's the case, I'm curious why they are held privately.

It could also be that CFAR isn't tracking an outcome metric like big 5 trait change or Raven's Matrices performance. If that's case, I'm curious about why not – the surveys are cheap to administer, and it seems like they would yield valuable information about how CFAR is doing.

Center for Applied Rationality (CFAR)

Personal Blog

11

New Comment

Rendering 0/28 comments, sorted by

top scoring

(show all) Click to highlight new comments since: Today at 10:33 AM

Moderation Log

11 No standard metric for CFAR workshops?

by ioannes

6th Sep 2018

3 min read

11

Update: CFAR used standard metrics in its 2015 study, which I didn't know about when drafting this post. It doesn't appear that they tracked these metrics in their most recent impact report.

My outstanding questions are in this comment.

Update #2: CFAR replies to outstanding questions here.

It seems strange that CFAR doesn't use a standardized metric to track the impact of its workshops over time.

From CFAR's mission statement:

CFAR exists to try to make headway in this domain – the domain of understanding how human cognition already works, in practice, such that we can then start the process of making useful changes, such that we will be better positioned to solve the problems that really matter.

A couple of frameworks from psychology could serve as useful metrics for assessing progress towards this mission:

Was there a change in a participant's big 5 personality traits after workshop attendance?
Was there a change in a participant's Raven's Progressive Matrices performance after workshop attendance?

As far as I can tell, CFAR's thesis is that cognitive changes will drive changes in behavior (and correspondingly, impact on the world).

I'd expect big cognitive changes to result in changes in big 5
personality traits. Specifically, I'd expect improved cognition to result in decreased neuroticism & increased conscientiousness.

I'd also expect big cognitive changes to result in improved performance on the Raven's Matrices.

As far as I know, workshop participants aren't being assessed on measures like this, so it's hard to know what impact the workshops are actually having.

CFAR current metric is "increase in expected impact" or IEI:

In May 2016, we set out to count the number of alumni who have had an increase in expected impact due to their involvement with CFAR by sending out a survey to our alumni...

For each person’s responses, we manually coded whether it seemed like 1) their current path was high-impact, 2) their current path was substantially better than their old path, and 3) CFAR played a significant role in this change. We counted someone as having an “increase in expected impact” (IEI) if they met all three criteria.

18% of workshop participants surveyed had an IEI.

A metric like IEI is better than no metric at all, but it suffers from limitations:

IEI relies on retrospective self-report (i.e. at the time of survey, respondent thinks back to what they were doing before their workshop, what the workshop was like, and what they did afterwards. They then synthesize all this into a story about what change the workshop had.)

In contrast, a big 5 trait survey relies on immediate self-report (i.e. at time of survey, respondent answers about how things are for them right then).

This strikes me as more reliable than retrospective self-report.

A Raven's Matrices test relies on cognitive performance at time of survey, which seems even higher signal than a big 5 survey.

IEI was developed in-house, so it is hard to compare the CFAR workshop to other interventions on the basis of IEI.

In contrast, many interventions measure change in big 5 traits & Raven's Matrices performance.

I'm bringing this up because I found the lack of a standardized, well-validated metric surprising when I thought about it.

It seems plausible that CFAR is already tracking metrics like this privately. If that's the case, I'm curious why they are held privately.

Center for Applied Rationality (CFAR)

Personal Blog

11

Mentioned in

42Survey Results: 10 Fun Questions for LWers

New Comment

Rendering 0/28 comments, sorted by

top scoring

(show all) Click to highlight new comments since: Today at 10:33 AM

Moderation Log

More from ioannes

Curated and popular this week

28Comments

No standard metric for CFAR workshops? — LessWrong

Comment Permalink

Ben Pace8y340

For people ignorant of statistics (like I am), I checked and $d$ measures the change of mean between two distributions, proportional to the variance (i.e. $(m e a n_{1} - m e a n_{2}) \div s t a n d a r d D e v i a t i o n$ ).

An effect size in the range $0.2 - 0.5$ is considered 'small' by Cohen, and conscientiousness was increased (with $p < 0.001$ ) with $d = 0.24$ which seems to me notable for such a small intervention (~4 days), though I haven't got much intuition around these things and maybe it's actually quite a big intervention. Neuroticism decreased with $d = 0.17$ .

If you want to jump to the bit in the above article that lists all the effect sizes measured, search for 'The table below summarizes the study results'. The biggest effect sizes were in the places (I think) where CFAR tried to come up with their own metrics to measure what they're trying to do better.

The biggest effect size was on one of these measures, for the metric 'Emotions Help Rather Than Hinder', and I'll quote that section in full below.

Emotions Help Rather Than Hinder

We created a single item measure of how participants relate to their emotions: “When you feel emotions, do they mostly help or hinder you in pursuing your goals?” The CFAR workshop emphasizes the value of emotions as sources of data and motivation, and includes techniques for increasing the alignment between one’s emotions, goals, and behavior. Research on emotion regulation provides some of the theoretical background for this approach, with the findings that it generally is not helpful to suppress emotions and it often is helpful to reframe situations so that they elicit different emotions (Gross, 2002).

There was an increase in the extent to which participants evaluated their emotions as helping them rather than hindering them, d = 0.41 (t(129) = 4.13, p < .001).

I feel like I'll be bringing this up in conversation any time that anyone says the rationality community is too focused on hyper-logical/S2 thinking.

Anyway, this is just me highlighting some stuff I found interesting, to overcome trivial inconveniences for others, but there's a ton more interesting stuff in the post.

See in context