Entropic Thoughts recently reanalyzed the data I'd shared on the relationship between mask requirements and dance attendance we've seen at BIDA. They conclude:
Other sources account for most of the variation in dance attendance, and masking only plays a small part. The amount of the total variation contributed by masking requirements is 5 %. This number is called the coefficient of determination, and its square root is the correlation: 0.21. This correlation is low enough that we cannot conclude that masking has a significant effect on attendance.
...
"But it still looks like masking has an effect!!" It does. It's just that if the effect is there, it is small enough that we cannot statistically prove an effect with just 44 dances. Assuming the coefficient of determination really is 5 %, and we are aiming for a traditional significance level of 0.05, the sample size curves tell us we would need over 80 dances to be sure of the effect of masking.
The approach they took in their post involves some statistics that make assumptions about the distribution of the data. While these assumptions may well be right, now that we have fast computers we can often use simulations to avoid this. I decided to have a go at analyzing this data with a permutation test.
In this approach, you permute (shuffle) the labels and then check what fraction of permutations led to an outcome at least as extreme as observed. In this case the attendance numbers were:
I decided to operationalize "at least this extreme" as the ratio of the average attendance at mask required to mask optional dances and wrote some code to simulate. With 10M simulations, which coded in very lazy python run for 27s on my Mac, I found that 19.3% of simulations passed that test, which can be expressed as p=0.193.
The bottom line is the same, however: while the pattern we saw is more likely in worlds where mask-optional dances are more popular, there's enough variation from other sources that with 23 dances there's still a decent chance that this apparent difference isn't real.