krzhang

Posts

Sorted by New

Wiki Contributions

Comments

Understanding Simpson's Paradox

Hi Vaniver! =D

On the commentary: your eyeballing seems good, but I don't think I ever said anything about relative comparisons between correlation coefficients (namely just overall correlation is positive). As you observed, I could easily make all 3 correlations (blue-only, green only, or blue+green) positive. I don't have any interesting things to say about their relative degrees.

I don't quite see the difference in interpretation from this writing. I agree with basically all the stuff you've written? The fact that the slicing "behaves as a filter", if I interpret it correctly, is exactly the problem here.

I don't know what "have a different origin than Simpson's paradox" means exactly, but here are a few ways they differ and why I say they are "different":

  • a fundamental assumption on Simpson's paradox is that there's some imbalance with the denominators; in your 2x2x2 matrix you can't arbitrarily scale the numbers arbitrarily; all the examples you can construct almost relies on (let's say we are using the familiar batting averages example) the fact that the denominators (row sums) are different.

  • the direct cause of the reversal effect is, as you said, the noise; I don't think Simpson's paradox has anything related to the noise.

Idea: my steel-man version of your argument is that reversal effects arise when you have inhomogenous data, and this is definitely the more general common problem in both situations. In that case I agree. (this is how I teach this class at SPARC, at least).

Welcome to Less Wrong! (July 2012)

Haha hey QC. Remind me sometime to learn the "get ridiculously high points in karma-based communities and learn a lot" metaskill from you... you seem to be off to a good start here too ;)

Welcome to Less Wrong! (July 2012)

I am Yan Zhang, a mathematics grad student specializing in combinatorics at MIT (and soon to work at UC Berkeley after graduation) and co-founder of Vivana.com. I was involved with building the first year of SPARC. There, I met many cool people at CFAR, for which I'm now a curicculum consultant.

I don't know much about LW but have liked some of the things I have read here; AnnaSalamon described me as a "street rationalist" because my own rationality principles are home-grown from a mix of other communities and hobbies. In that sense, I'm happy to step foot into this "mainstream dojo" and learn your language.

Recently Anna suggested I may want to cross-post something I wrote to LW and I've always wanted to get to know the community better, so this is the first step, I suppose. I look forward to learning from all of you.