Open thread, September 25 - October 1, 2017

Thomas

If it's worth saying, but not worth its own post, then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should start on Monday, and end on Sunday.

4. Unflag the two options "Notify me of new top-level comments on this article" and "Make this post available under..." before submitting

If it's worth saying, but not worth its own post, then it goes here.

Notes for future OT posters:

1. Please add the 'open_thread' tag.

2. Check if there is an active Open Thread before posting a new one. (Immediately before; refresh the list-of-threads page before posting.)

3. Open Threads should start on Monday, and end on Sunday.

4. Unflag the two options "Notify me of new top-level comments on this article" and "Make this post available under..." before submitting

I'm not a statistician, but I happen to have some intuitions and sometimes work out formulas or find them on the web.

I have a bunch of students that took a test each day. The test of each day had a threshold score out of, say, 100 points. Scores under the threshold are considered insufficient.

I don't know whether of the two is true:

I can either use the tests to evaluate the students, or the students to evaluate the tests.
I can evaluate the students using the tests and the tests using the students at the same time.

The option 2. seems counterintuitive at first sight, especially if one wants to be epistemically sound. It seems more intuitive at second sight, though. I think it might be analogous to how you can evaluate a circular flow of feedback by using linear algebra (cfr. LW 2.0 discussions).

Some other context: In my evaluation model I would rather not only consider whether the scores were sufficient or not, but consider how much they were sufficient or insufficient, possibly after opportunely transforming them. Also, I want the weights of the scores to decay exponentially. I would also rather use a bayesian approach.

Is this reasonable, and where can I find instructions on how to do so?

From a Bayesian perspective, you calculate P(S|T) and P(T|S) at the same time, so it doesn't really matter. What does matter, and greatly, are your starting assumptions and models: if you have only one for each entity, you won't be able to calculate how much some datum is evidence of your model or not.

3IlyaShpitser9y

You have an experimental design problem: https://en.wikipedia.org/wiki/Design_of_experiments. The way that formalism would think about your problem is you have two "treatments" (type of test, that you can vary, and type of student), and an "outcome" (how a given student does on a given test, typically some sort of histogram that's hopefully shaped like a bell). Your goal is to efficiently vary "treatment" values to learn as much as possible about the causal relationship between how you structure a test, and student quality, and the outcome. ---------------------------------------- There's reading you can do on this problem, it's a classical problem in statistics. Both Jerzy Neyman and Ronald Fisher wrote a lot about this, the latter has a famous book. In fact, in some sense this is the problem of statistics, in the sense that modern statistics could be said to have grown out of, and generalized from, this problem.