Hey, I am a Data Science undergraduate and recently I've gotten involved in AIS. I read this post and ended up wanting to build the measurement-error layer you flagged (and that Alexander Barry mentioned in his note). Apparently the raw per-run data is already on the same METR repo.
Having done that I have found at least three things you'll find interesting:
1. You suggested trying a Weibull instead of log-normal for the human timing distribution. What I got is that real within-task residuals are right-skewed with heavy tails (skew +1.09, excess kurtosis 8.2... (read more)
Hey, I am a Data Science undergraduate and recently I've gotten involved in AIS. I read this post and ended up wanting to build the measurement-error layer you flagged (and that Alexander Barry mentioned in his note). Apparently the raw per-run data is already on the same METR repo.
Having done that I have found at least three things you'll find interesting:
1.
You suggested trying a Weibull instead of log-normal for the human timing distribution. What I got is that real within-task residuals are right-skewed with heavy tails (skew +1.09, excess kurtosis 8.2... (read more)