Outside of the interpretability research space, do you know of other interesting examples of different techniques being graded on different curves?
Electric vehicles? Early electric vehicles were worse than gas cars on all axis other than the theoretical promise of the technology. However, they were (and still are, ie formula E) graded on separate curves. The fairly straight-forward analogy I'm trying to make is that maybe it's worthwhile treating early technologies gently, as now I think most people are pretty impressed by electric cars.
Although obviously there are significant differences here (consumer market vs helping engineers, etc), I think this could be a useful metaphor to try out arguments in these sequences on to judge their reasonableness.
Electric vehicles? Early electric vehicles were worse than gas cars on all axis other than the theoretical promise of the technology. However, they were (and still are, ie formula E) graded on separate curves. The fairly straight-forward analogy I'm trying to make is that maybe it's worthwhile treating early technologies gently, as now I think most people are pretty impressed by electric cars.
Although obviously there are significant differences here (consumer market vs helping engineers, etc), I think this could be a useful metaphor to try out arguments in these sequences on to judge their reasonableness.