I agree this is better (the system rewards only a subset of what it does), but it is still overgeneralizing. Systems could have a different purpose but fallible reward structure (also Goodharting). You could be analyzing the purpose-reward link at the wrong level: political parties want change, so they seek power, so they need donations. This makes it look like the purpose is to just get donations because of rewards to bundlers, but it ignores the rewards at other levels and confuses local rewards with global purpose. Just as a system does a lot of things, so it rewards a lot of things.
That's true and a very important point I wish I had included. I assumed consciousness and some unstated degree of able-bodiedness. A good hit to the head on the way in and/or certain physical limitations, and mere inches of depth will be the determinant.
using urban firewood prices for what was mostly rural consumption.
Bernard Stanford: If you value all informal economy firewood production at market price, and then compare it to extant GDP estimates, you need to make sure ALL informal economic production is similarly valued, or you’ll massively overestimate firewood’s share of GDP. Seems to be what happened!
The approach seems to have a serious flaw in assuming that THIS sector of the informal economy was underestimated, but surely not any OTHER sector.
There are two different problems being raised here.
The outside quote is Zvi pointing out the issue of what price is used to multiply by quantity output to get value estimates. This is a good point because we want to multiply by a representative price, not a too-high price.
The inside quote is a related but distinct issue, which I disagree with. Bernard is arguing that if you are going to include an informal element in GDP, you have to include all informal elements. Sure, it's the ideal, but 1) that's not what's going on here, and 2) even if it were, I'd be fine with it. First, firewood is already semi-included in GDP, this process is just attempting to get a better estimate. Imagine BLS sends people out to grocery stores to price milk and estimate milk consumption, but they only end up getting urban prices and only end up observing 1/3 of the milk consumed in the US. It'd actually be great to add back in the missing consumption. How do you price it? I guess with the only price you have (and Zvi is right that it's a problematic price to use). Second, imagine BLS sends people out to start to price household dishwashing and estimate household dishwashing service provision, but they don't bother to collect household vacuuming. It'd actually still be great to do that, though it'd be difficult to pull off.
So, I think what the authors did is actually a nice exercise, but the price issue means 28% could be an overestimate. How bad of one? The paper discusses Figure A11, recreating Barger's (1955) estimates (though it only goes back to 1869), so what was the GDP share using that one? In the 1870s, the NBER paper in question reports a share of about 8% (on $500M of firewood). Barger reports about $350M of firewood (because of the lower price index), which would imply a share of about 5.4%. If the 28% was a 50% overestimate, that would mean the "true" share in the 1830s (if you assume Barger's price index is "true") could be more like 18.7%. That's still quite high and far from "half an order of magnitude" overestimate, more like 1/5 of one. Is it absurd? Figure 6 here [pdf via academia.edu] has British and Swedish shares at 20% and 40%, respectively.
I too have grown increasingly skeptical that meta-analysis in its typical form does anything all that useful.
Unfortunately, people can be bad at understanding meta-analyses. If you have studies that disagree like 50/50, it's not necessarily true that half did something wrong. It's possible there is a legitimate hidden moderator that changes the effect of the variable (probably being revealed by the meta-analysis but not picked up sufficiently by popular reporting). Or even revealing that half have a fatal flaw would be a contribution of the meta-analysis! Sometimes the effects are not totally comparable, in which case that should either be modeled/adjusted, or excluded (probably already considered by the meta-analyst [though the salient counter-examples where the researcher(s) screw up confirm that not all papers are perfect, though this is not unique to meta-analyses]). It is indeed problematic when a meta-analysis concludes the average effect is the effect, particularly with a bimodal distribution of effect sizes (would be crazy to conclude that in that case!).
The alternative "look at the studies individually" suffers from all the same issues: garbage in, garbage out; hidden moderators; non-comparability; over-concluding from an average. At least meta-analysis brings some systematic thinking to the evaluation of the literature. A strong meta-analysis interacts with these issues and hopefully avoids the pitfalls because it does what a weak meta-analysis does not - it explains inclusion criteria, considers variability, and explains differences rather than just reporting a mean.
Yes, that's what I meant, thanks.
I'm glad she's totally fine. Maybe even a net positive for her and the family on future water safety. It showcases the importance of thinking about bodies of water beyond the prototypes.
A somewhat similar event occurred last weekend with my toddler in a pool. I was less than a foot away from him, as intended, and he was walking around in waist-deep water. Lost his feet but his waist is taller than his arms are long - so he needed me to intervene. He swallowed a little water in the less than two seconds he was sloshing around, but he otherwise didn't care.
The lesson is the same: the bottom needs to be reliable enough that they can regain footing + footing may mean arm lengths + you basically can't count on buoyancy/control (even if he had his arms straight out he might not know how to properly keep his head out, he might not know to hold his breath, and also he might be panicking) = don't go swimming alone.
I think it would be great to formally lay out alternative hypotheses for why this behavior arises, so we can begin to update their likelihoods based on evidence as it accumulates. I have several:
It would be valuable to figure out how variations across models would reflect on these hypotheses. Plus, what other hypotheses are there and what patterns of variation do they predict?
Yep, frequentist null hypothesis significance testing often gets critiqued for the epistemic failures of the people who use it, even when the core of the approach is fine. By the way, the term you're describing is familywise error rate. The key questions are, do we care about that error rate, and who is in the family (answers depend on the research questions)?
Alternatively or in addition to this, you can embrace AI and design new tasks and assignments that cause students to learn together with the AI.
This magical suggestion needs explication.
From what I've seen via Ethan Mollick, instead of copy-pasting, the new assignments that would be effective are the same as the usual - just "do the work," but at the AI. Enter a simulation, but please don't dual-screen the task. Teach the AI (I guess the benefit here is immediate feedback, as if you couldn't use yourself or friend as a sounding board), but please don't dual-screen the task. Have a conversation (again, not in class or on a discussion board or among friends), but please don't dual-screen the task. Then show us you "did it." You could of course do these things without AI, though maybe AI makes a better (and certainly faster) partner. But the crux is that you have to do the task yourself. Also note that this admits the pre-existing value of these kinds of tasks.
Students who will good-faith do the work and leverage AI for search, critique, and tutoring are...doing the work and getting the value, like (probably more efficiently than, possibly with higher returns than) those who do the work without AI. Students who won't...are not doing the work and not getting the value, aside from the signaling value from passing the class. So there you have it - educators can be content that not doing the assignment delivers worse results for the student, but the student doesn't mind as long as they get their grade, which is problematic. Thus, educators are not going quietly and are in fact very concerned about AI-proofing the work, including shifts to in-person testing and tasks.
However, that only preserves the benefit of the courses and in turn degree (I'm not saying pure signaling value doesn't exist, I'm just saying human capital development value is non-zero and under threat). It does not insulate the college graduate from competition in knowledge work from AI (here's the analogy: it would obviously be bad for the Ford brand to send lemons into the vehicle market, but even if they are sending decent cars out, they should still be worried about new entrants).
I agree, the meta-point of selection bias is valid but the direction of bias is unclear.