dividing problems into theory and empiricism, specifying your theory exactly then looking for evidence to either confirm or deny the theory, or finding evidence to later form an exact theory.
The main advantage of having this feedback loop is that the deviations from the search for "truth" are self-correcting. Less-than-rational thinking may slow you down, but it will still let you get to a better model of the world eventually. On the other hand, even the most rational thinking without the feedback inevitably results in very inaccurate models.
Basically, the scientific method alone completely trumps rationality alone, though of course you get best results when combining the two.
Good post.
A couple of nitpicks...
Similarly, a lot of modern medicine is rational, but not too scientific. A doctor sees something and it looks like a common ailment with similar symptoms they've seen often before, so they just assume that's what it is. They may run a test to verify their guess.
Actually, this illustrates scientific thinking; the doctor forms a hypothesis based on observation and then experimentally tests that hypothesis.
Even math curriculums are structured around calculus instead of the much more useful statistics and data science placing ridiculous hurdles for the typical non-major that most won't surmount.
Actually, the natural sciences (physics in particular) are heavily dependent on calculus. Ditto for engineering. In fact, a solid understanding of Bayesian statistics requires a grounding in calculus. So, I don't think it is true that statistics and data science are "much more useful" than calculus.
I was struggling to word the doctor parapgraph in a manner which was succinct but still got the idea across. I think query worded it better.
On math curriculum, that advanced classes build off of calculus is a function of current design. They could recenter courses around statistics and have calculus be an extension of it. Some of the calculus course would need to be reincorporated into the stats courses, but a lot of it wouldn't. You're going to have a hard time convincing me that trigonometry a̶n̶d̶ ̶v̶e̶c̶t̶o̶r̶s̶ are a necessary precursor for regression analysis or Bayes' theorem. The minority of students in physics and engineering that need both calculus and statistics should not dictate how other majors are taught. Fixing the curriculum isn't an easy problem, but they've had more than a century to solve it and there seems to be little movement in this direction.
You're going to have a hard time convincing me that... vectors are a necessary precursor for regression analysis...
So you're fitting a straight line. Parameter estimates don't require linear algebra (that is, vectors and matrices). Super. But the immediate next step in any worthwhile analysis of data is calculating a confidence set (or credible set, if you're a Bayesian) for the parameter estimates; good luck teaching that if your students don't know basic linear algebra. In fact, all of regression analysis, from the most basic least squares estimator through multilevel/hierarchical regression models up to the most advanced sparse "p >> n" method, is built on top of linear algebra.
(Why do I have such strong opinions on the subject? I'm a Bayesian statistician by trade; this is how I make my living.)
On math curriculum, that advanced classes build off of calculus is a function of current design.
Not really; Bayesian statistics really does build on calculus. This is true of the Bayesian methodology itself - not just curriculum design. Once you get beyond introductory probability problems using Bayes' rule, Bayesian statistics quickly gets into probability density functions, sampling from posterior distributions and so forth; all of this is based on calculus. I'm pretty sure a student trying to work through an introductory work on Bayesian data analysis (Kruschke, for example) without a year of freshman calculus under his/her belt is going to run in to some significant difficulty.
You're going to have a hard time convincing me that trigonometry and vectors are a necessary precursor for regression analysis or Bayes' theorem.
Your statement, in a post about scientific thinking, was that statistics and data science are much more useful than calculus. This is not true; as stated previously, calculus is of critical importance to many scientific fields. Moreover, vectors and trig (which, technically, one can study independently of calculus) are also of great importance in the natural sciences (at least physics) and certainly in engineering. I am surprised that you find this point to be controversial.
The minority of students in physics and engineering that need both calculus and statistics should not dictate how other majors are taught.
I'm not sure what you are saying here. Are you claiming that the minority of physics and engineering students need both calculus and statistics? All physics students and all engineers (at least in the traditional engineering fields) need calculus, so I don't know why you would claim that only the minority need both calculus and statistics. I don't think that you mean to claim that only the minority of these students need statistics, but that would follow logically from your claim.
Or, you might be saying that only the minority of students are in either physics or engineering. That may be true. But, since the OP is about scientific thought and scientific education, I'd say that the physics students (and engineers) are a minority that we ought to consider. And, natural sciences beyond physics also require calculus. Plus, as already stated, getting beyond the introductory level of Bayesian stats itself requires calculus.
Fixing the curriculum isn't an easy problem, but they've had more than a century to solve it and there seems to be little movement in this direction.
I honestly think that you are trying to fix something that is not broken. What's wrong with having kids learn calculus and statistics? Albeit I can see a role for a introductory "statistics-light" class for non-STEM majors that does not require calculus as a prerequisite. But, I think many colleges have this already (e.g. Tulane's Probability and Stats I does not have a calculus prerequisite).
Regardless of the above, I agree with a lot of your OP. In particular,
What's most damning is that our scientific curriculum in schools don't teach a lot of scientific thinking
seems to be true, at least in most of K-12. I was actually fortunate enough to have an outstanding physics teacher in 12th grade, who was able to convey some sense of the scientific method to the students. However, prior to that, much of the K-12 science curriculum seemed to consist of learning facts (or stamp collecting, as Ernest Rutherford would have said.)
Moreover, vectors and trig (which, technically, one can study independently of calculus)
Well, at least we agree there is leeway for a redesign; that's one problem solved.
What's wrong with having kids learn calculus and statistics?
TINSTAAFL
I'm not sure what you are saying here.
That physics and engineering majors represent only a minority of the student body.
calculus is of critical importance to many scientific fields. Moreover, vectors and trig (which, technically, one can study independently of calculus) are also of great importance in the natural sciences (at least physics) and certainly in engineering.
Even taking all natural science and engineering majors into account (which is a stretch since many natural science majors are going to end up in medicine or an unrelated career, and electrical engineering is a bit different from mechanical engineering) you've still got only 16% of the student body.
I'd say that the physics students (and engineers) are a minority that we ought to consider.
If all students need A and some students need B, then go to A first and the students who need B can still go to B afterwards.
Not really; Bayesian statistics really does build on calculus.
Most colleges are still focused around frequentist statistics in undergrad to the best of my knowledge. That's a separate debate entirely.
I'm pretty sure a student trying to work through an introductory work on Bayesian data analysis (Kruschke, for example) without a year of freshman calculus under his/her belt is going to run in to some significant difficulty.
Well, typically colleges are expecting you to take 3 semesters of calculus (although this 3rd course varies somewhat) as a prerequisite for just about everything, so if you can agree it should only be two, that's another problem solved. But I would go much further.
Yes, sets, series, and sequences are used in advanced stats, but there's no reason to teach those and trig at the same time. It's just a century old convention that no one ever corrected. If the 3 calculus courses were condensed down to what is actually used in say, 2 or 3 courses of stats, I'd bet you wouldn't even be left with a semester worth of material.
Albeit I can see a role for a introductory "statistics-light" class for non-STEM majors that does not require calculus as a prerequisite.
I don't see this single stats course as sufficient. But if a student wants to go further than this basic course, they generally have to take 3 calculus courses first. And then the other programs expect you to take the basic stats class and then calc I and II because they know it's not practical to expect every student to have the equivalent of a math minor just so they can take more than 1 course in data analysis.
Well, at least we agree there is leeway for a redesign; that's one problem solved.
There is no redesign needed. I first studied vectors and trig in high school before I'd ever had calculus. Its been a while, but I believe I studied vectors in 10th grade, trig in 11th and calculus in 12th. Admittedly, colleges seem to treat calculus as a prerequisite for linear algebra (at least mine did) for no apparent (to me) reason.
electrical engineering is a bit different from mechanical engineering
This is true. However, both require calculus, vectors and trig (from a fundamental level, not just a curriculum design level).
Well, typically colleges are expecting you to take 3 semesters of calculus (although this 3rd course varies somewhat) as a prerequisite for just about everything, so if you can agree it should only be two, that's another problem solved.
If we're just debating whether 2 or 3 semesters of calc is needed for statistics, then I agree; there is no argument. From what I can remember of my calc courses (its been a few years), I suspect 2 semesters of calc should be fine for most introductory to intermediate stats courses.
I don't see this single stats course as sufficient. But if a student wants to go further than this basic course, they generally have to take 3 calculus courses first.
If you are advocating requiring a lot of stats courses for non STEM students; I'm not sure I agree with that. As far as I can see, most non-STEM students are not going to want to take, nor will they benefit from, more than an introductory stats-lite course. Of course there are exceptions (e.g. some business/marketing, economics and sociology students for example might want more advanced stats courses). But, any kid with the aptitude and desire for intermediate stats courses is not going to have too much trouble with calc 1 & 2, and will need these to really get a handle on the stats (frequentists stats, like Bayesian stats, deals with concepts (probability density functions and the like) that are based on calc). And, of course more advanced stats classes may require additional calc past 1 & 2.
Of course there are exceptions (e.g. some business/marketing, economics and sociology students for example might want more advanced stats courses).
Business, social, and behavioral sciences represent over a third of students. They're more than double the size of STEM. It's a pretty big exception.
You're going to have a hard time convincing me that trigonometry and vectors are a necessary precursor for regression analysis or Bayes' theorem.
If you just type a command into R then you can do regression analysis in R but the most important lesson about regression analysis might be: Don't believe in the results. They often don't replicate.
If you do principle component analysis then you do need to understand what vectors are and what it means when they are orthogonal to each other. I'm not sure that you can understand well what degrees of freedom in a data set are without that background.
Actually, this illustrates scientific thinking; the doctor forms a hypothesis based on observation and then experimentally tests that hypothesis.
Most interactions in the world are of the form "I have an idea of what will happen, so I do X, and later I get some evidence about how correct I was". So, taking that as a binary categorization of scientific thinking is not so interesting, though I endorse promoting reflection on the fact that this is what is happening.
I think the author intends to point out some of the degrees of scientiificism by which things vary: how formal is the hypothesis, how formal is the evidence gathering, are analytical techniques being applied, etc. Normal interactions with doctors are low on scientificism in this sense, though they are heavily utilizing the output of previous scientificism to generate a judgement.
Most interactions in the world are of the form "I have an idea of what will happen, so I do X, and later I get some evidence about how correct I was".
Perhaps, but a the doctor in the OP did not just happen to later get some evidence about how correct he/she was; instead, after formulating a hypothesis, the doctor ran a test specifically to test the hypothesis. That is practically a textbook example (albeit a fairly short/simple one) of the scientific method at work.
though I endorse promoting reflection on the fact that this is what is happening
And that was really my point. It is worth noting that the scientific method is really just a very rigorous formalization of common sense reasoning. I think that demystifying science among the non scientifically sophisticated population is actually a step in the direction in which the OP gestures.
Normal interactions with doctors are low on scientificism in this sense, though they are heavily utilizing the output of previous scientificism to generate a judgement.
This also is true; even if one can't expect the full-on House M.D. treatment each time one goes in with a sinus infection or strep throat, many of the protocols that the doctor follows and the medicines that he/she prescribes were developed/tested with a high degree of scientific rigor.
Being to vague to be wrong is bad. Especially when you want to speak in favor of science.
I don't see any mention of formalism in the OP. There no reason to say "well maybe the author meant to say X" when he didn't say X.
Being to vague to be wrong is bad. Especially when you want to speak in favor of science.
I agree, it's good to pump against entropy with things that could be "Go Science!" cheers. I think the author's topic is not too vague to discuss, but his argument isn't strong or specific enough that you should leap to action based solely on it. I think it's a fine thing to post to Discussion though; maybe this indicate we have ideal different standards for Discussion posts?
There no reason to say "well maybe the author meant to say X" when he didn't say X.
Sure there is! Principle of charity, interpreting what they said in different language to motivate further discussion, rephrasing for your own understanding (and opening yourself to being corrected). Sometimes someone waves their hands in a direction, and you say "Aha, you mean..."
Above the author says "I think query worded it better", which is the sort of thing I was aiming to accomplish.
Above the author says "I think query worded it better", which is the sort of thing I was aiming to accomplish.
That result didn't include a discussion about the value of including formalism in the definition of science. The question about whether "formalism" is a central part of science is one that's to be had on LW.
Instead of saying: "I think you meant to include formalism." it's better to say: "I think formalism should be part of the our definition of science because of X, Y and Z."
That would make the discussion less vague and more concrete. Maybe someone agrees with your reasons. Maybe people disagree. In both cases there's productive discussion.
I think it's a fine thing to post to Discussion though
Criticizing an argument is not the same as objecting to it being posted. If vague ideas are posted in discussion then, it makes sense to have a discussion with the goal of getting the ideas less vague.
In addition to the question of "formalism" the OP's definition of science also lacks public challenge of ideas. Is that a conscious decision? I don't know.
As it stands the post is not address any of the concerns in the topic that exist in LW culture. To teach Bayesian statistics well you need calculus. Most statistics 101 classes teach frequentists statistics with p-values. Commonly they are taught in a memorize the teacher password way, that doesn't leave students with real understanding. Doctors have their statistics 101 classes but they mostly just memorize it and then forget it afterwards.
Does statistics 101 teaches students to expose themselves to empiric feedback? I don't think it does.
Simply saying: We need to teach more students statistics 101 ignores all that previous discussion.
He claims that LW is mainly about logical consistency and biases. I don't think that's the case.
Rationality!CFAR2015 seems to be: Your system I and system II are aligned in a way that if it's rational to get up at 7 o'clock your brain wakes you up at 7 o'clock without you needing an alarm clock. Then you work on the most important thing in your life.
We are not planning of publishing papers because academia with it's ethical review boards is too bureaucratic.
The post ends up with the sentence "It's disappointing that most people don't seem to understand even the basics of science." There are two ways to think about this. The "Go Science" way is to think that it's somehow obvious what those basics are. The author and the readers who what they are and as it's about statistics a statistics 101 course should solve the issue.
The other is to say, that actually it's not obvious what the basics are. It doesn't seem obvious for the author that systematization or formalization is an essential part of science that has to be in the definition. It's not obvious to the author that public criticism of ideas is an essential part of science and belongs into the core definition.
Including those criteria is not a matter of "wording it better" but a matter of substance.
If you see science as sacred and want team science to spread the gospel, then admitting questioning whether you actually are clear about the basics is emotionally very hard. It's hard enough that the question doesn't get asked much.
I don't bring this up to make FrameBenignly feel bad or to say that the post has no place (I upvoted it) but at the same time I find it important to actually engage with the issue.
Even professors, who do excellent research otherwise, often suddenly stop thinking analytically as soon as they step outside their domain of expertise. And some professors never learn the proper method.
As as as I know there no evidence for the thesis that thinking analytically always provides an improvement. A lot of expert thinking isn't analytic.
I've been often wondering why scientific thinking seems to be so rare. What I mean by this is dividing problems into theory and empiricism, specifying your theory exactly then looking for evidence to either confirm or deny the theory, or finding evidence to later form an exact theory.
Do you think you engaged into that type of thinking while writing this post?
As as as I know there no evidence for the thesis that thinking analytically always provides an improvement.
Not only did I not claim thinking analytically in the manner I'm describing always provides an improvement, I noted multiple exceptions when it doesn't improve it.
A lot of expert thinking isn't analytic.
If you want specific examples of what I'm referring to in regards to scientists screwing up, read Andrew Gelman's or James Coyne's blogs.
Do you think you engaged into that type of thinking while writing this post?
This is just vague.
I noted multiple exceptions when it doesn't improve it.
It sounded to me like you were criticizing doctors for not being scientific. Is your thesis that it's alright that doctors aren't?
If you want specific examples of what I'm referring to in regards to scientists screwing up, read Andrew Gelman's or James Coyne's blogs.
So on the one hand you advocate scientific thinking and then you come and provide anecdotal evidence? Decision science is a field.
This is just vague.
Why is the question of whether a particular form of reasoning follows the standards of "Thinking like a Scientist" vague? If you would have clear standards of what "Thinking like a Scientist" means, I think you should be able to answer the question.
If you don't have clear standards, then you are right it's vague. That means there a problem. Being to vague to be wrong is bad. The paradigm of science would say that removing vagueness from your theory would improve it.
"What do you actually mean when you say 'Thinking like a Scientist'" is to me the better response than 'Yeah science".
I do not claim that in-depth scientific analysis is always necessary. I claim that heuristics are often the rational approach, and gave examples in which I stated they were rational. I do not claim that all or even most scientists screw up their analysis. I do not claim how frequently it occurs. I do claim that I see it often enough that it worries me. I do claim that among non-scientists, the problem is much more common. I do not claim that personal experiences are useless. I claim that statistical analysis is superior and that people will often not note this in their personal assessment. I claim that I see these problems often enough that I can determine heuristically that it is likely to be extremely common.
I do claim that among non-scientists, the problem is much more common.
Non-scientists don't often engage in writing scientific papers. In what instances do you believe they screw things up and should engage in in-depth scientific analysis?
Reasoning from a statistical outlier is probably the most common error I see. Most news stories do this.
Mistakes on LessWrong tend to be more varied. Likely due to LessWrong's focus on Cognitive Psychology, I see a lot of people repeating the mistakes of Psychoanalysis; coming up with elaborate theories about mental processes with no reasonable means of verification of their ideas.
Most news stories do this.
They do this because it's good storytelling and they want to sell papers. A have fairly low confidence that teaching the authors statistics helps in any way. If you think it helps can you explain why you think so?
coming up with elaborate theories about mental processes with no reasonable means of verification of their ideas.
Why do you think those posts need "reasonable means of verification of their ideas" while you haven't provided one for the post you written and think it's okay based on heuristics? Aren't those people not also simply using heuristics instead of structured scientific thinking?
Not the authors; the readers; though I don't think the authors are generally aware of the problem either.
Verifiability is not heuristics. I'm combining in the term verification the two scientific concepts of direct observation and falsification. By elaborate theories, I'm referring to occam's razor.
My post isn't a theory post. It contains a few ideas and a lot of observations, but the assumptions are pretty straightforward and they're related to the central concept of scientific analysis, but not dependent on one another. The general statement about schools is not dependent on the specific statement about math, nor is the argument about math dependent on the argument about whether rationality is sufficient. And I try to be exact with my phrasing to specify my uncertainty where it exists.
What's most damning is that our scientific curriculum in schools don't teach a lot of scientific thinking.
There is an IQ prerequisite for that which most students are unable to fulfill.
The lower-IQ version of scientific thinking are the good-oldie, common-sense proverbs and maxims like "measure twice, cut once". They have a problem of being kinda boring and "uncool".
Kipling's poem The Gods of Copybook Headings is a good example what a low-IQ version of scientific thinking could be. I think it was abandoned because it was "uncool" and "boring".