I agree that trying to treat the symptoms without a thorough analysis, modeling and testing would likely be a wasted effort. To start, I'd ask questions like "When did the problem first appear? How did research work before p-hacking, publish-or-perish and fight for tenure become a staple of academic life? What worked then, why did it need to change, what changes were introduced, what effects they had.." This is not a simple problem to solve, so obvious simple solutions are unlikely to just work.
Too many people want a job that is interesting or meaningful, and nobody is willing to support all of them financially
Meaningful is an interesting word here. Most PHD students don't do work that has very much meaning or potential to have impact on the world.
If people want to work on jobs with a lot of meaning, why did Hamming need to as them why they didn't work on the most meaningful problems?
For the most part it seems to me that people are scared to work on problems that are actually meaningful.
We don't really have a metric for meaning or impact though.
And even if we had decent metrics they would gain value with time since the impact of a discovery becomes evident only after a while (think patents, landmark papers, new disciplines).
For the most part it seems to me that people are scared to work on problems that are actually meaningful.
It appears to me that the incentives system is the real issue here. UBI or some basic job might release a lot of people from their publishing cages, allowing them to work on research fundamentals: gathering good data, working on theory and methodology, replicating studies.
In some disciplines, like working in an Amazon warehouse, intrinsic motivation is low and both cheating and legitimate but zero sum efforts are unlikely. There, ever greater competition for economic and social rewards generate ever better results. In other domains, like the creation of knowledge and especially the evaluation of knowledge, intrinsic motivation is relatively strong, cheating is easy or likely to succeed or zero sum situations are the default. In those domains, competition leads to the displacement of necessary activities by highly optimized forms of fraud. At the extreme, nobody argues that we need profit maximization by judges or by generals. Unfortunately, using pharmceutical companies to produce medical innovation isn't very much better and ever more commercialized and competitive academia isn't much better than that.
Nobody is confused about this. We don't have incentives this terrible by accident. Rather, there are constituencies in favor of accurate shared information and constituencies against. The latter, unfortunately have been working unopposed for a long time
France does this, at least in mathematics. Many people go straight from PhD to a permanent CNRS research position with no responsibilities. Short term postdoctoral funding was only introduced in 2005, but before that a lot of people did foreign postdocs. Alain Connes talks about this here.
But mathematicians are cheap. What can you do in other fields? What does CNRS do?
UBI is a fascinating and likely hugely beneficial idea to improve individual welfare by decoupling choice (and ability) of work from being able to eat. I do worry that it reduces the incentive to choose professions which are in demand.
The fact that we have far more people who want to do academic research than we have an appetite to pay them to do so is a fact that UBI won't change, it will instead only obscure it, making it easier for people to do this as a semi-hobby.
I do worry that there's an efficiency loss in letting people be less productive as an academic subsidized by UBI than they would be if they went into the private sector and used their skills and knowledge to more directly meet paying customer needs.
I'll also point out that it's _REALLY_ hard to distinguish luck from talent in cases like this (tenure, film stars, drug kingpins, other winner-take-all labor markets with a whole lot of entrants and very few big winners). My model of the publication->tenure path isn't that it's selecting the lucky ones whose papers worked out, but for the talented ones who sniff out more likely topics well. Sure, there's luck involved, but there's also a very-hard-to-test-separately talent. Treating it as a pure lottery is likely to lead to very wrong policy decisions.
UBI is a fascinating and likely hugely beneficial idea to improve individual welfare by decoupling choice (and ability) of work from being able to eat. I do worry that it reduces the incentive to choose professions which are in demand.
The idea is that "UBI + job" would still pay better than "UBI only". It reduces the incentive for people who hate the job, but would hate more to see their families starve. The people who like the job and/or like the extra money would stay. And there is also an option to change the working conditions so that less people will hate the job.
Without the metaphorical gun at their heads, fewer people would be motivated to choose the professions in demand, but it would only be a partial reduction. The working conditions would probably improve, and the prices increase. The exact numbers would depend on the specific profession.
The question is, whether the world where jobs suck less but we can afford less things would be a net improvement. I guess that also depends on specific numbers.
It certainly sucks when you need a surgery, and there are not enough surgeons, because everyone prefers to meditate instead. (On the other hand, perhaps with improved working conditions, less surgeries would be needed.)
It also sucks when you don't need a surgery and go to a doctor with financial pressures to operate you and remove flexibility from your spine or remove an organ and you believe that doctor and get the surgery.
Oversupply of surgeries is as much of a problem as undersupply when you have doctors who aren't motivated by helping patients but primarily motivated by money.
Sure. This is likely to happen for any identity-based motivation, not just financial. If someone gets good feelings of contribution and effectiveness by performing surgeries, they're very likely to recommend it in borderline or not-fully justified cases.
If you don't need surgery, don't go to a surgeon. In theory, the price system gives victims of over-surgery a bit of additional motivation to stay away, but the repugnance of calculations about health is such that it doesn't really work that way in most countries.
Good feelings alone don't produce the institutional pressures that you have in current Western hospitals that are for-profit businesses.
You currently have a state of affairs where many doctors do have goals about the amount of surgeries that they do from their employer, that they are expected to reach.
I have a friend who worked as an anesthetist. According to his opinion he was part of operations that weren't medically justified and when he complained with his colleges about it they didn't justify their decision to operate on medical grounds.
In theory, the price system gives victims of over-surgery a bit of additional motivation to stay away, but the repugnance of calculations about health is such that it doesn't really work that way in most countries.
That's not how health care economics works in theory. People pay for health care to show that they care and paying more signals that they care more, see Hanson. There's also a lot of research about sacred values in decision theory and most people treat health as a sacred value.
I suspect there'll be enough surgeons - doctors get paid in prestige as well as money. We're going to run out of orderlies and nurses, though, so the surgeons are far less effective.
Agreed that it depends on specific numbers, and how much of the population satisfices near the UBI level vs how many are still motivated to take unpleasant work.
As I tend to think of it, a UBI is only supposed to cover the basic necessities you need to live and maybe a little on top of that, but little else. If this means that a UBI disincentivizes people to work some jobs, then that means that they were previously only doing those jobs because they were literally the kinds of jobs that you would only do if your only other option was to starve to death on the street.
I tend to think that if a UBI eliminates the need to do those kinds of jobs, then good riddance. Yes, maybe those kinds of jobs are necessary for society to function. In that case, if they really are critical ones, installing a UBI should force us to find a way to improve their working conditions (by e.g. offering a better pay) rather than allowing us to continue cruising on with the system where some people do them because they've got no better choice.
This is part of 'specific numbers matter'. In first-world countries, almost nobody _literally starves on the street_ already, so that UBI number would be 0. Most casual discussion I hear assumes UBI will be somewhere near a state minimum-wage job, not enough to live in a nice part of town, but enough to crowd out at least some current jobs.
In the abstract, I like your "good riddance to bad jobs" attitude. Unfortunatlely, I don't know which concrete low-paying jobs I'd rather leave undone than to have an unhappy worker doing them.
Unfortunatlely, I don't know which concrete low-paying jobs I'd rather leave undone than to have an unhappy worker doing them.
Me neither. It's possible that this is a bad idea, and that we'd end up with a worse society overall; but in general I'm skeptical of this, since you can make those jobs more attractive by increasing their pay; and if they really are valuable then they should still produce net value even if they were more expensive to fund. This logic doesn't work in every possible case and it's not that hard to think of counterexamples, but in general there's a big gap between productivity and wages, and a UBI could be something that would help fix that.
But of course nobody knows how it actually works out until we try.
I think the last parahraph of your comment agrees with the post:
"... if you try to measure skill and then pick people based on how well they performed on your measure, you’re actually selecting for skill + luck rather than pure skill."
Right, but that phrasing is misleading. Selecting by pure skill isn't available, and a lot of luck might be a hidden skill. Selecting by outcome is NOT luck-free, but it's better than any other selection mechanism.
The problem isn't the selecting by outcome part, the problem is the part where one tries to discriminate more than the measure allows.
Suppose that you are making people take an exam on which they can score between 0 and 100. If you took their exam scores literally as indicative of their skill, you might be tempted to use the scores to choose the people in the top 1% (say). But the exam probably isn't capable of measuring skill that precisely; the error of the measure may be such that realistically, it's only reliably picking up on skill differences on the level of 10-point increments. If you try to use it to pick the top 1% anyway, you are just measuring noise.
So if you want to make things fair in the sense of rewarding everyone who actually has enough skill, you should set the cutoff so that everyone who scores in the top decentile gets rewarded. (This does mean that some people who actually aren't that skilled will also be rewarded by luck, but that's an unavoidable trade-off.)
If people didn't respond to incentives, then you wouldn't need to care about making things fair in this sense. After all, anyone who's in the top 1% is in the top 10% too, and picking the people in the top 1% does mean that you're very unlikely to get someone who's actually in the top 89% but made it over the 90% cutoff by luck. But people do respond to incentives; in particular, if they realize that they can't make it to your cutoff just by being talented and hard-working but have to be exceptionally lucky as well, then they will feel the pressure to cheat and deliver results that look good but are actually false.
In fact, if this is common enough, then picking the people that the measure says are in the top 1% may actually get you people who are worse, at least in the sense of being more willing to fake their results, if it's more likely that you'll get into the top 1% by cheating than by being lucky.
I find it useful to think of this literally as an exam situation: you know that you need to score in the top percentile to be rewarded. You also know that skill only gets anyone over the 90 point mark and the rest is luck. Furthermore, you know that a number of people also taking the exam - some equally skilled as you, some less - are going to cheat on it, and that the vast majority of people who do make it to the top percentile only succeed at that because they cheated. If you don't cheat, it doesn't matter how skilled or hard-working you are: you almost certainly won't make it.
Would you cheat?
If no, you'll get eliminated from the system and have to do something else. If yes, you're contributing to the situation and increasing the incentive for others to cheat, and helping establish implicit norms where it's taken for granted that anyone who made it past this round of elimination is a cheater, so cheating can't be that bad.
Whereas if the cutoff was at the 90th percentile level, some people would still get in because they were cheats, but the system would no longer be selecting strongly against people with integrity, and the environment would be much more conducive for establishing norms where cheating was looked unfavorably on.
I think there's a crux here we haven't made explicit. I don't think the universe is particularly fair, and I don't intend to give up much efficiency in order to correct that unfairness. I don't want to reward skill, I want to eat fresher strawberries in the winter. I often signal to people that the best way to get my resources is to have and develop skill, but this doesn't contradict the fact that there's a lot of luck which neither I nor they can control.
I don't think most economic choices are very similar to school tests. They are simply not of the form "try to find the hidden intrinsic values of someone". No single human decides the grading mechanism, and it's not arbitrary proof of work to some authority. Economic choices are more of an uncertain time/energy/resource allocation problem - we MUST make choices of how to (try to) provide value to someone with resources, and how to expend those resources to best get the things we want. For this, even noisy measures are better than random. It's the best (and most resistant to Goodheart's Law) predictor of future delivery of value we have - even if it's mostly noise, there's some signal there.
My fear with UBI (note that on balance I'm still in favor) is that it doesn't change any fundamentals here - all these choices are still noisy and luck-entangled - it just removes a little bit of signal, making it a little bit less likely to optimize the outcomes.
For this, even noisy measures are better than random. It's the best (and most resistant to Goodheart's Law) predictor of future delivery of value we have - even if it's mostly noise, there's some signal there.
I agree, and don't think that I've said anything that would imply otherwise.
it just removes a little bit of signal, making it a little bit less likely to optimize the outcomes.
Hmm, how does it remove a little bit of signal? To me, the signal of an academic's worth is their publication/research history; the existence of a UBI is not going to remove that.
I still resist the idea that this is a signal of worth, and certainly that it's a signal of worth in a field. It's a signal of demand for a given output and of what an individual should do in order to give value to other humans.
The signal is _not_ how good of an academic one is. The signal is whether more academics are needed, vs other professions entirely. This signal is somewhat muted if the recipient is less dependent on market income.
Ah, we have been talking about different signals then.
I think we should be careful about lost purposes here: the function of an economic system should, IMO, be to ensure that everyone has the possibility to live a life that they experience as flourishing, while distracting them from that goal as little possible. If the productivity of an economic system is such that there's a sizable group of people who are happy and capable of having a flourishing life with a minimum wage-equivalent UBI, then that's a sign that the economic system is starting to be done. Not that it would be done - obviously it wouldn't work yet if everyone just lived off the UBI and only did things that they personally found meaningful. But those people are starting to pave the way towards a broader societal change that allows a life only doing personally meaningful things, as their presence and example will encourage others to make the same transition and also create cultural institutions that support such a lifestyle, paving the way for a broader transition to such a society. Including structures that support people doing socially-needed things due to intrinsic motivation and because voluntarily doing useful things becomes valued, rather than socially-needed things only being done because you get money for them.
My model here would take too long to explain in detail, but briefly, I suspect that a lot of the thing about jobs being something that you only do grudgingly is because people have been forced from an early age to do things they don't want to and come to hate it, whereas much more work would be done out of a volunteer basis and because people want to be useful if people had grown up in a society where the opportunity to do the-kinds-of-things-we-now-think-of-as-work was considered a joy and a privilege. See e.g. the research on SDT and how relatedness and competence are basic needs behind intrinsic motivation, but how extrinsic motivation tends to suppress intrinsic motivation.
an economic system should, IMO, be to ensure that everyone has the possibility to live a life that they experience as flourishing, while distracting them from that goal as little possible
Oh, yeah, that's definitely a point of confusion between us. IMO, an economic system is an evolved mechanism for communicating individual desires and negotiating behavior and resource constraints without having to pay the cost of the underlying violence behind the idea of ownership and the enforcement of necessary cooperation.
I broadly agree that we're in an unpleasant equilibrium of what's "necessary" and "unpleasant". I don't have a picture of a better equilibrium that supports this density of humans with the same distribution of capability and the same or better amount of individual autonomy. I prefer that there are enough porta-potty cleaners who choose that as their self-actualized goal, but I don't have a lot of faith that it's possible with current levels of human development.
Is there enough research to go round all who would like to do it? A quote attributed to Dirac in various forms, speaking of the early days of quantum mechanics, says, "It was very easy in those days for any second-rate physicist to do first-rate work. There has not been such a glorious time since then. It is very difficult now for a first-rate physicist to do second-rate work."
It is easy to list the big problems one might work on. Which of them offer opportunities for all of the first-class people to do first-rate work on them — opportunities that look promising for actually solving them, rather than merely working on solving them?
To me the world looks like there are many problems on which can be worked productively. Most of them are not in the prime focus of the existing academic communities.
I'm sometimes even surprised that problems that seem easily approachable don't get studied. Scott wrote in his latest Melatonin post that there are good reasons to expect it to be useful to take Melatonin right after waking up for people who wake up to early.
You don't need a first-rate scientist to run a study to investigate that phenomena.
The current evidence base suggests that taking Zinc on the first day of developing common flue reduce the length of the flue. However nobody went around to run studies to determine the correct amount of zinc that should be taken for that purpose.
Given the anecdotal evidence we have that taking Vitamin D3 in the morning instead of the evening is better for sleep it would be very useful if someone would run a study to investigate the phenomena.
Eliezer's idea of treating SAD with a higher amount of daylight lamps isn't studied by any scientist.
Those four problems can easily be approached by second-rate scientists and would all have the potential of significant positive impact.
I don't know about physics, which is a rather mature field, but it's my understanding that in fields like psychology most established researchers have way more research ideas than time to pursue them all.
Very interesting thoughts Kaj, thank you for your post. The part about skill measurement was very instructive.
I was recently wondering about UBI (universal basic income) and why it seemed to be an increasly important idea. Considering technlogicals advencement in automation and AI, jobs are just moving away from human's skills.
Humanity is loosing the job interview to machines.
Of course, theses advencements create new jobs, but they're the first in line to be replaced by algorythms.
My point was that research jobs (not particularly PhD level jobs, even very simple tasks i.e. counting birds, pick up metheorologicals data, grow plants, whatever), may be a major source of jobs in the future, if fuel by UBI.
This is not a deeper root to the actual science issue of course. But it may be a good way to fight raising inegalities and induce massive progress in spreading rationality. Plus, collecting data in fields like biology are still a major obstacle to scientific breakthrough. Mostly based on volontarism, theses task could be soon funded thus allowing people to focus on science.
Isn't a "researcher's basic income" just another word for, um... tenure? I think the proper solution is to tighten standards for what's considered "good" research (fix the replication crisis) and to increase the status of other sorts of scholarship which aren't highly valued at present (at least in STEM) but are very much needed, such as review articles and in-depth monographs. These things don't have the problem where only an unambiguously "positive" result demonstrates the value of one's scholarship, and reaching positive results is largely a matter of luck.
The author does suggest a system where "academic researchers are rewarded for running high quality studies with these sorts of attributes, regardless of outcome", but, barring highly-selective preregistration, I'm not sure how this can work - other things being equal, an unambiguous outcome does signal a higher-quality study, so researchers will always prefer clear (i.e. "positive") outcomes.
Tenured professors have a lot of responsibilities like holding lectures and supervising PHD students that don't allow them to focus on research.
Isn't a "researcher's basic income" just another word for, um... tenure?
Tenure means doing it successfully the wrong way for years, and afterwards as a reward being allowed to do it right. The proposed researcher's basic income is to allow people doing it the right way much sooner.
I ran across Ed Hagen’s article “Academic success is either a crapshoot or a scam”, which pointed out that all the methodological discussion about science’s replication crisis is kinda missing the point: yes, all of the methodological stuff like p-hacking is something that would be valuable to fix, but the real problem is in the incentives created by the crazy publish-or-perish culture:
In my field of anthropology, the minimum acceptable number of pubs per year for a researcher with aspirations for tenure and promotion is about three. This means that, each year, I must discover three important new things about the world. […]
Let’s say I choose to run 3 studies that each has a 50% chance of getting a sexy result. If I run 3 great studies, mother nature will reward me with 3 sexy results only 12.5% of the time. I would have to run 9 studies to have about a 90% chance that at least 3 would be sexy enough to publish in a prestigious journal.
I do not have the time or money to run 9 new studies every year.
I could instead choose to investigate phenomena that are more likely to yield strong positive results. If I choose to investigate phenomena that are 75% likely to yield such results, for instance, I would only have to run about 5 studies (still too many) for mother nature to usually grace me with at least 3 positive results. But then I run the risk that these results will seem obvious, and not sexy enough to publish in prestigious journals.
To put things in deliberately provocative terms, empirical social scientists with lots of pubs in prestigious journals are either very lucky, or they are p-hacking.
I don’t really blame the p-hackers. By tying academic success to high-profile publications, which, in turn, require sexy results, we academic researchers have put our fates in the hands of a fickle mother nature. Academic success is therefore either a crapshoot or, since few of us are willing to subject the success or failure of our careers to the roll of the dice, a scam.
The article then suggests that the solution would be to have better standards for research, and also blames prestigious journal publishers for exploiting their monopoly on the field. I think that looking at the researcher incentives is indeed the correct thing to do here, but I’m not sure the article goes deep enough with it. Mainly, it doesn’t ask the obvious question of why researchers have such a crazy pressure to publish: it’s not the journals that set the requirements for promotion or getting to the tenure track, that’s the universities and research institutions. The journals are just exploiting a lucrative situation that someone else created.
Rather my understanding is that the real problem is that there are simply too many PhD graduates who want to do research, relative to the number of researcher positions available. It’s a basic fact of skill measurement that if you try to measure skill and then pick people based on how well they performed on your measure, you’re actually selecting for skill + luck rather than pure skill. If the number of people you pick is small enough relative to the number of applicants, anyone you pick has to be both highly skilled and highly lucky; simply being highly skilled isn’t enough to make it to the top. This is the situation we have with current science, and as Hagen points out, it leads to rampant cheating when people realize that they have to cheat in order to make the cut. As long as this is the situation, there will remain an incentive to cheat.
This looks hard to fix; two obvious solutions would be to reduce the number of graduate students or to massively increase the number of research jobs. The first is politically challenging, especially since it would require international coordination and lots of nations view the number of graduating PhDs as a status symbol. The second would be expensive and thus also politically challenging. One thing that some of my friends also suggested was some kind of a researchers’ basic income (or just a universal basic income in general); for fields in which doing research isn’t much more expensive than covering the researchers’ cost of living, a lot of folks would probably be happy to do research just on the basic income.
A specific suggestion that was thrown out was to give some number of post-docs a 10-year grant of 2000 euros/month; depending on the exact number of grants given out, this could fund quite a number of researchers while still being cheap in comparison to any given country’s general research and education expenses. The existence of better-paid and more prestigious formal research positions like university professorships would still exist as an incentive to actually do the research, and historically quite a lot of research has been done by people with no financial incentive for it anyway (Einstein doing his research on the side while working at the patent office maybe being the most famous example); the fact that most researchers are motivated by the pure desire to do science is already shown by the fact that anyone at all decides to go to academia today. A country being generous handing out these kinds of grants also has the potential to be made into an international status symbol, creating the incentive to actually do this. Alternatively, this could just be viewed as yet another reason to just push for a universal basic income for everyone.
EDIT: Jouni Sirén made the following interesting comment in response to this article: “I think the root issue goes deeper than that. There are too many PhD graduates who want to do research, because money and prestige are insufficient incentives for a large part of the middle class. Too many people want a job that is interesting or meaningful, and nobody is willing to support all of them financially.” That’s an even deeper reason than the one I was thinking of!