If anyone reading this wants me to build them a custom D&D.Sci scenario (or something similar) to use as a test task, they should DM me. If the relevant org is doing something I judge to be Interesting and Important and Not Actively Harmful - and if they're okay with me releasing it to the public internet after they're done using it to eval candidates - I'll make it for free.
I mostly agree with this, but I definitely think that there's an important caveat: if the thing you are looking for involves multiple dimensions of a person to be at least adequate on a certain threshold and the sum of the dimensions to be as high as possible... that's a different restriction than just high-sum-with-no-thresholds. The more restrictive version means that you may in fact have a much smaller potential hiring pool than you imagine.
If, for example, you need the person to be highly trustworthy / value aligned, and reliable (generally doesn't flake on their commitments, accomplishes what they say they will), and technically competent in multiple ways. If being really good at two of those, but failing on the third means that the person overall wouldn't be a good hire... you might actually get a lot of value out of spending more on the hiring search process in order to get fewer false negatives.
What might this look like? I think you actually want to go to the effort of making a few different tests (at least 2, more if you can afford it), and allowing people to take a different test after failing (if they are sufficiently motivated). If the tests are sufficiently different from each other, this gives you a chance to catch a few false negatives who might otherwise have gotten ruled out due to an unlucky mismatch with the first test.
Here is another consideration: have something to give them in return. Ask why would very talented people want to work for you. I had a very bad experience working for people who were not as experienced as me, nor as resourceful or with good leadership skills. Other than the salary, I didn't feel like I was getting much from them. I eventually quit and in hindsight, I shouldn't have accepted the job.
Yeah I guess that's another prereq. I think you can make up for it some by having good work for people to do. I would rather work on a cool or valuable thing with amateurs than something lame with pros.
TLDR: use a very difficult timed 8-hour task to screen candidates. You should fail your own test.
I'm no expert but I've made some recommendations that turned out pretty well -- maybe like 5 ever. This post would probably be better if I waited 10 years to write it. Nonetheless, I think my method is far better than what most orgs/corps do. If you have had mad hiring success (judging by what your org accomplished) then please comment!
Half-remembered versions of Paul Graham's taste thing and Yudkowsky's Vinge's Law have lead some folks to think that judging talent above your own is extremely difficult. I do not think so.
Prereqs:
(Probably you don't meet the prereqs. Probably your org strongly depends on connection and reputation more than talent; probably your raises depend on you not out-hiring yourself; etc. Don't feel bad — it is totally ok to be a normal org/human! Being a goal psycho often sucks in every way except the accomplished goals.)
If you do meet the prereqs, then good news, hiring is almost easy. You just need to find people who are good at doing exactly what you need done. Here's the method:
Measure it yourself. Make up a test task. You need something that people can take without quitting their jobs or much feedback from you; you and the candidate should not become friends during the test; a timed 8-hour task is a reasonable starting point. Most importantly, you must be able to quickly and easily distinguish good results from very good results. The harder the task, the easier it is to judge the success of top attempts.
If you yourself cannot complete the task at all, then congratulations, you now have a method to judge talent far above your own. Take that, folk Vinge's law.
Important! Make the task something where success really does tell you they'll do the job well. Not a proxy IQ test or leetcode. The correlation is simply not high enough. Many people think they just need to hire someone generally smart and capable. I disagree, unless your org is very large or nebulous.
This task must also not be incredibly lame or humiliating, or you will only end up hiring people lacking a spine. (Common problem.) Don't filter out the spines.
It can be hard to think of a good test task but it is well worth all the signal you will get.
Say you are hiring someone to arrange all your offices. Have applicants come arrange a couple offices and see if people like it. Pretty simple.[1]
Say you are hiring someone to build a house. Have contractors build a shed in one day. Ten sheds only cost like 5% of what a house costs, but bad builders will double your costs and timeline.[2]
Pay people as much as you can for their time and the stress. Also helps you avoid the guilt that may lead you to get sloppy on hiring protocol.
You're going to set up some arbitrary filters. Very talented people will often fail them because they weren't prepared to do the exact random thing you asked them to do. Accept this. You only need 1 (or n) people to succeed. You are not running a charity. Or if you are running a charity, then hiring people isn't part of the charity. Or if it is then you're reading the wrong post.
You make the task very difficult to make sure that only "true positives" (ie definitely super talented people) get further into the pipeline where everyone will meet them and probably get attached. Firing people sucks super bad! So you eat all the false negatives. You will probably have a lot more false negatives than true positives. You gotta eat it.
You can communicate this to candidates early & often. Eg in the job ad: "We hire based on the results of some short but very difficult tests which most people, including most qualified candidates, do not pass. Test is paid well!"
Why give candidates a hard test before looking at other stuff?
You might expect resumes and references and interviews and so on to give you strictly more information. Really, they should just give you a better picture. But 90% of your brain matter is dedicated to finding allies etc and only 5 parts per million is tasked with carrying out abstract objectives on the outside world. Folks will switch political parties for love and respect! So be damned sure your brain will throw the quarterly targets in the waste bin in exchange for some direct personal social value. You have to blind this part of your brain from what's going on until candidates are screened.
To really spell this out, let me list all the orgs which say they only hire "top talent": all of them. Let me list all of the orgs where all the people are very good at their jobs: very few. Let me list all the orgs which are plagued by office politics: all of them. Let me list all the candidates which understand that office politics exists and are (possibly subconsciously) trying to game it: 88.88%. For example, almost no candidates are mean to the boss, but many candidates are mean to others.
After people have passed the first screen, you are free to pick and choose people based on professionalism or experience or whatever. You only have to be a little patient.
There is one little issue. You probably don't have enough time and money to administer the main test task to all your applicants. Someone must do a first-pass filter. It is a very good idea to make this a different person from the main test administrator/judge.
My opinion on how to do the first pass: aim for heterogeneity. For example, many high school graduates are better at coding or electronics or math or ad writing than the average college graduate in those areas. I don't know exactly where the bell curves lie, but they certainly overlap. There's also lots of people in their 40s that are really really good at jobs that usually are reserved for the young. Coding is an example. Recent college graduates and young professionals might be the most likely place to find a good hire, but then you'll be searching the same spot in the river as all the other orgs.
This is where it gets murky. There's actually two requirements for any job: good at it, and works hard. Unfortunately, this latter thing is much harder to judge at an arm's length. It is also liable to change with time and circumstance.
I have no good answer here. Almost any way you can accurately judge hardworkingness will lead to attachments all around. You have to use your gut here I guess. And maybe portfolios, but plagiarism is commonplace.
The psychopath (eg Amazon's) approach is to continually watch everyone's productivity and try to get lazy people to quit, or else fire them. I do not like this much.
Before you go crazy trying to hire super talented people, make sure that's what you actually need & want (it probably isn't). But such hiring is kind of easy if you stay focused on it and don't let your heart get in the way before the difficult test is done & graded. You will likely find some unusually talented folks this way. After the test is passed, pick among the candidates however you want.
Have you ever noticed how most offices kind of suck? The window shades are ugly and the desks are in bad places and there's no plants. Arranging an office is quite cheap! Why is it so bad? I think folks usually just do not look at the work their interior designer has done before. They certainly don't trial several candidates. ↩︎
My stepmother added a chunk to her house a few years ago. She got the "best guy in town" — her friends all recommended this guy for some reason — and she's been in a real-estate-adjacent work for 20 years — AND she probably personally knew more than a handful of folks who could do it. The best guy in town accidentally made the basement ceiling 2 feet too low. So she had to find some random other contractor to tear everything up and redo it. I never asked her what the mistake cost exactly, but you can imagine. ↩︎