Tyler Cowen will often praise a book by saying it is interesting on every page. Together with Daniel Gross, he has written a book that very much matches this description.

The topic is talent: How to search for it, how to identify it, interview and otherwise evaluate it, how to recruit it, how to buy it low and sell it high. The book asks, in various ways, what matters? What makes one person succeed or provide lots of value, while another fails and provides little? How can we put together a portfolio of the first type of people at (in all senses) a good price?

There is a unifying philosophy and way of looking at people and the world underlying all of it. I mostly subscribe to the bulk of it, and especially to a core kind of economic thinking.

As part of this way of thinking, the goal is not to interview or evaluate people in an adversarial framework. You don’t want to ‘get’ them by tricking them and making them look foolish. What does that accomplish? You want to ‘get’ them by understanding who they are, and you learn by asking in a situation where they feel they can trust you, about things you both care about.

Don’t approach interviewing as a process where you are trying to trick or trip up the other person. First, that is intrinsically a bad way to behave. Second, once the person realizes what you’re doing, they’ll stop trusting you and, in most cases, become guarded. That will make it harder for you to determine whether the person is a good match for your job or assignment, not to mention harder to land them should you decide they are. (415)

…

Most importantly, it should be a question you really want to know the answer to. So if the person worked in a ball bearings factory in Cleveland, ask yourself if you care more about ball bearings or more about Cleveland. And your follow-up questions should also reflect real interests of yours. (420)

…

There is only one real and credible protection against those phoniness-spotting abilities: to be actually worthy of someone’s trust. When we are genuinely curious it draws people out, and as we show interest, they begin to trust us. This doesn’t mean you shouldn’t ask hard questions and challenge answers that don’t add up—it’s simply that you must be genuine. (427)

I strongly agree with this. You’re still going to want to test them and provide challenges so you can see how they react, but there’s essentially never any call for an actually adversarial situation, which only poisons the well and muddies the waters.

Another of Tyler’s principles, in general and that is applied frequently in Talent, is to attempt to crack as many cultural codes and modes of thinking as possible. I again strongly agree. I get great value out of the ones I have and wish I had the ability and time to acquire more. Yet it is also important that these codes are not created equal. The more I explore and model the world, the more desperately I want to spread understanding of the basic economic cultural code, to think in terms of value, supply and demand and compound interest, above any other.

That economic code is the core language of this book, and the book could not have been written without it. The core hypothesis is that you want to look for good deals and good investments with high expected returns. That means finding people who:

1. Will improve over time, and improve at improving.
2. Spend time often, ideally every day, working on improving.
3. Are extremely driven, high-energy and ambitious.
4. Have a chance of being extraordinary.
5. Others will have a hard time finding or evaluating.
6. Others won’t appreciate or find strange or off putting.
7. The market will underpay for other reasons.
8. Have something wrong with them that you can handle.
9. Prefer working for or with you for unusual reasons.
10. Are idea-seekers rather than status-seekers.
11. Avoid the bureaucratic approach that minimizes loss and values consensus.
12. You took the time to understand as complete people and who you holistically think will be good fits for the task at hand.
13. Has the right ‘five factor’ personality values, although I am highly unconvinced that this is a useful tool or even what we should be looking for directionally.

Among other things. That’s the biggest macro idea.

In this mode, the whole point of an interview is to truly get to know the person on a deep level. How they think. What they care about. Where they are strong and weak and how smart they are, sure, but it’s more about the quirky stuff that makes us unique.

That of course assumes you can make good use of such information. A lot of the interview questions Talent suggests are the kinds that don’t have right or wrong answers, but if that’s true how do you know which candidate is right or wrong? You need to be able to turn that understanding of the candidate into an analysis into whether or not they would make a good hire, including avoiding equating ‘good hire’ with ‘would then make money and get promoted/funded going forward’ since your goals are different with (although also correlated with) that.

Part of this is that when you discover something ‘bad’ in some sense, that can be good because this will make them underpriced in the market, and something ‘good’ can likewise be bad if it is overly legible and overpriced.

My worry is that most people, myself included, would if they did this well ask a bunch of questions, have an interesting conversation where they get to know someone, then be confused what to do with all of this information. Are these attributes overpriced in various senses? Would they lead to good results in this endeavor? What actually matters?

Tyler and Daniel are true ‘power users’ of talent. Who else would be writing such a book? They both devoted tons of time to evaluating, following, cultivating and promoting talent in various forms. Someone like me has done more of it than most people, but much much less than Tyler or Daniel. So I have enough experience and world modeling to talk extensively about the micro of these questions and explore them, but remain highly uncertain about being able to benefit from the approach.

Having thought and written about a bunch of the micro ideas, I notice I worry that the thing being thought about as ‘talent’ is often ‘this person will be Successful™' and thus becomes talent for navigating moral mazes and selling oneself, or having successful future funding rounds or simply getting higher pay.

If you say you want talent that is idea based instead of status based and that focuses on production, but you measure production by markers of success, what you’re actually optimizing for is the production of the markers of success. Certainly all these are correlated with more generally useful talent, but the more the world is rewarding the wrong things, the more you will reward those same wrong things too. The even bigger danger is that at some point you risk crossing from Goodhart’s Law problems to 2nd level Goodhart’s Law problems, where you are effectively optimizing a metric that was originally predicting who would be able to successfully game the original metric, and so on.

I think about this a lot.

You absolutely want extreme forms of ambition and self-improvement, but in a sufficiently broken and hyper-competitive world this starts to select against caring about anything else, especially if those around you are running forms of meta-Goodhart and punishing caring about anything else. This seems like a big worry for the approach being taken, especially by Daniel.

There’s a lot of emphasis on compounding returns to working on self-improvement, looking for an obsession with improvement and one’s work, and being suspicious of those who spend much time on other things via ‘free time’ and ‘social life.’

I certainly see the argument. I’ve had jobs that were all-consuming, where I was obsessed with putting everything into them that I could, and in many cases I was very good. Whereas the times that I wasn’t doing that, it was a bad sign on multiple levels – I wasn’t putting in the work, and I wasn’t sufficiently motivated to be putting it in.

Tyler Cowen thinks about talent through the lens of Emergent Ventures, while Daniel Gross thinks about talent through the lens of venture capital. They notice.

Keep in mind that the venture capital or “Silicon Valley” approach to talent search worries much more about “sins of omission” than “sins of commission.” (203)

Thus both are on the lookout for big hits that lead to big wins even if there are a lot of misses. It is important to keep this in mind when thinking about your situation, whether you are searching for talent or you are the talent. Most situations have more limited upside and more serious downside. It is not usually so easy to walk away from or limit the damage from one’s bad talent choices. Also I suspect the bad VC choices matter more than VCs realize due to incentive effects, both globally for all VCs and in terms of selection effects and how candidates pitch themselves and select among possible funders for specific VCs.

As talent seekers and writers of a book called Talent, they of course talk their book.

Furthermore, the unavailability of needed skills and talent is judged to be the number one threat to businesses. When we speak to CEOs, nonprofit directors, or venture capitalists, lack of proper talent—and how to go about finding more of it—is an obsessive concern of theirs (111)

I had a chance to be CEO, and I did indeed spend a lot of that time obsessed with finding, recruiting, retaining and utilizing talent. It was most certainly not the number one threat to the business, which was lack of money and our inability to raise it (along with our inability to sell our product for a lot of money). I then had a chance at a different company to essentially be the technical cofounder, and yes we have a huge problem finding great engineers, but also once again the number one threat to our business involves some combination of raising money and being able to sell products.

The last few years have made the talent claim more plausible, with a lot more talk (in EA circles, and in my other circles more generally) of being ‘talent constrained’ and the primary problem being lack of people capable of running projects rather than a lack of the necessary funds. Finding people capable of Doing Thing and even more than that of Leading the Doing of Thing is plausibly the limiting factor on many things now.

They also point out that if you are the talent people are searching for, that makes it important to know how other people are searching, so you can help the right ones to find you.

Don’t just think of talent search as a problem faced by “the boss” or by human resource departments. If you are hoping to be discovered, one of the most valuable things you can learn is how other people are thinking about talent (or how they should be thinking about talent), in case you can showcase exciting and valuable traits that potential employers otherwise might miss. You have to worry about talent judgments at least as much as the boss does. Just about everyone is on a quest to find talent in others or to show off their own (122)

This does not imply that you should be reading Talent for this. That’s a harder question. My hunch is that depends on how ambitious you want to be.

Talent is focused on the best ways to find talent. It has some things to say about how most people looking for it actually find and evaluate talent, but mostly to help you avoid their mistakes and know what things will be undervalued and thus underpriced. There is information there that can be applied when you are the talent, but in practice if one is looking to play it relatively straight because one is facing talent seekers playing it straight, one is better off with a book that plays it more straight.

If you are an exceptional talent looking for an exceptional match, that advice might not apply, but it also might apply anyway. My experience with several (very good) talent evaluators who were seeking extraordinary talent were doing rather different things than are talked about in Talent. In the case where I failed, I still failed for ordinary-interview-failure reasons.

The core message the book has for talented folks trying to be found is: Mostly don’t waste time playing signaling games. Instead cultivate your talent, do real things, be open and be aggressive about seeking out other talent and talent seekers when it would be useful. Which as always implies its own signaling games, and knowing when to set aside that advice for the moment – but it is the right mistake to be making.

That’s the macro-level overview and synthesis as I see it, as opposed to the micro.

Micro and Macro

What is interesting on every page of Talent is the micro, the dive into particular questions and strategies and details. How should we think about a particular personality trait, what does it predict or have to offer? When is this general disadvantage actually an advantage? What are good and bad answers to this job interview question, and how should you follow up tactically to ensure the interview goes places that will inform you?

What excites me about writing this review is the opportunity to dive into those micro questions. Which then allows one to figure out the important macro things they are pointing at, and incorporate them into the first section above.

In the longer sections that follow, I’ll be focusing on the micro. By all means pick and choose the parts of that that seem most interesting to you.

I only managed to get about halfway through my highlights this week, and I’m declaring myself out of time, so I’m going to (at least for now) stop there. If I feel sufficiently motivated, I will do a Part 2.

For all quotes from the book, the number in parenthesis at the end is the quote’s location on the Kindle.

Interview Questions

The key point of the whole interview approach is to get people ‘off script’ which means putting them at ease and asking questions they did not prepare for, ideally that they could not have prepared for. Which is more anti-inductive than the book wants to think, especially if you’re using questions from this book, but that move is likely mostly safe for now.

Still, at the very least, you are getting “the real version of the fake person,” and that is still more valuable than trying to process prepared interview answers. (437)

There’s at least two ways to think about each question. You can ask: What is my answer? What should be my answer versus what is my real answer? And you can also ask: How would I evaluate answers? Do I agree with the frame and thinking behind the question?

Tell Me a Story

Before getting to the strict questions, there’s the concept that asking people to tell stories can be good for eliciting more truth.

A simple question designed to elicit a story, such as “What did you do this morning?,” is a good way to begin to get to know a person without being threatening. The stories you hear will reflect how the candidate organizes ideas, adds emotional valence, drives a narrative arc, and selects what is important. (442)

…

If you ask, “Tell us a detailed story about how a co-worker expressed appreciation for you,” virtually everyone will tell a version of what actually happened. (449)

My answer:

I woke up around 5:30, then I fed the baby while watching the Mets game, made myself French Toast and caught up on emails, Twitter and a work project until the nanny got here. Then I played Rogue Legacy 2 a bit until my wife Laura was fully awake, after which I started writing, with a pause to take Gideon to preschool.

On the question:

I can see how that gives you a lot to go on, while also being a very typical morning and my choosing not to have embellished or gone into extra detail anywhere. The decision of what level of detail to use is of course part of the question. Still, I can’t help but think it would be a little weird and unfair/off-putting? As if I was preparing for a test and then the test wanted to know how I was studying for it? That’s kind of weird, right?

The coworker thing doesn’t seem right to answer on a blog, but also I notice that when I thought about answering this in an interview I got a very strong ‘I really don’t want to answer this, this is making me stressed and uncomfortable’ response. I simultaneously get the feeling ‘I’d be bragging’ and ‘I’m going to be judged based on my ability to find the best possible brag and they’ll assume this is the best I could do’ and ‘my memory sucks I can’t think of anything oh no’ and also ‘that’s none of your damn business.’ I wonder if that’s common.

The ultimate pure version of all this is the question “How Did You Prepare For This Interview?”

Here are some questions that not only will elicit stories but also might yield relatively interesting answers: “How did you spend your morning today?” “What’s the farthest you’ve ever been from another human?” “What’s something weird or unusual you did early on in life?” “What’s a story one of your references might tell me when I call them?” “If I was the perfect Netflix, what type of movies would I recommend for you and why?” “How do you feel you are different from the people at your current company?” “What views do you hold religiously, almost irrationally?” “How did you prepare for this interview? (467)

When I hear that last question, my reaction is something like – that’s cheating. That’s not allowed. That’s profane, you’re fighting on holy ground. Which is, of course, a very informative answer in its own way.

My guess is that in many cases, my answer real would be something like ‘aside from at most learning who you are and what you’re about I simply showed up’ and I would then get highly positive selection based on the reaction. There are two ways one can view that answer. One can say that’s a terrible answer, almost the worst possible, you’re not taking this seriously and you don’t work to game systems so you won’t be able to game systems, you’re worthless, go away. Or you can say the opposite, that this is the best possible answer, because the whole point of an interview is to see if there’s a good fit and it’s not a competitive process, so actions designed to ‘prepare’ for an interview distort that process and are wasteful. It’s great to find someone who is focused on other things instead and also you can evaluate the rest of the interview in light of this.

I want my life to draw from that second group of thinkers.

What’s a story one of your references might tell me when I call them? How do you feel you are different from the people at your current company?

These question from the above list stand out because they seems straightforwardly quite bad? If you are preparing and gaming the system via your references you’ll pick your references based on what you want them to say and how they are going to pitch you, so you’ll have a good idea what stories you’d want them to say on your behalf. If you’re preparing for a standard interview you’ll be all over the ‘why are you different’ type of question.

Whereas if the target didn’t do that and played it the way the system wants to work, this is still probably the kind of freeform request for selling oneself and if you resist that temptation then it’s kind of a request to tell a story at all or make a self-pitch at all, whereas a more specific request would go better.

So I wouldn’t ask these.

I also wouldn’t ask:

What are ten words your spouse or partner or friend would use to describe you? (590)

I’m trying to put my finger on exactly what feels wrong here, why it feels like a ‘standard interview question’ and also makes me want to not answer it. Is it that it feels like something that’s about to be graded, where there’s ‘better’ and ‘worse’ responses? That’s obviously not strictly true, you want to present an accurate picture in many senses. There has to be more to the self-reported-impression thing somewhere, as a demand for you to be strategic and choose a signaling strategy, where it feels like there is strong social pressure to not give a true ‘honest’ answer, where you’re being kind of ordered to sell and brag, and all that, and it all again feels like a strategic move in a game in the bad sense.

And the ‘ten words’ thing seems like some sort of trigger to push all that stuff up a notch or three? As if it wants to be certain you notice you’re playing a game and being quizzed.

I have a more mixed attitude towards this next one:

What’s the most courageous thing you’ve done?

My answer:

Among the things I could post online, I had to think for a long time. I’ve done a bunch of stuff that looked courageous. I gambled for a living, I founded multiple companies, stuff like that, but it always seemed like the natural thing to do at the time. Maybe that’s a lot of how courageous people do what they do, they set it up to seem natural.

Then there’s the actual most courageous things I’ve done, which I am absolutely not going to talk about here and no I wouldn’t tell (most of) you if you asked. I will only say that it worked out. Have fun guessing.

On the Question:

There’s a few things going on here, beyond simply ‘this should be a good story.’ First, you’re seeing if they can point to a concrete particular courageous thing, because a lot of people haven’t done any such thing, and therefore can’t, and therefore probably won’t do any such thing in the future. Similarly you’re checking to see what they think it means to be courageous, what they think takes courage. These kind of go opposite ways in some sense, you want someone who will be courageous but you also for any given action worth doing would prefer they not think it required courage – someone who needs to be ‘so brave’ or ‘so courageous’ to get through the day is in that state because something is less than ideal.

Then there’s the ‘what they think would sound impressive’ kind of angle, which gives me some pause. I’d definitely worry this is a question someone would prepare for, and to me it gives off the ‘sell me on you’ vibe for sure.

If I was the perfect Netflix, what type of movies would I recommend for you and why?

My Answer:

Mostly I’m putting this in because I want to say out loud: If you were the perfect Netflix you’d go back to doing what you used to do when you had access to everything ever made and also I had star ratings and predict everything based on whether or not you think I’m going to like it, which you actually used to do a great job doing and if you wanted to you could do so much better now, except also I’d extend that to allow ‘power users’ to basically give the system any and all information they wanted to, and in general wow there’s so much you can do I really want to do this some day but there’s so much other stuff to do. I even think it would be useful for AI Alignment to be able to solve recommendation system alignment in this way. And for now I’ll stop there but yeah I find the whole thing fascinating and someone needs to start the open-source fully-power-usable-and-customizable universal-recommendation-engine project outside of the platforms themselves.

What kind of movies would this recommend to me? I presume a wide variety of actually-good movies, but wouldn’t push me towards particular types because that’s something I would tell it and then it would optimize based on that – I’d essentially ‘tell it the mood’ and then it would offer me the choices.

On the Question:

To me the interesting part of the question is the rabbit hole of ‘what would a perfect Netflix be like?’ so much so that I get fully nerd sniped.

There’s also value in learning what types of things people want to be watching, but I’m not sure what one does with that information exactly?

Practicing Scales

Daniel recalls that he first learned from Tyler this question for prospective hires: “What is it you do to practice that is analogous to how a pianist practices scales?” You learn what the person is doing to achieve ongoing improvement, and perhaps you can judge its efficacy or even learn something from it. You also learn how the person thinks about continual self-improvement, above and beyond their particular habits.

If a person doesn’t practice much, they still might be a good hire, but then you are much more in the world of “what you see is what you get,” which is valuable information on its own. If the person does engage in daily, intensive self-improvement, perhaps eschewing more typical and more social pursuits, there is a greater chance they are the kind of creative obsessive who can make a big difference. (45)

My answer:

It’s changed over the years.

When I was a Magic player I would shuffle cards and take test draws, and of course you play a ton of matches under various conditions. And you analyze decks, strategies, metagames and particular plays in detail, you post-mortem your games.

When I was gambling on sports, my business partner and I would make odds on anything and everything, whether or not it had anything to do with sports. If we weren’t sure if we’d arrive on time, one person would set odds and the other would bet.

As a rationalist, one assigns probabilities to things, which is very similar, but also importantly different. A gambling line is not only about what you believe, it is about what you believe others believe, what they are likely to do, how you will adjust the odds and protect your risk and what degree of uncertainty you have about the right probability and how much your answer changes if someone disagrees with you strongly. A probability on its own contains none of that, it is pure and non-adversarial instead.

As a trader it was trading, as various people on the desk would make markets. This is different from setting odds because others can undercut your market, and also it uses a different set of language and convention. But it’s not a coincidence that all three had very similar kinds of scales.

Now, I haven’t stopped thinking in terms of probability, but that’s not my scales anymore. Instead, I model and optimize every little thing I see, more explicitly and consciously than I used to, and look for surprise. Anything I encounter is a puzzle to be reasoned out and solved for its own sake. What knobs could be turned? What does this provide evidence for and against?

On the question:

I like this question, as the answer will tell you a lot about how someone thinks about self improvement, what skills they want to develop and how they view the core thing that they are doing or attempting to do.

What bothers me somewhat are what I see as the assumptions that give the impression they are embedded in the suggested interpretation a given answer, even if strictly speaking they are not. Indeed, one could say that many of the best answers defy those assumptions explicitly and are all the better for it.

In particular, there is the assumption that you can and should extensively goal factor, self-improving sometimes and working at other times, as opposed to constantly doing both interchangeably in a way very different from piano scales.

There are certainly great benefits to deliberate drilling and to dedicating time explicitly to practice and self-improvement that does not produce any other useful output. I’ve done plenty of it in the past, and still do when preparing for what gaming competitions I still enter.

Yet I consider this to be a second best solution to the problem, the same way school is at best a second best solution to the problem of learning.

The first best solution is, whenever possible, to do things that are useful in their own right but that may have also been chosen with an eye towards giving you an opportunity to learn and improve. As a Magic player, you enter a tournament with an experimental deck and a new sideboard strategy, then extensively analyze the games between rounds. Rather than making markets or setting probabilities on arbitrary things, focus on questions where you want to know the fair price, while also paying attention to your refining your methods. Figure out how to optimize your day, then figure out how to optimize figuring out how to optimize it, and consider not stopping there.

And so on. Instead of going to school, set out to do something real, and learn what you need to learn along the way, with a mindset of deliberate practice and continual self-improvement in all things – there are sufficiently urgent situations where one would suspend that until after things are over, but they’re rare.

That’s my real answer. The problem with the question is that it doesn’t seem likely to elicit that answer, and also I worry it is looking for the ‘wrong’ answer in some sense.

The exception is when one is engaging in a finite game of absolute competition, where you can absolutely describe exactly what matters. That happens to describe sports perfectly.

the very top players in the game, such as Ronaldo and Lionel Messi, have an extra obsession with self-improvement that puts them above the rest. Moritz realizes that the most important prospects do not always arrive with big successes under their belts, but rather are determined to work harder than anyone else to reach the top. (267)

Magic: The Gathering wasn’t quite as good a fit, but was a much better match to ‘you had to be obsessional about explicit improvement and doing the work to win’ than when I was doing other things.

As for the part about not pursuing typical social pursuits, I do think you want to see at least some of this.

Another question raised here is: If you do see someone who doesn’t have an obvious drive or focus on self-improvement, does that mean they won’t improve, or won’t improve that much? Does what you see therefore equal (in the best case) what you get?

The chances are obviously lower, but I think no because it is very easy to miss kind of improvement that comes from treating work as at least something of a deliberate practice or otherwise learning by doing. In some sense, as long as someone is striving to do a good job and is given tasks that are challenging it is hard not to learn.

I think this is one reason that good interviews involve doing some amount of work or problem solving. That gives you a chance to see whether someone uses a challenge as an opportunity to learn.

What Are Your Open Browser Tabs Right Now?

My answer:

1. A spreadsheet covering all the individual card quests for the game I’m working on, Emergents (now in closed beta!), as we work to get all of them functioning properly. Only 156 more to go.
2. A spreadsheet tracking my Storybook Brawl results.
3. This list of 2022 Favorites to Date I hadn’t gotten around to yet.
4. This post from Construction Physics I have open because I still haven’t decided how deeply I want to go into the details of infrastructure and construction, and I haven’t figured out if that blog is useful yet, but I should add that to my links list and close that, so I’m doing that now.
5. This YouTube talk from the Chairmen of the Council of Economic Advisors about how they are currently thinking about policy. Haven’t listened yet.
6. The page for the podcast Meditations with Zohar, the episode with Matt Levine was so good but not sure it will transfer over to the others. He does ask very good questions but one has to be interested and have the necessary context – for Matt’s episode this worked great but I tried one other and it didn’t fully work. I transferred it over to the Podcast app for now – I use CastBox but am hoping to find something better.
7. A Google Document containing my giant list of links to be used in near term posts, which is pretty out of hand and up to 37 pages some of which are other things I put there for easy access but the majority of it is links. I really need to sort through and clean that thing up, work through the backlog.
8. The Birthday Mailbag from Matthew Yglesias that talked about the Dredge Act and mentioned my post on it.
9. The spreadsheet I use to generate the graphs and charts of cases and deaths in my Covid posts.
10. Vitalik’s recent post on Stablecoins.
11. Scott Sumner’s post ‘who are the homeless’ cause I keep meaning to follow up and dig down into the numbers and I don’t want it lost in my pile of links.
12. The draft of this post that I’m writing in now.
13. My Kindle highlights from Talent.
14. My Gmail inbox.
15. My other Gmail inbox for work.
16. My Google Calendar.
17. The latest post from Dominic Cummings, Snippets 4.
18. A Tweet about speeding up review for green energy projects, that should just go on the pile of links under policy, so completely insane we haven’t done this.
19. A thread about where in Europe guests do and don’t get served food, seemed like it might be interesting.
20. An old post of mine, Seemingly Popular Model of Covid-19 is Obvious Nonsense, which I was linking to in a draft of another post but then I found out I’d already written a version of it back in 2010 so I didn’t have to write it and could instead build upon that and write something better later.
21. A Google Document that is planned to contain my views on ideal federal government policy, for those who find such a thing interesting. Still in the early stages. Will be at least book-length if it finishes. 2 copies so I could navigate while editing.
22. The previous, less formal, more here-is-what-rhetoric-I-would-use-for-it rather than two-sided-here-are-the-relevant-considerations version of that same document, which ended up being 37 pages and has 4 extra pages of other stuff floating around in it right now. Two copies, one to see the outline at the top and one to write it.

One good part about doing this at home where you can actually look at the tabs is it gives you incentive to close some of them and reminds you of others. Having 22 open tabs is not so many these days, but I still consider it ‘too many’ and think it would be better to get that down to 5-10.

On the question:

As long as no one sees the question coming, I love the question because everyone is unique when it comes to what they keep open unless they are running a very tight ship. As the book suggests, you learn a lot about a person, how they think and organize, what they are interested in.

The natural question then is: What are ‘good’ answers versus ‘bad’ answers? In some sense this is of course a wrong question. You’re not looking to classify people as good/bad or better/worse, you’re looking to figure them out, so in that sense there aren’t wrong answers. In another sense of course there are wrong answers, and of course there are some things you’re pretty much always looking for.

In that second sense, as Tyler said in an interview with Megyn Kelly, what he’s looking for more than anything is a sense that the person is attempting to solve problems and figure things out. Megyn thought her tabs were bad, and Tyler reassured her that he fully approved because she had specific real world problems and was attempting to solve them.

He also looks for signs the person is working on their skills in their spare time.

Especially for higher-level jobs, the question of spare time is a critical one. The very best performers don’t stop practicing for very long, and if you hear or sense that a person doesn’t do much practicing and skill refining in his or her spare time, they probably are poorly suited to assume a top position or to meet very high expectations. The browser-tab question gets at this, but it is not as awkward or exaggeration-inducing as “How hard do you work anyway?” or “How much do you try to improve your skills in your spare time? (328)

It makes sense that working on whatever skills are relevant to anything is still an excellent sign, where the thing you want to most measure is ‘do you work on real skills in general’ rather than exactly which skills are being worked upon.

It is the absence of these things that Tyler thinks is a poor answer. In particular, he thinks not having open browser tabs at all is a bad answer.

Which to me is odd. Open browser tabs tell you a lot, but one thing they tell you is a person is using open browser tabs as bookmakers/placeholders of a sort. Is that actually a good idea? I read an article on Bloomberg recently about the Arc browser that intentionally tries to prevent open tabs, among other things auto-closing most tabs after 12 hours into an archive. The founder argues that Chrome and Safari are stagnant for fear of alienating existing users but that dozens of open tabs are an actively bad sign that something is very wrong, and instead they want to build a new one. They’re still in ‘join the waitlist’ mode so I won’t be trying it out yet but if they randomly sent me a copy after seeing this I’d give it a whirl.

I spend a lot of time organizing stuff, web-based and otherwise. I still end up using my inbox and open browser tabs as de facto to-do lists of a sort reasonably often, and it works but at least I have the decency to feel bad about it and like I should be doing something better, you know?

What counts as a good or bad answer, in such cases, I think a lot comes down in large part (far from all, but a large part) to what the candidate thinks is a good or bad answer. What parts of this are they proud of and why, versus which parts do they think are embarrassing or shouldn’t have been there? If they knew there was an interview with this question today, what tabs would they have chosen to open or close?

What Subreddits or Blogs Do You Read?

This is similar to the tabs question, getting at where someone’s true interests lie.

We both find during interviews that “downtime-revealed preferences” are more interesting than “stories about your prior jobs.” So for instance, “What subreddits or blogs do you read?” usually is better than “What did you do at your previous job? (338)

…

“What subreddits, blogs, or online communities do you enjoy?” “What is something esoteric you do?” (473)

My Answer:

I don’t read subreddits, as I don’t find the format useful – it’s either Twitter where I follow a variety of lists and people, or it’s blogs and podcasts.

For blogs, I took this opportunity to update my list of links on my Substack main page. For podcasts, I want to mostly save that for another time, but the big two I make sure to listen to are Odd Lots and Conversations with Tyler.

I remember ‘online communities’ but that all seems so old school now. One might be useful though. I’m considering starting a Discord server.

On the Question:

Together with the browser tabs I have no doubt these questions are highly useful in understanding someone or finding things to talk about, and mostly it seems good, but I also get a sense that there’s something wrong with all this? It’s an invasion of privacy, crossing the streams, threatening to mess people up or something?

Seems worth unpacking a bit. I know I’d find it completely unacceptable if someone were to actually invade my privacy and extract this kind of information without my consent – and that if I thought that was likely I’d probably change some of my choices in ways that would make my life and the world worse.

There’s a sense that some things are ‘fair game’ while other things are private, in any given context. Some things are subject to signaling games and are done in light of others’ potential Bayesian updates, while others need a safe space where that isn’t true. Open browser tabs seem like they are somewhat muddled, since you are indeed ‘leaving them open’ to some extent, and the (ideally) temporary nature of them prevents other issues from getting too severe, but then in other ways things get murkier.

When I’m choosing which things to read or do ‘on my own time’ it is all for the good if I am thinking about how this helps me be better at my job. I worry more if I’m thinking about whether someone knowing what I am up to in my spare time, or what I am reading, even if I am not worried about cancel-style guilt by association.

One acts so differently when others are looking or might be looking, bringing in a whole new set of considerations. It opens up new realms to more intense signaling games, And That’s Terrible. In some sense you want people to be in a totalizing game of self-improvement and accomplishment, with its increasing returns to investment, even if it eats into other pursuits, and you want to measure that. What you don’t want is even a hint of Goodhart’s Law along similar lines, which will rapidly eat away at absolutely everything and destroy people’s lives as humans along with the actual investments that generate the increasing returns. If one has to worry about looking like you’re investing, there aren’t resources left to actually invest.

The obvious response to this is that so long as such questions are a surprise, so long as they are rare, that all is fine. You will do what you would have done anyway. It takes a reasonably large probability of being asked about your browser tabs or blogroll – in ways you can choose to dodge or fudge if necessary – before your behaviors start getting seriously bent out of shape.

This then gets back to the question of underused approaches, finding what is overlooked and underpriced. Talent is full of strategies whose entire approach is built upon the assumption that you are doing something completely different, because almost everyone else is doing something completely different. When people are doing completely the same things instead, as they usually are, they have to be bounded within a finite game where they can be optimized for the inevitable adversarial dances.

What’s Wrong with Pioneer’s Tournament System?

Daniel wants to know.

Daniel also likes to ask interview candidates to share their criticisms of the Pioneer tournaments he runs to select eventual recipients of Pioneer’s investments. He is looking for a specific approach to their critique, as much as the substance of their response. He gets especially worried when they ramble on vaguely rather than providing a focused approach to feedback, or when they choose to unload their gripes on the broader world of tech (a common response that suggests a lack of focus). He seeks specificity and frankness that focus on how Pioneer’s tournaments can be improved. (490)

Ask for actual useful information? Don’t mind if I do. Alas, the book doesn’t talk about the tournaments so it’s hard to offer useful feedback. I do like the question both as valuable for its own sake and also as a check to see if someone is willing and able to offer specific criticism.

What Is Your Most Absurd Belief? (Or: Questions Whose Utility Have Expired)

Many interview questions have a limited period of usefulness, and so they eventually need to be discarded or modified. Consider one of the most renowned interview questions of all time: “What is it you believe to be true that other smart people you know think is crazy?” There are other versions of this question, typically of a contrarian bent, such as “What is your most absurd belief?” This type of question is often attributed to Peter Thiel, although it seems it first came from a blog post by Tyler in 2006, and it is part of a broader philosophic tradition of critical self-examination.11 In its early days, this question was magnificently effective. It caught respondents by surprise, and at the very least you could observe how they responded to an unexpected situation, which was a plus for gathering data about their performance abilities. Furthermore, their answers revealed a good bit about how they understood the world.

You had a chance to learn “What kind of kook is this person?”—and really, what would you rather know? That kind of information usually isn’t contained on a resume. You could see how quickly they could put together an argument, and how quickly they could make it sound plausible (or not). Alternatively, some people will confidently answer the question with a totally ordinary, orthodox belief, convinced they have stated something highly unusual. Those individuals may be good hires for their reliability and conformity, but don’t expect them to shake up any internal systems that need reforming. If they think “globalization extended too far” is their weird, radical, absurd belief, whether or not you agree, the proper conclusion is that they are not exposed to many truly radical, weird ideas at all and wouldn’t know one if it hit them on the head.

When Tyler first put the absurd beliefs question out there, his favorite (written) answer was: “I believe that if you go to the beach, but you do not give the ocean a chance to taste you, she will come take her taste when she chooses.” He later met the person who wrote that, and she turned out to be remarkably smart and productive, and she was underplaced in her job at the time. (673)

I would note that I would give different answers to the different forms of the question, and most others likely should too – the whole idea of ‘thing I think is true that people around me think is crazy’ is largely about believing that the people around you being crazy in a particular way, so I think it would be right to answer with something that most other people find relatively normal, rather than my most absurd belief (and I’m going to keep you guessing on that for now, sorry).

Tyler says once word got out, the canned answers were no good.

And a canned answer to that question, while it revealed skills of preparation, just wasn’t that revealing anymore. The interviewee would prep to demonstrate just the level of contrarianism that he or she thought the interviewer was looking for. And suddenly the question now was rewarding a perverse kind of conformity—conformity to the proper level of contrarianism—rather than the true contrarianism it was supposed to ferret out. (692)

I think this underestimates the dynamics, and assumes the answer to the question and that there is a ‘right’ response and the right answer is to seek the right response. In some cases the right answer is the wrong response, because it shows you weren’t intentionally choosing the right response. If you were a true contrarian, would you alter your level of contrarianism to your expectations of the expectations of the interviewer? I say no, you would not.

Also, one could say that the good answers are things you hadn’t previously even considered, so even if you don’t get such great info on the person you’re still getting an interesting new belief, or if you don’t get one you’re getting info that they don’t have one to offer, of any level of contrarianism.

“Which of your beliefs are you least rational about?” “What views do you hold almost irrationally?” (731) and “Which of your beliefs are you most likely wrong about?” (742)

My Answer:

I mean, if I knew that I wouldn’t believe them. Or would I? Hard to say. In the context of founding a company, the real answer would of course be ‘that this company will succeed’ because it is well known that this belief is usually wrong. Or perhaps it’s something related to rationality by way of AGI or immortality or some such. Or in another sense maybe it’s simple stuff, that I think my family is the absolute best and the Mets are great and so on? That could be it too.

On the Question:

In some sense I see this not as checking to what extent the person is keeping multiple sets of books, and where the person sees this as useful, along with the question of whether the person thinks one should strive to hold accurate (rational) beliefs at all. I find keeping some sense of multiple sets of books more useful than many others I know, but far less (yet far more intentionally and self-aware about it) than most others.

My gut expectation is that this is mostly going to elicit BS, but also tell you what the candidate thinks is the good kind of BS to go around with, so I suppose that is at least somewhat useful.

I also don’t remember the book mentioning it, but often the obviously correct response will be a religious belief, or religious belief in general, to which I assume replying ‘aside from that sort of thing’ is the next move, but they score points for self-awareness.

How Do You Think Is This Interview Going?

The most brutal of all the meta questions is: “How do you think this interview is going?” An even more brutal version of it is to have two separate people ask the same question at different points throughout the day, and later compare notes. We don’t recommend it for most settings, but let’s pause for a moment and think through what this question does.

You are in essence asking the candidate how much weakness he or she wishes to disclose. Should the candidate give an articulate account of her weaknesses so far, thereby impressing you with her insightfulness but also confirming your negative impressions? Or should the candidate stonewall and instead give an account of everything that has gone well? If nothing else, you will confront the candidate with a surprise situation and put her in a difficult position, but one that gives her a chance to shine in confronting a challenge. But here’s why we are not entirely enamored of this one. Let’s say the candidate opts for the risk-averse choice and simply stonewalls it on the negative, offering a strong positive account of what went well. How much should you hold this risk aversion against the candidate? (747)

How should you respond to this? It’s complicated.

As the interviewee, you want to maximize the chance you get hired for jobs you want to get hired for, and ideally to avoid jobs you don’t want to get hired for, and you don’t know what you will get punished or rewarded for saying. The intention may have been to see if you’re honest and self-aware, and to see what you think they’re thinking about, but that doesn’t mean highlighting your potential weaknesses won’t reinforce and draw focus to them. And you often have little or no idea how things are actually going, because you don’t know what they are looking for. If you are doing your job properly as an interviewee, of course, there will be a correlation between your performance and the prize – the more you should want the job, the better you’ll be doing.

If the prize isn’t worth having, then it doesn’t much matter what you answer, and in fact you want to get turned down (although it does feel good to get a yes anyway, so maybe not if you know not to accept the offer, and also you might want to use the offer as leverage). So you should presumably start by assuming the job is indeed good and you want it, which means the company wants someone like you for the role, which means you’re probably doing pretty well. And given that most interviews end without an offer, if you’re not doing pretty well then it likely matters little what you say, so you might as well be relatively optimistic.

Stonewalling on the negative could be thought of as the risk-averse choice, but that isn’t obvious. Usually there will be something that obviously didn’t go great, and also you can use this as an opportunity to show high self-expectations. There are always things you could have done better. If you don’t see anything that went wrong, that means they should evaluate you as if you’re doing about as well as you can do. That’s probably not an impression you want to give. Whereas if you hold yourself to a higher standard, and frame your performance as below your expectations for yourself, that seems like a positive.

It also seems to me like expressing uncertainty is almost always good, unless you want to get hired for a position where they punish uncertainty, which seems terrible?

I’d also expect ‘always only be in adversarial mode and only say positive things’ to be a mode that’s on or off across the interview, and this is a place where they are checking to see which state that switch is in.

So I’d say even more than it being a ‘risk-averse’ answer to stonewall, stonewalling tells me you think that stonewalling is risk-averse, which tells me what you think I am looking for, and also that you’re willing to be somewhat dishonest to avoid risk, and yes I think it seems reasonable to hold all that against you.

I wouldn’t plan on asking this, but there are times when it should seem like the logical next question. In both of the great interviews I’ve done, that question would have been insightful at the right time.

How Ambitious Are You?

Finally, here is another useful question used by Peter Thiel: “How successful do you want to be?” Or this variant favored by Tyler: “How ambitious are you?” Offhand, this one might sound a little silly, but it can be useful to ask an individual to put his or her cards on the table. If the essence of the answer is “I wish to ensconce myself as a midlevel manager in a secure company,” well, that is probably the truth. You would then be ill-advised to invest in that person as an ambitious entrepreneur or a founder.

You might want to invest in them as a midlevel manager, but even then be aware that promotion incentives may not motivate the person very much. When Tyler posed this question to one potential academic hire, the candidate simply responded: “I would like to publish some papers and get tenure.” (“That’s it!?” Tyler wanted to shriek.) Alternatively, a potential fellowship recipient might have the ambition to cure cancer; a person who actually has such an ambition will be able to articulate it pretty clearly and with some degree of conviction. Upon reflection, this question is surprisingly difficult to fake. If you really don’t have the ambition to cure cancer, it isn’t so easy to claim in an interview that you do and defend the reality of your vision. It will sound like mere bragging, not backed up by analysis or details about the planned execution of the vision. On the other hand, people with truly ambitious goals often are eager to tell this to the world, and often they have some plans already worked out. Of course, in some cases you will immediately sense that a person’s ambitions are set implausibly high, and that is reason for doubting their judgment and self-knowledge, however sincere their answer might be. World peace would be a wonderful thing, but we’re not sure you should hire the candidate who pledges to bring it about. (762)

One common theme is that good questions to ask are questions where honest answers are ideally rewarded, or at least seem like the thing to do at the time. One way to do that is to ask a question where the process that chooses the true answer is the same process, or similar to, the process that chooses what you think is a good thing to say in an interview. Thus, if asked how ambitious you are, you’ll probably think that the true answer is the ‘best’ answer. The person who said ‘I want to get tenure’ probably thinks the proper goal of life is tenure, and that anything more would sound grandiose. If someone is telling you about their grand ambition, it does seem rather unlikely that they are lying about having one.

If At First

the candidate avoids the question, ask it again. (531)

I notice I am confused about this idea. Sometimes it is absolutely the right thing. Sometimes you need to know. Sometimes repeating the question until they ‘get it right’ is all they need to work through it. Perhaps their refusal to answer tells you the answer is all the more important.

Yet sometimes them avoiding the question is answer enough? Or alternatively, if you wish to get and keep them relaxed and in a non-hostile interaction, perhaps you should accept their refusal to answer.

So I do endorse considering the option to ask again and refusing to accept a non-answer, but I think the majority of the time it’s right not to do that. Them avoiding the question is some combination of answer enough and a signal you asked the wrong question.

Then there’s the situation where you get an answer, but you keep asking anyway.

If you are going to ask achievement-oriented questions, avoid the ordinary by continuing to ask for successive instances of candidate success until the respondent can’t come up with any more. So don’t just ask “Give me an example of a time that you felt you went above and beyond the call of duty at work” one time, because there you are just testing for basic prep. Ask it again. And again. And again, until the candidate can’t come up with any more answers. And don’t look away, break the tension with a chuckle, or give the candidate a chance to divert attention and halt the questioning process. (634)

To what extent is this a test of memory and the keeping of records of achievement, especially specific stories one can use in such spots, versus a test of how often one actually goes above and beyond the call of duty?

It’s hard to avoid the ‘what would happen if someone did this to you personally’ lens when thinking about interview techniques, so my lousy memory for such things is focusing my worries there.

One could think of this as, you ask once you are asking for basic prep. When you ask more than once, what kind of prep is being tested? You are hoping to blow past anyone’s ability to prep answers, and you will likely do so for interview purposes but what about those who track their achievements for other purposes, which seems like a reasonable reason to stack up tangible examples? If someone comes up with answers two through ten ‘too quickly’ in some sense, or they are too organized or too polished, what does that tell you? If they have to think a bit, is that a good or bad sign?

I also believe some people think of ‘above and beyond’ as what others think of as ‘the basics of the job.’ Thus, there is a job I had where I could claim I went above and beyond every day, or I could kind of say I never did because that’s simply the job. When you are a CEO of a startup, is it even possible in theory to go above and beyond? The job is supposed to be all-encompassing. That’s about this specific example question, but a similar version likely applies to many others.

The first time you ask this question, most candidates will draw upon their preparation. Sustained repetition, however, will get the person out of prep sooner or later, usually sooner. The previously cached answers will be exhausted, leaving time for the real meat. You will then see the depth of the candidate’s intellectual resources and emotional resilience. How does the person respond when being challenged continually? How many instances can the person come up with? If the respondent feels stymied and truly has no further responses, how readily and how poorly is he or she willing to resort to BS? Finally, if the candidate really does have seventeen significant different work triumphs, maybe you do want to hear about what number seventeen looks like. (639)

There is this angle as well. You’re interested in how they respond to the challenge as much or more than you are interested in the content, although you might (or might not) also care about the content for its own sake. To some extent half the book’s suggestions seem to boil down to ‘the point of an interview is to figure out who you’re dealing with so get them reacting to things and see what happens.’

Change of Venue

The book suggests taking candidates to different locations.

to a coffee shop or restaurant. Take a walk or sit on a park bench. This can happen mid-interview, or you can hold the entire interview there. In any case, the different setting allows you to see how the candidate responds to unexpected change and can ease the move to a more conversational exchange. It also makes it harder for the candidate to keep up the protective interview mode. (565)

This certainly seems like ‘part of a balanced interview process.’ If I was doing one interview and had an hour or two, I usually wouldn’t want to do this, I’d be ‘spending my time’ inefficiently in terms of information gathering. Yes, you get to see (as they mention elsewhere) how this person interacts with the world in various ways, how they treat waiters and so on, but is that the most important thing to learn? How much of your time will be ‘wasted’?

If you have a whole day or more to evaluate someone, or at least the majority of one, then that is different. Everyone has to eat and you get more benefit from taking a break of sorts and changing the overall tone and setting. So I like this for later stages of high-level interviews, but I’d only do it once someone had my full attention.

The Premortem

“If you joined us and then in three to six months you were no longer here, why would that be?” Or ask the same question about five years down the line as well and see how the two answers differ. (592)

Everyone remembers ‘what is your greatest weakness?’ This is the better variation of that, asking what might make things not work out.

I’m more excited by the few months timeline than the few years. For a few years, the question feels like a loyalty or endurance test of some kind. It feels like it’s telling me to come up with a false story. Whereas if the timeline is only a few months, clearly it didn’t work out so the question is why things could fail that quickly, and you can give a more honest answer that gives insight into what the person cares about and worries about. Might they prove not up to the task? Uninterested? A bad fit? What would get them to pull the plug so quickly?

What Did You Like to Do as a Child?

This gets at what they really like to do, because it harks back to a time before the world started bossing them around. (596)

My answer:

I played a lot of games. Card games, board games, computer games.

But also…

On the question:

Before the world started bossing them around? Wait, WHAT?

Children are bossed around the most of anyone, by far.

The bulk of the day you are told exactly where to physically be, where you must do exactly what you are told, otherwise keeping still and saying nothing. If you do not spend a lot of the rest of your time doing the arbitrary work they assign you, they punish you further as will your parents. The rest of the time, your parents largely tell you what you can and must do. Where you can go isn’t up to. Who your friends are often isn’t up to you. What you eat isn’t up to you. Where you sleep isn’t up to you. What television you can watch or games you can play often isn’t up to you.

Meanwhile, you are told that everything you do bears upon whether you get into a good next school, which will determine your entire future – you are constantly threatened that failing at one of thousands of things will ruin your entire life. And then that you’ll be judged based on arbitrary hidden and random criteria, so you should fill your life with things that ‘look good on an application.’

I already went on too long, but basically I cannot fathom someone thinking children aren’t being bossed around.

I mean, I get the intention behind the statement. Kids, one hopes at least, are still spending some amount of time ‘being kids’ although I increasingly worry that they are increasingly not being allowed to do this. And what people look back upon fondly, whether or not they had any say in its selection, gives you insight into the person, sure.

What is an example of a mistake you have committed but did not come to regret for a long time? (1,079)

My Answer:

Oh man I have at least two really good answers to this one.

The first is that when I founded MetaMed, I was running a startup, but I got confused and thought I was running a business. Thus, I tried to make the company be profitable and work as a business, rather than realizing that such concerns were irrelevant and I should be focused on creating proper hockey stick graphs that would enable additional fundraising rounds. By the time we realized that wasn’t going to work, it was too late.

The other, which predates that, is that the owners of a company told me that I was the future of the business and that if I kept going around making the place profitable and working my ass off they would transition ownership and leadership of the company over time, and I could do this remotely.

And I believed them.

Whoops. Yeah, no. That’s not how the world works. That’s on me.

I mean, it worked out fine, and I don’t even know I would have really wanted to move onsite so I could have had a shot of actually taking over – certainly I’m happy things turned out the way they did overall – but oh what might have been.

On the Question:

It’s a weird one, because the null response – that you have made mistakes and learned from them but don’t regret a thing – seems highly valid. As does not having been at a job for a long time to have such a mistake to regret.

If someone does answer, there’s the obvious ‘maybe they tell me something awful’ aspect but mostly it’s asking what long term mistakes are most salient to people, what do they notice and learn from and consider important.

Do You Deserve To Succeed?

Variations of this that bring moral desert into it get weird.

Peter Thiel, the book reports, asks whether someone morally deserves to succeed, and uses that as part his funding decision. This matches my other experiences and knowledge of Thiel. I know of examples where he knew that someone or something was failing, could have offered that information to make failure less likely, but he did not do that, and my model of why is something like ‘if they can’t figure it out on their own they deserve to fail’ and it would be somehow wrong to stop that.

I don’t agree, but I do understand how one can get there. I have the instinct, at least a little. Part of this is that if you ‘rescue’ the situation, perhaps that means you’re going to have failure down the line instead when it’s more expensive? And part of it really is some sort of feeling of justice.

Questions like this seem kind of bizarre to me:

“In the context of the workplace, what does the concept of deliberate sin really mean? And how does it differ from a mere mistake? Can you illustrate this from the experience of one of your co-workers?” Note that the invocation of the co-worker makes the inquiry less threatening and is more likely to induce an honest response. If you wish to go direct, you might try: “When have you experienced great regret in the workplace and why? How much were you at fault in that interaction?” (1,081)

What does deliberate sin mean? It means you’re using a moralistic model of the world. There’s certainly a sense in which some things are accidents and some things are deliberate, although not as much as one would think – much of the time, if an ‘accident’ happened it was kind of accidentally on purpose or because you didn’t much care if it happened or not, or you wanted it to happen but couldn’t do it ‘on purpose,’ and so on. It’s a continuum.

Asking people to ‘tell on themselves’ in these ways seems like it goes against the principle of having a cooperative relaxed interaction where you get to know someone, as it puts someone in a game-style bind where they are induced to go into a tactical defensive optimization mode. If you can convince people to respond naturally and honestly to such questions they’ll have good value, but that seems super hard, and if you are good enough to do that you’ll be fine either way.

What Would You Ask Me?

Another set of meta questions reverses the tables. Try this one: “What criteria would you use for hiring?” Again, you are testing an individual’s understanding of the job, of him- or herself, and of the interview process itself. It can be very instructive to ask, “What questions do you have for me?” or “What questions do you have about us?” or similar variants on this theme. The goal is to get the candidate to voice information about what he or she really cares about, and it also tests how well the candidate knows the job or project under consideration. On top of all that, it is testing for whether the candidate has penetrating insight into things you know about and, furthermore, whether the candidate might someday be in a position to insightfully interview, and perhaps hire, other people.

Still, this question only works some of the time, in part because so many people prep for it. It works best when you’ve already covered what they might have asked about from their prep, or when you can phrase the query in such a specific way that they can’t fall back on their prep. Something like: “During the middle of this discussion, we chatted about [a very particular project]. What questions do you have about that project?

If you simply ask them to present you with questions and they come back with “What does your benefits package look like?,” then you have tested only for prep. That is an entirely reasonable comeback on their part, but usually you are looking for a greater revelation of information. (798)

When I was doing what preparing I have done, part of that is being told not to ask about the benefits package during the interview. The benefits package is for after you get the job offer, there will be plenty of time then to ask such questions or negotiate about them. Asking now shows you don’t understand this or are focused on the benefits side, which is a bad sign, so if someone is evaluating your prep I’d expect them to think you’d failed. At a minimum, it means you don’t have a better question, either in terms of info you want or a way to benefit yourself, or both. Which either means again a lack of preparation, or worse.

Asking for a question on a particular thing avoids the issue entirely, it is true. The honest answer most of the time would normally of course be ‘no, I don’t have questions, I don’t care’ but presumably a good candidate can decide to be curious and see where it takes them. In good interviews the questions and scenarios are interesting, so I ended up asking questions like ‘what did you actually end up doing in that spot and why?’ or asking about whether I’d made any mistakes or missed anything important or what not. Continuous improvement checks, of a sort.

Asking what criteria you would use for hiring is another of those could-be-walking-into-prep but maybe it’s fine class of questions. Written tests are not obviously less revealing than ones off the cuff, and I want to hear the answer. Might even learn something.

Seeking Different Talent

The Spark

note whether the interviewee uses unusual expressions, seems to be coining their own phrases, explains basic concepts in a way different from what you might hear in the mainstream, speaks as if they are developing useful memes, has unusual rhythmic patterns to their speech, or conjures up a unique worldview. (536)

…

Demonstrating a language of one’s own is not always a positive sign, especially if you are hiring for a job requiring low creativity and high conscientiousness. But if you are looking for a founder, an entrepreneur, a maverick, or a highly productive intellectual to lead a venture to the next level, creating and commanding one’s own language may be an important positive feature. It may suggest that you have found a truly creative, charismatic, and generative individual, one of the 1 or 2 percent of people who are capable of creating something grand. It’s not that having a special language always translates into successful results; if someone creates and opens up their own world with language, there’s no guarantee that such a new and original world is a good one, or a good match for what you want to achieve as an employer or backer. But it is a sign that further investigation into the person is warranted, for there’s a chance that they might be an earth-shattering, pathbreaking genius. (549)

The way I would put it is that if they create their own language they don’t live in the consensus reality. Sometimes that means they live in a different reality. Sometimes it even has a relationship to physical reality. Be especially on the lookout for when they can draw others into the one they see, and on the flip side be on the lookout for it not having any connection to physical reality at all.

If you’re hiring for a position requiring low creativity and high conscientiousness, this book’s techniques are mostly not for you. That’s a different puzzle with different solutions.

If anything the most interesting thing here is the suggestion that more than 1% of people are capable of creating something grand. That’s a lot. It would be very good news if true, even if only true at a relatively young age, but that does not match my experience. Yes, at least 1% of people I have met were at one point capable, but there’s a lot of selection effects going on there.

This makes more sense in the context of an existing search for genius-level talent, which is one way of describing the job of a venture capitalist. Within that group, your prior on someone being a pathbreaking genius is orders of magnitude higher (at least one, likely two or more) and thus a 1-2% chance is plausible, while noting that most people capable of it will ultimately fail even if supported.

Buying Low

A common mystery about discrimination is that if some group was being systematically discriminated against, you could hire them at discount prices and you’d make lots of money, so why don’t we see this happening?

This applies to the usual suspects, but also unusual ones.

When we think of discrimination, we often think in terms of race, gender, and sexuality. Those remain very real problems, and the issue is so deeply rooted that there are many other areas in which American society has made mistakes in allocating talent. Circa 1970, were we really using nerds and introverts to the greatest and most productive degree possible? How about people with disabilities, or recent immigrant arrivals, or short individuals? Prejudices were—and still are—distorting many of our talent allocation decisions. (175)

Talent suggests thus seeking out ways to take advantage of this inefficiency.

That makes sense. This is where you can buy low and reap the benefits. The whole idea of the book is to expand this ‘buy low’ principle to a wide variety of attributes, including negative ones.

This assumes labor markets follow at least loosely some version of the law of one price when it comes to such attributes, but that the prices involved are often wrong. I think that’s basically correct, and the question is the magnitude of these effects and how best to take advantage.

I can verify that these strategies still work for groups like nerds and introverts. As a member of the Magic: The Gathering professional community, I knew firsthand that it was full of great talent that wasn’t being given good opportunities. These people had fierce desire to win and improve, were willing to work hard, had skills that transfer remarkably well to many domains, knew how to manage risk and loss and were wicked smart. And they’d chosen to pursue Magic rather than things that looked good on a resume. It was obvious they were massively underpriced.

I recruited a bunch of them, and they very much did not disappoint. Then when the time came to actually make a game I got to do it again, putting together a dream team by asking nicely.

They directly challenge the value of the Master’s Degree, which is increasingly becoming required for jobs.

Another way of asking the question: By requiring a master’s degree for those positions, are we potentially overlooking people with more relevant skills and talents who might be better for the job? Credentialism plays an important role in helping us narrow down who is best for the job. But when it misses the mark, it hurts the candidate and employer, limits the economic and social mobility of those who can’t afford an advanced degree, and encourages overinvestment in formal education. (198)

I would double down and apply the same logic to college degrees. Talent doesn’t go into the problem with forcing everyone to go to college, which seems much larger in magnitude than the graduate school problem, or that the law forces employers’ hands here to a large extent – requiring a college (or post-college) degree has a highly disparate impact, yet alternative means of measuring the same attributes signaled by a college degree are illegal because of their disparate impacts.

The Usual Questions

File under the interesting question being the opposite of the one being implied.

There is a whole class at Stanford, for engineers, on how to best prep for interviews and answer interview questions. (627)

Given how important the interview is in terms of one’s success, why is it unusual to have such a class as opposed to it being rare that this is only one class. And where is the class that’s for non-engineers? Where’s the one specifically for founders?

Why isn’t Introduction to Algorithms better described as Introduction to Algorithmic Interview Questions? Why don’t their tests take place on white boards?

Presumably some combination of focus on ‘real learning’ being considered superior and so the departments and students involved can tell a superior story. And because employers would be expected to eventually notice such systems and adjust, although my guess is they mostly wouldn’t.

I am, of course, happy that these classes mostly don’t exist.

The Usual Data

In particular, such technologies would divert your attention toward what the AI can measure well and distract you from the noisier pieces of information. And if many institutions have access to the same AI technologies, as is likely to be the case, it may be precisely the idiosyncratic, noisy pieces of information that become the most important for you. (1,133)

This seems important and reminds me of my concept of Zeroing Out, in the sense that all the AI-style legible measurements that everyone is making can kind of be combined together and then set aside or even count in the opposite direction, because they’re more known than average and thus likely overnoticed, overvalued and overpriced.

You would still then need to find a way to measure the ‘market price’ of the individual, in order to not have to pay attention to this info. In some sense that is extra incentive to let the AI do its job as a black box here, so you can verify and know that market price for context then focus on the unique information available.

Other Interesting Claims

Email Response Time

I’d seen this story before, but it’s worth repeating.

ALTMAN: You know, years ago I wrote a little program to look at this, like how quickly our best founders—the founders that run billion-dollar-plus companies—answer my emails versus our bad founders. I don’t remember the exact data, but it was mind-blowingly different. It was a difference of minutes versus days on average response times.

I am of two minds about this. With one of them, I super strongly agree.

Given you are realizing you have an email at all, and your state is being interrupted, responding immediately is absolutely the right thing to do if you are going to respond at all and you don’t have to (by nature of the email) do something else time intensive first before responding. People who are responsive in this way are much easier to coordinate with, much more likely to be worth engaging with further and generate positive feelings, it’s all great, more people should do this. That is all the more so for important connections like a founder’s connection to Sam Altman.

The flip side of that is the importance of deep work and maintaining state. There are times when it is vital that one not be disturbed, ideally for several hours or sometimes even days, and I don’t check or answer emails on Saturday for this reason. If you respond to emails within a few minutes, it means you’re getting a notification that will potentially interrupt your state or constantly checking email. So you either need to be setting up damn good filtering on when you get such an alert, and/or have someone whose job at any given time is to be the one who gets interrupted and then decides who else to interrupt.

Zoom Versus Zoomless

Why are person-to-person interactions often more informative than Zoom calls?

…

In which ways might a Zoom call be more informative than a person-to-person interaction? (844)

…

So all other things being equal, online trust will be lower. Consequently, edgy interview questions are harder to pull off in these settings. As an interviewer, you are more likely to appear obnoxious, or overly pushy, or simply “off,” and in any case your intentions will be harder to read. So you may be forced to use fewer such questions or to blunt their hard edges. That is one reason online interviews tend to be less informative, and it is also a factor you need to respect when choosing your angle of approach. (871)

…

Very often, online interactions are fine for specific, focused discussions based around a concrete problem or issue or point of contention. They seem to function much less well as a way of generating spontaneous interaction or as a means of eliciting non-goal-specific information, (891)

I don’t think this would have been exactly my hypothesis if I’d been asked, but it seems roughly right to me, and it’s at least close. I definitely have noticed that if you want a sufficiently ‘rich’ interaction, where creativity shines and people truly get to know each other and build trust and all that, it is much better in person.

Zoom still basically always beats the phone for such matters, in my book. Phone calls still have their place, but the bandwidth drop of having zero video is extreme.

For focused interaction with known goals between already-known parties, Zoom is fine and might even be superior to in person, as it lets everyone look at everyone else and have computer access with screen share. It also eliminates all travel and lets arrangements be made on the fly and so on, so there’s that too.

Then there’s the question of… relative status signals?

Another notable feature of online interaction is that it drains away many of the traditional markers of status. (911)

…

One of the hardest mental adjustments for people to make is to realize how much their positive affect relies on their in-person projection of high social status. To give a simple example, you might not be as witty as you think! (944)

I wonder how the authors would compare this to a phone. On the phone, it is not always obvious who is higher status, but often it is at least very clear who is claiming higher status and who is not, through voice tone, who lets who speak and so on. If anything, Zoom perhaps obscures those signals by telling us to focus on the visual cues because we have video, then it takes the visual cues away and we still miss the audio ones.

I wonder if that has something to do with this move he uses:

One of Daniel’s suggestions to avoid Zoom fatigue is simply to shut off the video and just use the audio, like a phone call. Many times the video is misleading anyway, and phone calls can be more intimate than video calls. Furthermore, this decision may prevent you from getting sick of Zoom and other video calls altogether, assuming there will be cases when you need to use them whether you wish to or not. When it comes to talking heads, sometimes it is better just to say no (1,019)

The rest of us do this too, for the simple reason that being ‘on’ and paying attention to where you’re looking and your facial expressions and so on can be exhausting, but the status thing is there too. Most of the time, Daniel will be higher status than those with whom he is stalking. On Zoom he is prevented from enjoying the full benefits thereof, which is exhausting – getting and keeping high status is exhausting and not obviously worthwhile, butit’s good in the moment to be the king and have high status. By turning off the video, Daniel gets to be the king again.

Yet there is still projection of status of a sort, a lot of which is because many people feel compelled to somehow be honest about their assessments of their own status.

There is not necessarily anything wrong with a candidate who has a mediocre background image, but still, it is one piece of information about that person’s self-presentation to the outside world—namely, that you are more likely to succeed with this match if you are hiring for a “substance job” than for a “flair job.” (970)

Another way of saying this is that the entire frame here is buying into the idea that flair is high status and substance is low status.

If you are trying to hire a “cold call” salesperson who will go around knocking on doors, put less weight on the Zoom call than if you are trying to hire a programmer. (1,017)

Thus, you can hire for low status jobs like engineers on Zoom but should be worried about high status jobs like salesman, which is high status because it requires you to project status, and caring about status is the essence of status?

What about the idea that Zoom has the advantage of leaving misleading information out?

Most interviewers consider a person’s gait, microexpressions, and interactions with third parties when judging how trustworthy that person really is, whether those judgments are fair or not. And we don’t know how accurate those signals really are. So if the online interview equalizes those signals across candidates somewhat, you may end up making a better decision. (1,147)

What’s weird is that the book elsewhere encourages you to set up encounters with third parties exactly so you can observe them, yet here it questions whether those signals are accurate and maybe it would be better not to have them. If we are encouraging people to get out of the office and into the world, that’s the opposite of Zoom, creating a richer, more nuanced and detailed physical world to interact with as opposed to a sterile fixed interview room.

My model is that by default these signals are useful as long as you are not overconfident about your ability to read them, and you don’t treat them as more important than they are, but also they could be overpriced. There is value there, and even a relatively imprecise interpretation is going to be better than random – if you properly used your new info you’d be better off. The problem is that if you overvalue the information, and also others are overvaluing the information, and especially if your mistakes are often correlated, you’re going to be overpaying and your decisions get worse, whereas letting others all overpay largely at random while all you notice is the big payment will end up going well for you.

This also has the ‘we don’t know how accurate’ thing, which is odd because the whole book is full of things whose accuracy is unknown. That’s the wheelhouse, so it’s odd to make a fuss about it here.

There’s also the thought that face to face is inherently more resonant because of its inconvenience.

Alternatively, imagine that Oculus technologies develop very rapidly and that your virtual “Paris holiday” is almost as good as the real thing, except cheaper and more convenient. Your virtual-reality interview through Oculus still won’t have the same emotional resonance as would a face-to-face interaction. If nothing else, your virtual interaction won’t have the potential inconveniences of the face-to-face encounter. (1,115)

It’s hard to actually imagine the technology properly, with its full potential attached. You’d need to be feeling the breeze on your shoulder and the heat of the sun, to smell the Paris air, to have the visuals be indistinguishable from normal vision and so on, in order to get to this point. That’s not to say getting there is even implausible, I do expect the tech to get there eventually in worlds where AGI is difficult.

In that world, will the Oculus version be less resonant and memorable because it wasn’t inconvenient? Yes, at least somewhat, that’s definitely a thing, but I think a thing that is generally overvalued, and that would shrink in worlds where Oculus technology improved. We definitely have a bug in our system where having paid additional costs has this effect, but I consider it mostly to be a bug, and one we can correct over time as it makes even less sense and gets less reinforcement.

Who Are You Performing For?

As the candidate tells their story, Daniel continuously asks himself: Whom is this person responding to or used to performing for? Whom do they view as important to impress? Their parents? A particular peer? High school friends? A former boss? This is revealed at moments when they disclose some angles of their past successes and failures rather than others. You might be surprised how often this information comes through in the context of an interview. For instance, a person may refer to college teachers who scorned her or did not appreciate her innovations, or a person may still be wrapped up in how he was viewed as a child by his parents. (493)

I initially focused on the question of who you were trying to impress and the whole thing felt alien. When I reorient to who you are performing for, or even more than that who or what you think will pass judgment, suddenly it makes sense. It is still presented here largely in terms of trying to impress, though:

If they are still trying to impress their high school peers, for instance, they might have focus but they are unlikely to understand the broader picture behind your company or grasp its global ambitions. Most importantly, be alert for the distinction between those who are stuck in their past and those who learned from it but are moving forward and seeking to expand the sphere of people they can impress. (499)

To me, if the judgement you are looking for is whether or not you impress then there is a problem no matter what is supposed to be impressed. ‘Look at how impressive this is’ seems like a wrong goal, whereas it makes sense to think of some algorithm or person or system as being the one that passes judgment. This passage makes me wonder whether that perspective is weirder than I realized.

If someone is stuck in the past, ask which past and in which ways they are stuck. Some pasts are in many ways superior.

Ask yourself what kind of impression this person seems to be trying to make in the interview and how they are trying to do that. How well is that strategy working? This will help you build out a more insightful image of the person you are talking to. (524)

Moral Non-Dilemma

This was the first statement in the book I strongly disagreed with:

Finally, we recognize that spotting talent can be a morally uneasy enterprise. For most jobs or posts, there are many more applicants than winners, and so often you are making only one person feel good about the outcome. (306)

Most of the time, in my experience, firing someone feels terrible and is stressful. Bryan Caplan believes, I think correctly, that this causes a lot of necessary firings not to happen. Yes, occasionally there is someone who richly deserves it, and it is indeed good to be willing to fire people when it is needed. Still, something is very wrong about you if “You’re Fired” is a line in which you take joy.

Hiring people is the opposite. It feels great to hire someone. It even feels great to try to hire someone. Even if they turn you down, as long as it was a decent offer, it usually feels like you made their day. When people offer me jobs, I’ve always taken it as a complement, even if there was never any danger I’d say yes. It’s nice to be asked.

Yes, the people you are not hiring are often going to be unhappy. But not all of them. If you’re ‘doing it right’ then many of those who you do not hire will also be happy with the experience.

If you do a good interview, often the assessment of a poor fit will be mutual. They won’t want the job any more than you want to offer them the job, both of which reinforce each other. At other times, they will be sad they didn’t get the job, but they’ll understand that they were unlikely to be the best candidate – they know how they did. In many cases, they should take not getting a job offer as evidence they should be happy not to have one. It’s not welcome news, but it’s fine. Sometimes the job interview was mostly for practice – either the job isn’t something they wanted and/or it was always out of reach – so there’s no harm in being turned down.

If you do a great interview, they’ll be happy they went through the process because the process was inherently worthwhile. The two great interview processes I went through fall into this category – one was a great use of time even though it didn’t get me a job offer, and the other was a great use of time that also ended in a job offer.

The book notices this phenomenon, that the great interview experiences I’ve had gave me very strong positive impressions of my time spent and also of the companies involved:

Interviews also play a crucial role in recruiting candidates and helping spread a positive impression of you and your company, even in cases where you don’t end up hiring the person. (410)

If the position you are offering is special everything is great. The more you are special and looking for special talent and using techniques like the one in the book, the more likely those who don’t make it are to consider this time well spent. One can think of this as a (stretch) goal, to make everyone happy in hindsight.

What about the ones who are unhappy? I would say that the people chose to come in for the job interview, which means they saw a lot of value in trying. When they get the interview, are most of them not happy? That’s a strong revealed preference. And I’d say that if you had to guess whether the average person interviews too rarely for a new job or too often, I’d be shocked if the answer wasn’t that they apply and interview far too rarely. The upside of a new job is huge, the downside of failing is low (aside from one’s pride) and in failing one often learns a lot or has an interesting experience or story.

In my experience, ‘I applied for the dream job and didn’t make it’ feels better than not applying. This is true even for the time I interviewed for what at the time was a dream job, left thinking I definitely had it, then got informed two weeks later that I didn’t. The finding out sucked, but I’m so happy that I cold emailed them and got my chance.

Even if I did believe I was making the people who applied but got turned down unhappy, I don’t see a moral dilemma or uneasiness here at all. Competitions are great, everyone entered knowing the odds were against them but the payoffs were worth it.

What about jobs that are, shall we say, less exciting? What if it’s a standard entry level position, and you have 100 applicants for one spot, and a lot of them would be fine and none of this is all that fun or interesting? If everyone involved is kind of desperate and worn down and unhappy about having to be there?

I still say it’s rather easy to hold your head high. Everyone is there for a reason. At the end of the day you got someone a job, and they got to stop looking for one. Even if it’s not the best job? That’s still pretty cool.

The Reference

The book isn’t big on references, but does offer some guidelines, advice and questions on what to do with them if one were so inclined.

big fans of interviewing a person’s references, especially for higher-level jobs. Here are some basic tips for doing that effectively: 1. The person you are calling probably wants to get off the phone quickly, help or not harm the candidate, and also not lie too badly. However, they are probably willing to bend the truth. 2. As with the interview itself, the key is to move out of the realm of the bureaucratic and into a more conversational mode. (801)

…

Ask questions about specific quantitative comparisons, because in those instances, even a truth-bending reference is unlikely to lie. (809)

…

If the candidate’s reference list doesn’t contain leaders (or at least links in the chain of direct reports) of the prior organization, that’s a bad sign. 5. Keep the person talking, in part by being interesting yourself. And if you don’t come away from those conversations with a sense of the candidate’s drawbacks, know that you have failed in your investigation so far. (813)

…

Is this person so good that you would happily work for them? Can this person get you where you need to be way faster than any reasonable person could? When this person disagrees with you, do you think it will be as likely you are wrong as they are wrong? (820)

A lot of the problem with references is legal worries or other ass covering. Making sure you are not sued and don’t cause an incident is often the order of the day. Beyond that, yeah, do what you can most of the time, I suppose.

None of these suggestions seem likely to move one into conversation mode. It all seems like a kind of a bidding game. The person does not want to lie too much and wants to not spend too long. You ask if they would be willing to bid a claim of how good the person is, you get how eagerly they do that or don’t do that. You keep going until you sense where they stop and where they start having to hedge or think about it, and you get your information. Your goal is to find the places they won’t bid, which become the weaknesses, and such.

Yet a lot of these questions seem like requests for very high bids. The last set are questions that should not get a genuine yes so often. How many people are there where, if they disagree with me, I think they are as likely right as wrong? Depends on context – some people are likely disagreeing to correct a spelling error or what not, so they’re usually right, but a meaningful disagreement that was sustained upon reflection? I’m not convinced anyone goes on that list.

Who is the Competition?

My general instinct is that if you have a counter-intuitive idea of who you are competing against, either that’s about your internal drive or you’re probably very wrong.

The most famous example of this is when Reed Hastings, the CEO of Netflix, said his competition was ‘sleep, and we’re winning.’

That’s an interesting sideways insight into an aspect of the business. It’s also insane.

Tyler and Daniel also have interesting less insane thoughts on their competition.

At the Mercatus Center, a research center with an emphasis on studying markets, Tyler argues to his co-workers that their competitor is often Google, not some other research center. If people want to know things, they might just go to Google rather than to any particular research center. So you had better place your work well on Google—or, better yet, induce people to go to you before they go to Google.

For Daniel’s Pioneer, the biggest competitor is not another venture capital firm but rather the risk that top prospects might prefer to take a steady salary at a good company and forget about any bigger plans. (612)

I think Tyler is mostly right.

I want Daniel to also be mostly right, but I am skeptical.

The thesis here implies that success as a founder is not so correlated with ability to convince venture capitalists to give you funding, at least in the current round. It’s saying that mostly other VCs are not interested in the same prospects as Pioneer. That would mean that Pioneer was providing lots of value, by identifying worthy founders and giving them a start they wouldn’t otherwise get.

Yet this is highly suspicious. If they are such top prospects why wouldn’t any other VCs be willing to fund them? Why would they be willing to give up? These imply that such would-be founders are not so driven and that they are not so good at fundraising and convincing VCs they are strong candidates. A true top prospect won’t have to and wouldn’t take the safe job offer.

So all this only makes sense if Pioneer is identifying counter-intuitive (to VCs and to themselves) candidates, and that is where they are getting their value. It’s not impossible, but I notice I am skeptical.

The best prospects are, I wager, mostly obviously great prospects, and mostly have no trouble finding interested VCs. The good deals are where VCs are directly competing to get into the round (or at least they are in good times like a few months ago) and they’ve managed to convince founders not to simply jack up the price to market clearing rates when this happens. If no one else is interested, that’s a reason to be worried on multiple levels.

I am doubly skeptical because the book reminds us that a primary motivator of VCs is the fear of missing out (FOMO). You get in trouble when you miss out on the next Facebook or Uber. Yet if someone doesn’t found the company, while you might have missed out on the next big thing, no one will ever know because it doesn’t exist. You won’t get the blame.

More generally, perhaps as a bit of a digression, I am very open to the idea of your competition not being what you think it is. As a game designer, I would often face people whose default assumption was that other games were ‘competition’ and that their success would be bad for Emergents. That the games were fighting over the same players. There isn’t none of this, but I firmly believe that mostly good games are good for games, and good collectable card games are good for collectable card games. They grow the market, they draw people in, they give us all a good name. Without Hearthstone having been created, I think Magic would have fewer players, not more. Whenever I discover a great game, like Storybook Brawl or Slay the Spire or Rogue Legacy, it makes it far more likely to play something else that is similar. That’s the sane version of the Netflix argument.

And just as I was writing this I came across another good example of the ‘you think you’re in competition with others like you but mostly you’re simply trying to be good enough’:

In my experience this is very right except that competition means there are existing pairings that are differentially likely to take the best pairings off the market, although they also induce market entry where people are willing to put themselves out there. Standards you have to meet go up and perhaps the quality of the pool facing you go down as competition increases, but you want to be where there’s more ‘competition’ not less.

I’d like to think of VC that way, where more better VCs means people get more excited to make more good companies (and to fund VC firms) and so more good VCs are good for VCs, and places like Google or Jane Street Capital with their good jobs are the true competition. I am not convinced.

Status-Seeking, Idea-Seeking and IQ

Status-seekers focus on maximizing attention from the perceived elite. Idea-seekers, on the other hand, want to advance knowledge and stimulate curiosity, speaking to the entire room and holding the attention of the group. Intrigue is their reserve currency, and conjectures are often framed as questions, not statements. (97)

Note that both groups here are seekers at all, which is better than not seeking.

The broader claim is that you should seek out idea-seekers rather than status-seekers. For most purposes in life this is clearly true. If my friends were not idea-seekers I would urgently need new and better friends.

Alas, this can be difficult to put this into practice. When one needs to interact with the elite or do business, will one run into more people who are paying attention to ideas or to the elite?

Daniel Gross is a venture capitalist. He wrote a book with Tyler Cowen, and the story of the book shows him being drawn to Tyler’s idea-seeking. But no matter what he writes in a book, when he decides who to fund, does he reward idea-seekers or status-seekers? Which should he do if he wants to make good investments?

A repeated theme of Talent is that IQ is overrated and overpriced. Everyone is looking for and recruiting for IQ, so you’re not going to find neglected talent by looking for IQ.

It is pointed out that IQ doesn’t matter much for earnings.

One classic study of IQ and earnings, by economists Jeffrey S. Zax and Daniel I. Rees and based on data from Wisconsin, finds that, on average, one IQ point predicts less than a 1 percent increase in lifetime earnings. Overall, associating one IQ point with about a 1 percent increase in lifetime earnings is a slightly generous estimate of the correlation, as in some studies the point higher in IQ correlates only with a 0.5 percent increase in lifetime earnings. In sum, more IQ just doesn’t convert into that much more money. (1353)

Or for another result, a study co-authored by Nobel laureate in economics James Heckman found that moving from the 25th to 75th percentile for intelligence was correlated with a 10 to 16 percent boost in earnings. (1358)

Alternatively, you might consider a recent study on Canadian earnings. The math here is a little more complex, but the main result is that a one-standard-deviation increase in cognitive ability corresponds to an earnings boost of 13 to 16 percent. (1363)

Also offered as evidence is that Swedish CEOs, on average, are not so smart.

The main results are that the median or “most typical” small-company CEO is above 66 percent of the Swedish population in cognitive ability, and the median large-company CEO is above 83 percent of the Swedish population in cognitive ability. In both cases those individuals are smarter than average, but they are not in the top 5 percent, much less the top 1 percent. So at least when it comes to CEOs, even very high achievers, at least as measured by their intelligence scores, are not as smart as you might think. (1370)

Higher IQ people have more tendency to idea-seek, and they have more comparative advantage in idea-seeking versus in status-seeking. CEOs are smart, but not so smart. The ‘CEOs of idea-land’ are much smarter in terms of raw IQ. Whey aren’t they rich?

Idea-seeking does not make you rich (in money, anyway) the way status-seeking makes you rich. It does not help you climb the corporate ladder and become CEO. It helps your start-up, but I would be surprised if it helps you as much as status-seeking. Status-seeking behavior helps you raise a round, in part by giving people the impression you will be able to raise another round.

Thus, while Tyler and Daniel both realize while idea-seeking that they should be looking for idea-seekers, they are still looking for attributes that predict outcomes that are correlated with (and caused by) status seeking.

I am reminded of Paul Graham and Y Combinator. Paul Graham often goes into idea-seeking mode, in which he writes excellent essays in which he believes one set of things. He gives advice to people to do the things that would be rewarded if he acted according to the principles in his essays. Then he goes out and (according to all reports I’ve seen, including occasionally his self-reports) does a different set of things in practice, occasionally admitting that he’d do better if he did more of the things he writes on paper. Part of the way he seeks talent is therefore seeing who can tell the difference.

Consider this quote as well:

Daniel believes that if you wish to fund an applicant for venture capital, it is worth asking about the business plan to see how well the basic idea is presented and defended. If they can’t make a case for it to you, they’ll probably have trouble attracting talent to help them. The anti-interview crowd, many of whom are centered in academia, overlooks these obvious truths. (407)

What is a business plan for? In theory the business plan is so investors can decide whether or not they think the plan will work and whether it is something worth investing in. Sometimes that is even true.

More often, both in my experience and by admission of many VCs including Daniel here, they don’t care about the business plan for its own sake. The point of the business plan is to evaluate the founder, and its success is defined by its success at convincing you of its success, as part of systems of the form ‘we should fund this round if and only if we should fund another round.’

If you mess up making the case for the business plan, that’s important because it means you can’t make cases for things. Making cases for things is important. If you can’t make a proper polished business plan that does the things I’m expecting, you won’t be able to properly give people polished things that do what they are expecting to find. Being able and willing to do that is important too. And so on.

Is this a better version of the method, since recruiting talent is a key skill that isn’t self-referential? A little, but mostly I think the explanation here is a rationalization, a way to pretend one is ‘doing it for the talent.’ Yes, you have to sell recruits on your company, but that’s a shoehorn of the concern about whether the founder can be an effective salesperson of the company’s prospects in general. Yes, you can say that in particular recruits have to be sold on the company and its prospects and I can confirm this is to some extent true but mostly the ways in which you do that are very different from the Way of the Business Plan.

I also don’t see how it interacts with academia’s anti-interview claims, since those (as far as I can tell) refer to being anti-interview in situations where the person’s job won’t centrally be about salesmanship or about recruitment.

Avoiding the Usual

good rule of thumb is this: if you found your question in a job interview book or on a website, it is likely you are simply testing the candidate’s preparation level. Again, that is fine up to a point, but don’t confuse it with additional insight. (587)

I expected the next line to be ‘Remember that this is in some sense an interview book.’

Testing someone’s preparation level and approach is indeed somewhat useful as a matching system. If you’re testing the results rather than asking for how they prepared (which as I mentioned earlier feels seriously ‘out of bounds’ to me), it is obviously in bounds but the same dynamic applies – do you want them to be ‘properly’ prepared, or not? If you ask a standard interview question, and get a good standard answer, is that a good or bad response? Which again, to me, goes back to whether you want someone who puts in effort to signal and to cram and to game systems in such ways, or if you want to avoid such folks.

Who Gets Funded?

The question is always what are the big winners? Or rather, what does your algorithm think are the big winners, or plausible enough candidates to get the funding?

This seems telling:

But in venture, one odd emotion Daniel focuses on is fear—specifically those moments when a founder launches their pitch and Daniel begins to feel a subtle fear, brought on by the person’s brazen ambition and drive, that they will do anything to succeed. It’s not that the founder is trying to scare him; rather, they ooze ambition, and Daniel picks up on that. If Daniel feels subtly afraid of them, he will pay attention. (212)

It should be a red flag when you notice that you are rewarding fear. Rewarding super-high levels of ambition and drive makes sense, but I strongly suspect that this is not what inspires the fear and thus is not the thing Daniel’s emotional responses are primarily measuring. It is instead the third thing Daniel mentions – that ‘they will do anything to succeed.’

This sense of ‘they will do anything’ is indeed a central reason I experience fear when encountering some people. And yes, a lot of the people who give off this vibe are in some central sense highly ‘successful’ in business or politics.

The mode of operation is highly useful on multiple levels.

When given the opportunity to betray someone, or their own principles and other preferences, for their own success, they will do so. That directly helps one succeed in the moment one does it.

In a good world, the intention to not keep one’s word, to not care about what is good and what is not so good, to reliably betray one’s friends and allies would it be convenient, to have every intention of selling out one’s cofounders, would be considered a negative when one is deciding whether to be one of the people to be later sold out.

Instead, in these worlds, it is considered a positive.

Others see that you are like this, and think you would be a good investment and a good ally, a good leader, someone to back.

Others see this, and they worry that you will become powerful or successful and they’ll miss out, and/or will use what power you already have against them if they don’t cooperate. Yes, they’ll screw you over anyway, but less, right?

They will constantly seek advantage, and use the threat of what they will do to constantly seek advantage in order to seek further advantage. They will alter the deal, you can’t turn your back on them for a second. But you can prey they don’t alter it any further.

Thus, such people actively work to trigger this fear response. They know that those deciding to whom to award power have learned to reward the invoking of fear, to reward those who show themselves to be this kind of monster and to lack any sort of other principle when it counts.

I have never met a truly successful public-facing politician, but it is clear that many successful ones are very much doing this – to avoid distraction I won’t name any and ask those commenting also not to do so.

I have met a number of billionaires and other rich businesspeople, and most of them show a central pattern of either being this thing, rewarding this thing or both. There are exceptions, in particular based on my experiences Sam Bankman-Fried has the drive and ambition but as far as I can tell is not doing this.

I’ve been an early employee or cofounder in three start-ups where I wasn’t the CEO that got as far as hiring a bunch of people. Two of them were very much this. Both, in the end, betrayed me and betrayed the business in massively destructive ways. The exception was, I strongly suspect, severely punished in fundraising for her not being this, and we had to give up most of the company to keep things going. It got us mostly expropriated.

This is a perfectly understandable algorithm to be using when everyone around you is using it – if your goal is to make money and others are going to reward a founder by funding them later and people will want to help them succeed then you want to be funding them now, even if there’s a good chance they’ll do everything in their power to screw you over later. The big hits will still often be quite big. So I get it.

Even if I was going to be part of another such company some day, the temptation to back another such person is not small – yes the downsides are obvious but at least you’re in the game.

Yet our eyes must be open about what types of people emerge at the end of such a process, and also what we are encouraging our most talented and ambitious people to become.

Invest in IQ? Overrated or Underrated?

An often repeated claim in the book is that IQ is, with notably rare exceptions, overrated and overpriced.

It’s worth saying up front that I am of course heavily biases against this claim.

Might it be that talk of intelligence tends to induce overly hierarchical thinking and ranking? After a long back-and-forth, we’ve concluded that intelligence usually is overrated, most of all by people who are smart. But research also suggests some very important cases where intelligence really matters. (1,165)

…

The most striking results are for the job category of inventors. If you are looking for inventors, IQ is by far the most significant of all the measurable variables we have. Furthermore, at higher levels of measured IQ the probability of becoming an inventor rises all the more. The relevant IQ measure here, by the way, is a variant of the Ravens test, which focuses on visuospatial skills rather than verbal facility. (1,175)

…

all individuals in the highest IQ decile would, statistically speaking, involve a 183 percent rise in the number of inventors compared to the status quo. (1,184)

I am surprised the 183% rise in inventors would be so low and it makes me wonder how many inventions are being misattributed. Invention seems to be super G-loaded to me.

In other places the correlations are not so high between ‘success’ and IQ, such as in the Swedish CEOs, and this is used as the main evidence that IQ is overpriced.

One could easily argue the opposite, if one has other reasons to believe that high IQs are valuable.

If IQ is not commanding a high price in the marketplace in the form of promotions and high pay, but we are confident it is highly valuable, that suggests that IQ is underpriced.

That seems like the more natural interpretation, that something commanding an unexpectedly high price is overpriced, and something commanding an unexpectedly low price is underrated.

Certainly in the jobs I have chosen to do I am confident that IQ is highly valuable and thus underpriced. That doesn’t mean it is the only factor, or that all other things combined don’t matter more – insufficiently high IQ would be a dealbreaker, but so would a bunch of other things – but if I wanted good work I would want to be long IQ.

As a clean example in the other direction, take height, which I was surprised wasn’t mentioned in the book that I can recall. Height is known to be correlated with ‘success’ and compensation. Taller political candidates win more often, taller employees win promotions more often and all that.

From that, we could conclude that being tall represents talent and thus is underpriced. Or we could conclude that there is a bias in favor of tall people, and that while this is worth knowing in contexts where you want to take advantage of that – you might well choose to run a political candidate or fund a startup founder because they are tall knowing others will respond positively to them being tall – you would mostly want to be suspicious that tall people were overpriced, and be excited to hire underpriced shorter people.

I’d be curious to see whether or not this could be used as a proxy for the degree to which merit is winning out at companies over status and superficiality and politics. Thus, the Long Short People ETF would evaluate the C-suite and board of companies, and invest in those that are on average unusually short. Would it have alpha? I’ve never seen the paper studying this.

The question comes down to why the high IQ people aren’t succeeding – a literal case of “If you’re so smart, why aren’t you rich?” How much of that is performance, versus smart people not being inclined to learn and play the social-political games, not being excited to participate in maze behaviors and so on, being different and thus seeming weird and so on.

And how much of it is active political selection against IQ? When selecting a CEO, I can imagine someone thinking that you wouldn’t want someone ‘too smart’ for any of several reasons.

What else do we have?

In that same data set, if you are considering what does explain who becomes a doctor or a lawyer, parental education (and not IQ) is the main explanatory variable, accounting for 39 and 52 percent of those career decisions, respectively. (1198)

That makes sense, especially if you don’t expect doctor or lawyer to be the comparative advantage of high-IQ people and also if you think high-IQ is partly causing parental education. It’s something though.

One carefully done study considers the connection between IQ and lifetime wages at the very top of the IQ distribution, in this case in the top 0.5 percent. The data come from children initially chosen based on a survey in California schools in 1921–22, 856 men and 672 women, and the study traces how much those individuals earned over their lifetimes. In this study, one additional point of IQ correlates with earnings of about 5 percent higher, or in this data set about $184,100 more over the course of a lifetime. In other words, even within the category of people with very good test scores, being “smarter yet” correlates with a noticeable boost in pay. (1211)

…

And here is the critical point: among this high-IQ group, that is a steeper pay/IQ gradient than we find for the population as a whole. That steeper gradient at the top is consistent with our view that intelligence probably matters most for the very high achievers. (1216)

So this is the flip side of the CEO study and other evidence that high IQ doesn’t pay – IQ actually pays quite a lot in this sample. But that’s directly contradicted here is perhaps simply an outlier?

One classic study of IQ and earnings, by economists Jeffrey S. Zax and Daniel I. Rees and based on data from Wisconsin, finds that, on average, one IQ point predicts less than a 1 percent increase in lifetime earnings. Overall, associating one IQ point with about a 1 percent increase in lifetime earnings is a slightly generous estimate of the correlation, as in some studies the point higher in IQ correlates only with a 0.5 percent increase in lifetime earnings. In sum, more IQ just doesn’t convert into that much more money. (1353)

…

Or for another result, a study co-authored by Nobel laureate in economics James Heckman found that moving from the 25th to 75th percentile for intelligence was correlated with a 10 to 16 percent boost in earnings. (1358)

…

Alternatively, you might consider a recent study on Canadian earnings. The math here is a little more complex, but the main result is that a one-standard-deviation increase in cognitive ability corresponds to an earnings boost of 13 to 16 percent. (1363)

I don’t defy the data – I believe that the smaller numbers are probably accurate. But that doesn’t mean accepting that this means the market is pricing this correctly, or that it means that IQ doesn’t much matter.

Instead, it tells me that smart people are not being properly utilized.

It is not obvious to me whether even the expensive value overprices or underprices IQ, as it depends on the circumstances. If you could ‘buy’ more IQ for your employees with higher pay at the super high price of 5% per point, would you? Certainly I agree it matters most at the very top because you can have dramatic oversized value generation, so I would happily pay 10x (or 100x, or if I could afford it 1000x) to get Von Neuman if I could put him to good use.

The more interesting question is if you buy your way from 110 to 120? In most jobs probably not, but in others definitely yes – I’d certainly rather have less engineers that were smarter. Other places my willingness to pay is a lot lower, but still mostly in the 1%+ range.

If I could buy IQ points at 0.5% per point? I’m a buyer for almost any task.

The earnings premium almost certainly overestimates the cost of ‘buying IQ’ because the higher pay comes in large part from what profession and job are chosen. So if the position is fixed, the marginal cost of more IQ is going to be far lower, if you can get it with higher pay. Alas, most of the time your pay scale is fixed and it’s more like you are trading off against other traits, so it’s harder to know if the deal is good.

I’ll go ahead and say that if the ‘market price’ of IQ is only 0.5%-1% per point, then I’m willing to go ahead and say this is crazy, IQ is dramatically underpriced, and you should try to get as long IQ as you can.

(Once again, reminder that I’m probably very biased.)

And outside of the very top of the game of chess, where the champions are indeed very smart, the evidence does not support a very strong link between chess achievement and intelligence (1244)

One hypothesis for this observation is that to become a not-very-top chess player, you have to be smart enough to play chess well, but you also have to not be smart enough to realize you have better life options than chess. Tyler often points out that being a chess player who isn’t at the very top is a poor life choice.

I had that choice to make. As a child, I was winning local tournaments and going to chess camp, once I came in 18th at the Elementary School Nationals, but I clearly had a choice to make. Either I could get super serious about my chess at the expense of other things, or I could accept that my life would consist of something else. I made the right choice, I like to think due in part to my (and my family’s) intelligence.

I do still notice I am surprised the link isn’t stronger, but chess-in-practice includes a lot of other skills that are not so IQ-bound, and also chess below the top mostly doesn’t reward innovation. My experience in Magic (where oh boy was success IQ loaded when I was playing professionally) is that it was the need for continuous innovation, the solving of new situations and finding new ideas, that was the big filter. Lots of players could practice in a static situation and get good, with success a lot more based on discipline and dedication and so on, which matches chess.

On the lower end, of course, for non-expert players, IQ is super correlated with chess ability with or without controlling for time spent on the game, and one does not need a study to know this.

That brings us to this claim.

Consider an analogy from finance. What if someone told you to “buy up the stock of companies full of really smart people”? That is not good advice. You can believe in the importance of smarts all you want, but quality companies also tend to have expensive share prices, and that will be true of these companies full of smart people too, at least insofar as those smarts really matter. Economists have known for a long time there are no extra gains to be had from investing by running after positive qualities but neglecting price. The key instead is to find undervalued companies, and that means companies with hidden virtues. (1385)

I have not seen or searched for a study on this, but I would strongly disagree. If I knew a company was run by lots of unexpectedly smart people, yes I would be more excited to invest. Certainly it is no guarantee of anything and likely increases the chances of a true blow up – see Enron’s story being called The Smartest Guys In The Room. At one point the MetaMed team was called ‘the highest IQ team in the valley’ and it did not get us very far.

It’s still the way I would bet, so why am I challenging the EMH here?

The mechanics here imply to me the EMH will fail here, and also I mostly don’t buy the stronger versions of the EMH in general.

Note that the book uses the framework of EMH-style efficient markets in many places, but fundamentally says that the talent market is deeply inefficient and much or even most great talent isn’t even noticed.

There are, as far as I can tell, two basic ways to argue that there won’t be Alpha in investing in IQ.

The first is the idea that CEOs and other high-up assets of companies are hired in a competitive market, and IQ is being correctly priced, so they’re paying the cost in a combination of more compensation and sacrificing other traits.

I think that is simply wrong. These markets are not remotely efficient, and also one asset of companies is their ability to attract better talent and hire better. I think mostly smarter people at the top aren’t trading off against much of anything except perhaps slightly higher pay and the higher pay is priced in and also not that important.

The second is the idea I think is being claimed above, that the market has assessed the intelligence of these folks, that’s public information, so it’s already priced in or maybe even overpriced.

I think this is a great example of why we shouldn’t expect a lot of public information to be properly priced in. Most people who are actively investing aren’t looking at all at these questions, except maybe in exceptional cases of superstar CEOs. So most of the money moving the market around, and most (or all or almost all) of the mathematical models don’t have this as a variable. The money that does look is likely estimating its value roughly properly, so it won’t fully adjust the price. The way this wouldn’t be true is if the people looking often dramatically overvalue intelligence, but I doubt this.

The natural market dynamic outcome should be for this to be underpriced.

Not investment advice, of course, but I think this is much more of a ‘hidden virtue’ than it is being given credit for. I can get into the opposite mindset with effort, but it seems absurd.

The book holds out the most promise in terms of hiring for IQ when the market forces are not so much in play.

The value of pursuing intelligence also is relatively high when you are the first one on the scene and there is no general competition to hire that same talent. That means intelligence is a better indicator of promise for the very young, for individuals from remote or economically underexplored areas, and for individuals being brought into networks for the very first time. In contrast, intelligence is a worse indicator of hire quality if you are considering a sixty-year-old individual with an established track record. (1255)

This does seem right to me, although I’d expand the mechanism beyond the one stated here. Intelligence is a lot of what lets someone expand and develop, which young people are much more likely to do than older people, and people who haven’t had opportunity are more likely to do than people who have already acclimated, which reinforces this effect. That’s on top of the ‘we already controlled for that and also adjusted the market price’ effect.

The Whole Package

It is easy to think of other examples of the nonlinear importance of intelligence at the very top tiers of achievement. Being a top CEO, baseball pitcher, or Nobel Prize–winning scientist may require a combination of numerous traits, where again, the total is greater than the sum of its parts. We call this the whole package. (1246)

This is a very real phenomenon, where the top achievers seem to be world class at many different things at once, and have lots of exceptional level traits at once as well, in ways that compound, and that they file away big win after big win. I have learned that if someone is exceptional in some ways, one shouldn’t expect a trade-off where they’re underwhelming in other ways. One should instead expect to find more ways they are exceptional, and for them to figure it out whatever it may be.

This happens often enough that ‘this is a remarkable confluence of skills’ seems like the wrong explanation. It’s no accident, the exceptionalism feeds into more exceptionalism (not always, but the ones where it doesn’t we don’t hear from them much after a while). Knowing that one can excel that much and having the drive required combines to keep them pushing forward.

Thus the Whole Package to me is more about package assembly than the package, plus noting whether there is a key flaw or weakness that the process isn’t going to be able to fix.

Not To Start a Political Argument

But within that range of individuals, how much do smarts really matter for the job? Well, the data do not obviously support the view that the smarter presidents were the better presidents. Historians, for instance, typically have regarded Woodrow Wilson, Richard M. Nixon, and Jimmy Carter as three of the intellectually smartest presidents of the twentieth century. We don’t intend to start political arguments here, but the records of these individuals are at best mixed by most reasonable standards. (1409)

I think this is actually a big hint on what is going on. Intellectual smarts have huge advantages, but also increase distance between you and regular people.

It was before my time, but looking backwards Jimmy Carter always felt somewhat off. He was that thing called out of touch, where he didn’t appreciate how people thought or what they would think was important. Carter won the presidency (as I understand it) by being the first person to understand the importance of the primary schedule and focus on early states – being smart let him game the system in a way that later became standard – and then he won a general election under favorable conditions, but he was never a presidential-level politician. He essentially ‘forced his way up’ through intelligence, but didn’t have The Whole Package.

Richard Nixon had a similar thing, as I understand it, where he was fundamentally uncharismatic. No one liked Tricky Dick, he was the opposite of the Teflon that many great politicians seem to enjoy. Yet he still became president, again by using his abstract intelligence to punch above his weight in various ways, and being willing to do what it takes in various ways to make that happen – I think because he thought he could do enough good in power that the ends justified the means. Eventually it all caught up to him, but it’s not like plenty of other Presidents didn’t do pretty terrible stuff, and I suspect a lot of it really was that Nixon was fundamentally unlikable. And a bunch of the rankings I looked at didn’t have him as that smart overall.

Woodrow Wilson has been cancelled for being a racist, and also did various other things like throw the first amendment and other civil liberties onto a dumpster fire, but those were his choices and it’s weird to blame that on his being smart either way. Wilson’s big failure, the one that interacts strongly with his IQ, was the Treaty of Versailles. Wilson’s intelligence allowed him to propose an actually really good peace – the Fourteen Points were not perfect but were damn good, the problem was they were too good and overreached his ability to get his way at the conference, and the final outcome was quite bad when he likely could have done better by being more realistic and less ambitious. But it was very directionally right, and this seems like a case where he simply lacked the skills that mattered most.

Book smarts were not the right kinds of smarts.

Whereas the first hit on Google actually had Bill Clinton as smarter than all three of them. He wasn’t as ‘book smart’ but it was obvious he was super smart, and he often played things very smart, and it paid off in many ways. And when I look at the ‘lower ranks’ of the presidents, I see a lot of names where I’d say their lack of intelligence was a major handicap.

Partisans will disagree about which American presidents were successful, but Franklin Delano Roosevelt and Ronald Reagan are two common picks (obviously the choice depends on the party and ideology of the person being asked). Neither was intellectual in the traditional sense, and while each certainly had plenty of shrewdness, they are not typically considered the smartest of U.S. presidents either, at least not as might be measured by standardized testing. (1419)

These are sufficiently partisan choices that I won’t go into details. I include the quote here for completeness.