Trying to understand my own cognitive edge
6TsviBT
6Darklight
4Morpheus
4interstice
5Wei Dai
5interstice
5Wei Dai
2interstice
3interstice
4Mateusz Bagiński
18Wei Dai
2Mateusz Bagiński
1foodforthought
1foodforthought
1foodforthought
1foodforthought
1marcohmuniz
New Comment
A flip side of this analysis is that the detrimental effects of the aforementioned cognitive distortions might be much higher than is usually supposed or realized, perhaps sometimes causing multi-year/decade delays in important approaches and conclusions, and can't be overcome by others even with significant IQ advantages over me. This may be a crucial strategic consideration, e.g., implying that the effort to reduce x-risks by genetically enhancing human intelligence may be insufficient without other concomitant efforts to reduce such distortions.
This is non-researched speculation, but my guesses would be:
- There are many cognitive dimensions that importantly affect performance in one or another important domain.
- Most of these effects are substantively, though far from completely, fungible with more IQ. In other words, to make up a totally fictional example, you could have someone with IQ 130 and a lot of calm-spacious-attuned-nimble-empathy, who is able to follow along with another person as they struggle though conflicting mental elements, and to help that person untangle themselves by inserting relevant tricks, tools, perceptions, etc., while being sensitive to things that might be upsetting, etc. etc. On the other hand you could have someone with IQ 155, and only somewhat of this calm-spacious-attuned-nimble-empathy; and they basically perform as well as the first therapist at the overall task of helping a client come out of the therapy session with more thriving, on a better cognitive trajectory by their own lights, etc. Even thought Therapist 2 has somewhat less intuitive following-along with the client as does Therapist 1, Therapist 2 is able to make up for that by being able to generate more varied hypotheses quicker and "manually" updating quicker and thinking of better tools and communicating more clearly.
- If you get a lot more people with really high IQs, you also get a bunch more people who are [high on other important cognitive traits, and also high IQ]. (How relevant this argument is, depends on what the numbers look like--how quick is the uptake of, say, reprogenetics technology, how high is the threshold of IQ and of other cognitive dimensions for a given performance, etc.)
Anyway, I definitely would want to genomically vector for these other traits, e.g. wisdom, but it's harder. I do think that argues in favor of working on psychometrics for personality traits as a higher marginal priority than IQ; I think that argument goes through pretty strongly. (Though some people have expressed special worry about personality traits--some of which, e.g. obedience/agreeability, might be targets for oppressive regimes. IDK what to think of that; it feels "far" / unlikely / outlandish, but I don't want to be dismissive and haven't thought about it enough.) But, I think
- The hardest part of any of this is the biotech, not the psychometrics. A crash course to get strong reprogenetics would be really hard and expensive and might not work; a crash course on psychometrics would probably somewhat work, well enough to get significant chunks of benefit. (But, not confident of any of that.)
- Even if you can just vector for IQ, that's still very positive in EV (though my belief here has substantial "instability", i.e. EV>0 has cruxes with high volatility on their probabilities, or something).
I've read a lot of your posts in the past and find you to be reliably insightful. As such, I find it really interesting that with such an IQ in the 99th percentile (or higher), you still initially thought you weren't good enough to do important AI safety work. While I haven't had my IQ properly tested, I did take the LSAT and got a 160 (80th percentile), which is probably around an IQ of merely 120ish. I remember reading a long time ago that the average self-reported IQ of Less Wrongers was 137, which, combined with the extremely rigorous posting style of most people on here, was quite intellectually intimidating (and still is to an extent).
This makes me wonder if there's much point in my own efforts to push the needle on AI safety. I've interviewed in the past with orgs and not gotten in, and occasionally done some simple experiments with local LLMs and activation vectors (I used to work in industry with word vectors so I know a bunch about that space), but actually getting any sort of position or publishable result seems to be very hard. I've had the thought a lot of times that the resources that could go to me would be better off given to a counterfactual more talented researcher/engineer, as it seems like AI safety is more funding constrained than talent constrained.
A flip side of this analysis is that the detrimental effects of the aforementioned cognitive distortions might be much higher than is usually supposed or realized, perhaps sometimes causing multi-year/decade delays in important approaches and conclusions, and can't be overcome by others even with significant IQ advantages over me. This may be a crucial strategic consideration, e.g., implying that the effort to reduce x-risks by genetically enhancing human intelligence may be insufficient without other concomitant efforts to reduce such distortions.
Since I have been working on germline engineering, I have been thinking about the same thing. My intuition is that if I could magically just increase everyone's IQ by 5 points that would result in a marginally saner world. But creating a few babies with ~160+ IQs doesn't seem obviously beneficial. Even if 3/4 get coordination problems etc. what if the fourth one decides to work at OpenAI or Anthropic on capabilities, because working on capabilities is just so much more exciting. If GWAS for personality were working better I'd be more optimistic about selecting for something like capacity to gather wisdom? With editing, you could also go for cognitively enhanced clones of people we consider wise (Main bottleneck with this option would be PR). Problem is that I am going to disagree with people whom we consider wise. Although perhaps we can all agree, we would not like to clone the CEOs of the AGI labs. Perhaps really good education would also help. Not an expert on education.
1Just spitballing here, but one thing that strikes me about a lot of your ideas is that they seem correct but impractical. So for example, yes it seems to be the case that a rational civilization would implement a long pause on AI, in a sense that's even "obvious", but in practice, it's going to be very hard to convince people to do that. Or yes, in theory it might be optimal to calculate the effect of all your decisions on all possible Turing machines according to your mathematical intuition modules, but in practice that's going to be very difficult to implement. Or yes, in theory we can see that money/the state are merely an arbitrary fixed-point in what things people have agreed to consider valuable, but it's gonna be hard to get people to adopt a new such fixed-point.
So the question arises, why are there few people with a similar bent towards such topics? Well, because such speculations are not in practice rewarded, because they are impractical! Of course, you can sometimes get large rewards from being right about one of these, e.g. bitcoin. But it seems like you captured a lot less of the value from that than you could've, such that the amount of resources controlled by people with your cognitive style remains small. Perhaps because getting the rewards from one of those large sparse payoffs still depends on a lot of practical details and luck.
Yet another way of formulating this idea might be that the theoretically optimal inference algorithm is a simplicity prior, but in practice that's impossible to implement, so people instead use approximations. In reality most problems we encounter have a lot of hard-to-compress detail, but there is a correspondingly large amount of "data" available(learned through other people/culture perhaps) so the optimal approximation ends up being something like interpolation from a large database of examples. But that ends up performing poorly on problems where there the amount of data is relatively sparse(but for which there may be large payoffs).
So this then raises the question of how cognitive styles that depend on large but sparse rewards can defend/justify themselves to styles that benefit from many small consistent rewards.
In terms of financial rewards, those also way exceeded my early expectations. Independent of Bitcoin, I was able to retire (stop working a paying job) comfortably in my late 20s from a software product I developed earlier, and during the last 5 years I multiplied my NW by 8x, by putting 10% of NW into S&P500 puts shortly before the COVID market crash, which doubled my NW, and then the rest from trading full time for 2-3 years (starting as a complete beginner who had only invested in index funds before, and putting my intellectual interests almost completely aside for that time) during the post-COVID market craziness, and from the general subsequent market rise.
So this then raises the question of how cognitive styles that depend on large but sparse rewards can defend/justify themselves to styles that benefit from many small consistent rewards.
I think the results (both intellectual and financial) speak for themselves?
I think the results (both intellectual and financial) speak for themselves?
I mean, it still seems to be the case that people with a less philosophical style control vastly more resources/influence, and are currently using them to take what are from your perspective insanely reckless gambles on AGI, no? I'm saying from an ecological perspective this is due to those cognitive styles being more useful/selected-for[well, or maybe they're just "easier" to come up with and not strongly selected against] on more common "mundane" problems where less philosophical reflection is needed(abstractly, because those problems have more relevant "training data" available)
I think in terms of wealth, it's just because there's a lot more of them to start with (so you end up with a much larger number of outliers with high wealth who could invest into AGI), but on a per person basis, it seems hard to argue that my cognitive style isn't financially very rewarding.
But in terms of gaining influence, it does seem that my style is terrible (i.e., being largely ignored for up to a decade or two on any given topic). Seems like an important point to bring up and consider, so thanks.
I think in terms of wealth, it's just because there's a lot more of them to start with
Ah yes, but why is that the case in the first place? Surely it's due to the evolutionary processes that make some cognitive styles more widespread than others. But yeah I think it's also plausible that there is net selection pressure for this and there just hasn't been enough time(probably the selection processes are changing a lot due to technological progress as well...)
Another thing which I wasn't sure how to fit in with the above. I framed the neglect of your "philosophizing" cognitive style as being an error on the world's part, but in some cases I think this style might ultimately be worse at getting things done, even on its own terms.
Like with UDT or metaphilosophy my reaction is "yes we have now reached a logical terminus of the philosophizing process, it's not clear how to make further progress, so we should go back and engage with the details of the world in the hope that some of them illuminate our philosophical questions". As a historical example, consider that probability theory and computability theory arose from practical engagement with games of chance and calculations, but they seem to be pretty relevant to philosophical questions(well, to certain schools of thought anyway). More progress was perhaps made in this way than could've been made by people just trying to do philosophy on its own.
the main thing that appears to have happened is that I had exceptional intuitions about what problems/fields/approaches were important and promising
I'd like to double-click on your exceptional intuitions, though I don't know what questions would be most revealing if answered. Maybe: could you elaborate on what you saw that others didn't see and that made you propose b-money, UDT, the need for an AI pause/slowdown, etc?
E.g., what's your guess re what Eliezer was missing (in his intuitions?) in that he came up with TDT but not UDT? Follow-up: Do you remember what the trace was that led you from TDT to UDT? (If you don't, what's your best guess what it was?)
b-money: I guess most people working on crypto-based payments were trying to integrate with the traditional banking system, and didn't have the insight/intuition that money is just a way for everyone to "keep tabs" of how much society as a whole owes to each person (e.g. for previous services rendered), and therefore a new form of money (i.e. not fiat or commodity) could be created and implemented as a public/distributed database or ledger.
UDT: I initially became interested in decision theory for a very different reason than Eliezer. I was trying to solve anthropic reasoning, and tried a lot of different ideas but couldn't find one that was satisfactory. Eventually I decided to look into decision theory (as the "source" of probability theory) and had the insight/intuition that if the decision theory didn't do any updating then we could sidestep the entire problem of anthropic reasoning. Hal Finney was the only one to seriously try to understand this idea, but couldn't or didn't appreciate it (in fairness my proto-UDT was way more complicated than EDT, CDT, or the later UDT, because I noticed that it would cooperate with its twin in one-shot PD, and added complications to make it defect instead, not questioning the conventional wisdom that that's what's rational).
Eventually I got the idea/hint from Eliezer that it can be rational to cooperate in one-shot PD, and also realized my old idea seem to fit well with what Nesov was discussing (counterfactual mugging), and this caused me to search for a formulation that was simple/elegant and could solve all of the problems known at the time, which became known as UDT.
I think Eliezer was also interested in anthropic reasoning, so I think he was missing my move to look into decision theory for inspiration/understanding and then making the radical call that maybe anthropic reasoning is unsolvable as posed, and should be side-stepped via a change to decision theory.
need for an AI pause/slowdown: I think I found Eliezer convincing when he started talking about the difficulty of making AI Friendly and why others likely wouldn't try hard enough to succeed, and just found it implausible that he could with a small team win a race against the entire world who was spending much less effort/resources on trying to make their AIs Friendly. Plus I had my own worries early on that we needed to either solve all the important philosophical problems before building AGI/ASI, or figure out how to make sure the AI itself is philosophically competent, and both are unlikely to happen without a pause/slowdown (partly because nobody else seemed to share this concern or talked about it).
1
1Thanks!
The entire thing seems to have a very https://www.lesswrong.com/posts/bhLxWTkRc8GXunFcB/what-are-you-tracking-in-your-head vibes, though that's admittedly not very specific.
What stands out to me in the b-money case is that you kept tabs on "what the thing is for"/"the actual function of the thing"/"what role it is serving in the economy", which helped you figure out how to make a significant improvement.
Very speculatively, maybe something similar was going on in the UDT case? If the ideal platonic theory of decision-making "should" tell you and your alt-timeline-selves how to act in a way that coheres (~adds up to something coherent?) across the multiverse or whatever, then it's possible that having anthropics as the initial motivation helped.
It seems impractical to recommend that someone spend a few years in cryptography
That's too literal. How about: always try to play chess/tennis with someone better than you, when you can. Get your early training -- when it's your full time job to study -- in the field you struggle to keep up in, not the one you clearly dominate in. Be the small fish in a big pond. You learn most from people who are better than you. You will learn epistemic humility and self-skepticism. You hopefully learn not to stake your ego on your genius. you learn how to make yourself useful to people you want to learn from.
(But don't pick the field you can't keep up in at all; you won't learn anything when you are utterly lost; you'll likely be demoralized by the attempt, even if everyone is kind; and you risk being a true burden, toward whom kindness won't always be extended).
Are there others who could make a similar claim of having exceptionally good and hard to explain intuitions
Yes. PM me.
Not only is it rare but there seems to be a surprisingly large gap between my intuitions and the next closest person's.
This is a natural consequence of having many interests and switching fields many times; you bring a unique set of factual knowledge, concepts, heuristics, theoretical frameworks or insights to the problem at hand. You pay the cost of not likely being the most-expert or most-competent in most or all the fields you care about, but you reap the benefit of having uncommon insights, which tend to be orthogonal to the rest of the field (and perhaps for this reason incomprehensible to them).
Weak analogy: If reality could be mapped to one orthogonal basis set, it's as if each specialization only has a few basis functions; the good fields have at least one basis function that explains a large fraction of the power in at least the signal set [dimension of reality] they study. But other parts of reality that don't happen to project onto their set are entirely invisible to them. Interdisciplinary field-switchers get to accumulate basis functions as they go; so they always have some at their disposal that are orthogonal to the set in use within the current discipline. The more disparate two disciplines, the less overlap in their endogenous basis sets (but perhaps that is a tautological statement, a definition of 'disparate').
It seems hard to explain using anything we know from cognitive science. Standard explanations for good intuitions include that they're distilled from extensive prior experience or reasoning, but I moved from field to field and as a result was often a newcomer.
This could be exactly why you were able to have exceptionally good intuitions, see https://www.lesswrong.com/posts/Afdohjyt6gESu4ANf/most-people-start-with-the-same-few-bad-ideas?commentId=JBHcLYgk77c7vxeGk
I think outside of grit and intelligence you also display high openness to experience which has been shown to increase or at least highly correlate with research productivity.