the main thing that appears to have happened is that I had exceptional intuitions about what problems/fields/approaches were important and promising
I'd like to double-click on your exceptional intuitions, though I don't know what questions would be most revealing if answered. Maybe: could you elaborate on what you saw that others didn't see and that made you propose b-money, UDT, the need for an AI pause/slowdown, etc?
E.g., what's your guess re what Eliezer was missing (in his intuitions?) in that he came up with TDT but not UDT? Follow-up: Do you remember what the trace was that led you from TDT to UDT? (If you don't, what's your best guess what it was?)
b-money: I guess most people working on crypto-based payments were trying to integrate with the traditional banking system, and didn't have the insight/intuition that money is just a way for everyone to "keep tabs" of how much society as a whole owes to each person (e.g. for previous services rendered), and therefore a new form of money (i.e. not fiat or commodity) could be created and implemented as a public/distributed database or ledger.
UDT: I initially became interested in decision theory for a very different reason than Eliezer. I was trying to solve anthropic reasoning, and tried a lot of different ideas but couldn't find one that was satisfactory. Eventually I decided to look into decision theory (as the "source" of probability theory) and had the insight/intuition that if the decision theory didn't do any updating then we could sidestep the entire problem of anthropic reasoning. Hal Finney was the only one to seriously try to understand this idea, but couldn't or didn't appreciate it (in fairness my proto-UDT was way more complicated than EDT, CDT, or the later UDT, because I noticed that it would cooperate with its twin in one-shot PD, and added complications to make it defect instead, not questioning the conventional wisdom that that's what's rational).
Eventually I got the idea/hint from Eliezer that it can be rational to cooperate in one-shot PD, and also realized my old idea seem to fit well with what Nesov was discussing (counterfactual mugging), and this caused me to search for a formulation that was simple/elegant and could solve all of the problems known at the time, which became known as UDT.
I think Eliezer was also interested in anthropic reasoning, so I think he was missing my move to look into decision theory for inspiration/understanding and then making the radical call that maybe anthropic reasoning is unsolvable as posed, and should be side-stepped via a change to decision theory.
need for an AI pause/slowdown: I think I found Eliezer convincing when he started talking about the difficulty of making AI Friendly and why others likely wouldn't try hard enough to succeed, and just found it implausible that he could with a small team win a race against the entire world who was spending much less effort/resources on trying to make their AIs Friendly. Plus I had my own worries early on that we needed to either solve all the important philosophical problems before building AGI/ASI, or figure out how to make sure the AI itself is philosophically competent, and both are unlikely to happen without a pause/slowdown (partly because nobody else seemed to share this concern or talked about it).
Thanks!
The entire thing seems to have a very https://www.lesswrong.com/posts/bhLxWTkRc8GXunFcB/what-are-you-tracking-in-your-head vibes, though that's admittedly not very specific.
What stands out to me in the b-money case is that you kept tabs on "what the thing is for"/"the actual function of the thing"/"what role it is serving in the economy", which helped you figure out how to make a significant improvement.
Very speculatively, maybe something similar was going on in the UDT case? If the ideal platonic theory of decision-making "should" tell you and your alt-timeline-selves how to act in a way that coheres (~adds up to something coherent?) across the multiverse or whatever, then it's possible that having anthropics as the initial motivation helped.
I applaud Eliezer for trying to make himself redundant, and think it's something every intellectually successful person should spend some time and effort on. I've been trying to understand my own "edge" or "moat", or cognitive traits that are responsible for whatever success I've had, in the hope of finding a way to reproduce it in others, but I'm having trouble understanding a part of it, and try to describe my puzzle here. For context, here's an earlier EAF comment explaining my history/background and what I do understand about how my cognition differs from others.[1]
In terms of raw intelligence, I think I'm smart but not world-class. My SAT was only 1440, 99th percentile at the time, or equivalent to about 135 IQ. (Intuitively this may be an underestimate and I'm probably closer to 99.9th percentile in IQ.) I remember struggling to learn the GNFS factoring algorithm, and then meeting another intern at a conference who had not only mastered it in the same 3 months that I had, but was presenting an improvement on the SOTA. (It generally seemed like cryptography research was full of people much smarter than myself.) I also considered myself lazy or not particularly hardworking compared to many of my peers, so didn't have especially high expectations for myself.
(An illustration of this is that when I, as a freshman CS major, became worried about eventual AI takeover after reading Vernor Vinge's A Fire Upon the Deep, I thought I wasn't smart or conscientious enough to contribute to a core field like AI safety, i.e., that there would eventually be plenty of people much smarter and harder working than me contributing to it. As a result I didn't even take any AI courses, but instead decided to focus my education and career on applied cryptography, as a way to contribute to reducing AI x-risk from the periphery, by increasing overall network security.)
It seems safe to say that I exceeded[2] my own expectations, and looking back, the main thing that appears to have happened is that I had exceptional intuitions about what problems/fields/approaches were important and promising, and then used my high but not world-class intelligence to pick off some low hanging fruits or stake out some positions destined to become popular later. Others ignored them for a long time, even after I published my ideas. In several cases they were ignored for so long that I had given up hope of getting significant validation or positive feedback for them, until they were eventually rediscovered and/or made popular by others.
The questions that currently puzzle me:
It occurs to me as I'm writing this, that maybe what I have (or had) is not exceptionally good intuitions, but good judgment that comes from a relatively high baseline reasoning ability and knowledge base, buffed by a lack of the usual cognitive distortions, specifically overconfidence (which leads to a tendency to latch onto the first seemingly good idea that one thinks of, instead of being self-skeptical and trying hard to find flaws in one's own ideas) and institutional pressures/incentives that result from one's employment.
My self-skepticism probably came from the early career in cryptography, where often the only way to minimize risk of public humiliation is to scrupulously examine one's own proposals for potential flaws, and overconfidence is quickly punished. Security proofs are often not possible or themselves potentially flawed, e.g. due to use of wrong assumptions or models. Also, the flaws are often extremely subtle and difficult to find, but hard to deny once pointed out, further incentivizing self-skepticism and scrutiny.
My laziness may have paradoxically helped, by causing me to avoid joining the usual institutions that someone with my interests might have joined (e.g. academia and other research institutes) to instead pursue a "pressure-free" life of thinking about whatever I want to think about, saying whatever I want to say.
(This life probably has its own cognitive distortions, e.g., related to status games that people play in online discussion forums, but perhaps they're different enough from the usual cognitive distortions that I was able to see a bunch of blind spots that other people couldn't see.)
Re-reading my 2-year-old EAF comment (copied as footnote [1] below), I had already mentioned my self-skepticism and financial/organizational independence as factors for my intellectual success, but apparently still felt like there was a puzzle to be explained. Perhaps the main realization/insight of this post is that the effect size from a combination of these 2 factors could be large enough to explain/constitute all or most of my "edge", and there may not be a further mystery of "exceptionally good intuitions" that needs to be explained.
I'll probably keep thinking about this topic, and welcome any thoughts or perspectives from others. It's also not quite clear what practical advice to draw from this, assuming my "plausible answer" is true. It seems impractical to recommend that someone spend a few years in cryptography, but I'm not sure if anything less onerous than that would have a similar effect, nor can I say with any confidence that even such experience will produce the same kind of general and deep-seated self-skepticism that it apparently did in me. Being financially/organizationally independent also seems impractical or too costly for most people to seriously pursue. I would welcome any suggestions on this front (of practical advice) as well.
One implication that occurs to me is that if the advantages of these cognitive traits accumulate multiplicatively (as they seem to), then the cost of gaining the last piece of the puzzle might be well worth paying for someone who already has the others. E.g., if someone already has a >99th percentile IQ, wide-ranging intellectual background and interests, and one of self-skepticism and independence, then the marginal value of gaining the other trait might be very high and hence worth its cost.
A flip side of this analysis is that the detrimental effects of the aforementioned cognitive distortions might be much higher than is usually supposed or realized, perhaps sometimes causing multi-year/decade delays in important approaches and conclusions, and can't be overcome by others even with significant IQ advantages over me. This may be a crucial strategic consideration, e.g., implying that the effort to reduce x-risks by genetically enhancing human intelligence may be insufficient without other concomitant efforts to reduce such distortions.
Copying here for completeness/archival purposes:
I thought about this and wrote down some life events/decisions that probably contributed to becoming who I am today.
A lot of these can't really be imitated by others (e.g., I can't recommend people avoid making friends in order to have more free time for intellectual interests). But here are some practical advice I can think of:
ETA: Oh, here's a recent LW post where I talked about how I arrived at my current set of research interests, which may also be of interest to you.
Copying my main accomplishments here:
With the notable exceptions of Nick Szabo who invented his BitGold at nearly the same time as b-money, Cypherpunks who thought b-money was interesting/promising but didn't spend much effort developing it further, and Hal Finney who perhaps paid the most attention to my ideas pre-LW, including by developing RPOW, trying to understand my early decision theory ideas, and writing up UDASSA in a publicly presentable form.