Rationality Research Report: Towards 10x OODA Looping?
20moridinamael
7Raemon
1azergante
14Elizabeth
11DaystarEld
10romeostevensit
2Raemon
6romeostevensit
9Elizabeth
4Raemon
4Steven Byrnes
4Kaj_Sotala
2Raemon
2Kaj_Sotala
2Raemon
4Raemon
4Steven Byrnes
41a3orn
6Raemon
4Raemon
21a3orn
2Raemon
3junk heap homotopy
1Review Bot
1mike_hawke
1mike_hawke
New Comment
One way of viewing planning is as an outer-loop on decision theory.
My approach to the general problem of planning skills was to start with decision theory and build up. In my Guild of the Rose Decision Theory courses was to spend time focusing on slowly building the most fundamental skills of decision theory. This included practicing manipulation of probabilities and utilities via decision trees, and practicing all these steps in a variety of both real and synthetic scenarios, to build an intuition regarding the nuances of how to set up decision problems on paper. The ultimate goal was to get the practitioners to the point where they usually don't need to draw up a decision tree on paper, but rather to leverage those intuitions to quickly solve decision problems mentally, and/or recognize when a decision problem is actually tricky enough to merit breaking out the spreadsheet or Guesstimate project.
In my experience, even long-time rationalists are so incredibly bad at basic decision theory that trying to skip the step of learning to correctly set up a basic decision tree might actually be counterproductive. So my inclination is to focus on really mastering this art before attempting planning.
Another way of viewing planning is that planning is search.
For computationally bounded agents like us, search involves a natural tradeoff of breadth versus depth. Breadth is essentially idea generation, depth is idea selection and refinement. The tricky think about planning, in general, is that if 100x solutions exist, then those solutions are going to be found by spending the majority of the time on breadth-search, i.e. blue sky brainstorming for ways that the plan could look wildly different from the default approach, but that most situations don't admit 100x plans. Most things in life, especially in our technological civilization, are already sort of optimized, because there is some existing refined solution that has already accommodated the relevant tradeoffs. I could get to work faster if I flew there in a helicopter, but considering in costs, the Pareto optimum is still driving my car on the freeway. Most things look like this. Well-considered Pareto solutions to real-world problems tend to look boring!
Therefor, if you spend a lot of time looking for 100x solutions, you will waste a lot of time, because these solutions usually won't exist. Then, after failing to find a truly galaxy-brain solution, you will spend some amount of time refining the probably-already-obvious plan, realize that there are a lot of unknown-unknowns, and that the best way to get clarity on these is to just start working. Then you will realize that you would have been better off if you had just started working immediately and not bothered with "planning" at all, and you will either be Enlightened or depressed.
It gives me no pleasure to say this! Ten years ago I was all fired up on the idea that rationalists would Win and take over the world by finding these clever HPJEV-esque lateral thinking solutions. I have since realized that one creative rationalist is usually no match for tens of thousands of smart people exploring the manifold through natural breadth-first and then refining on the best solutions organically.
I am not actually completely blackpilled on the idea of scenario planning. Clearly there are situations for which scenario planning is appropriate. Massive capital allocations and long-term research programs might be two good examples. Even for these types of problems, it's worth remembering that the manifold probably only admits to marginal optimizations, not 100x optimizations, so you shouldn't spend too much time looking for them.
Both of these thoughts are pretty interesting, thanks.
I'd be interested in hearing a bunch more detail about how you trained decision theory and how that went. (naively this sounds like overkill to me, or "not intervening at the best level", but I'm quite interested in what sort of exercises you did and how people responded to them)
re: "how useful is planning", I do think this is specifically useful if you have deep, ambitious goals, without well established practices. (i.e. Rationality !== Winning in General).
Most things in life, especially in our technological civilization, are already sort of optimized
I want to nuance that point: in my experience, as soon as I stray one iota from the one size fits all (or no one) products provided by the mass market, things either suck, don't exist or are 10x the price.
Even the so-called optimized path sucks sometimes, for reasons described in Inadequate Equilibria. A tech example of that is Wirth's law:
Wirth's law is an adage on computer performance which states that software is getting slower more rapidly than hardware is becoming faster.
There is a lot of software that is literally hundreds of times slower than it could be, because for example it runs on top of bloated frameworks that run on top of toy languages designed in 10 days (cough Javascript cough) that run on top of virtual machines, that run on top of OSes and use protocols designed for a bygone era.
I think that as civilization leverages economies of scale more and more, the gap between the quality/price ratio of custom goods and mass-produced goods increases, which leads to the disappearance of artisans, which means that as time goes on civilization is optimizing a narrower and narrower number of goods, and that sucks when you want a product with specific features that are actually useful for you.
Back to your point, I would say that civilization is often not optimized: we can literally do a hundred times better, but the issue is that often there is no clear path from "creating a better (or a custom) product" to "earning enough money to live".
Lord grant me the strength to persevere when things are hard the courage to quit when things are impossible and the wisdom to know the difference.
I'm running a small rationality dojo to try to approach this issue from the rat-for-rat-sake direction in a few weeks, trying to incorporate the things I learned from my Seasons of Growth, my Executive Function research, and stuff like Logan's Naturalism sequence (not to mention years of teaching at rat camps and workshops). I plan to do a writeup after, but would also love to chat sometime about this, either before or after.
One of the things that helped a lot with the predictions part was reading Judea Pearl's Heuristics. It seemed to make me better at noticing that a big part of my problem solving was split into two things: my representation of the problem space, and then my traversal of that space. I would notice more readily when I had stuck myself with an intractably sized space for the traversal speed available, and conclude that I needed to switch to trying to find a different representation that was tractable. Others might get very different insights out of the book, the search-inference framework is pretty flexible (also covered in Baron's Thinking and Deciding).
1The cleanest example is during Ravens testing, noticing that checking a particular set of hypotheses one by one is taking too long. Zooming out and seeing them as a class of hypotheses, what they have in common, and then asking what else is possible. If the different moving parts of the puzzle are slot machines, then it's an explore exploit problem.
But it's somewhat broader. I think "could I 10x my plans?" can be useful frame even if you feel averse to "what's literally the most important problem I could focus on?".
Even more baby-step version: come up with two plans instead of one and choose between them. The second plan probably won't be 10x better, but count of two (2) is easier than 10x, and builds the necessary muscles of looking for alternatives and choosing.
Yeah something like this has already come up as a necessary stepping stone.
See also: ‘have a plan, at all’
something that's really weirded me out about the literature on IQ, transfer learning, etc, is that... it seems like it's just really hard to transfer learn. We've basically failed to increase g, and the "transfer learning demonstrations" I've heard of seemed pretty weaksauce.
You might be referring to the skeptical take on transfer learning, summarized as follows in Surfaces and Essences by Hofstadter & Sander:
Experimental studies have indeed demonstrated that subjects who are shown a source situation and who are then given a target situation are usually unable to see any connection between the two unless they share surface-level traits. Furthermore, in such experiments, when two situations have a superficial resemblance, then the second one invariably brings the first one to mind, no matter whether it is appropriate or not (that is, irrespective of whether there are deeper reasons to connect the two cases). For instance, if subjects first tackle an arithmetic problem concerning items bought in a store, then any other problem concerning purchases will instantly remind them of the initial problem. But if the theme of the first problem is experimentally manipulated say it becomes a visit to a doctor’s office instead of a store — then the participants will almost surely see no link between the two stories, even if the solution method for the first problem applies perfectly to the second problem.
But then the authors argue that this skeptical take is misleading:
Unfortunately, the source–target [experimental] paradigm [in the studies above] has a serious defect that undermines the generality of the conclusions that experiments based upon it produce. This defect stems from the fact that the knowledge acquired about the source situation during the twenty minutes or so of a typical experiment is perforce very limited — often consisting merely in the application of a completely unfamiliar formula to a word problem. By contrast, when in real life we are faced with a new situation and have to decide what to do, the source situations we retrieve spontaneously and effortlessly from our memories are, in general, extremely familiar. We all depend implicitly on knowledge deeply rooted in our experiences over a lifetime, and this knowledge, which has been confirmed and reconfirmed over and over again, has also been generalized over time, allowing it to be carried over fluidly to all sorts of new situations. It is very rare that, in real life, we rely on an analogy to a situation with which we are barely familiar at all. To put it more colorfully, when it comes to understanding novel situations, we reach out to our family and our friends rather than to the first random passerby. But in the source–target paradigm, experimental subjects are required to reach out to a random passerby—namely, the one that was imposed on them as a source situation by the experimenter.
And so, what do the results obtained in the framework of this paradigm really demonstrate? What they show is that when people learn something superficially, they wind up making superficial analogies to it.
To rephrase: The problem is that, in the experimental protocol, the subjects only ever wind up with a crappy surface-level understanding of the source situation, not a deep mental model of the source situation reflective of true familiarity / expertise. When people do have real comfort and familiarity with the source situation, then they find deep structural analogies all over the place.
For example (these are my examples), if you talk to an economist about some weird situation, they will easily notice that there’s a supply-and-demand way to look at it, and ditto gains-from-trade and so on. And physicists will analogize random things to superpositions and fourier-space and so on, etc. Of course, the main thing that everyone is an “expert” in is “intuitive everyday life stuff”, and hence our thinking and speech is full of constant non-surface-level analogies to traveling, seasons, ownership, arguments, etc. etc.
I’m not sure if this is relevant to what you were saying, just thought I’d share.
1I've seen claims that the failure of transfer also goes in the direction of people with extensive practical experience and familiarity with math failing to apply it in a more formal context. From p. 64-67 of Cognition in Practice:
Like the AMP, the Industrial Literacy Project began with intensive observational work in everyday settings. From these observations (e.g. of preloaders assembling orders in the icebox warehouse) hypotheses were developed about everyday math procedures, for example, how preloaders did the arithmetic involved in figuring out when to assemble whole or partial cases, and when to take a few cartons out of a case or add them in, in order to efficiently gather together the products specified in an order. Dairy preloaders, bookkeepers and a group of junior high school students took part in simulated case loading experiments. Since standardized test data were available from the school records of the students, it was possible to infer from their performance roughly the grade-equivalent of the problems. Comparisons were made of both the performances of the various experimental groups and the procedures employed for arriving at problem solutions.
A second study was carried out by cognitive psychologists investigating arithmetic practices among children selling produce in a market in Brazil (Carraher et al. 1982; 1983; Carraher and Schliemann 1982). They worked with four boys and a girl, from impoverished families, between 9 and 15 years of age, third to eighth grade in school. The researchers approached the vendors in the marketplace as customers, putting the children through their arithmetic paces in the course of buying bananas, oranges and other produce.
M. is a coconut vendor, 12 years old, in the third grade. The interviewer is referred to as 'customer.'
Customer: How much is one coconut?
M: 35.
Customer: I'd like ten. How much is that?
M: (Pause.) Three will be 105; with three more, that will be 210. (Pause) I need four more. That is ... (pause) 315 ... I think it is 350.
The problem can be mathematically represented in several ways. 35 x 10 is a good representation of the question posed by the interviewer. The subject's answer is better represented by 105 + 105 + 105 +35, which implies that 35 x 10 was solved by the subject as (3 x 35) + 105 + 105 +35 ... M. proved to be competent in finding out how much 35 x 10 is, even though he used a routine not taught in 3rd grade, since in Brazil3rd graders learn to multiply any number by ten simply by placing a zero to the right of that number. (Carraher, Carraher and Scldiemam. 1983: 8-9)
The conversation with each child was taped. The transcripts were analyzed as a basis for ascertaining what problems should appear on individually constructed paper and pencil arithmetic tests. Each test included all and only the problems the child attem pted to solve in the market. The formal test was given about one week after the informal encounter in the market.
Herndon, a teacher who has written eloquently about American schooling, described (1971) his experiences teaching a junior high class whose students had failed in mainstream classrooms. He discovered that one of them had a well-paid, regular job scoring for a bowling league. The work demanded fast, accurate, complicated arithmetic. Further, all of his students engaged in relatively extensive arithmetic activities while shopping or in after-schooljobs. He tried to build a bridge between their practice of arithmetic outside the classroom and school arithmetic lessons by creating "bowling score problems," "shopping problems," and "paper route problems." The attempt was a failure, the league scorer unable to solve even a simple bowling problem in the school setting. Herndon provides a vivid picture of the discontinuity, beginning with the task in the bowling alley:
... eight bowling scores at once. Adding quickly, not making any mistakes (for no one was going to put up with errors), following the rather complicated process of scoring in the game of bowling. Get a spare, score ten plus whatever you get on the next ball, score a strike, then ten plus whatever you get on the next two balls; imagine the man gets three strikes in a row and two spares and you are the scorer, plus you are dealing with seven other guys all striking or sparing or neither one. I figured I had this particular dumb kid now. Back in eighth period I lectured him on how smart he was to be a league scorer in bowling. I pried admissions from the other boys, about how they had paper routes and made change. I made the girls confess that when they went to buy stuff they didn't have any difficulty deciding if those shoes cost $10.95 or whether it meant $109.50 or whether it meant $1.09 or how much change they'd get back from a twenty. Naturally I then handed out bowling-score problems, and naturally everyone could choose which ones they wanted to solve, and naturally the result was that all the dumb kids immediately rushed me yelling, "Is this right? I don't know how to do it! What's the answer? This ain't right, is it?" and "What's my grade?" The girls who bought shoes for $10.95 with a $20 bill came up with $400.15 for change and wanted to know if that was right? The brilliant league scorer couldn't decide whether two strikes and a third frame of eight amounted to eighteen or twenty-eight or whether it was one hundred eight and a half. (Herndon 1971: 94-95)
People's bowling scores, sales of coconuts, dairy orders and best buys in the supermarket were correct remarkably often; the performance of AMP participants in the market and simulation experiment has already been noted. Scribner comments that the dairy preloaders made virtually no errors in a simulation of their customary task, nor did dairy truck drivers make errors on simulated pricing of delivery tickets (Scribner and Fahrmeier 1982: to, 18). In the market in Recife, the vendors generated correct arithmetic results 99% of the time.
All of these studies show consistent discontinuities between individuals' performances in work situations and in school-like testing ones. Herndon reports quite spectacular differences between math in the bowling alley and in a test simulating bowling score "problems." The shoppers' average score was in the high 50s on the math test. The market sellers in Recife averaged 74% on the pencil and paper test which had identical math problems to those each had solved in the market. The dairy loaders who did not make mistakes in the warehouse scored on average 64% on a formal arithmetic test.
Both Claude and Perplexity claimed that these results have been consistently replicated, e.g. Perplexity's answer included a link to a Nature paper from February:
Children’s arithmetic skills do not transfer between applied and academic mathematics
Many children from low-income backgrounds worldwide fail to master school mathematics1; however, some children extensively use mental arithmetic outside school2,3. Here we surveyed children in Kolkata and Delhi, India, who work in markets (n = 1,436), to investigate whether maths skills acquired in real-world settings transfer to the classroom and vice versa. Nearly all these children used complex arithmetic calculations effectively at work. They were also proficient in solving hypothetical market maths problems and verbal maths problems that were anchored to concrete contexts. However, they were unable to solve arithmetic problems of equal or lesser complexity when presented in the abstract format typically used in school. The children’s performance in market maths problems was not explained by memorization, access to help, reduced stress with more familiar formats or high incentives for correct performance. By contrast, children with no market-selling experience (n = 471), enrolled in nearby schools, showed the opposite pattern. These children performed more accurately on simple abstract problems, but only 1% could correctly answer an applied market maths problem that more than one third of working children solved (β = 0.35, s.e.m. = 0.03; 95% confidence interval = 0.30–0.40, P < 0.001). School children used highly inefficient written calculations, could not combine different operations and arrived at answers too slowly to be useful in real-life or in higher maths. These findings highlight the importance of educational curricula that bridge the gap between intuitive and formal maths.
This sounds like it might be a Paper Trauma -ish thing, which might have a different specific mechanism.
The bit in the Nature paper saying that the formal -> practical direction goes comparably badly as the practical -> formal direction would suggest that it's at least not only that. (I only read the abstract of it, though.)
1I was going off a vague sense from having talked to a few people who had scanned the literature more than I.
Right now I'm commissioning a lit review about "transfer learning", "meta learning", and things similar to that. My sense so far is that there aren't a lot of super impressive results, but part of that looks like it's because it's hard to teach people relevant stuff in a "laboratory"-esque setting.
My implicit model is something like "in addition to g factor, there'd turn out to be an 's factor' (i.e. "slow intelligence") that is a product of both "g" and "general reasoning skills."
The old posts on mathematical talent by JonahS (1,2,3) seem maybe related to that? Although I took JonahS to be arguing that people like Grothendieck score highly in “ability to find / build really great mental models (albeit not necessarily quickly)”, which is neither g-factor nor skill-at-planning-and-pivoting, I think. I’m not sure though. I wish JonahS had written more.
This is less of "a plan" and more of "a model", but, something that's really weirded me out about the literature on IQ, transfer learning, etc, is that... it seems like it's just really hard to transfer learn. We've basically failed to increase g, and the "transfer learning demonstrations" I've heard of seemed pretty weaksauce.
But, all my common sense tells me that "general strategy" and "responding to novel information, and updating quickly" are learnable skills that should apply in a lot of domains.
I'm curious why you think this? Or if you have a place where you've explained why you think this at more length? Like my common sense just doesn't agree with this -- although I'll admit my common sense was probably different 5 years ago.
Overall a lot of the stuff here seems predicated on there being a very thick notion of non-domain specific "rationality" or "general strategy" that can be learned, that then after being learned speed you up in widely disparate domains. As in -- the whole effort is to find such a strategy. But there seems to be some (a lot? a little?) evidence that this just isn't that much of a thing, as you say.
I think current ML evidence backs this up. A Transformer is like a brain: when a Transformer is untrained, nearly literally the same architecture could learn to be a language model; to be an image diffusion model; to play Starcraft; etc etc. But once you've trained it, although it can learn very quickly in contexts to which it is adapted, it basically learns pretty poorly outside of these domains.
Similarly, human brains start of very plastic. You can learn to echolocate, or speak a dozen languages, or to ride a unicycle, or to solve IMO problems. And then brains specialize, and learn a lot of mostly domain-specific heuristics, that let them learn very quickly about the things that they already know. But they also learn to kinda suck elsewhere -- like, learning a dozen computer languages is mostly just going to not transfer to learning Chinese.
Like I don't think the distinction here I'm drawing is even well-articulated. And I could spend more time trying to articulate it -- there's probably some generality, maybe at the level of grit -- but the "learn domain-non-specific skills that will then speed up a particular domain" project seems to take a position that's sufficiently extreme that I'm like... ehhhh seems unlikely to succeed? (I'm in the middle of reading The Secret of Our Success fwiw, although it's my pre-existing slant for this position that has inclined me to read it.)
I think two main threads here:
- I think I just have tried to learn 'how to think on purpose', and have basically succeeded (like, somewhat, not necessarily amazingly, but enough to know there's a "there" there)
- Even in the world where skills don't transfer, some skills seem just useful in more places, or in "more useful places."
Re: 1
Most of the time, I'm not thinking strategically, I'm just doing some sort of pattern-matchy-find-the-nearest-reasonable-thing-to-do-and-then-do-it. My current guess is this is what most people (and, probably, ML algorithms?) are doing most of the time.
But, there's clusters of habits that seem pretty useful for solving novel problems, like asking:
- What is my goal here?
- what seem like the main inputs into that goal?
- what resources are available that compound?
- original seeing on the stimuli I'm looking at
- what skills are required here? what subskills make them up? what's the skill-tree?
- what would give me good feedbackloops for gaining those subskills, or, checking if I'm making progress towards my goal?
Each of those feel like "skills" to me, which I've practiced and cultivated, and once cultivated, can be chained into habits.
Re: 2
If you learn to play piano, I'd expect some weak transfer into: hand-finger coordination, understanding chord progression / musical structure, etc. If you learn a couple different instruments you probably have an easier time picking up new instruments. This can pave the way towards... being really good at music, and maybe some related things.
If you learn arithmetic and algebra, you have a building block skill that applies to science, engineering, and business. These things seem more world-changing than music.
(I think music can be world changing, but I think the skill-tree there is more like 'songwriting' and 'connecting with a muse and speaking to the heart of people's souls', which I think is pretty different from piano playing)
Point #1 is sort of a subset of point #2: analyzing your goals, breaking things down into subgoals, breaking down skills into subskills, are all "skills" that I expect to generalize quite a lot in a lot of domains.
...
How much is this worth?
I do think a point you made that stands out is "well, there's only so much you can specialize. If you specialize at meta-skills, i.e. "specialize in being a generalist", does that trade off against being better specialist?
Probably.
I think it depends on how early you pick up the meta-skills – it seems like a travesty that children aren't taught these skills at like age ~10 so that they get to apply them sooner/faster to more domains. If you're 30ish (like me), I don't think it's that obvious, in all cases, that you should "level up at meta". I spent the last month learning "meta", and I could have been learning ML, or math proofs, or web design, and it would have been more immediately applicable.
(See: Rationality !== Winning)
The reason I think this is important is because I think "how do we safely create a superintelligence" (or, avoid doing so in a reliable/safe fashion), are very confusing questions. It isn't obvious if I'm (or others) are supposed to learn ML, or math proofs, or geopolitics. And meta-skills seem more necessary for figuring out how to navigate that, and what specialist skills to learn, and how to apply them. i.e. Specializing in Problems We Don't Understand.
(This does all have implications in what sort of ML training regimes I'd expect to produce a general mind, although I think that's, like, bad and you shouldn't do it. Also it does still look like ML is still bottlenecked more on something like 'g' than something like 's' at the moment).
So I agree with some of what you're saying along "There is such a thing as a generally useful algorithm" or "Some skills are more deep than others" but I'm dubious about some of the consequences I think that you think follow from them? Or maybe you don't think these consequences follow, idk, and I'm imagining a person? Let me try to clarify.
There's clusters of habits that seem pretty useful for solving novel problems
My expectation is that there are many skills / mental algorithms along these lines, such that you could truthfully say "Wow, people in diverse domains have found X mental algorithm useful for discovering new knowledge." But also I think it's probably true that the actually shared information between different domain-specific instances of "X mental algorithm" is going to be pretty small.
Like, take the skill of "breaking down skills into subskills, figuring out what subskills can be worked on, etc". I think there's probably some kind of of algorithm you can run cross-domain that does this kind of thing. But without domain-specific pruning heuristics, and like a ton of domain-specific details, I expect that this algorithm basically just spits back "Well, too many options" rather than anything useful.
So: I expect non-domain specific work put into sharpening up this algorithm to run into steeply diminishing returns, even if you can amortize the cost of sharpening up the algorithm across many different domains that would be benefitted. If you could write down a program that can help you find relevant subskills in some domain, about 95% of the program is going to be domain-specific rather than not domain specific, and there are something like only ~logarithmic returns to working on the domain-specific problem. (Not being precise, just an intuition)
Put alternately, I expect you could specify some kind of algorithm like this in a very short mental program, but when you're running the program most mental compute goes into finding domain-specific program details.
Let me just describe the way the world looks to me. Maybe we actually think the same thing?
-- If you look throughout the history of science, I think that most discoveries look less like "Discoverer had good meta-level principles that let them situate themselves in the right place to solve the issue" and more like "Discoverer happened to be interested in the right chunk of reality that let them figure out an important problem, but it was mostly luck in situating themselves or their skills in this place." I haven't read a ton of history of science, but yeah.
-- Concretely, my bet is that most (many?) scientific discoverers of important things were extremely wrong on other important things, or found their original discovery through something like luck. (And some very important discoveries (Transformers) weren't really identified as such at the time.)
-- Or, concretely, I think scientific progress overall probably hinges less on individual scientists having good meta-level principles, and more on like...whatever social phenomena is necessary to let individuals or groups of scientists run a distributed brute-force search. Extremely approximately.
-- So my belief is that so far we humans just haven't found any such principles like those you're seeking for. Or that a lack of such principles can screw over your group (if you eschew falsifiability to a certain degree you're fucked; if you ignore math you're fucked) but that you can ultimately mostly raise the floor rather than the ceiling through work on them. Like there is a lot of math out there, and different kinds are very useful for different things!
-- I would be super excited to find such meta-level principles, btw. I feel like I'm being relentlessly negative. So to be clear, it would be awesome to find substantive meta-level principles such that non-domain specific work on the meta-level principles could help people situate themselves and pursue work effectively in confusing domains. Like I'm talking about this because I am very much interested in the project. I just right now... don't think the world looks like they exist? It's just in that in the absence of seeing groups that seem to have such principles, nothing that I know about minds in general makes me think that such principles are likely.
Or maybe I'm just confused about what you're doing. Really uncertain about all the above.
I totally agree with how science normally works. I'm sitting here being like "whelp, doesn't seem like the way science normally works can solve the problems I care about in time."
It's a serious question on my end "can I raise the ceiling, or just the floor?" and "Does raising the floor matter?". Thinking about that led to me re-examining "can I actually help senior researchers?", and feeling like I had at least some traction on that, which output the "Help Senior Researchers with Targeted Problems", which indeed feels most important insofar as it's tractable.
My sense is that most senior researchers at least "know, and sometimes think about, all the meta-level principles I've thought about so far." But, they don't always keep them in their "context window". Some things I current expect (at least some) senior researchers to not being attending to enough:
- not actually maximizing their working memory tools.
- not consistently steering towards the most hard-and-uncertain-but-important parts of their problem, so they can falsify early and move on to the next idea
- relatedly: pursuing things that are shiny and nerdsnipy.
- not attending much to "deliberately cultivate their meta-strategies", even in ways that just make sense to them. (My guess is often they'll have decent taste for what they should do more of, if prompted, but they don't prompt themselves to think about it as often as is optimal
Also, I think a bunch of them have various executive dysfunction stuff or health issues, which isn't what I'm currently focused on but seems important.
(note: I think "pursue things that are shiny/nerdsnipy" is an important motivational system that I'm not sure how to engage with, without breaking important things. But, my guess here is something similar to "if you want to marry into wealth, hang out around rich people and then marry for love". i.e. sink your attention into places where the shiny nerdsnipy problems are important, and then pick research directions based off excitement)
It'd be cool if a second group also worked towards "rationality skill assessment."
This was my project at last year's Epistea, but I sort of had to pause it to work full-time on my interp upskilling experiment.
I only got as far as implementing ~85% of an app to facilitate this (as described here), but maybe a quick chat about this would still be valuable?
1The LessWrong Review runs every year to select the posts that have most stood the test of time. This post is not yet eligible for review, but will be at the end of 2025. The top fifty or so posts are featured prominently on the site throughout the year.
Hopefully, the review is better than karma at judging enduring value. If we have accurate prediction markets on the review results, maybe we can have better incentives on LessWrong today. Will this post make the top fifty?
They also attempt to generate principles to follow from, well, first principles, and see how many they correctly identify.
Second principles?
========
I'm really glad to see you quoting Three Levels. Seems important.
If I'm building my own training and tests, there's always the risk of ending up "teaching to the test", even if unintentionally. I think it'd be cool if other people were working on "Holdout Questions From Holdout Domains", that I don't know anything about, so that it's possible to test if my programs actually output people who are better-than-baseline (controlling for IQ).
I am hoarding at least one or two fun facts that I have seen smart rationalists get wrong. Specifically, a claim was made, I ask, "huh, really?" they doubled down, and then later I go look it up and find out that they were significantly wrong. Unfortunately I think that if I had read the book first and started the conversation with it in mind, I might not have discovered that they were confidently incorrect. Likewise, I think it would be hard to replicate this in a test setting.