[rough babble on prerequisites for front-loading conceptual work, in the context of "maybe we should just wait until we have robustly superhuman AIs in a conceptual domain / wait until we have much higher bandwidth BCIs so that verification loops are quicker. Is just inefficient to attack certain problems now". this is unedited, will perhaps turn this into a more polished post later]
Perhaps some domains are "fundamentally serially bottlenecked", either on thought cycles or real-world iteration cycles. I don't quite understand how to operationalize "being serially bottlenecked" in a way that is ~basically not substantially accelerated by AI automation. Perhaps certain forms of human coordination fall into this category? Things "we want to do ourselves"?
We have a small or negative compute gap between generation and verification. E.g., substantial evidence for an insight making "philosophical progress" or being a "correct definition" is accumulated through human generation of the thought ("lives as detritus in your head", is scored according to coherence with existing neural heuristics and in conjunction with the rest of one's world model"). The verification process for some set of fuzzy tasks involves reconstructing these from the generated output (e.g. math paper / philosophical text) and when the verification involves internally checking this parsimony or coherence a human runs a similar process to generating the thought themselves. "Understanding mathematics is not too dissimilar from proving new mathematics oneself." That said, I'd be surprised if this wasn't at least a 10x gap in most domains of conceptual importance?
Some thoughts are strategically important. In catastrophic scenarios, often individuals are looking to "implement a plan off the shelf" and so having correct plans off the shelf is high-value. This alone doesn't justify doing the work now (generation could become 1000x faster) but if coordination is a ~fixed cost or path-dependent strategy implies that having the correct ideas now has much higher value than having the correct ideas on a given topic later, then this consideration could dominate.
There are some domains where the AIs will just stay subhuman for too long, either because the spiky capability profiles reach critical capability thresholds in e.g. ML R&D sooner or because of constraints on current architectures (e.g. "lacking algebraicity / ability to autonomously generate novel interesting concepts"), and if work in these domains is important then we have to do it.
I am curious to find high-quality arguments for/against humans doing conceptual work of various forms in late-2026. (Where conceptual work is work in the category of ARC / Redwood / Forethought etc). Of course there are many different kinds of "conceptual work" (e.g. question-generation, direct theorem-proving, "digestion") so the operationalization of this question matters quite a bit.
I am currently just quite skeptical of the argument provided here that direct AIS work makes ~no sense and effort should instead be directed at making solutions politically ... (read more)