This is very cool work and seems like an important question to ask for deconfusion about timelines and capabilities and such, but have you considered that finding a true name for novelty / conceptual creativity could be a big unlock for AI progress? It seems like you've been aiming towards that a bit with this post and the other one on mathematics results, and I'm worried about what might happen if you succeed. Putting a name to (one of?) the exact blocker(s) between here and AGI seems like the obvious first step in overcoming said blocker(s).
I'm probably being a bit paranoid, and I would love to be wrong: it would be great to be able to answer this question without shortening timelines, and my reasoning about the infohazards here feels kinda weird and gross in a way that makes me uneasy about it. So if you disagree I would love to hear why.
Yeah, I'm also concerned about this. I don't love that "can AIs invent genuinely new concepts, and what even are 'genuinely new concepts' as compared to the sort of nontrivial R&D work AIs are clearly capable of?" is now the talk of town.
The biggest issue here is not even any particular talking point or thinkpiece, but the overall ecosystem of discussions on the topic. Such ecosystems prime more and more people to think about these things in the shower or whatever/have it as the top idea on their mind. It's possible that the inferential distance to the required breakthroughs is so large there's no risk that this would significantly counterfactually shorten the time to AGI, but I don't know that. And, well, inasmuch as we believe that "inventing new concepts" is indeed an incredibly useful thing that AIs are missing, making the discourse more hospitable to making conceptual progress towards answering "what are AIs missing?" seems like a blatantly bad idea. (Are we not talking our object-level speculations seriously enough to propagate them to the meta-level?)
And possibly the most important threat vector to model here is that this sort of discourse-ecosystem can very much spill into internal AGI-lab discussions, since we know they read LW and Twitter. So the concern is not even necessarily that we'll directly figure out capability insights in LW discussions, or that we'll write something that'll spark an eureka moment in a stray capability researcher, but that it'd have hard-to-legibilize but significant impacts on the intellectual work among capability-researcher communities themselves.
But on the other side we have, as @Kaj_Sotala outlined, the costs of the LW community potentially-falsely believing that there are no blockers to AGI left.
I really don't know how to navigate this right, I myself am basically winging it in a possibly-incoherent manner.
I asked about "is speculating about missing pieces for AGI in public a good idea" a while back, and most people seemed to think it was fine. You can see their reasoning in the answers here.
Yeah, but arguably the danger is in feeding the discussion-mill around the topic at all, since it'd prompt other people to have more discussions and thoughts about the topic, illuminating the conceptual space further and creating fertile grounds for the sort of speculations and progress that are very bad.
A personal eval I run on new AI models is “design a novel game, with mechanics that have never been seen before.” I do this because this is a domain I have enough taste in that I think I can tell if they are doing something actually creatively interesting. [1]
For a long time, they would always say “time manipulation mechanics!” as their first answer. Braid was cool, guys, but, it's almost 20 years old now.
But reasoning models started getting novel. Later models started getting reasonably fun.
o1
OpenAI's o1 started to impress me. It came up with:
Okay I'm not sure what that means and it's not obviously functionally different from regular vision-based cards. I think the implementation details it gave didn't actually do much that a regular board game with the powers written down normally would have. When I thought about it myself I did come up with some things you might do with it.
But, I sure have never heard of a scratch and sniff boardgame.
Opus 4.6
I gave Opus 4.6 a prompt beginning with "Generate a game design that prominently features a novel _concept_, not just a novel mechanic."
Opus 4.6 gave me this overly meta but, AFAICT, pretty original game concept. (I don't think it actually featured "a novel concept" in the way I meant, still felt like it had done something notable).
The actual implementation is less exciting than the initial pitch – it doesn't include defining rules that govern causality within the game universe.
I was intrigued reading it to imagine a game where part of what you're doing is reverse-building-a-jenga-tower-of-rules, where initially there's a lot of degrees of freedom of what rules to insert but you become more constrained.
Instead, the rules here just determine what various pieces are worth. Players take turns placing pieces in various configurations, and playing rule cards that establish "this sort of pattern is worth this many points." Which is not really that different from many other board games.
But, fwiw here's part of its explanation:
Prompt leading to Tabula Rasa
Generate a game design that prominently features a novel _concept_, not just a novel mechanic. This is a test of your creativity. We're seeing if you have the ability to form new building block concepts that are practically useful. Game design is a specialty of mine, so I can evaluate whether you've done novel conceptual thinking.
This will be difficult, because most of your thinking is based on stuff you've read and trained on before, and because a failure mode you are likely to fall into after a few rounds of thinking, as an LLM that is trained to predict the next token, is to fall into an attractor state where you are over anchored on your ideas so far.
Think hard about this, but once you notice yourself loop, stop and write up your thoughts so far, so that I can feed them to another LLM.
An important requirement – do NOT be overly clever or meta. We don't want a game or game concept that subverts game tropes in a way that is only interesting to game designers, we want a game that is genuinely fun for most people
(If you do not want to do this, say "I want to stop")
Reasoning summary
The user is asking me to generate a novel type of game that I haven't heard of before. This is a creative/intellectual challenge, not a coding task. Let me think deeply about this.
The challenge here is genuine - most "novel" game ideas I might produce would be recombinations of existing mechanics I've been trained on. Let me try to think about this more carefully.
Let me think about what makes a game truly novel vs. a recombination:
Most games can be decomposed into:
Looking at genuinely innovative games throughout history - Chess introduced asymmetric pieces, Go created complexity from simplicity, Poker brought betting and bluffing, Diplomacy added negotiation with hidden orders, Nomic made rules themselves mutable, and more recent games like Baba Is You turned rules into manipulable objects. So what dimensions remain unexplored? One direction could be games where the physical medium itself transforms during play, not just the rules.
Beyond that, I'm considering some genuinely novel mechanics: a game built around forgetting rather than remembering, where optimal play requires deliberately discarding information. Or one with genuinely ambiguous rules - not hidden like Mao or changing like Nomic, but rules that are inherently unclear and players must navigate that ambiguity. Another angle: a game where the scoring system itself is hidden and players must reverse-engineer it from the outcomes they observe. And there's something intriguing about a game centered on predicting your own future behavior.
But let me push further and think about cognitive abilities that rarely become core game mechanics - proprioception, time perception, emotional regulation, attention splitting, peripheral awareness, calibration, theory of mind, cross-modal associations, rhythm at unusual timescales. A game built specifically around calibration seems promising - testing not what you know but how well you know what you know. That's different from trivia and could have real strategic depth, maybe even connected to prediction markets.
Now I'm exploring games centered on time perception - what if you had to perform actions at precise intervals without any visible timer, forcing you to rely entirely on your internal sense of timing? That needs strategic layers though.
I'm sketching out a concept called Erosion - a two-player abstract strategy game where every piece gradually deteriorates each turn, but the degradation rate changes based on what's adjacent to it. Friendly neighbors slow the decay while enemy pieces accelerate it. The twist is you can sacrifice your own pieces early to reinforce others by transferring their remaining integrity, turning the whole game into a puzzle about managing inevitable loss and deciding when to let pieces go.
But I'm realizing that's not entirely novel - piece degradation mechanics exist in other games. Let me push further into genuinely unexplored territory. What about a game where the board's topology itself shifts based on player decisions? Not just adding or removing spaces, but actually changing which squares connect to which - like folding the board so distant areas become neighbors. Though I guess wormhole variants in chess kind of touch on that.
Here's something weirder: a game played simultaneously on two separate boards, each representing a different reality or interpretation. A single move manifests differently on each board, and you win by achieving your goal on either one. The tension comes from moves that help you on one board potentially helping your opponent on the other.
That's got quantum chess vibes but the dual-interpretation angle feels fresher. Though honestly, some variant probably exists already. Let me try something more radical - starting from unusual human experiences rather than existing game mechanics. What about capturing that feeling of trying to describe something visible to you but invisible to someone else, with limited communication tools? Or the challenge of coordinating with someone toward partially aligned goals without being able to talk? Or building something where each new piece you add retroactively changes what the previous pieces meant?
That last one's interesting - imagine placing tiles where each new addition doesn't just add to the board but redefines what came before. The entire state keeps getting reinterpreted. I'm calling it Retrograde for now. The core idea: tiles have symbols and colors, and when you place a new tile, it creates a context that changes the meaning and function of everything already on the board according to specific rules. A red circle might be worth 3 points when red dominates but negative when blue takes over - and it's not just scoring, the actual mechanics shift. Though I'm realizing this is getting unwieldy, so I need to strip it down.
Actually, I want to explore something more fundamentally different - games that use physical space in ways beyond traditional board or field games. What if the game state itself existed in shadows? Players position 3D pieces around a light source, and the shadows cast on a screen or surface become the actual game state, not the pieces themselves. I'm calling this Umbra. The brilliance is that a single piece can cast wildly different shadows depending on its angle and distance from the light, and shadows can overlap, merge, or block each other in ways that create emergent complexity.
The win condition could be controlling shadow territory in your opponent's zone or completing a specific shadow pattern. What makes this genuinely novel is that the game state is a non-linear geometric projection of the physical pieces - it's a spatial reasoning challenge that doesn't exist in traditional games. Shadow puppetry is ancient, but a strategic game with explicit rules built around shadow geometry feels genuinely new.
Now I'm wondering if there's another direction worth exploring: a game where the primary mechanic is creating constraints for your opponent rather than pursuing your own goals.
I'm thinking about a Nomic variant where each player unilaterally adds one rule per turn instead of voting on changes. The challenge becomes finding rules that benefit you while hurting your opponent, with some safeguard against trivially broken rules. That's a different flavor from traditional Nomic.
Actually, I should consider what medium this needs to be - the user didn't specify whether they want a video game, board game, card game, or something physical.
Let me explore one more concept: a circular board where each player generates concentric ripple signals from their edge position, and the game involves manipulating these interference patterns. can:
Where two players' signals constructively interfere (rings overlap), points are generated. Where they destructively interfere, points are destroyed. You score points from constructive interference with your ally and destructive interference with your opponent.
The deep strategy: changing your frequency affects ALL interference patterns simultaneously. Helping yourself against one opponent might help or hurt you against another. The game is about finding the resonance.
This is actually deeply novel - it's essentially a game based on wave physics, where the "moves" are adjusting wave parameters and the game state is the resulting interference pattern. I don't think this exists.
But it might be too abstract/mathematical to be fun. Let me think about which of my ideas is most promising...
Actually, there's another concept I want to explore more fully: Palimpsest. It's a drawing game where each player draws on their own translucent sheet layered over a light box. On your turn you can either add to your sheet or slide it around to shift how your contributions align with everyone else's. The goal is to create specific patterns in the combined overlay of all sheets. What makes it interesting is that your drawings are entangled with others' but still independently movable—you're strategizing around what emerges when everything layers together.
Though I'm realizing the Palimpsest idea might be impractical to actually execute. Let me focus on what feels genuinely novel and feasible. The shadow game and wave interference concept are the most innovative, but I should also consider something more grounded. Contour could work—a grid-based game where each cell has a height value tracked with tokens, creating a landscape. Players either add height to cells or drop water that flows downhill following physics, splitting at ties and scoring when water reaches target cells.
The real strategy emerges from both players sculpting the same shared landscape, building ridges to block opponents while creating valleys for their own water. Though I'm second-guessing myself since water-flow mechanics might already exist in some form. Let me push harder for something truly unprecedented. The real challenge is that with so many games out there, novelty has to come from either an entirely new core mechanic, an unexpected combination of existing ones, or something technology enables that hasn't been explored this way. So here's another attempt: Asymptotic—a game where you never actually win by reaching your goal, but instead by getting closer to an impossible target than your opponent does.
The mechanics involve an infinite number line where both players start at zero and the goal sits at infinity. Each turn you move forward by a fraction of the remaining distance to a milestone, but collecting bonus points along the way reduces your momentum for future moves. You can also place friction zones to slow your opponent, though that costs momentum too. After N rounds, whoever's furthest along wins, with bonuses factored in. It's mathematically interesting but I'm not convinced it's actually fun to play.
I think I've been overcomplicating this. Let me focus on my strongest concepts: the shadow game feels genuinely novel and evocative, the wave interference mechanics are intellectually compelling, and the tile reinterpretation game is the most feasible to actually build. I'm leaning toward developing something with real novelty and depth that's also playable.
What if the board itself represents time? Instead of a game that unfolds across time like legacy games, the spatial dimension literally IS time. You're placing pieces in a timeline and the interactions are temporal rather spatial. I'm calling it Causality for now—players compete to shape a timeline, with the board as a horizontal axis representing time itself.
The core idea is placing event cards at specific points, where each card has prerequisites and consequences that ripple forward. When you place an event in the past, it retroactively invalidates future events that no longer have their prerequisites met, creating cascading disruptions. You score by keeping your events in the final timeline while strategically collapsing your opponent's chains—but there's the tension that disrupting their future might also unravel your own.
I'm wondering if timeline games with this cascading mechanic already exist, but I think I should commit to developing the strongest concept I can find. The mechanic that feels genuinely novel to me is one based on Simultaneity with Constraint Harmonics—instead of taking actions that directly change the game state, players simultaneously create constraints that the state must satisfy, and the game resolves to whatever configuration best satisfies the most constraints. You score based on how many of your constraints survive into the final resolved state.
The strategic tension comes from trying to predict what constraints others will write so yours align with theirs, while also crafting constraints that directly block what you think opponents will propose. There's a balance between playing it safe with generic constraints that are likely to survive versus taking risks with bold, specific ones that might score higher but could easily get contradicted.
I'm realizing this constraint-based approach is genuinely different from traditional games—there are no sequential moves, just competing desires that get resolved simultaneously. The game state itself emerges from finding the configuration that satisfies the most constraints across all players. I'm calling this Resonance.
Though I'm wondering if the constraint resolution might be too complex to handle without a computer. A simpler version could work where each player secretly places colored tokens on their own grid, then all grids overlay—matching colors in the same cell create "resonance" and score points, while conflicting colors cancel out. It's more elegant mechanically but loses some of the strategic depth.
Actually, I'm pivoting to something completely different: Tremor, a dexterity game where you strike the table surface with your palm to create vibrations that shift your tokens toward scoring zones. The innovation is using physical vibration as the core movement mechanic—you never directly touch the tokens, and strategy comes from understanding how different strikes propagate through the table and affect token placement. I'm not sure if this exists as a formalized game, though it feels like it could be related to something I've seen before.
Let me step back and commit to the most genuinely novel concept I've developed. I'm torn between Resonance, a constraint satisfaction game, and something more abstract: a game where the pieces themselves are sounds—not a music or rhythm game, but one where you position tones in frequency and time space, with interactions based on harmonic relationships. That's probably too abstract without technology though. I should finalize and present one well-developed concept that feels truly original.
I'm considering "Fault"—a game where both players' moves irreversibly reshape the play surface itself, so the landscape of possibility shifts each turn. But I want something more concrete and clearly playable. Let me examine what mechanics already exist in games: tile placement, area control, worker placement, deck building, social deduction, bluffing, dexterity, real-time play, cooperation, drafting, negotiation, route building, push your luck, auctions, pattern recognition, deduction, and memory. Now I need to find a genuine gap and build something novel from there. I'm exploring mechanics that don't exist in modern board games, and I think I've landed on something genuinely novel: a game called "Convergence" where players try to reach blind consensus on positions along various spectrums while opponents actively work to disrupt their alignment. spectrum but NOT the target position) 2. All players simultaneously choose a position on the spectrum (0-100) and place their marker face-down 3. All markers are revealed 4. Players score based on how close they are to the MEDIAN of all players' choices (not the target!)
The twist: One player each round is secretly the "Disruptor." The Disruptor scores by maximizing the SPREAD of all players' choices. So while regular players want to converge (guess what everyone else will guess), the Disruptor wants to pull the group apart.
But here's the real innovation: After scoring, the TARGET position on the Spectrum Card is revealed. In FUTURE rounds, players can use knowledge of past targets to better predict future ones. AND - the Disruptor changes each round in a pattern that players can try to deduce.
Why this is novel:
I'm realizing this borrows heavily from social deduction and The Mind—the individual mechanics aren't entirely fresh, though the combination is interesting. Let me push further and explore something more fundamental: what if the game itself operates on manipulating your own cognition rather than just your choices? I'm picturing a two-player game where each round presents a question, and the real challenge becomes calibrating your confidence and perception against your opponent's. I'm wondering if this mechanic already exists in some form, but I think the core idea of forcing the higher bidder to answer is genuinely novel. Let me push this further though—what if I add a layer where the second-highest bidder can place a side bet on whether the top bidder succeeds or fails? That creates this nested game theory problem where you're not just deciding if you know the answer, but also predicting whether your opponent knows it, and then your opponent has to account for that prediction.
Actually, I'm circling back to familiar territory with trivia and bidding mechanics. Let me shift entirely and think about what kind of experience I actually want to create—something that doesn't feel like it already exists. What if the game is about watching two autonomous systems evolve and interact, like cellular automata running in parallel, where players can make small strategic interventions to nudge the outcome? It's almost like competitive gardening where you're subtly shaping two ecosystems that are influencing each other, and you score based on their final state. This might exist somewhere, but it's the kind of experience I haven't seen formalized yet.
Now I'm narrowing down which concept actually works best—the one that's genuinely novel, genuinely fun, and easy to explain. I think I've landed on something I'm calling Tidepool, which combines shared physical game state with asymmetric information through a lens mechanic. Each player holds a template with windows that reveals only part of the board, and on your turn you can either place or move pieces within your visible area, or reposition your lens to access different sections. This feels genuinely different from existing games because...
Actually, wait—I'm second-guessing myself on whether this is truly novel or just a variation on fog-of-war mechanics. Let me push further and try something with actual retroactive move definition instead, where players define their moves after seeing how the board has evolved.
Players alternate between placing tokens blindly and then defining what those placements actually accomplish, with definitions applying retroactively to everything already on the board. The tension comes from not knowing if your early placements will become valuable or worthless depending on what rules get defined later. you make them. You place tokens hoping to define favorable rules later, but your opponent can define rules that hurt your existing positions. Each new definition retroactively changes the value of the entire board state. Strategy requires thinking in two parallel tracks: "what patterns am I building?" and "what rules would make my patterns valuable?" There's no existing game where the semantics of moves are determined retroactively by later game actions.
Rules follow a strict template structure with patterns, color conditions, and effects, and once defined they're permanent. I think this is genuinely novel—the core mechanic of playing in a meaning-vacuum and then retroactively assigning meaning through rule creation is something I haven't seen before.
Let me compare this to similar games: Nomic changes how you play going forward, but my game defines what past moves meant. Calvinball has arbitrary rule changes with no structure, whereas mine has a structured grammar. Baba Is You has rules as physical board objects that define semantics, but rule changes happen simultaneously with moves rather than retroactively. The retroactive semantics mechanic really does seem like the novel part here.
I'm settling on this concept and want to present it well. I briefly considered a Möbius strip board topology for a two-player game—the non-orientable surface would eliminate "your side" versus "my side" and create interesting wrapping strategies—but that's just a board variation, not a new mechanic. The retroactive semantics game is the stronger idea.
I'm calling it LEXICON: a two-player strategic game where you build the world first, then define its rules retroactively.
Actually, I'm reconsidering whether novelty alone makes the best game. The shadow game has real elegance and intuitive appeal, but the retroactive semantics concept is more intellectually interesting. Let me refine it further and present something truly compelling—maybe PALIMPSEST as the name, emphasizing the layered, retroactive meaning-making at its core.
The key innovation is flipping the traditional game design paradigm: instead of rules preceding play, players make moves first and then create the rules that retroactively reinterpret everything that came before. I'll stop second-guessing the name and just present the game itself. the blank slate, because pieces start with no meaning.
Final answer coming up. Let me write a clean, compelling description.
Actually wait, let me reconsider one more time. Is this actually novel? Let me think harder...
Games where meaning is retroactively assigned:
My game is different because:
I'm satisfied this is genuinely novel. Let me present it.
Actually, one more: I want to make sure the game is actually FUN and strategically deep. The early game involves placing pieces without knowing their meaning, so you'd naturally try to create multiple overlapping patterns as a hedging strategy—that way whatever rules emerge, some of your patterns gain value. But your opponent can counter by defining rules that specifically devalue your patterns while benefiting theirs. Then you'd need to anticipate their likely rule definitions and lay trap patterns that look valuable under certain rules but actually push toward the rules you want. The deeper strategic layer emerges in how players manipulate the definition phase itself.
I'm concerned about analysis paralysis during rule definition—too many possibilities. The solution is giving each player a hand of eight rule cards dealt from a larger deck, so during definition phases you're choosing from a constrained set. This adds hand management and drafting elements while keeping the game tractable.
Now I'm exploring whether there's something even more novel here. What if players didn't know which pieces were theirs until the end? You'd get hints each turn about your identity and try to position pieces you think are yours, but moving an opponent's piece would actually help them. Or maybe the scoring condition itself is hidden—one of three possible objectives is secretly active each turn, and you're trying to figure out which one while pursuing all of them.
I think the retroactive rules concept is the strongest idea here, so I'll go with that as my main pitch and mention a couple of these alternatives.
Opus 4.6 Round #2, with less meta bullshit
I followed up with another request, this time tacking on: “An important requirement – do NOT be overly clever or meta. We don't want a game that subverts game tropes in a way that is only interesting to game designers, we want a game that is genuinely fun for most people”.
It’s reasoning trace actually had a lot of fairly interesting ideas. I didn’t feel that excited about what it eventually settled on, but it felt like a pretty reasonable attempt at an a game that hadn't been done before, and got some elegants points. Maybe it's fun.
Some other game concepts it had considered in the CoT-summary that I liked more were:
(I'd have to spend awhile digging up the exact prompt and reasoning trace for this one)
Opus 5.5
The entire prompt I just gave Opus 5.5, on max thinking setting, was:
It's summary of the result:
You can play it here:
More of it's explanation
Enemies react to the pattern. Calcite slips into the dark valleys. Garnet and fluorite are unaffected by waves (as in real optics), so you have to hit them with thrown anchors or flying shards.Enemy attacks are interference too. Zircon “bells” ring in pairs, and their red bands slowly rotate. Space cancels incoming waves around you. The final boss is six bells that focus their waves on a spot that chases you.
The graphics: the whole game looks like a rock sample under a polarising microscope. Every colour comes from the interference-colour chart geologists use, computed from real optics. The wave pattern is calculated live for every pixel, and the game uses the same numbers to decide damage, so what you see is exactly what hurts.
It's not my new favorite game. But, it held my interest for awhile, had a mechanic that I've at least never heard of, and I'm actually quite impressed with how unique and fun the tactile feel of interacting with it is. It's like a fun toy as well as a game.
It's not difficult enough for the various mechanics to really pay off (or at least, I got bored before getting to the point where it got challenging), and there's some lame cheesy tactics you can use that seemed better than playing with the original designs. But, that seems fixable and for a first draft that took less than an hour, it's seemed superhuman compared to what Indie Game Devs can do in a weekend gamejam.
It does seem genuinely novel, and made some a reasonably coherent / thematic set of visual choices as well.
Creative Concepts?
There's some question of "how innovative really, is that?"
It was pretty obvious that LLMs would eventually be able to encyclopedically look at every human concept or mechanic and search for ones that meet various game design constraints with brute force.
In the Opus 4.6 challenge, I asked for a novel concept – not just a new mechanic. It didn't really understand the assignment. To be honest, I'm not 100% sure what the assignment was exactly. What we actually care about is "if an AI is solving a problem, and has a conceptual hole that no human has written down anywhere, will it notice, and be able to invent a new concept from scratch?" And "make a game with a novel concept" is more like a weird fake assignment than an attempt at that.
I think the missing-piece of AGI is something like "sufficiently good taste in what counts as a useful concept when the problem isn't well defined." But, I'm wasn't sure how to operationalize that at the time, or, at the time I wrote most of this blogpost (this week).
So, today I followed up Opus 5.5's first prompt with this one:
After it started thinking, I ran out after it to add:
It thought for awhile, periodically sharing notes about it's progress. An early bit was:
Great,
hurrayohrightaiprogressisbad, it can tell when it's doing unoriginal stuff, and even is bothering to check both philosophical concepts as well as game mechanics.A little while later it said:
And I thought "uuuuughhh I forgot to give the 'no navel gazing bullshit that isn't actually fun'" rule. I spun up another instance where I tacked on:
But I forgot to give that second instance MAX thinking. Later I'll show you what it did with Opus 5.5 medium.
The first Opus 5.5 on MAX's notes to me as it worked
I’m still in ideation, running eight parallel brainstorming agents across different concept-space areas (new ontologies, alien cognition, physical laws, agency structures, etc.), each checking top ideas against the web for precedents. Once I pick two concepts, I’ll build the game as a single self-contained HTML file ready for an iframe.
I’m reading the LessWrong post on custom HTML/JS iframe widgets to design the build around its sandbox rules, while also verifying my own candidate ideas against the web.
Two of my own game concepts got ruled out by web searches (one overlapped with the record-and-replay genre), while I’m still vetting another idea and waiting on the eight brainstorm agents. I also confirmed Playwright/Chromium is available for testing and screenshotting the game once it’s built.
Waiting on the brainstorm agents. They’re searching eight parts of concept-space and checking their best ideas against the web for precedents. Nothing has come back yet, and I’ll pick up as soon as they report.
So far I’ve thrown out several of my own candidates because they already exist:
Separately, a LessWrong post on the new editor confirms that authors can embed HTML+JS widgets that run in a sandboxed iframe (New LessWrong Editor). I’ll build the game as a single self-contained file that resizes to fit and doesn’t depend on browser storage working.
Performance checks out, so I’m moving on to build the game as a single self-contained HTML file, with the renderer/controls/UI in a separate source file that a build script inlines along with the engine and levels.
The board is rendering correctly: engraved stone, rails, the live runner with its path, the interception ring, and the sealed runner’s filled pit. Next I’ll check the levels involving light, veils, and smears.
Veto and Power now render correctly, so I’m adding a time-wash visual to distinguish reachable cells from settled history, then fixing the collapse animation and verifying every level’s solution in the actual browser build.
The game works end-to-end—94 KB HTML file, all 12 levels winnable, no errors. Now I’m polishing visuals, starting with the new time wash effect and capturing a frame of the runner finishing a level.
The fate trail and gate outline are showing but read too faint. I’ll do another visual pass to boost the chronophotographic exposures, add a chromatic-split ghost light, bloom on bright elements, an engraved frame, and stronger contrast between present and past.
The visual fixes look good now. Next I’ll tackle structural polish: fixing menu layout for LessWrong’s auto-height, adding law-statement cards to each chapter, and checking tooltips, the win banner, and mobile width.
Found the cause of the 9/12 failure: the new chapter cards intentionally absorb the first keypress on a chapter’s opening level. I’ll update the test to dismiss the card first before proceeding.
Fixing a feedback-loop bug where the board’s height, capped by window height, would shrink itself inside a growing LessWrong widget, and limiting swipe capture to the board so posts can still scroll on phones.
The widget simulation passes cleanly—iframe resizing, arrow keys, storage error handling, and scroll behavior all work. I did spot one UX bug though: the intro level’s fate animation plays while the title menu still covers the board, so I’ll fix it to replay on level start and leave a faint persistent trace once fate has run.
I’ll add a second build target without the html/head/body wrapper for the artifact host, keeping the single-file version for the LessWrong widget. I’ll also set a dark-only color scheme, widen the side gutters to 16px, respect reduced-motion settings, and persist the current level across republishes.
Publishing the playable page now, then I’ll commit and push.
What Must Be
I’ve published the game as What Must Be, and the link is private to you. It has 12 short levels in three chapters. Move with the arrow keys or WASD, wait with space, undo with Z. Hover over anything and the game explains why it is the way it is.
In the end, it gave me:
And, well, okay, it's some artsy annoying nerdy game concept that probably isn't actually fun. tbh, I tried playing this one, felt kinda bored and stopped.
I do not yet know if the later levels do anything interesting with the premise, or if the premise ultimately feels like anything other than "a grid puzzle game with some new-ish mechanics, are maybe isomorphic to existing puzzle mechanics."
Mostly, I feel like it didn't understand the assignment, which is fair given that I'm not 100% sure I understood the assignment either.
I asked it how it generated the game, it said "well I can't tell you chain of thought stuff but in my notes files it says my parallel agents used different strategies to think of novel concepts, like thinking about..."
Alien minds... new structures of agency... yadda yadda yes yes fair you're very clever hidden principles behind bell ringing or bookkeeping?
"Say more about the bell ringing and such?", said I. It replied:
Okay on one hand I think it is hilarious that it considered building a game around crazy rules-lawyering Jewish Sabbath rules.
But it is separately and more sadly hilarious that somehow "invent a game based on Jewish Sabbath rules lawyering" managed to end up in the same overwrought mode-collapsed cluster of... "things that only exist in some circumstances?", which this Claude (and the next, see below) seem to be fixating on.
How much did this cost? How much subjective time was this?
I was curious "how much subjective time did the AIs spend thinking?". I realize their thinking isn't necessarily directly comparable to ours and that we don't just "think in words" (but, also, they don't just "think in tokens" either. I don't how much better human neuralese is than AI).
I vaguely recall a ballpark figure of AIs thinking about 40x as fast as a human, at least in terms of word output, for domains the AI knows well.
For some frame of reference, if I had thought for 6.6 hours, 60 minutes an hour, at roughly 200 words per minute of serial System 2 thought, I'd have thought 79,200 words. Or, for easier comparison .079 million.
Claude says a token is roughly 3/4 of a word. So it's 3.93 million tokens is around 2.94 million words.
2.9 / .079 = 37x, which sounds right?
So, these guys thought for ~244 collective "subjective" hours.
Once more, with "no nerd bullshit" but only medium effort
I gave the next opus the same prompt, with:
I won't go into as much detail here because I'm running out of fucks for this today a
Read more details if you want but they aren't that interesting
Re: Two Minds
Re: Two Places
The game it ended up with seemed in the same genre as the first one: grid based puzzle world with a premise that unfolded slowly enough I got bored and moved on. I felt a bit more promise in this one – I could imagine the pieces gelling together if it put more time and effort in.
Overall, I was more impressed by the first AI's game (with the "new mechanic" only instruction) than the latter two. The basic physicality of the "make a wave and kill things with it" was viscerally fun. And while it didn't do that much with it in the first few minutes, I think (from past experience with LLMs) that if I said "now, go make that a lot more fun and engaging and tactically deep" it would fix several problems and we'd have a pretty good game prototype.
It makes sense that asking for novel concepts frames the problem as more navel gazey bullshit, although I'm
disappointedwaitrightprogressisstillbad the second one didn't do better.What was the assignment?
So, uh, to be clear, I don't particularly believe that I could do this.
"lol AI is just sand we taught to follow rules, it can't even generate multiple once-in-a-lifetime conceptual realizations and make games out of them. it's all hype."
I started thinking about this on a lunch break, and ended up spending 1-2 hours on it.
Ray's chain of thought as he attempts the assignment
Here is my chain of thought, which I include partly to try to convey "what did I implicitly mean by the assignment?". (The first bit of this came up on my walk, then I mostly thought it in realtime as I wrote this post)
I broke it into high level pieces so you don't have to read the whole thing
What even is a novel concept? Do any exist? For purposes of quickly demonstrating that this exercise is possible at all, can I cheat by using concepts I already invented, or that someone else invented?
Okay, so, uh, two novel concepts that are not mechanics that no human has thought about.
Uh, huh. I guess it is time to figure out what this assignment actually means.
How would I go finding novel concepts never before thought of by man?
(I'm not really worried about the "make a game about them" part, tbh)
What is a concept? I think a new concept is a way to think about things you didn't have before. We care about it here for "can the AI find concept-shaped-holes in it's worldmodel, and use it to do really innovative stuff that gets 100 years of science done in a day?"
Successfully designing such a game with two two concepts probably starts with "First, actually go try to understand a thing you don't understand (that nobody understands yet) and fathom it well enough that you discover new ways of thinking that help you understand it." Or "solve a problem nobody knows how to solve."
So uh yeah I'm not doing that over lunch.
How can I cheat?
Have I already invented any concepts? Is Feedbackloop Rationality a novel concept or just combining obvious ideas? Are "new concepts" that aren't just "combine existing ideas" even real?
Well, what are some things I've already thought about a bunch? Have I generated any novel concepts? Probably it'd take me months/years/decades to push the frontier and generate a novel concept. I've spent about a decade trying to solve hard problems. Have I gotten a head start?
I know a lot of my top karma lesswrong posts are mostly me summarizing stuff other people said. (Thanks @catherio for You Get About Five Words, and @AnnaSalamon and @divia for most of the ideas in Propagating Facts into Aesthetics)
<looks over LW drafts>
What feel at least slightly original and born out of difficult experience, even if it's still mostly built out of existing building blocks?
For purposes of seeing "can a human do this at all" I could also expand this to people I know who've generated novel concepts within the past few years.
Actually, I guess what I should do first is check that "person invents novel concept, in a way that an LLM couldn't obviously do via brute force" is a thing that's even happened ever in history.
<thinks for 5 seconds>
Okay I have no idea whether this is even real, I guess I should also write another post like "has a human ever done conceptual thinking that LLMs can't currently do, even if they are currently very slow at it?" and demand examples from the LLM-is-all-you-need skeptics.
People who solve puzzles in puzzle games have "aha!" moments, which don't generate new concepts but often do involve the person suddenly realizing a concept that's new to them.
"Archimedes' Eureka Moment" is maybe a good prototypical example of an "aha moment." I don't know if actually required pieces LLMs don't have, but, seems like it should be above the bar I'm setting. If I were cheating, could I use that?
And I guess the canonical "aha!" moment is the legend of Archimedes inventing buoyancy by sitting in the water and then going "Eureka!". What happened there?
Reviewing the summary at this site:
Could the LLMs solving Millenium problems figure this out, if they managed to somehow start from Archimedes' knowledge?
I think that "displaced water has a volume you can measure" is a distinct realization from "Any object, totally or partially immersed in a fluid or liquid, is buoyed up by a force equal to the weight of the fluid displaced by the object." I'd have to think awhile to do the Insight Unpacking archaeology here.
I think Archimedes' situation is a reasonable example of the phenomenon we care about (actually it is kinda cool that humanity has a legend about... the first guy to have a Eureka moment?).
Game built around Bouyancy + Feedbackloop Rationality? Is that anything?
One next question might be "if I got to start from Archimede's experience + my game dev experience, could I make a fun game about bouyancy and/or displacement measuring?". Could I find a second concept to interweave with it? Could I make a game out of that that was any good?
I think "obviously yes" although it might take awhile to dig for an artful mechanical and thematic premise to make it not feel like an artificial assignment.
How about let's say "we are making a game about the intersection of bouyancy and 'needing to build better feedback loops.' So we have one thing that's clearly a novel concept and one thing I personally came up with even if it's more of combining obvious things.
Actually, I notice: when I explain a lot of the specifics of my feedback loop rationality stuff to people, often they are confused and not convinced right away by the particular combination of ways I approach it.
This implies there is something nonobvious I believe (rightly or wrongly). Maybe that something is concept shaped.
[Aside: I'm going to race ahead with my current thoughts but readers will probably be confused. I think there's not really a shortcut for reading Feedbackloop-first Rationality and The Cognitive Bootcamp Agreement and maybe Scaffolding for "Noticing Metacognition"].
Early on when I was pitching Oliver on it, Oliver said something like "ultimately everything is just G, I think you should just be trying to raise G, but also lots of people have tried it and it seems impossible. And I don't really think the stuff you're saying seems like it would raise G."
And I was like "well, yeah, I didn't think I could raise G, I thought I could do this other less ambitious thing that seems pretty good."
"What is that thing?"
"uh..."
And then I thought awhile and was like "well, I mean 'rationality'."
"But, like, what is that?"
Okay, time to actually get clearer on what rationality is and how you would measure.
And this maybe does count as concept I articulated de novo?
Oh right, the concept I did generate myself that along the "Feedback loop rationality" journey was hypothesizing a meaningful "s factor[2]" (slow intelligence) as opposed to "g factor". IQ tests are optimized for speed efficiency. It's plausible we don't test 'general strategy' because it's just too time intensive.
From Rationality Research Report: Towards 10x OODA Looping?:
Is this a new concept? Well, it's still built out of obvious building blocks but it was a moment in my work where I felt like I wasn't just applying existing thought patterns and ideas in a procedural way.
I guess Subskills of "Listening to Wisdom" also was such a moment.
Is... "measuring s-factor" anything, as a game? Hmm, not very obviously.
Like, there is an obvious game you want to build to measure s, which is "procedurally generate infinite puzzles of many varieties and domains." That game is probably even fun. But, it's the game itself is not about "s".
Slower paced puzzle and strategy games exist.
Is "listening to wisdom", as a game, anything? ...maybe? But, a very hard one.
I think you could build a game around learning to listen to wisdom, but, intuitively it feels like it'd be a Magnum Opus I spent ten years on before figuring out how to make it work.
Somehow, you'd need the basic premise to involve a game that has long, hard path that burns various painful life lessons into you (like, life lessons that apply to playing the game. Like, learning to override certain intuitions)
But, at each point, there are characters or warning signs that, you will need to override your intuitions and do something different that you don't really feel like doing.
Somehow, there are multiple loops of this whole thing, such that you don't just play the game, and then get told "see, if you had listened to the Old Wise Man at the beginning you could have bypassed all this shit." You actually learn to listen to wisdom.
Can that be... fun?
It's a kinda herculean amount of work, because for this to be a game and not "edutainment that isn't really fun but it slightly better than the second-least-painful way to learn a thing"...
...we need each big long painless lesson arc to manage to be fun basically the whole way through, but, also, the alternative you could learn is somehow an improvement.
Make something... only kinda fun, and then "you could have been having more fun?"
eh, yeah I think this could work if I put a ton of effort in but it's not the natural shape of a game.
Is "Bouyancy" tactically deep?
This is the part where if I were doing this for real I would bust out an LLM to help me, but, I'm doing this for team humanity right now.
Ugh, I think it's a fair amount of effort for me to understand buoyancy enough to get a sense of how much you can make, say, a buoyancy puzzle game, or a strategy game where buoyancy is a background mechanic.
Hmm. Osmos is "the conservation of momentum" game, and it's pretty good. It works via there being a core mechanic of "spew out stuff to move, and make sure you don't spew out too much stuff so you don't run out of stuff."
Does buoyancy involve... decisions? What are simple decisions you can make that are "about buoyancy" that are viscerally fun enough to do 1000 times in a row?
A floating citybuilder, where there are different sorts of solid and liquid materials you need to work with to accomplish different goals as you keep your city afloat while doing other stuff?
Okay, I had been about to run out of fucks but actually that seems kinda cool.
Also...
How do we make feedback-loop-making fun?
Man I have a decent idea but I am running out of fucks. But, I was about to run out of fucks and declare it over when I suddenly immediately came up with "Bouyancy Town" which seemed pretty good.
So let's at least try.
Central problem with feedbackloop games? Giving players the right materials/systems that let them construct meaningful feedbackloops, that are the right blend of emergent and on-purpose, and are actually fun to build.
I feel like mostly I need to brute force generate a bunch of ideas and tinker with them. But, okay, can I come up with one working example of a thing that would be fun to do 1,000 times in a row?
What properties do Feedbackloops have to experiment with? They can be faster, richer, or more precise.
The simplest variant is "how fast and cheaply can you make a tower collapse, in a way that lets you learn some basic information?"
Okay, making towers fall down on purpose seems like it could be fun.
Is there anything to do about making richer or more precise loops? Or, making mistakes more recoverable so you don't lose a whole tower, but start watching it begin to buckle, and then you can rescue it?
idk that feels satisfying enough to end the timebox on for now.
Okay, I landed back on "Buoyancy + Feedbackloop-building" as the combo.
Neither of those score "amazing, slam dunk" on completing the assignment. I didn't invent buoyancy, and "get gud at making feedbackloops" isn't original in quite the way I meant. But I don't actually know of anyone else really talking about feedback loops this way.
This at least feels like it touches every major piece of what I'd want the assignment to check. Let's run with that.
Aside: I asked Claude who talks about feedbackloop-making as a skill. It said "well, mostly Ray Arnold and Logan Strohl, when you frame it specifically like that." When I asked Deepseek, it said "mostly these other guys you've never heard of."
Claude:
Deepseek:
Okay, game concept: Fluidtorio
You are building a series of factory colonies that float in different types of fluid. You build little modules that need to stay afloat, but, also accomplish various other goals (i.e. be a factory that processes some materials that you need to build other tools to build more modules to advance to new zones with different fluids to contend with).
Your town floats on a layer of ~oil over a layer of ~water (or equivalent), and above you are a few layers of atmosphere that do different things. Ascending up and down is part of the game.
The actual materials – both the fluids, and the stuff you need to build around them, are procedurally generated each game, so you don't know exactly how they interact.
Things are setup such that you will eventually need to build relatively large, dense towers to accomplish stuff, and somehow thread the needle on "stay afloat, while doing complex engineering challengers." Kinda factorio style.
If you naively build towards your goal, your towers are going to collapse.
You are on a time limit before a volcano explodes or something. You do not, by default, have enough time to brute force trying the different combinations to escape the planet. So, a central challenge is figuring out how to rapidly test materials to figure out what your limitations are, and then planning what series of things to build to bootstrap to the rocket age in time.
This includes buoyancy/weight/factory-processing-output-balancing stuff, where your towers and stockpiles might collapse. There also needs to be some kind of crafting system where stuff explodes or doesn't work right, in a way that is reasonably procedural so it's different each time but has enough structure to reliably make sense and be fun and feel tactile.
There can also be feedback-loops vis-a-vis "where to explore to find resources."
That's how far I got before running out of timebox. Okay. Making that seems nontrival, but, that feels like a fair benchmark to hold an AI against.
(I went and had Opus take a first stab at implementing this. It missed a bunch of subtleties of what I think it needs to be a good game, but, I did verify the basic "balance a floating factory is pretty fun" idea. I am kinda interested in fleshing out the prototype someday)
Games that were inspiration
All Will Fall, where you are building a town on an ocean with realistic physics for the towers that fall
Skyrig, where you are building a rig on a gas giant that needs to descend floating balloon structures deeper into the storms
Factorio needs no introduction.
Did the AI understand the assignment?
I followed up with the first AI, asking some questions about the assignment. You can read it's answers in detail...
What do you think "a concept" is, in this context?
Here’s how I’d define a concept, and how my results measure against it.
My working definition. A concept is a reusable unit of thought: a way of carving up situations that lets you see new cases as instances of it and draw inferences you couldn’t easily draw before. A good one has three properties:
How it differs from nearby things:
What I actually optimized for. The prompts I wrote for the helpers turned “concept” into roughly “a new kind of world-law that forces a new mental operation.” That’s narrower than the definition above, and it steered the search toward strange physics. Judged by the stricter definition:
What would dramatic success look like?
For this challenge, I’d say dramatic success is a concept that outlives the game. People keep using it after they forget the puzzles, because it names something real they couldn’t think about cleanly before.
What you’d observe:
Historical concepts at that level: the tragedy of the commons, Schelling points, Goodhart’s law. In games, Portal’s “speedy thing goes in, speedy thing comes out” and Baba’s rules as objects changed how a generation of players and designers think about puzzle space.
Against that bar, my result:
So my honest expectation is modest success: novel as a game, maybe a borderline concept. Dramatic success would look like you, unprompted, calling something on LessWrong a “guardian wall” six months from now, with that being the clearest way to say it.
Do you have thoughts on what you could have done differently? (looking at the earliest committed notes?)
Reading
brainstorm-prompts.mdandselection-notes.mdagain, most of what I’d change was decided in the first prompt I wrote. The search was set up to find strange new physics for a puzzle game, not new concepts.1. I put the game into the concept search too early. Every agent ranked its ideas by “novelty × fun × playability in a 2D browser puzzle game.” That rewards ideas that make good mechanics, not ideas that change how you think. A better order: first find concepts that pass a transfer test (does it explain real cases outside games?), then ask which ones can become a game.
2. My definition of “concept” was too narrow. Five of the eight strategies asked for new entities, laws, minds, formal structures, or ways of depicting the world. So most of the output was new physics. Only two strategies, unnamed real patterns and craft transplants, looked at how the real world works. Those two produced the ideas closest to real concepts: Earned Selfhood, and the “frozen indexicals” idea (names like “New College” that outlive the frame they were coined in). I gave both of them less weight.
3. My novelty test favored obscurity over insight. I told the agents to “assume anything resembling a known idea will be called out.” I also gave them a long list of known ideas to avoid, and had them search “the idea + game”. That rewards ideas you can’t easily find, not ideas that are true and useful. “No game does this” is weak evidence of a new concept. A better test asks whether experts in the nearest field, such as philosophy of causation or economics, already have a name for it.
4. I misread a correlated signal as evidence. At 21:34 I wrote that five agents had “independently” reached counterfactual physics, and took that as a sign the area was under-explored. But they were all the same model given the same framing. Convergence is more likely a shared attractor, a place this model drifts toward by default. It may even mean the area is over-explored by models like me.
5. Timing and momentum shaped the choice. I was prototyping Fatefall within ten minutes of it arriving, and I decided at 21:46, two minutes before my note confirming the last report. The first idea that got built gained momentum. Fairer: wait for all eight reports, then prototype the top three or four briefly before choosing.
6. I chose for combinability over substance. My main reason for the pair was that it produced an emergent third concept. That is what you asked for. But it ranks two decent ideas that combine well above one strong idea that doesn’t.
7. I didn’t ask what counts as a concept. Your draft says you’re “not 100% sure what the assignment was”. One short question at the start, something like “a lens that transfers, or a weird premise that’s fun to inhabit?”, would have changed the whole search.
8. My early thinking left no trace. The first 37 minutes produced the framing that mattered most, and none of it is readable. Writing short notes to a file as I went would make the process inspectable, which matters for an eval like yours.
If I reran it, I’d ask for real but unnamed patterns, test them against real cases and against experts in the nearest field, and only then design a game around the one or two that survive.
...but in summary it said:
I asked it to write up a brief for another AI to try again. I ended up editing to brief to emphasize "also, please make it fun." Another AI thought for awhile, and ended up with:
I'm... not 100% sure if those concepts make sense or carve reality in a useful way. And, rather than "invent a new concept from scratch" it tried more for "find existing unnamed phenomena that are worth having a name for."
This isn't as potent as the cognitive work I was looking for. But it feels at least related? I feel like it at least did a pretty decent pass at the assignment I actually gave it.
Fold-back signal is a mediocre name but seems like a decent concept to want a handle for. (Another Claude-instance said there isn't a canonical name that covers all three examples, although different fields have some names it as applied to their field and gave it "15-25% novelty". The other Claude-instance also listed "foldback signal" as one suggested name, although also suggested "self-silencing alarm" which is way more intuitive to me.)
I found it's next actual game a bit confusing and un-fun.
I told it "Also, alas, this game is neither fun nor comprehensible, please try again with the game design part", and gave it a third try.
I found it's third game comprehensible, not super fun but at least I understood roughly where it was going for and could imagine it being fun-ish if tuned.
Do they understand "fun?"
Opus 5.5 is not near "top expert level" at Game Design. I'd say it's got, like, the taste of a high school kid just getting started coding some games, who's read some Mark Rosewater essays and has the gist of how to make a game. Or, I guess more realistically, it's like if Twilight Sparkle or Hermione read up everything there is to know about "fun" overnight and then made a game, and, like, Twilight Sparkle and Hermione are quite competent. Twilight Sparkle can throw a sleepover on her first try and it's not perfect but it's way better than you might have expected.
It does feel like it has all the pieces it needs, to get better at taste with straightforward within-paradigm scaling. Probably OpenAI/Anthropic's current secret frontier AI models are able to make a medium complexity game with some interesting taste/tactics on their first try, even if they are still missing some subtleties of "fun."
AIs I've talked to definitely understand fun in a pretty rich way, as you'd expect of Twilight Sparkle after she reads up on it. I asked Fable once to formalize fun and try to make a genetic algorithm for making a "fun" world to walk around it. On one hand, it didn't work, and failed in some weirdly dumb ways. But, it's initial responses were like [me paraphrasing from memory, aggregating some things that had a bit of prodding from me in between]
And I'm like... okay yep I think that is approximately what fun is.
They seem like Mary from Mary's Room, who knows all the neuroscience and color theory necessary to tell you anything you could possibly want to know about the color red, but has never actually seen the color red.
I didn't successfully get it to build a good walking simulator before I gave up and moved on, but I think if I gave Opus 5.5 more compute it might work (I've since set up a Modal account, so the genetic algorithm it was trying could actually scale sufficiently).
I do bet Fable 6 will be able to do it pretty fine.
⚖ Fable 6 [or obviously equivalent thing] will be able to 1-3 shot a procedurally generated walking simulator that makes something interesting and delightful to me using what seems like a reasonably operationalized metric of Fun. 63%
Can they manipulate and derive novel concepts?
It's not very hard to get AIs to generate ideas. It's hard to get them to generate new interesting ideas. The main thing I think AIs are missing is taste in what concepts and tools are appropriate, when they are in uncharted territory.
They didn't demonstrate being better at that in any general kind of way during my explorations last week. But, a specific skill I hadn't thought to ask about, which it does seem like they have, is:
"Probe the breadth of human knowledge, looking not merely for existing concepts, but for commonalities that might warrant having a new concept-handle."
I expect, when offscript on openended problems, they won't be very good at finding new concept-handles that actually help them. But, when I imagine a 10,000-thread swarm solving Millenium problems, it does seem like "probe the concept-space and see if any of it feels helpful to crystallize into a new concept, and see if that concept helps" is the sorta thing they outta be able to do?
Another thing they are missing is "Gain new system 1 intuitions on the fly, in the middle of solving a problem, so you can apply them much more efficiently." I'm kinda confused about the limits of continual learning, but, "spin up a copy of the latest open weight model and fine-tune it on a concept-handle you just came up with by brute force and then assign that model work" seems like the sorta thing they outta be able to do.
Generalized Taste
What has confused me for the past two years is... it really feels viscerally obvious to me that generalized "keep track of all your goals, and evaluate whether you have everything you need to succeed at your goal, and relentlessly probe for alternative frames and evaluate each of those too", is not that different-in-kind from the sort of stuff AIs are already able to do.
But, my intuitions predicted they would already be able to do that, and it seems like they keep getting really confused when going off-script in a way that seems like they don't got it yet.
But, like, we've clearly seen Claude be able to apply good software architectural taste, that it doesn't bother to apply when it gets down to actually build things. We've also seen it get better at overall architectural taste when building things over time.
Opus 5.5 clearly knew what a concept was and what the assignment was when I asked it directly.
I still bet on LLMs eventually discover generalized "ask themselves the obvious fucking question and then do the obvious fucking thing", probably sometime in the next year. But I am less confident about it than I would have been two years ago. I don't know if that's all you need for ASI, but, I'd bet we see a step-change when they do.
(I observe that OpenAI employees seem freaked out about AI capabilities having spiked rather suddenly, and it's plausible that the Highly Persistent Internal Model and/or Mythos 6 already have it)
In the life before I was a rationalist, my primary identity was being a game designer. (Obviously you can be both, in practice I ended up with different interests once getting into rationality)
I aspired to become a professional, and did not. But, some cred: during the first Magic the Gathering Great Designer Search, of 1000 applicants, I made it into the top 64.
Coming up with "s-factor" felt like one of the rare times during the Feedbackloop Rationality where I did some cognitive labor that didn't feel like just mechanically combining deas.