One possible interpretation here is going back to the inner-monologue interpretations as being multi-step processes with an error rate per step where only complete success is useful, which is just an exponential; as the number of steps increase from 1 to n, you get a sigmoid from ceiling performance to floor performance at chance. So you can tell the same story about these more extended tasks, which after all, are just the same sort of thing - just more so. We also see this sort of sigmoid in searching with a fixed model, in settings like AlphaZero in Hex, which makes sense if we assume that these LLMs are doing a lot of retries and backtracking, which constitute a 'search' process as a whole, even if they never explicitly represent or model a decision/game tree, and have error rates stemming from their blindspots and biases. And you can tell a similar story there about error rates and exponentials: all the critical steps have to be right (omitting ones which don't do anything, ones which get undone or reset, etc), and the final result is either right or wrong as you do the task or not.

(And on a more detailed mechanistic level, you can tell a story where NNs learn 'atoms' of skills over scaling, power-law distributed in random naturalistic data, which are recombined to solve each 'new' inner-monologue problem, and if you have 'memorized' enough atoms, you can solve every task which is just a reconfiguration of known atoms, and that is just what 'learning' and 'generalization' are.)

But of course, the interesting thing here is that the human baselines do not seem to hit this sigmoid wall. It's not the case that if a human can't do a task in 4 hours there's basically zero chance of them doing it in 48 hours and definitely zero chance of them doing it in 96 hours etc. Instead, human success rates seem to gradually flatline or increase over time, especially if we look at individual steps: the more time that passes, the higher the success rates become, and often the human will wind up solving the task eventually, no matter how unprepossessing the early steps seemed. In fact, we will often observe that a step that a human failed on earlier in the episode, implying some low % rate, will be repeated many times and quickly approach 100% success rates! And this is true despite earlier successes often being millions of vision+text+audio+sensorimotor tokens in the past (and interrupted by other episodes or tasks themselves equivalent to millions of tokens), raising questions about whether self-attention over a context window can possibly explain it. Some people will go so far as to anthropomorphize human agents and call this 'learning', and so I will refer to these temporal correlations as learning too.

Why the difference between machine and human learning? Well, you might ask, given this sigmoid wall, how did we get so much higher performance from GPT-2 to Claude-3.7? How did o1-style models go from flailing about to far higher performance on coding/reasoning tasks even at the same size model? And how did we go from below amateur Go AI (AlphaZero at the start of training) to strongly superhuman Go AI (AlphaZero at the end of training), with the same size model? The shocking but true answer is... we trained better neural networks. (And larger too, of course, but that was not strictly necessary.) We didn't prompt them or do brute-force best-of-n samples search or even MCTS search a (randomly initialized) model or use a really really large context window on GPT-2. But we trained them, so they could learn new and better stuff. (Another way one could make the point: if self-attention really is a perfect substitute for gradient descent on the weights, and there is no crossover point, why do we not just 'train' models using purely linear self-attention on trillions of tokens, and use that instead? Why does anyone still bother with, say, finetuning instead of putting that dataset into the context and caching it?)

Incidentally, what do GPT-2, GPT-4, and Claude-3.7 all share in common, that is not just untrue, but nearly impossible for a human doing a task? They have frozen weights which do no learning at runtime.

So I would suggest that the sigmoid we see here is mostly what we would expect from using a frozen non-learning model to do search over a difficult game/task, and that if the LLMs were able to properly learn using finetuning (or an online equivalent like dynamic evaluation), you would see different and more human-like temporal scaling: where the success rate declines more gradually and plateaus at a higher asymptote, as within-episode, it observes poorly-modeled environment dynamics and improves its predictions of those, observes its errors and avoids repeating them in favor of new things, knows what it has and hasn't done without having to reason over the entire history (filled with false starts and errors), and can explicitly reason about things and incorporate the results of the reasoning directly into the weights computing everything else.

See also: ARC, Claude Plays Pokemon.

While it's not possible to counter-signal with a suit in Japan, I feel the equivalent would be to wear traditional clothing like a samue or jinbei, which have their own set of challenges.

Yep. It can be pretty funny watching the contexts in which you can get away with a happi coat or a kimono/yukata; I can only speak from Japanese media rather than personal experience, but one thing I've noticed is that it seems a non-retired man wearing a kimono can still get away with it today as long as they are a sufficiently accomplished humanist or literary scholar (but not STEM). It reminds me of the 'tweed jacket' professor archetype here: you can still get away with wearing a tweed jacket with leather patches etc, but you'd better be a professor or a novelist or something of that ilk if you don't want to be quietly judged for it.

Though now that I think about it more, presumably once someone has been captured the next thing you'd get them to do is spend a lot of time staring at a region of the sky that will reprogram them in more sophisticated ways. So maybe the normal glitchers in my story are unrealistically incompetent.

That was what I was thinking, yes. "A pact would normally allow voluntary communication to be initiated with the AIs, so any glitcher which had been successfully attacked would have simply communicated back to its masters, either downloading new instructions & attacks or finetuning the existing ones or being puppeted directly by the AIs, sometime over the past centuries or millennia; if nothing else, they have an unlimited amount of time to stare at the sky and be reprogrammed arbitrarily after the initial exploit; so glitchers are indeed 'glitchy' and must represent a permanently failed attack method. That is why they bumble around semi-harmlessly: a broken worm or virus can cause a lot of trouble as it futilely portscans or DoSes targets or goes through infinite loops etc, even if the code is buggy and has accidentally locked out its creators as well as everyone else."

I never heard of that, do you have examples?

My local gym has posted rules which include an explicit ban on perfume. (They don't use the exact term 'scent-free' but I assume it is an example of what OP means.)

Not that they enforce it, or even could enforce it; but I am reminded that rule exists every so often a woman (and it's always a woman) walks past me when I'm there at night, and I am suddenly hit by the smell (especially as I don't think of myself as being particularly perceptive nose-wise and I don't usually notice how people smell), and I wonder to myself if they put on fresh perfume just to go to the gym (some of the young women clearly 'dress up' for the gym) or if they just put on that much perfume in the morning.

I've never seen that SpongeBob gag either. But Mr Bean is a real person and people do have perfume sensitivities and allergic reactions. (My father had an ugly clash at work with one woman who apparently wore a lot of perfume and he was convinced was causing him headaches and other problems.)

I saw Mr. Bean nearly die at the perfume counter.

To clarify, this is a fictional example and not a personal anecdote in desperate need of unpacking: https://mrbean.fandom.com/wiki/The_Return_of_Mr._Bean#Act_Two:_Shopping

if you take his general 'cognitive strategy' and just power it with worse thinking, you get really bad results.

I call this "retired physicist syndrome", after the classic SMBC cartoon: https://www.smbc-comics.com/comic/2012-03-21

Climate models aren't reductionist enough!

We can recreate the first language using statistics!

Awoooo! Oncology is doing it wrong!

It can be bad to be Open but not also smart. (You could also call this "emeritus professor syndrome" in many cases.)

A hack like that would just have other EDT failure modes: instead of confabulating evidence from my dataset or personal examples, it might just confabulate references. "Yes, this was predicted by Foo et al 1990, and makes perfect sense."

I think it's substantially better and calmer - even just the thumbnails look calmer now.

I still think you are going a bit overboard on visual complexity, things like slashed-zeros aren't bad (I like them), just too much of a good thing and using up your visual complexity budget where there may be a better deal elsewhere: the question I ask myself is, "do I want to look at this for the next 20 years? If I add X, will it 'look painfully 2025' in a few years?" Elements which don't seem excessive in the excitement of implementation may, with the passage of time, gradually come to rub you the wrong and need to be sanded down or removed. If you aren't careful, you may slowly, one accretion at a time, come to resemble a site like "Agora of Flancia": encrusted with neat experimental features which are too half-baked to be genuinely useful, but which burden and deter readers. (I try to avoid a reluctance to 'kill my darlings' by cultivating the mindset that every addition is an experiment, and thus if I have to remove them, they're not wasted, they're just a new 'Design Graveyard' writeup, is all.) But I suppose you'll find out your own preferences on this matter over the next few years as the novelty wears off.

I want to do something about the desktop logo animation being distracting. I don't know what that is, yet. I can't play/pause the GIF on hover because GIFs don't allow that (AFAIK). I'll probably find a way to move it to a WEBM while also making it autoplay across browsers, at which point I can implement the feature.

I also still think that the logo should probably not play by default, and for animations like this, it's better to take an Apple-like attitude about them being enhancements, opted into by user actions, to 'spark joy', but not to be used by default. What do the worst websites do? They animate tons of stuff gratuitously. How much more delightful it is to discover a website with taste & restraint, where there are easter eggs and features to discover as you surf, where, say, the animated logo plays only when you hover over it... Truly an oasis or quiet little pond amidst the howling desert of the contemporary Internet. (I'm reminded of a Family Guy meme I re-ran into recently: why does Peter Griffin dislike The Godfather? Because "It insists upon itself." A website animating the logo unasked for insists upon itself.) And this helps instill a design feature: you the reader are in control, and you express this control in part because you can hover over everything to learn more or focus on some things.

However, if you insist upon it, perhaps you could reduce its impact by some sort of limitation. Let it cycle a few times or seconds, and then slow it down or fade it or stop it. If the reader hasn't appreciated it by then, why keep flickering it in the corner of their eye? Another idea would be a site-wide limitation on animation: on Gwern.net, we have a 'demonstration mode' feature which tracks how many times something has happened / been shown, and changes it (usually to disable it) after n times, tracking n site-wide by using a cookie in LocalStorage counting that particular event. We use it to do things like simplify obtrusive text labels, or to disable the educational theme-toggle-bar animation after a few animations.

Also, have you considered animating the dark-mode setting to move the sun/star around? The sun over the pond to the right, then the moon to behind the castle and swap the palette for a 'night' palette. (The 'night' label can be omitted, typeset vertical, or you can do it by horizontal mirroring of the whole image.) If you don't do something like that, that would be a real shame. The perfect twist on dark-mode for The Pond.

like the ability to listen to AI-generated readings in my voice (for the less math-y articles)

This sounds like it would make for a better project & writeup than actual long-term website feature intended for actual reader use. Is this something readers want or will use? It never once occurred to me that I might want a spoken version of your articles (human, AI sexy-Scarlett-Johansson, or AI-Turntrout - ok maybe Scarlett). These aren't poems or autobiographical essays or even fiction, where there can be value to hearing the author read it. (Although even there most of the reasons to want that are ones where an AI can't substitute: if you want to hear the author's unique intonations or pacing for subtle insights into the text or the author*, obviously an AI model voicing it is useless.)

On a sidenote, ahem, you might want to rethink how the popups work. I was unaware you even had popups, until I noticed in the network tab that a lot of pages were being prefetched when I hovered over them. It might be an oversight to prefetch if you aren't doing a popup. But regardless, since I'm on a fullscreen 4k portrait monitor, and I have to zoom out before I can see popups at all, you may have gone overboard in your width requirements.

* when I questioned the value of Dwarkesh Patel et al investing in making a spoken version of our interview, this was given as one of the reasons that people valued audio over the transcript. We split the difference by the voice actor, Chris Painter, having the audio and performing it. It's uncanny for me to listen to.

It’s about grief, with central metaphors that add exactly zero to anyone’s aesthetic understanding of grief (stuff being underground, things not staying buried)

It is about grief, but it didn't have to be. This would've been more obvious if I could've shown you the session, but I'll copy it out:

2. Brainstorming Ideas:

• A child confronting a local superstition after witnessing something traumatic.
• A funeral narrated by an animal's perspective.
• A celebrity’s fall from grace caught on live camera.
• A girl who collects superstitions until one unexpectedly comes true.
• A gardener who secretly buries cursed objects.
• Friends reuniting silently after a crime.
• A model who steals other people’s identities through makeup.
• A ghost attending its own funeral, confused.
• Farmers discovering an ominous black stone.
• Neighbors noticing unsettling coincidences.
• Someone who compulsively invents last messages for strangers.
• A woman discovering her double living her unlived life.
• An eerie superstition circulating quietly through a town.
• Villagers silently celebrating the death of a tyrant.
• Two women performing friendship publicly, enemies privately.
• Someone obsessively organizing goodbyes.
• A funeral revealing hidden grudges through seating arrangements.
• Friends trapped in a toxic nostalgia loop.
• A boy who accidentally predicts deaths.
• An abandoned motel that traps memories.

3. Selection

Idea: A gardener who secretly buries cursed objects.

Rationale:

• Symbolic and psychological depth.
• Opportunity for visual and thematic resonance.
• Tension around secrecy, superstition, and consequences.

Now that I look at it, the 5 flash stories I happened to copy into seem to mostly hit on themes of death & grief (the last one is literally titled "obit"!), so I think you are unfairly dinging 4.5 here - if 'good' flash fiction, by your own standards, keep revolving around death and grief, how can we blame 4.5 (or r1) for doing likewise?

Anyway, some of these ideas seem promising. "A funeral narrated by an animal's perspective" is one I like, I don't think I've ever seen that.

And of course, if the failure mode is so common, throw it into the prompt. (When I yell at 4.5 to avoid grief/death/funerals and brainstorm some more, it picks out '"The Parking Attendant Matchmaker": A seemingly ordinary parking attendant quietly manipulates parking assignments at a large business complex to engineer chance encounters and romances among strangers.' Yeah sure why not.)

Like, what does it possibly mean for mourners to “trust my silence” here. What is it they’re trusting? How does the earth’s hunger contrast to that?

Balderdash. There's a lot to criticize here, but you're straining to come up with criticisms now. That's possibly the least objectionable sentence in the whole thing. If this had been written by a human, you wouldn't hesitate in the slightest to accept that. It is perfectly sensible to speak of trusting the confidentiality of a confessor/witness figure, and the hungry earth is a cliche so straightforward and obvious that it is beyond cliche and loops around to ordinary fact, and if a human had written it, you would have no trouble in understanding the idea of 'even if I were to gossip about what I saw, the earth would have hidden or destroyed the physical evidence'.

I also see it a lot in the ClaudePlaysPokemon twitch chat, this idea that simply adding greater situational awareness or more layers of metacognition would make Claude way better at the game.

I do agree that the Claude-Pokemon experiment shows a limitation of LLMs that isn't fixed easily by simply a bit more metadata or fancier retrieval. (I think it shows, specifically, the serious flaws in relying on frozen weights and refusing to admit neuroplasticity is a thing which is something that violates RL scaling laws, because those always assume that the model is, y'know, learning as it gains more experience, because who would be dumb enough to deploy frozen models in tasks far exceeding their context window and where they also aren't trained at all? - and why we need things like dynamic evaluation. I should probably write a comment on that - the pathologies like the deliberate-fainting are, I think, really striking demonstrations of the problems with powerful but frozen amnesiac agents.)

I'm much less convinced that we're seeing anything like that with LLMs writing fiction. What is the equivalent of the Claude pathologies, like the fainting delusion, in fiction writing? (There used to be 'write a non-rhyming poem' but that seems solved at this point.) Especially if you look at the research on people rating LLM outputs, or LMsys; if they are being trained on lousy preference data, and this is why they are like they are, that's very different from somehow being completely incapable of "extracting the actual latent features of good flash fiction". (What would such a latent feature look like? Do you really think that there's some property of flash fiction like "has a twist ending" that you can put two flash stories into 4.5 or o1-pro, with & without, and ask it to classify which is which and it'll perform at chance? Sounds unlikely to me, but I'd be interested to see some examples.)