Seems a lot harder to write a post a day if one is not holed up in Lighthaven. 

Heard that story many times by or about exchange students to the US.

What gives you the impression of low integrity?

There's an interestingly pernicious version of a selection effect that occurs in epistemology, where people can be led into false claims because when people try to engage with arguments, people will drop out at random steps, and past a few steps or so, the people who believe in all the arguments will have a secure-feeling position that the arguments are right, and that people who object to the arguments are (insane/ridiculous/obviously trolling), no matter whether the claim is true:

I find this difficult to parse: people, people, people, people, people. 

These seem to be at least three different kind of people: The evangelists, the unconvinced (who drop out) and the believers (who don't drop out). Not clearly distinguishing between these groups makes the whole post more confusing than necessary. 

The function of the feedforward components in transformers is mostly to store knowledge and to enrich the token vectors with that knowledge. The wider you make the ff-network the more knowledge you can store. The network is trained to put the relevant knowledge from the wide hidden layer into the output (i.e. into the token stream). 

I fail to see the problem in the fact that the hidden activation is not accessible to future tokens. The ff-nn is just a component to store and inject knowledge. It is wide because it has to store a lot of knowledge, not because the hidden activation has to be wide. The full content of the hidden activation in isolation just is not that relevant.

Case in point: Nowadays the ff-nns actually look different than in GPT-3. They have two hidden layers with one acting as a gating mechanism: The design has changed to allow the possibility to actively erase part of the hidden state! 

Also: This seems very different from what you are talking about in the post, it has nothing to do with "the next run". The hidden layer activations aren't even "accessible" in the same run! They are purely internal "gears" of a subcomponent. 

 It also seems to me like you have retreated from 

with its intermediate states ("working memory") completely wiped.

to "intermediate activations of ff-components are not accessible in subsequent layers and because these are wider than the output not all information therein contained can make it into the output".

What I was pointing to was the fact that the feed forward networks for the new token don't have access to the past feed-forward states of the other tokens [...] When curing cancer the second time, it didn't have access to any of the processing from the first time. Only what previous layers outputted for previous tokens.

That is the misconception. I'll try to explain it in my words (because frankly despite knowing how a transformer works, I can't understand Radford Neal's explanation).

In the GPT architecture each token starts out as an embedding, which is then in each layer enriched with information from previous tokens and knowledge stored in the nn itself. So you have a vector which is modified in each layer, let's call the output of the $n$-th layer: $v_n$

The computation of $v_n$ accesses the v${}_{n-1}$ of all previous tokens! So in your example, if in layer $n-1$ at some token the cure for cancer is discovered, all following tokens will have access to that information in layer $n$. The model cannot forget this information. It might never access it again, but the information will always be there for the taking. 

This is in contrast to a recurrent neural network that might actually forget important information if it is unfortunate in editing its state. 

I think even in the case that AI 2027 is directionally correct (very fast AI progress) the concrete details are likely to be wrong, so I'm not sure how impressed one should be if your predictions turn out to be correct.

About "it's all just vibes": AI 2027 is strongly based on the METR time horizon analysis. I think it would be more fruitful to critique and analyse that. Stuff like the time from SC to SAI seems like epicycles. Though the biggest uncertainty in AI 2027 probably comes from the assumption of recursive improvement. 

I am not sure how fruitful the "shallow vs deep thinking" terminology is. What you explain in more detail is what I call "knowledge integration" and "learning while problem solving" which is both about humans having more powerful representations that can be modified while mulling stuff over and improved by integrating data from other domains. 

Your algorithmic explanation for LLM shortcomings seems to be wrong and based on a misunderstanding of how LLMs work: 

As joseph_c already mentioned the human brain (as an nn architecture) is much, much wider and shallower than a GPT. One of your examples, coming up with clever jokes, also doesn't require enough time for humans to engage in a lot of recursive thought. 

Also, LLMs do actually keep the entire earlier state around, that's what the KV-cache is. The computation of each new token does access the fine-grained vector representation of earlier tokens. There is no memory wiping going on. 

I think the opposite is correct: LLMs are not nearly wide enough. As a consequence their representation of the "the problem" or "the situation" is impoverished.

I think this insight is really interesting! Especially the potential connection to LLMisms. 

But I don't really understand why you chose these experiments. It seems to me the things to check or prove are:

1. current tokenizers do actually tokenize typical training data so that short tokens are more common
2. current models do produce text that recapitulates this bias
3. how the k for topk-sampling exacerbates this bias depending on k
4. how this changes some typical completions

You do significantly more work to show the effect in a toy setting that may or may not bear on the real case. And I think the outcome of your experiments is already clear before you do them because the effect of top-k sampling on tokens with low/high probability is not complicated (and well explained by you in the post). 

because tokens are too low bandwidth

That's also my impression: https://www.lesswrong.com/posts/KrgBkqeChtAWuPsLP/what-llms-lack

The 4-month doubling trend implies getting 8h+ horizon length by early 2026 and an order of magnitude more by mid-2027. If the best time horizon length in mid-2027 would be 9h, would you feel like you had won the argument, even if you had won the bet?