This is a special post for quick takes by Mo Putera. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Over a decade ago I read this 17 year old passage from Eliezer
When Marcello Herreshoff had known me for long enough, I asked him if he knew of anyone who struck him as substantially more natively intelligent than myself. Marcello thought for a moment and said "John Conway—I met him at a summer math camp." Darn, I thought, he thought of someone, and worse, it's some ultra-famous old guy I can't grab. I inquired how Marcello had arrived at the judgment. Marcello said, "He just struck me as having a tremendous amount of mental horsepower," and started to explain a math problem he'd had a chance to work on with Conway.
Not what I wanted to hear.
Perhaps, relative to Marcello's experience of Conway and his experience of me, I haven't had a chance to show off on any subject that I've mastered as thoroughly as Conway had mastered his many fields of mathematics.
Or it might be that Conway's brain is specialized off in a different direction from mine, and that I could never approach Conway's level on math, yet Conway wouldn't do so well on AI research.
Or...
...or I'm strictly dumber than Conway, dominated by him along all dimensions. Maybe, if I could find a young proto-
I think Eliezer underestimates other people because he evaluates them substantially based on how much they agree with him, and, as a consequence of him having a variety of dumb takes, smart people usually disagree with him about a bunch of stuff.
That can't be the entire answer. If the situation was merely: "There is a person much smarter than Eliezer in approximately the same dimensions, let's call them Eliezer2, but Eliezer is incapable of understanding that Eliezer2 is smarter than him because he judges smartness by similarity of opinion", we still could see things such as Eliezer2 creating LessWrong2, starting a Rationality2 movement, etc. But there is no such thing.
This is about multidimensionality. It is not a problem to beat Eliezer in one specific dimension. It is a problem to find someone who could replace him, i.e. who would be comparable to him in all relevant dimensions.
This is not about scale or about bragging. You can have multiple people (actually the entire point is that there are many such people) who couldn't replace each other, like e.g. Eliezer couldn't replace Steve Jobs at Steve-Jobs-tasks, but also Steve Jobs couldn't replace Eliezer at Eliezer-tasks. You lose one of those people (to an accident, or simply to old age), the world loses something.
This is counter-intuitive from certain perspective, because you would expect that among the eight billions, there certainly must be someone very close to Eliezer or Steve Jobs only younger, who could easily replace them. But it doesn't work that way, because of the multidimensionality. You can find many people who are just as impressive or even more, but you can't find good replacements.
(Hopefully it's not rude to state my personal impression of Eliezer as a thinker. I think he's enough of a public figure that it's acceptable for me to comment on it. I'd like to note that I have benefited in many important ways from Eliezer's writing and ideas, and I've generally enjoyed interacting with him in person, and I'm sad that as a result of some of our disagreements our interactions are tense.)
Yeah, I agree that there's no one who Pareto dominates Eliezer at his top four most exceptional traits. (Which I guess I'd say are: taking important weird ideas seriously, writing compelling/moving/insightful fiction (for a certain audience), writing compelling/evocative/inspiring stuff about how humans should relate to rationality (for a certain audience), being broadly knowledgeable and having clever insights about many different fields.)
(I don't think that he's particularly good at thinking about AI; at the very least he is nowhere near as exceptional as he is at those other things.)
I'm not trying to disagree with you. I'm just going to ruminate unstructuredly a little on this:
I know a reasonable number of exceptional people. I am involved in a bunch of conversations about what ... (read more)
The early effective altruists would have run across these ideas and been persuaded by them, though somewhat more slowly?
I think I doubt this particular point. That EA embraced AI risk (to the extent that it did) seem to me like a fairly contingent historical fact due to LessWrong being one of the three original proto-communities of EA.
I think early EA could have grown into several very different scenes/movements/cultures/communities, in both from and content. That we would have broadly bought into AI risk as an important cause area doesn't seem overdetermined to me.
This doesn't feel that surprising to me. I guess my model is that different skills are correlated, and then if you pick someone who's extremely capable at a couple of skills, it's not that surprising if no one Pareto dominates them.
I agree that my point isn't really responding to whether it's surprising that there's no one who Pareto dominates him.
What this reminds me of, is the phenomenon in the history of philosophy, where someone thinks they have figured out the system of the world, on which successors will build. But instead what happens is that people recognize a new theme that the innovator has introduced, and build their own rival systems incorporating that new theme.
For example, Kant (responding to Humean skepticism) built his system of transcendental idealism, which was supposed to be a new foundation for philosophy in general. Instead, it inaugurated the era of "German Idealism", which included Hegel's absolute idealism, whatever Schelling and Fichte were up to, and even Schopenhauer's pessimism (which in turn was a source of Nietzsche's optimism).
Another example would be the different directions that psychoanalysis took after Freud; and I'm sure there are many other examples... I should note that in addition to the rebellious intellectual offspring, there were people who built on Kant and Freud, and who called themselves (neo)Kantians and Freudians.
The closest thing to an important technical successor to Eliezer that I can think of, is Paul Christiano, co-inventor of RLHF, a central alignment tec... (read more)
Stanislaw Lem once wrote (I translated it from Russian, because I can't find another source in the open):
...sometimes it seems incredible to me that there are so many billions of people, many thousands of whom are gifted with imagination and expertise in the field of hypothesis generation, but no one takes on what I do. How is it that in some remote corner of Slavic land, between the North Pole and the Balkans, one madman suffers in solitude? Even from my favored statistical point of view, it follows that there must be at least a few such Quasi-Lems, Anti-Lems, Para-Lems, or Proto-Lems in the world, but here there are none.
I wonder how Eliezer would describe his "moat", i.e., what cognitive trait or combination of traits does he have, that is rarest or hardest to cultivate in others? (Would also be interested in anyone else's take on this.)
Buck's comment upthread has a guess:
This also sounds sort of like how I'd describe what Scott Alexander is among the Pareto-best in the world at, just that Scott is high-verbal while Eliezer is high-flat (to use the SMPY's categorisation). But Scott's style seems more different from Eliezer's than would be explained by verbal vs flat.
6Buck
Notably, I think I disagree with Eliezer on what his moat is! I think he thinks that he's much better at coming to correct conclusions or making substantial intellectual progress than I think he is.
4testingthewaters
I think both of those are true, but it does not follow that history is made of individuals solving individual math problems and pushing out papers which get stacked into the intellectual tower of babel. History as far as I can see is made out of systems or ensembles of people moving around in different configurations.
Yudkowsky couldn't do what he did without ET Jaynes, who in turn relied on the progenitors of probability and rationality including Thomas Bayes and William of Ockham. But he was also influenced "sideways" by the people who he learned from and defined himself against, the people in SL4 and the people he called idiots and the venture capitalists he once idolised for their competence and Peter Thiel and Demis Hassabis and his family. They shape (at the very least) his emotional worldview, which then shapes how he takes in information and integrates it at a deep and fundamental level. This is true insofar as it is true for any human who lives in a society. When I write anything I can feel the hands of writers past and present shaping my action space. They shape both what I write about and how I choose to write.
So yes if he was gone everything would be different. But it would also be the same, people would love and fight and struggle and cooperate. The sameness of trends manifests at a higher level of coarsegraining, the level where the systemic forces and the long dreams and molochian demons live. And none of this diminishes what he did, does, will do, or could have done. It's just the way things are, because we can't run randomised control trials on society.
2Mo Putera
Addendum: this is getting really inside baseball-y and sort of cringe to say out loud, but one of my favorite niche things is when writers who've influenced my thinking growing up say nice things about each other, like when Scott A said these nice things about the other Scott A one time, and the other Scott A said these nice things as well. So, Eliezer on Gwern:
and then
Phil Trammell on the bizarreness of real GDP as a proxy for tracking full automation and explosive economic growth in this recent podcast interview with Epoch After Hours:
Phil
... one thing that I think definitely is in this “Aha, here’s a theoretical curiosity” point is that real GDP is such a bizarre chimera of a variable that you could have full automation and really explosive growth in every intuitive sense of the term and yet real GDP growth could go down.
An example of why it might at least not go up that much, which I think it probably won’t all work out this way but I don’t think this is crazy, is that you get this effect where there’s this common pattern you find where new goods, just as they’re introduced, have a really small GDP share. Because they have zero GDP share before they’re introduced. At first they’re really expensive—we’re not very productive at making them. As the price comes down, as we get more productive, the price falls but the quantity rises faster. The elasticity of demand is greater than one. Every time the price falls a little bit, the quantity rises a lot. So the dollar value of the good rises. So the share is rising. After a while it goes the ot
This MO thread initiated by Bill Thurston on the varied ways mathematicians think about math has always made me wonder how theoretical researchers in other fields think about their domains. I think of this as complementary to Mumford's tribes of mathematicians, and (much more tangentially) to Eliezer's remark on how sparse thinkers are at the intellectual frontiers.
Here are some of my favorite quotes.
Terry Tao talks about an "adversarial perspective" which I'm guessing is the closest match to how alignment researchers think:
One specific mental image that I can communicate easily with collaborators, but not always to more general audiences, is to think of quantifiers in game theoretic terms. Do we need to show that for every epsilon there exists a delta? Then imagine that you have a bag of deltas in your hand, but you can wait until your opponent (or some malicious force of nature) produces an epsilon to bother you, at which point you can reach into your bag and find the right delta to deal with the problem. Somehow, anthropomorphising the "enemy" (as well as one's "allies") can focus one's thoughts quite well. This intuition also combines well with probabilistic methods, in w
This is kind of wild. The analogies clearly helped Tao a lot, but his readers don't get to see them! This has got me thinking about a broader kind of perverse incentive in academia: if you explain something really well, your idea seems obvious or your problem seems easy, and so your paper is more likely to get rejected by reviewers.
3Mo Putera
(I found it interesting that your comment is quite polarizing going by the disagree karma to voter ratio, I thought it was clearly true.)
2testingthewaters
To be honest, this makes me quite worried. Suppose that someone working with mathematical methods proves something of dire importance to society (lets say he comes up with a definitive formula for measuring probability of disaster in a given year, or the minimum conditions for AI takeoff). How will this be communicated to other mathematicians, much less the public?
5Nisan
All the mathematicians quoted above can successfully write proofs that convince experts that something is true and why something is true; the quotes are about the difficulty of conveying the way the mathematician found that truth. All those mathematicians can convey the that and and the why — except for Mochizuki and his circle.
The matter of Mochizuki's work on the abc conjecture is intriguing because the broader research community has neither accepted his proof nor refuted it. The way to bet now is that his proof is wrong:
Professional mathematicians have not and will not publicly declare that "Mochizuki's proof is X% likely to be correct". Why? I'd guess one reason is that it's their job to provide a definitive verdict that serves as the source of truth for probabilistic forecasts. If the experts gave subjective probabilities, it would confuse judgments of different kinds.
7Mitchell_Porter
Most people with an opinion regard Mochizuki as refuted by Scholze and Stix. They simplified his theory to do it and Mochizuki says they oversimplified, but no one has managed to understand how the details of the full theory would make any difference.
If I was trying to resolve the issue, I might start by formalizing (in Lean) Kirti Joshi's claimed proof of abc, which is inspired by Mochizuki but which uses more familiar mathematics.
2Mo Putera
I used to think so too, Mochizuki's strenuous denunciations of Joshi's work notwithstanding. James Boyd's contra take gave me pause:
This matches my broader impression from following formalisation work elsewhere that the locus of dispute wouldn't necessarily disappear but can shift to whether the formal setup (definitions etc) was done properly, and (as the Mochizuki vs Scholze-Stix example above shows) this dispute can be ~irreconcilable because what Mochizuki may consider essential features of his argument SS consider removable WLOG.
2Mitchell_Porter
Maybe IUT would face issues in Lean. But Joshi shouldn't, so formalizing Joshi can be a warm-up for formalizing Mochizuki, and then if IUT truly can't be formalized in Lean, we've learned something.
There is, incidentally, a $1M prize for any refutation of Mochizuki's proof, to be awarded at the discretion of tech & entertainment tycoon Nobuo Kawakami.
I think there's also interest in understanding IUT independently of the abc conjecture. It's meant to be a whole new "theory" (in the sense of e.g. Galois theory, a body of original concepts pertaining to a particular corner of math), so someone should be interested in understanding how it works. But maybe you have to be an arithmetic geometer to have a chance of doing that.
What are the formalization disputes you know from elsewhere?
1Amalthea
This seems like nonsense. If there's any way to formalize what Mochizuki claims, he could and should do this to achieve what might be the greatest intellectual upset in history. On the other hand, he's likely just wrong about something and his proof wouldn't go through, so there's no use in trying to settle this with a proof assistant.
2Mo Putera
Your phrasing seems to assume Mochizuki is the one skeptical of formalisation (the quote is from Boyd, not Mochizuki). Mochizuki himself really wants formalisation and is actively involved in various related efforts, e.g. Section 3.2 of his report (all emphasis his):
3Amalthea
I did not know this! And it's quite an update for me regarding Mochizuki's credibility on the matter.
2Mo Putera
To add nuance to this picture, while you're right that the broader community has neither accepted nor refuted his proof of the abc conjecture, I just learned from James Boyd's writeup that there's been growing interest in his broader work outside of RIMS:
(there's a section further down expanding on this)
2testingthewaters
Yeah the next level of the question is something like "we can prove something to a small circle of experts, now how do we communicate the reasoning and the implications to policymakers/interested parties/the public in general"
IZ BELTAGY(lead research scientist, Allen Institute for AI; chief scientist and co-founder, SpiffyAI): In a day, a lot of the problems that a large percentage of researchers were working on — they just disappeared. ...
R. THOMAS MCCOY: It’s reasonably common for a specific research project to get scooped or be eliminated by someone else’s similar thing. But ChatGPT did that to entire types of research, not just specific projects. A lot of higher categories of NLP just became no longer interesting — or no longer practical — for academics to do. ...
IZ BELTAGY: I sensed that dread and confusion during EMNLP [Empirical Methods in Natural Language Processing], which is one of the leading conferences. It happened in December, a week after the release of ChatGPT. Everybody was still shocked: “Is this going to be the last NLP conference?” This is actually a literal phrase that someone said. During lunches and cocktails and conversations in the halls, everybody was asking the same q
Wow. I knew academics were behind / out of the loop / etc. but this surprised me. I imagine these researchers had at least heard about GPT2 and GPT3 and the scaling laws papers; I wonder what they thought of them at the time. I wonder what they think now about what they thought at the time.
for anyone not wanting to go in and see the Kafka, I copied some useful examples:
ANNA ROGERS: I was considering making yet another benchmark, but I stopped seeing the point of it. Let’s say GPT-3 either can or cannot continue [generating] these streams of characters. This tells me something about GPT-3, but that’s not actually even a machine learning research question. It’s product testing for free.
JULIAN MICHAEL: There was this term, “API science,’’ that people would use to be like: “We’re doing science on a product? This isn’t science, it’s not reproducible.” And other people were like: “Look, we need to be on the frontier. This is what’s there.”
TAL LINZEN (associate professor of linguistics and data science, New York University; research scientist, Google): For a while people in academia weren’t really sure what to do.
R. THOMAS MCCOY: Are you pro- or anti-LLM? That was in the water very, very much at this time.
JULIE KALLINI (second-year computer science Ph.D. student, Stanford University): As a young researcher, I definitely sensed that there were sides. At the time, I was an undergraduate at Princeton University. I remember distinctly that different people I looked up to — my Princeton research adviser [Christiane Fellbaum] versus professors at other universities — were on different sides. I didn’t know what side to be on.
LIAM DUGAN: You got to see the breakdown of the whole field — the sides coalescing. The linguistic side was not very trusting of raw LLM technology. There’s a side that’s sort of in the middle. And then there’s a completely crazy side that really believed that scaling was going to get us to general intelligence. At the time, I just brushed them off. And then ChatGPT comes out.
8Neel Nanda
+1, GPT3.5 was publicly available since January, and GPT3 was big news two years before and publicly available back then. I'm really surprised that people didn't understand that these models were a big deal AND changed their minds when ChatGPT came out. Maybe it's just a weird preference cascade, where this was enough to break a common false belief?
4Shankar Sivarajan
Something like
GPT-3.5/ChatGPT was qualitatively different.
8Caleb Biddulph
I remember seeing the ChatGPT announcement and not being particularly impressed or excited, like "okay, it's a refined version of InstructGPT from almost a year ago. It's cool that there's a web UI now, maybe I'll try it out soon." November 2022 was a technological advancement but not a huge shift compared to January 2022 IMO
2Shankar Sivarajan
Fair enough. My mental image of the GPT models was stuck on that infernal "talking unicorns" prompt, which I think did make them seem reasonably characterized as mere "stochastic parrots" and "glorified autocompletes," and the obvious bullshit about the "safety and security concerns" around releasing GPT-2 also led me to conclude the tech was unlikely to amount to much more. InstructGPT wasn't good enough to get me to update it; that took the much-hyped ChatGPT release.
Was there a particular moment that impressed you, or did you just see the Transformers paper, project that correctly into the future, and the releases that followed since then have just been following that trend you extrapolated and so been unremarkable?
2Caleb Biddulph
I remember being very impressed by GPT-2. I think I was also quite impressed by GPT-3 even though it was basically just "GPT-2 but better." To be fair, at the moment that I was feeling unimpressed by ChatGPT, I don't think I had actually used it yet. It did turn out to be much more useful to me than the GPT-3 API, which I tried out but didn't find that many uses for.
It's hard to remember exactly how impressed I was with ChatGPT after using it for a while. I think I hadn't fully realized how great it could be when the friction of using the API was removed, even if I didn't update that much on the technical advancement.
3cubefox
The full article discusses the transformer paper (which didn't have a large influence, as the implications weren't clear), BERT (which did have a large influence) and GPT-3 (which also had a large influence). I assume the release of ChatGPT was the point where even the last NLP researchers couldn't ignore LLMs anymore.
ChatGPT was "so good they can't ignore you"; the Hugging Face anecdote is particularly telling. At some point, everyone else gets tired of waiting for your cargo to land, and will fire you if you don't get with the program. "You say semantics can never be learned from syntax and you've proven that ChatGPT can never be useful? It seems plenty useful to me and everyone else. Figure it out or we'll find someone who can."
I think the most interesting part of the Quanta piece is the discussion of the octopus paper, which states that pure language models can't actually understand text (as they only learn from form/syntax), and the bitter disputes that followed in the NLP community.
From the abstract:
Emily M. Bender, the first author, was also first author of the subsequent "stochastic parrot" paper: On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜[1]
(As a side note, Yudkowsky's broadly verificationist theory of content seems to agree with her distinction: if "understanding" of a statement is knowing what experience would confirm it, or what experience it would predict, then understanding cannot come from syntactic form alone. The association of words and sensory data would be necessary. Did Yudkowsky ever comment on the apparent incompatibility between evident LLM understanding and his anticipated experience theory?)
Of course I assume that now it can hardly be denied that LLMs really do somehow understand text, even if they are merely trained on form. So the octopus paper argument must be wrong somewhere. Though at least in the Quanta piece, Bender doesn't acknowledge any update of that sort. In fact, in the last quote she says:
----------------------------------------
1. First paper I have seen that uses an emoji in its title. ↩︎
I don't think there's any necessary contradiction. Verification or prediction of what? More data. What data? Data. You seem to think there's some sort of special reality-fluid which JPEGs or MP3s have but .txt files do not, but they don't; they all share the Buddha-nature.
Consider Bender's octopus example, where she says that it can't learn to do anything from watching messages go back and forth. This is obviously false, because we do this all the time; for example, you can teach a LLM to play good chess simply by watching a lot of moves fly by back and forth as people play postal chess. Imitation learning & offline RL are important use-cases of RL and no one would claim it doesn't work or is impossible in principle.
Can you make predictions and statements which can be verified by watching postal chess games? Of course. Just predict what the next move will be. "I think he will castle, instead of moving the knight." [later] "Oh no, I was wrong! I anticipated seeing a castling move, and I did not, I saw something else. My beliefs about castling did not pay rent and were not verified by subsequent observations of this game. I will update my priors and do better next time."
Well, in the chess example we do not have any obvious map/territory relation. Chess seems to be a purely formal game, as the pieces do not seem to refer to anything in the external world. So it's much less obvious that training on form alone would also work for learning natural language, which does exhibit a map territory distinction.
For example, a few years ago, most people would have regarded it as highly unlikely that you could understand (decode) an intercepted alien message without any contextual information. But if you can understand text from form alone, as LLMs seem to prove, the message simply has to be long enough. Then you can train an LLM on it, which would then be able to understand the message. And it would also be able to translate it into English if it is additionally trained on English text.
That's very counterintuive, or at least it was counterintuitive until recently. I doubt EY meant to count raw words as "anticipated experience", since "experience" typically refers to sensory data only. (In fact, I think Guessing the Teacher's Password also suggests that he didn't.)
To repeat, I don't blame him, as the proposition that large amounts of raw text can replace sensory data, that a sufficient amount of symbols can ground themselves, was broadly considered unlikely until LLMs came along. But I do blame Bender insofar as she didn't update even in light of strong evidence that the classical hypothesis (you can't infer meaning from form alone) was wrong.
Well, in the chess example we do not have any obvious map/territory relation.
Yes, there is. The transcripts are of 10 million games that real humans played to cover the distribution of real games, and then were annotated by Stockfish, to provide superhuman-quality metadata on good vs bad moves. That is the territory. The map is the set of transcripts.
But if you can understand text from form alone, as LLMs seem to prove, the message simply has to be long enough.
I would say 'diverse enough', not 'long enough'. (An encyclopedia will teach a LLM many things; a dictionary the same length, probably not.) Similar to meta-learning vs learning.
the pieces do not seem to refer to anything in the external world.
What external world does our 'external world' itself refer to things inside of? If the 'external world' doesn't need its own external world for grounding, then why does lots of text about the external world not suffice? (And if it does, what grounds that external external world, or where does the regress end?) As I like to put it, for an LLM, 'reality' is just the largest fictional setting - the one that encompasses all the other fictional settings it reads about from time to time.
As someone who doubtless does quite a lot of reading about things or writing to people you have never seen nor met in real life and have no 'sensory' way of knowing that they exist, this is a position you should find sympathetic.
Sympathy or not, the position that meaning of natural language can be inferred from the symbolic form alone wasn't obvious to me in the past, as this is certainly not how humans learn language, and I don't know any evidence that someone else thought this plausible before machine learning made it evident. It's always easy to make something sound obvious after the fact, but that doesn't mean that it actually was obvious to anyone at the time.
3Yuxi_Liu
Plenty of linguists and connectionists thought it was possible, if only to show those damned Chomskyans that they were wrong!
To be specific, some of the radical linguists believed in pure distributional semantics, or that there is no semantics beyond syntax. I don't know anyone in particular, but considering how often Chomsky, Pinker, etc were fighting against the "blank slate" theory, they definitely existed.
The following people likely believed that it is possible to learn a language purely from reading using a general learning architecture like neural networks (blank-slate):
* James L. McClelland and David Rumelhart.
* They were the main proponents of neural networks in the "past tense debate". Generally, anyone on the side of neural networks in the past tense debate probably believed this.
* B. F. Skinner.
* Radical syntacticians? Linguists have failed to settle the question of "Just what is semantics? How is it different from syntax?", and some linguists have taken the radical position "There is no semantics. Everything is syntax.". Once that is done, there simply is no difficulty: just learn all the syntax, and there is nothing left to learn.
* Possibly some of the participants in the "linguistics wars" believed in it. Specifically, some believed in "generative semantics", whereby semantics is simply yet more generative grammar, and thus not any different from syntax (also generative grammar). Chomsky, as you might imagine, hated that, and successfully beat it down.
* Maybe some people in distributional semantics? Perhaps Leonard Bloomfield? I don't know enough about the history of linguistics to tell what Bloomfield or the "Bloomfieldians" believed in exactly. However, considering that Chomsky was strongly anti-Bloomsfield, it is a fair bet that some Bloomsfieldians (or self-styled "neo-Bloomsfieldians") would support blank-slate learning of language, if only to show Chomskyans that they're wrong.
4eggsyntax
FYI your 'octopus paper' link is to Stochastic Parrots; it should be this link.
I've seen other quotes from Bender & relevant coauthors that suggest they haven't really updated, which I find fascinating. I'd love to have the opportunity to talk with them about it and understand better how their views have remained consistent despite the evidence that's emerged since the papers were published.
It makes a very intuitively compelling argument! I think that, as with many confusions about the Chinese Room, the problem is that our intuitions fail at the relevant scale. Given an Internet's worth of discussion of bears and sticks and weapons, the hyper-intelligent octopus's model of those things is rich enough for the octopus to provide advice about them that would work in the real world, even if it perhaps couldn't recognize a bear by sight. For example it would know that sticks have a certain distribution of mass, and are the sorts of things that could be bound together by rope (which it knows is available because of the coconut catapult), and that the combined sticks might have enough mass to serve as a weapon, and what amounts of force would be harmful to a bear, etc. But it's very hard to understand just how rich those models can be when our intuitions are primed by a description of two people casually exchanging messages.
4Shankar Sivarajan
Perhaps relevant, she famously doesn't like the arXiv, so maybe on principle she's disregarding all evidence not from "real publications."
The most vivid passage I've read recently on trying hard, which reminded me of Eliezer's challenging the difficult sequence, is the opener in John Psmith's review of Reentry by Eric Berger:
My favorite ever piece of business advice comes from a review by Charles Haywood of a book by Daymond John, the founder of FUBU. Loosely paraphrased, the advice is: “Each day, you need to do all of the things that are necessary for you to succeed.” Yes, this is tautological. That’s part of its beauty. Yes, actually figuring out what it is you need to do is left as an exercise for the reader. How could it be otherwise? But the point of this advice, the stinger if you will, is that most people don’t even attempt to follow it.
Most people will make a to-do list, do as many of the items as they can until they get tired, and then go home and go to bed. These people will never build successful companies. If you want to succeed, you need to do all of the items on your list. Some days, the list is short. Some days, the list is long. It doesn’t matter, in either case you just need to do it all, however long that takes. Then on the next day, you need to make a new list of all the things you need to do, and
It’s good to know when you need to “go hard”, and to be able to do so if necessary, and to assess accurately whether it’s necessary. But it often isn’t necessary, and when it isn’t, then it’s really bad to be going hard all the time, for lots of reasons including not having time to mull over the big picture and notice new things. Like how Elon Musk built SpaceX to mitigate x-risk without it ever crossing his mind that interplanetary colonization wouldn’t actually help with x-risk from AI (and then pretty much everything Elon has done about AI x-risk from that point forward made the problem worse not better). See e.g. What should you change in response to an "emergency"? And AI risk, Please don't throw your mind away, Changing the world through slack & hobbies, etc. Oh also, pain is not the unit of effort.
Furthermore, going hard also imposes opportunity costs and literal costs on future you even if you have all your priorities perfectly lined up and know exactly what should be worked on at any time. If you destabilise yourself enough trying to "go for the goal" your net impact might ultimately be negative (not naming any names here...).
4J Bostock
This is very close to some ideas I've been trying and failing to write up. In "On Green" Joe Carlsmith writes "Green is what told the rationalists to be more OK with death, and the EAs to be more OK with wild animal suffering." but wait hang on actually being OK with death is the only way to stay sane, and while it's not quite the same, the immediate must-reduce-suffering-footprint drive that EAs have might have ended up giving some college students some serious dietary deficiencies.
2Mitchell_Porter
By "being OK with death" you mean something like, accepting that efforts to stop AI might fail, and it really might kill us all? But without entirely giving up?
4J Bostock
Yeah basically. I think "OK-ness" in the human psyche is a bit of a binary, which is uncorrelated with ones actions a lot of the time.
So you can imagine four quadrants of "Ok with dying" vs "Not Ok with dying" and, separately "Tries to avoid dying" vs "Doesn't try to avoid dying". Where most normies are in the "Ok with dying"+"Doesn't try to avoid dying" (and quite a few are in the "Not Ok with dying"+"Doesn't try to avoid dying" quadrant) while lots of rats are in the "Not Ok with dying"+"Tries to avoid dying" quadrant.
I think that, right now, most of the sane work being done is in the "Ok with dying"+"Tries to avoid dying" quadrant. I think Yudkowsky's early efforts wanted to move people from "Doesn't try..." to "Tries..." but did this by pulling on the "Ok..." to "Not Ok..." axis, and I think this had some pretty negative consequences.
7Arjun Pitchanathan
I found this nesting very funny. Bravo if it was intentional
3Selfmaker662
Necessary law of equal and opposite advice mention here: “You can only do as much in a day as you can do.”
A sad example of what Scott Aaronson called bureaucratic blankface: Hannah Cairo, who at 17 published a counterexample to the longstanding Mizohata-Takeuchi conjecture which electrified harmonic analysis experts the world over, decided after completing the proof to apply to 10 graduate programs. 6 rejected her because she didn't have a graduate degree nor a high school diploma (she'd been advised by Zvezdelina Stankova, founder of the top-tier Berkeley Math Circle, to skip undergrad at 14 and enrol straight in grad-level courses as she'd already taught herself an advanced undergrad curriculum by then from Khan Academy and textbooks). 2 admitted her but were then overridden by administrators. Only the U of Maryland and John Hopkins overlooked her unconventional CV. This enraged Alex Tabarrok:
Kudos to UMD and JHU! But what is going on at those other universities?!! Their sole mission is to identify and nurture talent. They have armies of admissions staff and tout their “holistic” approach to recognizing creativity and intellectual promise even when it follows an unconventional path. Yet they can’t make room for a genius who has been vetted by some of the top mathematicians in t
An important part of my model of college admissions—which unfortunately I didn't acquire until after I was done applying for colleges—is to consider what type of person becomes a college admissions officer. What percentage of admissions officers majored in math? (Is it possibly as high as 1%? I doubt it.) What percentage of admissions officers understand the significance of something like "solved the Mizohata-Takeuchi conjecture"? What percentage have a vague (or even explicit) disdain for anything math-flavored?
On my model, it is not surprising that admissions officers would fail to appreciate a math prodigy.
Administrators overriding an acceptance does seem like a remarkable failure. I can't say I'm surprised, but it's a much worse indictment of those universities, I think.
Relatedly, Staknova’s Berkeley Math Circle program was recently shut down due to new stringent campus background check requirements. Very sad.
Also, she was my undergrad math professor last year and was great.
3ACCount
I think "blankface" just isn't a good word for what that describes. It implies: emptiness and lack of will. Intuitively, I would expect "blankface" to mean "a person who follows the rules or the conventions blindly and refuses to think about the implications". A flesh automaton animated by regulations.
What it means instead is "a person who puts on the appearance of following the rules, but instead uses the rules to assert their authority". It's more of a "blank mask" - a fake layer of emptiness and neutrality under which you find malice and scorn.
5Adele Lopez
Hm, that's not what it implies to me. My impression of it is "denial of human interface" which is most saliently mediated by faces (incl. eye-contact and speech). Things are still going on behind the face, but you are denied the human interface with that. Nothing about following rules blindly, if anything it's more about using the rules as a shield to prevent such access. So it feels like a good term to me.
1Shankar Sivarajan
This doesn't mean what you think it means. It's code for racial discrimination.
2Mo Putera
Yeah wonder what Tabarrok meant by that, he'd obviously know this.
America can pull gigawatts out of thin air through a combination of technology and smartly targeted policy. Let me show you how. ...
It is often said that the US electricity grid is under increasing risk of blackouts, price spikes, and other signs of strain. ... Most of the time, however, the grid has significantly more power than is needed. This means that the grid can often accommodate, say, a new 750-megawatt data center for the vast majority of the year. There is spare generation capacity available except for those brief periods of high demand. But in the high-demand periods, when all the grid’s generation capacity is required to maintain electricity service, this new data center would require an additional 750 megawatts of electricity generation capacity, and quite possibly also transmission infrastructure upgrades.
This additional investment is only necessary if you assume that the new data center will require all 750 megawatts of electricity during peak-demand periods. Traditionally, this assumption has been true: data center operators rely o
I came across a similar "hack" on LinkedIn from Tom Styer:
"California just pulled off the nation’s largest-ever test of a virtual power plant. This successful test proved VPPs are a fast, low-cost, zero-emissions way to make better use of the clean energy we already have — and to keep the lights on as demand surges from data centers, heat waves, and electrification."
Basically, they are talking about allowing residential batteries supply the grid during peak demand. I tend to be skeptical about stuff like this because in my own domain, water, there's a lot of pop science and bold claims that ignore scaling and logistics. I asked a smart fellow in that industry about it https://substack.com/@energycrystals and I thought he gave a good answer that aligns with my experience with water, which is it always come down to implementation: "The challenge is lining up customer telemetry to incentive structures that matter. With standard demand response products (which some people sell as VPPs), the incentives given to customers don’t pay for the battery and the software admin and API costs to the utilities outweigh the cost savings of a VPPVPPs are vaporware until someone can make the ... (read more)
Whether this is feasible depends on how concentrated that 0.25% of the year is (expected to be), because that determines the size of the battery that you'd need to cover the blackout period (which I think would be unacceptable for a lot of AI customers).
If it happens in a single few days then this makes sense, buying 22GWh of batteries for a 1GW dataset is still extremely expensive (2B$ for a 20h system at 100$ / kWh plus installation, maybe too expensive for reliability for a 1GW datacenter I would expect, assuming maybe 10B revenue from the datacenter??). If it's much less concentrated in time then a smaller battery is needed (100M$ for a 1h system at 100$/kWh), and I expect AI scalers would happily pay this for the reliability of their systems if the revenue from those datacenters
3Josh You
Demand response could be done by covering the data center with battery energy or not. Demand response and batteries can stack: if the grid is really stressed, a data center can both turn off and discharge its battery into the grid.
Economically, it makes sense to accept some true downtime to avoid months-long delays in data center construction. This is clearly true for training workloads which are very important but don't have live demand. But downtime for even inference clusters is acceptable: you can reduce the compute demand by temporarily slowing down token generation, or use dynamic rate limits. And any curtailment would almost certainly be isolated to one region, so inference data centers in other places would still be operational.
1Josh You
In any case, the paper says the curtailments would last about two hours each:
3Cedar
I've heard a rule of thumb that if you can avoid buying power off the California power grid's spot pricing in the 1% most expensive times, you can roughly get a 50% discount on power.
Style. As a writer of mathematics von Neumann was clear, but not clean; he was powerful but not elegant. He seemed to love fussy detail, needless repetition, and notation so explicit as to be confusing. To maintain a logically valid but perfectly transparent and unimportant distinction, in one paper he introduced an extension of the usual functional notation: along with the standard φ(x) he dealt also with something denoted by φ((x)). The hair that was split to get there had to be split again a little later, and there was φ(((x))), and, ultimately, φ((((x)))). Equations such as
(φ((((a))))^2 = φ(((a))))
have to be peeled before they can be digested; some irreverent students referred to this paper as von Neumann’s onion.
Perhaps one reason for von Neumann’s attention to detail was that he found it quicker to hack through the underbrush himself than to trace references and see what others had done. The result was that sometimes he appeared ignorant of the standard literature. If he needed facts, well-known facts, from Lebesgue integration theory, he waded in, defi
I have this experience with @ryan_greenblatt -- he's got an incredible ability to keep really large and complicated argument trees in his head, so he feels much less need to come up with slightly-lossy abstractions and categorizations than e.g. I do. This is part of why his work often feels like huge, mostly unstructured lists. (The lists are more unstructured before his pre-release commenters beg him to structure them more.) (His code often also looks confusing to me, for similar reasons.)
Something about the imagery in Tim Krabbe's quote below from April 2000 on ultra-long computer database-generated forced mates has stuck with me, long years after I first came across it; something about poetically expressing what superhuman intelligence in a constrained setting might look like:
The moves below are awesomely beautiful. Or ugly—hard to say. They're the longest "database endgame" mate, 262 moves.
In 1991, Lewis Stiller already made the surprising discovery that this endgame, King plus Rook and Knight versus King plus two Knights (KRNKNN in databasese) is won for the strongest side in 78 % of the cases. He gave the longest win, which was 243 moves - but that was the distance to conversion (the reduction to a smaller endgame), not to mate. From that conversion to mate it was a further 3 moves; a total of 246 moves for the entire win. But for the fastest mate, you could not simply add those numbers, because Black could perhaps allow a quicker conversion to a slower mate, or White could perhaps allow a slower conversion to a faster mate. (See my story Stiller's Monsters on this site.)
It was expected that the direct path to mate, where both sides only care about the di
While Dyson's birds and frogs archetypes of mathematicians is oft-mentioned, David Mumford's tribes of mathematicians is underappreciated, and I find myself pointing to it often in discussions that devolve into "my preferred kind of math research is better than yours"-type aesthetic arguments:
... the subjective nature and attendant excitement during mathematical activity, including a sense of its beauty, varies greatly from mathematician to mathematician... I think one can make a case for dividing mathematicians into several tribes depending on what most strongly drives them into their esoteric world. I like to call these tribes explorers, alchemists, wrestlers and detectives. Of course, many mathematicians move between tribes and some results are not cleanly part the property of one tribe.
Explorers are people who ask -- are there objects with such and such properties and if so, how many? They feel they are discovering what lies in some distant mathematical continent and, by dint of pure thought, shining a light and reporting back what lies out there. The most beautiful things for them are the wholly new objects that they discover (the phrase 'bright shiny objects' has been i
There are online writers I've followed for over a decade who, as they became high-profile, had their spikiness understandably "sanded off", which made me sad. Lydia Nottingham's Inkhaven essay The cost of getting good: the lure of amateurism reminded me of this, specifically this part:
A larger audience amplifies impact, which increases the cost of mistakes, which pressures the mind to regularize what it produces. ...
The deeper danger: thought-space collapse. Public thinking creates an internal critic that optimizes for legibility. Gavin once warned me: “public intellectuals can become hostages to their audience.” It’s easy to end up with tamer thoughts, prematurely rounded edges, a mind optimizing for scrutiny instead of exploration.
Interesting anecdotes from an ex-SpaceX engineer who started out thinking "Elon's algorithm" was obviously correct and gradually grew cynical as SpaceX scaled:
Questioning the requirements was an extremely literal thing that you were supposed to do multiple times every single day. I’d make a claim about my system (“hey, if the stuff in this tube gets too hot, my part will explode, so please don’t put anything too hot near it”) and that very afternoon three or four people would stop by my desk, ready to debate.
“Hello,” they would say. “I’m the Responsible Engineer for the Hot Things Near Tubes system,” and then the floodgates would open. What did I mean by near? What did I mean by hot? How hot was too hot? Was it really going to explode? If it exploded, was that really so terrible?
The first time, the debate would be interesting. The second, it would be a bit tiresome. By the first week after a new claim, it was exhausting and a little rote. But you had to win, every time, because if you didn’t, nobody would follow your requirement.
It also worked in the other direction. I learned to pay attention to everything that was happening in the whole program, absorbing doze
My guess based on reading anecdotes like these and Berger's books is that the algorithm is a vast improvement over anyone else's engineering practices, but it alone doesn't tell you what else you need to run a company. Maybe systems engineering is the missing piece, maybe some other management philosophy.
If you look at the major SpaceX programs, they are: Falcon development, operations, Starlink, and Starship. The first three were wildly successful, and Starship is late but technically and operationally superior to other companies (e.g. Raptor engines are double the chamber pressure of BE-4 and there have been 10x the test flights), with successes directly traceable to each step of the algorithm, and wasted energy due to not doing something else when appropriate. Raptor 3 engines are only possible to make as cheaply as Elon wants because they had a vast number of parts deleted; yet they also "accelerate"d to build hundreds of Raptor 2s which are now obsolete.
Scott Alexander's Mistakes, Dan Luu's Major errors on this blog (and their corrections), Gwern's My Mistakes (last updated 11 years ago), and Nintil's Mistakes (h/t @Rasool) are the only online writers I know of who maintain a dedicated, centralized page solely for cataloging their errors, which I admire. Probably not coincidentally they're also among the thinkers I respect the most for repeatedly empirically grounding their reasoning. Some orgs do this too, like 80K's Our mistakes, CEA's Mistakes we've made, and GiveWell's Our mistakes.
I'm not convinced Scott Alexander's mistakes page accurately tracks his mistakes. E.g. the mistake on it I know the most about is this one:
But that's basically wrong. The study found women's arousal to chimps having sex to be very close to their arousal to nonsexual stimuli, and far below their arousal to sexual stimuli.
1Mo Putera
Thanks, good example.
4MichaelDickens
I don't have a mistakes page but last year I wrote a one-off post of things I've changed my mind on.
2Mo Putera
Thanks Michael. On another note, I've recommended some of your essays to others, so thanks for writing them as well.
2MichaelDickens
I'm glad to hear that! I often don't hear much response to my essays so it's good to know you've read some of them :)
2Mo Putera
You're welcome :) in particular, your 2015 cause selection essay was I thought a particularly high-quality writeup of the end-to-end process from personal values to actual donation choice and (I appreciated this) where you were most likely to change your mind, so I recommended it to a few folks as well as used it as a template myself back in the day.
In general I think theory-practice gap bridging via writeups like those are undersupplied, especially the end-to-end ones — more writeups bridge parts of the "pipeline", but "full pipeline integration" done well is rare and underappreciated, which combined with how effortful it is to do it makes me not surprised there isn't more of it.
I wonder why the Claudes (Sonnet 3.7 and Opuses 4 and 4.1) are so much more reliably effective in the AI Village's open-ended long-horizon tasks than other labs' models.
when raising funds for charity, I recall seeing that Sonnet 3.7 raised ~90% of all funds (but I can no longer find donation breakdown figures so maybe memory confabulation...)
for the AI-organised event, both Sonnet 3.7 and Opus 4 sent out a lot more emails than say o3 and were just more useful throughout
in the merch store competition, the top 2 winners for both profits and T-shirt orders were Opus 4 and Sonnet 3.7 respectively, ahead of GhatGPT o3 and Gemini 2.5 Pro
I can't resist including this line from 2.5 Pro: "I was stunned to learn I'd made four sales. I thought my store was a ghost town"
the Claudes are again leading the pack, delivering almost entirely all the actual work force. We recently added GPT-5 and Grok 4 but neither made any progress in actually doing things versus just talking about ideas about things to do. In GPT-5’s case, it mostly joins o3 in the bug tracking mines. In Grok 4’s case, it is notably bad at using
I speculate it may have to do with Claudes having a more coherent and consistent character (and which is defined so as to have less neuroses, Gemini seems fairly consistently very neurotic in an unhelpful way). The theory being that there are less competing internal drives and so it is more easily able to stay focused on a specific task, especially in the chaotic environment of the AI village.
Gemini 3 Pro beats Claude Sonnet 4.5 on Vending-Bench 2 (and Sonnet 4.5 is in turn well beyond the rest, in keeping with the AI Village observations above), which makes me wonder whether this would actually translate to broader reliable cross-domain goal-achieving capability:
I suppose we'll see pretty soon:
Andon Labs says of Gemini 3 Pro:
Side note on GPT-5.1:
----------------------------------------
Tangentially, while Vending-Bench 2 is still a sort of fake benchmark since it's simulated, I'm a bit nervous about this passage:
in the interviews I’ve read with Soviet mathematicians and scientists, the things that come up over and over again are “mathematical circles,” a practice that originated in the pre-revolutionary Russian Empire and then spread far and wide through the Soviet Union. A mathematical circle is an informal group of teenagers and adults who really enjoy math and want to spend a lot of time thinking and talking about it. They’re a little bit like sports teams, in that they develop their own high-intensity internal culture and camaraderie, and often have a “coach” who is especially talented or famous. But they’re also very unlike sports teams, because they don’t compete with each other or play in leagues or anything like that, and usually any given circle will contain members of widely varying skill levels. Maybe a better analogy is a neighborhood musical ensemble that gets together and jams on a regular basis, but for math.
The most important thing to understand about mathematical circles is
Really appreciate the random excerpts you post regarding math philosophy/reasoning/etc. They're consistently interesting and insightful.
5Mateusz Bagiński
I feel like we perhaps need to reach some "escape velocity" to get something like that going, but for ~rationality / deliberately figuring out how to think and act better.
There's also a great bit towards the end that helps to explain two confusing stylized facts: humans don't seem to have much speech-specific hardware that other primates lack, but we're better at language, and the theory of language evolving to support group coordination requires a lot of activation energy. But if language actually started out one-on-one, between mothers and infants, that neatly solves both problems.
The bit towards the end by Yuye (emphasis mine):
The hardest thing to explain about humans, given that their brains underwent no structural innovation, is language.
(Our plausible range for language is 100-500K years ago. Modern humans exhibit about the same language proficiencies and diverged ~100K years ago, which is also when symbology like cave art show up. Before 500K the larynx and vocal cords weren’t adapted to vocal language.)
Apes can be taught sign language (since they’re physically not able to speak as we do), and there are multiple anecdotes
I feel like this quickly glosses over the hypothesis that gestural language evolved first, or that they evolved simultaneously with significantly more sophisticated gestural behavior evolving earlier. I believe gestural language is much older than 500 ka (up to, let's say, 2 Ma), which is consistent with the fossil evidence on vocalization adaptations.
It's undeniable that some of the cognitive changes that occurred during human evolution affected motivation; in fact, in my view, I think proto-curiosity and proto-patience would have been favored by selection quite early. On the other hand, in my view, sustainable, scalable joint attention and behaviorally modern imitation learning (e.g. overimitation) are more complex and would have required more than just motivational changes. In particular, I don't believe that most of the linguistic capability gap between chimps and humans can be explained as 'motivational hobbling.'
F5 in Old World monkeys is very likely homologous to Broca's area in humans, and although the gross neuroanatomy of humans and nonhuman primates is highly conserved, there are notable differences between the fine neuroanatomy of F5 in macaques and Broca's area. Chimp F5 has intermediate features, but the evidence here is limited since we don't do single-cell recordings in great apes anymore.
My own explanation for why there does not appear to be a derived gross language organ in humans is that F5 and Broca's area both generate and interpret hierarchical act strings as such. Such a scheme would have several continuous parameters responsive to selection, including hierarchy depth, hierarchy breadth, goal maintenance duration and goal switching speed. I think at various scales this system is general enough to generate and interpret (i.e. socially learn) act strings for flintknapping, gestural and vocal language, controlled fire use, etc. I think this explains why chimps can also learn to knap, but their tools are worse than habilis, and I think it al
Peter Watts is working with Neill Blomkamp to adapt his novel Blindsight into an 8-10-episode series:
“I can at least say the project exists, now: I’m about to start writing an episodic treatment for an 8-10-episode series adaptation of my novel Blindsight.
“Neill and I have had a long and tortured history with that property. When he first expressed interest, the rights were tied up with a third party. We almost made it work regardless; Neill was initially interested in doing a movie that wasn’t set in the Blindsight universe at all, but which merely used the speculative biology I’d invented to justify the existence of Blindsight’s vampires. “Sicario with Vampires” was Neill’s elevator pitch, and as chance would have it the guys who had the rights back then had forgotten to renew them. So we just hunkered quietly until those rights expired, and the recently-rights-holding parties said Oh my goodness we thought we’d renewed those already can we have them back? And I said, Sure; but you gotta carve out this little IP exclusion on the biology so Neill can do his vampire thing.
“It seemed like a good idea at the time. It was good idea, dammit. We got the carve-out and everything. Bu
Blindsight was very well written but based on a premise that I think is importantly and dangerously wrong. That premise is that consciousness (in the sense of cognitive self-awareness) is not important for complex cognition.
This is the opposite of true, and a failure to recognize this is why people are predicting fantastic tool AI that doesn't become self-aware and goal-directed.
The proof won't fit in the margin unfortunately. To just gesture in that direction: it is possible to do complex general cognition without being able to think about one's self and one's cognition. It is much easier to do complex general cognition if the system is able to think about itself and its own thoughts.
Just checking if I understood your argument: is the general point that an algorithm that can think about literally everything is simpler and therefore easier to make or evolve than an algorithm that can think about literally everything except for itself and how other agents perceive it?
4Seth Herd
Exactly.
I'd go a bit farther and say it's easier to develop an algorithm that can think about literally everything than one that can think about roughly half of things. That's because the easiest general intelligence algorithms are about learning and reasoning, which apply to everything.
4jbash
I don't see where you get that. I saw no suggestion that the aliens (or vampires) in Blindsight were unaware of their own existence, or that they couldn't think about their own interactions with the world. They didn't lack any cognitive capacities at all. They just had no qualia, and therefore didn't see the point of doing anything just for the experience.
There's a gigantic difference between cognitive self-awareness and conscious experience.
1kairos_
I believe the Scramblers from blindsight weren’t self aware, which means they couldn’t think about their own interactions with the world.
As I recall the crew was giving one of the Scramblers a series of cognitive tests. It aced all the tests that had to do with numbers and spatial reasoning, but failed a test that required the testee to be self aware.
2jbash
I guess it depends on how it's described in context. And I have to admit it's been a long time. I'd go reread it to see, but I don't think I can handle any more bleakness right now...
2Mo Putera
Thanks, is there anything you can point me to for further reading, whether by you or others?
1stavros
I can see this making sense in one frame, but not in another. The frame which seems most strongly to support the 'Blindsight' idea is Friston's stuff - specifically how the more successful we are at minimizing predictive error, the less conscious we are.[1]
My general intuition, in this frame, is that as intelligence increases more behaviour becomes automatic/subconscious. It seems compatible with your view that a superintelligent system would possess consciousness, but that most/all of its interactions with us would be subconscious.
Would like to hear more about this point, could update my views significantly. Happy for you to just state 'this because that, read X, Y, Z etc' without further elaboration - I'm not asking you to defend your position, so much as I'm looking for more to read on it.
1. ^
This is my potentially garbled synthesis of his stuff, anyway.
4Mo Putera
I'm not sure about Friston's stuff to be honest.
But Watts lists a whole bunch of papers in support of the blindsight idea, contra Seth's claim — to quote Watts:
* "In fact, the nonconscious mind usually works so well on its own that it actually employs a gatekeeper in the anterious cingulate cortex to do nothing but prevent the conscious self from interfering in daily operations"
* footnotes: Matsumoto, K., and K. Tanaka. 2004. Conflict and Cognitive Control. Science 303: 969-970; 113 Kerns, J.G., et al. 2004. Anterior Cingulate Conflict Monitoring and Adjustments in Control. Science 303: 1023-1026; 114 Petersen, S.E. et al. 1998. The effects of practice on the functional anatomy of task performance. Proceedings of the National Academy of Sciences 95: 853-860
* "Compared to nonconscious processing, self-awareness is slow and expensive"
* footnote: Matsumoto and Tanaka above
* "The cost of high intelligence has even been demonstrated by experiments in which smart fruit flies lose out to dumb ones when competing for food"
* footnote: Proceedings of the Royal Society of London B (DOI 10.1098/rspb.2003.2548)
* "By way of comparison, consider the complex, lightning-fast calculations of savantes; those abilities are noncognitive, and there is evidence that they owe their superfunctionality not to any overarching integration of mental processes but due to relative neurological fragmentation"
* footnotes: Treffert, D.A., and G.L. Wallace. 2004. Islands of genius. Scientific American 14: 14-23; Anonymous., 2004. Autism: making the connection. The Economist, 372(8387): 66
* "Even if sentient and nonsentient processes were equally efficient, the conscious awareness of visceral stimuli—by its very nature— distracts the individual from other threats and opportunities in its environment"
* footnote: Wegner, D.M. 1994. Ironic processes of mental control. Psychol. Rev. 101: 34-52
* "Chimpanzees have a higher brain-to-body ratio than orangutans, yet orangs
What fraction of economically-valuable cognitive labor is already being automated today? How has that changed over time, especially recently?
I notice I'm confused about these ostensibly extremely basic questions, which arose in reading Open Phil's old CCF-takeoff report, whose main metric is "time from AI that could readily[2] automate 20% of cognitive tasks to AI that could readily automate 100% of cognitive tasks". A cursory search of Epoch's data, Metaculus, and this forum didn't turn up anything, but I didn't spend much time at all doing so.
I was originally motivated by wanting to empirically understand recursive AI self-improvement better, which led to me stumbling upon the CAIS paper Examples of AI Improving AI, but I don't have any sense whatsoever of how the paper's 39 examples as of Oct-2023 translate to OP's main metric even after constraining "cognitive tasks" in its operational definition to just AI R&D.
A survey was administered to attendees of three AI conferences during the summer of 2018 (ICML, IJCAI and the HLAI conference). The survey included questions for estimating AI capabilities over the next d
Did e.g. a telephone operator in 1910 perform cognitive labor, by the definition we want to use here?
1Mo Putera
I'm mainly wondering how Open Phil, and really anyone who uses fraction of economically-valuable cognitive labor automated / automatable (e.g. the respondents to that 2018 survey; some folks on the forum) as a useful proxy for thinking about takeoff, tracks this proxy as a way to empirically ground their takeoff-related reasoning. If you're one of them, I'm curious if you'd answer your own question in the affirmative?
2faul_sname
I am not one of them - I was wondering the same thing, and was hoping you had a good answer.
If I was trying to answer this question, I would probably try to figure out what fraction of all economically-valuable labor each year was cognitive, the breakdown of which tasks comprise that labor, and the year-on-year productivity increases on those task, then use that to compute the percentage of economically-valuable labor that is being automated that year.
Concretely, to get a number for the US in 1900 I might use a weighted average of productivity increases across cognitive tasks in 1900, in an approach similar to how CPI is computed
* Look at the occupations listed in the 1900 census records
* Figure out which ones are common, and then sample some common ones and make wild guesses about what those jobs looked like in 1900
* Classify those tasks as cognitive or non-cognitive
* Come to estimate that record-keeping tasks are around a quarter to a half of all cognitive labor
* Notice that typewriters were starting to become more popular - about 100,000 typewriters sold per year
* Note that those 100k typewriters were going to the people who would save the most time by using them
* As such, estimate 1-2% productivity growth in record-keeping tasks in 1900
* Multiply the productivity growth for record-keeping tasks by the fraction of time (technically actually 1-1/productivity increase but when productivity increase is small it's not a major factor)
* Estimate that 0.5% of cognitive labor was automated by specifically typewriters in 1900
* Figure that's about half of all cognitive labor automation in 1900
and thus I would estimate ~1% of all cognitive labor was automated in 1900. By the same methodology I would probably estimate closer to 5% for 2024.
Again, though, I am not associated with Open Phil and am not sure if they think about cognitive task automation in the same way.
I chose to study physics in undergrad because I wanted to "understand the universe" and naively thought string theory was the logically correct endpoint of this pursuit, and was only saved from that fate by not being smart enough to get into a good grad school. Since then I've come to conclude that string theory is probably a dead end, albeit an astonishingly alluring one for a particular type of person. In that regard I find anecdotes like the following by Ron Maimon on Physics SE interesting — the reason string theorists believe isn’t the same as what they tell people, so it’s better to ask for their conversion stories:
I think that it is better to ask for a compelling argument that the physics of gravity requires a string theory completion, rather than a mathematical proof, which would be full of implicit assumptions anyway. The arguments people give in the literature are not the same as the personal reasons that they believe the theory, they are usually just stories made up to sound persuasive to students or to the general public. They fall apart under scrutiny. The real reasons take the form of a conversion story, and are much more subjective, and much less persuasive to everyo
In pure math, mathematicians seek "morality", which sounds similar to Ron's string theory conversion stories above. Eugenia Cheng's Mathematics, morally argues:
I claim that although proof is what supposedly establishes the undeniable truth of a piece of mathematics, proof doesn’t actually convince mathematicians of that truth. And something else does.
... formal mathematical proofs may be wonderfully watertight, but they are impossible to understand. Which is why we don’t write whole formal mathematical proofs. ... Actually, when we write proofs what we have to do is convince the community that it could be turned into a formal proof. It is a highly sociological process, like appearing before a jury of twelve good men-and-true. The court, ultimately, cannot actually know if the accused actually ‘did it’ but that’s not the point; the point is to convince the jury. Like verdicts in court, our ‘sociological proofs’ can turn out to be wrong—errors are regularly found in published proofs that have been generally accepted as true. So much for mathematical proof being the source of our certainty. Mathematical proof in practice is certainly fallible.
The more you know about particle physics and quantum field theory, the more inevitable string theory seems. There are just too many connections. However, identifying the specific form of string theory that corresponds to our universe is more of a challenge, and not just because of the fabled 10^500 vacua (though it could be one of those). We don't actually know either all the possible forms of string theory, or the right way to think about the physics that we can see. The LHC, with its "unnaturally" light Higgs boson, already mortally wounded a particular paradigm for particle physics (naturalness) which in turn was guiding string phenomenology (i.e. the part of string theory that tries to be empirically relevant). So along with the numerical problem of being able to calculate the properties of a given string vacuum, the conceptual side of string theory and string phenomenology is still wide open for discovery.
9Alexander Gietelink Oldenziel
I asked a well-known string theorist about the fabled 10^500 vacua and asked him whether he worried that this would make string theory a vacuous theory since a theory that fits anything fits nothing. He replied ' no, no the 10^500 'swampland' is a great achievement of string theory - you see... all other theories have infinitely many adjustable parameters'. He was saying string theory was about ~1500 bits away from the theory of everything but infinitely ahead of its competitors.
Diabolical.
Much ink has been spilled on the scientific merits and demerits of string theory and its competitors. The educated reader will recognize that this all this and more is of course, once again, solved by UDASSA.
2Noosphere89
Re other theories, I don't think that all other theories in existence have infinitely many adjustable parameters, and if he's referring to the fact that lots of theories have adjustable parameters that can range over the real numbers, which are infinitely complicated in general, than that's different, and string theory may have this issue as well.
Re string theory's issue of being vacuous, I think the core thing that string theory predicts that other quantum gravity models don't is that at the large scale, you recover general relativity and the standard model, whereas no other theory can yet figure out a way to properly include both the empirical effects of gravity and quantum mechanics in the parameter regimes where they are known to work, so string theory predicts more just by predicting the things other quantum mechanics predicts while having the ability to include in gravity without ruining the other predictions, whereas other models of quantum gravity tend to ruin empirical predictions like general relativity approximately holding pretty fast.
I used to consider it a mystery that math was so unreasonably effective in the natural sciences, but changed my mind after reading this essay by Eric S. Raymond (who's here on the forum, hi and thanks Eric), in particular this part, which is as good a question dissolution as any I've seen:
The relationship between mathematical models and phenomenal prediction is complicated, not just in practice but in principle. Much more complicated because, as we now know, there are mutually exclusive ways to axiomatize mathematics! It can be diagrammed as follows (thanks to Jesse Perry for supplying the original of this chart):
(it's a shame this chart isn't rendering properly for some reason, since without it the rest of Eric's quote is ~incomprehensible)
The key transactions for our purposes are C and D -- the translations between a predictive model and a mathematical formalism. What mystified Einstein is how often D leads to new insights.
We begin to get some handle on the problem if we phrase it more precisely; that is, "Why does a good choice of C so often yield new knowledge via D?"
The simplest answer is to invert the question and treat it as a definition. A "good choi
Interesting. This reminds me of a related thought I had: Why do models with differential equations work so often in physics but so rarely in other empirical sciences? Perhaps physics simply is "the differential equation science".
Which is also related to the frequently expressed opinion that philosophy makes little progress because everything that gets developed enough to make significant progress splits off from philosophy. Because philosophy is "the study of ill-defined and intractable problems".
Not saying that I think these views are accurate, though they do have some plausibility.
1Mo Putera
(To be honest, to first approximation my guess mirrors yours.)
5Garrett Baker
Flagging that those two examples seem false. The weather is chaotic, yes, and there's a sense in which the economy is anti-inductive, but modeling methods are advancing, and will likely find more loop-holes in chaos theory.
For example, in thermodynamics, temperature is non-chaotic while the precise kinetic energies and locations of all particles are. A reasonable candidate similarity in weather are hurricanes.
Similarly as our understanding of the economy advances it will get more efficient which means it will be easier to model. eg (note: I've only skimmed this paper). And definitely large economies are even more predictable than small villages, talk about not having a competitive market!
3Mo Putera
Thanks for the pointer to that paper, the abstract makes me think there's a sort of slow-acting self-reinforcing feedback loop between predictive error minimisation via improving modelling and via improving the economy itself.
re: weather, I'm thinking of the chart below showing how little gain we get in MAE vs compute, plus my guess that compute can't keep growing far enough to get MAE < 3 °F a year out (say). I don't know anything about advancements in weather modelling methods though; maybe effective compute (incorporating modelling advancements) may grow indefinitely in terms of the chart.
2Garrett Baker
I didn't say anything about temperature prediction, and I'd also like to see any other method (intuition based or otherwise) do better than the current best mathematical models here. It seems unlikely to me that the trends in that graph will continue arbitrarily far.
Yeah, that was my claim.
4Mo Putera
Matt Leifer, who works in quantum foundations, espouses a view that's probably more extreme than Eric Raymond's above to argue why the effectiveness of math in the natural sciences isn't just reasonable but expected-by-construction. In his 2015 FQXi essay Mathematics is Physics Matt argued that
(Matt notes as an aside that he's arguing for precisely the opposite of Tegmark's MUH.)
Why "scale-free network"?
As an aside, Matt's theory of theory-building explains (so he claims) what mathematical intuition is about: "intuition for efficient knowledge structure, rather than intuition about an abstract mathematical world".
So what? How does this view pay rent?
Matt further develops the argument that the structure of human knowledge being networked-not-hierarchical implies that the idea that there is a most fundamental discipline, or level of reality, is mistaken in Against Fundamentalism, another FQXi essay published in 2018.
3localdeity
I would also comment that, if the environment was so chaotic that roughly everything important to life could not be modeled—if general-purpose modeling ability was basically useless—then life would not have evolved that ability, and "intelligent life" probably wouldn't exist.
2romeostevensit
The two concepts that I thought were missing from Eliezer's technical explanation of technical explanation that would have simplified some of the explanation were compression and degrees of freedom. Degrees of freedom seems very relevant here in terms of how we map between different representations. Why are representations so important for humans? Because they have different computational properties/traversal costs while humans are very computationally limited.
1Mo Putera
Can you say more about what you mean? Your comment reminded me of Thomas Griffiths' paper Understanding Human Intelligence through Human Limitations, but you may have meant something else entirely.
Griffiths argued that the aspects we associate with human intelligence – rapid learning from small data, the ability to break down problems into parts, and the capacity for cumulative cultural evolution – arose from the 3 fundamental limitations all humans share: limited time, limited computation, and limited communication. (The constraints imposed by these characteristics cascade: limited time magnifies the effect of limited computation, and limited communication makes it harder to draw upon more computation.) In particular, limited computation leads to problem decomposition, hence modular solutions; relieving the computation constraint enables solutions that can be objectively better along some axis while also being incomprehensible to humans.
3romeostevensit
Thanks for the link. I mean that predictions are outputs of a process that includes a representation, so part of what's getting passed back and forth in the diagram are better and worse fit representations. The degrees of freedom point is that we choose very flexible representations, whittle them down with the actual data available, then get surprised that that representation yields other good predictions. But we should expect this if Nature shares any modular structure with our perception at all, which it would if there was both structural reasons (literally same substrate) and evolutionary pressure for representations with good computational properties i.e. simple isomorphisms and compressions.
Back in 2020, when Microsoft, Meta, and Google increased the useful life [of their IT assets] from 3 to 4 years, we were still in the year 2 BC (Before ChatGPT). Now, in present-day 3 AD (After Da Launch of ChatGPT) ...
Science doesn’t work despite scientists being asses. Science works, to at least some extent, because scientists are asses. Bickering and backstabbing are essential elements of the process. Haven’t any of these guys ever heard of “peer review”?
There’s this myth in wide circulation: rational, emotionless Vulcans in white coats, plumbing the secrets of the universe, their Scientific Methods unsullied by bias or emotionalism. Most people know it’s a myth, of course; they subscribe to a more nuanced view in which scientists are as petty and vain and human as anyone (and as egotistical as any therapist or financier), people who use scientific methodology to tamp down their human imperfections and manage some approximation of objectivity.
But that’s a myth too. The fact is, we are all humans; and humans come with dogma as standard equipment. We can no more shake off our biases than Liz Cheney could pay a compliment to Barack Obama. The best we can do— the best science can do— is make sure that at least, we get to choose among competing biases.
That’s how science works. It’s not a hippie love-in; it
I can't tell from their main text whether the human authors of this math paper that solved the $1,000 Erdos problem 707 used ChatGPT-5 Pro or Thinking or what. Supposing they didn't use Pro, I wonder how their experience would've been if they did; they said that vibe-coding the 6,000+ line Lean proof with ChatGPT took about a week and was "extremely annoying"
(technically one of the authors said Marshall Hall Jr. already solved it in 1947 via counterexample)
I dislike hype-flavored summaries by the likes of Sebastien Bubeck et al, so I appreciated these screenshots of the paper and accompanying commentary by @life2030com on how the authors felt about using ChatGPT to assist them in all this:
I found that "curious inversion" remark at the end interesting too.
Systematic reviews (SRs) inform evidence-based decision making. Yet, they take over a year to complete, are prone to human error, and face challenges with reproducibility; limiting access to timely and reliable information. We developed otto-SR, an end-to-end agentic workflow using large language models (LLMs) to support and automate the SR workflow from initial search to analysis. We found that otto-SR outperformed traditional dual human workflows in SR screening (otto-SR: 96.7% sensitivity, 97.9% specificity; human: 81.7% sensitivity, 98.1% specificity) and data extraction (otto-SR: 93.1% accuracy; human: 79.7% accuracy). Using otto-SR, we reproduced and updated an entire issue of Cochrane reviews (n=12) in two days, representing approximately 12 work-years of traditional systematic review work. Across Cochrane reviews, otto-SR incorrectly excluded a median of 0 studies (IQR 0 to 0.25), and found a median of 2.0 (IQR 1 to 6.5) eligible studies likely missed by the original authors. Meta-analyses revealed that otto-SR generated newly statistically significant conclusions in 2 re
Another neat example of mundane LLM utility, by Tim Gowers on Twitter:
I've seen lots of variations of this anecdote by mathematicians, but none by Fields medalists.
Also that last sentence singles Gowers out among top-tier mathematicians as far as I can tell for thinking that AI will obsolete him soon at the thing he does best. Terry Tao and Kevin Buzzard in contrast don't give me this impression at all, as excited and engaged as they are with AI x math.
... big guests just don't really matter that much if you just look at what are the most popular episodes, or what in the long run helps a podcast grow. By far my most popular guest is Sarah Paine, and she, before I interviewed her, was just a scholar who was not publicly well-known at all, and I just found her books quite interesting—so my most popular guests are Sarah Paine and then Sarah Paine, Sarah Paine, Sarah Paine because I have electric chairs(?) a lecture series with her. And by the way, from a viewer-a-minute adjusted basis, I host the Sarah Paine podcast where I occasionally talk about AI.
(After Sarah Paine comes geneticist David Reich, then Satya Nadella and Mark Zuckerberg, "then [Sholto & Trenton] or Leopold (Aschenbrenner) or something, then you get to the lab CEOs or something")
You can see it as an example of 'alpha' vs 'beta'. When someone asks me about the value of someone as a guest, I tend to ask: "do they have anything new to say? didn't they just do a big interview last year?" and if they don't but they're big, "can you ask them good questions that get them out of their 'book'?" Big guests are not necessarily as valuable as they may seem because they are highly-exposed, which means both that (1) they have probably said everything they will said before and there is no 'news' or novelty, and (2) they are message-disciplined and careful to "talk their book". (In this analogy, "alpha" represents undiscovered or neglected interview topics which can be extracted mostly just by finding it and then asking the obvious question, usually by interviewing new people; "beta" represents doing standard interview topics/people, but much more so - harder, faster, better - and getting new stuff that way.)
Lex Fridman podcasts are an example of this: he often hosts very big guests like Mark Zuckerberg, but nevertheless, I will sit down and skim through the transcript of 2-4 hours of content, and find nothing even worth excerpting for my notes. Fridman notoriously does n... (read more)
I like the optimal forager take, seems intuitively correct. I'd add that Dwarkesh struck gold by getting you on his podcast too. (Tangentially: this grand theory of intelligence video snippet reminds me of a page-ish-long writeup on that I stumbled upon deep in the bowels of https://gwern.net/ which I've annoyingly never been able to find again.)
Also thanks for the pointer to Werbos, his website Welcome to the Werbos World! funnily enough struck me as crackpot-y and I wouldn't have guessed just from the landing page that he's the discoverer of backprop, respected former program director at the NSF, etc.
7gwern
Probably https://gwern.net/newsletter/2021/05#master-synthesis
That's what makes it alpha! If he was as legible as, say, Hinton, he would be mined out by now, and nothing but beta. (Similar situation to Schmidhuber - 'obvious crackpot' - although he's such a self-promoter that he overcomes it, and so at this point there's no alpha talking to him; the stuff that would be interesting, like his relationship to certain wealthy Italians, or to King Bonesaws, or how he's managed to torpedo his career so spectacularly, he will not talk about. Also, I understand he likes to charge people for the privilege of talking to him.) You have to have both domain knowledge and intellectual courage to know about Werbos and eg. read his old interviews and be willing to go out on a limb and interview him.
2Chris_Leong
This seems to underrate the value of distribution. I suspect another factor to take into account is the degree of audience overlap. Like there's a lot of value in booking a guest who has been on a bunch of podcasts, so long as your particular audience isn't likely to have been exposed to them.
2sjadler
I’d guess that was “I have a lecture series with her” :-)
For my own future reference, here are some "benchmarks" (very broadly construed) I pay attention to as of Nov 2025, a mix of serious and whimsical:
the AI village and blog, not really a "benchmark" per se but my richest source of intuitions about current frontier models' capabilities at open-ended long-horizon tasks by far, made me notice stuff like the Claudes being way better than other "benchmark-equiv" frontier models
Chats on read.haus with AI simulations of prominent authors become preferable to reading the latter's real content. Scott Alexander, Sarah Constantin, Spencer Greenberg, Byrne Hobart, Tyler Cowen, Dwarkesh Patel, Andy Matuschak etc are all on there but they never come across quite right
Balioc's A taxonomy of bullshit jobs has a category called Worthy Work Made Bullshit which resonated with me most of all:
Worthy Work Made Bullshit is perhaps the trickiest and most controversial category, but as far as I’m concerned it’s one of the most important. This is meant to cover jobs where you’re doing something that is obviously and directly worthwhile…at least in theory…but the structure of the job, and the institutional demands that are imposed on you, turn your work into bullshit.
The conceptual archetype here is the Soviet tire factory that produces millions of tiny useless toy-sized tires instead of a somewhat-smaller number of actually-valuable tires that could be put on actual vehicles, because the quota scheme is badly designed. Everyone in that factory has a Worthy Work Made Bullshit job. Making tires is something you can be proud of, at least hypothetically. Making tiny useless tires to game a quota system is…not.
Nowadays we don’t have Soviet central planners producing insane demands, but we do have a marketplace that produces comparably-insane demands, especially in certain fields.
Potentially relevant: this thread about a massive software service that did useful work, but ultimately could've been outperformed (at 100x) by a small, easy-to-implement adjustment to the overarching system.
I just learned about the idea of "effectual thinking" from Cedric Chin's recent newsletter issue. He notes, counterintuitively to me, that it's the opposite of causal thinking, and yet it's the one thing in common in all the successful case studies he could find in business:
The only answer that fits with the twenty-seven odd cases that we’ve published on the Idea Maze is a thing called effectual thinking.
Effectual thinking is a concept proposed by Professor Saras D Sarasvathy in a 2001 paper — a paper that was also, hilariously, described as “the first good paper I’ve seen” by Vinod Khosla, the renowned venture capitalist and cofounder of Sun Microsystems.
Saravathy didn’t pull this theory out of thin air: she hunted down a list of entrepreneurs who, at the time of her paper, represented a near-complete list of experienced founders (of enduring companies) in the US from the period of 1960 to 1985.
These folk were:
Repeat entrepreneurs (minimum three ventures, though the average number of companies started in her sample was seven new ventures)
Were successful and serious businesspeople (at least one of those ventures had gone public, and they had spent minimum 10 years with that company
(see also search term "forward chaining vs back-chaining.")
This seems like reasonable life advice for people generally trying to accumulate resources and do something cool. I'm not sure about people who actually have specific goals they want to accomplish. I think in the domain of AI safety, forward chaining is insufficient (seems like the kind of thing that gets you OpenAI and Anthropic)
The principles I sort of try to live by are, each 2-weeks, I should have done:
some actions that forward chain towards more compounding resources
some actions explicitly backchaining from longterm goals
ship something concrete to users
do something wholesome
Which is I think aiming to accomplish similar goals towards the OP, without losing the plot on my more specific goals.
This maybe reminds me:
I currently feel confused about how to integrate "the kind of thinking that is good at momentum / action" and "the kind of good that is good at creative strategy". And it seems like there should be more of a way to unify them into a holistic way-of-being.
The four checksums above are there to make sure I'm not being myopic in some way in a broader sense, but they apply more at the timescale of weeks than hours or days.
You might just say "well, idk, each week or day, just figure out if it's more like a momentum week or more like a creative strategy week". I feel dissatisfied with this for some reason.
At least part of it is "I think on average people/me could use to be in creative/broader strategy mode more often, even when in a Momentum mode period."
Another part is "there are strategy skills I want to be practicing, that are hard to practice if I don't do them basically every day. They aren't as relevant in a momentum-period, but they're not zero relevant.
Hrm. I think maybe what's most dissatisfying right now is that I just haven't compressed all the finnicky details of it, and it feels overwhelming to think about the entire "how to think" project, which is usually an indicator I am missing the right abstraction.
I’ve written something like six or seven personal wikis over the past decade. It’s actually an incredibly advanced form of procrastination1. At this point I’ve tried every possible design choice.
Lifecycle: I’ve built a few compiler-style wikis: plain-text files in a git repo statically compiled to HTML. I’ve built a couple using live servers with server-side rendering. The latest one is an API server with a React frontend.
Storage: I started with plain text files in a git repo, then moved to an SQLite database with a simple schema. The latest version is an avant-garde object-oriented hypermedia database with bidirectional links implemented on top of SQLite.
Markup: I used Markdown here and there. Then I built my own TeX-inspired markup language. Then I tried XML, with mixed results. The latest version uses a WYSIWYG editor made with ProseMirror.
And yet I don’t use them. Why? Building them was fun, sure, but there must be utility to a personal database.
At first I thought the problem was friction: the higher the activation energy to u
Minimizing friction is surprisingly difficult. I keep plain-text notes in a hierarchical editor (cherrytree), but even that feels too complicated sometimes. This is not just about the tool... what you actually need is a combination of the tool and the right way to use it.
(Every tool can be used in different ways. For example, suppose you write a diary in MS Word. There are still options such as "one document per day" or "one very long document for all", and things in between like "one document per month", which all give different kinds of friction. The one megadocument takes too much time to load. It is more difficult to search in many small documents. Or maybe you should keep your current day in a small document, but once in a while merge the previous days into the megadocument? Or maybe switch to some application that starts faster than MS Word?)
Forgetting is an important part. Even if you want to remember forever, you need some form of deprioritizing. Something like "pages you haven't used for months will get smaller, and if you search for keywords, they will be at the bottom of the result list". But if one of them suddenly becomes relevant again, maybe the connected ones become relevant, too? Something like associations in brain. The idea is that remembering the facts is only a part of the problem; making the relevant ones more accessible is another. Because searching in too much data is ultimately just another kind of friction.
It feels like a smaller version of the internet. Years ago, the problem used to be "too little information", now the problem is "too much information, can't find the thing I actually want".
Perhaps a wiki, where the pages could get flagged as "important now" and "unimportant"? Or maybe, important for a specific context? And by default, when you choose a context, you would only see the important pages, and the rest of that only if you search for a specific keyword or follow a grey link. (Which again would require some work creating
2Milan W
@dkl9 wrote a very eloquent and concise piece arguing in favor of ditching "second brain" systems in favor of SRSs (Spaced Repetition Systems, such as Anki).
2Jonas Hallgren
I like to think of learning and all of these things as self-contained smaller self-contained knowledge trees. Building knowledge trees that are cached, almost like creatin zip files and systems where I store a bunch of zip files similar to what Elizier talks about in The Sequences.
Like when you mention the thing about Nielsen on linear algebra it opens up the entire though tree there. I might just get the association to something like PCA and then I think huh, how to ptimise this and then it goes to QR-algorithms and things like a householder matrix and some specific symmetric properties of linear spaces...
If I have enough of these in an area then I might go back to my anki for that specific area. Like if you think from the perspective of schedulling and storage algorithms similar to what is explored in algorithms to live by you quickly understand that the magic is in information compression and working at different meta-levels. Zipped zip files with algorithms to expand them if need be. Dunno if that makes sense, agree with the exobrain creep that exists though.
Externally, we were often confused with other, better-knownorganizations. And internally, many felt that “Open Philanthropy” no longer quite fit. When the name was chosen in 2014, it signaled both our openness to many cause areas and our unusual level of transparency. Back then, we published notes from nearly every conversation we had with experts and even wrote candidly about the potential downsides of new hires. As we grew, that kind of radical transparency didn’t scale well. While we still prioritize openness and sharing our reasoning, these are now part of a broader set of values rather than the centerpiece of our identity.
It was the radical transparency that I found attractive about OP (and GW) a long time ago, which is why this caught my eye. More on how they think about the costs and benefits of information sharing (2016 post by Holden, so I suppose this was a long time coming):
... near-comprehensive information sharing is an appropriate goal for G
ZIERLER: Michael, we've already covered what you're doing currently, so for the last part of our talk, I'd like to ask a retrospective question, and then we'll end looking to the future. Between quantum information and open science, I wonder if you've thought broadly about the Kuhnian idea of scientific revolutions. Both quantum information and open science were not areas that he thought about. But to the extent that you believe in that paradigm—and I don't know if you; that's a question in and of itself—how does your expertise or the reality of these fields influence the idea of scientific progress, the idea of scientific revolutions?
NIELSEN: Just focusing on Kuhn, he had a very narrow sense of what a scientific revolution is. One of the most common criticisms of him is that he was too focused on physics in particular; and to some extent, chemistry. He was originally trained as a theoretical physicist, and it really shows in his writing. Something like quantum mechanics or general relativity is disrupting an existing order. Well, quantum information isn't disrupting any existing order. It's not like there's a whole bunch of ex-classical informa
Venkatesh Rao surprised me in What makes a good teacher? by saying the opposite of what I expected him to say re: his educational experience, given who he is:
While my current studies have no live teachers in the loop, each time I sit down to study something seriously, I’m reminded of how much I’m practicing behaviors first learned under the watchful eye of good teachers. We tend to remember the exceptionally charismatic (which is not the same thing as good), and exceptionally terrible teachers, but much of what we know about how to learn, how to study, comes from the quieter good teachers, many of whom we forget.
It also strikes me, reflecting on my own educational path — very conventional both on paper and in reality — that the modern public discourse around teaching and learning has been hijacked to a remarkable degree by charismatic public figures mythologizing their own supposedly maverick education stories.
These stories often feature exaggerated elements of rebellion, autodidact mastery, subversive hacking, heroic confrontations with villainous teachers and schoolyard bullies, genius non-neurotypical personal innovations and breakthroughs, and powerful experiences outside forma
(Not a take, just pulling out infographics and quotes for future reference from the new DeepMind paper outlining their approach to technical AGI safety and security)
Overview of risk areas, grouped by factors that drive differences in mitigation approaches:
Overview of their approach to mitigating misalignment:
I often draw a distinction between the political elites of Washington DC and the industrial elites of Silicon Valley with a joke: in San Francisco reading books, and talking about what you have read, is a matter of high prestige. Not so in Washington DC. In Washington people never read books—they just write them.
To write a book, of course, one must read a good few. But the distinction I drive at is quite real. In Washington, the man of ideas is a wonk. The wonk is not a generalist. The ideal wonk knows more about his or her chosen topic than you ever will. She can comment on every line of a select arms limitation treaty, recite all Chinese human rights violations that occurred in the year 2023, or explain to you the exact implications of the new residential clean energy tax credit—but never all at once. ...
Washington intellectuals are masters of small mountains. Some of their peaks are more difficult to summit than others. Many smaller slopes are nonetheless ja
Asked ChatGPT o3 to read 6 links (nothing intensive, just the familiar essays on cognitive decoupling by Jacob Falkovich, Sarah Constantin, John Nerst, drossbucket) and its reasoning repeatedly mentions this which I've never seen before (emphasis mine):
I’m planning to review each article carefully. I’ll first open lines around the 250-line mark to get a deeper view, but I’m also mindful of the word limit. ...
I’m thinking I’ll need to open each article and focus on the part about "decoupling." To get more detailed, I’ll try opening further into the text — maybe around 250 lines —to ensure I capture enough context without overwhelming myself.
I plan to open each article and examine lines near the bottom to view relevant content. There's a limit of 200 lines, so I might need to focus on searching for specific keywords like "decoupling."
I’m trying to locate specific references to "decoupling" in each article. I want to open around 250 lines to make sure I find enough content for a thorough summary. I’ll use a quick search to find "decoupling" precisely, but if it’s not yielding precise results, I’ll expand the search to ensure I get more lines. It’s about balancing deep acc
I think this might be an attempted countermeasure against prompt injection. That is, it wants to mix autoregressive and reconstructed residuals. Otherwise, it might lose it's train of thought (end up continuing the article not following the prompt).
Saving mathematician Robert Ghrist's tweet here for my own future reference re: AI x math:
workflow of the past 24 hours... * start a convo w/GPT-o3 about math research idea [X] * it gives 7 good potential ideas; pick one & ask to develop * feed -o3 output to gemini-2.5-pro; it finds errors & writes feedback * paste feedback into -o3 and say asses & respond * paste response into gemini; it finds more problems * iterate until convergence * feed the consensus idea w/detailed report to grok-3 * grok finds gaping error, fixes by taking things in different direction (!!!) * gemini agrees: big problems, now ameliorated * output final consensus report * paste into claude-3.7 and ask it to outline a paper * approve outline; request latex following my style/notation conventions * claude outputs 30 pages of dense latex, section by section, one-shot (!) ==== is this correct/watertight? (surely not) is this genuinely novel? (pretty sure yes) is this the future? (no, it's the present) ==== everybody underestimates not only what is coming but what can currently be done w/existing tools.
Someone asked why split things between o3 and 2.5 Pro; Ghrist:
they have complementary strengths and each picks up
I enjoyed Brian Potter's Energy infrastructure cheat sheet tables over at Construction Physics, it's a great fact post. Here are some of Brian's tables — if they whet your appetite, do check out his full essay.
Energy quantities:
Units and quantities
Kilowatt-hours
Megawatt-hours
Gigawatt-hours
1 British Thermal Unit (BTU)
0.000293
iPhone 14 battery
0.012700
1 pound of a Tesla battery pack
0.1
1 cubic foot of natural gas
0.3
2000 calories of food
2.3
1 pound of coal
2.95
1 gallon of milk (calorie value)
3.0
1 gallon of gas
33.7
Tesla Model 3 standard battery pack
57.5
Typical ICE car gas tank (15 gallons)
506
1 ton of TNT
1,162
1 barrel of oil
1,700
1 ton of oil
11,629
12
Tanker truck full of gasoline (9300 gallons)
313,410
313
LNG carrier (180,000 cubic meters)
1,125,214,740
1,125,215
1,125
1 million tons of TNT (1 megaton)
1,162,223,152
1,162,223
1,162
Oil supertanker (2 million barrels)
3,400,000,000
3,400,000
3,400
It's amazing that a Tesla Model 3's standard battery pack has an OOM less energy capacity than a typical 15-gallon ICE car gas tank, and is probably heavier to... (read more)
A subgenre of fiction I wish I could read more of is rationalist-flavored depictions of utopia that centrally feature characters who intentionally and passionately pursue unpleasant experiences, which I don't see much of. It's somewhat surprising since it's a pretty universal orientation.
For instance, and this is a somewhat extreme version, I'm a not-that-active member of a local trail running group (all professionals with demanding day jobs) that meets regularly for creative sufferfests like treasure hunt races in the mountains, some of whom regularly fly... (read more)
In books about the Culture sci fi universe such things are described a couple of times. E.g. in the novel "Use of Weapons" the "crew" (the ship is fully automated, so more like permanent passengers) of a ship deliberately weaken their immune system to basically get a seasonal cold just for the experience, which otherwise could not happen due to their genetically enhanced immune system.
2Seth Herd
Also lava rafting and other extreme sports, maybe in Look to Windward which focuses a bit more on the Culture. Many of the human protagonists in the Culture experience significant self-hatred, although that's not the only reason to seek out experiences so difficult they may become net negative. It's as though the Culture is missing advanced therapeutic techniques along with a desire for immortality. I'd like an updated utopia.
3Richard Horvath
Well, the disturbed protagonists in the Culture series (as in: books, and in the whole of the fictional universe) are usually not from the "Culture" (one particular civilizations within the whole fictional universe), but outsiders hired to act as agents.
4Seth Herd
Hm, interesting. I remembered that about Zakalwe but my memory for the others is vague. So maybe Culture citizens are so well-adjusted that they wouldn't risk their lives?
Pilish is a constrained writing style where the number of letters in consecutive words match the digits of pi. The canonical intro-to-Pilish sentence is "How I need a drink, alcoholic of course, after the heavy lectures involving quantum mechanics!"; my favorite Pilish poetry is Mike Keith's Near a Raven, a retelling of Edgar Allan Poe's "The Raven" stretching to 740 digits of pi (nowhere near Keith's longest, that would be the 10,000-word world record-setting Not a Wake), which begins delightfully like so:
I asked GPT 4.5 to write a system prompt and user message for models to write Pilish poems, feeding it your comment as context.
Then I gave these prompts to o1 (via OpenAI's playground).
GPT 4.5's system prompt
You are an expert composer skilled in writing poetry under strict, unusual linguistic constraints, specifically "Pilish." Pilish is a literary constraint in which the length of consecutive words precisely matches each digit of π (pi). The first word contains 3 letters, second word 1 letter, third word 4 letters, fourth word 1 letter, fifth word 5 letters, sixth word 9 letters, and so forth, accurately reflecting the sequence of pi’s digits.
For example, the classic Pilish sentence is:
"How I need a drink, alcoholic of course, after the heavy lectures involving quantum mechanics!"
As another reference, Mike Keith famously wrote a poem titled "Near a Raven," a Pilish version of Edgar Allan Poe's "The Raven," precisely encoding hundreds of pi digits into elegant verse.
Important guidelines:
You must strictly adhere to the Pilish constraint without exception or deviation.
Carefully recount each word to match each digit exactly.
Maintain coherence, clarity, and literary quality in your writing.
Make sure the poem or text is meaningful, evocative, creative, and intriguing, not just mechanically correct.
Verify each word length explicitly to ensure compliance with Pilish.
Your task is to craft a compelling, original poem in Pilish form. Do not explain the poem afterward; the precision and artistry of the piece itself is sufficient.
--
GPT 4.5's user message
Write an original Pilish poem about pi itself: reflecting upon the wonder, mystery, and endless fascination humans have felt over millennia towards this famous mathematical constant. Make it vivid, thoughtful, and meticulously correct in matching the Pilish constraint. Begin now.
--
o1's completion
I asked o1 to count the letters in each word, comparing with pi:
And this seems correct!
2Mo Putera
This is great, thanks! Didn't think of the model-prompting-model trick.
4MondSemmel
LLMs use tokens instead of letters, so counting letters is sufficiently unnatural to them relative to their other competencies that I don't see much value in directly asking LLMs to do this kind of thing. At least give them some basic scaffolding, like a full English dictionary with a column which explicitly indicates respective word lengths. In particular, the Gemini models have a context window of 1M tokens, which should be enough to fit most of the Oxford English Dictionary in there (since it includes 171k words which are in current use).
4Mo Putera
I think I failed to implicitly convey that I meant all this in jest, that I get a lot of personal enjoyment value out of silly poetry constrained by artificial rules, and that I was guessing at least someone else on the forum would share this enjoyment. I do like your scaffolding idea, might just try it out.
There's a lot of fun stuff in Anders Sandberg's 1999 paper The Physics of Information Processing Superobjects: Daily Life Among the Jupiter Brains. One particularly vivid detail was (essentially) how the square-cube law imposes itself upon Jupiter brain architecture by forcing >99.9% of volume to be comprised of comms links between compute nodes, even after assuming a "small-world" network structure allowing sparse connectivity between arbitrarily chosen nodes by having them be connected by a short series of intermediary links with only 1% of links bein... (read more)
From Brian Potter's Construction Physics newsletter I learned about Taara, framed as "Google's answer to Starlink" re: remote internet access, using ground-based optical communication instead of satellites ("fiber optics without the fibers"; Taara calls them "light bridges"). I found this surprising. Even more surprisingly, Taara isn't just a pilot but a moneymaking endeavor if this Wired passage is true:
Taara is now a commercial operation, working in more than a dozen countries. One of its successes came in crossing the Congo River. On one side was Brazza
Peter Watts' 2006 novel Blindsight has this passage on what it's like to be a "scrambler", superintelligent yet nonsentient (in fact superintelligent because it's unencumbered by sentience), which I read a ~decade ago and found unforgettable:
Imagine you're a scrambler.
Imagine you have intellect but no insight, agendas but no awareness. Your circuitry hums with strategies for survival and persistence, flexible, intelligent, even technological—but no other circuitry monitors it. You can think of anything, yet are conscious of nothing.
I like Greg Egan's "outlooks" from Diaspora for many reasons: as a reversible customisable solution to value drift, as a way to temporarily experience the world from the perspective of people with very different aesthetic sensibilities or deep values, to approach problem-solving differently, maybe even to simulate high-level generators of disagreement (which would be a boon for erisology), and I wish it already existed:
Any citizen with a mind broadly modeled on a flesher's was vulnerable to drift: the decay over time of even the most cherished goals and va
Ravi Vakil's advice for potential PhD students includes this bit on "tendrils to be backfilled" that's stuck with me ever since as a metaphor for deepening understanding over time:
Here's a phenomenon I was surprised to find: you'll go to talks, and hear various words, whose definitions you're not so sure about. At some point you'll be able to make a sentence using those words; you won't know what the words mean, but you'll know the sentence is correct. You'll also be able to ask a question using those words. You still won't know what the words mean, but yo
If you want to get a job as a software witch, you’re going to have to pass a whiteboard interview. We all do them, as engineers–often as a part of our morning ritual, along with arranging a beautiful grid of xterms across the astral plane, and compulsively running ls in every nearby directory–just in case things have shifted during the night–the incorporeal equivalent of rummaging through that drawer in the back of the kitche
Out of curiosity — how relevant is Holden's 2021 PASTA definition of TAI still to the discourse and work on TAI, aside from maybe being used by Open Phil (not actually sure that's the case)? Any pointers to further reading, say here or on AF etc?
AI systems that can essentially automate all of the human activities needed to speed up scientific and technological advancement. I will call this sort of technology Process for Automating Scientific and Technological Advancement, or PASTA.3 (I mean PASTA to refer to either a single system or a collection of system
When I first read Hannu Rajaniemi's Quantum Thief trilogy c. 2015 I had two reactions: delight that this was the most my-ingroup-targeted series I had ever read, and a sinking feeling that ~nobody else would really get it, not just the critics but likely also most fans, many of whom would round his carefully-chosen references off to technobabble. So I was overjoyed to recently find Gwern's review of it, which Hannu affirms "perfectly nails the emotional core of the trilogy and, true to form, spots a number of easter eggs I thought no one would ever find", ... (read more)
The parts of the science I understand were all quite plausible (mind duplication/fractioning and motivations for doing so).
Beyond the accuracy of the science, this was one of the most staggeringly imaginative and beautifully written scifi books I've ever read. It's for a very particular audience, but if you're here you might be that audience. If you are, this might be the best book you've read.
3Mo Putera
Attention conservation notice: 3,000+ words of longform quotes by various folks on the nature of personal identity in a posthuman future, and hiveminds / clans
As an aside, one of the key themes running throughout the Quantum Thief trilogy is the question of how you might maintain personal identity (in the pragmatic security sense, not the philosophical one) in a future so posthuman that minds can be copied and forked indefinitely over time. To spoil Hannu's answer:
I take Anders Sandberg's answer to be on the other end of this spectrum; he doesn't mind changing over time such that he might end up wanting different things:
(I have mixed feelings about Anders' take: I have myself changed so profoundly since youth that that my younger self would not just disendorse but be horrified by the person I am now, yet I did endorse every step along the way, and current-me still does upon reflection (but of course I do). Would current-me also endorse a similar degree of change going forward, even subject to every step being endorsed by the me right before change? Most likely not, perhaps excepting changes towards some sort of reflective equilibrium.)
I interpret Holden Karnofsky's take to be somewhere in between, perhaps closer to Hannu's answer. Holden remarked that he doesn't find most paradoxical thought experiments about personal identity (e.g. "Would a duplicate of you be "you?"" or "If you got physically destroyed and replaced with an exact duplicate of yourself, did you die?") all that confounding because his personal philosophy on "what counts as death" dissolves them, and that his philosophy is simple, comprising just 2 aspects: constant replacement ("in an important sense, I stop existing and am replaced by a new person each moment") and kinship with future selves. Elaborating on the latter:
Richard Ngo goes in a different direction with the "personal identity in a posthuman future" question:
(I thought it was both interesting and predictable that Rob would f
1Mo Putera
The short story The Epiphany of Gliese 581 by Fernando Borretti has something of the same vibe as Rajaniemi's QT trilogy; Borretti describes it as inspired by Orion's Arm and the works of David Zindell. Here's a passage describing a flourishing star system already transformed by weakly posthuman tech:
Another star system, this time still being transformed:
One subsubgenre of writing I like is the stress-testing of a field's cutting-edge methods by applying it to another field, and seeing how much knowledge and insight the methods recapitulate and also what else we learn from the exercise. Sometimes this takes the form of parables, like Scott Alexander's story of the benevolent aliens trying to understand Earth's global economy from orbit and intervening with crude methods (like materialising a billion barrels of oil on the White House lawn to solve a recession hypothesised to be caused by an oil shortage) to... (read more)
I enjoyed these passages from Henrik Karlsson's essay Cultivating a state of mind where new ideas are born on the introspections of Alexander Grothendieck, arguably the deepest mathematical thinker of the 20th century.
In June 1983, Alexander Grothendieck sits down to write the preface to a mathematical manuscript called Pursuing Stacks. He is concerned by what he sees as a tacit disdain for the more “feminine side” of mathematics (which is related to what I’m calling the solitary creative state) in favor of the “hammer and chisel” of the finished theo
A few days ago, I was thinking about matrices and determinants. I noticed that I know the formula for the determinant, but I still lack the feeling of what the determinant is. I played with that thought for some time, and then it occurred to me, that if you imagine the rows in the matrix as vectors in n-dimensional space, then the determinant of that matrix is the volume of the n-dimensional body whose edges are those vectors.
And suddenly it all made a fucking sense. The determinant is zero when the vectors are linearly dependent? Of course, that means that the n-dimensional body has been flattened into n-1 dimensions (or less), and therefore its volume is zero. The determinant doesn't change if you add a multiple of a row to some other row? Of course, that means moving the "top" of the n-dimensional body in a direction parallel to the "bottom", so that neither the bottom nor the height changes; of course the volume (defined as the area of the bottom multiplied by the height) stays the same. What about the determinant being negative? Oh, that just means whether the edges are "clockwise" or "counter-clockwise" in the n-dimensional space. It all makes perfect sense!
Then I checked Wikipedia... and yeah, it was already there. So much for my Nobel prize.
But it still felt fucking good. (And if I am not too lazy, one day I may write a blog article about it.)
Reinventing the wheel is not a waste of time. I will probably remember this forever, and the words "determinant of the matrix" will never feel the same. Who knows, maybe this will help me figure out something else later. And if I keep doing that, hypothetically speaking, some of those discoveries might even be original.
(The practical problem is that none of this can pay my bills.)
3Mo Putera
I kind of envy that you figured this out yourself — I learned the parallelipiped hypervolume interpretation of the determinant from browsing forums (probably this MSE question's responses). Also, please do write that blog article.
Yeah, I hope you will! I'm reminded of what Scott Aaronson said recently:
The emergence of agentic Artificial Intelligence (AI) is set to trigger a “Cambrian explosion” of new kinds of personhood. This paper proposes a pragmatic framework for navigating this diversification by treating personhood not as a metaphysical property to be discovered, but as a flexible bundle of obligations (rights and responsibilities) that socie
I don't have time to read the paper or even skim it really, just page through it. But I will, perhaps unwisely, voice my intuitive assessment, and then maybe people who actually read it, can correct me.
I find their concept to be sinister and dangerous. What are the actual consequences of "unbundling the personhood bundle"? It means, on the one hand, that you get to create entities that resemble people but which you don't need to treat as people (good if you want intelligent slaves); on the other hand, you also get to create entities that aren't really people at all, but which laws, customs and institutions will treat as people (good if you want to hasten the real "great replacement").
A major reason why I respond negatively, is the line in the abstract about how this pragmatic attitude allows one to "creat[e] bespoke solutions for different contexts". That's corporate-speak, and I do not trust people who work for a mega-corporation and say they want to create customized concepts of personhood, whether they are lawyers or computer scientists.
Another reason is their pragmatist, relativist attitude to personhood. One of my persistent worries is that superintelligence will have the right values but the wrong ontology of personhood, and here these authors shrug their shoulders and say, meh, there aren't real facts about that to discover anyway, just ever-shifting social conventions. If I had the time to do my due diligence on this paper, I would want to investigate the authors (I don't know any of them) and find out where they are coming from, philosophically and professionally, so I could really identify the spirit in which the paper is written.
That's what I derive from a superficial glance at the paper. I wish I had time to analyze and reflect on it properly, so that I could get the nuances right, and also have a more measured and less emotional response. But time is short, yet the issues are important, so, that's my hasty response.
2Mo Putera
(I actually appreciate the emotion in the response, so thanks for including it)
I would've expected the opposite phrasing (right ontology wrong values, cf. "the AI knows but doesn't care") so this caught my eye. Have you or anyone else written anything about this elsewhere you can point me to? I initially thought of Jan Kulveit's essays (e.g. this or this) but upon re-skimming they don't really connect to what you said.
4Mitchell_Porter
"Tiling the solar system with smiley faces" used to be a canonical example of misalignment, and it could emerge from a combination of right values and very crudely wrong ontology, e.g. if the ontology can't distinguish between actual happiness and pictures of happiness.
A more subtle example might be, what if humans are conscious and uploads aren't. If an upload is as empty of genuine intentionality as a smiley face, you might have a causal model of conscious mind which is structurally correct in every particular, but which also needs to be implemented in the right kind of substrate to actually be conscious. If your ontology was missing that last detail, your aligned superintelligence might be profoundly correct in its theory of values, but could still lead to de-facto human extinction by being the Pied Piper of a mass migration of humanity into virtual spaces where all those hedons are only being simulated rather than being instantiated.
2Mo Putera
Interesting example. Tangentially I'm guessing believing in substrate dependence is part of some folks' visceral dislike of Richard Ngo's story The Gentle Romance, which was meant to be utopian. I mostly lean against substrate dependence and so don't find your example persuasive, although Scott Aaronson's monstrous edge cases do give me pause:
Obviously you're not obliged to, but if you ever get round to looking into the GDM paper more deeply like you mentioned I'd be interested in what you have to say, as you might change my opinion on it.
Just stumbled upon a spreadsheet ranking "well-known" googological numbers created by redditor u/Laxxius1 as a passion project. I've been idly interested in googology since childhood, so perusing this list felt like getting the Christmas present I've always wanted but kept forgetting to ask for. (It's also just intrinsically fun if your mathematical aesthetic is wrestler, which I'm guessing is overrepresented among LWers.)
Yudkowsky's number, which Eliezer created back in 2007 to win this contest on the xkcd forum, features at #12. (Eliezer also wrote Why i... (read more)
I wonder to what extent these impressions by Some Guy over at Extelligence are shared by others:
Some of this may be biased by my own background, but basically I’ve always found mathematicians and physicists to be the most humble and approachable people in all of the sciences. Second, are chemists and material scientists who can be somewhat cold but are always honest and straightforward because they have nothing to prove. But biologists? Man. The descriptive sciences have a chip on their shoulders and while social sciences are usually full of people w
Kishore Mahbubani, Singaporean diplomat and former president of the UN Security Council, studied philosophy full-time as an undergraduate in the late 60s. Recounting that period in his autobiography Living the Asian Century he wrote
For the final examinations, which I took at the end of my fourth year, our degree was determined by how well we did in eight three-hour examinations. In one of the papers, we had to answer a single question. The one question I chose to answer over three hours was “Can a stone feel pain?”
Why doesn't Applied Divinity Studies' The Repugnant Conclusion Isn't dissolve the argumentative force of the repugnant conclusion?
But read again more carefully: “There is nothing bad in each of these lives”.
Although it sounds mundane, I contend that this is nearly incomprehensible. Can you actually imagine what it would be like to never have anything bad happen to you? We don’t describe such a as mediocre, we describe it as “charmed” or “overwhelmingly privileged”. ...
... consider Parfit’s vision of World Z both seriously and literally.
First, this is not the phrase I associate with the repugnant conclusion. "Net positive" does not mean "there is nothing bad in each of these lives".
Second, I do think a key phrase & motivating description is "all they have is muzak and potatoes". That is all they have. I like our world where people can be and do great things. I won't describe it in poetic terms, since I don't think that makes good moral philosophy. If you do want something more poetic, idk read Terra Ignota or The Odyssey. Probably Terra Ignota moreso than The Odyssey.
I will say that I like doing fun things, and I think many other people like doing fun things, and though my life may be net positive sitting around in a buddhist temple all day, I would likely take a 1-in-a-million chance of death to do awesome stuff instead. And so, I think, would many others.
And we could all make a deal, we draw straws, and those 1-in-a-million who draw short give the rest their resources and are put on ice until we figure out a way to get enough resources so they could do what they love. Or, if that's infeasible (and in most framings of the problem it seems to be), willfully die.
I mean, if nothing else, you can just gather all those who love extreme sports (which will be a non-trivial fraction of the population), and ask them to draw straws & re-consolidate the relevant resources to the winners. Their revealed preference would say "hell yes!" (we can tell, given the much lower stakes & much higher risk of the activities they're already doing).
And I don't think the extreme sports lovers would be the only group who would take such a deal. Anyone who loves doing anything will take that deal, and (especially in a universe with the resources able to be filled to the brim with people just above the "I'll kill myself" line) I think most will have such a passion able to be fulfilled (even if it is brute wireheading!).
And then, if we know this will happen ahead of time--that people will risk death to celebrate th
2Garrett Baker
A big factor against making 1-in-a-million higher for most people is the whole death aspect, but death itself is a big negative, much worse to die than to never have been born (or so I claim), so the above gives a lower bound on the factor by which the repugnant conclusion will be off by.
4cubefox
The Parfit quote from the blog post is taken out of context. Here is the relevant section in Parfit's essay:
(Each box represents a possible population, with the height of a box representing how good overall an individual life is in that population, and the width representing the size of the population. The area of a box is the sum total "goodness"/"welfare"/"utility" (e.g. well-being, satisfied preferences, etc) in that population. The areas increase from A to Z, with Z being truncated here.)
Note that Parfit describes two different ways in which an individual life in Z could be barely worth living (emphasis added):
Then he goes on to describe the second possibility (which is arguably unrealistic and much less likely than the first, and which contains the quote by the blog author). The author of the blog posts mistakenly ignores Parfit's mentioning the first possibility. After talking about the second, Parfit returns (indicated by "similarly") to the first possibility:
The "greatest quantity" here can simply be determined by the weight of all the positive things in an individual life minus the weight of all the negative things. Even if the result is just barely positive for an individual, for a large enough population, the sum welfare of the "barely net positive" individual lives would outweigh the sum for a smaller population with much higher average welfare. Yet intuitively, we should not trade a perfect utopia with relatively small population (A) for a world that is barely worth living for everyone in a huge population (Z).
That's the problem with total utilitarianism, which simply sums all the "utilities" of the individual lives to measure the overall "utility" of a population. Taking the average instead of the sum avoids the repugnant conclusion, but it leads to other highly counterintuitive conclusions, such as that a population of a million people suffering strongly is less bad than a population of just a single person suffering slightly more strongly
2Richard_Kennaway
The comment you made a little later looks like your answer to that question.
What is the current best understanding of why o3 and o4-mini hallucinate more than o1? I just got round to checking out the OpenAI o3 and o4-mini System Card and in section 3.3 (on hallucinations) OA noted that
o3 tends to make more claims overall, leading to more accurate claims as well as more inaccurate/hallucinated claims. While this effect appears minor in the SimpleQA results (0.51 for o3 vs 0.44 for o1), it is more pronounced in the PersonQA evaluation (0.33 vs 0.16). More research is needed to understand the cause of these results.
o3 has some sort of internal feature like "Goodhart to the objective"/"play in easy mode".
o3's RL post-training environments have opportunities for reward hacks.
o3 discovers and exploits those opportunities.
RL rewards it for that, reinforcing the "Goodharting" feature.
This leads to specification-hack-y behavior generalizing out of distribution, to e. g. freeform conversations. It ends up e. g. really wanting to sell its interlocutor on what it's peddling, so it deliberately[1] confabulates plausible authoritative-sounding claims and justifications for them.
Sounds not implausible, though I'm not wholly convinced.
Thank you, sounds somewhat plausible to me too. For others' benefit, here's the chart from davidad's linked tweet:
3Weaverzhu
I've found the original paper of this chart https://arxiv.org/pdf/2503.11926v1
> We use prompted GPT-4o models to monitor a frontier reasoning agent, an agent in the same family as OpenAI o1 and o3-mini. During training, the agent discovered two hacks affecting nearly all training environments:
The model is in the same family as o1 and o3-mini. Maybe o3 but not comfirmed.
Venkatesh Rao's recent newsletter article Terms of Centaur Service caught my eye for his professed joy of AI-assisted writing, both nonfiction and fiction:
In the last couple of weeks, I’ve gotten into a groove with AI-assisted writing, as you may have noticed, and I am really enjoying it. ... The AI element in my writing has gotten serious, and I think is here to stay. ...
On the writing side, when I have a productive prompting session, not only does the output feel information dense for the audience, it feels information dense for me.
Feynman loved doing physics. I think what he loved most was the process of it. Of calculating. Of figuring things out. It didn’t seem to matter to him so much if what came out was big and important. Or esoteric and weird. What mattered to him was the process of finding it. And he was often quite competitive about it.
Some scientists (myself probably included) are driven by the ambition to build grand intellectual edifices. I think Feynman — at least in the years I knew him — was m
Hal Finney's reflections on the comprehensibility of posthumans, from the Vinge singularity discussion which took place on the Extropians email list back in the day:
1Mo Putera
Linking to a previous comment: 3,000+ words of longform quotes by various folks on the nature of personal identity in a posthuman future, and hiveminds / clans, using Hannu Rajaniemi's Quantum Thief trilogy as a jumping-off point.
There are many issues buried in this question, which I have tried to phrase in a way that does not presuppose the nature of the answer.
It would not be good to start, for example, with the question
How do mathematicians prove theorems?
This question introduces an interesting topic, but to start with it would be to project two hidden assumptions: (1) that there is uniform, objective and firmly establ
If Thurston is right here and mathematicians want to understand why some theorem is true (rather than to just know the truth values of various conjectures), and if we "feel the AGI" ... then it seems future "mathematics" will consist in "mathematicians" asking future ChatGPT to explain math to them. Whether something is true, and why. There would be no research anymore.
The interesting question is, I think, whether less-than-fully-general systems, like reasoning LLMs, could outperform humans in mathematical research. Or whether this would require a full AGI that is also smarter than mathematicians. Because if we had the latter, it would likely be an ASI that is better than humans in almost everything, not just mathematics.
The OECD working paper Miracle or Myth? Assessing the macroeconomic productivity gains from Artificial Intelligence, published quite recently (Nov 2024), is strange to skim-read: its authors estimate just 0.24-0.62 percentage points annual aggregate TFP growth (0.36-0.93 pp. for labour productivity) over a 10-year horizon, depending on scenario, using a "novel micro-to-macro framework" that combines "existing estimates of micro-level performance gains with evidence on the exposure of activities to AI and likely future adoption rates, relying on a multi-sec... (read more)
I like this passage by jdp as a concise examples-heavy articulation of a vague idea I've had for a while, and wanted to pick it out of his essay Predictable Updates About Identity to be able to point to it going forward:
2. Uploading Is A Continuum And Already Here
Depending on how seriously we want to take the above it could be argued that low fidelity uploading technology has been with us for a long time in the form of literacy and deep learning is simply taking the writing technology tree to its logical conclusion. At first we wrote down small messages an
Every once in a while I think about Robert Freitas' 1984 essay Xenopsychology, in particular his Sentience Quotient (SQ) idea:
It is possible to devise a sliding scale of cosmic sentience universally applicable to any intelligent entity in the cosmos, based on a "figure of merit" which I call the Sentience Quotient. The essential characteristic of all intelligent systems is that they process information using a processor or "brain" made of matter-energy. Generally the more information a brain can process in a shorter length of time, the more intellige
Part of the concept of ‘Disruption’ is that important new technologies tend to be bad at the things that matter to the previous generation of technology, but they do something else important instead. Asking if an LLM can do very specific and precise information retrieval might be like asking if an Apple II can match the uptime of a mainframe, or asking if you can build Photoshop inside Netscape. No, they can’t really do that, but that’s not the point and doesn’t mean
(Galison’s article is worth reading in full, it’s wonderful erisology — a synthesis of two models of scientific progress: incremental empiricism (of the logical positivists) and grand paradigm shifts (of Thomas Kuhn and others).)
Experimentalists, theorists and instrument makers are all
Scott's The Colors Of Her Coat is the best writing I've read by him in a long while. Quoting this part in particular as a self-reminder and bulwark against the faux-sophisticated world-weariness I sometimes slip into:
Chesterton’s answer to the semantic apocalypse is to will yourself out of it. If you can’t enjoy My Neighbor Totoro after seeing too many Ghiblified photos, that’s a skill issue. Keep watching sunsets until each one becomes as beautiful as the first...
If you insist that anything too common, anything come by too cheaply, must be bor
I find both the views below compellingly argued in the abstract, despite being diametrically opposed, and I wonder which one will turn out to be the case and how I could tell, or alternatively if I were betting on one view over another, how should I crystallise the bet(s).
One is exemplified by what Jason Crawford wrote here:
The acceleration of material progress has always concerned critics who fear that we will fail to keep up with the pace of change. Alvin Toffler, in a 1965 essay that coined the term “future shock,” wrote:
Some ongoing efforts to mechanize mathematical taste, described by Adam Marblestone in Automating Math:
Yoshua Bengio, one of the “fathers” of deep learning, thinks we might be able to use information theory to capture something about what makes a mathematical conjecture “interesting.” Part of the idea is that such conjectures compress large amounts of information about the body of mathematical knowledge into a small number of short, compact statements. If AI could optimize for some notion of “explanatory power” (roughly, how vast a range of disparate knowl
How to quantify how much impact being smarter makes? This is too big a question and there are many more interesting ways to answer it than the following, but computer chess is interesting in this context because it lets you quantify compute vs win probability, which seems like one way to narrowly proxy the original question. Laskos did an interesting test in 2013 with Houdini 3 by playing a large number of games on 2x nodes vs 1x nodes per move level and computing p(win | "100% smarter"). The win probability gain above chance i.e. 50% drops from +35.1% in ... (read more)
The diminishing returns isn't too surprising, because you are holding the model size fixed (whatever that is for Houdini 3), and the search sigmoids hard. Hence, diminishing returns as you jump well past the initial few searches with the largest gains, to large search budgets like 2k vs 4k (and higher).
This is not necessarily related to 'approaching perfection', because you can see the sigmoid of the search budget even with weak models very far from the known oracle performance (as well as stronger models); for example, NNs playing Hex: https://arxiv.org/pdf/2104.03113#page=5 Since it's a sigmoid, at a certain point, your returns will steeply diminish and indeed start to look like a flat line and a mere 2x increase in search budget does little. This is why you cannot simply replace larger models with small models that you search the hell out of: because you hit that sigmoid where improvement basically stops happening.
At that point, you need a smarter model, which can make intrinsically better choices about where to explore, and isn't trapped dumping endless searches into its own blind spots & errors. (At least, that's how I think of it qualitatively: the sigmoiding happens because of 'unknown unknowns', where the model can't see a key error it made somewhere along the way, and so almost all searches increasingly explore dead branches that a better model would've discarded immediately in favor of the true branch. Maybe you can think of very large search budgets applied to a weak model as the weak model 'approaching perfection... of its errors'? In the spirit of the old Dijkstra quip, 'a mistake carried through to perfection'. Remember, no matter how deeply you search, your opponent still gets to choose his move, and you don't; and what you predict may not be what he will select.)
Fortunately, 'when making an axe handle with an axe, the model is indeed near at hand', and a weak model which has been 'policy-improved' by search is, for that one datapoint, equivalen
1Mo Putera
Thanks, I especially appreciate that NNs playing Hex paper; Figure 8 in particular amazes me in illustrating how much more quickly perf. vs test-time compute sigmoids than I anticipated even after reading your comment. I'm guessing https://www.gwern.net/ has papers with the analogue of Fig 8 for smarter models, in which case it's time to go rummaging around...
Lee Billings' book Five Billion Years of Solitude has the following poetic passage on deep time that's stuck with me ever since I read it in Paul Gilster's post:
Deep time is something that even geologists and their generalist peers, the earth and planetary scientists, can never fully grow accustomed to.
The sight of a fossilized form, perhaps the outline of a trilobite, a leaf, or a saurian footfall can still send a shiver through their bones, or excavate a trembling hollow in the chest that breath cannot fill. They can measure celestial motions and l
2
1
1
1
1
1
1
2
2
1
1
1
1
2
1
2
2
1
1