LESSWRONG
LW

Mo Putera's Shortform — LessWrong

Mo Putera's Shortform

16th Jan 2025

1 min read

5

This is a special post for quick takes by Mo Putera. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.

Mo Putera's Shortform

9Alexander Gietelink Oldenziel

154 comments, sorted by

top scoring

Click to highlight new comments since: Today at 7:54 PM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]Mo Putera5mo530

These quotes from When ChatGPT Broke an Entire Field: An Oral History stood out to me:

On November 30, 2022, OpenAI launched its experimental chatbot. ChatGPT hit the NLP community like an asteroid.
IZ BELTAGY (lead research scientist, Allen Institute for AI; chief scientist and co-founder, SpiffyAI): In a day, a lot of the problems that a large percentage of researchers were working on — they just disappeared. ...
R. THOMAS MCCOY: It’s reasonably common for a specific research project to get scooped or be eliminated by someone else’s similar thing. But ChatGPT did that to entire types of research, not just specific projects. A lot of higher categories of NLP just became no longer interesting — or no longer practical — for academics to do. ...
IZ BELTAGY: I sensed that dread and confusion during EMNLP [Empirical Methods in Natural Language Processing], which is one of the leading conferences. It happened in December, a week after the release of ChatGPT. Everybody was still shocked: “Is this going to be the last NLP conference?” This is actually a literal phrase that someone said. During lunches and cocktails and conversations in the halls, everybody was asking the same q

... (read more)

[-]Daniel Kokotajlo5mo3933

Wow. I knew academics were behind / out of the loop / etc. but this surprised me. I imagine these researchers had at least heard about GPT2 and GPT3 and the scaling laws papers; I wonder what they thought of them at the time. I wonder what they think now about what they thought at the time.

[-]lc5mo140

The full article sort of explains the bizarre kafkaesque academic dance that went on from 2020-2022, and how the field talked about these changes.

5Yuxi_Liu5mo

for anyone not wanting to go in and see the Kafka, I copied some useful examples: ANNA ROGERS: I was considering making yet another benchmark, but I stopped seeing the point of it. Let’s say GPT-3 either can or cannot continue [generating] these streams of characters. This tells me something about GPT-3, but that’s not actually even a machine learning research question. It’s product testing for free. JULIAN MICHAEL: There was this term, “API science,’’ that people would use to be like: “We’re doing science on a product? This isn’t science, it’s not reproducible.” And other people were like: “Look, we need to be on the frontier. This is what’s there.” TAL LINZEN (associate professor of linguistics and data science, New York University; research scientist, Google): For a while people in academia weren’t really sure what to do. R. THOMAS MCCOY: Are you pro- or anti-LLM? That was in the water very, very much at this time. JULIE KALLINI (second-year computer science Ph.D. student, Stanford University): As a young researcher, I definitely sensed that there were sides. At the time, I was an undergraduate at Princeton University. I remember distinctly that different people I looked up to — my Princeton research adviser [Christiane Fellbaum] versus professors at other universities — were on different sides. I didn’t know what side to be on. LIAM DUGAN: You got to see the breakdown of the whole field — the sides coalescing. The linguistic side was not very trusting of raw LLM technology. There’s a side that’s sort of in the middle. And then there’s a completely crazy side that really believed that scaling was going to get us to general intelligence. At the time, I just brushed them off. And then ChatGPT comes out.

8Neel Nanda5mo

+1, GPT3.5 was publicly available since January, and GPT3 was big news two years before and publicly available back then. I'm really surprised that people didn't understand that these models were a big deal AND changed their minds when ChatGPT came out. Maybe it's just a weird preference cascade, where this was enough to break a common false belief?

4Shankar Sivarajan5mo

Something like GPT-3.5/ChatGPT was qualitatively different.

8Caleb Biddulph5mo

I remember seeing the ChatGPT announcement and not being particularly impressed or excited, like "okay, it's a refined version of InstructGPT from almost a year ago. It's cool that there's a web UI now, maybe I'll try it out soon." November 2022 was a technological advancement but not a huge shift compared to January 2022 IMO

2Shankar Sivarajan5mo

Fair enough. My mental image of the GPT models was stuck on that infernal "talking unicorns" prompt, which I think did make them seem reasonably characterized as mere "stochastic parrots" and "glorified autocompletes," and the obvious bullshit about the "safety and security concerns" around releasing GPT-2 also led me to conclude the tech was unlikely to amount to much more. InstructGPT wasn't good enough to get me to update it; that took the much-hyped ChatGPT release. Was there a particular moment that impressed you, or did you just see the Transformers paper, project that correctly into the future, and the releases that followed since then have just been following that trend you extrapolated and so been unremarkable?

2Caleb Biddulph5mo

I remember being very impressed by GPT-2. I think I was also quite impressed by GPT-3 even though it was basically just "GPT-2 but better." To be fair, at the moment that I was feeling unimpressed by ChatGPT, I don't think I had actually used it yet. It did turn out to be much more useful to me than the GPT-3 API, which I tried out but didn't find that many uses for. It's hard to remember exactly how impressed I was with ChatGPT after using it for a while. I think I hadn't fully realized how great it could be when the friction of using the API was removed, even if I didn't update that much on the technical advancement.

3cubefox5mo

The full article discusses the transformer paper (which didn't have a large influence, as the implications weren't clear), BERT (which did have a large influence) and GPT-3 (which also had a large influence). I assume the release of ChatGPT was the point where even the last NLP researchers couldn't ignore LLMs anymore.

[-]gwern5mo115

ChatGPT was "so good they can't ignore you"; the Hugging Face anecdote is particularly telling. At some point, everyone else gets tired of waiting for your cargo to land, and will fire you if you don't get with the program. "You say semantics can never be learned from syntax and you've proven that ChatGPT can never be useful? It seems plenty useful to me and everyone else. Figure it out or we'll find someone who can."

2cubefox5mo

I think the most interesting part of the Quanta piece is the discussion of the octopus paper, which states that pure language models can't actually understand text (as they only learn from form/syntax), and the bitter disputes that followed in the NLP community. From the abstract: Emily M. Bender, the first author, was also first author of the subsequent "stochastic parrot" paper: On the Dangers of Stochastic Parrots: Can Language Models Be Too Big? 🦜[1] (As a side note, Yudkowsky's broadly verificationist theory of content seems to agree with her distinction: if "understanding" of a statement is knowing what experience would confirm it, or what experience it would predict, then understanding cannot come from syntactic form alone. The association of words and sensory data would be necessary. Did Yudkowsky ever comment on the apparent incompatibility between evident LLM understanding and his anticipated experience theory?) Of course I assume that now it can hardly be denied that LLMs really do somehow understand text, even if they are merely trained on form. So the octopus paper argument must be wrong somewhere. Though at least in the Quanta piece, Bender doesn't acknowledge any update of that sort. In fact, in the last quote she says: ---------------------------------------- 1. First paper I have seen that uses an emoji in its title. ↩︎

[-]gwern5mo2419

I don't think there's any necessary contradiction. Verification or prediction of what? More data. What data? Data. You seem to think there's some sort of special reality-fluid which JPEGs or MP3s have but .txt files do not, but they don't; they all share the Buddha-nature.

Consider Bender's octopus example, where she says that it can't learn to do anything from watching messages go back and forth. This is obviously false, because we do this all the time; for example, you can teach a LLM to play good chess simply by watching a lot of moves fly by back and forth as people play postal chess. Imitation learning & offline RL are important use-cases of RL and no one would claim it doesn't work or is impossible in principle.

Can you make predictions and statements which can be verified by watching postal chess games? Of course. Just predict what the next move will be. "I think he will castle, instead of moving the knight." [later] "Oh no, I was wrong! I anticipated seeing a castling move, and I did not, I saw something else. My beliefs about castling did not pay rent and were not verified by subsequent observations of this game. I will update my priors and do better next time."

4cubefox5mo

Well, in the chess example we do not have any obvious map/territory relation. Chess seems to be a purely formal game, as the pieces do not seem to refer to anything in the external world. So it's much less obvious that training on form alone would also work for learning natural language, which does exhibit a map territory distinction. For example, a few years ago, most people would have regarded it as highly unlikely that you could understand (decode) an intercepted alien message without any contextual information. But if you can understand text from form alone, as LLMs seem to prove, the message simply has to be long enough. Then you can train an LLM on it, which would then be able to understand the message. And it would also be able to translate it into English if it is additionally trained on English text. That's very counterintuive, or at least it was counterintuitive until recently. I doubt EY meant to count raw words as "anticipated experience", since "experience" typically refers to sensory data only. (In fact, I think Guessing the Teacher's Password also suggests that he didn't.) To repeat, I don't blame him, as the proposition that large amounts of raw text can replace sensory data, that a sufficient amount of symbols can ground themselves, was broadly considered unlikely until LLMs came along. But I do blame Bender insofar as she didn't update even in light of strong evidence that the classical hypothesis (you can't infer meaning from form alone) was wrong.

[-]gwern5mo*1912

Well, in the chess example we do not have any obvious map/territory relation.

Yes, there is. The transcripts are of 10 million games that real humans played to cover the distribution of real games, and then were annotated by Stockfish, to provide superhuman-quality metadata on good vs bad moves. That is the territory. The map is the set of transcripts.

But if you can understand text from form alone, as LLMs seem to prove, the message simply has to be long enough.

I would say 'diverse enough', not 'long enough'. (An encyclopedia will teach a LLM many things; a dictionary the same length, probably not.) Similar to meta-learning vs learning.

the pieces do not seem to refer to anything in the external world.

What external world does our 'external world' itself refer to things inside of? If the 'external world' doesn't need its own external world for grounding, then why does lots of text about the external world not suffice? (And if it does, what grounds that external external world, or where does the regress end?) As I like to put it, for an LLM, 'reality' is just the largest fictional setting - the one that encompasses all the other fictional settings it reads about from time to time.

As someone who doubtless does quite a lot of reading about things or writing to people you have never seen nor met in real life and have no 'sensory' way of knowing that they exist, this is a position you should find sympathetic.

5cubefox5mo

Sympathy or not, the position that meaning of natural language can be inferred from the symbolic form alone wasn't obvious to me in the past, as this is certainly not how humans learn language, and I don't know any evidence that someone else thought this plausible before machine learning made it evident. It's always easy to make something sound obvious after the fact, but that doesn't mean that it actually was obvious to anyone at the time.

3Yuxi_Liu5mo

Plenty of linguists and connectionists thought it was possible, if only to show those damned Chomskyans that they were wrong! To be specific, some of the radical linguists believed in pure distributional semantics, or that there is no semantics beyond syntax. I don't know anyone in particular, but considering how often Chomsky, Pinker, etc were fighting against the "blank slate" theory, they definitely existed. The following people likely believed that it is possible to learn a language purely from reading using a general learning architecture like neural networks (blank-slate): * James L. McClelland and David Rumelhart. * They were the main proponents of neural networks in the "past tense debate". Generally, anyone on the side of neural networks in the past tense debate probably believed this. * B. F. Skinner. * Radical syntacticians? Linguists have failed to settle the question of "Just what is semantics? How is it different from syntax?", and some linguists have taken the radical position "There is no semantics. Everything is syntax.". Once that is done, there simply is no difficulty: just learn all the syntax, and there is nothing left to learn. * Possibly some of the participants in the "linguistics wars" believed in it. Specifically, some believed in "generative semantics", whereby semantics is simply yet more generative grammar, and thus not any different from syntax (also generative grammar). Chomsky, as you might imagine, hated that, and successfully beat it down. * Maybe some people in distributional semantics? Perhaps Leonard Bloomfield? I don't know enough about the history of linguistics to tell what Bloomfield or the "Bloomfieldians" believed in exactly. However, considering that Chomsky was strongly anti-Bloomsfield, it is a fair bet that some Bloomsfieldians (or self-styled "neo-Bloomsfieldians") would support blank-slate learning of language, if only to show Chomskyans that they're wrong.

4eggsyntax5mo

FYI your 'octopus paper' link is to Stochastic Parrots; it should be this link. I've seen other quotes from Bender & relevant coauthors that suggest they haven't really updated, which I find fascinating. I'd love to have the opportunity to talk with them about it and understand better how their views have remained consistent despite the evidence that's emerged since the papers were published. It makes a very intuitively compelling argument! I think that, as with many confusions about the Chinese Room, the problem is that our intuitions fail at the relevant scale. Given an Internet's worth of discussion of bears and sticks and weapons, the hyper-intelligent octopus's model of those things is rich enough for the octopus to provide advice about them that would work in the real world, even if it perhaps couldn't recognize a bear by sight. For example it would know that sticks have a certain distribution of mass, and are the sorts of things that could be bound together by rope (which it knows is available because of the coconut catapult), and that the combined sticks might have enough mass to serve as a weapon, and what amounts of force would be harmful to a bear, etc. But it's very hard to understand just how rich those models can be when our intuitions are primed by a description of two people casually exchanging messages.

4Shankar Sivarajan5mo

Perhaps relevant, she famously doesn't like the arXiv, so maybe on principle she's disregarding all evidence not from "real publications."

[-]Mo Putera2mo406

Dean Ball's Out of Thin Air: A proposal for the grid is the most "big if true" thing I've read on AI infrastructure recently:

America can pull gigawatts out of thin air through a combination of technology and smartly targeted policy. Let me show you how. ...
It is often said that the US electricity grid is under increasing risk of blackouts, price spikes, and other signs of strain. ... Most of the time, however, the grid has significantly more power than is needed. This means that the grid can often accommodate, say, a new 750-megawatt data center for the vast majority of the year. There is spare generation capacity available except for those brief periods of high demand. But in the high-demand periods, when all the grid’s generation capacity is required to maintain electricity service, this new data center would require an additional 750 megawatts of electricity generation capacity, and quite possibly also transmission infrastructure upgrades.
This additional investment is only necessary if you assume that the new data center will require all 750 megawatts of electricity during peak-demand periods. Traditionally, this assumption has been true: data center operators rely o

... (read more)

[-]title222mo135

I came across a similar "hack" on LinkedIn from Tom Styer:

"California just pulled off the nation’s largest-ever test of a virtual power plant.
This successful test proved VPPs are a fast, low-cost, zero-emissions way to make better use of the clean energy we already have — and to keep the lights on as demand surges from data centers, heat waves, and electrification."

Basically, they are talking about allowing residential batteries supply the grid during peak demand. I tend to be skeptical about stuff like this because in my own domain, water, there's a lot of pop science and bold claims that ignore scaling and logistics. I asked a smart fellow in that industry about it
https://substack.com/@energycrystals
and I thought he gave a good answer that aligns with my experience with water, which is it always come down to implementation:
"The challenge is lining up customer telemetry to incentive structures that matter. With standard demand response products (which some people sell as VPPs), the incentives given to customers don’t pay for the battery and the software admin and API costs to the utilities outweigh the cost savings of a VPPVPPs are vaporware until someone can make the ... (read more)

4Hoagy2mo

Whether this is feasible depends on how concentrated that 0.25% of the year is (expected to be), because that determines the size of the battery that you'd need to cover the blackout period (which I think would be unacceptable for a lot of AI customers). If it happens in a single few days then this makes sense, buying 22GWh of batteries for a 1GW dataset is still extremely expensive (2B$ for a 20h system at 100$ / kWh plus installation, maybe too expensive for reliability for a 1GW datacenter I would expect, assuming maybe 10B revenue from the datacenter??). If it's much less concentrated in time then a smaller battery is needed (100M$ for a 1h system at 100$/kWh), and I expect AI scalers would happily pay this for the reliability of their systems if the revenue from those datacenters

3Josh You2mo

Demand response could be done by covering the data center with battery energy or not. Demand response and batteries can stack: if the grid is really stressed, a data center can both turn off and discharge its battery into the grid. Economically, it makes sense to accept some true downtime to avoid months-long delays in data center construction. This is clearly true for training workloads which are very important but don't have live demand. But downtime for even inference clusters is acceptable: you can reduce the compute demand by temporarily slowing down token generation, or use dynamic rate limits. And any curtailment would almost certainly be isolated to one region, so inference data centers in other places would still be operational.

1Josh You2mo

In any case, the paper says the curtailments would last about two hours each:

3Cedar2mo

I've heard a rule of thumb that if you can avoid buying power off the California power grid's spot pricing in the 1% most expensive times, you can roughly get a 50% discount on power.

[-]Mo Putera6mo400

Interesting anecdote on "von Neumann's onion" and his general style, from P. R. Halmos' The Legend of John von Neumann:

Style. As a writer of mathematics von Neumann was clear, but not clean; he was powerful but not elegant. He seemed to love fussy detail, needless repetition, and notation so explicit as to be confusing. To maintain a logically valid but perfectly transparent and unimportant distinction, in one paper he introduced an extension of the usual functional notation: along with the standard φ(x) he dealt also with something denoted by φ((x)). The hair that was split to get there had to be split again a little later, and there was φ(((x))), and, ultimately, φ((((x)))). Equations such as
(φ((((a))))^2 = φ(((a))))
have to be peeled before they can be digested; some irreverent students referred to this paper as von Neumann’s onion.
Perhaps one reason for von Neumann’s attention to detail was that he found it quicker to hack through the underbrush himself than to trace references and see what others had done. The result was that sometimes he appeared ignorant of the standard literature. If he needed facts, well-known facts, from Lebesgue integration theory, he waded in, defi

... (read more)

[-]Buck6mo140

I have this experience with @ryan_greenblatt -- he's got an incredible ability to keep really large and complicated argument trees in his head, so he feels much less need to come up with slightly-lossy abstractions and categorizations than e.g. I do. This is part of why his work often feels like huge, mostly unstructured lists. (The lists are more unstructured before his pre-release commenters beg him to structure them more.) (His code often also looks confusing to me, for similar reasons.)

[-]Mo Putera8mo353

While Dyson's birds and frogs archetypes of mathematicians is oft-mentioned, David Mumford's tribes of mathematicians is underappreciated, and I find myself pointing to it often in discussions that devolve into "my preferred kind of math research is better than yours"-type aesthetic arguments:

... the subjective nature and attendant excitement during mathematical activity, including a sense of its beauty, varies greatly from mathematician to mathematician... I think one can make a case for dividing mathematicians into several tribes depending on what most strongly drives them into their esoteric world. I like to call these tribes explorers, alchemists, wrestlers and detectives. Of course, many mathematicians move between tribes and some results are not cleanly part the property of one tribe.
Explorers are people who ask -- are there objects with such and such properties and if so, how many? They feel they are discovering what lies in some distant mathematical continent and, by dint of pure thought, shining a light and reporting back what lies out there. The most beautiful things for them are the wholly new objects that they discover (the phrase 'bright shiny objects' has been i

... (read more)

[-]Mo Putera7mo*301

Scott Alexander's Mistakes, Dan Luu's Major errors on this blog (and their corrections), Gwern's My Mistakes (last updated 11 years ago), and Nintil's Mistakes (h/t @Rasool) are the only online writers I know of who maintain a dedicated, centralized page solely for cataloging their errors, which I admire. Probably not coincidentally they're also among the thinkers I respect the most for repeatedly empirically grounding their reasoning. Some orgs do this too, like 80K's Our mistakes, CEA's Mistakes we've made, and GiveWell's Our mistakes.

While I prefer dedicated centralized pages like those to one-off writeups for long content benefit reasons, one-off definitely beats none (myself included). In that regard I appreciate essays like Holden Karnofsky's Some Key Ways in Which I've Changed My Mind Over the Last Several Years (2016), Denise Melchin's My mistakes on the path to impact (2020), Zach Groff's Things I've Changed My Mind on This Year (2017), Michael Dickens' things I've changed my mind on, and this 2013 LW repository for "major, life-altering mistakes that you or others have made", as well as by orgs like HLI's Learning from our mistakes.

In this vein I'm also sad to see m... (read more)

7tailcalled7mo

I'm not convinced Scott Alexander's mistakes page accurately tracks his mistakes. E.g. the mistake on it I know the most about is this one: But that's basically wrong. The study found women's arousal to chimps having sex to be very close to their arousal to nonsexual stimuli, and far below their arousal to sexual stimuli.

1Mo Putera7mo

Thanks, good example.

4MichaelDickens7mo

I don't have a mistakes page but last year I wrote a one-off post of things I've changed my mind on.

2Mo Putera7mo

Thanks Michael. On another note, I've recommended some of your essays to others, so thanks for writing them as well.

2MichaelDickens7mo

I'm glad to hear that! I often don't hear much response to my essays so it's good to know you've read some of them :)

2Mo Putera7mo

You're welcome :) in particular, your 2015 cause selection essay was I thought a particularly high-quality writeup of the end-to-end process from personal values to actual donation choice and (I appreciated this) where you were most likely to change your mind, so I recommended it to a few folks as well as used it as a template myself back in the day. In general I think theory-practice gap bridging via writeups like those are undersupplied, especially the end-to-end ones — more writeups bridge parts of the "pipeline", but "full pipeline integration" done well is rare and underappreciated, which combined with how effortful it is to do it makes me not surprised there isn't more of it.

4Rasool7mo

Another good blog: https://nintil.com/mistakes

1Mo Putera7mo

Thanks! Added to the list.

[-]Mo Putera4mo240

I really liked this extended passage on math circles from John Psmith's REVIEW: Math from Three to Seven, by Alexander Zvonkin, it made me wish math circles existed in my home country when I was younger:

in the interviews I’ve read with Soviet mathematicians and scientists, the things that come up over and over again are “mathematical circles,” a practice that originated in the pre-revolutionary Russian Empire and then spread far and wide through the Soviet Union. A mathematical circle is an informal group of teenagers and adults who really enjoy math and want to spend a lot of time thinking and talking about it. They’re a little bit like sports teams, in that they develop their own high-intensity internal culture and camaraderie, and often have a “coach” who is especially talented or famous. But they’re also very unlike sports teams, because they don’t compete with each other or play in leagues or anything like that, and usually any given circle will contain members of widely varying skill levels. Maybe a better analogy is a neighborhood musical ensemble that gets together and jams on a regular basis, but for math.
The most important thing to understand about mathematical circles is

... (read more)

8Thane Ruthenis4mo

Really appreciate the random excerpts you post regarding math philosophy/reasoning/etc. They're consistently interesting and insightful.

5Mateusz Bagiński4mo

I feel like we perhaps need to reach some "escape velocity" to get something like that going, but for ~rationality / deliberately figuring out how to think and act better.

[-]Mo Putera6mo212

Peter Watts is working with Neill Blomkamp to adapt his novel Blindsight into an 8-10-episode series:

“I can at least say the project exists, now: I’m about to start writing an episodic treatment for an 8-10-episode series adaptation of my novel Blindsight.
“Neill and I have had a long and tortured history with that property. When he first expressed interest, the rights were tied up with a third party. We almost made it work regardless; Neill was initially interested in doing a movie that wasn’t set in the Blindsight universe at all, but which merely used the speculative biology I’d invented to justify the existence of Blindsight’s vampires. “Sicario with Vampires” was Neill’s elevator pitch, and as chance would have it the guys who had the rights back then had forgotten to renew them. So we just hunkered quietly until those rights expired, and the recently-rights-holding parties said Oh my goodness we thought we’d renewed those already can we have them back? And I said, Sure; but you gotta carve out this little IP exclusion on the biology so Neill can do his vampire thing.
“It seemed like a good idea at the time. It was good idea, dammit. We got the carve-out and everything. Bu

... (read more)

[-]Seth Herd6mo*140

Blindsight was very well written but based on a premise that I think is importantly and dangerously wrong. That premise is that consciousness (in the sense of cognitive self-awareness) is not important for complex cognition.

This is the opposite of true, and a failure to recognize this is why people are predicting fantastic tool AI that doesn't become self-aware and goal-directed.

The proof won't fit in the margin unfortunately. To just gesture in that direction: it is possible to do complex general cognition without being able to think about one's self and one's cognition. It is much easier to do complex general cognition if the system is able to think about itself and its own thoughts.

5Viliam6mo

Just checking if I understood your argument: is the general point that an algorithm that can think about literally everything is simpler and therefore easier to make or evolve than an algorithm that can think about literally everything except for itself and how other agents perceive it?

4Seth Herd6mo

Exactly. I'd go a bit farther and say it's easier to develop an algorithm that can think about literally everything than one that can think about roughly half of things. That's because the easiest general intelligence algorithms are about learning and reasoning, which apply to everything.

4jbash6mo

I don't see where you get that. I saw no suggestion that the aliens (or vampires) in Blindsight were unaware of their own existence, or that they couldn't think about their own interactions with the world. They didn't lack any cognitive capacities at all. They just had no qualia, and therefore didn't see the point of doing anything just for the experience. There's a gigantic difference between cognitive self-awareness and conscious experience.

1kairos_6mo

I believe the Scramblers from blindsight weren’t self aware, which means they couldn’t think about their own interactions with the world. As I recall the crew was giving one of the Scramblers a series of cognitive tests. It aced all the tests that had to do with numbers and spatial reasoning, but failed a test that required the testee to be self aware.

2jbash6mo

I guess it depends on how it's described in context. And I have to admit it's been a long time. I'd go reread it to see, but I don't think I can handle any more bleakness right now...

2Mo Putera6mo

Thanks, is there anything you can point me to for further reading, whether by you or others?

1stavros6mo

I can see this making sense in one frame, but not in another. The frame which seems most strongly to support the 'Blindsight' idea is Friston's stuff - specifically how the more successful we are at minimizing predictive error, the less conscious we are.[1] My general intuition, in this frame, is that as intelligence increases more behaviour becomes automatic/subconscious. It seems compatible with your view that a superintelligent system would possess consciousness, but that most/all of its interactions with us would be subconscious. Would like to hear more about this point, could update my views significantly. Happy for you to just state 'this because that, read X, Y, Z etc' without further elaboration - I'm not asking you to defend your position, so much as I'm looking for more to read on it. 1. ^ This is my potentially garbled synthesis of his stuff, anyway.

4Mo Putera6mo

I'm not sure about Friston's stuff to be honest. But Watts lists a whole bunch of papers in support of the blindsight idea, contra Seth's claim — to quote Watts: * "In fact, the nonconscious mind usually works so well on its own that it actually employs a gatekeeper in the anterious cingulate cortex to do nothing but prevent the conscious self from interfering in daily operations" * footnotes: Matsumoto, K., and K. Tanaka. 2004. Conflict and Cognitive Control. Science 303: 969-970; 113 Kerns, J.G., et al. 2004. Anterior Cingulate Conflict Monitoring and Adjustments in Control. Science 303: 1023-1026; 114 Petersen, S.E. et al. 1998. The effects of practice on the functional anatomy of task performance. Proceedings of the National Academy of Sciences 95: 853-860 * "Compared to nonconscious processing, self-awareness is slow and expensive" * footnote: Matsumoto and Tanaka above * "The cost of high intelligence has even been demonstrated by experiments in which smart fruit flies lose out to dumb ones when competing for food" * footnote: Proceedings of the Royal Society of London B (DOI 10.1098/rspb.2003.2548) * "By way of comparison, consider the complex, lightning-fast calculations of savantes; those abilities are noncognitive, and there is evidence that they owe their superfunctionality not to any overarching integration of mental processes but due to relative neurological fragmentation" * footnotes: Treffert, D.A., and G.L. Wallace. 2004. Islands of genius. Scientific American 14: 14-23; Anonymous., 2004. Autism: making the connection. The Economist, 372(8387): 66 * "Even if sentient and nonsentient processes were equally efficient, the conscious awareness of visceral stimuli—by its very nature— distracts the individual from other threats and opportunities in its environment" * footnote: Wegner, D.M. 1994. Ironic processes of mental control. Psychol. Rev. 101: 34-52 * "Chimpanzees have a higher brain-to-body ratio than orangutans, yet orangs

[-]Mo Putera7mo210

What fraction of economically-valuable cognitive labor is already being automated today? How has that changed over time, especially recently?

I notice I'm confused about these ostensibly extremely basic questions, which arose in reading Open Phil's old CCF-takeoff report, whose main metric is "time from AI that could readily^[2] automate 20% of cognitive tasks to AI that could readily automate 100% of cognitive tasks". A cursory search of Epoch's data, Metaculus, and this forum didn't turn up anything, but I didn't spend much time at all doing so.

I was originally motivated by wanting to empirically understand recursive AI self-improvement better, which led to me stumbling upon the CAIS paper Examples of AI Improving AI, but I don't have any sense whatsoever of how the paper's 39 examples as of Oct-2023 translate to OP's main metric even after constraining "cognitive tasks" in its operational definition to just AI R&D.

I did find this 2018 survey of expert opinion

A survey was administered to attendees of three AI conferences during the summer of 2018 (ICML, IJCAI and the HLAI conference). The survey included questions for estimating AI capabilities over the next d

... (read more)

3faul_sname7mo

Did e.g. a telephone operator in 1910 perform cognitive labor, by the definition we want to use here?

1Mo Putera7mo

I'm mainly wondering how Open Phil, and really anyone who uses fraction of economically-valuable cognitive labor automated / automatable (e.g. the respondents to that 2018 survey; some folks on the forum) as a useful proxy for thinking about takeoff, tracks this proxy as a way to empirically ground their takeoff-related reasoning. If you're one of them, I'm curious if you'd answer your own question in the affirmative?

2faul_sname7mo

I am not one of them - I was wondering the same thing, and was hoping you had a good answer. If I was trying to answer this question, I would probably try to figure out what fraction of all economically-valuable labor each year was cognitive, the breakdown of which tasks comprise that labor, and the year-on-year productivity increases on those task, then use that to compute the percentage of economically-valuable labor that is being automated that year. Concretely, to get a number for the US in 1900 I might use a weighted average of productivity increases across cognitive tasks in 1900, in an approach similar to how CPI is computed * Look at the occupations listed in the 1900 census records * Figure out which ones are common, and then sample some common ones and make wild guesses about what those jobs looked like in 1900 * Classify those tasks as cognitive or non-cognitive * Come to estimate that record-keeping tasks are around a quarter to a half of all cognitive labor * Notice that typewriters were starting to become more popular - about 100,000 typewriters sold per year * Note that those 100k typewriters were going to the people who would save the most time by using them * As such, estimate 1-2% productivity growth in record-keeping tasks in 1900 * Multiply the productivity growth for record-keeping tasks by the fraction of time (technically actually 1-1/productivity increase but when productivity increase is small it's not a major factor) * Estimate that 0.5% of cognitive labor was automated by specifically typewriters in 1900 * Figure that's about half of all cognitive labor automation in 1900 and thus I would estimate ~1% of all cognitive labor was automated in 1900. By the same methodology I would probably estimate closer to 5% for 2024. Again, though, I am not associated with Open Phil and am not sure if they think about cognitive task automation in the same way.

[-]Mo Putera7mo210

I chose to study physics in undergrad because I wanted to "understand the universe" and naively thought string theory was the logically correct endpoint of this pursuit, and was only saved from that fate by not being smart enough to get into a good grad school. Since then I've come to conclude that string theory is probably a dead end, albeit an astonishingly alluring one for a particular type of person. In that regard I find anecdotes like the following by Ron Maimon on Physics SE interesting — the reason string theorists believe isn’t the same as what they tell people, so it’s better to ask for their conversion stories:

I think that it is better to ask for a compelling argument that the physics of gravity requires a string theory completion, rather than a mathematical proof, which would be full of implicit assumptions anyway. The arguments people give in the literature are not the same as the personal reasons that they believe the theory, they are usually just stories made up to sound persuasive to students or to the general public. They fall apart under scrutiny. The real reasons take the form of a conversion story, and are much more subjective, and much less persuasive to everyo

... (read more)

[-]Mo Putera7mo220

In pure math, mathematicians seek "morality", which sounds similar to Ron's string theory conversion stories above. Eugenia Cheng's Mathematics, morally argues:

I claim that although proof is what supposedly establishes the undeniable truth of a piece of mathematics, proof doesn’t actually convince mathematicians of that truth. And something else does.
... formal mathematical proofs may be wonderfully watertight, but they are impossible to understand. Which is why we don’t write whole formal mathematical proofs. ... Actually, when we write proofs what we have to do is convince the community that it could be turned into a formal proof. It is a highly sociological process, like appearing before a jury of twelve good men-and-true. The court, ultimately, cannot actually know if the accused actually ‘did it’ but that’s not the point; the point is to convince the jury. Like verdicts in court, our ‘sociological proofs’ can turn out to be wrong—errors are regularly found in published proofs that have been generally accepted as true. So much for mathematical proof being the source of our certainty. Mathematical proof in practice is certainly fallible.
But this isn’t the only

... (read more)

6Mitchell_Porter7mo

The more you know about particle physics and quantum field theory, the more inevitable string theory seems. There are just too many connections. However, identifying the specific form of string theory that corresponds to our universe is more of a challenge, and not just because of the fabled 10^500 vacua (though it could be one of those). We don't actually know either all the possible forms of string theory, or the right way to think about the physics that we can see. The LHC, with its "unnaturally" light Higgs boson, already mortally wounded a particular paradigm for particle physics (naturalness) which in turn was guiding string phenomenology (i.e. the part of string theory that tries to be empirically relevant). So along with the numerical problem of being able to calculate the properties of a given string vacuum, the conceptual side of string theory and string phenomenology is still wide open for discovery.

9Alexander Gietelink Oldenziel7mo

I asked a well-known string theorist about the fabled 10^500 vacua and asked him whether he worried that this would make string theory a vacuous theory since a theory that fits anything fits nothing. He replied ' no, no the 10^500 'swampland' is a great achievement of string theory - you see... all other theories have infinitely many adjustable parameters'. He was saying string theory was about ~1500 bits away from the theory of everything but infinitely ahead of its competitors. Diabolical. Much ink has been spilled on the scientific merits and demerits of string theory and its competitors. The educated reader will recognize that this all this and more is of course, once again, solved by UDASSA.

2Noosphere897mo

Re other theories, I don't think that all other theories in existence have infinitely many adjustable parameters, and if he's referring to the fact that lots of theories have adjustable parameters that can range over the real numbers, which are infinitely complicated in general, than that's different, and string theory may have this issue as well. Re string theory's issue of being vacuous, I think the core thing that string theory predicts that other quantum gravity models don't is that at the large scale, you recover general relativity and the standard model, whereas no other theory can yet figure out a way to properly include both the empirical effects of gravity and quantum mechanics in the parameter regimes where they are known to work, so string theory predicts more just by predicting the things other quantum mechanics predicts while having the ability to include in gravity without ruining the other predictions, whereas other models of quantum gravity tend to ruin empirical predictions like general relativity approximately holding pretty fast.

[-]Mo Putera7mo20-2

I used to consider it a mystery that math was so unreasonably effective in the natural sciences, but changed my mind after reading this essay by Eric S. Raymond (who's here on the forum, hi and thanks Eric), in particular this part, which is as good a question dissolution as any I've seen:

The relationship between mathematical models and phenomenal prediction is complicated, not just in practice but in principle. Much more complicated because, as we now know, there are mutually exclusive ways to axiomatize mathematics! It can be diagrammed as follows (thanks to Jesse Perry for supplying the original of this chart):

(it's a shame this chart isn't rendering properly for some reason, since without it the rest of Eric's quote is ~incomprehensible)

The key transactions for our purposes are C and D -- the translations between a predictive model and a mathematical formalism. What mystified Einstein is how often D leads to new insights.
We begin to get some handle on the problem if we phrase it more precisely; that is, "Why does a good choice of C so often yield new knowledge via D?"
The simplest answer is to invert the question and treat it as a definition. A "good choi

... (read more)

6cubefox7mo

Interesting. This reminds me of a related thought I had: Why do models with differential equations work so often in physics but so rarely in other empirical sciences? Perhaps physics simply is "the differential equation science". Which is also related to the frequently expressed opinion that philosophy makes little progress because everything that gets developed enough to make significant progress splits off from philosophy. Because philosophy is "the study of ill-defined and intractable problems". Not saying that I think these views are accurate, though they do have some plausibility.

1Mo Putera7mo

(To be honest, to first approximation my guess mirrors yours.)

5Garrett Baker7mo

Flagging that those two examples seem false. The weather is chaotic, yes, and there's a sense in which the economy is anti-inductive, but modeling methods are advancing, and will likely find more loop-holes in chaos theory. For example, in thermodynamics, temperature is non-chaotic while the precise kinetic energies and locations of all particles are. A reasonable candidate similarity in weather are hurricanes. Similarly as our understanding of the economy advances it will get more efficient which means it will be easier to model. eg (note: I've only skimmed this paper). And definitely large economies are even more predictable than small villages, talk about not having a competitive market!

3Mo Putera7mo

Thanks for the pointer to that paper, the abstract makes me think there's a sort of slow-acting self-reinforcing feedback loop between predictive error minimisation via improving modelling and via improving the economy itself. re: weather, I'm thinking of the chart below showing how little gain we get in MAE vs compute, plus my guess that compute can't keep growing far enough to get MAE < 3 °F a year out (say). I don't know anything about advancements in weather modelling methods though; maybe effective compute (incorporating modelling advancements) may grow indefinitely in terms of the chart.

2Garrett Baker7mo

I didn't say anything about temperature prediction, and I'd also like to see any other method (intuition based or otherwise) do better than the current best mathematical models here. It seems unlikely to me that the trends in that graph will continue arbitrarily far. Yeah, that was my claim.

4Mo Putera7mo

Matt Leifer, who works in quantum foundations, espouses a view that's probably more extreme than Eric Raymond's above to argue why the effectiveness of math in the natural sciences isn't just reasonable but expected-by-construction. In his 2015 FQXi essay Mathematics is Physics Matt argued that (Matt notes as an aside that he's arguing for precisely the opposite of Tegmark's MUH.) Why "scale-free network"? As an aside, Matt's theory of theory-building explains (so he claims) what mathematical intuition is about: "intuition for efficient knowledge structure, rather than intuition about an abstract mathematical world". So what? How does this view pay rent? Matt further develops the argument that the structure of human knowledge being networked-not-hierarchical implies that the idea that there is a most fundamental discipline, or level of reality, is mistaken in Against Fundamentalism, another FQXi essay published in 2018.

3localdeity7mo

I would also comment that, if the environment was so chaotic that roughly everything important to life could not be modeled—if general-purpose modeling ability was basically useless—then life would not have evolved that ability, and "intelligent life" probably wouldn't exist.

2romeostevensit7mo

The two concepts that I thought were missing from Eliezer's technical explanation of technical explanation that would have simplified some of the explanation were compression and degrees of freedom. Degrees of freedom seems very relevant here in terms of how we map between different representations. Why are representations so important for humans? Because they have different computational properties/traversal costs while humans are very computationally limited.

1Mo Putera7mo

Can you say more about what you mean? Your comment reminded me of Thomas Griffiths' paper Understanding Human Intelligence through Human Limitations, but you may have meant something else entirely. Griffiths argued that the aspects we associate with human intelligence – rapid learning from small data, the ability to break down problems into parts, and the capacity for cumulative cultural evolution – arose from the 3 fundamental limitations all humans share: limited time, limited computation, and limited communication. (The constraints imposed by these characteristics cascade: limited time magnifies the effect of limited computation, and limited communication makes it harder to draw upon more computation.) In particular, limited computation leads to problem decomposition, hence modular solutions; relieving the computation constraint enables solutions that can be objectively better along some axis while also being incomprehensible to humans.

3romeostevensit7mo

Thanks for the link. I mean that predictions are outputs of a process that includes a representation, so part of what's getting passed back and forth in the diagram are better and worse fit representations. The degrees of freedom point is that we choose very flexible representations, whittle them down with the actual data available, then get surprised that that representation yields other good predictions. But we should expect this if Nature shares any modular structure with our perception at all, which it would if there was both structural reasons (literally same substrate) and evolutionary pressure for representations with good computational properties i.e. simple isomorphisms and compressions.

[-]Mo Putera4mo170

Neat example of mundane LLM utility: Automation of Systematic Reviews with Large Language Models

Abstract
Systematic reviews (SRs) inform evidence-based decision making. Yet, they take over a year to complete, are prone to human error, and face challenges with reproducibility; limiting access to timely and reliable information. We developed otto-SR, an end-to-end agentic workflow using large language models (LLMs) to support and automate the SR workflow from initial search to analysis. We found that otto-SR outperformed traditional dual human workflows in SR screening (otto-SR: 96.7% sensitivity, 97.9% specificity; human: 81.7% sensitivity, 98.1% specificity) and data extraction (otto-SR: 93.1% accuracy; human: 79.7% accuracy). Using otto-SR, we reproduced and updated an entire issue of Cochrane reviews (n=12) in two days, representing approximately 12 work-years of traditional systematic review work. Across Cochrane reviews, otto-SR incorrectly excluded a median of 0 studies (IQR 0 to 0.25), and found a median of 2.0 (IQR 1 to 6.5) eligible studies likely missed by the original authors. Meta-analyses revealed that otto-SR generated newly statistically significant conclusions in 2 re

... (read more)

[-]Mo Putera6mo171

This remark at 16:10 by Dwarkesh Patel on his most recent podcast interview AMA: Career Advice Given AGI, How I Research ft. Sholto & Trenton was pretty funny:

... big guests just don't really matter that much if you just look at what are the most popular episodes, or what in the long run helps a podcast grow. By far my most popular guest is Sarah Paine, and she, before I interviewed her, was just a scholar who was not publicly well-known at all, and I just found her books quite interesting—so my most popular guests are Sarah Paine and then Sarah Paine, Sarah Paine, Sarah Paine because I have ~~electric chairs(?)~~ a lecture series with her. And by the way, from a viewer-a-minute adjusted basis, I host the Sarah Paine podcast where I occasionally talk about AI.

(After Sarah Paine comes geneticist David Reich, then Satya Nadella and Mark Zuckerberg, "then [Sholto & Trenton] or Leopold (Aschenbrenner) or something, then you get to the lab CEOs or something")

[-]gwern6mo*548

You can see it as an example of 'alpha' vs 'beta'. When someone asks me about the value of someone as a guest, I tend to ask: "do they have anything new to say? didn't they just do a big interview last year?" and if they don't but they're big, "can you ask them good questions that get them out of their 'book'?" Big guests are not necessarily as valuable as they may seem because they are highly-exposed, which means both that (1) they have probably said everything they will said before and there is no 'news' or novelty, and (2) they are message-disciplined and careful to "talk their book". (In this analogy, "alpha" represents undiscovered or neglected interview topics which can be extracted mostly just by finding it and then asking the obvious question, usually by interviewing new people; "beta" represents doing standard interview topics/people, but much more so - harder, faster, better - and getting new stuff that way.)

Lex Fridman podcasts are an example of this: he often hosts very big guests like Mark Zuckerberg, but nevertheless, I will sit down and skim through the transcript of 2-4 hours of content, and find nothing even worth excerpting for my notes. Fridman notoriously does n... (read more)

3Mo Putera6mo

I like the optimal forager take, seems intuitively correct. I'd add that Dwarkesh struck gold by getting you on his podcast too. (Tangentially: this grand theory of intelligence video snippet reminds me of a page-ish-long writeup on that I stumbled upon deep in the bowels of https://gwern.net/ which I've annoyingly never been able to find again.) Also thanks for the pointer to Werbos, his website Welcome to the Werbos World! funnily enough struck me as crackpot-y and I wouldn't have guessed just from the landing page that he's the discoverer of backprop, respected former program director at the NSF, etc.

7gwern6mo

Probably https://gwern.net/newsletter/2021/05#master-synthesis That's what makes it alpha! If he was as legible as, say, Hinton, he would be mined out by now, and nothing but beta. (Similar situation to Schmidhuber - 'obvious crackpot' - although he's such a self-promoter that he overcomes it, and so at this point there's no alpha talking to him; the stuff that would be interesting, like his relationship to certain wealthy Italians, or to King Bonesaws, or how he's managed to torpedo his career so spectacularly, he will not talk about. Also, I understand he likes to charge people for the privilege of talking to him.) You have to have both domain knowledge and intellectual courage to know about Werbos and eg. read his old interviews and be willing to go out on a limb and interview him.

2Chris_Leong6mo

This seems to underrate the value of distribution. I suspect another factor to take into account is the degree of audience overlap. Like there's a lot of value in booking a guest who has been on a bunch of podcasts, so long as your particular audience isn't likely to have been exposed to them.

2sjadler6mo

I’d guess that was “I have a lecture series with her” :-)

1Mo Putera6mo

D'oh, you're obviously right, thanks!

[-]Mo Putera5mo163

Balioc's A taxonomy of bullshit jobs has a category called Worthy Work Made Bullshit which resonated with me most of all:

Worthy Work Made Bullshit is perhaps the trickiest and most controversial category, but as far as I’m concerned it’s one of the most important. This is meant to cover jobs where you’re doing something that is obviously and directly worthwhile…at least in theory…but the structure of the job, and the institutional demands that are imposed on you, turn your work into bullshit.
The conceptual archetype here is the Soviet tire factory that produces millions of tiny useless toy-sized tires instead of a somewhat-smaller number of actually-valuable tires that could be put on actual vehicles, because the quota scheme is badly designed. Everyone in that factory has a Worthy Work Made Bullshit job. Making tires is something you can be proud of, at least hypothetically. Making tiny useless tires to game a quota system is…not.
Nowadays we don’t have Soviet central planners producing insane demands, but we do have a marketplace that produces comparably-insane demands, especially in certain fields.
This is especially poignant, and e

... (read more)

6Thane Ruthenis5mo

Potentially relevant: this thread about a massive software service that did useful work, but ultimately could've been outperformed (at 100x) by a small, easy-to-implement adjustment to the overarching system.

[-]Mo Putera3mo150

I just learned about the idea of "effectual thinking" from Cedric Chin's recent newsletter issue. He notes, counterintuitively to me, that it's the opposite of causal thinking, and yet it's the one thing in common in all the successful case studies he could find in business:

The only answer that fits with the twenty-seven odd cases that we’ve published on the Idea Maze is a thing called effectual thinking.
Effectual thinking is a concept proposed by Professor Saras D Sarasvathy in a 2001 paper — a paper that was also, hilariously, described as “the first good paper I’ve seen” by Vinod Khosla, the renowned venture capitalist and cofounder of Sun Microsystems.
Saravathy didn’t pull this theory out of thin air: she hunted down a list of entrepreneurs who, at the time of her paper, represented a near-complete list of experienced founders (of enduring companies) in the US from the period of 1960 to 1985.
These folk were:
Repeat entrepreneurs (minimum three ventures, though the average number of companies started in her sample was seven new ventures)
Were successful and serious businesspeople (at least one of those ventures had gone public, and they had spent minimum 10 years with that company

... (read more)

[-]Raemon3mo199

(see also search term "forward chaining vs back-chaining.")

This seems like reasonable life advice for people generally trying to accumulate resources and do something cool. I'm not sure about people who actually have specific goals they want to accomplish. I think in the domain of AI safety, forward chaining is insufficient (seems like the kind of thing that gets you OpenAI and Anthropic)

The principles I sort of try to live by are, each 2-weeks, I should have done:

some actions that forward chain towards more compounding resources
some actions explicitly backchaining from longterm goals
ship something concrete to users
do something wholesome

Which is I think aiming to accomplish similar goals towards the OP, without losing the plot on my more specific goals.

3Raemon3mo

This maybe reminds me: I currently feel confused about how to integrate "the kind of thinking that is good at momentum / action" and "the kind of good that is good at creative strategy". And it seems like there should be more of a way to unify them into a holistic way-of-being. The four checksums above are there to make sure I'm not being myopic in some way in a broader sense, but they apply more at the timescale of weeks than hours or days. You might just say "well, idk, each week or day, just figure out if it's more like a momentum week or more like a creative strategy week". I feel dissatisfied with this for some reason. At least part of it is "I think on average people/me could use to be in creative/broader strategy mode more often, even when in a Momentum mode period." Another part is "there are strategy skills I want to be practicing, that are hard to practice if I don't do them basically every day. They aren't as relevant in a momentum-period, but they're not zero relevant. Hrm. I think maybe what's most dissatisfying right now is that I just haven't compressed all the finnicky details of it, and it feels overwhelming to think about the entire "how to think" project, which is usually an indicator I am missing the right abstraction.

[-]Mo Putera8mo140

Unbundling Tools for Thought is an essay by Fernando Borretti I found via Gwern's comment which immediately resonated with me (emphasis mine):

I’ve written something like six or seven personal wikis over the past decade. It’s actually an incredibly advanced form of procrastination¹. At this point I’ve tried every possible design choice.
Lifecycle: I’ve built a few compiler-style wikis: plain-text files in a git repo statically compiled to HTML. I’ve built a couple using live servers with server-side rendering. The latest one is an API server with a React frontend.
Storage: I started with plain text files in a git repo, then moved to an SQLite database with a simple schema. The latest version is an avant-garde object-oriented hypermedia database with bidirectional links implemented on top of SQLite.
Markup: I used Markdown here and there. Then I built my own TeX-inspired markup language. Then I tried XML, with mixed results. The latest version uses a WYSIWYG editor made with ProseMirror.
And yet I don’t use them. Why? Building them was fun, sure, but there must be utility to a personal database.
At first I thought the problem was friction: the higher the activation energy to u

... (read more)

4Viliam8mo

Minimizing friction is surprisingly difficult. I keep plain-text notes in a hierarchical editor (cherrytree), but even that feels too complicated sometimes. This is not just about the tool... what you actually need is a combination of the tool and the right way to use it. (Every tool can be used in different ways. For example, suppose you write a diary in MS Word. There are still options such as "one document per day" or "one very long document for all", and things in between like "one document per month", which all give different kinds of friction. The one megadocument takes too much time to load. It is more difficult to search in many small documents. Or maybe you should keep your current day in a small document, but once in a while merge the previous days into the megadocument? Or maybe switch to some application that starts faster than MS Word?) Forgetting is an important part. Even if you want to remember forever, you need some form of deprioritizing. Something like "pages you haven't used for months will get smaller, and if you search for keywords, they will be at the bottom of the result list". But if one of them suddenly becomes relevant again, maybe the connected ones become relevant, too? Something like associations in brain. The idea is that remembering the facts is only a part of the problem; making the relevant ones more accessible is another. Because searching in too much data is ultimately just another kind of friction. It feels like a smaller version of the internet. Years ago, the problem used to be "too little information", now the problem is "too much information, can't find the thing I actually want". Perhaps a wiki, where the pages could get flagged as "important now" and "unimportant"? Or maybe, important for a specific context? And by default, when you choose a context, you would only see the important pages, and the rest of that only if you search for a specific keyword or follow a grey link. (Which again would require some work creating

2Milan W8mo

@dkl9 wrote a very eloquent and concise piece arguing in favor of ditching "second brain" systems in favor of SRSs (Spaced Repetition Systems, such as Anki).

2Jonas Hallgren8mo

I like to think of learning and all of these things as self-contained smaller self-contained knowledge trees. Building knowledge trees that are cached, almost like creatin zip files and systems where I store a bunch of zip files similar to what Elizier talks about in The Sequences. Like when you mention the thing about Nielsen on linear algebra it opens up the entire though tree there. I might just get the association to something like PCA and then I think huh, how to ptimise this and then it goes to QR-algorithms and things like a householder matrix and some specific symmetric properties of linear spaces... If I have enough of these in an area then I might go back to my anki for that specific area. Like if you think from the perspective of schedulling and storage algorithms similar to what is explored in algorithms to live by you quickly understand that the magic is in information compression and working at different meta-levels. Zipped zip files with algorithms to expand them if need be. Dunno if that makes sense, agree with the exobrain creep that exists though.

[-]Mo Putera2mo110

Interesting quote by Michael Nielsen:

ZIERLER: Michael, we've already covered what you're doing currently, so for the last part of our talk, I'd like to ask a retrospective question, and then we'll end looking to the future. Between quantum information and open science, I wonder if you've thought broadly about the Kuhnian idea of scientific revolutions. Both quantum information and open science were not areas that he thought about. But to the extent that you believe in that paradigm—and I don't know if you; that's a question in and of itself—how does your expertise or the reality of these fields influence the idea of scientific progress, the idea of scientific revolutions?
NIELSEN: Just focusing on Kuhn, he had a very narrow sense of what a scientific revolution is. One of the most common criticisms of him is that he was too focused on physics in particular; and to some extent, chemistry. He was originally trained as a theoretical physicist, and it really shows in his writing. Something like quantum mechanics or general relativity is disrupting an existing order. Well, quantum information isn't disrupting any existing order. It's not like there's a whole bunch of ex-classical informa

... (read more)

[-]Mo Putera5mo110

Venkatesh Rao surprised me in What makes a good teacher? by saying the opposite of what I expected him to say re: his educational experience, given who he is:

While my current studies have no live teachers in the loop, each time I sit down to study something seriously, I’m reminded of how much I’m practicing behaviors first learned under the watchful eye of good teachers. We tend to remember the exceptionally charismatic (which is not the same thing as good), and exceptionally terrible teachers, but much of what we know about how to learn, how to study, comes from the quieter good teachers, many of whom we forget.
It also strikes me, reflecting on my own educational path — very conventional both on paper and in reality — that the modern public discourse around teaching and learning has been hijacked to a remarkable degree by charismatic public figures mythologizing their own supposedly maverick education stories.
These stories often feature exaggerated elements of rebellion, autodidact mastery, subversive hacking, heroic confrontations with villainous teachers and schoolyard bullies, genius non-neurotypical personal innovations and breakthroughs, and powerful experiences outside forma

... (read more)

[-]Mo Putera6mo110

(Not a take, just pulling out infographics and quotes for future reference from the new DeepMind paper outlining their approach to technical AGI safety and security)

Overview of risk areas, grouped by factors that drive differences in mitigation approaches:

Overview of their approach to mitigating misalignment:

Overview of their approach to mitigating misuse:

Path to deceptive alignment:

How to use interpretability:

Goal	Understanding v Control	Confidence	Concept v Algorithm	(Un)supervised?	How context specific?
Alignment evaluations	Understanding	Any	Concept+	Either	Either
FaithfulReasoning	Understanding^∗	Any	Concept+	Supervised+	Either
DebuggingFailures	Understanding^∗	Low	Either	Unsupervised+	Specific
Monitoring	Understanding	Any	Concept+	Supervised+	General
Red teaming	Either	Low	Either	Unsupervised+	Specific
Amplified oversight	Understanding	Complicated	Concept	Either	Specific

Interpretability techniques:

Technique	Understanding v Control	Confidence	Concept v Algorithm	(Un)supervised?	How specific?	Scalability
Probing	Understanding	Low	Concept	Supervised	Specific-ish	Cheap
Dictionary learning	Both	Low	Concept	Unsupervised	General^∗	Expensive
Steering vectors	Control	Low	Concept	Supervised	Specific-ish	Cheap
Training data attribution	Understanding

... (read more)

[-]Mo Putera8mo113

I currently work in policy research, which feels very different from my intrinsic aesthetic inclination, in a way that I think Tanner Greer captures well in The Silicon Valley Canon: On the Paıdeía of the American Tech Elite:

I often draw a distinction between the political elites of Washington DC and the industrial elites of Silicon Valley with a joke: in San Francisco reading books, and talking about what you have read, is a matter of high prestige. Not so in Washington DC. In Washington people never read books—they just write them.
To write a book, of course, one must read a good few. But the distinction I drive at is quite real. In Washington, the man of ideas is a wonk. The wonk is not a generalist. The ideal wonk knows more about his or her chosen topic than you ever will. She can comment on every line of a select arms limitation treaty, recite all Chinese human rights violations that occurred in the year 2023, or explain to you the exact implications of the new residential clean energy tax credit—but never all at once. ...
Washington intellectuals are masters of small mountains. Some of their peaks are more difficult to summit than others. Many smaller slopes are nonetheless ja

... (read more)

[-]Mo Putera4mo100

Asked ChatGPT o3 to read 6 links (nothing intensive, just the familiar essays on cognitive decoupling by Jacob Falkovich, Sarah Constantin, John Nerst, drossbucket) and its reasoning repeatedly mentions this which I've never seen before (emphasis mine):

I’m planning to review each article carefully. I’ll first open lines around the 250-line mark to get a deeper view, but I’m also mindful of the word limit. ...
I’m thinking I’ll need to open each article and focus on the part about "decoupling." To get more detailed, I’ll try opening further into the text — maybe around 250 lines — to ensure I capture enough context without overwhelming myself.
I plan to open each article and examine lines near the bottom to view relevant content. There's a limit of 200 lines, so I might need to focus on searching for specific keywords like "decoupling."
I’m trying to locate specific references to "decoupling" in each article. I want to open around 250 lines to make sure I find enough content for a thorough summary. I’ll use a quick search to find "decoupling" precisely, but if it’s not yielding precise results, I’ll expand the search to ensure I get more lines. It’s about balancing deep acc

... (read more)

3Mis-Understandings4mo

I think this might be an attempted countermeasure against prompt injection. That is, it wants to mix autoregressive and reconstructed residuals. Otherwise, it might lose it's train of thought (end up continuing the article not following the prompt).

[-]Mo Putera5mo100

Saving mathematician Robert Ghrist's tweet here for my own future reference re: AI x math:

workflow of the past 24 hours...
* start a convo w/GPT-o3 about math research idea [X]
* it gives 7 good potential ideas; pick one & ask to develop
* feed -o3 output to gemini-2.5-pro; it finds errors & writes feedback
* paste feedback into -o3 and say asses & respond
* paste response into gemini; it finds more problems
* iterate until convergence
* feed the consensus idea w/detailed report to grok-3
* grok finds gaping error, fixes by taking things in different direction (!!!)
* gemini agrees: big problems, now ameliorated
* output final consensus report
* paste into claude-3.7 and ask it to outline a paper
* approve outline; request latex following my style/notation conventions
* claude outputs 30 pages of dense latex, section by section, one-shot (!)
====
is this correct/watertight? (surely not)
is this genuinely novel? (pretty sure yes)
is this the future? (no, it's the present)
====
everybody underestimates not only what is coming but what can currently be done w/existing tools.

Someone asked why split things between o3 and 2.5 Pro; Ghrist:

they have complementary strengths and each picks up

... (read more)

[-]Mo Putera8mo100

I enjoyed Brian Potter's Energy infrastructure cheat sheet tables over at Construction Physics, it's a great fact post. Here are some of Brian's tables — if they whet your appetite, do check out his full essay.

Energy quantities:

Units and quantities	Kilowatt-hours	Megawatt-hours	Gigawatt-hours
1 British Thermal Unit (BTU)	0.000293
iPhone 14 battery	0.012700
1 pound of a Tesla battery pack	0.1
1 cubic foot of natural gas	0.3
2000 calories of food	2.3
1 pound of coal	2.95
1 gallon of milk (calorie value)	3.0
1 gallon of gas	33.7
Tesla Model 3 standard battery pack	57.5
Typical ICE car gas tank (15 gallons)	506
1 ton of TNT	1,162
1 barrel of oil	1,700
1 ton of oil	11,629	12
Tanker truck full of gasoline (9300 gallons)	313,410	313
LNG carrier (180,000 cubic meters)	1,125,214,740	1,125,215	1,125
1 million tons of TNT (1 megaton)	1,162,223,152	1,162,223	1,162
Oil supertanker (2 million barrels)	3,400,000,000	3,400,000	3,400

It's amazing that a Tesla Model 3's standard battery pack has an OOM less energy capacity than a typical 15-gallon ICE car gas tank, and is probably heavier to... (read more)

[-]Mo Putera4mo90

A subgenre of fiction I wish I could read more of is rationalist-flavored depictions of utopia that centrally feature characters who intentionally and passionately pursue unpleasant experiences, which I don't see much of. It's somewhat surprising since it's a pretty universal orientation.

For instance, and this is a somewhat extreme version, I'm a not-that-active member of a local trail running group (all professionals with demanding day jobs) that meets regularly for creative sufferfests like treasure hunt races in the mountains, some of whom regularly fly... (read more)

[-]Richard_Ngo4mo124

Have you read The Metamorphosis of Prime Intellect? Fits the bill.

5Richard Horvath4mo

In books about the Culture sci fi universe such things are described a couple of times. E.g. in the novel "Use of Weapons" the "crew" (the ship is fully automated, so more like permanent passengers) of a ship deliberately weaken their immune system to basically get a seasonal cold just for the experience, which otherwise could not happen due to their genetically enhanced immune system.

2Seth Herd4mo

Also lava rafting and other extreme sports, maybe in Look to Windward which focuses a bit more on the Culture. Many of the human protagonists in the Culture experience significant self-hatred, although that's not the only reason to seek out experiences so difficult they may become net negative. It's as though the Culture is missing advanced therapeutic techniques along with a desire for immortality. I'd like an updated utopia.

3Richard Horvath4mo

Well, the disturbed protagonists in the Culture series (as in: books, and in the whole of the fictional universe) are usually not from the "Culture" (one particular civilizations within the whole fictional universe), but outsiders hired to act as agents.

4Seth Herd4mo

Hm, interesting. I remembered that about Zakalwe but my memory for the others is vague. So maybe Culture citizens are so well-adjusted that they wouldn't risk their lives?

[-]Mo Putera6mo90

Pilish is a constrained writing style where the number of letters in consecutive words match the digits of pi. The canonical intro-to-Pilish sentence is "How I need a drink, alcoholic of course, after the heavy lectures involving quantum mechanics!"; my favorite Pilish poetry is Mike Keith's Near a Raven, a retelling of Edgar Allan Poe's "The Raven" stretching to 740 digits of pi (nowhere near Keith's longest, that would be the 10,000-word world record-setting Not a Wake), which begins delightfully like so:

Poe, E.
Near a Raven
Midnights

... (read more)

7CAC6mo

I asked GPT 4.5 to write a system prompt and user message for models to write Pilish poems, feeding it your comment as context. Then I gave these prompts to o1 (via OpenAI's playground). GPT 4.5's system prompt You are an expert composer skilled in writing poetry under strict, unusual linguistic constraints, specifically "Pilish." Pilish is a literary constraint in which the length of consecutive words precisely matches each digit of π (pi). The first word contains 3 letters, second word 1 letter, third word 4 letters, fourth word 1 letter, fifth word 5 letters, sixth word 9 letters, and so forth, accurately reflecting the sequence of pi’s digits. For example, the classic Pilish sentence is: "How I need a drink, alcoholic of course, after the heavy lectures involving quantum mechanics!" As another reference, Mike Keith famously wrote a poem titled "Near a Raven," a Pilish version of Edgar Allan Poe's "The Raven," precisely encoding hundreds of pi digits into elegant verse. Important guidelines: You must strictly adhere to the Pilish constraint without exception or deviation. Carefully recount each word to match each digit exactly. Maintain coherence, clarity, and literary quality in your writing. Make sure the poem or text is meaningful, evocative, creative, and intriguing, not just mechanically correct. Verify each word length explicitly to ensure compliance with Pilish. Your task is to craft a compelling, original poem in Pilish form. Do not explain the poem afterward; the precision and artistry of the piece itself is sufficient. -- GPT 4.5's user message Write an original Pilish poem about pi itself: reflecting upon the wonder, mystery, and endless fascination humans have felt over millennia towards this famous mathematical constant. Make it vivid, thoughtful, and meticulously correct in matching the Pilish constraint. Begin now. -- o1's completion I asked o1 to count the letters in each word, comparing with pi: And this seems correct!

2Mo Putera6mo

This is great, thanks! Didn't think of the model-prompting-model trick.

4MondSemmel6mo

LLMs use tokens instead of letters, so counting letters is sufficiently unnatural to them relative to their other competencies that I don't see much value in directly asking LLMs to do this kind of thing. At least give them some basic scaffolding, like a full English dictionary with a column which explicitly indicates respective word lengths. In particular, the Gemini models have a context window of 1M tokens, which should be enough to fit most of the Oxford English Dictionary in there (since it includes 171k words which are in current use).

4Mo Putera6mo

I think I failed to implicitly convey that I meant all this in jest, that I get a lot of personal enjoyment value out of silly poetry constrained by artificial rules, and that I was guessing at least someone else on the forum would share this enjoyment. I do like your scaffolding idea, might just try it out.

[-]Mo Putera6mo80

There's a lot of fun stuff in Anders Sandberg's 1999 paper The Physics of Information Processing Superobjects: Daily Life Among the Jupiter Brains. One particularly vivid detail was (essentially) how the square-cube law imposes itself upon Jupiter brain architecture by forcing >99.9% of volume to be comprised of comms links between compute nodes, even after assuming a "small-world" network structure allowing sparse connectivity between arbitrarily chosen nodes by having them be connected by a short series of intermediary links with only 1% of links bein... (read more)

[-]Mo Putera7mo80

From Brian Potter's Construction Physics newsletter I learned about Taara, framed as "Google's answer to Starlink" re: remote internet access, using ground-based optical communication instead of satellites ("fiber optics without the fibers"; Taara calls them "light bridges"). I found this surprising. Even more surprisingly, Taara isn't just a pilot but a moneymaking endeavor if this Wired passage is true:

Taara is now a commercial operation, working in more than a dozen countries. One of its successes came in crossing the Congo River. On one side was Brazza

... (read more)

[-]Mo Putera8mo81

Peter Watts' 2006 novel Blindsight has this passage on what it's like to be a "scrambler", superintelligent yet nonsentient (in fact superintelligent because it's unencumbered by sentience), which I read a ~decade ago and found unforgettable:

Imagine you're a scrambler.
Imagine you have intellect but no insight, agendas but no awareness. Your circuitry hums with strategies for survival and persistence, flexible, intelligent, even technological—but no other circuitry monitors it. You can think of anything, yet are conscious of nothing.
You can't imagine such a

... (read more)

3quetzal_rainbow8mo

It's very funny that Rorschach linguistic ability is totally unremarkable comparing to modern LLMs.

[-]Mo Putera9mo81

Found an annotated version of Vernor Vinge's A Fire Upon The Deep.

3Rasool9mo

How interesting, I was curious about copyright etc but this is annotated by the author himself!

[-]Mo Putera7mo70

Ravi Vakil's advice for potential PhD students includes this bit on "tendrils to be backfilled" that's stuck with me ever since as a metaphor for deepening understanding over time:

Here's a phenomenon I was surprised to find: you'll go to talks, and hear various words, whose definitions you're not so sure about. At some point you'll be able to make a sentence using those words; you won't know what the words mean, but you'll know the sentence is correct. You'll also be able to ask a question using those words. You still won't know what the words mean, but yo

... (read more)

[-]Mo Putera2mo62

Kyle Kingsbury's technical interview pentalogy of short stories is unlike anything else I've read. Here's how the first story begins:

If you want to get a job as a software witch, you’re going to have to pass a whiteboard interview. We all do them, as engineers–often as a part of our morning ritual, along with arranging a beautiful grid of xterms across the astral plane, and compulsively running ls in every nearby directory–just in case things have shifted during the night–the incorporeal equivalent of rummaging through that drawer in the back of the kitche

... (read more)

[-]Mo Putera7mo60

Out of curiosity — how relevant is Holden's 2021 PASTA definition of TAI still to the discourse and work on TAI, aside from maybe being used by Open Phil (not actually sure that's the case)? Any pointers to further reading, say here or on AF etc?

AI systems that can essentially automate all of the human activities needed to speed up scientific and technological advancement. I will call this sort of technology Process for Automating Scientific and Technological Advancement, or PASTA.³ (I mean PASTA to refer to either a single system or a collection of system

... (read more)

[-]Mo Putera8mo62

When I first read Hannu Rajaniemi's Quantum Thief trilogy c. 2015 I had two reactions: delight that this was the most my-ingroup-targeted series I had ever read, and a sinking feeling that ~nobody else would really get it, not just the critics but likely also most fans, many of whom would round his carefully-chosen references off to technobabble. So I was overjoyed to recently find Gwern's review of it, which Hannu affirms "perfectly nails the emotional core of the trilogy and, true to form, spots a number of easter eggs I thought no one would ever find", ... (read more)

3Seth Herd8mo

The parts of the science I understand were all quite plausible (mind duplication/fractioning and motivations for doing so). Beyond the accuracy of the science, this was one of the most staggeringly imaginative and beautifully written scifi books I've ever read. It's for a very particular audience, but if you're here you might be that audience. If you are, this might be the best book you've read.

3Mo Putera8mo

Attention conservation notice: 3,000+ words of longform quotes by various folks on the nature of personal identity in a posthuman future, and hiveminds / clans As an aside, one of the key themes running throughout the Quantum Thief trilogy is the question of how you might maintain personal identity (in the pragmatic security sense, not the philosophical one) in a future so posthuman that minds can be copied and forked indefinitely over time. To spoil Hannu's answer: I take Anders Sandberg's answer to be on the other end of this spectrum; he doesn't mind changing over time such that he might end up wanting different things: (I have mixed feelings about Anders' take: I have myself changed so profoundly since youth that that my younger self would not just disendorse but be horrified by the person I am now, yet I did endorse every step along the way, and current-me still does upon reflection (but of course I do). Would current-me also endorse a similar degree of change going forward, even subject to every step being endorsed by the me right before change? Most likely not, perhaps excepting changes towards some sort of reflective equilibrium.) I interpret Holden Karnofsky's take to be somewhere in between, perhaps closer to Hannu's answer. Holden remarked that he doesn't find most paradoxical thought experiments about personal identity (e.g. "Would a duplicate of you be "you?"" or "If you got physically destroyed and replaced with an exact duplicate of yourself, did you die?") all that confounding because his personal philosophy on "what counts as death" dissolves them, and that his philosophy is simple, comprising just 2 aspects: constant replacement ("in an important sense, I stop existing and am replaced by a new person each moment") and kinship with future selves. Elaborating on the latter: Richard Ngo goes in a different direction with the "personal identity in a posthuman future" question: (I thought it was both interesting and predictable that Rob would f

1Mo Putera7mo

The short story The Epiphany of Gliese 581 by Fernando Borretti has something of the same vibe as Rajaniemi's QT trilogy; Borretti describes it as inspired by Orion's Arm and the works of David Zindell. Here's a passage describing a flourishing star system already transformed by weakly posthuman tech: Another star system, this time still being transformed:

[-]Mo Putera6mo50

One subsubgenre of writing I like is the stress-testing of a field's cutting-edge methods by applying it to another field, and seeing how much knowledge and insight the methods recapitulate and also what else we learn from the exercise. Sometimes this takes the form of parables, like Scott Alexander's story of the benevolent aliens trying to understand Earth's global economy from orbit and intervening with crude methods (like materialising a billion barrels of oil on the White House lawn to solve a recession hypothesised to be caused by an oil shortage) to... (read more)

[-]Mo Putera9mo50

I enjoyed these passages from Henrik Karlsson's essay Cultivating a state of mind where new ideas are born on the introspections of Alexander Grothendieck, arguably the deepest mathematical thinker of the 20th century.

In June 1983, Alexander Grothendieck sits down to write the preface to a mathematical manuscript called Pursuing Stacks. He is concerned by what he sees as a tacit disdain for the more “feminine side” of mathematics (which is related to what I’m calling the solitary creative state) in favor of the “hammer and chisel” of the finished theo

... (read more)

3Viliam8mo

A few days ago, I was thinking about matrices and determinants. I noticed that I know the formula for the determinant, but I still lack the feeling of what the determinant is. I played with that thought for some time, and then it occurred to me, that if you imagine the rows in the matrix as vectors in n-dimensional space, then the determinant of that matrix is the volume of the n-dimensional body whose edges are those vectors. And suddenly it all made a fucking sense. The determinant is zero when the vectors are linearly dependent? Of course, that means that the n-dimensional body has been flattened into n-1 dimensions (or less), and therefore its volume is zero. The determinant doesn't change if you add a multiple of a row to some other row? Of course, that means moving the "top" of the n-dimensional body in a direction parallel to the "bottom", so that neither the bottom nor the height changes; of course the volume (defined as the area of the bottom multiplied by the height) stays the same. What about the determinant being negative? Oh, that just means whether the edges are "clockwise" or "counter-clockwise" in the n-dimensional space. It all makes perfect sense! Then I checked Wikipedia... and yeah, it was already there. So much for my Nobel prize. But it still felt fucking good. (And if I am not too lazy, one day I may write a blog article about it.) Reinventing the wheel is not a waste of time. I will probably remember this forever, and the words "determinant of the matrix" will never feel the same. Who knows, maybe this will help me figure out something else later. And if I keep doing that, hypothetically speaking, some of those discoveries might even be original. (The practical problem is that none of this can pay my bills.)

3Mo Putera8mo

I kind of envy that you figured this out yourself — I learned the parallelipiped hypervolume interpretation of the determinant from browsing forums (probably this MSE question's responses). Also, please do write that blog article. Yeah, I hope you will! I'm reminded of what Scott Aaronson said recently:

[-]Mo Putera2mo40

Just stumbled upon a spreadsheet ranking "well-known" googological numbers created by redditor u/Laxxius1 as a passion project. I've been idly interested in googology since childhood, so perusing this list felt like getting the Christmas present I've always wanted but kept forgetting to ask for. (It's also just intrinsically fun if your mathematical aesthetic is wrestler, which I'm guessing is overrepresented among LWers.)

Yudkowsky's number, which Eliezer created back in 2007 to win this contest on the xkcd forum, features at #12. (Eliezer also wrote Why i... (read more)

[-]Mo Putera2mo40

I wonder to what extent these impressions by Some Guy over at Extelligence are shared by others:

Some of this may be biased by my own background, but basically I’ve always found mathematicians and physicists to be the most humble and approachable people in all of the sciences. Second, are chemists and material scientists who can be somewhat cold but are always honest and straightforward because they have nothing to prove. But biologists? Man. The descriptive sciences have a chip on their shoulders and while social sciences are usually full of people w

... (read more)

[-]Mo Putera3mo40

Kishore Mahbubani, Singaporean diplomat and former president of the UN Security Council, studied philosophy full-time as an undergraduate in the late 60s. Recounting that period in his autobiography Living the Asian Century he wrote

For the final examinations, which I took at the end of my fourth year, our degree was determined by how well we did in eight three-hour examinations. In one of the papers, we had to answer a single question. The one question I chose to answer over three hours was “Can a stone feel pain?”
From my exam results, I gained a fir

... (read more)

[-]Mo Putera4mo40

Why doesn't Applied Divinity Studies' The Repugnant Conclusion Isn't dissolve the argumentative force of the repugnant conclusion?

But read again more carefully: “There is nothing bad in each of these lives”.
Although it sounds mundane, I contend that this is nearly incomprehensible. Can you actually imagine what it would be like to never have anything bad happen to you? We don’t describe such a as mediocre, we describe it as “charmed” or “overwhelmingly privileged”. ...
... consider Parfit’s vision of World Z both seriously and literally.
These are live

... (read more)

5Garrett Baker4mo

First, this is not the phrase I associate with the repugnant conclusion. "Net positive" does not mean "there is nothing bad in each of these lives". Second, I do think a key phrase & motivating description is "all they have is muzak and potatoes". That is all they have. I like our world where people can be and do great things. I won't describe it in poetic terms, since I don't think that makes good moral philosophy. If you do want something more poetic, idk read Terra Ignota or The Odyssey. Probably Terra Ignota moreso than The Odyssey. I will say that I like doing fun things, and I think many other people like doing fun things, and though my life may be net positive sitting around in a buddhist temple all day, I would likely take a 1-in-a-million chance of death to do awesome stuff instead. And so, I think, would many others. And we could all make a deal, we draw straws, and those 1-in-a-million who draw short give the rest their resources and are put on ice until we figure out a way to get enough resources so they could do what they love. Or, if that's infeasible (and in most framings of the problem it seems to be), willfully die. I mean, if nothing else, you can just gather all those who love extreme sports (which will be a non-trivial fraction of the population), and ask them to draw straws & re-consolidate the relevant resources to the winners. Their revealed preference would say "hell yes!" (we can tell, given the much lower stakes & much higher risk of the activities they're already doing). And I don't think the extreme sports lovers would be the only group who would take such a deal. Anyone who loves doing anything will take that deal, and (especially in a universe with the resources able to be filled to the brim with people just above the "I'll kill myself" line) I think most will have such a passion able to be fulfilled (even if it is brute wireheading!). And then, if we know this will happen ahead of time--that people will risk death to celebrate th

2Garrett Baker4mo

A big factor against making 1-in-a-million higher for most people is the whole death aspect, but death itself is a big negative, much worse to die than to never have been born (or so I claim), so the above gives a lower bound on the factor by which the repugnant conclusion will be off by.

4cubefox4mo

The Parfit quote from the blog post is taken out of context. Here is the relevant section in Parfit's essay: (Each box represents a possible population, with the height of a box representing how good overall an individual life is in that population, and the width representing the size of the population. The area of a box is the sum total "goodness"/"welfare"/"utility" (e.g. well-being, satisfied preferences, etc) in that population. The areas increase from A to Z, with Z being truncated here.) Note that Parfit describes two different ways in which an individual life in Z could be barely worth living (emphasis added): Then he goes on to describe the second possibility (which is arguably unrealistic and much less likely than the first, and which contains the quote by the blog author). The author of the blog posts mistakenly ignores Parfit's mentioning the first possibility. After talking about the second, Parfit returns (indicated by "similarly") to the first possibility: The "greatest quantity" here can simply be determined by the weight of all the positive things in an individual life minus the weight of all the negative things. Even if the result is just barely positive for an individual, for a large enough population, the sum welfare of the "barely net positive" individual lives would outweigh the sum for a smaller population with much higher average welfare. Yet intuitively, we should not trade a perfect utopia with relatively small population (A) for a world that is barely worth living for everyone in a huge population (Z). That's the problem with total utilitarianism, which simply sums all the "utilities" of the individual lives to measure the overall "utility" of a population. Taking the average instead of the sum avoids the repugnant conclusion, but it leads to other highly counterintuitive conclusions, such as that a population of a million people suffering strongly is less bad than a population of just a single person suffering slightly more strongly

2Richard_Kennaway4mo

The comment you made a little later looks like your answer to that question.

[-]Mo Putera5mo40

What is the current best understanding of why o3 and o4-mini hallucinate more than o1? I just got round to checking out the OpenAI o3 and o4-mini System Card and in section 3.3 (on hallucinations) OA noted that

o3 tends to make more claims overall, leading to more accurate claims as well as more inaccurate/hallucinated claims. While this effect appears minor in the SimpleQA results (0.51 for o3 vs 0.44 for o1), it is more pronounced in the PersonQA evaluation (0.33 vs 0.16). More research is needed to understand the cause of these results.

as of ... (read more)

[-]Thane Ruthenis5mo103

This is one potential explanation:

o3 has some sort of internal feature like "Goodhart to the objective"/"play in easy mode".
o3's RL post-training environments have opportunities for reward hacks.
o3 discovers and exploits those opportunities.
RL rewards it for that, reinforcing the "Goodharting" feature.
This leads to specification-hack-y behavior generalizing out of distribution, to e. g. freeform conversations. It ends up e. g. really wanting to sell its interlocutor on what it's peddling, so it deliberately^[1] confabulates plausible authoritative-sounding claims and justifications for them.

Sounds not implausible, though I'm not wholly convinced.

^{^}
In whatever sense this term can be applied to an LLM.

4Mo Putera5mo

Thank you, sounds somewhat plausible to me too. For others' benefit, here's the chart from davidad's linked tweet:

1Weaverzhu5mo

I've found the original paper of this chart https://arxiv.org/pdf/2503.11926v1 > We use prompted GPT-4o models to monitor a frontier reasoning agent, an agent in the same family as OpenAI o1 and o3-mini. During training, the agent discovered two hacks affecting nearly all training environments: The model is in the same family as o1 and o3-mini. Maybe o3 but not comfirmed.

[-]Mo Putera6mo40

Venkatesh Rao's recent newsletter article Terms of Centaur Service caught my eye for his professed joy of AI-assisted writing, both nonfiction and fiction:

In the last couple of weeks, I’ve gotten into a groove with AI-assisted writing, as you may have noticed, and I am really enjoying it. ... The AI element in my writing has gotten serious, and I think is here to stay. ...
On the writing side, when I have a productive prompting session, not only does the output feel information dense for the audience, it feels information dense for me.
An example of th

... (read more)

2Viliam6mo

Most human fiction is only interesting to the human who wrote it. The popular stuff is but a tiny minority out of all that was ever written.

[-]Mo Putera6mo40

Nice reminiscence from Stephen Wolfram on his time with Richard Feynman:

Feynman loved doing physics. I think what he loved most was the process of it. Of calculating. Of figuring things out. It didn’t seem to matter to him so much if what came out was big and important. Or esoteric and weird. What mattered to him was the process of finding it. And he was often quite competitive about it.
Some scientists (myself probably included) are driven by the ambition to build grand intellectual edifices. I think Feynman — at least in the years I knew him — was m

... (read more)

[-]Mo Putera8mo40

This is a top-level comment collecting various quotes discussing the posthuman condition.

2Mo Putera8mo

Hal Finney's reflections on the comprehensibility of posthumans, from the Vinge singularity discussion which took place on the Extropians email list back in the day:

1Mo Putera8mo

Linking to a previous comment: 3,000+ words of longform quotes by various folks on the nature of personal identity in a posthuman future, and hiveminds / clans, using Hannu Rajaniemi's Quantum Thief trilogy as a jumping-off point.

[-]Mo Putera6mo30

Terry Tao recently wrote a nice series of toots on Mathstodon that reminded me of what Bill Thurston said:

1. What is it that mathematicians accomplish?
There are many issues buried in this question, which I have tried to phrase in a way that does not presuppose the nature of the answer.
It would not be good to start, for example, with the question
How do mathematicians prove theorems?
This question introduces an interesting topic, but to start with it would be to project two hidden assumptions: (1) that there is uniform, objective and firmly establ

... (read more)

2cubefox6mo

If Thurston is right here and mathematicians want to understand why some theorem is true (rather than to just know the truth values of various conjectures), and if we "feel the AGI" ... then it seems future "mathematics" will consist in "mathematicians" asking future ChatGPT to explain math to them. Whether something is true, and why. There would be no research anymore. The interesting question is, I think, whether less-than-fully-general systems, like reasoning LLMs, could outperform humans in mathematical research. Or whether this would require a full AGI that is also smarter than mathematicians. Because if we had the latter, it would likely be an ASI that is better than humans in almost everything, not just mathematics.

[-]Mo Putera6mo30

The OECD working paper Miracle or Myth? Assessing the macroeconomic productivity gains from Artificial Intelligence, published quite recently (Nov 2024), is strange to skim-read: its authors estimate just 0.24-0.62 percentage points annual aggregate TFP growth (0.36-0.93 pp. for labour productivity) over a 10-year horizon, depending on scenario, using a "novel micro-to-macro framework" that combines "existing estimates of micro-level performance gains with evidence on the exposure of activities to AI and likely future adoption rates, relying on a multi-sec... (read more)

[-]Mo Putera3mo20

Ben Evans' Are better models better? (from a business/consumer perspective, not LW/AF etc):

Part of the concept of ‘Disruption’ is that important new technologies tend to be bad at the things that matter to the previous generation of technology, but they do something else important instead. Asking if an LLM can do very specific and precise information retrieval might be like asking if an Apple II can match the uptime of a mainframe, or asking if you can build Photoshop inside Netscape. No, they can’t really do that, but that’s not the point and doesn’t mean

... (read more)

[-]Mo Putera4mo20

From John Nerst's All the World’s a Trading Zone, and All the Languages Merely Pidgins:

Trading Zone: Coordinating Action and Belief, by Peter Galison discusses how physicists with different specialties live in different (social) realities and what happens when they interact.
(Galison’s article is worth reading in full, it’s wonderful erisology — a synthesis of two models of scientific progress: incremental empiricism (of the logical positivists) and grand paradigm shifts (of Thomas Kuhn and others).)
Experimentalists, theorists and instrument makers are all

... (read more)

[-]Mo Putera6mo20

Scott's The Colors Of Her Coat is the best writing I've read by him in a long while. Quoting this part in particular as a self-reminder and bulwark against the faux-sophisticated world-weariness I sometimes slip into:

Chesterton’s answer to the semantic apocalypse is to will yourself out of it. If you can’t enjoy My Neighbor Totoro after seeing too many Ghiblified photos, that’s a skill issue. Keep watching sunsets until each one becomes as beautiful as the first...
If you insist that anything too common, anything come by too cheaply, must be bor

... (read more)

[-]Mo Putera7mo10

I find both the views below compellingly argued in the abstract, despite being diametrically opposed, and I wonder which one will turn out to be the case and how I could tell, or alternatively if I were betting on one view over another, how should I crystallise the bet(s).

One is exemplified by what Jason Crawford wrote here:

The acceleration of material progress has always concerned critics who fear that we will fail to keep up with the pace of change. Alvin Toffler, in a 1965 essay that coined the term “future shock,” wrote:
I believe that most human beings

... (read more)

[-]Mo Putera7mo10

Some ongoing efforts to mechanize mathematical taste, described by Adam Marblestone in Automating Math:

Yoshua Bengio, one of the “fathers” of deep learning, thinks we might be able to use information theory to capture something about what makes a mathematical conjecture “interesting.” Part of the idea is that such conjectures compress large amounts of information about the body of mathematical knowledge into a small number of short, compact statements. If AI could optimize for some notion of “explanatory power” (roughly, how vast a range of disparate knowl

... (read more)

[-]Mo Putera7mo10

How to quantify how much impact being smarter makes? This is too big a question and there are many more interesting ways to answer it than the following, but computer chess is interesting in this context because it lets you quantify compute vs win probability, which seems like one way to narrowly proxy the original question. Laskos did an interesting test in 2013 with Houdini 3 by playing a large number of games on 2x nodes vs 1x nodes per move level and computing p(win | "100% smarter"). The win probability gain above chance i.e. 50% drops from +35.1% in ... (read more)

3gwern7mo

The diminishing returns isn't too surprising, because you are holding the model size fixed (whatever that is for Houdini 3), and the search sigmoids hard. Hence, diminishing returns as you jump well past the initial few searches with the largest gains, to large search budgets like 2k vs 4k (and higher). This is not necessarily related to 'approaching perfection', because you can see the sigmoid of the search budget even with weak models very far from the known oracle performance (as well as stronger models); for example, NNs playing Hex: https://arxiv.org/pdf/2104.03113#page=5 Since it's a sigmoid, at a certain point, your returns will steeply diminish and indeed start to look like a flat line and a mere 2x increase in search budget does little. This is why you cannot simply replace larger models with small models that you search the hell out of: because you hit that sigmoid where improvement basically stops happening. At that point, you need a smarter model, which can make intrinsically better choices about where to explore, and isn't trapped dumping endless searches into its own blind spots & errors. (At least, that's how I think of it qualitatively: the sigmoiding happens because of 'unknown unknowns', where the model can't see a key error it made somewhere along the way, and so almost all searches increasingly explore dead branches that a better model would've discarded immediately in favor of the true branch. Maybe you can think of very large search budgets applied to a weak model as the weak model 'approaching perfection... of its errors'? In the spirit of the old Dijkstra quip, 'a mistake carried through to perfection'. Remember, no matter how deeply you search, your opponent still gets to choose his move, and you don't; and what you predict may not be what he will select.) Fortunately, 'when making an axe handle with an axe, the model is indeed near at hand', and a weak model which has been 'policy-improved' by search is, for that one datapoint, equivalen

1Mo Putera7mo

Thanks, I especially appreciate that NNs playing Hex paper; Figure 8 in particular amazes me in illustrating how much more quickly perf. vs test-time compute sigmoids than I anticipated even after reading your comment. I'm guessing https://www.gwern.net/ has papers with the analogue of Fig 8 for smarter models, in which case it's time to go rummaging around...

[-]Mo Putera8mo10

Just reread Scott Aaronson's We Are the God of the Gaps (a little poem) from 2022:

When the machines outperform us on every goal for which performance can be quantified,
When the machines outpredict us on all events whose probabilities are meaningful,
When they not only prove better theorems and build better bridges, but write better Shakespeare than Shakespeare and better Beatles than the Beatles,
All that will be left to us is the ill-defined and unquantifiable,
The interstices of Knightian uncertainty in the world,
The utility functions that no one has yet wr

... (read more)

[-]Mo Putera6mo*00

Lee Billings' book Five Billion Years of Solitude has the following poetic passage on deep time that's stuck with me ever since I read it in Paul Gilster's post:

Deep time is something that even geologists and their generalist peers, the earth and planetary scientists, can never fully grow accustomed to.
The sight of a fossilized form, perhaps the outline of a trilobite, a leaf, or a saurian footfall can still send a shiver through their bones, or excavate a trembling hollow in the chest that breath cannot fill. They can measure celestial motions and l

... (read more)

[+][comment deleted]3mo20

Moderation Log