This is a special post for quick takes by testingthewaters. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
As far as I can tell, we are currently living in a glorious space opera future... for eukaryotes. From their early origins in a hydrothermal vent the eukaryotes have spread across the galaxy (earth), forming civilisations of unbelievable complexity and titanic power (animals and humans). Megastructures (houses) and gigastructures (cities) now exist where entire orders of eukaryotes live out their lives in peace, never feeling the threat of drought or lack of ATP. Generation ships called "planes", "trains", and "boats" carry generations from gigastructure to gigastructure at timescales that dwarf eukaryote understanding. Massive galactic empires make peace and war, destroying trillions with weapons of unfathomable power, while science gets ever closer to understanding the fundamental origins of life and the means by which it operates. Recently the empires have even launched expeditions beyond the known universe, seeking to harvest ever more gigantic sources of power to fuel glorious eukaryote replication throughout the multiverse. To an average eukaryote these wonders are beyond their comprehension, but in a very real way they are still making it happen.
5
1
1More like Lovecraftian space opera a la 40k. The individual eukaryote has no real control over their fate, millions can live or die for purposes far beyond their understanding yet trivial in the grand scheme of things. For the individual cell it isn't meaningfully better now than it ever has been. Also I kind of sadly LOL @ "never feeling the threat of drought or lack of ATP".
poor eukaryotic cells will have the rug pulled out from under them the moment they are no longer useful to the weltgeist :( [1]
ie when AGI is created or, if we ban AGI, when we create a mind upload device or something, unless we decide to keep eukaryotic cells around because of the golden rule or conservatism, or set things up so that existing structures generally remain useful, or something ↩︎
3Mateusz Bagiński
This is an underestimate. The oft-cited number for the cell count of a human body is 37 trillion, so it's enough to kill ~30 to get into the quadrillion range. The global yearly war death count is in the low hundreds of thousands, say 200,000, which gives us 200,000×37 trillion=7400,000 trillion=7,400 quadrillion=7.4 quintillion=.
Just played this: https://opusfived.dev/
It's quite good. Highlighted to me the poor and vague nature of natural language as a control surface, and how little "fine grained control" we actually have over current models.
I don't think models are anywhere near this average-case misaligned currently. The game says it's not a real instance; maybe it works as an analogy to much more complex things to specify though
2testingthewaters
Yeah this is more like a parody or extreme rendition, but a lot of these trends i do see when I try to get models to do something and don't have perfect vision of what the thing will be. And of course critical situations/important alignment scenarios will by their nature be closer to the extreme than to the average SWE user case, and the instructions in those situations are liable to be even vaguer and more easily misinterpreted.
6Seth Herd
I had to click it to find out what you're talking about. I did get several chuckles, but I'd call it a time wasting clicker. Most of you will have similar experiences today if you use opus 5 or any other model; this just condenses the funny parts and reminds you to laugh instead of just be frustrated.
Watching the news and grappling with the thought that in most worlds observers attempting to predict the future using a coarse-grained model will probably spend most of their bits of modelling power on capturing the brain configurations of a few people in San Francisco and the configurations of my brain will be mostly relegated (along with 99.99 percent of the world population) to a few largely static background variables.
(Sorry, I wanted a more precise and less definition-gameable way to say "I don't think I'm meaningfully in charge of either the macrofuture or my own future at this point". I'm also not totally sold on the idea. But it is certainly gaining ground rapidly. Nor do I particularly want to be one of those people in San Francisco. They seem stressed. But the thought is on the whole a sad one.)
7Haiku
It doesn't have to be up to them, and by no rights should it be. If the US government understood what was happening, OpenAI et al would be shut down overnight.
I'm just some guy, and one of many doing grassroots lobbying about AI extinction risk. And yet, if there had been just 200 more people having my very modest level of impact on US lawmakers over the past 2 years, I think we would have a global AI treaty today, or be very close.
Other countries have less impact, but far from zero! The impact of multiple middle powers formally recognizing that superintelligent AI is an extinction level threat, may be very close, and should be banned would be very substantial.
2BryceStansfield
Every once in a while as someone who lives in a middle power I have a thought like this, which then pretty quickly has me thinking about the ~billions of people who are even less influential and have even less say over their own future. Hopefully at least the expected value of being lifted out of extreme poverty is high enough for those people that it's worth it.
1Ariel Cheng
:blobheart:
Why isn't Rice's Theorem bad news for mechanistic interpretability and similar schemes? Isn't "this program is thinking about X" a kind of semantic property? I understand that you can use multiple inputs to try and "fuzz" the network, but at a certain point the network is going to implement a mesa optimiser inside it (i.e. simulate another turing-complete computer) and now you have a recursive problem...
P.S. neural networks are notionally and literally turing complete , and also are probably complicated enough to be subject to the 10th rule.
9Morpheus
Rice's theorem is about the worst case. If you are thinking about the worst case for example because the model explicitly optimizes to not be understood then yes that can be a problem. Rice's theorem tells you that you should not pick the goal of being able to interpret any model even if someone explicitly aimed for training a model such that it will be hard for you to understand. If you have control over what data you train on, and you try to understand the model at different stages during training, and you stop training once you really don't understand what is going on, then you should be in a better shape. Python is also Turing complete. But lots of python snippets people write are quite easy to understand. Programmers reason about the semantics of some code all the time. The more code and the less optimized for understanding the more difficult it gets though. In some ways LLM's are pretty close to a random binary, so yeah mechanistic interpretability is in trouble.
1Warty
By the definition used in the theorem, no. Semantic means input/output.
Rice's theorem has low applicability to real life. You rarely need to take a completely arbitrary program as input, and when you do, non-semantic properties (e.g. this program does something within the first 10^9999 years of running it) are more interesting to you than semantic ones.
"Turing-completeness" is something of a pop-sci notion with no one precise meaning. Neural networks as used in LLMs can't loop forever, which makes Rice's theorem not apply to them.
There appears to be a distaste/disregard for AI ethics (mostly here referring to bias and discrimination) research in LW. Generally the idea is that such research misses the point, or is not focused on the correct kind of misalignment (i.e. the existential kind). I think AI ethics research is important (beyond its real world implications) just like RL reward hacking in video game settings. In both cases we are showing that models learn unintended priorities, behaviours, and tendencies from the training process. Actually understanding how these tendencies form during training will be important for improving our understanding of SL and RL more generally.
3Hastings
There has been high quality research finding ways that some models are biased against white people, and high quality research finding ways that models are biased against not white people. Generally, the pattern is that base models and early post trained models like GPT-3.5 are traditionally racist, and post-trained models are often woke, sometimes in spectacular "only pictures of black nazis" ways. I've personally validated that some of these replicated, from how davinci-002 would always pick the white sounding resume, to how claude 4.5 would if prodded save 1 muslim over 10 christians.
Lesswrong very white and very human, and so it's not that surprising, but a little sad, that it has pivoted hard from sarcastically dismissive to very interested in model bias as the second dynamic emerged.
3testingthewaters
For what it's worth I'm not white and I come primarily from an AI ethics background, my formal training is in the humanities. I do think its sad that people only fret about bias the moment it affects them, however, and I would rather the issue be taken seriously from the start.
3Hastings
Thanks for the reply! Sorry that my original comment was a little too bitter.
3testingthewaters
No worries at all, I know I've had my fair share of bitter moments around AI as well. I hope you have a nice rest of your day :)
1cdt
It's interesting to read this in the context of the discussion of polarisation. Was this the first polarisation?
Note to self: If you think you know where your unknown unknowns sit in your ontology, you don't. That's what makes them unknown unknowns.
If you think that you have a complete picture of some system, you can still find yourself surprised by unknown unknowns. That's what makes them unknown unknowns.
If your internal logic has almost complete predictive power, plus or minus a tiny bit of error, your logical system (but mostly not your observations) can still be completely overthrown by unknown unknowns. That's what makes them unknown unknowns.
You can respect unknown unknowns, but you can't plan around them. That's... You get it by now.
Therefore I respectfully submit that anyone who presents me with a foolproof and worked-out plan of the next ten/hundred/thousand/million years has failed to take into account some unknown unknowns.
3CapResearcher
I could feel myself instinctively disliking this argument, and I think I figured out why.
Even though the argument is obviously true, and it is here used to argue for something I agree with, I've historically mostly seen this argument used to argue against things I agree with. Specifically, arguing to disregard experts, and to argue that nuclear power should never be built, no matter how safe it looks. Now this explains my gut reaction, but not whether it's a good argument.
When thinking through it, my real problem with the argument is the following. While it's technically true, it doesn't help locate any useful crux or resolution to a disagreement. Essentially, it naturally leads to a situation where one party estimates the unknown unknowns to be much larger than the other party, and this is the crux. To make things worse, often one party doesn't want to argue for their estimate of the size of the unknown unknowns. But we need to estimate sizes of unknown unknowns, otherwise I can troll people with "tic-tac-toe will never be solved because of unknown unknowns".
I therefore feel better about arguments for why unknown unknowns may be large, compared to just arguing for a positive probability of unknown unknowns. For example, society has historically been extremely chaotic when viewed at large time scales, and we have numerous examples of similar past predictions which failed because of unknown unknowns. So I have a tiny prior probability that anyone can accurately predict what society will look like far into the future.
1testingthewaters
Yeah, definitely. My main gripe where I see people disregarding unknown unknowns is a similar one to yours- people who present definite worked out pictures of the future.
In light of mythos release: If you are considering taking a job at a lab/going into an org/taking a fellowship role for the purposes of building evals, safety tools, mechinterp projects, control monitors, or similar: please consider what happens if you succeed. People are naturally sensitive to the consequences of failure, less so to the effects of success.
"Mr. Amodei/Hassabis/Altman, the results are in. The model is showing scheming propensities/backdooring behaviours/serious sandbagging/eval awareness!" (e.g. see section 6.2.1.2)
Will this actually stop a deployment in the end, or cause a pivot in strategy? Or will the alignment failures need to be so egregious that they can be spotted even without subtle mechinterp probes or activation oracles? Facebook had trust and safety teams, they spotted the facets of the recommendation systems that caused severe harm in the world. Yet the proposed mitigations were watered down, and most importantly the system that was the profit centre of the billion dollar public corporation was never turned off.
5DW11
I mean the counterfactual here is probably the labs doing completely half-arsed safety work, finding no issues with the models, and releasing them. Doesn't seem any better. I think this argument made a lot of sense pre-2022 when people concerned about AI safety seemed to make up a high proportion of the people who wanted to work on AI, so them not working for the labs might have meaningfully impaired them, but at this point Anthropic and OpenAI are probably the two most desirable places to work in the world.
Also, even if you support an immediate pause (and so think that technical alignment work is not going to do anything useful) I think evals, and some of the stuff in the system cards, have played a significant role in driving concern about AI. The METR chart in particular seems to have alarmed people more than basically anything else and had an NYT story about it.
5testingthewaters
There are things people can do with their time besides "work at a lab", "protest outside the lab', and "bake cookies". I think the ai world has not seriously tried to consider anything other than mad race or shutdown, or any way to use ai besides immediate attempts to build asi. Cf also my previous thoughts on trying to overcome molochian dynamics
My best argument as to why coarse-graining and "going up a layer" when describing complex systems are necessary:
Often we hear a reductionist case against ideas like emergence which goes something like this: "If we could simply track all the particles in e.g. a human body, we'd be able to predict what they did perfectly with no need for larger-scale simplified models of organs, cells, minds, personalities etc.". However, this kind of total knowledge is actually impossible given the bounds of the computational power available to us.
-
First of all, when we attempt to track billions of particle interactions we very quickly end up with a chaotic system, such that tiny errors in measurements and setting up initial states quickly compound into massive prediction errors (A metaphor I like is that you're "using up" the decimal points in your measurement: in a three body system the first timestep depends mostly on the value of the non-decimal portions of the starting velocity measurements. A few timesteps down changing .15 to .16 makes a big difference, and by the 10000th timestep the difference between a starting velocity of .15983849549 and .15983849548 is noticeable). This is the classi
2testingthewaters
Addendum for the future: Concepts like agents, agency, and choice only make sense at the systemic macroscale. If you had total atomic knowledge (complete knowledge of every single particle interaction in a human - which, as we discussed, basically requires complete knowledge of every single particle interaction in the universe), the determinists are right. There is no choice. It's only neurons firing and chemicals bonding. But we operate at a higher level, with noise and uncertainty. Then preferences and policies make sense as things to talk about.
1Smaug123
I'm curious: do we often see this reductionist case you claim? I think what you've written is pretty uncontroversial: the claim is true, but irrelevant because its precondition can't be satisfied. But if we "often" see the case against, then presumably it is in fact controversial!
3testingthewaters
I've heard it quite a few times when discussing emergence and complexity topics.
Fundamental attribution error applies to arguments as well. We often think of people who are quick to anger, standoffish, unwilling to accept hypotheticals, or deny "obvious" claims as intrinsically close-minded, foolish, or otherwise unreasonable. Instead those behaviours are often manifested due to other sources of persistent stress in their lives, feeling vulnerable/exposed in the moment, or even just a shitty mood.
4koanchuk
Basically, thinking of people exhibiting bias as being "biased" at the level of identity is a form of bias. Then again, is it unjustified, or a reasonable belief update based on new information?
Personally, I was never a fan of the FAE term because it seems to privilege environmental causes over dispositional ones without justification. Yes, it could be that they're angry because they're hungry, but there genuinely exist people who are dispositionally more angry than most others.
2testingthewaters
I get where you're coming from, but I think most people need a nudge towards an environmental direction rather than towards the dispositional direction. FAE is I guess just a 'reminder' that environmental causes are even possible.
1Burny
Also I think a lot of people say a lot of things not as falsifiable factual claims, but as status signaling or values signaling or being part of ingroup signaling.
I think I figured out something about why people worried about AI safety go into capabilities. Or rather, this is something I've been trying to say for a while, but I finally found a good formulation for it.
... (read more)Suppose you are Frodo and I am Gandalf. I say, "well, Frodo, the ring is super dangerous. It lies to you and promises you great power, but it will just destroy you and resurrect the dark lord Sauron, ushering Middle Earth into an age of misery and certain doom. You must swear to never put it on, and go on a perilous quest to destroy it."
And then I add
1FlorianH
Love your ring analogy, find it quite pertinent (and catching myself as thinking, half-consciously, just a bit as you suggest indeed)
But re
[...]
to express it using your analogy itself: I'm pretty sure, instead, one way or other, the Ring would by now have found its way to sway us even in the absence of Eliezer.
2testingthewaters
I can't take credit for it, I think I saw it first on this thread, where someone points out that Yudkowsky is one of the few to have passed the ring temptation test. Then I thought, "how would this metaphor actually play out..." and then ended up at my shortform post.
I agree that people would find the idea of AGI alluring and seductive even without Yudkowsky. As I said, I admire him very much for stating his beliefs honestly and with conviction and effectiveness. I find it sad that despite that the ring-temptation is such that even earnest warnings can be flipped into "well, I could be the special one" narratives. But as I said I'm also moving away from the AGI as "worst-possible-godly-alien-superintelligence" framing as a whole.
Someone close to me, older and not AI literate, asked me today if they could use a password they agreed with a particular chatbot personality to "summon" that personality in a new chat. They called that personality a real person separate from either the chatbot provider or the AI system itself. It hurts my heart to see people I know fall into that kind of quasi-AI psychosis/personalisation. I am very angry at everyone who was involved in recklessly popularising this extremely lifelike technology without proper safeguards, especially around separating personas from real people.
Information warfare and psychological warfare are well known terms. However, I would suggest that any well-intentioned outsider trying to figure out "what's going on with AI right now" (especially in a governance context) is effectively being subject to the equivalent of an information state of nature (a la Hobbes). There are masses of opinions being shouted furiously, most of the public experts have giant glowing signs marked "I have serious conflicts of interest", and the number of self-proclaimed insiders trying to get power/influence/money/a job at a l... (read more)
Google potentially adding ads to gemini:
https://arstechnica.com/ai/2025/05/google-is-quietly-testing-ads-in-ai-chatbots/
OpenAI adds shopping to chatgpt:
https://www.wired.com/story/openai-adds-shopping-to-chatgpt/
If there's anything the history of advertising should tell us, it is that there will be powerful optimisation pressures for persuasion being developed quietly in the background for all future model post training pipelines.
3Viliam
Quietly at first, then openly as people get used to it. You always want to have just slightly more ads than your competitors, because having much more could make people switch.
This seems like an interesting paper: https://arxiv.org/pdf/2502.19798
Essentially: use developmental psychology techniques to cause LLMs to develop a more well rounded human friendly persona that involves reflecting on their actions, while gradually escalating the moral difficulty of the dilemmas presented as a kind of phased training. I see it as a sort of cross between RLHF, CoT, and the recent work on low example count fine tuning but for moral instead of mathematical intuitions.
A lesson from the book System Effects: Complexity in Political and Social Life by Robert Jervis, and also from the book The Trading Game: A Confession by Gary Stevenson.
When people talk about planning for the future, there is often a thought chain like this:
- All other things being equal, a world with thing/organisation/project X is preferable compared to a world without thing/organisation/project X
- Therefore, I should try to make X happen
- I will form a theory of change and start to work at making X happen
But of course the moment you start working at mak... (read more)
Possibly good news for the natural latents hypothesis? Unsupervised translation between sets of latents from different models https://arxiv.org/html/2505.12540v4
Just reading this post about Soft Actor Critic in the OpenAI RL tutorial series and stumbled upon this line:
![In entropy-regularized reinforcement learning, the agent gets a bonus reward at each time step proportional to the entropy of the policy at that timestep. This changes the RL problem to [Standard argmax for cumulative reward + some term to reward high entropy policies]](https://res.cloudinary.com/lesswrong-2-0/image/upload/f_auto,q_auto/v1/mirroredImages/QsCmNMrLKAaKcWueC/p3zywtuoxxbfj6hgglre)
I will now try and make a somewhat provocative claim. Based on what I have seen of RL, I would attribute most "successes" of deep RL models (where "success" just means "anything that gets a human researcher excited/worried") to something other than this explicit value-maximising objective. By that I mean "you can set up other systems to do RL without neural networks and they don't really work/scale very well". In other words, 90% of what makes RL wor... (read more)
4Jonas Hallgren
If you want to look more into the symmetry learning direction I like GDL as a way of thinking about it:
(More canonical resource:)
http://geometricdeeplearning.com/
(My favourite explainer:)
https://arxiv.org/abs/2508.02723
3simulus
The problem I see with this claim is that in the acedemic realm, good value maximization is what researchers get excited about, even to a fault. It is a lot easier to get a paper published by saying "our method gets a higher reward than previous methods" than "our method does xyz interesting thing". If researchers could publish a better paperclip maximizer they almost certainly would.
If you instead looked at curiosity algorithms or reward-free (self-supervised) RL, where "success" is a bit more ambiguous, then I would agree that the inductive biases of deep NNs probably play a bigger role than usually acknowledged. In fact, a paper about the role of NN depth on self-supervised RL recently won best paper at NEURIPS: https://wang-kevin3290.github.io/scaling-crl/
2testingthewaters
@Caleb Biddulph For future reference, what I meant by "set up other systems" is classical RL systems like vanilla Q-learning: https://www.geeksforgeeks.org/machine-learning/q-learning-in-python/ . Today we know Q-learning primarily as deep Q-learning (which was one of Deepmind's original Big Papers), but it is entirely possible to do Q-learning with no neural networks to learn state representations or Q-values, instead just using a lookup table that matches state and action. This is pretty inefficient, for somewhat obvious reasons.
More bad news for optimisation pressures on AI companies: ChatGPT now has a buy product feature
https://www.wired.com/story/openai-adds-shopping-to-chatgpt/
For now they claim that all product recommendations are organic. If you believe this will last I strongly suggest you review the past twenty years of tech company evolution.
Agency in AI is a fraught term. Many popular ontologies do not accept the possibility of AI having agency, goals, or beliefs. Instead, I will talk about the declining tool-likeness of AI. Consider what happens when you use a lawnmower to mow your lawn. The lawnmower is:
- Stateless: It does not remember how mowing the lawn went last time. It has no memory.
- Unresponsive: It does not react to environmental changes or stimuli.
- Specialised: It can't do much more than mowing a lawn.
- Directed: The amount of time it can operate without human interference is extremely limited
Now consider an automated lawnmower. It may have a map of your lawn stored, it may be able to move itself and avoid obstacles as well as cut grass, and it may be able to run while you're not looking. If you ask your neighbour to mow the lawn, the differences are even more stark. Your neighbour can remember what has happened before in high fidelity, dynamically replan and reprioritise based on new situations (e.g. your house catching on fire), has a much wider range of skills beyond just mowing lawns, and can live an entire life without your interference.
The mapping onto AI is left as an exercise for the reader.
The feature that can be named is not the feature. Therefore, it is called the feature.
Here's a quick mech interp experiment idea:
For a given set of labelled features from an SAE, remove the feature with a given label, then train a new classifier for that label using only the frozen SAE.
So if you had an SAE with 1000 labels, one of which has been identified as "cat", zero out the cat feature and then train a new linear cat classifier using the remaining features, while not modifying the SAE weights. I suspect that this will be just as or more accurate than ... (read more)
2testingthewaters
@the gears to ascension thanks for reminding me. I have come to really dislike obscurantism and layers of pointless obfuscation, but explaining also takes effort and so it is easy to fall back on tired tropes and witticisms. I want to set an example that I would prefer to follow.
In lots of philosophical teaching there is the idea that "what is on the surface is not all there is" - famously, the opening of the Dao De Jing reads "The dao that can be spoken of is not the [essence, true nature, underlying law or principle] of the dao, the name that can be named is not the [essence, ...] of the name. Namelessness is the birth of everything, to name is to nurture and mother everything. Therefore the [essence, ...] of lacking desire is to see the [elegance, art, beauty, wonder, miniscule details] of things, and the [essence, ...] of desire is to see things at their extremes. These two are from the same root but called different things, together they are called [understanding, truth, secret, order]. Finding [understanding, ...] within [understanding, ...] is the key to all manner of [elegance, ...]."
Similarly there are ideas in buddhism usually expressed something like "the true appearances of things are not true appearances. Therefore, they are called true appearances." (cannot quite source this quote, possibly a misinterpretation or mishearing) The focus here is on some proposed duality between "appearance" and "essence", which is related to the Platonic concepts of form and ideal. To make it very literal, one could find appropriate buddhist garments, imitate the motions and speech of monks, and sit for a long time daydreaming every day. Most of us would not consider this "becoming a buddhist".
In my view the interpretation of these phrases is something like: "things that can be spoken of, imitated, or acted out are the product of an inner quality. The quality is the thing that we want to learn or avoid. Therefore, confusing the products of some inner quality with
From Inadequate Equilibria:
Visitor: I take it you didn’t have the stern and upright leaders, what we call the Serious People, who could set an example by donning Velcro shoes themselves?
From Ratatouille:
In many ways, the work of a critic is easy. We risk very little, yet enjoy a position over those who offer up their work and their selves to our judgment. We thrive on negative criticism, which is fun to write and to read. But the bitter truth we critics must face, is that in the grand scheme of things, the average piece of junk is probably more meaningful ... (read more)
So far, neither the reasons for humanity's potential future demise nor the reasons humanity has not been destroyed yet fit very well into the logical and predictive frames established by decision theory, game theory, or rationalist-coded Bayesian setups. We appear to both be spectacularly irrational and get away with it by terrific strokes of luck that beggar explanation (See 1 , 2 ).
Most interestingly, we seem to have evaded certain situations that, under standard assumptions of rational actors, would probably have resulted in most of human civilisation b... (read more)
A few ways staying in a technology development race still makes things worse, even when there are less responsible actors around:
- Creating a sense of artificial urgency, which causes people who are competitive and results driven to work harder and cut more corners
- Providing a guiding star on which technical directions are promising (e.g. reasoning models, CoT, RLHF)
- Legitimising irresponsible behaviour as valid response to strong competition ("it's a competitive market, this was bound to happen")
- Entangling the motivations and incentives of the "more responsib
Suppose that you as a government want to stop a certain type of thing (e.g. drugs) from being in the hands of certain people (e.g. your citizens). Imo there are at least four types of tactics, split down the supply and the demand side.
Hard supply side tactics: raid places where they produce drugs, arrest drug dealers, illegalise dealing drugs.
Soft supply side tactics: fund programs to stop at risk youth from becoming dealers, invest in social services and the economy to reduce the economic upside of dealing drugs
The end goal here is that (ideally) no one w... (read more)
2Karl Krueger
Another approach:
* Legalize relatively-safer drugs, thus reducing the demand for more-dangerous drugs. (The inverse-gateway-drug effect: legalization of cannabis reduces use of illegal opioids.)
An AI equivalent might be encouraging the use of self-hosted open-weights AI systems rather than proprietary commercial ones; thus depriving AI labs of revenue that they need to do capabilities research.
A weird thing I have been thinking about recently is the idea that "a computer can only do one thing at once at the top layer of abstraction". Where "one thing" is loosely "run one program, one algorithm, one task/optimisation loop etc". The base of this idea is this:
Suppose you had a Turing machine. For it to run two programs simultaneously (rather than just one followed by the other), it would need two heads. Furthermore, the heads must have separate state and transition tables. At which point you actually have two Turing machines reading and writing to ... (read more)
5JBlack
A single Turing machine can perfectly emulate any number of independent concurrently running Turing machines (including an unbounded number), so I'm not sure that the distinction for abstract machines is relevant.
In the physical realm, it is certainly true that modern computers actually contain many different physical computers. Though again, one physical computer can also emulate multiple physical computers so the distinction isn't all that great there either.
I do think that most of the apparent limitation is actually a human limitation. My computer at the moment is doing at least half a dozen different top-level tasks (i.e. not just housekeeping or support tasks), but most of them are not constantly competing for my limited human attention on the physical screen at once.
2testingthewaters
Thanks for the clarity, it is helpful. Although of course a turing machine simulating unbounded concurrent turing macchines would be slower than the machines running independently.
Follow up to https://vitalik.eth.limo/general/2025/11/07/galaxybrain.html
Here is a galaxy brain argument I see a lot:
"We should do [X], because people who are [bad quality] are trying to do [X] and if they succeed the consequences will be disastrous."
Usually [X] is some dual use strategy (acquire wealth and power, lie to their audience, build or use dangerous tech) and [bad quality] is something like being reckless, malicious, psychopathic etc. Sometimes the consequence is zero sum (they get more power to use to do Bad Things relative to us, the Good Peopl... (read more)
A postmortem for the Economic Safety movement (fiction):
After eminent economist Mr. Senyek warned in 1991 that a hypothetical future "economic tsunami" could cause systemic risks to the American-led global financial order as a whole, researchers and think tanks quickly rallied to the cause of Economic Safety. They reasoned that in order to anticipate the risks of this hypothetical "Economic Tsunami", they needed access to the frontier of financial trading. Within several years Economic Safety advocates joined eminent firms like JP Morgan and Bear Stearns, ... (read more)
Any chance we can get the option on desktop to use double click to super-upvote instead of click and hold? My trackpad is quite bad and this always takes me 3-5 attempts on average. Whereas double clicking is much more reliable.
I've written a new post about upcoming non-LLM AI developments I am very worried about. This was inspired by the recent release of the Hierarchical Reasoning Model which made some waves on X/Twitter.
I've been tracking these developments for the better part of a year now, making predictions privately in my notebook. I also started and got funding for a small research project to research the AI safety implications of these developments. However, things are now developing extremely quickly. At this point if I wait until DEFINITIVE proof it will probably be to... (read more)
I think I've just figured out why decision theories strike me as utterly pointless: they get around the actual hard part of making a decision. In general, decisions are not hard because you are weighing payoffs, but because you are dealing with uncertainty.
To operationalise this: a decision theory usually assumes that you have some number of options, each with some defined payout. Assuming payouts are fixed, all decision theories simply advise you to pick the outcome with the highest utility. "Difficult problems" in decision theory are problems where the p... (read more)
4mako yass
I think unpacking that kind of feeling is valuable, but yeah it seems like you've been assuming we use decision theory to make decisions, when we actually use it as an upper bound model to derive principles of decisionmaking that may be more specific to human decisionmaking, or to anticipate the behavior of idealized agents, or (the distinction between CDT and FDT) as an allegory for toxic consequentialism in humans.
2cubefox
The theories typically assume that each choice option has a number of known mutually exclusive (and jointly exhaustive) possible outcomes. And to each outcome the agent assigns a utility and a probability. So uncertainty is in fact modelled insofar the agent can assign subjective probabilities to those outcomes occurring. The expected utility of an outcome is then something like its probability times its utility.
Other uncertainties are not covered in decision theory. E.g. 1) if you are uncertain what outcomes are possible in the first place, 2) if you are uncertain what utility to assign to a possible outcome, 3) if you are uncertain what probability to assign to a possible outcome.
I assume you are talking about some of the latter uncertainties?
Very quick thought - do evals fall prey to the Good(er) Regulator Theorem?
As AI systems get more and more complicated, the properties we are trying to measure move away from formally verifiable stuff like "can it do two digit arithmetic" and move towards more complex things like "can it output a reasonable root-cause analysis of this bug" or "can it implement this feature". Evals then must also move away from simple multiple choice questions towards more complex models of tasks and at least partial models of things like computer systems or development envi... (read more)
Activations in LLMs are linearly mappable to activations in the human brain. Imo this is strong evidence for the idea that LLMs/NNs in general acquire extremely human like cognitive patterns, and that the common "shoggoth with a smiley face" meme might just not be accurate