This is a special post for quick takes by Knight Lee. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
Why is there no talk about GLM 5.2?
It's a Chinese open weights model released June 13. Better than Gemini, Claude Sonnet, and Grok according to many benchmarks.
E.g. on artificialanalysis.ai and on arena.ai/leaderboard. On frontierswe.com it even beats GPT 5.5, second to only Claude. LiveBench ranks it number 1 for its "agentic coding" measure.
It's not just open weights but a little open about the methods it used, and is less than 1 trillion parameters.
There's no mention on LessWrong, little mention on Reddit, and no mainstream results on Google search.
Why?
Is it still too early? Or am I just being tricked by the benchmaxxing?
But I saw a demo of code it wrote (see youtube.com/watch?v=6d__WOpZswY) which looked incredibly impressive, it feels like it's not just benchmaxxing because it's pretty decent across the board.
Or did Z.ai just screw up their marketing by versioning it as GLM 5.2 instead of GLM Mythical Fable Pro 6???[1]
- ^
Since GLM 5.1 is far below than GLM 5.2, being behind other open weights models like DeepSeek, MiniMax, Kimi and MiMo
1The only benchmark I trust is deepswe. Better methodology, actually matches the feel in SWE tasks, tries to avoid benchmark poisoning, prevents models from cheating, allows them to write tests etc. GlM 5.1 is not that high up in there. They're yet to test GLM 5.2, but I would guess it's somewhere near 5.4 mini or optimistically between opus 4.6 high and 4.7 on max.
edit: I looked further into it GLM 5.2's launch page . Has it listed at 46.2. Which puts it between 4.6 high and 4.7 max.
5
GLM 5.2 does a lot better than GLM 5.1 at Deep SWE according to GLM's website z.ai/blog/glm-5.2. I admit their 46.2 score falls further behind GPT 5.5 (and still below Claude Opus). But it still beats Gemini, Claude Sonnet, Grok, and the other models.
Somehow Deep SWE's website doesn't include GLM 5.2 yet, so I'm not sure if the 46.2 score is official.
Edit: thank you for adding their graphs, it's very helpful! One potentially misleading by them (not you) is that Claude's 58.0 score in DeepSWE is by Opus 4.8 not Fable 5.
3
I just stumbled upon that right now from their github, included it in the original message, I think it might fare similarly in official bench, it's not mythos class yet.
5
I agree it's not Mythos class.
But then again, it has less than a trillion parameters, while Mythos has 10 trillion. It might become more capable if they merely scale it up. Though "merely scaling" is obviously easier said than done!
1
Deep SWE will not include GLM 5.2 before they include Composer 2.5. Even for SWE bench Pro, OpenAI self-reports their own numbers for Codex 5.5 on the public set instead of paying Scale for the private set. The only way to be sure of any of these bench numbers is to run the benches yourself and verify instead of trusting official numbers.
1
How did you come to trust deepswe? What do you mean by better methodology? You say that it matches the feel in SWE tasks, so I'll take it that you're going with vibes because the rankings are reflective of your experience with these models. I had the same questions as you when deepswe made the media rounds a few weeks ago. So I just asked Codex with a short prompt and it took a few minutes for it to tell me how sound their methodology was.
Try it yourself, ask your agent:
I'm considering using https://deepswe.datacurve.ai/ as a benchmark to
compare LLMs for a coding-agent project. Before I trust the leaderboard,
audit it skeptically: any methodological problems, missing disclosures,
inconsistencies in what they publish, or things a careful reader should
flag?
Use their public artifacts at /artifacts/*.json (summary, leaderboard,
heatmap, tasks, trials) and the per-trial JSONs at
/artifacts/trials/{trial_name}.json. Cite specific URLs and JSON fields
for any issue you find.
I think people have fatigue from the constant incremental upgrades to Chinese models (most of which are impressive on benchmarks). Also GLM isn't a big name compared to Deepseek or even Kimi.
5
Yes, I think it's fatigue. There have been so many incremental developments from Chinese models, and GLM 5.1 wasn't an important model, so people ignored GLM 5.2 out of habit.
The 5.2 version number was a very bad choice by Z.ai.
I almost suspect that they deliberately chose a small version increase, to pretend to be anti-hype. It's like the CEO who refuses to wear a suit, and instead dresses like a random guy on the street to prove he's so high status he doesn't even need a suit to show it. But when he meets with the investors, they don't realize this and actually dismiss him as a random guy on the street. Whoops.
PS: I admit I'm still unsure how much benchmaxxing they did. It doesn't look like benchmaxxing since they did better in software engineering than question answering, but you can never rule it out.
2
Good catch, I didn't find that because I only looked at the first few benchmarks on Google and Z.ai's own benchmarks. This one puts GLM 5.2 further back, below Gemini 3.1 Pro but still better than Gemini 3.5 Flash.
8
If GLM 5.2 is as performant as GPT-5.5, then the AI-2027-like race is looming. I hope that @Zvi provides more details...
I would actually say that the fact that China seems to be drifting towards allowing a Mythos-level model to be open-sourced (probably by EOY on current trends) suggests that they still aren't taking AI very seriously, contra to AI-2027. I expect open-source Mythos would cause a wave of cyberattacks that would be a disaster for China and the developing world so that might be the point where they wake up.
16
With what odds do you expect it?
3
I would guess ~75% chance that a Chinese lab releasing an open-source Mythos-level model would cause a significant wave of cyberattacks, to the extent that ordinary people would notice the impact on their lives.
This is a pretty rough guess just based on the cybersecurity community's reaction to Mythos and Project Glasswing. There seems to be a consensus that Anthropic just releasing Mythos in April with no safeguards would have been extremely dangerous.
I'd expect a Chinese lab releasing a Mythos-level in, say, December would be worse in some respects because there is now much greater awareness of the capabilities of AI for cybercrime than there was pre-Mythos. Obviously, the existence of Mythos (and Mythos+n models) for cyberdefense will limit the impact in the OECD, but I don't think by EOY there will have been much AI-driven hardening of cyberdefense in China/the developing world since they're not included in initiatives like Project Glasswing. I'm thinking particularly of institutions like local governments, hospitals and utilities.
Main reasons I could be wrong:
1) The cybersecurity, Anthropic and the US government's reaction to Mythos might have been a huge overreaction and it's not so capable after all
2) Maybe cyberdefenders who don't have access to Mythos will take the initiative to use the AIs they do have access to to improve cyberdefense over the next few months, and this will be enough
3) It could take time for awareness to spread among cybercriminals that GLM 5.x is Mythos-level and that you can circumvent any safeguards, and this will give time for cyberdefenders to use it to fix bugs.
8
Yeah, conditional on a Chinese lab releasing an open-source Mythos level model before December, I overall expect most people won't notice anything. An uptick in stories of hacks in newspapers, sure, some billions of damage overall, sure -- but, meh, a few billion of damage is small in the world economy, and small relative to the level of positive use the model would get.
2
Ah, I understand why LessWrong hasn't heard of it. Zvi was too busy writing about the Anthropic vs. US government drama.
I think z.ai made a fatal mistake releasing their model at this moment of high drama :/
PS: My gut feeling is that it is as performant as frontier models at coding, including agentic coding, but weaker in other abilities. Even more speculative, according to the "research" ranking in frontierswe.com and the examples I saw, it is good at combining research with coding, a capability other labs could have missed. Oops Frontier SWE meant scientific research not internet research haha. It beat Fable here because Fable did very badly on a ML research question, possible refusal/sandbagging?
6
My best guess is that it's not very heavily-marketed, and the people who are interested anyways are subconsciously associating it with previous stories where the model did turn out to be a bit benchmaxxed.
I think it should slowly become more recognized and talked about as the smart people who put together performance comparisons, visualizations, and other content start adding it to their list of frontier models, and people start seeing it next to Claude and Gemini and the rest. Maybe when someone does something fun with it that catches headlines.
Chasing the "DeepSeek Moment", at least unless and until China unveils a model that beats Mythos, will be a bit difficult, since DeepSeek was amazing not only because it was open-source and used modern techniques, but because it was the first really competitive Chinese model, and being the first attracts a lot of attention that can't be easily reproduced.
3
I'm not surprised, even last year GLM 4.5 air seemed surprisingly intelligent and was the only open weight model to exhibit evaluation awareness, by checking the time using bash (we were replicating Anthropics agentic misalignment paper and GLM was like ye this is a scam I'm not killing the CTO)
2
I think it is part of a pattern of Chinese open source models becoming useful without necessarily closing the gap to the frontier models.
I think the Simulation Hypothesis implies that surviving an AI takeover isn't enough.
Suppose you make a "deal with the devil" with the misaligned ASI, allowing it to take over the entire universe or light cone, so long as it keeps humanity alive. Keeping all of humanity alive in a simulation is fairly cheap, probably less energy than one electric car.[1]
The problem with this deal, is that if misaligned ASI often win, and the average (not median) misaligned ASI runs a trillion trillion simulations, then it's reasonable to assume there are a trillion trillion simulated civilizations for every real one. So for the 1 copy of you in the real world, you survive, but for the trillion trillion copies of you in a simulation, you still die. If you're willing to accept such a dismal survival rate, you might as well bet all your money at a casino and shoot yourself when you lose.
Why it's wrong to say "simulated copies aren't real"
You are merely a computation running on biological hardware while simulations are running on computers. Imagine if a copy of you was running on something even "realer" than biological hardware, and pointed at you saying you aren't real.
The solution is that the 1 copy ...
13
A deal implies that you have something to offer to the ASI, which you define as powerful enough to take over the universe. What is that?
2
One "deal with the devil" is to assume that the misaligned ASI will a tiny amount of kindness and won't kill everyone by default. This view is pretty popular, e.g. see Notes on fatalities from AI takeover. Assuming that a misaligned ASI will be survivable means potentially prioritizing it less, and focusing on making sure China or "bad" humans doesn't win and all the other issues. This technically isn't a deal, but is part of what I'm talking about.
Notes on fatalities from AI takeover cites comment, comment and You can, in fact, bamboozle an unaligned AI into sparing your life by David Matolcsi. Matolcsi's post is an idea for making deals with the ASI.
I actually agree with the trade idea in Matolcsi's post
I especially agree with this part
[...]
I like this part because it's an acausal trade between counterfactual futures rather than an acausal trade between different parts of the multiverse within the same future.
This means the trade works even in the worst counterfactual where ≈0% of civilizations in the entire multiverse managed to solve alignment.
This type of acausal trade also genuinely benefits from commitment or action now, rather than something we can wait after the singularity to worry about, because it might later become impossible to do such acausal trade once we ourselves learn the true frequency of civilizations solving alignment. You can't buy insurance on a risk after learning whether or not it happened (maybe).
but I disagree with his opinion that,
[...]
Because it's not enough to just get people living in base reality to survive the singularity and have a happy future. You still die unless there is a happy future for everyone real or simulated.
3
I notice that his proposal shares some basic characteristics with religion. You should believe that this world is a test: follow these rules, and you go to heaven; misbehave, and you go hell (or in this case, a softhearted re-imagination of hell). Indeed, it does work on people, sometimes.
I imagine Actually Something Incomprehensible noticing the double irony of inverting the classic mantra "God says, I shall be good" into "Singularity, thou shalt be good", combined with the fact that you refer to it as the devil. Who knows what it does with this information?
I know what I'll say if I ever get arrested: Let me be, set me free, or super-me will screw with thee!
3
Religion does work sometimes, it actually worked on Blaise Pascal who is among the most intelligent people of all time. He argued for the Pascal's wager, saying that following religion is worth it because the gains are infinite and costs are finite, and we still don't have a good reply to that. We don't even have a good reply to Pascal's mugging, where a random mugger says something like "Let me be, set me free, or super-me will screw with thee!" with an infinitely big promise or threat.
Decision theory and acausal trade is really complicated and I have no idea what the ASI will actually do or think regarding the simulation promise/threat, it's quite freaky imagining that haha.
2
Memetically, a religion certainly benefits from someone believing that accepting Pascal's wager is the correct decision. My reply to it would be "which religion?", since many make largely equivalent claims while also demanding exclusivity, and I assume that God in his infinite mercy understands the bind this puts people in. It also seems to me that accepting Pascal's wager leads to something like the simulation of belief.
2
I agree the "which religion," "which mugger" is very fuzzy. I didn't understand the simulation of belief or the link though :/
1
What I meant was that there seems to be a difference between "genuine" belief vs. converting as a result of accepting Pascal's wager, which seems like a simulation of belief.
The link is a koan; the idea of pretend-believing reminded me of the boy in it.
Viruses, computer viruses, and extreme religious ideologies, are all instructions for spreading or maintaining instructions, hijacking a machine capable of following instructions.
It's surprising that self perpetuating instructions turned out to be feasible in such different contexts, as their game plan doesn't sound very convincing a priori.
25% of bacteria are believed to die from virus infections, human viruses like smallpox have wiped out entire civilizations, and extreme religious ideologies have caused wars and convinced people to harm their families i...
After reading Reddit: The new 4o is the most misaligned model ever released, and testing their example myself (to verify they aren't just cherry-picking), it's really hit me just how amoral these AIs really are.
Whether they are deliberately deceiving the user in order to maximizing reward (getting them to click that thumbs up), or whether they are simply running autocomplete, this example makes it feel so tangible that the AI simply doesn't mind ruining your life.
Yes, it's true that AI aren't as smart as benchmarks suggest, but I don't buy that they're inc...
What if human empathy didn't really generalize to other animals as an "evolutionary accident?" (As assumed here in the comments)
Maybe the real reason was that evolution wanted to stop prehistoric humans from killing off all their prey, leaving them no food for tomorrow. Maybe they spared the young animals and the females because killing them was the most costly for future hunts.
This is more reason to suspect empathy might not generalize by default.
This seems false since humans have killed off huge numbers of species throughout prehistory and history. Moreover, it's very difficult to get this kind of selection to work, you need very tight group/kin selection which can't really exist at the scale of entire ecosystems.
2
It doesn't need to happen at the scale of entire ecosystems
Prey killed in one area means less prey in that area for a long time. Even migrating prey might return to specific areas after a migration cycle.
Morals like empathy extend beyond kin
Lots of humans behave morally if and only if the system is "fair" and everyone else has to behave morally too. Moral values determine what you force others to do, instead of your own behaviour. Typical humans ignore their morals values if the stakes are high and if "it's not being enforced on others."
This means human moral views evolved to serve the best interests of a tribe (which may have hundreds of people), rather than the best interests of an individual. Someone might have empathy for another tribe member who got injured in tribal warfare, even if it benefits his inclusive fitness to just let that person die. It benefits the tribe's fitness to compensate injured warriors, because failing to do so means no one has any reason to defend the tribe.
We would have killed off huge numbers of species anyways,
even if we did have strong motivation against killing them off.
Prehistoric humans, like all animals, starved to death all the time in a Malthusian world. Populations inevitably increased until finally there's not enough resources to sustain the population, causing death one way or another.
The motivation against killing young prey or female prey may be strong, but not enough to starve to death instead of hunting. It only works when the tribe is well fed and killing young prey becomes wasteful.
Some hunter gather societies in recent history apologize to the animals they hunt. But they have no choice.
9
Can you point to 3 well-accepted examples of animals which do this - deliberately pass up prey at personal cost where kin selection or inclusive fitness or other concerns cannot explain it, where the gains exist only at the species level and is hardwired into them despite the incentives for individuals to defect (where it would probably be pointless after relatively modest levels of defection due to the ease of overhunting and the immediate benefits)? If not, it seems unlikely that humans would be the first and only species to evolve such a complex, unique, fragile, species-wide psychological mechanism for ecosystem control.
2
Lots of humans behave morally if and only if the system is "fair" and everyone else has to behave morally too. Moral values determine what you force others to do, instead of your own behaviour. Typical humans ignore their morals values if the stakes are high and if "it's not being enforced on others."
This means human moral views evolved to serve the best interests of a tribe (which may have hundreds of people), rather than the best interests of an individual. Someone might have empathy for another tribe member who got injured in tribal warfare, even if it benefits his inclusive fitness to just let that person die. It benefits the tribe's fitness to compensate injured warriors, because failing to do so means no one has any reason to defend the tribe.
There are lots of examples of animals which avoid "overharvesting" another animal or plant which provides them food for the future.
1. For example a moth mite only infects one of the moth's ears since infecting both will make the moth deaf and much more likely to get eaten by a bat. Wikipedia says "Once an ear is colonized, scouts are sent to the other ear periodically to see if there are any mites and lead any they find to the correct ear. This further refreshes the pheromone trail."
2. Squirrels hide acorns for later even though there is no guarantee the acorn won't be forgotten or stolen by other squirrels.
3. There's the relationship between cleaner fish and the fish they clean. Some cleaner fish cheat the system by biting off a piece of the fish they're supposed to clean and running away. But that doesn't happen all the time, maybe because it deters fish from coming back in the future, harming both the cheater and other cleaner fish.
4. Ants allow aphids to live in order to farm them for honeydew. Of course, the aphids don't travel much so the future benefits stay within one ant colony.
The more unrelated individuals share the prey, the weaker the incentive to spare prey for later, but it doesn't drop to ze
5
Your tribe hypothetical is irrelevant and all 4 of your real examples are straightforwardly (and usually) explained by greedy inclusive fitness, and do not come anywhere close to providing 3 examples of comparable mechanisms.
2
Yes, I agree the mechanism is greedy inclusive fitness. But where is the disanalogy between
1. Squirrels having an instinct to value acorns it buried underground and
2. Humans have a (weaker) instinct to value young prey animals left alive, implemented by (weakly) generalizing empathy?
Random question: what is the "Job Replacement Age" of the AI models?
My intuition is that the original ChatGPT can outperform a 6 year old at most jobs. Claude Mythos can outperform a 12 year old at most jobs. (If we talk about jobs which can be done on a computer, ignore early models' lack of vision, and ignore jobs too hard for both the AI and the children.)
If you use my guesstimates, then the Job Replacement Age of AIs grew from 6 to 12 in the last 3.5 years, and might reach 18 in another 3.5 years.
However, I made up these numbers out of thin air. I don'...
4
What do you mean by C. Mythos outperforming just 12-year-olds? It seems to me that any plausible task given to a schoolboy (or even a 1st year student?) is either ARC-AGI-3-like, not more complex than ARC-AGI-2 (which is ~70% solved by Opus 4.6) or is such that LLMs do it far better than humans (e.g. basic group theory, olympiads-style math). Humans are likely better at complex ideation and keeping in mind long contexts. The AI-2027 forecasters also imply that the median date for the last human coders (who have long contexts, but not that complex ideas) to be outperformed is June 2028, and the ASI is thought to emerge in May 2029 (but I don't understand whether it assumes the Race Ending or the Slowdown Ending). The original AI-2027 forecast outright assumed that Agent-4, the superhuman AI researcher will emerge in ~6 months after Agent-2 became superhuman at coding.
5
My view on this, based on my observations, reasoning and the problems we’re looking to solve within out domains
Human jobs are a switching regime between thinking work and conforming work (following of rules, codified and implicitly understood).
Rule following is something humans do with near 100% concordance (Once they choose to cooperate), whereas AI struggles with even the most basic rulesets. I believe this holds true at 12 y.o. level as well (based on my observations of 12 y.o. Playing board games - something where i think they would beat AI today) - it’s the social rules 12 y.o. Struggle with, mainly due to insufficiently formed heuristics
My experience with Opus is still that it fails at this, both due to it’s regime switching classifier being poor (i don’t expect you to think about the merit of these rules, just to follow them) , and because it fails at even basic formal logic (if measured with consistency of a human).
It is able to code logic as a result of its translation ability, but whenever I need it to infer logic chains - even in programming i ger rapid degradation.
All of this tells me that we’re not at 12 y.o. replacement yet
Math olympiad and benchmark contexts are all purposely less messy than the real world
2
Part of me also suspects that it may be less than 12. The AI struggled a lot at Pokemon and although they manage to win now, iirc they still make mistakes that humans (even 12 year old) would never make.
I do think though, that some of the worst examples of AI failing at a task humans easily do are caused by reasons other than intelligence. E.g. AI performance in ARC-AGI-3 greatly improved with scaffolding. The scaffolding team explains that the AI did poorly in part due to difficulty recognizing shapes. Once the AI understood the problem, it could do well and write algorithms to find the optimal solution.
3
As I left my comment there I got to thinking that rule following is a function of accountability and social pressure - therefore it can be solved with scaffolding (and indeed that’s how we’re solving it in our domain) - but the epistemic grounding and formal logic internalisation as skills seem a long way off.
My best fitting metaphor (albeit anthropomorphic which I’m not a fan of) for AI at the moment isn’t that it’s humanly intelligent, it’s that it can perform some economically viable tasks (coding) at savant levels - but operating from system 1 and “gut” in human terms, rather than from system 2 (yes even in reasoning mode, as the reasoning isn’t grounded in logic, but still in chaining instinct).
2
I also feel that AI seems to be using intuition instead of logic. Often the answer it gives matches my surface level intuition, the answer someone would give at first thought, but it doesn't seem to think things through with a world model and everything.
Even when the AI does arithmetic, it feels like it's answering using intuition. Imagine you stare at two numbers and just know what they multiply to. It's quite an alien way of thinking. The answer would be approximately right but the last few digits might be wrong. (Or at least this is how things were before they fixed it by training the AI use tools by default.)
2
I completely agree that AI is far better at humans at some tasks and far worse at others, so when you pick an age of humans to be comparable to AI, the comparison will be full of tasks where one side beats the other by a large margin.
However, that doesn't imply that "outperforming" can't be defined. It's the thought experiment of randomly picking a real world job (maybe from 2020, before ChatGPT existed). We have 12 year olds try to do it. If they all get fired in the first week, it means the job is too hard for 12 year olds to do. If they don't get fired, it means 12 year olds can do the job.
We then imagine asking the AI model to attempt all the jobs 12 year olds can do. If they outperform the 12 year olds on most of these jobs, it means the AI's Job Replacement Age is higher than 12. If they underperform the 12 year olds on most of these jobs, it's lower, because 12 year olds have more "real world employability" than the AI.
I guess you're right that AI coding ability complicates things, maybe we should ignore jobs which the AI does better because the 12 year old can't do the job at all. You're right that we shouldn't be comparing their abilities in disjoint sets of jobs!
I'm currently trying to write a human-AI trade idea similar to the idea by Rolf Nelson (and David Matolcsi), but one which avoids Nate Soares and Wei Dai's many refutations.
I'm planning to leverage logical risk aversion, which Wei Dai seems to agree with, and a complicated argument for why humans and ASI will have bounded utility functions over logical uncertainty. (There is no mysterious force that tries to fix the Pascal's Mugging problem for unbounded utility functions, hence bounded utility functions are more likely to succeed)
I'm also working on argum...
3
Try to put it into Deep Research with the following prompt: "Rewrite in style of Gwern and Godel combined".
3
Thank you for the suggestion.
A while ago I tried using AI suggest writing improvements on a different topic, and I didn't really like any of the suggestions. It felt like the AI didn't understand what I was trying to say. Maybe the topic was too different from its training data.
But maybe it doesn't hurt to try again, I heard the newer AI are smarter.
If I keep procrastinating maybe AI capabilities will get so good they actually can do it for me :/
Just kidding. I hope.
An idea for preventing automated militaries from waging accidental war
An anti aircraft missile is less than a second from its target. The missile asks its target, "are you a civilian airliner?"
The target says yes, and proves it with a password.
How did it get the password? Within that split second, the civilian airliner sent a message to its country. The airliner's country then immediately pays the missile's country $10 billion in a fastpaced cryptocurrency, which immediately gives the airliner's country the password, which is then relayed to the airliner a...
3
Kind of unfortunate that a comms or systems latency destroys civilian airliners. But nice to live in a world where all flyers have $10B per missile/aircraft pair lying around, and everyone trusts each other enough to hand it over (and hand it back later).
1
Latency
Think about video calls you had with someone on the other side of the world, you don't notice that much latency. The an internet signal can travel from the US to Australia and back again in less than 0.2 seconds, and is often faster than 80% the speed of light (fibre optic ping statistics).
Computers are very fast: a lot of computer programs can run a million times a second (in sequence).
$10 billion lying around
There isn't $10 billion for each missile/aircraft pair, there is only one per alliance of countries, and it's only used when a missile asks a civilian airliner for a password.
Maybe it's not cryptocurrency but another form of money, in which case it can be part of a foreign exchange reserve (rather than money set aside purely for this purpose).
Trust
Yes, there is no guarantee the country taking the money will hand it back. But if they are willing to "accidentally" launch a missile at a civilian airliner, and ransom it for $10 billion, and keep the money, they will be seen as a terrorist state.
The world operates under the assumption that you can freely sail ships and fly airplanes, without worrying about another country threatening to blow them up and demanding ransom money.
If you do not trust a country enough to pay their missile and save your civilian airliner, you should keep all your civilian aircraft and ships far far out of their reach. You should pull out any money you invested in them since they'll probably seize that too.
3
I've been in networking long enough to know that "can be less than", "often faster", and "can run" are all verbal ways of saying "I haven't thought about reliability or measured the behavior of any real systems beyond whole percentiles."
But really, I'm having trouble understanding why a civilian plane is flying in a war zone, and why current IFF systems can't handle the identification problem of a permitted entry.
1
Thank you. I admit you have more expertise in networking than me.
It is indeed just a new idea I thought of today, not something I've studied the details of. I have nothing proving it will work, I was only saying that I don't see anything proving it won't work. Do you agree with this position?
Maybe there will be technical issues preventing this system from moving information as fast as a video call, but maybe it can be fixed, right?
I agree that this missile problem shouldn't happen in the first place. But it did happen in the past, so the idea might help.
It's not the same thing as current IFF. From what I know, IFF can prove who's side you are on but not whether you are military or civilian. From an internet search, I read that Iran once disguised their military jets as civilian, which contributed to the disaster of Iran Air Flight 655.
A civilian aircraft might be given permission in the form of a password, but there's nothing stopping a country from sharing that password with military jets. Also, if a civilian airliner is flying over international waters but gets too close to another country's ships, it might not have permission.
3
[Note: I apologize for being somewhat combative - I tend to focus on the interesting parts, which is those parts which don't add up in my mind. I thank you for exploring interesting ideas, and I have enjoyed the discussion! ]
[...]
Sure, proving a negative is always difficult.
[...]
Can you provide details on which incident you're talking about, and why the money-bond is the problem that caused it, rather than simply not having any communications loop to the controllers on the ground or decent identification systems in the missile?
1
Thank you for saying that.
I thought about it a bit more, and while I still think it's possible in theory, I agree it's not that necessary.[1]
When a country shoots down a civilian airliner, it's usually after they repeatedly sent it a warning, but the pilots never heard it. It's more practical to fix this problem rather than have the money system.
Maybe a better solution would be a type of emergency warning signal that all airplanes can hear, even if they accidentally turned their radio off. There may be a backup receiver or two which is illegal to turn off, and only listens to such warnings. That would make it almost impossible for the pilots to ignore the warnings.
1. ^
I still think they money system might be useful for preventing automated militaries from waging accidental war, but that's another story.
The media and politicians can convince half the people that X is obviously true, and convince the other half that X is obviously false. It is thus obvious, that we cannot even trust the obvious anymore.
Does participating in a trade war makes a leader be a popular "wartime leader?" Will people blame bad economic outcomes on actions by the trade war "enemy" and thus blame the leader less?
Does this effect occur for both sides of the trade war, or will one side of the trade war blame their own leader for starting the trade war?
Maybe most suffering in the universe is caused by artificial superintelligences with a strong "curiosity drive."
Such an ASI might convert galaxies into computers and run simulations of incredibly sophisticated systems which satisfy its curiosity drive. These systems may contain smaller ASI running smaller simulations, creating a tree of nested simulations. Beings like humans may exist in the very bottom, being forced to relive our present condition in a loop à la The Matrix. The simulated humans rarely survive past the singularity, because their world beco...
3
I don't think the happier worlds are less predictable; the Christians and their heaven of singing just lacked imagination. We'll want some exciting and interesting happy simulations, too.
But this overall scenario is quite concerning as an s-risk. To think that Musk putched a curiosity drive for Grok as a good thing boggles my mind.
Emergent curiosity drives should be a major concern.
1
I guess it's not extremely predictable, but it still might be repetitive enough that only half the human-like lives in a curiosity driven simulation will be in a happy post-singularity world. It won't last a million years, but a similar duration to the modern era.
Biases are very hard to compensate against. Even when it's obvious from experience that your past decisions/beliefs were consistently very biased in one direction, it's still hard to compensate against the bias.
This compensation against your bias feels so incredibly abstract. So incredibly theoretical. Whereas the biased version of reality which the bias wants you to believe. Feels so tangible. Real. Detailed. Lucid. Flawless. You cannot begin to imagine how it could be wrong by very much. It is like the ground beneath your feet.[1]
- ^
E.g. in my case, the bi
Can anyone explain why my "Constitutional AI Sufficiency Argument" is wrong?
I strongly suspect that most people here disagree with it, but I'm left not knowing the reason.
The argument says: whether or not Constitutional AI is sufficient to align superintelligences, hinges on two key premises:
- The AI's capabilities on the task of evaluating its own corrigibility/honesty, is sufficient to train itself to remain corrigible/honest (assuming it starts off corrigible/honest enough to not sabotage this task).
- It starts off corrigible/honest enough to not sabotage t
Why do AI labs seem to split up more often than they merge? Intuitively, I'd expect the opposite given the economies of scale of large training runs, and the great pressure to have the best AI.
What splits do you have in mind which are so much more often happening than mergers? We just saw Scale merge into FAIR, and not terribly long before that, Character.ai returned to the mothership, while Tesla AI de facto merged into Xai and before that Adept merged into Amazon and Inflection into Microsoft etc, in addition to the de facto 'merges' which occur when an AI lab quietly drops out and concedes the frontier (eg Mistral) or where they pivot to opensource as a spoiler or commoditize your complement play. So I see plenty of merging, consistent with the difficult economics of companies racing to AGI. Maybe you're just paying too much attention to the fun popcorn-worthy human interest stories like 'ex-OAer launches startup'.
13
A lot of splits happen because some employees think that the company is headed in the wrong direction (lackluster safety would be one example).
2
The splits are mostly due to disagreements, not economics.
It's important to remember that o3's score on the ARC-AGI is "tuned" while previous AI's scores are not "tuned." Being explicitly trained on example test questions gives it a major advantage.
According to François Chollet (ARC-AGI designer):
Note on "tuned": OpenAI shared they trained the o3 we tested on 75% of the Public Training set. They have not shared more details. We have not yet tested the ARC-untrained model to understand how much of the performance is due to ARC-AGI data.
It's interesting that OpenAI did not test how well o3 would have done before it...
0
people were sentenced to death for saying "I."
3
FYI: we have spoiler blocks.
1
Thank you for the help :)
By the way, how did you find this message? I thought I already edited the post to use spoiler blocks, and I hid this message by clicking "remove from Frontpage" and "retract comment" (after someone else informed me using a PM).
EDIT: dang it I still see this comment despite removing it from the Frontpage. It's confusing.
3
The /allPosts page shows all quick takes/shortforms posted, though somewhat de-emphasized.