Petropolitan

Sun-synchronous orbits are already very popular, have you considered how large a constellation can they safely sustain before everyone placing satellites would have to pay some kind of congestion fees?

Where does one get ~90T of non-useless text tokens?

That is a useful thing if implemented well, and indeed it is a thing I use (from OpenAI and Anthropic) more often than I use Google Search. But that thing is not Google Search.

Several hours ago I googled an uncommon steel grade (an alphanumeric designation with the word steel). In the late 2010s Google would have given me search results in milliseconds and at least one of the first two links would have had the specs I needed.

Today I got a page of garbage links which happened to have same number in different contexts, and then 30 seconds later after a lot o... (read more)

Strong upvote, the key here is "VR" in RLVR: there are no automatically verifiable rewards for good or convincing writing, only RLHF, the cost of which scales proportionally with the length of writing evaluated (and if you hire non-Americans as RLHF trainers for economy reasons the result is unlikely to fit well with stylistic preferences of Americans). The labs can use engagement as a metric but that will lead to "baiting" already very common in the social media and will not convince anyone

Somewhat tangential to the questions of whether this essay was AI-written and whether any human actually writes like a LLM, I think linguists now widely agree that LLMs picked a lot of traits from the formal register of African varieties of English when OpenAI (and later other American companies) hired Kenyans and Nigerians (possibly a lot of English teachers among them!) to do RLHF

And for the same copyright reasons the labs will never allow the users to see the pretraining data in any way

IDK about quantitative trading, but managing real sector companies like pharmaceutical labs requires plenty of skills CEOs and boards of AI labs (and of software companies in general) just don't have.

However, in February Anthropic hinted they are interested in transpiling legacy COBOL code, causing IBM shares to plunge. There surely is quite a lot of specialized competence and experience needed to disrupt the software sector, but plenty of people with both will be happy to work for OpenAI or Anthropic, and they speak the same IT jargon lab executives know ... (read more)

I realized there's one more way Eliezer's use of the term ASI is confusing: do we agree that Dario's "country of geniuses in a data center" count as the "real ASI" for the purpose of the thesis "You only get one real shot at real ASI"?

If you take Bostrom's definition of ASI, it obviously should qualify: "Speed superintelligence: A system that can do all that a human intellect can do, but much faster. [...] Collective superintelligence: A system composed of a large number of smaller intellects such that the system’s overall performance across many very gene... (read more)

The article does an excellent job for 2020 but has become a bit dated in regards to the static warfare due to the recent Drone Revolution, the author briefly addresses that in comments

Thank you for a good reply. I think the key of our disagreement is the definition of "the actual ASI". Many future AI systems are certain to be superhuman in many more aspects than the existing LLMs even with best current scaffolds, and will still be below humans in some important aspects, and thus will fail to take over. Why would you deny the rank of ASI to them? Others (Wagner's "clever AI") might destroy our civilization during a takeover attempt but still be below humans in less important aspects, why grant the rank to them before the attempt?

I'm argu... (read more)

More importantly, can the consumers afford the products? If there's increased supply of labor from robots, the real incomes from jobs will fall, but at the same time the increased demand for energy and raw materials from the AI boom will also cause cascading inflation throughout the real sector of the economy! Coupled with the lucrativeness of investing the capital into AI, the interest rates might skyrocket, which also has all sorts of negative effects on the real sector (for example, who bails out millions of people who had to downshift jobs due to automation and thus can't repay their debts but can't refinance due to rates either?)

Your line of argument in the other comment sounds convincing but I'm not sure how it answers my question! BTW in a war, there is also an option of a stalemate which is really a lose-lose situation for both sides (doesn't look like it can apply to an AI takeover for the first glance).

As of responses to failed AI takeover attempts, I believe it will depend on the number of casualties: if there are dozens of fatalities or worse, the humanity will probably treat it as a fire alarm and react accordingly (whether it would be too late is another question), while if no one dies, probably not

Not just "the takeover" but every takeover attempt in the history of humanity, that's very different from the "only one try" framing (cf. repeated game vs. single-shot game in game theory).

I am specifically worried about a scenario where multiple dumb failed AI takeover attempts discredit the idea that misaligned AIs can do significant harm but actually teach the future AIs how to take over, and by the time the decision-makers realize how serious the issue is it's too late.

E. g., first takeover attempts might be so ridiculous that the AIs fail at exfiltrat... (read more)

Chess is fully verifiable in silico, so the curse clearly does not apply.

Taking over an anthill is a poor comparison because the anthill is a sufficiently simple system with little feedback loops if at all, unlike the human society which is complex and unpredictable due to plenty of poorly understood feedback loops, often very nonlinear and often irrational.

You might disagree, but with the current very limited progress in AI alignment and quite unsafe practices in the frontier labs the first AGI/ASI attempting a takeover almost certainly will not be superh... (read more)

Does the "curse of oneshotness" apply to the unaligned AGI/ASI attempting a takeover? If no, why? If yes, does that imply the first AI takeover attempt would probably fail, thus seemingly contradicting the applicability of "oneshotness" to humanity developing ASI?

There was an actual theory of why the Chernobyl reactor was supposed to not explode, written down so that multiple people could read it, based on an understanding from first principles!

More specifically (and I don't think it's known outside of the Russian nuclear engineering-adjacent community), at least two people independently calculated and described in classified technical reports how RBMK could explode in the specific circumstances it actually exploded, and because the technical solution implemented after 1986 was at the time deemed too expensive for ... (read more)

The "Foom and Doom" hypothesis is popular here even now, when it was mostly succeeded by the "fast takeoff", and it was even more popular in the past. People who believe in these hypotheses tend to assume that the economy just won't have enough time to adapt to AGI/ASI so they disregard the possible economic effects.

And BTW, unemployment without an extinction could easily happen in a myriad more ways: through AI pauses and bans due to, e. g, economic crisis and public outcry, a failed AI takeover attempt, a "fire alarm" incident short of a takeover attempt... (read more)

An interesting work, let me compare it with my estimates from three weeks ago: for all eight GPT-5 series models I considered (5, 5 Pro, 5.1, 5.2, 5.2 Pro, 5.3, 5.4, 5.4 Pro) 2T total parameters fall within the 90% prediction interval brackets, and four more I didn't consider (4o, o1, o3, 4.1) fit as well. My 1.2T estimate for Sonnet is very close to Li's 1.7T, and my 4T estimate for Opus 4-series fits into the 90% PI bracket for all five versions. (Just to remind, on average, we should expect 1 true value out of 10 not to fit)

The list of problems with mineral traceability has never really included someone tampering with the data already in an external database. What it does include is the data entered into the database being false to begin with, while many participants of the projects have economic incentives to cover it up, and there are indeed geochemical fingerprinting attempts to fix that problem but they are entirely orthogonal to the issue of the data storage and access.

From what I know, I see no significant advantages of a blockchain against a public, well-audited relati... (read more)

I expanded my previous comment significantly after posting it, hope it didn't mess with your response.

I think we have somewhere in between because these issues are actually connected. I do believe AI superhuman in hard-to-verify tasks are plausible, but they won't have this particular problem anymore (maybe they would have some functional analog of shame working against it^[1] or maybe it will just go away with some advances in RL).

But if this issue isn't solved, AIs are unlikely to be able to run basic military procurement tasks fully autonomously (especia... (read more)