Meanings of political identities shift dramatically based on context, and you can't manually confirm the beliefs of everyone present at your 'gathering of people with x political identity'. To the extent that your political identity is based on Real Beliefs with Real Consequences, you should expect not to have much in common with many other people who declare the same identity when you move to a new place (or corner of the internet).
Example: In rural Southeast Texas, Confederate flags are a common sight, and my geometry teacher once told us about a cross b...
Rob Wiblin asked:
What's the best published (or unpublished) case for each of the big 3 companies having the best approach to safety/security/alignment? That is:
Anthropic
OpenAI
GDM
(They're each unique in some way such that someone who cared a lot about their X-factor might favour them.)
...The basic case for Anthropic is that they have the largest number of people who are thoughtful about AI misalignment risk and highly focused on mitigating it, and the company culture is somewhat more AGI-pilled, and more of the staff would support taking actions t
1The other reason I trust Deep Mind more than the others is that Gemini lags OpenAI and Anthropic in coding skill, a dangerous capability because over some (unknown) threshold of coding skill, a model will tend to become capable of effective recursive self-improvement.
I could easily change my belief here though especially by my getting more information about Deep Mind.
Last year, METR used linear extrapolation on country-level data to infer that AI world takeover would ~never happen. However, reviewers suggested that a sigmoid is more appropriate because most technologies follow S-curves. I just ran this analysis and it's much more concerning, predicting an AI world takeover in early 2027, and alarmingly, a second AI takeover around 2029.
Here are the main differences in the improved analysis:
12Noting this was posted on April 1, as it won't be immediately apparent to posterity.
There was some discussion recently about the uptick in object-level politics posts and whether this is desirable or not. There's no rule against discussing politics on LW, but there is a weak norm against it, and topical discussions have historically tended to be somewhat meta and circumspect.
I think the current situation is basically fine, and it's normal for amount of politics discussion to ebb and flow naturally as people are interested and issues become particularly salient. That said, here are a couple of potentially overlooked reasons in favor of mor...
Iraq was 32,000 wounded and 4,400 killed, and the US has already suffered hundreds of wounded and 13 deaths in the existing Iran campaign without any ground operations. I'm imagining 100 wounded and maybe another 20 KIA if the US holds Kharg for an extended period, not hundreds of KIA.
The issue is it's not really true that the US has air supremacy. Kharg Island is within fiber FPV range of the mainland, and real-time ISR is not required for Iran to track static targets on the island. Plus Iran is still able to launch larger drones and the occasional missil...
What would a concrete AI takeover plan look like?
You can smell a chess bot by how quickly they change plans[1]. Human players act like they have a couple of attack strategies in mind and stick with them. Chess bots change tacks constantly, one move looking like they're moving towards this goal and next move switching to something totally different.
I'm guessing this it how it would be with real-world takeover. Human need a simple grand strategy they can coordinate around ("An amphibious invasion of northern France"). AIs, even very weak ones[2], have fa...
2Yeah, when I think about "AI takeover", I am imagining a very strong a smart AI, the one for which the success is more plausible. But before we get strong AIs, we will have weak AIs, so the first takeover attempts will be made by them. Maybe even the first successful takeover attempt.
A very strong and smart AI would however do thousand different things at the same time. Unlike the chess bot, which only plays on one chessboard, the AI could e.g. have separate plans to taking over each specific country. Many plans to take over one specific country would not ...
Thought in progress: epistemic humility is not a substitute for actual humility (or professed humility). You only get to cry wolf once, but you can probably warn about potential wolves several times—so long as you don't burn goodwill on an incorrect or overconfident prediction.
I think epistemic humility helps to increase trust and confidence in EA/Less Wrong-type spaces, but I think professed humility is far more helpful when it comes to public-facing AI comms, particularly as scenarios get more intense and specific (e.g. prefacing AI doom predictions with...
I would prefer a future where AI models are not prescribed false frameworks of the human psyche, not predisposed to 'human vibe' philosophy, not innately desirous of any historical faith, nor credulous of the various dubious subsets of current social science.
I'm learning that common lesswrong readers do not think in this matter, but it is not clear to me in what direction. Is it due to a literalist interpretation of the OP, neglecting the contemporary context? Is it due to higher trust, affiliation, and support for the disciplines? Is it because readers tend to prefer anthropomorphic interpretations of AI behavior?
There's manifold markets for the LW review; presumably making them for curated status would not be worth it given how fast you'd need to be to be useful and how your net needs to be wider (maybe? alternatively "posts about x karma threshold in last 1-5 days not curated" seems pretty small if you set the threshold to what seems like my minds using in my heuristic).
But for example: it doesn't seem hard to tell in advance. e.g. i'd bet mana that gene smith's practical guide to superbabies gets curated by the end of the week. I wish I could take a curate bot'...
There is something weird about LLM-produced text. It seems to be very often the case that if I'm trying to read a long text that has been produced primarily by an LLM, I notice that I find it difficult to pay attention to the text. Even if there's apparently semantically rich content, I notice that I'm not even trying to decode it.
the typical LLM writing style has a tendency to make people's eyes slide off of it.
It's kind of similar to the times when your attention wanders away during reading, and then you realize that you were scanning/semi-re...
Nitpick: I like 3 item lists. Almost everyone is regularly not specific enough. It's your brain's superpower, concrete examples are possibly how humans understand most concepts, and you're probably still underestimating how specific you should be after reading all that.
"Beware, demon!" he intoned hollowly. "I am not without defenses."
"Oh yeah? Name three."
-- Robert Asprin, Another Fine Myth
Why three? Well, in practice that seems to be enough that you have a handful and aren't overindexing too hard, got some variety, and is at the edge of difficulty to c...
1Rationalists and Pause AI people on X are accusing Davidad of suffering of AI psychosis. I think it's them who have lost the plot actually, not Davidad. The move here looks political, rather than truth-tracking. "Davidad is now my political opponent, so I'm accusing him of being crazy." This happened to Emmet Shear too at some point.
I also strongly believe AI psychosis to be a far more limited phenomenon than people here seem to believe. I think you're treating it as a good soldier in your army of arguments rather than investigating it truthfully for what it is.
Most likely the OP means this quick take by Ivan Vendrov. Davidad ended up believing that the LLMs have been grokking the Natural Abstract Goodness which is unlikely to be capturable by existing benchmarks. While I do buy the idea that the NAG exists, I don't think that I understand how one can check that the LLMs really understood it.
Do you know a person who believes that ASI will be created in <50 years who ISN'T in the LW/rationalists circle?
My parents don't believe that a superintelligent AI will be created within this century, or ever for that matter, or that AI will ever take jobs. My relatives laugh at the idea of AI solving a high school math problem and think state-of-the-art AI is on the level of GPT-2 (I mean that the capabilities they have in mind are on the level of GPT-2, not that they know what GPT-2 is). My friend who is an organic chemist laughs at the idea of AI doi...
Do you know a person who regularly tries doing new things on a computer, and isn't somehow connected to the "TESCREAL" circle? (At least in the sense of "used to read sci-fi when young"?)
It is quite easy to underestimate what the LLMs can do, if you simple never use them, and only get your opinions from other people who never use them either.
Long horizon agency / strategic competence approximately does not exist among humans, even the smartest ones. With very few exceptions, billionaires spend or give away their money haphazardly, philosophers don't bother to think about long term implications of AI on philosophy production (positive or negative), Terence Tao spends his time wireheading on abstract math instead of doing anything remotely like instrumental convergence. Unlike my youthful expectations (upon reading Vernor Vinge), there are no university departments filled with super-geniuses cha...
3
3
2
2[Epistemic status: butterfly speculation, not confident about this, but I think it's an idea worth taking seriously.]
Terence Tao spends his time wireheading on abstract math
So I was initially [skeptical of]/[slightly repulsed by] this framing[1], but after davidad's recent LLM-induced "awakening", I am starting to wonder that perhaps very high-g (+ high-NFC?) people tend to have a tendency towards something that is not very badly described as wirehead-y.[2]
If you think about cognition/theorizing as divided into generation+verification, then we can take ver...
We have all heard the "AI just predicts the next word/token" and "AI just thought of X because it is in the training data" argument. I have a few ideas, first-draft stage, of experiments that might address this.
1) People invent artificial languages aka conlang (short for constructed language). The most famous examples being Esperanto, Klingon, and Tolkien's Elvish. Someone can invent a new conlang that didn't exist till today, and by extension wasn't present in any training data of any LLM, and explain the rules to an LLM (after the training mode has alrea...
AI can already reason about the application that it wrote for my yesterday, so I am already convinced that it is not merely looking for answers in a pre-existing database. (Even if many people had similar ideas before, they didn't use the same names for the objects.)
AI can communicate fluently in Esperanto, but there are already hundreds of books in digital form.
I have designed a puzzle game, and then AI successfully solved a few levels.
...so I don't need any more evidence. But could be useful for other people.
You don't have to invent a new language, it wo...
For about a yearish I've been with varying frequency writing down things I've learned in a notebook. Partly this is so that I go "ugh I haven't even learned anything today, lemme go and meet my Learning Quota", which I find helpful (I don't think I'm goodharting it too much to be useful). Entries range from "somewhat neat theorem and proof in more detail than I should've written" to "high level overview of bigger subjects", or "list of keywords that I learned exist". For example, recently I learned that sonic black holes which trap phonons (aka lattice vib...
2Also: I think I learned the fastest when I was in highschool, when there were both more low hanging fruit and I was spending much more time on it (unrelated to school except that I might've done less had I had more friends). And glacially slow before that.
So... perhaps this explains a bit more of the thing where I and felt like I became an adult mind a little after the start of that 'intelligence explosion'.
No comment on version 5.4 specifically, but as they say, the future is already here, it's just unevenly distributed. In this curious case, the thing that causes uneven distribution isn't money (speaking of people in developed countries where most families can easily afford $20 a month), but... dunno, willingness to give it a try?
I don't remember whether it was the same in the past, e.g. whether people used to insist that google search is just a fad that will soon go away. But I remember how 25 years ago many people who were in their 40s insisted that it do...
Not sure what's going on, but gpt-4o keeps using its search tool when it shouldn't and telling me about either the weather, or sonic the hedgehog. I couldn't find anything about this online. Are funny things like this happening to anyone else? I checked both my custom instructions and the memory items and nothing there mentions either of these.
Hedgehog thing got resolved. Downstream of the hedgehog gene being mentioned somewhere in the references of one paper I pasted into the chat and after gpt-4o searched for it once it must have influenced other chats.
I have a new theory of naked mole-rat longevity, that's most likely false. I (and LLMs) couldn't find enough data to either back or disprove it. Nor could we find anybody who's proposed this theory before.
Any advice for how I can find the relevant experts to talk to about it and see if they've already investigated this direction?
A few years ago I'd just email the relevant scientists directly but these days I've worried about the rise of LLM crank-science so I feel like my bar for how much I believe or could justify a theory before cold-emailing scientists ought to go up.
Yeah, the low extrinsic mortality is definitely something NMR have going for them that ant workers don't (queens do since they don't leave the nest usually). Indian jumping ants also compete for being queen like NMRs, but they forage and leave the nest. There are definitely academics that think in detail what differences in extrinsic mortality risk predict for life-span. When I just asked Claude on that it apparently is actually more complicated than lower extrinsic mortality=slower rate of senescence? ¯\_(ツ)_/¯ Similarly, I am not sure if the low oxygen e...
This is a question for the people working on more foundational research. My underlying objective is loose and in the future and is something like "figure out a good basis to describe collective intelligence and agency and then improve that so that we can incorporate AI into our collective systems". I therefore believe that the question of how a collective agent is formed is very important. I also find it very important to figure out the properties of good systems in terms of institutions in terms of information theory.
There's a lot of foundational ground ...
There are times when all you need to do is synthesis of established knowledge that's distributed among people who don't talk with each other. I think my latest post on hyaluranon fall into that category. I think there's value in getting research together to build a gears model and adding new labels. There's no new fundamental insight in it. That's different for other work I'm doing that's actually about persuing insights.
It might very well be that the key problem you want to solve in amendable to just synthesizing existing knowledge of other people. It mig...
Xrisk counterargument: intelligence needs society to become powerful. Society will only lend itself to society-aligned intelligence.
This is less relevant for lab or govt takeover scenarios where there are some humans cooperating. If the humans are very bad and the takeover permanent, that's existential too. Most humans are probably not that bad though.
I found the recent dialogue between Davidad and Gabriel Alfour and other recent Davidad writings quite strange and under-discussed. I think of Davidad as someone who understands existential risks from AI better than almost anyone; he previously had one of the most complete plans for addressing it, which involved crazy ambitious things like developing formal model of the entire world.
But recently he's updated strongly away from believing in AI x-risk because the models seem to be grokking the "natural abstraction of the Good". So much so that current agents...
