LESSWRONG
LW

Home All Posts Concepts Library Community

Quick Takes

Linch's Shortform

Linch3h116

Anthropic issues questionable letter on SB 1047 (Axios). I can't find a copy of the original letter online.

[+][comment deleted]1h10

Deleted by isabel, Today at 12:01 AM

Reason: whoops, duplicate

2Zach Stein-Perlman2h

Here's the letter: https://s3.documentcloud.org/documents/25003075/sia-sb-1047-anthropic.pdf I'm not super familiar with SB 1047, but one safety person who is thinks the letter is fine.

jacobjacob's Shortform Feed

jacobjacob4d7320

Someone posted these quotes in a Slack I'm in... what Ellsberg said to Kissinger:

“Henry, there’s something I would like to tell you, for what it’s worth, something I wish I had been told years ago. You’ve been a consultant for a long time, and you’ve dealt a great deal with top secret information. But you’re about to receive a whole slew of special clearances, maybe fifteen or twenty of them, that are higher than top secret.
“I’ve had a number of these myself, and I’ve known other people who have just acquired them, and I have a pretty good sense of w

kave6h42

I wish this quote were a little more explicit about what's going wrong. On a literal reading it's saying that some people who disagreed attended meetings and were made to feel comfortable. I think it's super plausible that this leads to some kind of pernicious effect, but I wish it spelt out more what.

I guess the best thing I can infer is that the author thinks public resignations and dissent would have been somewhat effective and the domesticated dissenters were basically ineffective?

Or is the context of the piece just that he's explaining the absence of prominent public dissent?

12Daniel Kokotajlo3d

I was talking about the immediate parent, not the previous one. Though as secrecy gets ramped up, the effect described in the previous one might set in as well. I have personal experience feeling captured by this dynamic, yes, and from conversations with other people i get the impression that it was even stronger for many others. Hard to say how large of an effect it has. It definitely creates a significant chilling effect on criticism/dissent. (I think people who were employees alongside me while I was there will attest that I was pretty outspoken... yet I often found myself refraining from saying things that seemed true and important, due to not wanting to rock the boat / lose 'credibility' etc. The point about salving the consciences of the majority is interesting and seems true to me as well. I feel like there's definitely a dynamic of 'the dissenters make polite reserved versions of their criticisms, and feel good about themselves for fighting the good fight, and the orthodox listen patiently and then find some justification to proceed as planned, feeling good about themselves for hearing out the dissent.' I don't know of an easy solution to this problem. Perhaps something to do with regular anonymous surveys? idk.

6TsviBT3d

So what am I supposed to do if people who control resources that are nominally earmarked for purposes I most care about are behaving this way?

Closed Limelike Curves's Shortform

Closed Limelike Curves8d9614

The Wikipedia articles on the VNM theorem, Dutch Book arguments, money pump, Decision Theory, Rational Choice Theory, etc. are all a horrific mess. They're also completely disjoint, without any kind of Wikiproject or wikiboxes for tying together all the articles on rational choice.

It's worth noting that Wikipedia is the place where you—yes, you!—can actually have some kind of impact on public discourse, education, or policy. There is just no other place you can get so many views with so little barrier to entry. A typical Wikipedia article will get more hit... (read more)

Showing 3 of 38 replies (Click to show all)

Closed Limelike Curves7h10

Permanent link that won't expire here. @the gears to ascension @Olli Järviniemi

1Closed Limelike Curves7h

https://discord.gg/skNZzaAjsC

3Closed Limelike Curves8h

I'm not annoyed by these, and I'm sorry if it came across that way. I'm grateful for your comments. I just meant to say these are exactly the sort of mistakes I was talking about in my post as needing fixing! However, talking about them here isn't going to do much good, because people read Wikipedia, not LessWrong shortform comments, and I'm busy as hell working on social choice articles already. From what I can tell, there's one substantial error I introduced, which was accidentally conflating the two kinds of IIA. (Although I haven't double-checked, so I'm not sure they're actually unrelated.) Along with that there's some minor errors involving strict vs. non-strict inequality which I'd be happy to see corrected.

Daniel Kokotajlo's Shortform

Daniel Kokotajlo9mo1312

Here's a gdoc comment I made recently that might be of wider interest:

You know I wonder if this standard model of final goals vs. instrumental goals has it almost exactly backwards. Would love to discuss sometime.

Maybe there's no such thing as a final goal directly. We start with a concept of "goal" and then we say that the system has machinery/heuristics for generating new goals given a context (context may or may not contain goals 'on the table' already). For example, maybe the algorithm for Daniel is something like:
--If context is [safe surroundings]+[n... (read more)

4Richard_Ngo2d

Relevant: my post on value systematization Though I have a sneaking suspicion that this comment was originally made on a draft of that?

Daniel Kokotajlo12h20

At this point I don't remember! But I think not, I think it was a comment on one of Carlsmith's drafts about powerseeking AI and deceptive alignment.

4Daniel Kokotajlo9mo

To follow up, this might have big implications for understanding AGI. First of all, it's possible that we'll build AGIs that aren't like that and that do have final goals in the traditional sense -- e.g. because they are a hybrid of neural nets and ordinary software, involving explicit tree search maybe, or because SGD is more powerful at coherentizing the neural net's goals than whatever goes on in the brain. If so, then we'll really be dealing with a completely different kind of being than humans, I think. Secondly, well, I discussed this three years ago in this post What if memes are common in highly capable minds? — LessWrong

TurnTrout's shortform feed

TurnTrout3dΩ18336

Effective layer horizon of transformer circuits. The residual stream norm grows exponentially over the forward pass, with a growth rate of about 1.05. Consider the residual stream at layer 0, with norm (say) of 100. Suppose the MLP heads at layer 0 have outputs of norm (say) 5. Then after 30 layers, the residual stream norm will be $100 \cdot {1.05}^{30} \approx 432.2$ . Then the MLP-0 outputs of norm 5 should have a significantly reduced effect on the computations of MLP-30, due to their smaller relative norm.

On input tokens $x$ , let ${A t t n}_{i} (x), {M L P}_{i} (x)$ be... (read more)

jake_mendel15h10

[edit: stefan made the same point below earlier than me]

Nice idea! I’m not sure why this would be evidence for residual networks being an ensemble of shallow circuits — it seems more like the opposite to me? If anything, low effective layer horizon implies that later layers are building more on the outputs of intermediate layers. In one extreme, a network with an effective layer horizon of $1$ would only consist of circuits that route through every single layer. Likewise, for there to be any extremely shallow circuits that route directly from... (read more)

1StefanHex3d

I like this idea! I'd love to see checks of this on the SOTA models which tend to have lots of layers (thanks @Joseph Miller for running the GPT2 experiment already!). I notice this line of argument would also imply that the embedding information can only be accessed up to a certain layer, after which it will be washed out by the high-norm outputs of layers. (And the same for early MLP layers which are rumoured to act as extended embeddings in some models.) -- this seems unexpected. I have the opposite expectation: Effective layer horizons enforce a lower bound on the number of modules involved in a path. Consider the shallow path * Input (layer 0) -> MLP 10 -> MLP 50 -> Output (layer 100) If the effective layer horizon is 25, then this path cannot work because the output of MLP10 gets lost. In fact, no path with less than 3 modules is possible because there would always be a gap > 25. Only a less-shallow paths would manage to influence the output of the model * Input (layer 0) -> MLP 10 -> MLP 30 -> MLP 50 -> MLP 70 -> MLP 90 -> Output (layer 100) This too seems counterintuitive, not sure what to make of this.

15Joseph Miller3d

Computing the exact layer-truncated residual streams on GPT-2 Small, it seems that the effective layer horizon is quite large: I'm mean ablating every edge with a source node more than n layers back and calculating the loss on 100 samples from The Pile. Source code: https://gist.github.com/UFO-101/7b5e27291424029d092d8798ee1a1161 I believe the horizon may be large because, even if the approximation is fairly good at any particular layer, the errors compound as you go through the layers. If we just apply the horizon at the final output the horizon is smaller. However, if we apply at just the middle layer (6), the horizon is surprisingly small, so we would expect relatively little error propagated. But this appears to be an outlier. Compare to 5 and 7. Source: https://gist.github.com/UFO-101/5ba35d88428beb1dab0a254dec07c33b

Ben Millwood's Shortform

Ben Millwood4d114

xAI has ambitions to compete with OpenAI and DeepMind, but I don't feel like it has the same presence in the AI safety discourse. I don't know anything about its attitude to safety, or how serious a competitor it is. Are there good reasons it doesn't get talked about? Should we be paying it more attention?

Vladimir_Nesov18h60

A new Bloomberg article says xAI is building a datacenter in Memphis, planned to become operational by the end of 2025, mentioning a new-to-me detail that the datacenter targets 150 megawatts (more details on DCD). This means the scale of 100,000 GPUs or $4 billion in infrastructure, a bulk of its recently secured $6 billion from Series B.

This should be good for training runs that could be said to cost $1 billion in cost of time (lasting a few months). And Dario Amodei is saying that this is the scale of today, for models that are not yet deployed. This p... (read more)

2Vladimir_Nesov4d

For some reason current labs are not running $10 billion training runs already, didn't build the necessary datacenters immediately. It would take a million H100s and 1.5 gigawatts, supply issues seem likely. There is also a lot of engineering detail to iron out, so the scaling proceeds gradually. But some of this might be risk aversion, unwillingness to waste capital where a slower pace makes a better use of it. As a new contender has no other choice, we'll get to see if it's possible to leapfrog scaling after all. And Musk has affinity with impossible deadlines (not necessarily with meeting them), so the experiment will at least be attempted.

17tylerjohnston4d

I've asked similar questions before and heard a few things. I also have a few personal thoughts that I thought I'd share here unprompted. This topic is pretty relevant for me so I'd be interested in what specific claims in both categories people agree/disagree with. Things I've heard: * There's some skepticism about how well-positioned xAI actually is to compete with leading labs, because although they have a lot of capital and ability to fundraise, lots of the main bottlenecks right now can't simply be solved by throwing more money at the problem. E.g. building infrastructure, securing power contracts, hiring top engineers, accessing huge amounts of data, and building on past work are all pretty limited by non-financial factors, and therefore the incumbents have lots of advantages. That being said, it's placed alongside Meta and Google in the highest liquidity prediction market I could find about this asking which labs will be "top 3" in 2025. * There's some optimism about their attitude to safety since Elon has been talking about catastrophic risks from AI in no uncertain terms for a long time. There's also some optimism coming from the fact that he/xAI opted to appoint Dan Hendrycks as an advisor. Personal thoughts: * I'm not that convinced that they will take safety seriously by default. Elon's personal beliefs seem to be hard to pin down/constantly shifting, and honestly, he hasn't seemed to be doing that well to me recently. He's long had a belief that the SpaceX project is all about getting humanity off Earth before we kill ourselves, and I could see a similar attitude leading to the "build ASI asap to get us through the time of perils" approach that I know others at top AI labs have (if he doesn't feel this way already). * I also think (~65%) it was a strategic blunder for Dan Hendrycks to take a public position there. If there's anything I took away from the OpenAI meltdown, it's a greater belief in something like "AI Safety realpolitik;" that

Linch's Shortform

Linch3d1616

Probably preaching to the choir here, but I don't understand the conceivability argument for p-zombies. It seems to rely on the idea that human intuitions (at least among smart, philosophically sophisticated people) are a reliable detector of what is and is not logically possible.

But we know from other areas of study (e.g. math) that this is almost certainly false.

Eg, I'm pretty good at math (majored in it in undergrad, performed reasonably well). But unless I'm tracking things carefully, it's not immediately obvious to me (and certainly not in... (read more)

Showing 3 of 10 replies (Click to show all)

2cubefox1d

Conceivability is not invoked for logical statements, or mathematical statements about abstract objects. But zombies seem to be concrete rather than abstract objects. Similar to pink elephants. It would be absurd to conjecture that pink elephants are mathematically impossible. (More specifically, both physical and mental objects are typically counted as concrete.) It would also seem strange to assume that elephants being pink is logically impossible. Or things being faster than light. These don't seem like statements that could hide a logical contradiction.

Linch18h20

Sure, I agree about the pink elephants. I'm less sure about the speed of light.

2Linch3d

Do you think ideal reasoning is well-defined? In the limit I feel like you run into classic problems like anti-induction, daemons, and all sorts of other issues that I assume people outside of our community also think about. Is there a particularly concrete definition philosophers like Chalmers use?

Alexander Gietelink Oldenziel's Shortform

Alexander Gietelink Oldenziel2d15-3

Crypticity, Reverse Epsilon Machines and the Arrow of Time?

[see https://arxiv.org/abs/0902.1209 ]

Our subjective experience of the arrow of time is occasionally suggested to be an essentially entropic phenomenon.

This sounds cool and deep but crashes headlong into the issue that the entropy rate and the excess entropy of any stochastic process is time-symmetric. I find it amusing that despite hearing this idea often from physicists and the like apparently this rather elementary fact has not prevented their storycrafting.

Luckily, computational mec... (read more)

15Lucius Bushnaq2d

It's time symmetric around a starting point t0 of low entropy. The further t is from t0, the more entropy you'll have, in either direction. The absolute value |t−t0| is what matters. In this case, t0 is usually taken to be the big bang. So the further in time you are from the big bang, the less the universe is like a dense uniform soup with little structure that needs description, and the higher your entropy will be. That's how you get the subjective perception of temporal causality. Presumably, this would hold to the other side of t0 as well, if there is one. But we can't extrapolate past t0, because close to t0 everything gets really really energy dense, so we'd need to know how to do quantum gravity to calculate what the state on the other side might look like. So we can't check that. And the notion of time as we're discussing it here might break down at those energies anyway.

cubefox1d20

See also the Past Hypothesis. If we instead take a non-speculative starting point as $t_{0}$ , namely now, we could no longer trust our memories, including any evidence we believe to have about the entropy of the past being low, or about physical laws stating that entropy increases with distance from $t_{0}$ . David Albert therefore says doubting the Past Hypothesis would be "epistemically unstable".

Arjun Panickssery's Shortform

Arjun Panickssery1d31

What's the actual probability of casting a decisive vote in a presidential election (by state)?

I remember the Gelman/Silver/Edlin "What is the probability your vote will make a difference?" (2012) methodology:

1. Let E be the number of electoral votes in your state. We estimate the probability that these are necessary for an electoral college win by computing the proportion of the 10,000 simulations for which the electoral vote margin based on all the other states is less than E, plus 1/2 the proportion of simulations for which the margin based on all other

... (read more)

jmh1d20

I would assum they have the math right but not really sure why anyone cares. It's a bit like the Voter's Paradox. In and of it self it points to an interesting phenomena to investivate but really doesn't provide guidance for what someone should do.

I do find it odd that the probabilities are so low given the total votes you mention, and adding you also have 51 electoral blocks and some 530-odd electoral votes that matter. Seems like perhaps someone is missing the forest for the trees.

I would make an observation on your closing thought. I think if one ... (read more)

Arjun Panickssery's Shortform

Arjun Panickssery1mo308

FiveThirtyEight released their prediction today that Biden currently has a 53% of winning the election | Tweet

The other day I asked:

Should we anticipate easy profit on Polymarket election markets this year? Its markets seem to think that
Biden will die or otherwise withdraw from the race with 23% likelihood
Biden will fail to be the Democratic nominee for whatever reason at 13% likelihood
either Biden or Trump will fail to win nomination at their respective conventions with 14% likelihood
Biden will win the election with only 34% likelihood
Even if gas fe

Eric Neyman1d20

I'm now happy to make this bet about Trump vs. Harris, if you're interested.

2Eric Neyman5d

Looks like this bet is voided. My take is roughly that: * To the extent that our disagreement was rooted in a difference in how much to weight polls vs. priors, I continue to feel good about my side of the bet. * I wouldn't have made this bet after the debate. I'm not sure to what extent I should have known that Biden would perform terribly. I was blindsided by how poorly he did, but maybe shouldn't have been. * I definitely wouldn't have made this bet after the assassination attempt, which I think increased Trump's chances. But that event didn't update me on how good my side of the bet was when I made it. * I think there's like a 75-80% chance that Kamala Harris wins Virginia.

3kairos_7d

Polymarket has gotten lots of attention in recent months, but I was shocked to find out how much inefficency there really is. There was a market titled "What will Trump say during his RNC speech?" that was up a few days ago. At 7 pm, the transcript for the speech was leaked, and you could easily find it by a google search or looking at the polymarket discord. Trump started his speech at 9:30, and it was immediately that he was using the script. One entire hour into the speech I stumbled onto the transcript on Polymarkets discord. Despite the word "prisons" being in the leaked transcript that Trump was halfway through, Polymarket only gave it a 70% chance of being said. I quickly went to bet and made free money. To be fair it was a smaller market with 800k in bets, but nonetheless I was shocked on how easy it was to make risk-free money.

davekasten's Shortform

davekasten2d70

A random observation from a think tank event last night in DC -- the average person in those rooms is convinced there's a problem, but that it's the near-term harms, the AI ethics stuff, etc. The highest-status and highest-rank people in those rooms seem to be much more concerned about catastrophic harms.

This is a very weird set of selection effects. I'm not sure what to make of it, honestly.

2Dagon1d

There are (at least) two models which could partially explain this: 1) The high-status/high-rank people have that status because they're better at abstract and long-term thinking, and their role is more toward preventing catastrophe rather than nudging toward improvements. They leave the lesser concerns to the underlings, with the (sometimes correct) belief that it'll come out OK without their involvement. 2) The high-status/high-rank people are rich and powerful enough to be somewahat insulated from most of the prosaic AI risks, while the average member can legitimately be hurt by such things. So everyone is just focusing on the things most likely to impact themselves. edit: to clarify, these are two models that do NOT imply the obvious "smarter/more powerful people are correctly worried about the REAL threats, and the average person's concerns are probably unimportant/uninformed". It's quite possible that this division doesn't tell us much about the relative importance of those different risks.

davekasten1d10

Yup1 I think those are potentially very plausible, and similar things were on my short list of possible explanations. I would be very not shocked if those are the true reasons. I just don't think I have anywhere near enough evidence yet to actually conclude that, so I'm just reporting the random observation for now :)

2[comment deleted]2d

Nathan Young's Shortform

Nathan Young1mo10

Here is a 5 minute, spicy take of an alignment chart.

What do you disagree with.

To try and preempt some questions:

Why is rationalism neutral?

It seems pretty plausible to me that if AI is bad, then rationalism did a lot to educate and spur on AI development. Sorry folks.

Why are e/accs and EAs in the same group.

In the quick moments I took to make this, I found both EA and E/acc pretty hard to predict and pretty uncertain in overall impact across some range of forecasts.

Showing 3 of 8 replies (Click to show all)

ProgramCrafter2d12

I disagree with "of course". The laws of cognition aren't on any side, but human rationalists presumably share (at least some) human values and intend to advance them; insofar they are more successful than non-rationalists this qualifies as Good.

2Said Achmiz26d

The laws of cognition are natural laws. Natural laws cannot possibly “drive towards flourishing” or toward anything else. Attempting to make the laws of cognition “drive towards flourishing” inevitably breaks them.

2cubefox21d

A lot of problems arise from inaccurate beliefs instead of bad goals. E.g. suppose both the capitalists and the communists are in favor of flourishing, but they have different beliefs on how best to achieve this. Now if we pick a bad policy to optimize for a noble goal, bad things will likely still follow.

quila's Shortform

quila2d74

I was looking at this image in a post and it gave me some (loosely connected/ADD-type) thoughts.

In order:

The entities outside the box look pretty scary.
I think I would get over that quickly, they're just different evolved body shapes. The humans could seem scary-looking from their pov too.
Wait.. but why would the robots have those big spiky teeth? (implicit question: what narratively coherent world could this depict?)
Do these forms have qualities associated with predator species, and that's why they feel scary? (Is this a predator-species-world?)
Most human

... (read more)

0Jay2d

I don't want to live in a world where there's only the final survivors of selection processes who shrug indifferently when asked why we don't revive all the beings who were killed in the process which created the final survivors. If you could revive all the victims of the selection process that brought us to the current state, all the crusaders and monarchists and vikings and Maoists and so, so many illiterate peasant farmers (on much too little land because you've got hundreds of generations of them at once, mostly with ideas that make Putin look like Sonia Sotomayor), would you? They'd probably make quite the mess. Bringing them back would probably restart the selection process and we probably wouldn't be selected again. It just seems like a terrible idea to me.

quila2d51

Some clarifications:

I'm thinking of this in the context of a post-singularity future, where we wouldn't need to worry about things like conflict or selection processes.
By 'the ones who were killed in the process', I was thinking about e.g herbivorous animals that were killed by predator species^[1], but you're correct that it could include humans too. A lot of humans have been unjustly killed (by others or by nature) throughout history.
I think my endorsed morals are indifferent about the (dis)value of reviving abusive minds from the past, though moral-patie

... (read more)

kromem's Shortform

kromem2d50

I'm surprised that there hasn't been more of a shift to ternary weights a la BitNet 1.58.

What stood out to me in that paper was the perplexity gains over fp weights in equal parameter match-ups, and especially the growth in the advantage as the parameter sizes increased (though only up to quite small model sizes in that paper, which makes me curious about the potential delta in modern SotA scales).

This makes complete sense from the standpoint of the superposition hypothesis (irrespective of its dimensionality, an ongoing discussion).

If nodes are serving mo... (read more)

Seth Herd's Shortform

Seth Herd8d113

A metaphor for the US-China AGI race

It is as though two rivals have discovered that there are genies in the area. Whichever of them finds a genie and learns to use its wishes can defeat their rival, humiliating or killing them if they choose. If they both have genies, it will probably be a standoff that encourages defection; these genies aren't infinitely powerful or wise, so some creative offensive wish will probably bypass any number of defensive wishes. And there are others that may act if they don't.

In this framing, the choice is pretty clear. If it's ... (read more)

kromem2d30

While I generally like the metaphor, my one issue is that genies are typically conceived of as tied to their lamps and corrigibility.

In this case, there's not only a prisoner's dilemma over excavating and using the lamps and genies, but there's an additional condition where the more the genies are used and the lamps improved and polished for greater genie power, the more the potential that the respective genies end up untethered and their own masters.

And a concern in line with your noted depth of the rivalry is (as you raised in another comment), the quest... (read more)

Leon Lang's Shortform

Leon Lang8d140

https://x.com/sama/status/1813984927622549881

According to Sam Altman, GPT-4o mini is much better than text-davinci-003 was in 2022, but 100 times cheaper. In general, we see increasing competition to produce smaller-sized models with great performance (e.g., Claude Haiku and Sonnet, Gemini 1.5 Flash and Pro, maybe even the full-sized GPT-4o itself). I think this trend is worth discussing. Some comments (mostly just quick takes) and questions I'd like to have answers to:

Should we expect this trend to continue? How much efficiency gains are still possible? C

2Leon Lang6d

One question: Do you think Chinchilla scaling laws are still correct today, or are they not? I would assume these scaling laws depend on the data set used in training, so that if OpenAI found/created a better data set, this might change scaling laws. Do you agree with this, or do you think it's false?

Vladimir_Nesov2d20

New data! Llama 3.1 report includes data about Chinchilla optimality study on their setup. The surprise is that Llama 3.1 405b was chosen to have the optimal size rather than being 2x overtrained. Their actual extrapolation for an optimal point is 402b parameters, 16.55T tokens, and 3.8e25 FLOPs.

Fitting to the tokens per parameter framing, this gives the ratio of 41 (not 20) around the scale of 4e25 FLOPs. More importantly, their fitted dependence of optimal number of tokens on compute has exponent 0.53, compared to 0.51 from the Chinchilla paper (which wa... (read more)

4Vladimir_Nesov6d

Data varies in the loss it enables, doesn't seem to vary greatly in the ratio between the number of tokens and the number of parameters that extracts the best loss out of training with given compute. That is, I'm usually keeping this question in mind, didn't see evidence to the contrary in the papers, but relevant measurements are very rarely reported, even in model series training report papers where the ablations were probably actually done. So could be very wrong, generalization from 2.5 examples. With repetition, there's this gradual increase from 20 to 60. Probably something similar is there for distillation (in the opposite direction), but I'm not aware of papers that measure this, so also could be wrong. One interesting point is the isoFLOP plots in the StripedHyena post (search "Perplexity scaling analysis"). With hybridization where standard attention remains in 8-50% of the blocks, perplexity is quite insensitive to change in model size while keeping compute fixed, while for pure standard attention the penalty for deviating from the optimal ratio to a similar extent is much greater. This suggests that one way out for overtrained models might be hybridization with these attention alternatives. That is, loss for an overtrained model might be closer to Chinchilla optimal loss with a hybrid model than it would be for a similarly overtrained pure standard attention model. Out of the big labs, visible moves in this directions were made by DeepMind with their Griffin Team (Griffin paper, RecurrentGemma). So that's one way the data wall might get pushed a little further for the overtrained models.

Daniel Kokotajlo's Shortform

Daniel Kokotajlo2y510

Surprising Things AGI Forecasting Experts Agree On:

I hesitate to say this because it's putting words in other people's mouths, and thus I may be misrepresenting them. I beg forgiveness if so and hope to be corrected. (I'm thinking especially of Paul Christiano and Ajeya Cotra here, but also maybe Rohin and Buck and Richard and some other people)

1. Slow takeoff means things accelerate and go crazy before we get to human-level AGI. It does not mean that after we get to human-level AGI, we still have some non-negligible period where they are gradually getting... (read more)

Showing 3 of 26 replies (Click to show all)

Richard_Ngo2d20

I disagree with the first one. I think that the spectrum of human-level AGI is actually quite wide, and that for most tasks we'll get AGIs that are better than most humans significantly before we get AGIs that are better than all humans. But the latter is much more relevant for recursive self-improvement, because it's bottlenecked by innovation, which is driven primarily by the best human researchers. E.g. I think it'd be pretty difficult to speed up AI progress dramatically using millions of copies of an average human.

Also, by default I think people talk ... (read more)

3Heighn2y

Less relevant now, but I got the "few years" from the post you linked. There Christiano talked about another gap than AGI -> ASI, but since overall he seems to expect linear progress, I thought my conclusion was reasonable. In retrospect, I shouldn't have made that comment.

1Heighn2y

Thanks for offering your view Paul, and I apologize if I misrepresented your view.

Daniel Kokotajlo's Shortform

Daniel Kokotajlo3d6629

Great quote, & chilling: (h/t Jacobjacob)

The idea of Kissinger seeking out Ellsberg for advice on Vietnam initially seems a bit unlikely, but in 1968 Ellsberg was a highly respected analyst on the war who had worked for both the Pentagon and Rand, and Kissinger was just entering the government for the first time. Here’s what Ellsberg told him. Enjoy:
“Henry, there’s something I would like to tell you, for what it’s worth, something I wish I had been told years ago. You’ve been a consultant for a long time, and you’ve dealt a great deal with top secret i

... (read more)

Showing 3 of 4 replies (Click to show all)

13Raemon2d

...in the last 24 hours? Or, like, awhile ago in a previous context?

Richard_Ngo2d91

In the last 24 hours. I read fast (but also skipped the last third of the Doomsday Machine).

2jmh2d

Seemed to jump out to me. While I don't always follow my own advice I do most of the time approach others from a view point that I can learn something from anyone and everyone.

tilek's Shortform

tilek2d10

AI-Caused Extinction Ingredients

Below is what I see is required for AI-Caused Extinction to happen in the next few tens of years (years 2024-2050 or so). In brackets is my very approximate probability estimation as of 2024-07-25 assuming all previous steps have happened.

AI technologies continue to develop at approximately current speeds or faster (80%)
AI manages to reach a level where it can cause an extinction (90%)
AI that can cause an extinction did not have enough alignment mechanisms in place (90%)
AI executes an unaligned scenario (low, maybe less than

... (read more)

tilek's Shortform

tilek2d10

"AI will never be smarter than my dad."

I believe ranked comparing intelligence between two artificial or biological agents can only be down subjectively with someone deciding what they value.

Additionally, I think there is no agreed upon whether the definition "intelligence" should include knowledge. For example, can you consider an AI "smart" if it doesn't know anything about humans?

On the other hand, I value my dad's ability to have knowledge about my childhood and have a model of my behavior across tens of years very highly. Thus, I will neve... (read more)