I think continual learning might be solved by giving an agent tool access to a database, and then training the agent to use the tool effectively. Rather than something with the weights.

My odds are:

• closer to small updates on weights [40%]
• closer to database queries [30%]
• unresolved [30%]

One is that many participants at ARENA have already done AI Safety research before participating.  Second evidence is that at least four ARBOx (a 2-week compressed version of ARENA) are doing elite AI safety fellowships (1 Anthopic Fellows Program, 2 LASR Labs, 1 MATS).

I don't think this is evidence that ARENA is about signalling:

1. MATS and LASR are completely different from ARENA. I wouldn't find anything fishy about someone doing ARENA after LASR/MATS.
2. If anything, someone doing ARENA after MATS is evidence against the hypothesis, because the signal of (MATS+ARENA) is no better than MATS alone.
3. It sounds a little odd to do ARBOx after Anthropic Fellows. But I'd guess they wanted a fun two weeks hanging out in Oxford with other young people. Not that they thought the signal of (AFP + ARBOx) was so much better than AFP alone (which sounds absurd to me).

We've all heard of "Safety Cases", i.e. structured arguments that an AI deployment has low chance of catastrophe. Should labs be required to make Benefit Cases, i.e. structured arguments for why their AI deployment has high expected benefits?

Otherwise, how do we know that the benefits outweigh the risks?

@Eli Tyre asked for an example:

An event-space was hosting a Christmas party in London. I arrived late, maybe 10pm. They had oversupplied food, and a large cake, topped with strawberries, had been abandoned in the corner. Rather than cutting a slice of cake, I simply took a strawberry from the top. I turned to my friend and said "This is unethical"^[1] and ate the strawberry.

1. ^{^}

   Clopus45 initially tells me that taking the strawberry was fine, but after some back-and-forth we've agreed on this assessment: Strawberries are the scarce, desirable resource; cake is the abundant substrate. The intended allocation bundles them together. By taking a strawberry without cake, you claim more than your proportional share of the good stuff while leaving strawberry-depleted cake for others. You might argue you weren't going to eat cake regardless—but someone else might have wanted a properly-topped slice. The strawberry you took was theirs. This is mitigated by the likelihood that the cake would go uneaten anyway, but not eliminated by it.

What's up with lesswrong and lumenators? It's not that rats are less susceptible to marketing, or better at finding products. (Or, not only.) It's something about reframing the problem from "I need blue light therapy" to "I need photons" and then sourcing photons from wherever they're cheapest.

This is related to More Dakka: "I need blue light therapy" isn't dakka-able because you're either doing the therapy or your not. Whereas "I need photons" is dakka-able -- it's easier to see what it would mean to 100x photons.

I think that, if you're about to do something that you know is wrong, it's better to loudly declare to yourself and others that it's wrong.

c.f. active inference, inoculation prompting, signalling, social memetics, etc, etc.

Two additional considerations in favour of long-dated call options:

1. (h/t  @Ryan Kidd) This acts as a hedge against the devaluation of your income. Imagine two investment portfolios: one is "AGI-pilled" (i.e. long-dated call options, semiconductors, etc) and the other is "business-as-usual" (i.e. real estate, S&P 500). Suppose further that they have the same distribution over returns, from your perspective. Then you should buy the AGI-pilled portfolio, to hedge against the loss of income due to AGI.
   1. Caveats: Maybe income is higher in AGI worlds than in BAU worlds? This isn't crazy, but unlikely. See Capital in the 22nd Century (Phil Trammell and Dwarkesh)
2. The value of the SP 500 is the market's expectation of the long-run returns. So when you buy a 2 year call option, you aren't making a bet that AGI will cause revenue to jump 2x, you are only making a bet that the market will expect future discounted revenue to jump by 2x. That is, this is more like timelines on when the market will become AGI-pilled rather than when AGI arrives.

But if you give them Iphones, they can always just decide they don't want to use them.

So there's a few thoughts which push in the opposite direction:

1. Something something free will is subjective. Like, you might describe something as "you offered them a choice between A and B and they chose A" but they might describe it as "you forced them into state A". This is a bit weird example but: if I stabbed you, I might describe this as me offering you an option to bleed on the floor.
2. There are (at least) two strategies to resolve the problem in (1). One strategy is kinda deontological: we introduce the concept of boundaries or «Boundaries». But this is a bit messy so IDK if it appears in our mature ethics. And if it does, it might have the entire island in a boundary.
3. The other strategy is kinda utilitarian: we say that you are harming someone when you do something to them that they wouldn't ideally want. And that includes offering them a free choice between A and B. And then the meat of the question is what "ideally" means.

IDK. I think I'm just confused about all of this.

But I think we shouldn't reward people for only making joking predictions instead of 100-page reports

I don't like this sentiment.

If he predicted ASI in 2028-2030 then I'm not 'punishing' him by believing he did; if he didn't then I'm not 'rewarding' him by believing he didn't.

Thanks for these considerations, I'll ponder on them more later.

Here are my immediate thoughts:

If you think that the amount of resources used on mundane human welfare post-singualarity is constant, then adding the Zambian child to the population leads to a slight decrease in the lifespan of the rest of the population, so it's zero-sum.

Hmm, this is true on impersonal ethics, in which the only moral consideration is maximising pleasurable person-moments. On such a view, you are morally neutral about killing 1000 infants and replacing them with people with the same welfare. But this violates common sense morality. And I think you should have some credence (under moral uncertainty) that this is bad.

If you think that the lightcone will basically be spent on the CEV of the humans that exist around the singularity, you might worry that the marginal child's vote will make the CEV worse.

Hmm, this doesn't seem clear-cut, certainly not enough to justify deviating so strongly from common-sense morality.

1. Just naively, it sounds crazy to me.
2. This consideration assumes that the child you save from malaria cares less about hedonium (or whatever weird thing EA's care about) than the average person. However, you might naively expect that they will care more about hedonium because they actually owe their lives to EA whereas almost no one else does.
3. This consideration assumes that the CEV is weighted equally among all humans, rather than weighted by wealth. If you assume it's weighted by wealth then the GiveDirectly donation has the same impact on CEV as the AMF donation.
4. This consideration predicts that someone is incentivised to kill as many people as possible just before the CEV procedure is executed. But a CEV procedure which incentivised people to murder would be terrible, so we wouldn't run it. We are more likely to run a CEV procedure which rewards people for saving the lives of the participants of the CEV.