Tejas

On bio: I think if you believe that we can build safe Superintelligence, at which point you almost certainly have bio-defense dominance, then you just want to get there as soon as possible, and any waiting or slowdown period makes someone using AI systems for bioterrorism, or some kind of national conflict escalating into bioweapons conflict more likely. More generally, I think the central driver of biorisk in a world where you aren't worried about loss of control is foreign nations and terrorists getting access to the models, so marginal time in the perio

For what it’s worth, I only recently started working full-time in AI safety, but have not encountered stages 6 and 7 frequently when hearing other junior staff articulate the “Constellation worldview.” Instead, I usually hear it articulated as “preventing an AI capable of making significant progress on very ambitious approaches to principled superintelligence alignment from engaging in reward-hacking, sandbagging, etc.,” i.e., stage 4 (but a broader vision of ambitious superalignment than just the agent foundations and CEV stuff), without getting to the st... (read more)

I’ve been wondering: imagine you had a very smart person with very limited ML background and essentially no background in AI safety, but high-potential in some way (e.g., a great philosopher with good mathematical intuitions and very smart). What would be your reading list for them be if they wanted to be an excellent strategic thinker about AI and futurism, and have the context necessary to work on reducing the risk of AI takeover in a productive way?

Thanks for writing this! I found it helpful.

Is the link to “the Problem -> Patch -> Make it smarter -> Whole New Problem -> New Patch loop” meant to be a specific tweet? Is the link to cultural evolution meant to be to the comment about reflection?

Some combination of:

• general work to promote better epistemics and integrity among people working on AI safety (e.g., through maintaining LessWrong),
• specific work to help maintain useful AI safety infrastructure (e.g., helping with design and increasing traffic to very valuable research projects, helping build good funding systems for AI safety that I trust),
• and more recent specific work to capitalize on recent attention on AI safety (e.g., callcongress.ai).

Donated $600 for now. My current (noncommital) guess is that I will continue supporting your work with small donations over the next few months if you remain somewhat funding-constrained.

I’ve been very impressed by Lightcone’s work recently. How valuable would you say small donations (say, on the order of ~$1k) are on the current margin?

Consider applying to a career transition grant.

That analogy is helpful, thanks. I think I still feel like there’s a gap between developing a good theory of decision-making and putting that theory into immediate practice. For instance, immediately after developing expected utility theory, I could imagine there’s a processing gap among someone who isn’t used to using it to make everyday decisions; I feel a meaningful amount of doubt that von Neumann was much less likely to read books while driving after internalizing expected utility theory (obviously, maybe his preference for doing this was just that st... (read more)

(sorry for the nitpick but I think you meant to say EDT with updatefulness double counts?)

I think their plan is more robust than you expect, because "solving agent foundations" is not like "writing down some code", it's more like "totally refactoring your ontology for how rational agents make decisions". From an outside view, having the best understanding of how it's rational to make decisions does seem like it'll very plausibly help you not make obvious mistakes. And from my inside view, I actually think that the solution to agent foundations is specifically action-guiding on questions like "when and how should I try to accumulate power?" (I e

This is the best initial attempt I’ve seen at answering this question (or something like it, anyway). I don’t think it has a clear answer to many of the thorny cases though.

I agree with that!

The idea that there are no more good ideas and it's all just scaling from here

Fwiw, this is not how I understand his take. I think he’s saying AI progress will be bottlenecked by compute (and human expert data), which I interpret to mean that the elasticity of substitution between compute/data and ideas isn’t high enough. In fact, I think this his view includes compute to run experiments to do AI research.

I feel like there’s some kind of disanalogy between the prime factorization case (where the goal itself is well-defined (the job is to “find the prime factorization, where N has a process-independent prime factorization”), and hence processes can be refined to become better processes in reaching the goal) and normative goals under the broadly-Humean framework (where the output of the procedure is the goal, and there’s no further goal to reach).

That said, I think there’s something in the vicinity of this that feels right to me. Maybe it has to do with the ... (read more)

It seems to me that:

• If you had a sudden influx of 100m high-skilled immigrants (for simplicity, just assuming this is ordinary very high-skilled immigrants, and let’s say they don’t have above-replacement fertility), the long-run growth rate wouldn’t actually change much. Besides the “declining research productivity” issue common to semi-endogenous models, I’d guess this would also be pretty bottlenecked by energy, housing/zoning laws, etc. A population level effect, which doesn’t substantially change the population growth rate, seems really unlikely to m

As a moral anti-realist, I’m sympathetic to something sort of like Humean constructivism, although with a substantial component of “self creation”/understanding that, at the bottom, it’s still on me. In that case, though, I kind of think the values I end up with upon reflection—if the reflection happens in a way I endorse—are what I’d consider my “true values.” This also means that, if the reflection process is underspecified, I get to specify the idealization process I’d like, as it is a process of making, and discovering, myself according to the filters ... (read more)