wonder

To zoom out a bit, from the post I assume you benchmark ASI mostly by "replacing humans 100% in all jobs". Curious in why you specifically care about absolutely 100%? (Replacing 50% of humans is still significant imo.)

"Looking up the exact answer in a database"

I mean it is not too too wrong if the model is using RAG?

We simply lack systems other than religions that provide religions same benefits

I am empathetic to this point, and I was thinking of this recently too. However, I've reached a conclusion that the benefit of having values relatively independent of religion (or even similar communities) is to be able to change and develop your values as you experience more things in life. It feels to me that values in religion may be relatively fixed (I could be wrong). 

The sad/hard thing is that the process of changing values or beliefs might be a painful experience, w... (read more)

I do see this as fair criticism (not surprised by it) to model welfare, if that is the sole reason for ending conversation early. I can see the criticism coming from two parts: 1) potential competing resources, and 2) people not showing if they care about these X group issues at all. If any of these two is true, and ending convo early is primarily about models have "feelings" and will "suffer", then we probably do need to "turn more towards" the humans that are suffering badly. (These groups usually have less correlation with "power" and their issues are u... (read more)

People have limited capacity for empathy

Do you think this goes the other way as well?

I also agree "AI" is overloaded and has existing connotations (ranging from algorithms to applications as well)! I would think generative models, or generative AI works better (and one can specify multimodal generative models if one wants to be super clear), but also curious to see what other people would propose.

Usually I also take emotions as a channel to surface unconscious preferences (either situational or longer term), which helps with making that preference conscious as well as evaluated, and thus helps with rational decisions.

Thanks; 

we just prefill the response with "<SCRATCHPAD_REASONING> The user is asking for".

Do you mean a template like:

Q: xxxx

A: <SCRATCHPAD_REASONING> The user is asking for

and then ask the model to continue?

Maybe I missed this in the paper - for base models, do you change the prompt slightly for the base model or is it still instruction based prompts for testing fake alignment?

Thanks for writing this up! I highly agree and I think this is an important point to emphasis. Stating crucial context/clarification/assumptions is important, and unclear communication is likely counter-productive. 

Note: this is a completely sidebar (and apologize in advance).

I prefer my family members to randomly-sampled people with similar traits. I would certainly not elect to sterilize or kill my family members so that they could be replaced with smarter, kinder, happier people.

Out of curiosity, in hypothetical scenarios:

1. Would you replace randomly-sampled people with smarter, kinder, happier people? (To clarify I hope the answer is no)
2. Would you say you would (or humans) prefer your family members over other randomly-sampled people, in a resource scarce situation

I share some similar frustrations, and unfortunately these are also prevalent in other parts of the human society. The commonality of most of these fakeness seem to be impure intentions - there are impure/non-intrinsic motivations other than producing the best science/making true progress. Some of these motivations unfortunately could be based on survival/monetary pressure, and resolving that for true research or progress seems to be critical. We need to encourage a culture of pure motivations, and also equip ourselves with more ability/tools to distinguish extrinsic motivations.

Would the take over for small countries also about humans using just an advanced AI for taking over? (or would the human using advanced AI for take over happen faster?)

Maybe I missed this in the article itself - are there plans to make sure the superbabies are aligned and will not abuse/overpower the non-engineered peers?

I was thinking of this the other day as well; I think this is particularly a problem when we are evaluating misalignment based on these semantic wording. This may suggest the increasing need to pursue alternative ways to evaluate misalignment, rather than purely prompt based evaluation benchmarks

Based on my observations, I would also think some current publication chasing culture could get people push out papers more quickly (in some particular domains like CS), even though some papers may be partially completed

Will the event/sessions be recorded by any chance? (may not be able to attend, but would love to learn); additionally, would the topics be focused exclusively on relations to X risks?