in short, it seems to me that the crux of the argument comes down to whether there is physiological continuity of self or 'consciousness' for lack of a better word.  

I suspect this will also actually have very relevant applications in field such as cryonics which adds an additional layer of complexity because all metabolic processes will completely cease to function.
 
Conducting the duplication experiment during sleep (or any altered state of consciousness) is interesting but nevertheless there is clearly physical (physiological) continuity of the original subject in an albeit altered state. It could be the case that some form of metabolic (or inslico equivalent) may be necessary to ensure the original self persists without question (eg. nano-tech enabled neuron by neuron and synapse by synapse replacement over an extended period of time). We do have some interesting existence proofs in the case of organisms like the Northern Tree Frog though that seem to retain memory and 'self' through periods of freezing with negligible metabolic activity.

I don't agree that associating multiple related ideas is necessarily 'not good', that said Governance and politics are not the focus of this article. In fact it specifically addresses that. 

The article was primarily intended to illustrate how economic alignment may very well be an additional area of alignment and risk that that we should more seriously consider.  If you disagree with this, I sincerely want to better understand objections so I'd welcome a thoughtful counter argument that addresses the claims and not the structure of the article. 

 

I'm not clear on what you mean by 'relevant probability', however,  yes, I do think that we will see AGI within two decades, and with respect to AI, 

P(massive job displacement) is high, perhaps 30-50%, 
P(millions die / acute catastrophe) is perhaps 3-6% and
P(billions die / doom) is perhaps 0.2-0.5%

So, I'd say P(catastrophe) is not negligible and will likely slowly rise over time (so long as Information technology generally improves). If it does happened, I would not be surprised if governments take drastic action including potential broad blanket internet blackouts which would increase the value of certain non-IT skills.

I do worry about economic stability during any AI takeoff because the lack thereof could severely inhibit our ability to respond.

I think the term 'prepper' skills is a tad derogatory and perhaps simplistic, but I do believe that we are slowly loosing many of those skill sets that contribute to self-sufficiency and I do believe some skills  associated with 'prepping' are valuable (i.e., basic first aid, CPR, orienteering, navigation, engineering, construction, carpentry, mechanical repair, basic agriculture, PPE, maintaining some food/water supply, etc.). Obviously, I am not talking about the more extreme fringes of prepping which becomes a different conversation. 
 

I'm not actually talking about a complete breakdown of economy or society, just significant shocks to retail, IT, and supply chains and longer term economic shifts. 

If I may  draw a historical parallel: we shut down our entire airline transportation industry for weeks after 9/11, reducing annual GDP about 0.5%.  Not to trivialize the ~3000 deaths and related suffering that occurred but, IMO, AI-facilitated deliberate attacks by malicious actors or nation states could easily be orders of magnitude worse. I honestly don't know exactly how capable and prepared the US government is of 'shutting down' the internet completely. This would be a huge decision but I would be very surprised if they don't already have a protocol in place for major catastrophic cyber scenarios that effectively does exactly this. 

I suspect it might only take one order of magnitude (i.e., 30,000 deaths) perhaps two, before the Federal Agencies that be, pull the plug at every ISP and every telecom data provider for months. 3 OOM, it could be years.  Yes, the consequences would be economically catastrophic.... and yes, there would probably be some hacky half-effective work-arounds via, decentralized networks, RF, satellite etc. but yes it could absolutely happened. 

Information and computing Technologies would still have limited function in society but it wouldn't be long before we saw some massive shifts in how digital tech-centric our economic future remained.

To preface, i didn't mean to make this a central point and it was mostly directed at families (but also anyone with the interest and means) considering how critical food availability would be in an prolonged AI gone bad scenario when internet connectivity would likely be cut.  There are many scenario's where food (and retail at large) could be significantly disrupted due to lack of electronic payment :  prolonged power outages, accidental cyber incidents, deliberate cyber attacks, internet shutoffs, and honestly: recognize how significantly our agricultural system is based on autonomous harvesting and AI.   In any of these scenarios, electronic forms of payment would be non-operational within days. Even if you have some cash on hand, many retailers might close their doors (with or without social unrest). 

Further, there are numerous reasons to think severe weather will worsen in many regions which can directly or indirectly interrupt food supplies.  The reason I bring this up in the context of families becuase once you have kids, you realize how important safe food supplies are...  most can source water fairly easily outside of dry desert regions but unless you're in alaska or hawaii, canada or have a few acres of farmland, you might not realize how vulnerable our food system could be.

Honestly, some of the mannerisms and vocal features certainly seem inspired by Ms. Johansen's likeness in the Movie Her... and Open AI probably should have not used a voice this close to Johansen's especially in light of her declining the first voice contract offer.  But the fact is, just because 4o invokes the spirit of Samantha in "Her", the voice is definitely different and across all the online chatter about GPT-4o last week, I didn't hear about anyone talking specifically about how the voice sounded exactly like Johansen's. It was only after Johansen made this big public statement and presumed legal threat that people were talking about Johansens likeness. I'm sure there might have been a few but does anyone have any actual evidence from before Johansen's statements?

Well said.  We've been contemplating expanding our family lately and I have to say, I've been secretly thinking many of the same things.   That said, if we want humanity to persist and have a chance of one day prospering alongside AI and other technologies to come, children seem like a pretty clear prerequisite (particularly from people like us who care about these bigger pictures).  I personally believe there will likely be non-trivial socioeconomic inequality and strife in the wake of AGI, however, I believe that these timescales will be on the order of decades (not weeks or months). In short, I believe that raising future generations to care about the future of humanity is incredibly important.  

On a brighter note, I personally think a few things could be worthwhile to think about in preparing our children for the uncertainty that will very likely come with a post-AGI world. Purely IMO and I realize these things are not available or practical for everyone but just wanted to share a few thoughts:

1. In an time when we increasingly can't believe everything we see and read, kids need to learn to question appropriately, to reason probabilistically, think critically and most of all think for themselves.  
2. AGI will more significantly disrupt white-collar job markets than blue collar job markets( with a few exceptions). Consequently, you might help your children develop some hard skills (eg. how to repair an appliance, build something with wood, patch clothing, change your car oil, an Arduino project etc.) 
3. it will become increasing valuable to be more of a generalist (and one who is comfortable with change). Teach resilience and get them thinking about emotional intelligence from a young age.  
4. I feel like it will be increasingly important to be involved with your local community and know your neighbors. 
5. IMHO, The problem with kids and technology is less with IT and computers as it is with social media and the darker sides of the internet. I would actually prefer that my kids engage intellectually with an offline language model than to drop them in the deep end of the internets. 
6. We frankly shouldn't take our economic supply chains and food industries for granted. Understand where things come from, how they're made, how to be more self sufficient. Grow a garden, tech kids basic horticulture, raise a pet chicken. If possible, lean towards living in an area with some local agriculture.
7. Not everyone needs to be a dedicated 'prepper'... but if you own a home, maybe think about solar and water storage.  Regardless, of where you live, minimal food stores not a bad idea. $100 of rice/beans and a bottle of multivitamins stored properly can feed a family for many months if desperate.   
8. This is personal but I think we're generally trending towards over structuring and micromanaging (ie. helicopter parenting) our kids which can lead to anxiety and lack of self-reliance. I think that is important to give younger children a bit more latitude and autonomy to learn to become comfortable on their own.  
9. Needless to say, I don't think social media (and most mainstream media) is healthy in any way for kids under 16ish, get them a flip phone. Get them involved with local groups for real-world socialization (eg. scouts, clubs, meetups, etc)
10. This isn't easy of feasible for many but if you have the option and inclination, consider raising kids in a more rural environment at least for a while. If you believe that AI will bring socioeconomic instability, dense urban metropolitan area will be most impacted.

alignment research is currently a mix of different agendas that need more unity. The alignment agendas of some researchers seem hopeless to others, and one of the favorite activities of alignment researchers is to criticize each other constructively

Given the risk-landscape uncertainty and conflicting opinions, I would argue that this is precisely the optimal high-level approach for AI Alignment research agendas at this point in time. 'Casting a broader net' can allow us to more quickly identify and mobilize resources towards areas of urgently-needed alignment research when they are identified with sufficient confidence.  IMHO, Constructive debate about research priorities is hard to argue against.  Moreover, much like the lack of publication of negative results in academic literature results in significant inefficiencies within scientific R&D, having even a shallow understanding of a broader 'space' of alignment solutions  has value in itself in that can identify areas that are ineffective or not applicable respective to certain AI capabilities.