Very interesting question, Anna.  In fact, it covers so much territory and is so deep I'm not sure where to start.  

To one of your core questions, "the difficulty we are having lately in forming/sustaining institutional cultures (especially, ones adequate to get much done)" - Yuval Levin has studied and written extensively on this.  

One of Levin's core diagnoses is that there is a new tendency of people to use institutions as platforms to launch/improve their own popularity rather than letting the institution form and guide them.  

Think of the relatively new phenomenon of new star journalists using, say the NYT, as the platform to leverage and boost their own brand and career independent of the NYT.  Or, say a politician who uses the R or D party as a platform for a media following rather than the old way of slowly working your way up the ranks of a party and gaining influence and prestige within the confines of the organizational norms.  

At the risk of blaming social media for yet another social ill, I do think SM enables this "platform using" behavior.  In today's world, a young upstart politician can get her own following on Twitter or FB, breaking through the limitations of the old party organization.  

===

Consider also larger societal and cultural trends that have been in play for decades now. On average Americans are much wealthier and have more opportunities than in past generations.  One of the side effects of higher wealth is people don't need or desire to be part of institutions as much as before.  Fifty years ago if someone was down on their luck, short on money, getting help from their local church or VFW Club would have been more common.  

Relatedly and additionally, a surge in government benefits also reduces the need and incentives to rely on those local institutions.  Why bother with the church or the Lion's Club if state and local gov. is providing checks no (or few) questions asked.  

I like to approach this question by thinking of incentives.  

I would add two more elements to your three listed reasons: (1) mature alternatives (2) risk aversion which causes general adoption of a new technologies to take a long time.  

While the promise of self-driving cars could bring lots of benefits, the current alternatives are so advanced and refined that there is not a huge urgency to switch to the new alternatives yet.  

People and governments resist new technologies and need lots of assurances before the new (driverless cars, etc.) is accepted.  

Consider an imperfect example/analogy:  Covid vaccines happened way faster than anything like it in history because of the extreme incentives involved - the world had ground to a halt and many thousands of people were dying daily.