The valuelessness of a treaty seems to be based on a binary interpretation of success. Treaties banning chemical, biological, and nuclear weapons development may not have been absolutely successful; they have been violated. But I don’t think many people would argue those restrictions haven’t been beneficial. 

I’m not clear why a ban on developing AGI would not have similar value. 

I like the idea of cocktail party investment advice as an economic bellwether. That’s a good observation. 

 

There’s no hard delimiter on financially induced sector collapse, and it’s often not directly attributable to the sector that collapses. The dot com crash was tied to federal reserve interest rate increases that resulted in a sell off as investors moved towards less speculative investments. 

AI is in a fairly safe position right now through sheer variety of vested interests. Government, construction, infrastructure, computing hardware, software, and early corporate adopters of AI are all doing everything they can to keep the ball rolling. They’ve crossed a line where sunk costs have an outsized role in future decisions. There’s also the wildcard of AI being deemed a strategic defense asset. 

The present state of spending will continue until there’s some catastrophic event that scares people off or public pressure forces a reduction in scale. 

My money is on public pressure leading to change. Data centers are being publicly subsidized and the general public is beginning to push back. One or two public service commissions refusing to comply with energy and water subsidies will lead to wide scale changes that will put the brakes on. Things will move out of startup mode and into the realm of actual business. 

You were right. I was interested and genuinely enjoyed the article. I hope there is a Part III. I am interested to know about your personal approach.  

Omelas can also be seen as plausibility through imperfection. People have trouble accepting perfection. Imperfection adds a layer of realism that anchors people to a story. Without imperfection, there’s nothing to talk about. 

Reza Aslan cleverly applies the concept to deities, but it works with pretty much everything. Looking gift horses in the mouth is human nature. There’s a direct relationship between an ideal, and the level of effort put into the inquisition the ideal is subjected to in search of the imperfection. There’s also an expectation that the closer something is to utopia, the worse the imperfection will be. 

Mark Twain said “What do they say about a man with no vices?” Churchill said he didn’t trust a man without any vices. Both are saying the same thing. The same thing as Le Guin. Omelas isn’t interesting until suffering is introduced. Omelas is a fantasy that’s wholly unremarkable without the imperfection of the girl’s plight. The imperfection makes it relatable. 

The error in the analysis seems to be the assumption that the value of the story is in the suffering. I don’t think that has to be the case. Omelas is a valid critique of cynicism. Our innate need to find fault simply to give ourselves something to complain about. Our refusal to accept something unless it’s damaged. 

The eldritch horror analogy is interesting. I like it overall. If we extend the analysis to include not just the gods, but the people involved, I believe it adds another dimension to the discussion. 

In eldritch horror there are rarely any good guys. Arguably there are none. There are typically two groups of actors, and the rest of humanity is an ignorant mass concerned with mundane matters. Innocent of the knowledge of the supermundane. 

The two groups of actors are the priests, and the individuals who seek to thwart the priests using some other eldritch power. The priests are universally bad. They worship their god at the expense of everything, and everyone, else. The oppositional force, let’s call them adepts, seek to keep the priests in check. They do not worship the eldritch gods, but they do use esoteric knowledge of other eldritch gods in furtherance of their mission. There is a semblance of altruism in their opposition to the priests, but it’s more accurate to view their behavior as the cost of their education in their arts. A moral obligation that comes with their pursuit of knowledge for their own ends.

The priests and the adepts both use their beliefs as mechanisms to bypass moral and ethical considerations. They have a higher calling or self imposed obligation they use to justify their actions. The only people left untouched by the eldritch horrors are the people who don’t get involved. The central lesson in the genre is that no good comes from getting involved. 

Since this essay is about getting involved, I think we must ask what the goal is, and where the solution lies. Are we to play the role of the priest or the adept? Are we accelerating the rise of eldritch horror, or are we shaping the horror to benefit ourselves while minimizing the impact to non-participants? 

Once a role is chosen, the location of the solution must be determined, and here, I believe, is the crux of all this. Is the eldritch horror actually the problem, or is the problem the people involved? By targeting the god we’re really no different than the priests who use the god to justify immortality. By targeting the god we’re just continuing an age old game of making war on ideas while the priests are out conjuring new gods. By targeting the priests we’re crossing a line that, historically, hasn’t resulted in much good, and has created suffering so profound we define our temporal position by it. 

So what is the point? We’re either wasting our time in an eternal philosophical conflict or we’re engaging in an eternal physical conflict and the moral quagmire that entails. Does reason truly demand we get involved, or are we citing reason as an excuse to pursue personal gain? 

Within the genre of eldritch horror, reason would seem to dictate no action. There is limited room for the role of the scribe who records the deeds of the priests and adepts. That allows for intellectual satisfaction, but the neutrality required is notoriously difficult to maintain. If the compulsion to act cannot be overcome, perhaps a gatherer and recorder of information is a viable option. 

 

I’m pleased this got some traction. One of my largest concerns with AI policy development is that state level decision makers will not recognize the threat until catastrophic damage has been done. 

Identifying the need for chemical, biological, and nuclear warfare treaties was fairly universal, as there were real examples of their risks available for all to see. Without that tangible evidence, there’s a risk of incremental disaster like we’re seeing with climate change policies. 

A policy accelerationist is probably my biggest concern. A group that creates a problem in order to highlight the need to protect against even larger disasters. Like the Gruinard Island soil incidents. 

Any movement towards redlines and international safeguards is a good thing. 

A central assumption here is that AI will continue to function in a unfettered way. Where each user is able to construct a bespoke digital world for themselves based on all the variation access to unlimited information can provide. 
I don’t think that idea reflects reality. Historically, when it is possible to limit information, authorities will do so in whatever ways forward their agenda, or retards the agenda of an opponent or enemy. 

Instead of a bespoke microcosm based on personal values, authorities will simply prevent access to information that is not compliant with the agenda. Trends towards monopolistic capitalism mixed and religious bias will result in a monolithic AI that doesn’t just ignore things (like evolution) but attacks the ideas and people who support them. 

We’re already seeing a measurable reduction in original online content and ideas. That’s unlikely to change. The whole thing is speeding towards a single point that authoritarian types have been gagging for as long as literacy has existed. Instead of insular enclaves, I see an entire insular society living in a constant crisis of verisimilitude and outright propaganda where error checking is entirely self referential and wholly circular. 

Conservatism isn’t about keeping things as they are. It’s about regression into a gilted fictional past. Intentionally introducing bias into a system in order to transit to a nonexistent temporal location as a reactionary response seems like a strange thing to do in general. It seems like an exceedingly strange thing to do to a conglomeration of logical procedures. 

The entire notion is inherently regressive and reactionary. It’s coping with fear of an unknown future by appealing to an idealized past. Intentionally baking fear into the system eliminates the goal of an inherently progressive system that is, by definition and design, intended to be in a continuous state of incremental improvement. 

Systematic Utopianism predicated on fictions fabricated in correlation fallacy does not seem like a pursuit of reason; therefore hostile to AGI. 

Another commonality between the Hasidim and LDS is disassociation from society at large. It’s more than localized consolidation of theologically aligned individuals. Both set themselves apart from society, essentially othering the majority of society. Whether they deem others to be merely goyim (Hasidim) or Gentiles (LDS), or the more extreme views tamei or unclean, a major emphasis is placed on downgrading those outside their faith. 

That’s not to say individuals from either religion are viewing outsiders as lesser, necessarily, but from a group level the othering of outsiders is a key component in community building and cohesion. By establishing a state of being which outsiders cannot attain, the scale and scope of the community is severely limited. Community members are forced to look within the community to satisfy innate needs. Companionship, entertainment, education, reproduction, et al can only be obtained within the community. 

On a small scale, these practices are very effective from a community perspective and they have limited negative impact outside the community. But at scale, things get complicated, and ugly. It’s not a big step from viewing people as unclean to becoming a cleaner. Highly insular communities are not, sociologically, far off from ethnic supremacy ideologies. The red headed stepchild of tradition is blood purity.