An LLM it a tool of communicative expression, but so it the written or spoken word, music etc. It is a medium throug which the intent travels. As a Dutchman, I have a preference of being direct and clear, but the impact of my words sometimes have the opposite effect, as my listeners do not have my context and can react emotionally to a worded message that is meant factually. If an LLM can help me translate such expression to a language that is better for my target audience to understand, then it is similar to translating into another language.
Still, the written word is no substitute for the full breadth of human expression couple with sufficient context. I find my communication often fails because I assume context (and sometimes intelligence) that my partners don't have.
Apart from that, communication is often not about hearing another one's mind, it can also, and often is, about trying to impose one's own worldview on another. Most broadcast media are: you are not intended to know anything about the mind of the creator of the message, that is in fact an irrelevant aspect of the broadcast message.
Our exploration system is very useful, but it takes a lot of energy (and anxiety), because of the inherent cost of failure which genetics baked into our brain. Hence, doing something new everyday in a society as complex and everchanging as our own is very useful, but very hard with our outdated brain hardware and software.
Add to that the distractions that hijack our outdated brain mechanisms: we have gotten better and better and such hijacking, creating an additional barrier. Doing this is comparably difficult to keeping to a strict diet and exercise regime while mouthwatering delicacies and relaxed convenciences are offered to you at every turn.
You are trying to break patterns (habits), but it is extremely hard to create a habit/pattern of newness, for habits/patterns are fundamentally opposed to doing things in a novel way.
1I think your claim the rudimentary abilities arrive before transformational ones cannot be applied to A.I. the same as human intelligence. While humans might have taken millennia to go from caveman painting to our current ability to produce artistic images, it is clear that A.I. became transformational very quickly in that particular field. You see the same transformational abilities in text writing, music and video too and software development is getting there.
Some of the more artistic of these abilities don't have a clear benchmark, but even with more fuzzy criteria for success, they already outcompute most humans.
Some of the building blocks of A.I. are fundamentally different from us, that is why the have difficuty with some tasks. Their metacognitive, learning and memory abilities has been improved significantly over the last couple of years, but it is still a pale shadow compared to what we are capable of. And in some of the transformational tasks, these abilities are essential.
Horizon length is an imperfect measurement of the lack of some of the abilities.
Your working paper, "Open Global Investment as a Governance Model for AGI." It provides a clear, pragmatic, and much-needed baseline for discussion by grounding a potential governance model in existing legal and economic structures. The argument that OGI is more incentive-compatible and achievable in the short term than more idealistic international proposals is a compelling one.
However, I wish to offer a critique based on the concern that the OGI model, by its very nature, may be fundamentally misaligned with the scale and type of challenge that AGI presents. My reservations can be grouped into three main points.
1. The Inherent Limitations of Shareholder Primacy in the Face of Existential Stakes
The core of the OGI model relies on a corporate, shareholder-owned structure. While you thoughtfully include mechanisms to mitigate the worst effects of pure profit-seeking (such as Public Benefit Corporation charters, non-profit ownership, and differentiated share classes), the fundamental logic of such a system remains beholden to shareholder interests. This creates a vast principal-agent problem where the "principals" (all of humanity) have their fate decided by "agents" (a corporation's board and its shareholders) who are legally and financially incentivized to prioritize a much narrower set of goals.
This leads to a global-scale prisoner's dilemma. In a competitive environment (even OGI-1 would have potential rivals), the pressure to generate returns, achieve market dominance, and deploy capabilities faster will be immense. This could force the AGI Corp to make trade-offs that favor speed over safety, or profit over broad societal well-being, simply because the fiduciary duty to shareholders outweighs a diffuse and unenforceable duty to humanity. The governance mechanisms of corporate law were designed to regulate economic competition, not to steward a technology that could single-handedly determine the future of sentient life.
2. Path Dependency and the Prevention of Necessary Societal Rewiring
You astutely frame the OGI model as a transitional framework for the period before the arrival of full superintelligence. The problem, however, is that this transitional model may create irreversible path dependency. By entrenching AGI development within the world's most powerful existing structure—international capital—we risk fortifying the very system that AGI's arrival should compel us to rethink.
If an AGI corporation becomes the most powerful and valuable entity in history, it will have an almost insurmountable ability to protect its own structure and the interests of its owners. The "rewiring of society" that you suggest might be necessary post-AGI could become politically and practically impossible, because the power to do the rewiring would have already been consolidated within the pre-AGI paradigm. The stopgap solution becomes the permanent one, not by design, but by the sheer concentration of power it creates.
3. Misidentification of the Ultimate Risk: From Distributing Wealth to Containing Unchecked Power
My deepest concern is that the OGI model frames the AGI governance challenge primarily as a problem of distribution: how to fairly distribute the economic benefits and political influence of AGI. This is why it focuses on mechanisms like international shareholding and tax revenues.
I fear the ultimate risk is not one of unfair distribution, but of absolute concentration. As you have explored in your own work, AGI represents a potential tool of immense capability. It is a solution to the game of power, allowing its controller to resolve nearly any game-theoretic dilemma in their favor. The single greatest check on concentrated power throughout human history has been the biological vulnerability and mortality of leaders. No ruler has been immortal; no regime has been omniscient. AGI could sweep those limitations away.
From this perspective, a governance system based on who can accumulate the most capital (i.e., buy the most shares) seems like a terrifyingly arbitrary method for selecting the wielders of such ultimate power. It prioritizes wealth as the key qualification for stewardship, rather than wisdom, compassion, or a demonstrated commitment to the global good.
In conclusion, while I appreciate OGI's pragmatism, I believe its reliance on a shareholder-centric model is a critical flaw. It applies the logic of our current world to a technology that will create a new one, potentially locking us into a future where ultimate power is wielded by an entity optimized for profit, not for the flourishing of humanity.
I don't think people in general react well to societal existential risks, regardless how well or courageous the message is framed. These are abstract concerns. The fact that we are talking about AI (an abstract thing in itself) makes it even worse.
I'm also a very big opponent of arguing by authority (I really don't care how many nobel laureates are of the opinion of something, it is the content of their argument I care about, now how many authorities are saying it). That is simply that I cannot determine the motives of these authorities and hence their opinions, while I can't argue with logic and facts)
Usually it is better to make people can understand the risks in terms of stories, in particular stories they can relate to, hence why people still think of Terminator when thinking of AI exctinction risks.
There is a real (and large) exctinction risk, sure. Then again, the Ape picking up the club in 2001: a Space Odyssey could just as well be accused of going down a path that very likely would result in extinction. But when is extinction risk accepable is a more interesting question, and a question most people are much more ready to answer.
Social messaging is fine balancing act: people like to offload responsibility and effort, especially if it doesn't come at the cost of status. And, to be honest, you don't know if your question would impose upon the other (in terms of cognitive load, social pressure or responsibility), so you it is smart to start your social bid low and see if the other wants to raise the price. Sometimes they work, creating a feedback loop similar to how superstitions evolve: if it is minimal effort and sometimes it is effective, better continue using it.
As a child, I despised a lot of these practices, to me it felt like people were lying all the time, or at least, hiding their true motivations or concerns. I tended to simply call out these adults on their bullshit. If somebody said "I'm fine with everything", I simply proposed something that I know that person is not fine with but that is absurd enough to indicate that I'm not being serious. As a child you can still get away with such behaviour, but many adults find it highly annoying. However, I still employ it among friends who I know don't judge me on that interaction or at least lace it with humor to make it socially acceptable.
However, I think such messaging can often turn into a social communication into a prisoner's dilemma type situation, where each party puts in the minimum succesfull effort resulting in a situation unsatisfactory to either party. I'm just not sure how (and if) we are able to recognize when a situation is a prisoner's dillema and when not. "How was your week?" is often a very welcoming question to me, but not for others.
Leaving things unspoken and relying on generally accepted principles can increase communication efficiency enormously, a lot of communication isn't a prisoner's dilemma type exchange after all, but it will run into issues occasionally, especially if the communicator do not share a set of unspoken rules.
Having grown up in Dutch culture, I was unusually direct (rude) for even a Dutch person, so travelling in Iran where things are absurdly polite at times was very interesting for me, for example. However, a society like Iran requires quite an amount of cognitive load for even simple issues.
Of course it is perfectly rational to do so, but only from a wider context. From the context of the equilibrium it isn't. The rationality your example is found because you are able to adjudicate your lifetime and the game is given in 10 second intervals. Suppose you don't know how long you have to live, or, in fact, now that you only have 30 seconden more to live. What would you choose?
This information is not given by the game, even though it impacts the decision, since the given game does rely on real-world equivalency to give it weight and impact.
Any Nash Equilibrium can be a local optimum. This example merely demonstrates that not all local optima are desirable if you are able to view the game from a broader context. Incidentally, evolution has provided us with some means to try and get out of these local optima. Usually by breaking the rules of the game or leaving the game or seemingly not acting rationally from the perspective of the local optimum.
Please keep in mind that the Chat technology is an desired-answer-predicter. If you are looking for weird response, the AI can see that in your questioning style. It has millions of examples of people trying to trigger certain responses in fora etc, en will quickly recognize what you really are looking for, even if your literal words might not exactly request it.
If you are a Flat Earther, the AI will do its best to accomodate your views about the shape of the earth and answer in a manner that you would like your answer to be, even though the developers of the AI have done their best to instruct it to 'speak as accurately as possible within the parameters of their political and PR views'
If you want to trigger the AI to give poorly written code examples with mistakes in them, it can. And you don't even have to ask it directly, it can detect your intention by carefully listening to your line of questioning.
Once again, it is a desired-answer-predicter/most-likely-response generator, that's its primary job, not to be nice or give you accurate information.
While I acknowledge this is important, it is a truly hard problem, as it often involves looking not just at first-order consequences, but also at second-order consequences and so on. People are notoriously bad at predicting, let alone managing side-effects.
Besides, if you look at it more fundamentally, human natures and technological progress in a broad sense has many of these side effects, where you basically need to combat human nature itself to have people take into account the side-effects.
We are still struggling coming to terms and accurately classifying things like environmental pollution, global warming and such. Understanding the illegible problems -> explaining them to policy makers -> thinking of legible solution -> convincing policy makers -> have policy makers convince their constituents -> having policy makers take action effective timeframe
I see so much issue with those, that I rather solve the issue on how to gett policy makers to take action within a reasonable timeframe, otherwise defining the illegible problems better will most likely only result in a I-told-you-so scenario.