Sorry, I should have clarified that the news was US GDP Growth: https://www.bea.gov/news/2019/gross-domestic-product-third-quarter-2019-second-estimate-corporate-profits-third-quarter
This idea has become part of my conceptual toolkit for discussing / describing a key failure mode.
(Note: the below part of the comment is from my Facebook comment about the article when it came out.)
There's a great connection you make to bureaucracy, and it's definitely worth exploring.
This gives me a good language to discuss something I've noted a number of times. I'd posit that selection pressure for bureaucracy limits how stupid the system gets as a function of the best simple alternative, and the difficulty of transitioning to it without turning off the system. This means that for critical systems where there is no incremental pathway to improve, it's near-permanent even if there are better alternatives - see the US healthcare system. For less critical systems, once an alternative is found, as long as the transition isn't too long/too expensive, and there are incentives that actually promote efficiency, it will happen. The critical fact is that the incentives need to exist for everyone that is involved - not just the end user. So if bob in accounting doesn't like the change, unless someone else can induce cooperation (like senior management,) it never happens.
This post has influenced my evaluations of what I am doing in practice by forcing me to consider lowering the bar for expected success for high return activities. Despite "knowing" about how to shut up and multiply, and needing to expect a high failure rate if taking reasonable levels of risk, I didn't consciously place enough weight on these. This helped move me more in that direction, which has led to both an increased number of failures to get what I hoped, and a number of mostly unexpected successes when applying for / requesting / attempting things.
It is worth noting that I still need to work on the reaction I have to failing at these low cost, high-risk activities. I sometimes have a significant emotional reaction to failing, which is especially problematic because the emotional reaction to failing at a long-shot can influence my mood for multiple days or weeks afterwards.
Until seeing this post, I did not have a clear way of talking about common knowledge. Despite understanding the concept fairly well, this post made the points more clearly than I had seen them made before, and provided a useful reference when talking to others about the issue.
This post has been a clear example of how rationality has and has not worked in practice. It is also a subject of critical practical importance for future decisions, so it frequently occurs to me as a useful example of how and why rationality does and does not help with (in retrospect) critical decisions.
This post has significant changed my mental model of how to understand key challenges in AI safety, and also given me a clearer understanding of and language for describing why complex game-theoretic challenges are poorly specified or understood. The terms and concepts in this series of posts have become a key part of my basic intellectual toolkit.
I don 't think this is straightforward in practice - and putting a cartesian boundary in place is avoiding exactly the key problem. Any feature of the world used as the item to minimize/maximize is measured, and uncorruptable measurement systems seems like a non-trivial problem. For instance, how do I get my GAI to maximize blue in an area instead of maximizing the blue input into their sensor when pointed at that area? We need to essentially solve value loading and understand a bunch of embedded agent issues to really talk about this.
There is also overhead to scaling and difficulty aligning goals that they want to avoid. (As above, I think my Ribbonfarm post makes this clear.) Once you get bigger, the only way to ensure alignment is to monitor - trust, but verify. And verification is a large part of why management is so costly - it takes time away from actually doing work, it is pure overhead for the manager, and even then, it's not foolproof.
When you're small, on the other hand, high-trust is almost unnecessary, because the entire org is legible, and you can see that everyone is (or isn't) buying in to the goals. In typical startups, they are also well aligned because they all have similar levels of payoff if things go really well.
My claim is that *competence* isn't the critical limiting factor in most cases because structure doesn't usually allow decoupling, not that it's not limited. When it IS the limiting factor, I agree with you, but it rarely is. And I think alignment is a different argument.
In EA orgs, alignment can solve the delegation-without-management problem because it can mitigate principal-agent issues. Once we agree on goals, we're working towards them, and we can do so in parallel and coordinate only when needed. In most orgs, alignment cannot accomplish this, because it's hard to get people to personally buy into your goals when those goals are profit maximization for a company. (Instead, you use incentive structures like bonuses to align them. But then you need to monitor them, etc.)
On your points about scaling, I mostly agree, but want to note that there are fundamental issues with scaling that I explained in a post here: https://www.ribbonfarm.com/2016/03/17/go-corporate-or-go-home/
The post is rather long. In short, however, I don't think that your Kingdom metaphor works, because large bureaucracies are big *not* because they have many mini-kingdoms doing similar things in parallel, but because they need to specialize and allow cross-functional collaboration, which requires lots of management.