The classical meaning for AGI is things like humans rather than chimps or calculators and the implied potential impact; and for ASI being qualitatively more intelligent and immediately more impactful than humanity. The explosion of discussion around LLMs eroded the terms, but these are still the key thresholds of impact.

AGI is a point where humans are no longer needed for some kind of civilization to keep developing towards superintelligence and further, and superintelligence is a point where humanity's efforts are washed away by its capabilities, and so it's definitely too late to fix anything the superintelligence wouldn't cooperate in fixing.

Drowning in nuance doesn't change the utility of these simple concepts where they centrally apply. But the words might need to change at some point to protect the concepts from erosion of meaning, if it gets too bad.

When we have a particular query at hand, we don't need the broad terms, and that's great. But we do need shorthands when defining a particular query doesn't fit the time/space/attention span. Having a wider variety of sub-terms within the community would be extremely helpful, but I'm not sure how to work toward that.

I think about this a fair amount. I think terminology matters, and having terms of art is practically necessary. We're not going to restate a functional definition in every article, let alone every comment. AGI centrally means "the type of AI that changes everything", which is the central topic of discussion in the safety community for good reason. 

I agree that the term AGI is used in so many different ways that it's problematic and probably not useful. I think we'd benefit a lot by having some standard reference terminology that's carefully defined.

To that end, I wrote a couple of articles trying to give a better definition, which includes talking about what type of cognitive capabilities are necessary to create a system that "thinks for itself" and is smarter than humans in most useful ways. I wrote Sapience, understanding, and "AGI" on this in 2023, and "Real AGI" more recently.

One problem I discovered in my attempts to create better defined terms is that the project was wrapped up in theories about exactly how AI will progress to have transformative properties. I expect a certain set of cognitive capacities to be the turning point, so I define it with those. Others have different theories. Which would be fine if we had multiple definitions in circulation. Transformative AI is an attempt to sidestep addressing the particulars of AI capabilities that will be important, but I think the relative disuse of that term suggests that addressing that question is pretty critical to most of the discussion.

I think the term ASI is usually used distincly (at least on LessWrong) and is pretty useful; it usually points to something more capable than the first things you'd call AGI, and can designate takeover-capable AI (one of my recently preferred terms for pointing to the important properties).

Maybe if somebody with sufficient reputation wrote a post saying "hey let's settle on some more specific sub-terms" that would work. But I suspect that would feel locally like a waste of time to that busy person.

There's a joke that academic philosophers would rather use each other's toothbrushes than each other's terminology. This tendency creates a profusion of different terms, and subtly different uses of the same terms. We could at least improve on that by having some reference posts intended to define the terms in particular ways.

One of the cruxes here is whether one believes that "AGI" is in fact a real distinct thing, rather than there just being a space of diverse cognitive algorithms of different specializations and powers, with no high-level organization. I believe that, and that the current LLM artefacts are not AGI. Some people disagree. That's a crux.

If you're one of the people who thinks that "general intelligence" is a thing, then figuring out whether a given system is an AGI or not, or whether a given paradigm scales to AGI, can be a way to figure out whether that system/paradigm is going to be able to fully automates the economy/AI research. An AGI would definitely be able to do that, so "LLMs are an AGI" implies "LLMs will be transformative". "LLMs are not AGI" does not imply "LLMs won't be transformative/omnicide-capable", so determining the AGI-ness doesn't give you all the answers (such as the exact limits of the LLM paradigm). Nevertheless, "is this an AGI?" is still a useful query to run^[1], and getting "no" does somewhat update you away from "LLMs will be transformative", since it removes some of the reasons they may be transformative. 

Tertiarily, one question here may be, "what the hell do you mean by a 'real AGI' as distinct from a 'good-enough AGI approximant'"? Here's a long-winded semi-formal explanation:

I deny the assumption that behavioral definitions of AGI/ASI are akin 
to denying the distinction between the Taylor-polynomial approximation of a function, and that function itself (at least if we assume infinite compute like this).

Consider a function $f\left(x\right)$ that could be expanded into a power series $p\left(x\right)$ with an infinite convergence radius, e. g. $f\left(x\right)=e^x$.^[2] The power series is that function, exactly, not an approximation; it's a valid way to define $f\left(x\right)$.

However, that validity only holds if you use the whole infinite-term power series, $p_{\infty }\left(x\right)$. Partial sums $p_n\left(x\right)$, with finite $n$, do not equal $f\left(x\right)$, "are not" $f\left(x\right)$. In practice, however, we're always limited to such finite partial sums. If so, then if you're choosing to physically/algorithmically implement $f\left(x\right)$ as some $p_n\left(x\right)$, the behavior of $p_n\left(x\right)$ as you turn the "crank" of $n$ is crucial for understanding what you're going to achieve.

Suppose that we define an AGI as some function $F\left(x\right)$. As above, $F$ can have many exactly-equivalent representations/implementations, call their set $R\left(F\right)$. One of them may be some entity $G_n\left(x\right)$ such that ${lim}_{n\to \infty }{G}_n\left(x\right)=F\left(x\right)$.

Does it matter which member of $R\left(F\right)$ we're working with, if we're reasoning about $F$'s behavior or capabilities? No: they are all exactly equivalent. Some members of $R\left(F\right)$ may define $F$ behaviorally, some may do it structurally; we don't care. We pick whichever definition is more convenient to work with.

However, implementing $F$ in practice – incarnating it into algorithms and physics – is a completely different problem. To do so, we have to pick some structural representation/definition from $R\left(F\right)$, and we have to mind its length/complexity, because it'll be upper-bounded (by our finite compute).

In this case, the pick of representation matters. Notably, for some $k\ne \infty$, $G_k$ is not a member of $R\left(F\right)$. If so, we have to start wondering about the error $\text{dist}(G_k;F)$, and how it behaves as we move $k$ around, and what is the largest $k$ such that $G_k$'s description complexity is below our upper bound.

For example, perhaps $G_n\left(x\right)$ is an expansion of $F\left(x\right)$ around some abstract "point" $x=a$, and while $G_{\infty }$ exactly equals $F$ everywhere, $G_k$ only serves as a good approximation around that point $a$, with the error growing rapidly as you move away from $a$. (We may call that "failure to generalize out of distribution".)

If so, nitpicking about the definition/representation/implementation of $F$ that is being used to incarnate $F$ absolutely matters.

For some members of $R\left(F\right)$, say $Q_{\infty }$, even their complexity-limited approximations $Q_k$ are good enough everywhere. Formally, we may say that $\forall x:\text{dist}\left(F\right(x);Q_k(x\left)\right)<ϵ$, for some small $ϵ$. The informal statement "$Q_k$ is a genuine AGI" is then only slightly imprecise.

For other members of $R\left(F\right)$, like $G_{\infty }$, their complexity-limited approximations $G_k$ are not that good. Formally, $\exists x:\text{dist}(F;Q_k)>T$, for some large $T$ and large sets of $x$. The informal statement "$G_k$ is an AGI" is then drastically incorrect: it essentially says $\forall x:\text{dist}\left(F\right(x);G_k(x\left)\right)<ϵ$. Translated into practical considerations, reasoning about $F$ is not a good way to predict $G_k$'s behavior.

(Edit: Further, if what you care about are practically feasible AGI designs, you may restrict the set $R\left(F\right)$ only to those definitions/representations whose small-error approximations have a finite description length. Hence you may somewhat-imprecisely say that "$G_n$ is not a real AGI" – since it's true for all finite $n$ and $n=\infty$ is implicitly ruled to be outside the consideration.

I note that this is what I was doing in this comment, and yeah, my using the practical and the theoretical definitions of $R\left(F\right)$ is unhelpful. I'll try to avoid that in the future.)

1. ^{^}

   Since figuring out whether it's an AGI/AGI-complete may be easier than predicting its consequences from the first principles.

   Formally, suppose all members of some set $S$ have a property $q$, and suppose we're wondering whether some entity $f$ has that property. Proving $f\in S$ is a valid way to prove that $f$ possesses $q$, and it may be easier than proving that in a more "direct" way (without referring to $S$).

2. ^{^}

   Formally, assume $f\left(x\right)$ is an entire function.