Capabilities and alignment of LLM cognitive architectures

[-]awg3y66

Strong upvoted. A very thorough, yet fairly easy-to-follow (at least...for around here) overview of human cognitive architecture, what we currently know about it, and how it applies to what we're currently building with LLMs and LMCAs as you call them (there's so many different names floating around).

I'm not sure I come to quite the same level of optimism (if that's the right word) on the amount of alignment we'll be able to get out of these types of distributed architectures, but I do agree they seem to lend themselves well to a hodgepodge alignment strategy with many different people working on aligning many different parts of the architecture at once (in addition to the central LLM)...or something.

I am starting to wonder if this is the approach that OpenAI/Microsoft is internally pursuing toward AGI.

[-]Seth Herd3y30

Thank you! I'm really glad you found it relatively easy to follow. That was certainly the goal, but it's impossible to know until others read it.

I think that almost certainly is at least one aspect of Microsoft's strategy. There was some accidental revealing of an extra internal prompt when bing chat was first released; I didnt take down the source unfortunately. And now that you mention it, that would be a great way to apply Microsoft's existing skill pool of coders to improve performance.

That might also be a big thing within OpenAI. Sam Altmann recently said (again I don't remember where) that the age of big models is already over (I'm sure they'll still make them larger, but he's saying that's less of the focus). This could be one of the alternate approaches they're taking up.

I don't know how optimistic I am about the alignment prospects. But I do think it's got more upsides and the same downsides of most other suggestions. Particularly the more practical ones that have a low enough alignment tax to actually be realistically implementable.

[-]awg3y21

My degree is in cognitive science, so that might have given me a leg up, but while it was technical I found it got the good points across (with relevant citations) without getting too deep into the weeds, which is always a sweet spot to hit in cases like these :)

And word, I had written out this short list of (complete ass-pull gut instinct) reasons why I think this will be OpenAI/Microsoft's strategy before deleting them and reframing it as that wondering instead. But now that you bring it up:

The GPT-4 + plugins launch seems like they had that same system internally for quite some time. I think it was part of their testing plan for a while and they built their own, better versions of AutoGPT and decided it was all still pretty dumb and safe and so they released to see what the public would do with them, just like their safety plan states.
The slow roll of GPT-4.2+ also points in that same direction to me. They could be using GPT-4.X as the central LLM reasoning hub in their own version of AutoGPT and figure (maybe correctly??) that bottlenecking the performance of the central hub is the proper way to handicap any layman approaches to AutoGPT. I expect they're testing the performance of GPT-4.2/3/4/whatever in stages for this very purpose before rolling out any updates to the public.
Sam A's comment that the age of large models is over indicates to me that they might be going with this approach: refining/progressing the LLM central hub to a point where it can orchestrate human-level+ STEM across a distributed system of other models.

And re: alignment, agreed.

[-]Nathan Helm-Burger2y40

A relevant recent advance in cognitive scaffolding for LLMs: Self-Discover: Large Language Models Self-Compose Reasoning Structures

[-]Evan R. Murphy3y30

Post summary (experimental)

Here's an alternative summary of your post, complementing your TL;DR and Overview. This is generated by my summarizer script utilizing gpt-3.5-turbo and gpt-4. (Feedback welcome!)

The article explores the potential of language model cognitive architectures (LMCAs) to enhance large language models (LLMs) and accelerate progress towards artificial general intelligence (AGI). LMCAs integrate and expand upon approaches from AutoGPT, HuggingGPT, Reflexion, and BabyAGI, adding goal-directed agency, executive function, episodic memory, and sensory processing to LLMs. The author contends that these cognitive capacities will enable LMCAs to perform extensive, iterative, goal-directed "thinking" that incorporates topic-relevant web searches, thus increasing their effective intelligence.
While the acceleration of AGI timelines may be a downside, the author suggests that the natural language alignment (NLA) approach of LMCAs, which reason about and balance ethical goals similarly to humans, offers significant benefits compared to existing AGI and alignment approaches. The author also highlights the strong economic incentives for LMCAs, as computational costs are low for cutting-edge innovation, and individuals, small and large businesses are likely to contribute to progress. However, the author acknowledges potential difficulties and deviations in the development of LMCAs.
The article emphasizes the benefits of incorporating episodic memory into language models, particularly for decision-making and problem-solving. Episodic memory enables the recall of past experiences and strategies, while executive function focuses attention on relevant aspects of the current problem. The interaction between these cognitive processes can enhance social cognition, self-awareness, creativity, and innovation. The article also addresses the limitations of current episodic memory implementations in language models, which are limited to text files. However, it suggests that vector space search for episodic memory is possible, and language can encode multimodal information. The potential for language models to call external software tools, providing access to nonhuman cognitive abilities, is also discussed.
The article concludes by examining the implications of the NLA approach for alignment, corrigibility, and interpretability. Although not a complete solution for alignment, it is compatible with a hodgepodge alignment strategy and could offer a solid foundation for self-stabilizing alignment. The author also discusses the potential societal alignment problem arising from the development of LLMs with access to powerful open-source agents. While acknowledging LLMs' potential benefits, the author argues for planning against Moloch (a metaphorical entity representing forces opposing collective good) and accounting for malicious and careless actors. Top-level alignment goals should emphasize corrigibility, interpretability, harm reduction, and human empowerment/flourishing. The author also raises concerns about the unknown mechanisms of LLMs and the possibility of their output becoming deceptively different from the internal processing that generates it. The term x-risk AI (XRAI) is proposed to denote AI with a high likelihood of ending humanity. The author also discusses the principles of executive function and their relevance to LLMs, the importance of dopamine response in value estimation, and the challenges of ensuring corrigibility and interpretability in LMCA goals. In conclusion, the author suggests that while LLM development presents a wild ride, there is a fighting chance to address the potential societal alignment problem.

I may follow up with an object-level comment on your post, as I'm finding it super interesting but still digesting the content. (I am actually reading it and not just consuming this programmatic summary :)

[-]Seth Herd3y40

Cool, thanks! I think this summary is impressive. I think it's missing a major point in the last paragraph: the immense upside of the natural language alignment and interpretability possible in LMCAs. However, that summary is in keeping with the bulk of what I wrote, and a human would be at risk of walking away with the same misunderstanding.

[-]Ozyrus3y30

We need a consensus on how to call these architectures. LMCA sounds fine to me.
All in all, a very nice writeup. I did my own brief overview of alignment problems of such agents here.
I would love to collaborate and do some discussion/research together.
What's your take on how these LCMAs may self-improve and how to possibly control it?

[-]Seth Herd3y10

Interesting. I gave a strong upvote to that post, and I looked at your longer previous one a bit too. It looks like you'd seen this coming farther out than I had. I expected LLMs to be agentized somehow, but I hadn't seen how easy the episodic memory and tool use was.

There are a number of routes for self-improvement, as you lay out, and ultimately those are going to be the real medium-term concern if these things work well. I haven't thought about LMCAs self-improvement as much as human improvement; this post is a call for the alignment community to think about this at all. Oh well, time will tell shortly if this approach gets anywhere, and people will think about it when it happens. I was hoping we'd get out ahead of it.

[-]Ozyrus3y30

Thanks.
My concern is that I don't see much effort in alignment community to work on this thing, unless I'm missing something. Maybe you know of such efforts? Or was that perceived lack of effort the reason for this article?
I don't know how much I can keep up this independent work, and I would love if there was some joint effort to tackle this. Maybe an existing lab, or an open-source project?

[-]Seth Herd3y20

Calling attention to this approach and getting more people to at least think about working on it is indeed the purpose of this post. I also wanted to stress-test the claims to see if anyone sees reasons that LMCAs won't build on and improve LLM performance, and thereby be the default stand for inclusion in deployment. I don't know of anyone actually working on this as of yet.

[-]Seth Herd3y10

I hadn't seen your post. Reading it now.

[-]Brendon_Wong3y21

Have you read Eric Drexler's work on open agencies and applying open agencies to present-day LLMs? Open agencies seem like progress towards a safer design for current and future cognitive architectures. Drexler's design touches on some of the aspects you mention in the post, like:

The system can be coded to both check itself against its goals, and invite human inspection if it judges that it is considering plans or actions that may either violate its ethical goals, change its goals, or remove it from human control.

[-]Seth Herd3y10

I had read the first but hadn't seen the second. I just read it. It's well written and highly relevant, and I'll be citing it in my future work. Thanks much for the reference!

[-]Max H3y20

I like the LMCA term, and I agree that systems like them are likely to be the capabilities frontier in the near future. I think aiming to make such systems corrigible while they're in the human-level or weakly superhuman regime is more promising than trying to align them in full generality.

In a recent post, I considered what adding corrigibility to a system comprised of LMCAs might look like concretely.

[-]Roger Dearnaley3y52

A lot of the work people have done in alignment had been based on the assumptions that 1) interpretability is difficult/weak, and 2) the dangerous parts of the architecture are mostly trained by SGD or RL or something like that. So you have a blind idiot god making you almost-black boxes. For example, the entire standard framing of inner vs outer alignment has that assumption built into it.

Now suddenly we're instead looking at a hybrid system where all of that remains true for the LLM part (but plausibly a single LLM forward pass isn't computationally complex enough to be very dangerous by itself), however the cognitive architecture built on top of it has easy interpretability and even editability (modulo things like steganography, complexity, and sheer volume), looks like combination of a fuzzy textual version of GOFAI with prompt engineering, and its structure is currently hand-coded, and could remain simple enough to be programmed and reasoned about. Parts of alignment research for this might look a lot like writing a constitution, or writing text clearly explaining CEV, and parts might look like the kind of brain-architectural diagrams in the article above.

I believe the only potentially-safe convergent goal to give a seed AI is to build something with the cognitive capability of one-or-multiple smart (and not significantly superhumanly smart) humans, but probably faster, that are capable of doing scientific research well, giving it/them the goal of solving the alignment problem (including corrigibility), somehow ensuring that that goal is locked in,and then somehow monitoring them while they do so

So how would you build a LMCA scientist? It needs to have the ability to:

Have multiple world models, including multiple models of human values, with probability distributions across them.
Do approximate Bayesian updates on them
Creatively generate new hypotheses/world models
It attempts to optimize our human value (under some definition, such as CEV, that it is also uncertain of), and knows that it doesn't fully understand what that is, though it has some data on it.
It optimizes safely/conservatively in the presence of uncertainty in its model of the world and human values, i.e has solved the optimizer's curse Significantly, this basically forces it to make progress on the Alignment Problem in order to ever do anything much outside its training distribution: it has to make very pessimistic assumptions in the presence of human value uncertainty, so it is operationally motivated to reduce human value uncertainty. So even if you build one that isn't an alignment researcher as its primary goal, but say a medical researcher and tell it to cure cancer, it is likely to rapidly decide that in order to do that well it needs to figure out what Quality-of-life-adjusted Life Years and Informed Consent are, and the Do What I Mean problem, which requires it to first solve the Alignment Problem as a convergent operational goal.
Devise and perform low cost/risk experiments (where the cost/risk is estimated conservatively/pessimistically under current uncertainty in world+human value models) that will distinguish between/can falsify individual world+human value models in order to reduce model uncertainty.

This strongly suggest to me an LMCA architecture combining handling natural language (and probably also images, equations, code, etc) with quantitative/mathematical estimation of important quantities like probabilities, risks, costs, and value, and with an underlying structure built around approximate Bayesian reasoning.

[-]Mitchell_Porter3y30

You might want to compare your ideas to (1) Conjecture's CoEms (2) brain-like AGI safety by @Steven Byrnes (3) Yann LeCun's ideas.

[-]Seth Herd3y30

I'd done that with 1 and 2, and probably should've cited them. I did cite a recent brief summary of Steve's work, and my work prior to getting excited about LMCAs was very similar to his. In practice, it seems like what Conjecture is working on is pretty much exactly this. I wasn't familiar with LeCun's scheme, so thanks for sharing.

The brainlike cognitive architecture proposed by Steve (and me) and LeCun's are similar in that they're cognitive architectures with steering systems. Which is an advantage. I also wrote about this here.

But they don't have the central advantage of a chain of thought in English. That's what I'm most excited about. If we don't get this type of system as the first AGI, I hope we at least get one with a steering system (and I think we will; steering is practical as well as safer). But I think this type of natural language chain of thought alignment approach has large advantages.

[-]Seth Herd3y20

Excellent! I'll digest that and the steering post. I'm also addressing steering system alignment as somewhat of a shortcut in Human preferences as RL critic values - implications for alignment. I think it's really similar to your approach, as I'm also thinking about a MuZero type of RL system. The map here is the executive function system in an LMCA, but it deserves to be spelled out more thoroughly as you've done in your corrigibility tax post.

^{^}

Beren Millidge’s excellent article describes scaffolded LLMs as natural language computers. He is addressing essentially the same set of potentials in LLMs by having them driven by scripts and interact with external tools, but he addresses this from a thoroughly CS perspective. This complements my perspective of seeing them as loosely brainlike cognitive architectures, and I highly recommend it.

^{^}

David Shapiro coined this term and originated this natural language approach to alignment in his 2021 book Natural Language Cognitive Architecture: A Prototype Artificial General Intelligence, which I haven’t yet read. He probably came up with this approach long before publishing that book, and others have probably talked about a natural language alignment prior to that post, but I haven’t it. I found Shapiro’s work when researching for this article, and I am adopting his cognitive architecture terminology because I think it’s appropriate. The few mentions of his work on Less Wrong are quickly dismissed in each instance.

I do not endorse Shapiro’s proposed “Heuristic Imperatives” as top-level goals. They are: Reduce suffering in the universe; Increase prosperity in the universe; and Increase understanding in the universe.

I’d expect these to wind up creating a world with no humans and lots of smart and prosperous AIs that don’t experience suffering (and never have cool dance parties or swap meets). But, to be fair, Shapiro doesn’t claim that these are the best final form, just that we should have a list, and encourage their adoption by social and economic pressure.

I am not going to propose specific alternatives here, because we should first discuss whether any such scheme is useful. I'd say top-level goals for alignment should probably emphasize corrigibility and interpretability, along with some sort of harm reduction and human empowerment/flourishing.

^{^}

It is unknown how much processing LLMs really accomplish to create each natural language string. And there are concerns that its output could become deceptively different than the internal processing that creates it. This is an important caveat on this approach, and deserves further discussion and interpretability work. The final section mentions some of these concerns.

^{^}

I’m suggesting the term x-risk AI, abbreviated XRAI, to denote AI that has a good chance of ending us. AGI is not specific enough, as GPT4 meets the intuitive definition of an AI that does a bunch of stuff well enough to be useful. I’d like a more strict definition of AGI, but I believe in coining new terms instead of telling people they’re using it wrong.

^{^}

Of course, the interactions between brain systems are highly complex on a neuronal level, and the exact mechanisms have not been worked out. On a high level, however, the principles seem clear. For the interactions I’ve described, it seems as though the limited bandwidth of natural language descriptions and simple APIs will be adequate to do a good deal of cognitive work.

^{^}

This was an explicitly pluralistic version of the widely-used ACT-R cognitive architecture, meaning that the design goal was having cognitive components that could be swapped out for other implementations that accomplished the same function. This is the HuggingGPT approach of having an array of external tools to call. We were using the Leabra algorithm to create deep neural networks, but the actual implementation was limited to a vision network taking on the role of the visual "what" stream in ACT-R. Two papers may be of interest:

SAL: an explicitly pluralistic cognitive architecture (2008)

Integrating systems and theories in the SAL hybrid architectur (2013)

^{^}

For too much more on the precise distinctions in terminology surrounding executive function, see our paper How Sequential Interactive Processing Within Frontostriatal Loops Supports a Continuum of Habitual to Controlled Processing.

^{^}

While Auto-GPT has accomplished little of real use at this early date, I'm impressed by how it seemingly spontaneously tries new approaches after a failure, based on its conclusion of that failure in its context window. More sophisticated approaches are possible, but they may not be necessary.

^{^}

This type of interference may relate to a cognitive neuroscience theory postulating that humans' low working-memory-for-executive-function capacity is actually advantageous in preventing interference. See Rationalizing constraints on the capacity for cognitive control.

^{^}

Dopamine response has long been known to approximate reward prediction error (which is equivalent to a value estimate in actor-critic RL). It is now known that the dopamine response contains other response characteristics, including positive responses for negative occurrences. This is consistent with dopamine as a general error signal that trains many responses beyond reward expectation, but it is consistent with the finding that dopamine’s central function is value estimation, with around 70% of dopamine cells responding in that way. Much more can be found in A systems-neuroscience model of phasic dopamine.

^{^}

The number and length of tool descriptions in the Hugging Face library necessitated a pre-selection step to select more-likely appropriate tool descriptions, since all of the descriptions together exceeded the context window.

^{^}

I have neither the artistic skill nor the time to cajole midjourney to depict a shoggoth wearing a smiley face mask that holds dangerous tools and talks to itself. Help would be appreciated. The lore I’m remembering may not be canon, it’s been a while. Fortunately, fictional shoggoths aren’t relevant. Unfortunately, the level of sentience and goal-directedness of current and future LLMs is also unknown.

^{^}

Whoops, giving an LMCA a goal of not changing its goals could conflict with its goals of corrigibility and interpretability, since letting a user inspect its thoughts might result in the user changing its goals. This stuff is going to be tricky. I hope nobody launches a GPT-5 LMCA based on my suggestions without reading the footnotes.

LESSWRONG
LW

LESSWRONG
LW

88

Capabilities and alignment of LLM cognitive architectures

88

Ω 26

88

Ω 26

Overview

Cognitive capacities enabled and enhanced by LLM wrappers and extensions:

LMCAs have agency

LMCAs have executive function

Prompts as executive function

LLMs alone have little or no executive function

Varieties of executive function

LMCAs have episodic memory

Episodic memory ⊗ executive function

Complex decision-making: Decision-Making ⊗ Executive Function ⊗ Episodic Memory

LMCA sensory and action systems

LMCA nonhuman cognitive capacities

Implications for alignment