Introduction

In the Spring of 2022, Stuart Russell wrote an essay entitled If We Succeed, in which he questioned whether and how the field of AI might need to pivot from its historical goal of creating general-purpose intelligence to a new goal, of creating intelligence that would be provably beneficial for humans. He noted that although the former goal had driven a great deal of progress, it was rapidly on its way to becoming myopic, counterproductive, and even dangerous. 

It is not rational for humans to deploy machines that pursue fixed objectives when there is a significant possibility that those objectives diverge from our own. 

-Stuart Russell

I believe that a similar situation may now be arising in neuroscience. It is a case where neuroscientists may need to pivot from focusing on the historical goals of their field to thinking about what happens as the world actually gets closer to achieving them. 

But why should anyone on Less Wrong be concerned about what is happening in neuroscience? Readers of this forum tend to be most concerned about progress in AI, and the need for AI researchers to pivot, which Russell talked about. And, that is for understandable reasons. The rapid ascent of AI models up the leaderboards, which can feel so concerning, is being driven by AI researchers and AI companies, not neuroscientists. What is happening in neuroscience might seem quaint or irrelevant by comparison.

However, there have turned out to be remarkable parallels between what is happening in neuroscience and AI, or at least, the subfield of neuroscience now often known as neuroAI. Both neuroAI scientists and AI scientists have gotten to the present moment by pursuing very similar methods. These are the methods of deep learning and deep neural networks. As such, neuroscientists and AI scientists are facing similar problems, and the perspectives they have developed are complimentary and synergistic.

For example, one problem that concerns AI researchers currently, which is forcing them to reconsider their priorities, is the question of how AI programs should be interpreted. If we can't even interpret these programs, as we so rapidly improve them, then that increases the risks that they might pose significant risks and dangers. 

But neuroscientists have pursued a very similar interpretability question, themselves, in their own way. That's because they have sought to build deep neural networks into the form of models of brain regions. As such, they have become experts in interpreting just how well deep neural networks (or more generally, AI programs) can be mapped onto neurobiological substrates. And since 2014, they have found astonishing evidence in favor of the existence of such mappings. They have found that AI programs can be made that have extremely deep correspondences with certain brain regions, in the form of signal correlations, functional correlations, and explanatory relationships.

Because of these connections, I do not think that what is going on in neuroscience (or especially neuroAI) should be seen as quaint or irrelevant to AI safety. But also, I think that the concerns of AI safety should likely start to be taken seriously within neuroscience. Because it would seem that neuroscientists face a worst case scenario that is very similar to the scenario faced by AI researchers, in the near term. For example, whereas AI experts like Russell might believe we should be concerned about unaligned general intelligences, the neuroscience evidence suggests that we should be concerned about the creation of what I will call hyperintelligent machine sociopaths (HMSs). 

The question is whether or not both of these worst case scenarios will be realized in the form of modern AI programs. However, whereas the concept of an unaligned general intelligence has the advantage of being a powerful, general abstraction, the HMS concept has the advantage of being much easier to explain to non-experts. More importantly, it appears as though it may be just as rigorously supported by the research—for the particular tech tree that humanity has happened to go down.

In the rest of this essay, I will do my best to explain and justify why neuroscience might be facing a similar situation as Russell alluded to. I am bound to get some things wrong or incorrect, or to overstate or to understate certain things, and so on. I'd appreciate your feedback.

But if I could add just one more thing, upfront, to this introduction, it would be to emphasize that the issue that Russell described with the field of AI is as much a social and an economic issue as it is technical. It is not just that no one knows how to create 'provably beneficial artificial intelligence.' It is also that there are many social and economic obstacles to doing so. In many ways, I admire the users of this forum for making an admirable effort to pushback against those issues.

But we should note that in neuroscience (or neuroAI), the social and economic barriers are just as great, or even greater. Perhaps for these reasons, there has yet to be a comparable figure in neuroscience to Stuart Russell, who has written about the need of their own field to pivot, and why such a pivot might be so critical. And my feeling is that the importance of these two fields for each other is being overlooked by many. But that should perhaps not be surprising. Neuroscience and AI have been undergoing rapid changes, and the biological sciences especially are used to moving much more slowly.

However, this is not to say that no one in neuroscience is thinking about AI safety. Steven Byrnes has written an impressive series on the subject. The Part 3 of his series may be particularly relevant. And, other researchers in neuroscience, here and there (to cite just a couple examples), appear to be gradually taking a greater interest.

My main goal here is to make the story of what has been happening in neuroscience a little more accessible. I was forced to do this, for myself, when I came across this subject as a science journalist, and took an interest in understanding it. That led me to try to undertake a journalism book project, which I released a couple weeks ago, but doesn't look likely to receive funding. That's disappointing, but my biggest concern has always been just to try to write a story about what's been happening, and why it might be so interesting and important—perhaps most of all, for AI safety.

Progress in Neuroscience

Why is it that neuroscientists might be facing a challenge in their field similar to the one that Russell described? To answer this question, it is helpful to zoom out and first look at their field's central objectives. 

Historically, the central objective of neuroscience has been to understand the mind, and especially, the way that the brain and nervous system give rise to it. 

One of the main ways they have pursued this objective, just like all other scientists, is by building models and simulations of their objects of interest, like neurons, brain regions, and even whole brains. 

It is obvious that if neuroscientists had started building realistic simulations of whole brains, then the field would rapidly need to pivot to thinking about ethical, moral, and safety considerations. They would be creating computer versions of intelligent individuals who could both cause and sustain great negative impacts, absent any interventions or regulations.

However, historically speaking, there has been little need for neuroscientists to think about making such a pivot, because their grand objectives have always seemed so distant.

But recently, the frontier of neuroscience has been shifting. Neuroscientists have started to make highly significant progress towards achieving some of their core objectives, like creating realistic models and simulations of large-scale brain regions—if not whole brains. These regions include the visual cortex, responsible for all our visual capabilities, and the so-called language network, which allows for the basic language comprehension and generation.

Needless to say, these brain regions or cortexes are some of the most powerful and important parts of what make us human. To build realistic models or simulations of them would be a historically unprecedented achievement. It would be so unprecedented that indeed, most neuroscientists are still coming to grips with how to evaluate their own progress, which is more partial, more in a gray area, more complicated, and perhaps most importantly, extremely recent. 

As a consequence, many neuroscientists consider current state-of-the-art models of these regions as dubious, especially when it comes to more recent work, as for example, building models of the language network. Emblematic of this perspective, they do not tend to refer to their models of these brain regions as 'simulations,' except with a few notable exceptions. However, that is a liberty that I will be intentionally taking, for the remainder of this essay, and something I say much more about in my project, because I believe it is an important part of the overall social situation, which in a sense, I am seeking to question. 

The field of neuroscience is undergoing a seismic shift, from seeing the brain as an arcane mechanical object of almost unspeakable complexity, to seeing it as something explained remarkably well as a product of optimization. 

-Me, Chapter 1, AI: How We Got Here—A Neuroscience Perspective

I will go into more depth on the neuroscience evidence in the last section of this essay, and I also go into it in great depth in my project materials. But from my personal point of view, based on my reading of the evidence, it feels likely that what is happening in neuroscience may be a matter of great social and ethical importance; or if it is not yet, it will be soon, say, in the next five years. 

Because if we have actually obtained the ability to build brain regions, or even if we have only gotten close to obtaining it, then this, by itself, seems to pose not only unprecedented capabilities, but also the potentials for serious dangers. 

The Problems with Progress

The overarching problem with progress in neuroscience, which parallels the problem in AI, is that this progress has been uneven. Namely, neuroscientists have made great strides in building models of some brain regions and some brain systems, like the visual cortex, auditory cortex, and the language network, but not all of them. And yet these other, still mysterious machineries of our brain may be what are most important.

For example, there are many other distinctive brain regions, like the so-called theory of mind network, responsible for social reasoning, that neuroscientists have little clarity on how to build yet. Further, neuroscientists are not yet sure how all these systems—even if you could build them—are integrated. And, there are many other systems of great importance, like the neurotransmitter system, involving the well known biochemicals like dopamine and serotonin, which are totally missing from existing models and simulations. (Reflective of these gaps in knowledge, neuroscientists still do not know how to create a realistic simulations of any whole nervous system, in its entirety, such as that belonging to the worm, C. elegans.)

But, as we know, humans are known to rely on the composite functioning of all these different brain regions and brain systems for healthy functioning. When these regions are missing or dysfunctional, then we consider medical interventions necessary. For example, about one in eight members of the United States population takes drugs known to target neurotransmitter chemistry. They do this not for fun, but because they consider it imperative to being a healthy human.

To consider an isolated but large-scale brain region is therefore to consider something highly lobotomized, fragmented, and problematic, compared to a human. We would refer to a human possessing such a system as being severely brain damaged. Such a human might possess a human-level mastery of language, for example, but only a superficial facsimile of emotional reasoning processes, which might cause them to make gravely harmful decisions.

But if neuroscientists were only creating such isolated brain regions in the laboratory, or analogs of them, then it would probably not be a problem. Neuroscientists are specialists, with a wealth of experience in dealing with ethical questions. And, these models are not conscious. They are more like single, isolated cognitive modules, the likes of which are not found in nature. (Again, this is another reason why it is still early for the neuroscience community to be talking about them. They simply don't have the language.)

Consider however that neuroscientists have made these putative brain region simulations in the form of deep neural network programs—exactly the same sort of programs made by AI researchers. And, the programs made by AI scientists and neuroscientists share a great deal in common. 

For example, the first model that neuroscientists discovered to be capable of explaining some of the measured signals from the human language network was—guess what—a language model, adapted from the work of AI scientists (see e.g. Schrimpf et al, 2021). 

To state things differently, neuroscientists have shown that deep neural networks can in certain cases be rigorously interpretable as closely related to real brain regions. 

But then, if neuroscientists have shown the science of DNNs to be the science of making braintech, then that means a fragmentary and distorted form of synthetic braintech has already become totally widespread and proliferated, through the actions of the commercial AI industry. 

I use the word synthetic to describe the creations of the AI industry, because these are distinctive from the models of neuroscientists, for example, in their connection graphs, their scale of training data, and in their numbers of artificial neurons. The evidence from AI suggests that such synthetic models are in some ways already hyperintelligent—when seen as single modules—compared to the single brain regions of humans. 

To make matters worse, the commercial AI industry is now actively in the process of integrating these powerful synthetic cortexes—or what you might think of as 'brain-class' programs—into crude and simplistic agent architectures. In other words, the suggestion from neuroscience seems to be that we are already in the process of mass producing hyperintelligent machine sociopaths (HMSs).

We don't tend to look at modern AI programs in this way, perhaps because these things have been branded as chatbots, and given a veneer of submissiveness and docility, due to the application of a finishing stage, where we train such programs to reproduce human preferences. But consider that it would be just as easy to create—using the exact same, standard methods—an LLM of extremely evil language appearances. The LLM does not really care about what it's saying; it has none of the machinery of ethics or emotions, or at least, that looks to be the finding.

To reiterate, this interpretation of LLM agents as HMSs is not just some kind of intuitive, hand waving, ambiguous claim. It is based on hardcore evidence from neuroscience, including the tendency for models trained on single goals (like text prediction) to most often show close signal and functional correspondences with only certain, isolated brain regions (see e.g. Mahowald et al, 2024, section 'Non-augmented LLMs fall short on functional linguistic competence'). 

To be clear, the evidence for HMSs is still only recent, and in much greater need of study. It is still far from clear how optimizing a DNN for a certain training goal winds up making it related to certain parts of the brain, and not others, or whether a single optimization goal might lead to the emergence of multiple brain regions. 

But to summarize what is known with greater certainty, existing agentic systems—and those currently in development—may be expected to possess much brain-level machinery for cognition, but nothing like the brain's machinery for safe agency, ethics, and volition. 

Thus, to be a little speculative, the old sci-fi visions of things like cold and heartless terminators seem to have been strangely prescient. Perhaps humanity has always known, within itself, what are our most important components, and what therefore might be the most difficult to replicate—to our own detriment.

An Absence of Discourse

Obviously, if you take this reading of the neuroscience research, then it would seem imperative for neuroscientists to bring their insights about AI into the AI safety discussion. And also, for neuroscientists to start thinking about AI safety, and taking it as a major focus. To do otherwise would be to make a giant dereliction and disregard of nature's solution to the alignment problem, which was obviously so important. 

But if such a problem is really happening, then why wouldn't more neuroscientists be entering the discussion? I believe it's fair to say that of the thousands or tens of thousands of members of the community, only an extreme minority is expressing interest in this topic. 

I'll admit, I'm not sure why so few neuroscientists have gotten involved in the discourse around AI safety. I do know that some of them have, and are actively trying to do so. Perhaps I'm mischaracterizing the field, or overgeneralizing. I certainly can't speak for the entirety of the neuroscience community, and wouldn't try to do so. I can only comment on what I have experienced and seen myself, as a science journalist, working on the project for only a limited time frame.

But, if I could speculate on the reasons for this silence, I would suggest that the reasons keeping neuroscientists from participating in the discussions are both simple and complex. The simplest reasons might be that the community simply hasn't gotten there yet. Neuroscientists might simply still feel that more evidence is needed, or, they might have different judgements about the evidence, and may not yet have come to consensus. 

However, I would argue that it might be a time for the community to be a little more speculative and forward looking. There would seem to be substantially less risk in doing that, then in going ahead and letting HMSs potentially come about, without saying anything. 

The more complex reasons why neuroscientists aren't speaking up are what are most interesting. But before I mention a little about all these various technical, economic, and social considerations, I think it would probably be helpful for me to describe a little about how I came to consider them, in the first place.

My Personal Observations

I first came across some hints of what was going on in neuroscience in the fall of 2021, while working as a science journalist. More specifically, I was working on a story about the advent of powerful new language models, like GPT-3, and the new methods of creating them, such as scaling. For that story, I interviewed many scientists, including Ilya Sutskever, who mentioned that if you took the poetic analogy between biological and artificial neural networks seriously, then it provided an intuitive way to interpret why the scaling paradigm had been so successful. 

These initial discussions about links between BNNs and ANNs made me curious. I understood, from my basic technical background, that ANNs were never expected to be very closely related to real brain regions. They seemed far too over simplistic. But, I wanted to understand what the experts were saying, and so, I started googling the neuroscience literature. What I found felt very surprising.

As I learned, since 2014, neuroscientists had begun to discover that highly optimized neural networks provided a powerful abstraction for making models of large-scale brain regions. In a now classic study from 2014, neuroscientists at MIT discovered that the signals in the macaque monkey visual cortex, measured while the monkey was engaged in recognizing objects, could be well explained by the signals in neural networks, which had been highly optimized for performing the same task.

Although neural networks had long been used in the biological sciences, before this point, and especially by the group of practitioners who called themselves connectionists, I came to understand that this 2014 study (and others from around that time) marked a major shift in their application. Formerly, the neural network model had always been seen as something crude, something vaguely inspired by real biological neural networks, something vaguely allusive of them. Perhaps useful for information processing, yes, but not all that relevant to understanding or modeling brain systems. 

But beginning in 2014, researchers had begun to show that artificial neural networks—when coupled with the methods of deep learning, like depth, and highly efficient optimization—were capable of displaying great correspondences with real brain regions. Not just functional or superficial correspondences, but deeply biophysical links between them, like the correlations of neural activity.

As I learned, from that time onwards, neuroscientists have discovered ever more evidence to suggest that AI programs can be made in the image of brain regions—and to a degree that is extremely striking. Using such AI programs, they have now made the first predictive models of many different distinctive brain regions. And, an entirely new subfield has sprang up to focus on the study of these models, called neuroconnectionism, or NeuroAI, or cognitive computational neuroscience, with new professors of the subject being minted almost everywhere, in all the world's leading neuroscience faculties.

However, one thing I noticed as a journalist, as I came across all this progress, was that the members of this new subfield were also largely reserved in how they talked about what was happening. I simply never came across anyone, in my interviews with 30+ neuroscientists, who questioned where all this progress was headed, and whether it might be problematic, or who ever described their models as simulations. Why was that?

Technical Fixations

As best as I can tell, the biggest single reason for the silence of neuroscientists on the AI safety subject has been a fixation on the technical developments. In other words, the science has just been too recent, and the community has not yet come to a consensus that it should shift to a focus on the social implications.

And, this technical fixation is pretty understandable, especially in a historical context. Research in this area has been extremely recent, complex, and difficult to interpret. Even still, it has been difficult and time-consuming for the community to make progress, and the biological sciences are slow moving, anyways. Even the existing evidence has led to substantial arguments and disagreements, as when trying to establish whether existing models of the visual cortex should actually be seen as good models of it. Given this abundance of technical questions in the present, it has been difficult for the field to think about the forward looking implications.

Consider that it wasn't until 2021 that neuroscientists first discovered that there were significant correlations between language models and the language network (see e.g. Schrimpf et al, 2021). And, it is only recently that neuroscientists have begun to reflect on the highly fragmentary nature of such models—the way they tend to correspond to isolated brain regions (see e.g. Mahowald et al, 2024, section 'Non-augmented LLMs fall short on functional linguistic competence'). It is only recently that they have started asking whether such models might possess a greater 'emergent modularity' than single human brain regions.

There is just so much that we don't know yet. 

The Pull of Russell's Argument

From my own point of view, I have to admit—perhaps a little shamefully—I hardly ever questioned this fixation of neuroscientists on the technical issues. Given the recency of the research, and the lack of consensus, it could seem in poor taste and speculative to ask about the far-reaching implications of it. It was understandable, at least to me, for neuroscientists to be cautious and reserved when assessing it.

So it was that when I released my journalism project around three weeks ago, on January 15, I myself had barely thought about the problematic implications of neuroscience progress, or the need for neuroscientists to pivot. In fact, I took the unevenness of progress as a reason to be optimistic. Because I figured that if we were still far from creating true human-like intelligences, then that suggested that we need not be as worried about AI progrress.

However, as I sought to tell others about the project, and share why the neuroscience research seemed important, I couldn't help but feel the pull of Russell's thinking on the subject. Because the inability to build whole brains, when you think about it, does not stop you from building profoundly dangerous HMSs. And when you look at what's going on in AI from a neuroscience perspective, it seems like that's what might be happening. 

So, in other words, it has started to feel impossible for me to ignore the problematic implications. From my own point of view, this simply does not feel, in other words, like a story of a dry and abstract scientific achievement, which is frankly the way that 99% of neuroscientists talk about the academic research in this area. This is a story that, for me, at this moment, feels a little more like the invention of gunpowder, or maybe even the atom bomb, in terms of its potential for profoundly negative impacts.

In other words, and not to compare myself, a lowly science journalist, with Stuart Russell, one of the world's most distinguished AI scientists, but I've had to realize that perhaps what's most important about the neuroscience is exactly the things that have tended to be omitted.

Other Reasons for Silence

Again, there is the question that if these links between neuroscience and AI are so important, then why so little discourse?

Ultimately, I can really only speculate. I've ran out of funding to ask these questions. I would tend to guess that many neuroscientists would consider my thinking on this subject to be overly speculative, or, they might dispute the evidence. And, I would consider that to be entirely fair feedback—and I'd love to hear it. There is so much I do not understand. So, at this point, until I find more debate with neuroscientists, I would say it is altogether possible that I am making a mistake in my assessment. 

However, what I think is more likely is that the worldwide community of neuroscience has been struggling to overcome social, historical, and technical factors, and that those in fact might be the greatest reasons for why most neuroscientists have been quiet.

For example, consider that the biological sciences are historically extremely conservative, compared to the AI sciences. It is very difficult to make progress in neuroscience, compared to AI science, because of the costliness and difficulty of doing experiments on living organisms.

And yet, even those in AI who have raised questions about risks and concerns have often been met with skepticism and derision. It doesn't seem like scientific communities like hearing about such things. For example, Russell only penned his essay in 2022, and at that point, many in AI were still being mocked (and are still being mocked, today) for thinking of their new language programs as anything more than trivial 'pattern recognizing' engines. There are also profoundly qualified individuals (John Carmack being one who comes to my mind) who appear to be more-or-less unconcerned about AI safety problems, and their opinions should be taken into account. 

In neuroscience, the social situation is that much more complex. There are still so many technical questions with the neuroscience research that it makes it extremely difficult to raise one's head to the horizon, and to look where things are headed. Neuroscientists have still not even come to consensus that their models of brain regions should even be thought of as simulations—even when these models are bottom up models of biological neural networks, predictive of the signals from electrocorticography and MRI experiments, show deep functional correspondences with isolated brain regions, have been optimized for the same tasks as biological neural networks, etc. etc. etc.

And, it is practically a tautology, but consider that if one wishes to question the  the ethical implications of current progress, then this question must be directly based on evaluating and assessing the extent of current progress. But, for a working neuroscientist, especially one who is early in their career, it is extremely difficult to make claims of progress. They face enormous pressure to respect the norms of the field, and to be extremely cautious about making such claims. Perhaps it is fair to say they are all a little traumatized by their peer review process? And in public, any such claim that is deemed to go too far is to be expected to be met with strong protest and pushback. This simply is the social milieu of neuroscience, and probably most sciences. 

All of these complex social dynamics are something I would love to investigate. But unfortunately, I feel in hindsight, I did not prioritize them as much as I should have. I was mainly trying to parse out the evidence, without considering the absurd-seeming possibility that even the neuroscientists might not be focusing on what was actually most important. But, in hindsight, that was exactly the sort of thing that Russell was talking about. 

Where We Go From Here

If I had to draw forth a conclusion from my humbling adventure working on this project, it may be that we could all greatly benefit from a better understanding of what's been happening in neuroscience, and a greater involvement of neuroscientists in the discussions about AI safety. I know many in this forum already feel similarly, and I was inspired when someone directed me to the writing of Steven Byrnes in his Intro to brain-like-AGI safety series. 

For example, in Part 3 of his work, Byrnes articulates a view of neuroscience in which he observes progress in building cortical systems, or what he calls the 'learning system,' as massively outpacing progress in building other brain systems, which he calls the 'steering system'. As such, he already raised a flag of sorts—several years ago—of a very similar nature to what I am describing, where an unevenness of progress has led to problems.  

Nevertheless, I was somewhat startled to come across Byrnes' account, because even though I feel that much of what I'm talking about is steeped in controversy, you could also see it as a totally non-controversial effort to try to corroborate and expand on various parts of what others, like him, were saying. Albeit, doing so in a way that was much more non-technical, conventional, and geared towards non-experts.

If researchers travel down the most easy and natural path—the path that looks like the AI and neuroscience R&D community continuing to behave in ways that they behave right now—we will wind up being able to make AGIs that do impressive things that their programmers want, for a while, but are driven by radically alien motivation systems.

-Steven Byrnes, [Intro to brain-like-AGI safety] 3. Two subsystems: Learning & Steering

I'll probably no longer be working on this project, as I've run out of funding. However, the Kickstarter fundraiser will still be active for a few more days, and I would greatly appreciate your support. Or more generally, I'd appreciate feedback and discussion. 

But, regardless of my personal involvement in this research, greater rapprochement between neuroscience and AI seems necessary. 

Ultimately, if we as a society are going to achieve safe and beneficial general purpose intelligence, then I do think we all need to work together, and we need to make room for various scientific communities to catch up, as it were, in the way I have described here. We also need to all catch up to their research, which may offer so much, for example, in interpreting the potential problems of AI safety.

Anyways, that neuroscience should have so much to offer to AI safety should not be surprising. Neuroscientists are exactly the community of people who already specialize in asking ethical questions about intelligence. Their input would be indispensable, and if society was to try to put together a scientific dream team to go to the United Nations, and demand for AI regulation, then it certainly seems like one or more neuroscientists going on that committee would make it far stronger and more persuasive. 

Perhaps the biggest challenge is simply going out and asking neuroscientists these hard questions—asking them whether they need to pivot, and whether they need to grapple with the implications of their own successes, which had for so long looked so far away.