(This post is probably only of interest to neuroscientists. I’m mostly writing it in the hopes that someone more knowledgeable will chime in and help me out. There’s a comments section at the bottom, or email me.)

(See updates at the very bottom—I might have an answer now.)

tl;dr

In animals, specific innate reactions are reliably triggered by corresponding specific smells—for example, odors associated with natural predators tend to trigger avoidance behavior, even in the absence of any prior experience of those odors. In order for this to work, I think odor information needs to get from the nose to either the hypothalamus or brainstem, without passing through any of a long list of regions that includes the amygdala and the whole cortex. I’m struggling to figure out what this pathway is, if any. I offer my best current guesses as to what’s going on.

Background

Why I expect direct projections of smell (like all other senses) to the “Steering Subsystem”

It’s well-known that animals have numerous specific innate reactions that are triggered by specific smells. For example, odors associated with species-typical predators or unhealthy food may trigger avoidance, odors associated with species-typical healthy food may trigger approach and eating, odors emitted by conspecifics may trigger mating, aggression, or other behaviors, and so on.

Meanwhile, I continue to believe that a large fraction of the brain, which I call the “Learning Subsystem”, including the whole cortical mantle, striatum, cerebellum, and some other stuff, “learn from scratch”, a term that I’m using in a very specific way defined here; and meanwhile I think the rest of the brain, which I call the “Steering Subsystem”, particularly including the hypothalamus and brainstem, is a repository of innate “business logic” such as “if I’m fertile, increase my sex drive”, as discussed here.

For sensory input processing, there’s a nice story that goes along with that two-subsystems picture. The sensory input (I claim) has to split, with one copy going to the Learning Subsystem, and another going to the Steering Subsystem. The former system treats the input as input data for a learning algorithm, and the latter system uses that input to calculate specific ecologically-relevant things to trigger corresponding reactions. This split is critical, for theoretical reasons explained in §3.2.1 here (I won’t repeat it here). And this hypothesis seems to work really well for other senses: For example, visual information goes both to visual cortex in the Learning Subsystem and the superior colliculus in the Steering Subsystem (via the thalamus^[1]); taste goes to both gustatory cortex in the Learning Subsystem and the gustatory nucleus of the medulla in the Steering Subsystem; and so on.

Relevant basics on smell neuroanatomy

…But I’m more confused about smell—particularly how it gets to the Steering Subsystem.

Let’s start with some background on smell. The first step is “olfactory sensory neurons” which can actually detect odorants. Quoting a textbook:

The sensory neurons are embedded in a specialized olfactory epithelium that lines part of the nasal cavity, approximately 5 cm${}^2$ in area in humans. … The axons of olfactory sensory neurons project to the ipsilateral olfactory bulb [where they] terminate on the dendrites of olfactory bulb neurons within bundles of neuropil called glomeruli that are arrayed over the bulb’s surface…. In each glomerulus, the sensory axons make synaptic connections with three types of neurons: mitral and tufted projection (relay) neurons…and periglomerular interneurons, which encircle the glomerulus.…In each glomerulus, the axons of several thousand sensory neurons converge on the dendrites of approximately 40 to 50 relay neurons. … Each glomerulus, and each mitral and tufted relay neuron connected to it, receives input from just one type of odorant receptor. The result is a precise arrangement of sensory inputs from different odorant receptors, one that is similar between individuals. (source)

Incidentally, it’s not too relevant here, but some readers might be wondering: what’s the deal with pheromones? At the first stage of processing, the olfactory epithelium and its olfactory sensory neurons are split into a “vomeronasal organ” (where pheromones are detected) and a main compartment (where every other smell is detected). Then at the second stage of processing, the olfactory bulb, there’s a corresponding split into an “accessory olfactory bulb” for the pheromones and a “main olfactory bulb” for every other smell. If memory serves, the pheromone system is present in humans, but there’s evidence that it’s inactive and vestigial (i.e., not actually connected to anything). But there’s some controversy there. I dunno. For present purposes, I’m equally interested in both.

OK, so far we have taken the first step from olfactory sensory neurons to the mitral and tufted neurons of the olfactory bulb. And we’re still in a great position to set up innate reactions—there has been no randomization, as far as I can tell, and therefore a specific predator urine scent (for example) will excite the same specific mitral and tufted cells in every individual of a species. So in theory, those specific mitral and/or tufted cells could trigger an appropriate innate reaction (avoidance, in this case). My question is: how exactly?

So what’s the next step? Where do those mitral and tufted cells go when they leave the olfactory bulb? The answer seems to be: they go multiple places. Typical of the cortex-supremacist neuroscience literature, the source that I quoted above only talks about the projections from olfactory bulb to the cortex. So we have to turn instead to other sources.

Scattered and sketchy claims in the literature of an olfactory-bulb-to-hypothalamus direct projection

The thing that most cleanly fits my prior beliefs would be a direct projection from the olfactory bulb to the hypothalamus or brainstem. And pretty early on in my investigation of this topic, I found a claim of exactly that! (In rats.) The paper is: “Ghrelin-containing neurons in the olfactory bulb send collateralized projections into medial amygdaloid and arcuate hypothalamic nuclei: neuroanatomical study” by Cristina Russo et al. (2018) (link). They seem to state pretty clearly that they found evidence (by retrograde tracing) of a direct connection from olfactory bulb mitral cells to the arcuate nucleus of the hypothalamus.

So, upon finding this paper a year or two ago, I ran a little victory lap and mentally filed innate smell reactions into the category of “I seem to have at least a vague working hypothesis for what’s going on here, even if I’m hazy on the details”. But then recently I was trying to figure out something, and needed to know more details about innate smell, so I revisited the topic, and decided that my victory lap had been premature. Here are some of my newfound concerns:

• The neurons they measured are very likely specific to food-related odors evoking hunger-related behaviors. I say this based on the fact that they were measuring ghrelin-producing neurons, and they project to the arcuate nucleus which contains ghrelin-detecting neurons which seem to specifically trigger hunger (this is the population of neurons I was talking about here). So this connection probably isn’t directly relevant to other innate smell-triggered behaviors like predator aversion, mating, and so on. It’s suggestive that maybe they found one of several similar direct connections, but if the other connections exist, then nobody has ever measured them, as far as I can see.
• Their “discussion” section has a head-scratching sentence: “There is still no evidence of direct connections from [main olfactory bulb mitral cells] to the arcuate core of the hypothalamus….” Huh?? Did they not read their own paper? My best guess right now is that, in this sentence, they were intending to describe the state of the literature before their own study, and it’s just badly worded. But I remain concerned that I’m wildly misinterpreting the paper.
• We can weigh this paper against a mountain of literature on the olfactory system that doesn’t mention any direct connection from the olfactory bulb to the hypothalamus. What am I supposed to make of that? Did lots of researchers look for it and find that it wasn’t there? Did they read this Russo 2018 paper but didn’t believe it? What are we expecting? I’m not sure, but this surely counts as some nonzero evidence that I should be skeptical of this paper. Regrettably, I lack the expertise to read a neural tracing paper and judge whether the authors were carefully following best practices. By and large, my sense is that different tracing experiments tend to agree on major neuroanatomical connections, but I find that it is not uncommon for two different groups to report conflicting results about the presence or absence of a minor direct connection. So I figure, these measurements must be possible to mess up somehow, even if I don’t know the details.

The next paper I found (cited by Russo 2018) is “Hypothalamus-olfactory system crosstalk: orexin A immunostaining in mice” by Jean Gascuel et al. (2012) (link). They offer the following helpful diagram:

I’m looking for direct connections from “OB” (olfactory bulb) or “OE” (olfactory epithelium) to anywhere in the big gray area marked “hypothalamus”. The only one shown is the blue arrow labeled “EBOP”. What’s that? The authors write: “The EBOP (Extra Bulbar Olfactory Pathway) is made up of sensory neurons that project directly to the POA in the hypothalamus. However, this pathway has only been identified in lower vertebrates.” (I believe “lower vertebrates” mostly means “fish”.) I’m mostly interested in mammals, so I guess this isn’t relevant. Unless there’s secretly an EBOP in mammals, that nobody has noticed yet. Seems pretty unlikely to me, but who knows, I guess.

So putting aside fish, all I can find is Russo 2018 above—one lonely paper in the entire neuroscience literature (to my knowledge) claiming that there’s a direct projection from the olfactory bulb to the hypothalamus, and moreover making that claim in a weirdly ambiguous way. Every other source I’ve seen doesn’t mention any such projection, although again I’m not sure how much they’ve looked for it and found it to be absent.

So, maybe this paper is best ignored. What are my other options?

Other possibilities

What if there’s a direct connection from the olfactory bulb to some other part of the Steering Subsystem?

I think this would entail either:

• There’s a direct connection from the olfactory bulb to the brainstem, and either nobody knows about it or nobody mentions it (or I’m reading the wrong papers, misinterpreting them, etc.).
• …Or I’m wrong about which parts of the brain are in the Learning Subsystem versus Steering Subsystem.
• …Or I’m just wildly confused about everything.

The second bullet point is highly plausible! It’s happened before! Including within the last year.

So, to pick an example, there are direct connections from the olfactory bulb to the olfactory tubercle. I’ve been assuming that the olfactory tubercle is “kinda like striatum” and therefore presumably part of the Learning Subsystem, mostly on the basis of shared embryological origin and cytoarchitectural similarity. But maybe that’s wrong! It’s not an area I know much about.

Probably more relevant than olfactory tubercle is the amygdala. It seems to be a popular opinion in the olfaction literature (example, example) that part of the amygdala (specifically the “medial amygdaloid nucleus”) is part of the core innate smell pathway. I currently have a very strong belief that some parts of the amygdala are Learning Subsystem, where they do supervised learning of dozens or hundreds of little trained models for things like “in thus-and-such situation, it’s a good time to emit digestive enzymes”. But “the amygdala” is really a heterogeneous mish-mosh of stuff that lumps together structurally- and functionally-unrelated neurons, just because they happen to be near each other (details). (Back when early scientists were making up anatomical terms, they were often confused about the natural, embryological structure of the brain.) I don’t actually know much about the medial amygdaloid nucleus per se, instead I was calling it “Learning Subsystem” by tentatively generalizing from other parts of the amygdala that I’m more familiar with (basolateral and central)—which I just said is a problematic thing to do. So, it’s possible, as far as I know right now, that the medial amygdala is actually full of Steering Subsystem innate “business logic” stuff, not supervised learning algorithms. I haven’t found much helpful literature on this so far.

What if some of the olfactory-bulb-to-Learning-Subsystem signals are “supervisory” rather than “context”?

(For background on what I mean by “supervisory” and “context”, see here.)

Based on various properties of the mitral and tufted cells—for example, the fact that they don’t produce dopamine—I would be a bit surprised if this proposal were the case. But I don’t rule it out. I’m finding very little information on the gory details of how those neurons connect to their downstream targets.

Something else?

I can think of at least one other weird implausible possibility, and maybe there are more things I’m not thinking of.

Where I’m at right now

These are weakly-held opinions, but I guess my current odds are:

• 60% that if I were to look much more into the medial amygdala, I would decide to redistrict it into the Steering Subsystem, unlike many other parts of the amygdala which I remain quite confident are Learning Subsystem.
• 50% that the Russo 2018 paper is right, and there really are direct projections from the olfactory bulb to the hypothalamus and/or brainstem, despite their apparent almost-complete absence from the literature. 
• 20% something else.

(These add up to more than 100% because more than one could be true.)

(Update Nov 29 2023: Here’s another idea. AFAICT, the medial amygdala and olfactory tubercle are striatum-like. Could they be a learning algorithm that's initialized as a trivial identity function (i.e. a relay that trivially maps each input onto a corresponding output)? Maybe! I think the striatum is a much simpler learning algorithm than e.g. the cortex and cerebellum, with only one “layer” (in the ML sense) and no pattern-separation. So the idea that its freshly-initialized state is kinda an information-preserving “relay”—and that freshly-initialized state can then get modified by a striatum-like learning algorithm—strikes me as plausible, or at least no worse than anything else I’ve come up with, based on what I know right now.)

(Update Dec 8 2023: Ooh, this is promising. There’s a poorly-studied part of the rodent olfactory bulb called “the necklace glomeruli”. This 2016 thesis by Taralyn Tan has a nice discussion. Upstream of these glomeruli is a set of olfactory sensory neurons that seem to be tailored to detecting behaviorally-significant odors—predator odors, pheromones, and so on—and which have some unusual properties including violating the usual “one receptor-one neuron” rule typical of other olfactory sensory neurons. What about downstream of these glomeruli? Well, according to this thesis, the necklace glomeruli (unlike other parts of the olfactory bulb) project to at least one place that I have long been thinking of as Steering Subsystem (i.e., the same broad category as the hypothalamus and brainstem), namely the medial septum / diagonal band area. Very intriguing! That could be the answer I was looking for—or at least, part of it. Still need to look into it more, but it’s a great lead. Thanks to Alex Wiltschko for the tip, and to Adam Marblestone for putting me in touch with Alex!)

(Update Aug 19 2024: Hey, Russo et al. 2018 is not alone in the literature! I have now found a second claim that there are olfactory bulb neurons that project directly to the hypothalamus: “Connectivity from OR37 expressing olfactory sensory neurons to distinct cell types in the hypothalamus” (Bader et al., 2012).)

1. ^{^}

   Is the thalamus in the Learning Subsystem or Steering Subsystem? Umm, it’s complicated. But the relevant thing here is that the retina-to-thalamus-to-superior colliculus (brainstem) pathway is capable of doing specific innate calculations on sensory inputs, because (this part of) the thalamus is a clean predictable relay. In other words, despite the stopover at the thalamus, there is still a deterministic 1-to-1 relation between specific signal lines from the retina and corresponding signal lines entering the superior colliculus. So the superior colliculus can correspondingly do specific innate visual calculations like “is there something in the visual field that’s scuttling like a spider?”. Yes, I know, the thalamus does some stuff to the signal in between the retina and the superior colliculus, but I don’t think it does so much that the superior colliculus can’t do that job via a specific genetically-specified scuttling-detector calculation circuit. By contrast, a connection through any part of the cortical mantle (as an example) would scramble the signal way too much for that to work, I think—thanks to things like pattern separation.