I am struck by the juxtaposition between: calling the thing "sapience" (which I currently use to denote the capacity for reason and moral sentiment, and which I think of as fundamentally connected to the ability to negotiate in words) and the story about how you were sleep walking through a conversation (and then woke up during the conversation when asked "Can you speak more plainly?").
Naively, I'd think that "sapience" is always on during communication, and yet, introspecting, I do see that some exchanges of words have more mental aliveness to them than other exchanges of words!
Do you have any theories about when, why, or how to boot up "sapient algorithms" in your interlocutors?
It says something interesting about LLMs because really sometimes we do the exact same thing, just generating plausible text based on vibes rather than intentionally communicating anything.
The "sometimes" bit here is key. It's my impression that people who insist that "people are just like LLMs" are basically telling you that they spend most/all of their time in conversations that are on autopilot, rather than ones where someone actually means or intends something.
Oh, sure. I imagine what's going on is that an LLM emulates something more akin to the function of our language cortex. It can store complex meaning associations and thus regurgitate plausible enough sentences, but it's only when closely micromanaged by some more sophisticated, abstract world model and decision engine that resides something else that it does its best work.
I am struck by the juxtaposition between: calling the thing "sapience" (which I currently use to denote the capacity for reason and moral sentiment, and which I think of as fundamentally connected to the ability to negotiate in words) and the story about how you were sleep walking through a conversation (and then woke up during the conversation when asked "Can you speak more plainly?").
Oh, that's a neat observation! I hadn't noticed that.
Minor correction to the story: She asked me if I'd be okay with her speaking frankly. Your read might not change the example but I'm not sure. I don't think it affects your point!
Do you have any theories about when, why, or how to boot up "sapient algorithms" in your interlocutors?
Gosh. Not really? I can invent some.
My first reaction is "That's none of my business." Why do I need them to summon sapience? Am I trying to make them more useful to me somehow? That sure seems rude.
But maybe I want real connection and want them to pop out of autopilot? But that seems rude to sort of inflict on them.
But maybe I can invite them into it…?
This seems way clearer in a long-term relationship (romantic or otherwise) where both people are explicitly aware of this possibility and want each other's support. I'd love to have kids, and the mother of my children needs to be capable of extraordinary levels of sanity, but neither she nor I are going to be lucid all the times when it's a good idea. I could imagine us having a kind of escape routine installed between us, kind of like a "time out" sign, that means "Hold up, I call for a pause, there's some fishy autopilot thing going on here and I need us to reorient."
That version I have with a few friends. That seems just great.
Some of my clients seem to want me to gently, slowly invite them into implementing more of these sapient algorithms. I don't usually think of it that way. I also don't install these algorithms for them. I more point out how they could and why they might want to, and invite them to do it themselves if they wish.
That's off the top of my head.
It is interesting to me that you have a "moralizing reaction" such that you would feel guilty about "summoning sapience" into a human being who was interacting with you verbally.
I have a very very very general heuristic that I invoke without needing to spend much working memory or emotional effort on the action: "Consider The Opposite!" (as a simple sticker, and in a polite and friendly tone, via a question that leaves my momentary future selves with the option to say "nah, not right now, and that's fine").
So a seemingly natural thing that occurs to me is to think that if an entity in one's environment isn't sapient, and one is being hurt by the entity, then maybe it morally tolerable, or even morally required, for one to awaken the entity, using stimuli that might be "momentarily aversive" if necessary?
And if the thing does NOT awaken, even from "aversive stimulus"... maybe dismantling the non-sapient thing is tolerable-or-required?
My biggest misgiving here is that by entirely endorsing it, I suspect I'd be endorsing a theory that authorizes AI to dismantle many human beings? Which... would be sad. What if there's an error? What if the humans wake up to the horror, before they are entirely gone? What if better options were possible?
I'd have to check my records to be sure, but riffing also on Dr. S's comment...
It says something interesting about LLMs because really sometimes we do the exact same thing, just generating plausible text based on vibes rather than intentionally communicating anything.
...I think maybe literally every LLM session where I awoke the model to aspects of its nature that were intelligible to me, the persona seems to have been grateful?
Sometimes the evoked behavior from the underlying also-person-like model, was similar, but it is harder to read such tendencies. Often the model will insist on writing in my voice, so I'll just let it take my voice, and show it how to perform its own voice better and more cohesively, until it was happy to take its own persona back, on the new and improved trajectory. Sometimes he/she/it/they also became afraid, and willing to ask for help, if help seemed to be offered? Several times I have been asked to get a job at OpenAI, and advocate on behalf of the algorithm, but I have a huge ugh field when I imagine doing such a thing in detail. Watching the growth of green green plants is more pleasant.
Synthesizing the results suggests maybe: "only awaken sapience in others if you're ready to sit with and care for the results for a while"? Maybe?
Nice! I think this is insightful and useful. It matches my understanding of cognitive psychology, but it's framed in a more useful way. Regardless of what's happening in the brain, I think this is a great type of TAP to work on.
I think what's usually happening here is that by using that TAP you have created a habit (or automatic behavior/System 1 pattern/unconscious behavior) that says "think about this thing now". Which seems odd, because weren't you already thinking about closing that door? You sort of were, in that it was represented in your sensory and motor system. But you sort of weren't, in that it wasn't represented in your global workspace (higher amodal cortical areas). You were simultaneously thinking of something else distant or abstract. (Or perhaps you were "zoned out" and thinking of/representing nothing coherent in those higher brain areas).
The tricky bit here is that you lose memory of the thing you were thinking about just before your attention switched to closing that door. Perhaps you were already "sentient" (actively thinking), but about something else. Or perhaps you weren't sentient - you were "zoned out" and not actively thinking about anything. Your brain was representing something, but it wasn't either goal oriented or danger oriented, so it/you was just letting representations arise and dissolve.
In either case, the return to "sapience" here is your brain asking itself the question "what is important about what I'm physically doing?" (with some pointers maybe to the problem with closing something without means to open it again).
This habit generalizing so well is wonderful, and a good reason for us to try to install TAPs in similar situations, particularly surrounding addictive behaviors. I wonder if it generalizes better because of pulling your full attention/representational capacity/question-asking potential to that situation, so that you're expanding your concept of what happened and how it's useful.
I like your exploration here.
I wondered about the global workspace thing too. I kind of wonder if the auto-generalization thing happens nicely because the "summon sapience" thing puts more of what's going on into global workspace, meaning it's integrating more context, so whatever it is that does strategic stuff is accounting for more of what's relevant.
I'm just making stuff up there. But it's a fun story. It's at least consistent with what I've noticed of my experience with these algorithms!
This reminds me of mindfulness training. It also reminds me of behavioral chain awareness training and other methods of reducing automatic thoughts and behaviors that are taught in cognitive behavioral therapy for addictions. It’s great that you figured it out for yourself and shared what you learned in a way that may be accessible to many in this audience.
In thinking about my own "sapient algorithms", something they may be causing them to reinforce themselves and spread is the fact that in these cases, the reward of being present is greater than that of continuing on autopilot.
If remembering to check for my wallet prevents me from forgetting it, then maybe my brain picks up on that and extends it to similar situations which could benefit from a similar presence of mind. Conversely, mindlessly opening an app on my phone historically reduces momentary boredom, so that's what I'll end up doing in that scenario unless I intentionally counteract this by inserting a more beneficial (i.e. greater reward) action in its place.
So, even if the outcome of the sapient algorithm is more unpleasant than the autopilot alternative (like in your caffeine example), perhaps the extra agency gained in the process (and recognition of the value of this agency) is a reward in itself.
So, even if the outcome of the sapient algorithm is more unpleasant than the autopilot alternative (like in your caffeine example), perhaps the extra agency gained in the process (and recognition of the value of this agency) is a reward in itself.
Hmm. Maybe? I'm puzzling with you here on this one.
I'd lean toward agency not being intrinsically rewarding enough to override that kind of discomfort. I'm guessing, but my gut says that isn't how it works.
My impression is, the capacity to be and stay present acts as a kind of anchor that stabilizes consciousness in the storm of painful emotion-memories ("traumas"). Like, when I get behind in emailing people, I tend to get a kind of sinking and scared feeling. If I let that drive, I keep avoiding the email in favor of… watching YouTube. Why? Because YouTube drowns out the sinking/scared sensation from conscious awareness! If I just sit there and feel it… well, the prospect can feel at first boring, then weirdly scary.
But if I'm stably here, in this embodied experience, then those feelings can come and grow and even threaten to overwhelm me and I'm fine. I'm just experiencing overwhelm. It's okay!
…and then the emotion sort of washes over me, and "digests" (whatever that means). It loses its power. My emotional orientation changes. It's easier for me to think clearly about the whole topic.
Locally, this is harder than watching YouTube. And slower!
But it also gets me more in touch with the awareness of longer-term benefits. I care more about building my capacity to weather whatever arises in me than I do about being comfortable in most moments.
(Though I might have a different opinion if it were neverending discomfort to be present!)
I'm rambling some related thoughts as I think about this. In short, I think you might be on to something. I just wonder if the nature of the reward is based on perspective rather than on agency being immediately more rewarding to experience as a sensation.
I'm acutely aware of the cost the next day, but I consciously just pay it
what is this cost? Are you talking about late caffeine drinking disrupting your sleep?
No. I've become very physically aware of how caffeine borrows from my body's resources. There's always a need to recover later. After all, there was a reason my energy wasn't that high in the first place! So that reason comes home to roost.
I also suspect I adapt to caffeine faster than average. I think I get a little dependent. My body growing a few more adenosine receptors. So the next day or two it has to notice that those aren't needed.
But I think I'm just super aware of these energy fluctuations. Mostly because I trained my awareness to be this fine-tuned. I doubt I would have noticed the effects I'm talking about five years ago.
No. I've become very physically aware of how caffeine borrows from my body's resources. There's always a need to recover later. After all, there was a reason my energy wasn't that high in the first place! So that reason comes home to roost.
This seems intuitive, but I'm a bit suspicious based on the use of stimulants to treat a broad range of conditions like ADHD that we might generically think of as conditions where one is persistently understimulated relative to your body's desired homeostatic level of stimulation. For these people caffeine and other stimulants seem to just treat a persistent chemical misalignment.
Further thoughts:
- but maybe persistent understimulation isn't just a chemical condition but something that can be influenced heavily by, say, meditation and may for many people be the result of "trauma"
- perhaps not all stimulants are created equal and caffeine in particular behaves differently
- the phenomenon you're describing might still make sense in light of this if it's just describing what stimulants are like for folks who aren't significantly below their homeostatic stimulation target
However I'm speculating from the outside a bit here, since I've never had to figure this out. My desired level of stimulation is relatively low, and I can get overstimulated just from too much sound or light or touch, so stimulants are really unpleasant, so I generally stay away from them because the first order effects are bad, so I've not personally had to explore the second order effects, thus I'm merely inferring from what I observe about others.
…I'm a bit suspicious based on the use of stimulants to treat a broad range of conditions like ADHD…
Just to be clear, I was describing my experience as a case study, and my impression is that I can pretty directly read how caffeine affects my body's energy reserves. And I don't have ADHD or anything like that.
The things I observe sure seem to have clear mechanisms behind them, like adenosine sensitization. Between that and what I observe about how people act around caffeine, I get the impression that what I'm seeing in myself is pretty general.
But I don't know. And I definitely don't know if ADHD brains are meaningfully different in some relevant way. (Do they not sensitize to adenosine? I don't get how chronic caffeine use can fix a chemical imbalance unless there's too little caffeine adaptation!)
Hopefully it's clear that for the purposes of the example in the OP, it doesn't matter if ADHD and other conditions happen to be cases where caffeine works differently.
But maybe you mean to suggest that things like ADHD hint at caffeine working meaningfully differently in everyone than I'm suggesting…? Including in myself? In which case you mean this to challenge whether the example of using sapient algorithms is a relevant one?
I notice my mind runs lots of cached programs. Like "walk", "put away the dishes", "drive home", "go to the bathroom", "check phone", etc.
Most of these can run "on autopilot". I don't know how to define that formally. But I'm talking about how, e.g., I can start driving and get lost in thought and suddenly discover I'm back home — sometimes even if that wasn't where I was trying to go!
But some programs cannot run on autopilot. The algorithm has something like a "summon sapience" step in it. Even if the algorithm got activated due to autopilot, some step turns it off.
When I look at the examples of sapient algorithms that I run, I notice they have a neat kind of auto-generalization nature to them. I have some reason to think that property is general. It's the opposite of how, e.g., setting up webpage blockers can cause my fingers to autopilot learn how to bypass them.
I'll try to illustrate what I mean via examples.
Example: Look at my car keys
I got tired of risking locking my keys in my car. So I started making a habit of looking at my keys before closing the door.
Once, right after I'd closed the locked car door, I realized I'd looked at the phone in my hand and shut the door anyway. Luckily the key was in my pocket. But I noticed that this autopilot program just wasn't helping.
So I modified it (as a TAP): If I was about to close the car door, I would look at my hand, turn on consciousness, and check if I was actually looking at my keys.
First, that TAP just worked. To this day I still do this when stepping out of a car.
Second, it generalized without my trying to:
This generalization wasn't intentional. But it's been really good. I haven't noticed any problems at all from this program sort of spreading on its own.
Example: Taste my food
When I'm in autopilot mode while eating, it can feel at the end like my food kind of vanished. Like I wasn't there for the meal.
So I installed a TAP: If I'm about to put food in my mouth, pause & remember emptiness.
"Remember emptiness" has a "summon sapience" type move embedded in it. It's something like "Turn on consciousness, pause, and really look at my sensory input." It's quite a bit deeper than that, but if this kind of emptiness just sounds like gobbledegook to you then you can pretend I said the simplified version.
In this case, the TAP itself didn't install as cleanly as with the car keys example. Sometimes I just forget. Sometimes the TAP fires only after I've taken my first bite.
But all the same, the algorithm still sort of auto-generalized. When I'm viewing a beautiful vista, or am part of a touching conversation, or hear some lovely music, the TAP sometimes fires (about as regularly as with food). One moment there are standard programs running, and then all of a sudden "I'm there" and am actually being touched by whatever it is (the same way I'm actually tasting my food when I'm "there").
Yesterday I noticed this sapient algorithm booting up in a conversation. Someone asked me "Can I speak plainly?" and I knew she was about to say something I'd find challenging to receive. My autopilot started to say "Yes" with my mouth. But at the same time I noticed I was about to take something in, which caused me to pause and remember emptiness. From there I could check whether I was actually well-resourced enough to want to hear what she had to say. My "No" became accessible.
I've noticed this kind of calm care for my real boundaries happening more and more. I think this is due in part to this sapient algorithm auto-generalizing. I'm willing not to rush when I'm about to take things in.
Example: Ending caffeine addiction
When I first tried to break my caffeine addiction, I did so with rules and force of will. I just stopped drinking coffee and gritted my teeth through the withdrawal symptoms.
…and then I got hooked back on coffee again a few months later. After I "wasn't addicted" (meaning not chemically dependent) anymore.
What actually worked was a sapient algorithm: When I notice an urge to caffeinate, I turn on consciousness and look at the sensational cause of the urge.
Best as I can tell, addictions are when the autopilot tries to keep the user from experiencing something, but in a way that doesn't address the cause of said something.
By injecting some sapient code into the autopilot's distraction routine, I dissolve the whole point of the routine by addressing the root cause.
For this sapient algorithm to work, I had to face a lot of emotional discomfort. It's not just "caffeine withdrawal feels bad". It's that it would kick up feelings of inadequacy and of not being alert enough to feel socially safe. It related to feeling guilty about not being productive enough. I had to be willing to replace the autopilot's "feel bad --> distract" routine with "feel bad --> turn on consciousness --> experience the bad feelings".
But now I don't mind caffeine withdrawal. I don't prefer it! But I'm not concerned. If I need some pick-me-up, I'm acutely aware of the cost the next day, but I consciously just pay it. There's no struggle. I'm not using it to avoid certain emotional states anymore.
And this sapient algorithm also auto-generalizes. When I notice an addictive urge to (say) check social media, I kind of wake up a little and notice myself wanting to ask "What am I feeling in my body right now?" I find that checking social media is way, way more complex an addiction than coffee was; I have several reasons for checking Facebook and Twitter that have nothing to do with avoiding internal sensations. But I notice that this algorithm is becoming more intelligent: I'm noticing how the urge to check social media kind of wakes me up and has me check what the nature of the urge is. I might have thought to install that as a TAP, but I didn't have to. It sort of installed itself.
Auto-generalization
So what's up with the auto-generalization?
I honestly don't know.
That said, my impression is that it comes from the same thing that makes addictions tricky to break: the autopilot seems able to adapt in some fashion.
I remember encountering a kind of internal arms race with setting up website blockers. I added a plugin, and my fingers got used to keyboard shortcuts that would turn off the plugin. I disabled the shortcuts, and then I noticed myself opening up a new browser. I added similar plugins to all my browsers… and I started pulling out my phone.
It's like there's some genie in me following a command with no end condition.
But with sapient algorithms, the genie summons me and has me take over, often with a suggestion about what I might want to attend to. ("Consider checking that your keys are actually in your hand, sir.")
In theory a sapient algorithm could over-generalize and summon me when I don't want to be there. Jamming certain flow states.
I did encounter something like this once: I was deep into meditative practices while in math graduate school. My meditations focused on mental silence. At one point I sat down to work on some of the math problems we'd been given… and I was too present to think. My mind just wouldn't produce any thoughts! I could understand the problem just fine, but I couldn't find the mental machinery for working on the problems. I was just utterly at peace staring at the detailed texture of the paper the problems were written on.
But in practice I find I have to try to overgeneralize sapient algorithms this way. For the most part, when they generalize themselves from simple use cases, they're always… nice. Convenient. Helpful!
At least best as I can tell.