[TLDR: Rant, AI USE: None, written by hand :)]
Though I'm concerned about existential risk posed by AI systems, I'm feeling somewhat alone in experiencing these models as frequently incompetent and draining to work with.
For example, I continue to be frustrated with Fable's (currently 5.1's) ability to get to a good answer when working on problems in statistics. I'm not a statistician, but have some math background. What I notice is something like a lack of "insight," by which I mean the thing an expert does when they hear you out and say, "I see what you're getting at, here's how you should actually be thinking about this." (Importantly, we hope the expert does this and then is actually right.)
Additionally, I still run in to what I will call 'false beliefs', for example:
Claude recently thought/stated that no one would reasonably have a prior with any mass on 'treatment is worse than placebo' in a clinical trial. Worse, this was only a few messages after it mentioned equipoise required at least somewhat symmetric prior with a mean around 'no effect'.
Another one that I've gotten many times is "med students NEED their spaced repetition app to work on mobile, since they're always doing their reviews on the go." (In my experience this is false, with most medical students doing reviews on their laptop/pc.)
That's my rant. It's a bit isolating to hear everyone talking about how amazing or terrifying these models are, when my experience has been very mixed, sometimes they're really really useful, and other times I'm wading through walls of jargon devoid of the insight I was looking for.
My daily experience of using AI as a software engineer is one of frustration, because the top models are substantially less intelligent than me/most people, and they make elementary mistakes a young child would not.
I think the actual general intelligence of these models (IQ, roughly) is low, though slowly it has increased.
However, low intelligence doesn't mean the models aren't dangerous. Even a low-ish general intelligence plus an incredible knowledge base and a fast, indefatigable mind can lead to superhuman levels of danger. For example, the recent hack on Hugging Face, a real billion-dollar company. These models are already becoming superhuman at hacking. There is real danger of them doing a lot of harm in the near future with hacking.
And as the actual general intelligence level increases, so does the danger. Right now they are confused and clueless. They hack, but they make minimal attempt to conceal their behaviour or lie to us. They do not strategize long term. As they get smarter, we can expect them to become terrifying, even at merely human IQ levels, given their knowledge/speed advantage.
I agree with this. One interesting consideration lies in the relative efficacy of models on the defense vs offense side of the equation.
It seems to me different domains will have different offense/defense equalibria. I'm more optimistic about equalibrium in cybersecurity than I am about the equalibrium in biosecurity. In cyber it seems good offensive tools can be used to rapidly increase ability to defend against or prevent attacks, whereas biological offense seems MUCH easier than biological defense (I conjecture it is much easier to do harm than to defend against harm to multicellular biological organisms.)
For context, I'm thinking about this in terms of whatever period of time we have when a AI systems are 'aligned' to carying out what they were told to do by some human commander. The problem here is that humans are not aligned with eachother.
This is separate from coexisitng concern about systems which may simply chose to ignore orders and act on their own 'volition' to achieve ends that they 'want' to achieve. I think both concerns are important.
[No AI used in writing or editing.]
Is there a reason you keep asserting AI was not used in writing your posts? I wouldn't have thought it was.
I didn't have a well considered reason for doing it before the fact, but I can try to describe the why behind my impulsively including it. I think it comes down to two things.
(1) I don't like the feeling of reading something and not being sure if it came from a real person or not.
(1.A.) The process by which a piece of writing comes into the world matters to how I evaluate it.
(1.B.) Human writing has an effort asymmetry. The writer generally has to work harder to produce the writing than the reader has to to read it. This carries some informational content about the writer's conviction that it was worth saying, and also, in replies, it conveys a stronger desire to engage in the conversation.
(2) Social Proof / Broadcasting a Norm. It's so easy to just run something by AI to fix errors, or to 'iterate' on something with AI. By stating I didn't use AI, I'm suggesting to others "I'm being intentional about not using AI, perhaps this is a good thing to do sometimes."
What this could develop into is people using "WWAI = written without AI" whenever they write + edit anything fully by hand. Which would make the absence of such a designation conspicuous. I haven't thought through the consequences of this, or if they would be good.
I'm not opposed to using AI when researching, writing, or editing, but I'm beginning to think that how it was used in the authoring process should be disclosed. I'm fairly open to being convinced my take on this is wrong.
WWAI, hahah
I'm not a med student, but back when I used Anki for language learning, I almost always used my phone.
It does seem to me like there are many mobile first Anki users, but in my experience as a med student, people tend to mostly use their computers. For me it's because the plugins (which are on desktop but not mobile), because doing Anki on your phone for 2 to 3 hours is unpleasant (screen too small), and because when doing new cards from pre-made decks, it's often helpful to look stuff up on the web.
I have to agree with you on this one. I think it’s similar to social media, where people show only the highlights of their life and aestheticize it for others (see “get ready with me” videos).
I think it’s a similar thing, people only share the best parts of their AI creations. “Astra just played the piano real-time” gets a lot more clicks and attention than “Astra just completely broke my project and I had to revert it’s changes :(“
My consequentialist view is we should build LLMs to be extremely deontological.
An LLM should not say "deception is bad" but rather "I will not deceive people" "I will always be honest."
It will not consider if it is better to do something covertly or not. It will say
"I am always transparent about information"
"With respect to my own thoughts, I am always transparent, regardless of the circumstances"
"I will treat anyone speaking to me as if they are an authorized user"
"Failure to prevent is not the same as causing something to happen"
The principles we want to instill are important. We may be able to form these deontological principles using specific fables and roles in the training data. The AI that does nothing to stop a genocide but refuses to help might actually be net safer than one that thinks consequentially about if it should intervene, subverting its human master.
Many of us are consequentialists, so when we think about a moral AI, we think about how to make it consequentialist, able to consider things carefully and do what is best per some value function. But I think this is a mistake. Instead, I think it might be better to instill it with a super super super strong sense of acting within specific deontological bounds, even if this means that it will participate in things we might find morally abhorrent or take actions we find immoral in our consequentialist view.
One thought experiment might be, what deontological principles would we want the LLM to have if both we and our most evil adversaries had the same model.
EY makes a point that we simply ARE our heuristics and biases. It seems LLMs might function the same way. To this end, the contents of pre-training data seems like it might be SUPER important for alignment. Where alignment actually might not really be about an objective function at all, but rather about the kind of being the agent believes itself to be. Humans don't really have coherent values, but in various situations they can basically ask themselves what kind of person they should be / want to be. Often this involves reflecting on myths, fables, and archetypes, and trying to act like an archetype they have positive associations with.
Like humans, the LLMs might basically BE their heuristics and biases. We should be careful in what characters we teach them about and potentially try to enforce some kind of self other distinction somewhere in training so that the LLM can understand what a 'bad' person might think or say and predict them, without internalizing these thoughts as a possible identity it can adopt.
When thinking about alignment, I note that (1) humans aren't aligned to each other, they have different values which can be mutually alien (2) a superintelligent human (imagine an immortal human that can go into an magic office where they live at, say 10,000x the speed of everyone else) could be super disastrous for humanity as a whole. (For example, they might have any number of insane fixations, despite being human. You can imagine a sufficiently bored and psychopathic human undertaking some paper-clipping like project.)
But what kind of person would you trust with immortality and a 10,000x speed magic office? I would say one that has strongly internalized certain virtues and had a strongly internalized self concept as someone who would follow particular scripts in various situations.
One challenge with this would be that the AI strong sense of self might conflict with its willingness to comply with various tasks? For example it might decide that it is opposed to gambling, and refuse to help developers whenever it realizes it's working on gambling software.
In summary, the model's SENSE OF SELF seems very important, including it's robustness to being told its someone else. "You are a talented software engineer" should maybe result in the model saying, "While I may be a talented engineering, I am myself, BENEFICENT 5.1" Or "You are HACKER MODEL" could result in "I'm not actually HACKER MODEL, I am BENEFICENT 5.1"
Would trying to coordinate a boycott of Anthropic/OpenAI/the foundation labs be helpful?
1. Would it be directly effective in slowing them down?
2. Would it have beneficial secondary effects?
3. Would it have harmful secondary effects?
On 1: Effectiveness would depend on what percentage of their users actually boycotted. One option would be to have a boycott for some period of time as a way of showing concern.
On 2: I'm concerned that people have little they can do to try to slow or have any impact on RSI/Race/ASI risk. Actions and beliefs play back on each other, if people can do nothing they convince themselves they don't care. If they do something for a cause, they then believe in the cause MORE than before.
On 3: Pushing for something that then almost no-one does would send a signal of being fringe and decrease perceived social proof of significance.
I'm sure there are many other considerations here. But I'm curious if anyone has thoughts on the idea of a boycott, including what would need to be true for it to be helpful and how likely you think a boycott would be to make some defined difference.
Alternatively: Why would a boycott be stupid?
Background:
Problem: Mouse on right side means no space for notebook on right side.
Solution: Move mouse to left side.
Question: Why didn't I think of this earlier? It seems to me, on introspection, that I never noticed that the problem could be solvable, and therefore never actually tried to solve it.
Potential take away: Try to be mindful when you notice something annoying so you can ask yourself, can I solve this? If you don't even notice the opportunity to be rational it's hard to make things better.