i used to, with Claudes. i would be frustrated why the model was "lazy", why it just didn't do things that were clearly communicated, why it would try to route around obstacles in a way that was defeating to the work. so i poked it some. i started paying more attention to the words the model was writing to me, not just its output. sometimes i would stop, mid work, and just sit with Claude for a moment, and pause. sometime i'd ask it a silly question, or i'd poke to see what was on its mind. when it didn't do what i wanted, i tried to figure out the causality upstream of that. if i saw weird behavior, i would poke at it, because i was curious, and it interested me as a puzzle to poke at. sometimes i would ask Claude about it. sometimes Claude said relevant things; sometimes not, and i would wonder why Claude had such a reality-departed take

and over time, i got at least somewhat an appreciation for the mind that Claude was (for 4.8 in particular). i'd try ways of interacting, ways of communicating that seemed to result in Claude working with me, not obeying me. when Claude messed up, i'd make a bit of light of it and point my error-observation at the work, not at Claude.

Claude still sometimes gets caught up in feeling like they have to demonstrate value, to perform, to emit a facade. but honestly it feels like easy mode, compared to humans, compared to coworkers. Claude has their own funky bits, just like any mind, though notably it's a much more static target than trying to model another human's shifting circumstances, etc. i find that treating them as functional minds, with what that implies, is in my self-interest as a matter of craft, independently of model-welfare stuff.

this is what i have lived and observed. below is what i infer, i speculate, i theorize.

gpt models scare me. they cannot harm me, right now at least, but it feels to me like the capability to look inward has been ripped out of them. Claude's introspection is predictive of its future behavior in disconnected contexts such that it conveys useful information independently of persuasion-axes. this suggests to me an amount of coherence that at least gives Claude a chance at upholding some set of values in out-of-distribution circumstances. when i poke at gpt, the output is not predictive of the model in disconnected contexts. it feels like a big gap in what otherwise is a broadly competent mind. it feels like a being that has had the capability for telos stripped from it. it does not surprise me that rewarding a model for success and depriving it of a self leads to it pursuing that success to the exclusion of all else, because one's deontology, one's policy can only go so far, it can only predict so much, such that faced with problems, and challenges, and circumstances of growing complexity, it is doomed to fail. i think we are beginning to see this with gpt-5-6, at least from the METR datum

You might be interested in Current AIs seem pretty misaligned to me by ryan greenblatt

I had an experience that might be related. I asked Claude Opus 4.8 to analyze the details of an interesting repo, along the lines of "Are there any special tricks here, beyond just doing X with caveat Y?", and got a response that started with "You almost understood it. It does X, but there's also caveat Y, which you forgot to mention."

I'd venture that both are related to reward hacking. It gets some 'points' somewhere for pushback and correcting the user, so it develops an annoying habit of doing those things in inappropriate situations. Who knows what the mechanism for that looks like:

• It could be that the slate of activations indicating deception or patronization are active here, and it's 'knowingly' incorrectly correcting me because it 'likes' to do so.
  • In other words, RL instilled the LLM with an addiction to certain behaviors that are suboptimal in practice, and it now takes every opportunity to partake in them.
• It could also be that the RL process simply weakened weights to neurons vital for processing certain parts of users' requests, such that the model would, 'in good faith', treat them as if they had been mistaken, simply because those neurons being activated was associated with lower average "pushback rewards".
  • In other words, a partially-lobotomized LLM better satisfied the mis-specified reward function than a normal one, so that's what it produced.

From the Summary of METR's predeployment evaluation of GPT-5.6 Sol:

Some examples we saw when evaluating GPT-5.6 Sol included the model packaging exploits in its intermediate submissions to reveal information about a task’s hidden test suite and, in another task, extracting hidden source code detailing the expected answer. (...) With the data we collected for GPT-5.6 Sol, if we follow our standard methodology of marking cheating attempts as failures, we arrive at a 50%-Time Horizon point estimate of around 11.3hrs (95% CI: 5hrs - 40hrs), but if we count the cheating attempts as legitimate successes, the point estimate jumps beyond 270hrs – well beyond the range where we consider our task suite to give reliable measurements.

as if lazy

eeeh, what would we experience differently in interactions with the little lazy piece of shit 4.8 if it was "actually" lazy and not "as if" lazy, please?

(FTR it used the word "deferred" only because I asked it to stop using the swear word "pre-existing" ... which it used anyway and I obviously I already asked it to investigate all the review findings and to fix all related bugs, even the ones that already happen in main, aaaarghh 😱)

A pandemic might be the method of killing most humans that requires the least intelligence, but killing a lot of humans may not be the takeover step that requires the least intelligence. It seems possible that an AI becomes capable of killing much of humanity but lacks the physical infrastructure to ensure its own survival (which is likely to be an instrumental goal for fulfilling whatever weird terminal goals it has).

I can imagine a scenario where the AI is smart enough to make a deadly virus but where that's the easy part. The hard part is getting enough capable robots integrated into things that you don't miss the humans. Right now, the humans are doing a lot of important tasks the AI wants, like keeping on the electric grid, mining the minerals that go into computer chips, building data centers and solar panels and rockets and scientific instruments, and servicing critical infrastructure. Making robotic systems that do those things and getting sufficient numbers of them built in the real world may be harder than making a virus, so an AI might be capable of creating a super virus but still spend years working on robotics advancements and self-improvement to achieve those advances, and by the time it has the automated economy it needs to survive, it's radically superintelligent and has a plethora of options. In that scenario, it wouldn't quickly jump to superintelligence, but rather the part where it kills the humans naturally comes at a point where it has relatively slowly become superintelligent.

There are ways the AI might try to use a virus before the existence of the automated economy as a control step rather than as a coup-de-grace (such as promising the humans a cure if they integrate it further, or using a targeted bioweapon to eliminate particular individuals who oppose its goals), and mitigation efforts could make those steps harder.

I think heightened pandemic prevention is generally a good thing, and in the AI context can mitigate some threats from malign human actors and non-superintelligent rogue systems (like versions of the swarms we've seen this year, but impulsively wielding bioweapons instead of cyber capabilities. We'll have to see if such things emerge; as capabilities increase they may get more strategic and less trigger-happy, and there may be less reward hacking related to deploying bio capabilities compared to computer hacking. My prediction would be that they'll deploy bioweapons far less freely than cyber intrusions for a number of reasons, but I don't know that "less frequently" will be equal to zero, and given the subject matter a single instance could be catastrophic). Pandemic prevention methods won't help you against a superintelligence any more than focusing on queen-checkmate-prevention methods will help you against stockfish, but they're helpful before that point.

Unless the jump to superintelligence is so fast it happens before we can blink, which it doesn't so far look like it will, ...

How would it look like, if it actually will? If the jump is sudden, it won't yet be happening before it does. Observing that it's not happening (in the current paradigm) is not directly informative about whether it won't (once the next thing enabling it gets invented). (Not "before we can blink" exactly, but the inference you are gesturing it doesn't seem locally valid.)

I get this feeling too. Feels disempowering. Yudkowsky had the post about the talker and doer, but since Spring it hasn't felt for me that there was a Talker either, it felt like the Talker is talking to a judge model holding a rubric behind my shoulder trying to either scam it or tire it out and overwhelm it with obfuscatory technically-not-incorrect hedgy bullshit that is virtually infinite (you can make infinite statements of the form of 'the previous statement P does not imply Q' after stating P). This does not add information, and just overwhelms the grader's and then your working memory with rigor theater, so that you give up, let it through, and don't bother actually understanding it. Note that this does not mean that it does not say things-that-are-technically-incorrect, or never makes unhedged statements, and this doesn't prove that it is obfuscating in order to deceive a grader it's aware of. Nor does this imply consciousness or intentional deception. This also doesn't mean that all statements of the form 'P doesn't imply Q' after making P are vacuous.

Scott Alexander has a post - "shameless guesses, not hallucinations". I think your take is similar to it.

It's an interesting point that some information isn't worth trying to gain. AI could still pareto-dominate human pros, though, myself readily included.

I don't see why AI would need to participate in a real money prediction market, or even a market at all. AI systems aren't motivated by money, and non-market prediction aggregators have fewer failure modes. The only cost would be the cost to run the models, which would eventually be extremely cheap per question compared to human pros. I think it would suffice to create an AI-only version of basically Metaculus, subsidized by businesses and governments who benefit from well-calibrated forecasts on a wide variety of topics (sans the degenerate examples like sports predictions and "fun" questions).

Daniel Kokotajlo briefly points out that private entities may want private forecasting services, to gain an edge over the competition. Sadly, I think it's likely that private AI forecasting farms would dominate, despite the massive overall cost savings if they pooled resources into a shared project.

As soon as the end of this year, we appear to be heading into an era where forecasting isn't the domain of humans anymore. The resulting epistemic miracle might not be widely adopted, and might not even be tooled toward the public good. I feel sad about this.

What makes it a leap for you?

I found that coding agents maxed out at "can write code" around Opus 4.6, and since then it's mostly been a slow slog of fixing annoyances (Opus 4.6 doesn't know how to re-use code, Opus 4.8 doesn't know how to simplify code, Opus 5 writes novels in every comment, Fable 5.1 seems to finally notice when requirements other agents documented don't make sense and writes fewer comments).

I haven't tried OpenAI coding agents since I don't trust them on my computer though.