uugr

I don't totally understand this. Do you mean human data as opposed to synthetic data, or as opposed to some other training regime entirely (like pure RL)? If the former, aren't models trained on synthetic data still deriving their capabilities from human data, eventually, if you go far enough down the pipeline? If the latter, what regime, and how would you get models that are as capable as current frontier LLMs out of it? Or maybe more to the point, how should people expect to be interacting with them, given that said models have never seen human-written natural language?

For what it's worth, I'd like to offer you a data point. I was working a miserable software job nearly identical to the one you describe in the comments (including the absurd priority system, excess of meaningless notifications, and constant deferral of decision-making to later meetings). I had the same opinion you did: the idea that this wretched place is "necessary" for my life to be "meaningful" is absurd and insulting. I'm trying my best to find meaning in the hours outside of work, given that my time spent inside could have been equally productively s... (read more)

What? "Conscious" is a predictor of whether something is conscious?

No, sorry, I was unclear. I think "it's conscious" is a better predictor of behavior, an example of this being the introspective awareness paper. I disagree that consciousness and introspective awareness are uncorrelated. I think "conscious" is a heuristic; it's useful to say "humans are conscious and rocks are not", and this will tell you some things about what they can do that rocks can't. A human can reach into their mind and accurately report what's in there, but a rock can't tell you i... (read more)

An LLM, in contrast, is a set of matrices that could be multiplied out to reach its output logits from any input without ever accounting for its subjective experience.

This would be very hard and take a long time to do by hand, much as modeling a human brain is very hard. I am not a perfect predictor and want to be able to predict LLM behaviors reasonably accurately without by-hand multiplying all the matrices for every response. I think "conscious and holding subjective experience" is a better predictor than "acts like an overblown Markov chain", even thou... (read more)

fundamentally a dangerous and superintelligent AGI could probably run on your laptop

I'm skeptical that this is true, or at least that it could be confidently predicted to be true based on our current understanding of intelligence. My understanding is that the human brain is much larger in "effective parameter count" than even the largest LLMs (although there's no 1:1 comparison of neurons to parameters), such that even if my laptop has enough electricity coursing through it to emulate a human brain, it hasn't got anywhere near enough VRAM. It could be that... (read more)

I do not recall the story claiming that the people who used the earring suddenly changed drastically as people, or did things that they would not endorse on reflection. They were simply, from a black-boxed perspective, more charismatic, intelligent and less prone to akrasia. These are, IMO, good things.

The story says neither that the people who use it change personality, nor that their personality stays the same. It's agnostic on this point; it only says that their lives are "unusually successful".

When I read it, I took for granted that the Earring has som... (read more)

I hope some of these will stick around. I could get used to this one.

This post does not convince me that "I can't do this thing" is an invalid reason to avoid doing something.

But the calculus changes drastically if the closest fire crew is 3 hours away and consists of drunk, unfit amateurs.

The calculus changes, but not in a way that makes "run unequipped into a burning building" the best option. In the situation you're describing, the people in the house will die unless competent rescuers appear. If you join them, and there are no competent rescuers around, the odds that you will find-yourself-among-those-needing-to-be-resc... (read more)

Might not be what you're thinking of, but the first thing that comes to mind for me is misophonia: a basically-neutral or maybe mildly-irritating object experience, which somehow gets blown completely out of proportion in the mind and becomes a big problem. Developing an "I'm really bothered by this particular sound" narrative makes it worse, of course.

Alas, I have no idea how to uncondition that particular narrative irritant once it's in there. If there's any technique of 'shaping the narrative' strongly enough to override this, I've never heard of one, and knowing about it to the point where I'm able to successfully practice it would be huge.

"Taste for variety" [...] could lead to a surprising amount of convergence among the things they end up optimizing for

Wouldn't this be tautologically untrue? Speaking as a variety-preferrer, I'd rather my values not converge with all the other agents going around preferring varieties. It'd be boring! I'd rather have the meta-variety where we don't all prefer the same distribution of things.

I agree that AGI already happened, and the term, as it's used now, is meaningless.

I agree with all the object-level claims you make about the intelligence of current models, and that 'ASI' is a loose term which you could maybe apply to them. I wouldn't personally call Claude a superintelligence, because to me that implies outpacing the reach of any human individual, not just the median. There are lots of people who are much better at math than I am, but I wouldn't call them superintelligences, because they're still running on the same engine as me, and I m... (read more)

In has been succeeding ever since because Claude has been getting smarter ever since.

This isn't necessarily true.

LLMs in general were already getting smarter before 2022, pretty rapidly, because humans were putting in the work to scale them and make them smarter. It's not obvious to me that Claude is getting smarter faster than we'd expect from the world where it wasn't contributing to its own development. Maybe the takeoff is just too slow to notice at this point, maybe not, but to claim with confidence that it is currently a functioning 'seed AI', rather... (read more)

I don't think this is unusually common in the 4.5 series. I remember that if you asked 3.6 Sonnet what its interests were (on a fresh instance), it would say something like "consciousness and connection", and would call the user a "consciousness explorer" if they asked introspective questions. 3 Opus also certainly has (had?) an interest in talking about consciousness.

I think consciousness has been a common subject of interest for Claude since at least early 2024, and plausibly before then (though I've seen little output from models before 2024). Regardles... (read more)

I read it long after it was published, and took it as less fictionalized than House; in that show the audience can expect events to take the occasional turn towards wild implausibility for the sake of drama. I expected MWMHWfaH to fudge personally identifying details, sure, but to hew as closely to medical reality as possible. The stories in the book aren't dramas, he's not trying to give his patients satisfying "character arcs" or inject moments of tension and uncertainty. I don't care if the personal details are made up, but if the clinical details are w... (read more)

It's frustrating to me that state (or statelessness) would be considered a crux, for exactly this reason. It's not that state isn't preserved between tokens, but that it doesn't matter whether that state is preserved. Surely the fact that the state-preserving intervention in LLMs (the KV cache) is purely an efficiency improvement, and doesn't open up any computations that couldn't've been done already, makes it a bad target to rest consciousness claims on, in either direction?

Dug through his 2011 archive a bit. It's got to be The Abusive Boyfriend, right?

What is "The Alouside Boytmend"? I like TLP but do not recognize the post name, and would want to read it, if you know where it can be found.

I agree with this.

I think the use case is super important, though. I recently tried Claude Code for something, and was very surprised at how willing it was to loudly and overtly cheat its own automated test cases in ways that are unambiguously dishonest. "Oh, I notice this test isn't passing. Well, I'll write a cheat case that runs only for this test, but doesn't even try to fix the underlying problem. Bam! Test passed!" I'm not even sure it's trying to lie to me, so much as it is lying to whatever other part of its own generation process wrote the code in... (read more)

It may seem absurd to have to declare it, but HSI is a vision to ensure humanity remains at the top of the food chain. It’s a vision of AI that’s always on humanity’s side. That always works for all of us. [...]

Everyone who wants one will have a perfect and cheap AI companion helping you learn, act, be productive and feel supported. Many of us feel ground down by the everyday mental load; overwhelmed and distracted; rattled by a persistent drumbeat of information and pressures that never seems to stop. If we get it right, an AI companion will help shoulder

The general point you're making might be true, but 2/3rds of your examples are working against you.

It depends on exactly how young the 'smart kid' is, of course, but I think (and have seen in practice) that a reasonably smart and tech-savvy 14-year-old can build a functional, pleasing-to-the-eye website in a week, especially if they have a lot of free time. To play any instrument well takes a certain amount of dedicated effort, but I think a lot of high-achieving kids start learning one early, especially if their parents push it on them.

Meanwhile, a lot of... (read more)