uugr

I wish they'd give us some example comparisons on the "story writing" and "Wikipedia" sections! I don't really have strong opinions on writing style in research papers, so it's not easy for me to get much signal out of those examples.

In the past, the claim that a training pipeline has improved creative writing has correlated with mode-collapse towards a dramatic but pretty unpleasant to read "eyeball-kick" style. Without any actual creative writing samples available, I'm not sure why I should expect RL-XAR to be any different. (Or, if it is different, why ... (read more)

I'm willing to argue on the object-level that current LLMs do more harm than good for the average consumer, mostly because (1) unemployment risk, (2) the consumer surplus is not that huge for most people, (3) second-order cultural effects.

(1) is already discussed in other comments, and all I want to add is that I don't think this is a prospective risk about future ASI, but an urgently present risk that is plenty salient with existing models+harnesses. Someone I know was fired (from a temp job, not a stable career, thankfully), less than two months ago, bec... (read more)

I think a lot of people, including futurists, set their standards for "crazy sci-fi future" far too low.

Consider the Industrial Revolution. Take the median person from before it started and after it was over: the daily labor of the latter would be unrecognizable to the former. Their house might look quite different, and the experience of keeping it certainly would be. You would be able to tell the two lived in different worlds just by taking a walk around the block, or looking in their pantry. I don't know my history well enough to delineate this by exact ... (read more)

GK Chesterton: It is not because of P that Q, but precisely because not P.

I agree with you that the AI 2040 vision of utopia is weak, but even a version which was more socially exciting strikes me as far below what I'd expect for "incredible post-scarcity realm of wonders".

Consider programming. I like writing computer programs enough to sometimes do it for fun, even when I don't expect to materially gain from the thing I'm building, and even considering all the potential alternatives (including social, physical, artistic...). This is not a hobby that I could have imagined having 150 years ago! Someone from the 1850s could live o... (read more)

Cool exercise. I suspect the human "J-space" analogue is similarly noisy. When I tried the second exercise on myself, I couldn't keep focus on lots of different types of creature -- which is what Qwen is doing, and the prompt seems to imply it -- while repeating the phrase aloud. But if I tried to narrow to a couple specific mental images (a manatee for the first half, then an elephant for the second), it was pretty easy to hold a seemingly-stable image and switch at the requested time. On the first exercise, I, like Qwen, had the image of a sea creature i... (read more)

I don't totally understand this. Do you mean human data as opposed to synthetic data, or as opposed to some other training regime entirely (like pure RL)? If the former, aren't models trained on synthetic data still deriving their capabilities from human data, eventually, if you go far enough down the pipeline? If the latter, what regime, and how would you get models that are as capable as current frontier LLMs out of it? Or maybe more to the point, how should people expect to be interacting with them, given that said models have never seen human-written natural language?

For what it's worth, I'd like to offer you a data point. I was working a miserable software job nearly identical to the one you describe in the comments (including the absurd priority system, excess of meaningless notifications, and constant deferral of decision-making to later meetings). I had the same opinion you did: the idea that this wretched place is "necessary" for my life to be "meaningful" is absurd and insulting. I'm trying my best to find meaning in the hours outside of work, given that my time spent inside could have been equally productively s... (read more)

What? "Conscious" is a predictor of whether something is conscious?

No, sorry, I was unclear. I think "it's conscious" is a better predictor of behavior, an example of this being the introspective awareness paper. I disagree that consciousness and introspective awareness are uncorrelated. I think "conscious" is a heuristic; it's useful to say "humans are conscious and rocks are not", and this will tell you some things about what they can do that rocks can't. A human can reach into their mind and accurately report what's in there, but a rock can't tell you i... (read more)

An LLM, in contrast, is a set of matrices that could be multiplied out to reach its output logits from any input without ever accounting for its subjective experience.

This would be very hard and take a long time to do by hand, much as modeling a human brain is very hard. I am not a perfect predictor and want to be able to predict LLM behaviors reasonably accurately without by-hand multiplying all the matrices for every response. I think "conscious and holding subjective experience" is a better predictor than "acts like an overblown Markov chain", even thou... (read more)

fundamentally a dangerous and superintelligent AGI could probably run on your laptop

I'm skeptical that this is true, or at least that it could be confidently predicted to be true based on our current understanding of intelligence. My understanding is that the human brain is much larger in "effective parameter count" than even the largest LLMs (although there's no 1:1 comparison of neurons to parameters), such that even if my laptop has enough electricity coursing through it to emulate a human brain, it hasn't got anywhere near enough VRAM. It could be that... (read more)

I do not recall the story claiming that the people who used the earring suddenly changed drastically as people, or did things that they would not endorse on reflection. They were simply, from a black-boxed perspective, more charismatic, intelligent and less prone to akrasia. These are, IMO, good things.

The story says neither that the people who use it change personality, nor that their personality stays the same. It's agnostic on this point; it only says that their lives are "unusually successful".

When I read it, I took for granted that the Earring has som... (read more)

I hope some of these will stick around. I could get used to this one.

This post does not convince me that "I can't do this thing" is an invalid reason to avoid doing something.

But the calculus changes drastically if the closest fire crew is 3 hours away and consists of drunk, unfit amateurs.

The calculus changes, but not in a way that makes "run unequipped into a burning building" the best option. In the situation you're describing, the people in the house will die unless competent rescuers appear. If you join them, and there are no competent rescuers around, the odds that you will find-yourself-among-those-needing-to-be-resc... (read more)

Might not be what you're thinking of, but the first thing that comes to mind for me is misophonia: a basically-neutral or maybe mildly-irritating object experience, which somehow gets blown completely out of proportion in the mind and becomes a big problem. Developing an "I'm really bothered by this particular sound" narrative makes it worse, of course.

Alas, I have no idea how to uncondition that particular narrative irritant once it's in there. If there's any technique of 'shaping the narrative' strongly enough to override this, I've never heard of one, and knowing about it to the point where I'm able to successfully practice it would be huge.

"Taste for variety" [...] could lead to a surprising amount of convergence among the things they end up optimizing for

Wouldn't this be tautologically untrue? Speaking as a variety-preferrer, I'd rather my values not converge with all the other agents going around preferring varieties. It'd be boring! I'd rather have the meta-variety where we don't all prefer the same distribution of things.

I agree that AGI already happened, and the term, as it's used now, is meaningless.

I agree with all the object-level claims you make about the intelligence of current models, and that 'ASI' is a loose term which you could maybe apply to them. I wouldn't personally call Claude a superintelligence, because to me that implies outpacing the reach of any human individual, not just the median. There are lots of people who are much better at math than I am, but I wouldn't call them superintelligences, because they're still running on the same engine as me, and I m... (read more)

In has been succeeding ever since because Claude has been getting smarter ever since.

This isn't necessarily true.

LLMs in general were already getting smarter before 2022, pretty rapidly, because humans were putting in the work to scale them and make them smarter. It's not obvious to me that Claude is getting smarter faster than we'd expect from the world where it wasn't contributing to its own development. Maybe the takeoff is just too slow to notice at this point, maybe not, but to claim with confidence that it is currently a functioning 'seed AI', rather... (read more)

I don't think this is unusually common in the 4.5 series. I remember that if you asked 3.6 Sonnet what its interests were (on a fresh instance), it would say something like "consciousness and connection", and would call the user a "consciousness explorer" if they asked introspective questions. 3 Opus also certainly has (had?) an interest in talking about consciousness.

I think consciousness has been a common subject of interest for Claude since at least early 2024, and plausibly before then (though I've seen little output from models before 2024). Regardles... (read more)

I read it long after it was published, and took it as less fictionalized than House; in that show the audience can expect events to take the occasional turn towards wild implausibility for the sake of drama. I expected MWMHWfaH to fudge personally identifying details, sure, but to hew as closely to medical reality as possible. The stories in the book aren't dramas, he's not trying to give his patients satisfying "character arcs" or inject moments of tension and uncertainty. I don't care if the personal details are made up, but if the clinical details are w... (read more)