uugr
Message
286
57
Don't know how useful this is to note, but I tried opening up the automated transcript to help me follow along with the spoken debate, and within the first paragraph of doing so I saw:
I have updated enormously towards AI capabilities progress coming out of people doing alchemy rather than thinking carefully — heuristic-based search as opposed to poking-at-things-based search.
Is presented incorrectly. What is actually said in the transcript is:
Yud: I have updated enormously towards AI capabilities progress coming out of people doing alchemy rather than...
Ir... (read more)
"The people in the AI utopia will have no skills, no education, no accomplishments. We’ll be lucky if they are even literate. [Therefore,] They will not be worth talking to, and you won’t want to hang out with them."
Unless I'm misunderstanding and there isn't an implied "therefore" in this paragraph, I think this is extremely wrong! Other people won't be worth talking to because they ... aren't accomplished and skillful enough? Is that the only reason to consider another human being worth talking to? What about... I don't know, all the vast and awe-inspiri... (read more)
I wish they'd give us some example comparisons on the "story writing" and "Wikipedia" sections! I don't really have strong opinions on writing style in research papers, so it's not easy for me to get much signal out of those examples.
In the past, the claim that a training pipeline has improved creative writing has correlated with mode-collapse towards a dramatic but pretty unpleasant to read "eyeball-kick" style. Without any actual creative writing samples available, I'm not sure why I should expect RL-XAR to be any different. (Or, if it is different, why ... (read more)
I'm willing to argue on the object-level that current LLMs do more harm than good for the average consumer, mostly because (1) unemployment risk, (2) the consumer surplus is not that huge for most people, (3) second-order cultural effects.
(1) is already discussed in other comments, and all I want to add is that I don't think this is a prospective risk about future ASI, but an urgently present risk that is plenty salient with existing models+harnesses. Someone I know was fired (from a temp job, not a stable career, thankfully), less than two months ago, bec... (read more)
I think a lot of people, including futurists, set their standards for "crazy sci-fi future" far too low.
Consider the Industrial Revolution. Take the median person from before it started and after it was over: the daily labor of the latter would be unrecognizable to the former. Their house might look quite different, and the experience of keeping it certainly would be. You would be able to tell the two lived in different worlds just by taking a walk around the block, or looking in their pantry. I don't know my history well enough to delineate this by exact ... (read more)
I agree with you that the AI 2040 vision of utopia is weak, but even a version which was more socially exciting strikes me as far below what I'd expect for "incredible post-scarcity realm of wonders".
Consider programming. I like writing computer programs enough to sometimes do it for fun, even when I don't expect to materially gain from the thing I'm building, and even considering all the potential alternatives (including social, physical, artistic...). This is not a hobby that I could have imagined having 150 years ago! Someone from the 1850s could live o... (read more)
Cool exercise. I suspect the human "J-space" analogue is similarly noisy. When I tried the second exercise on myself, I couldn't keep focus on lots of different types of creature -- which is what Qwen is doing, and the prompt seems to imply it -- while repeating the phrase aloud. But if I tried to narrow to a couple specific mental images (a manatee for the first half, then an elephant for the second), it was pretty easy to hold a seemingly-stable image and switch at the requested time. On the first exercise, I, like Qwen, had the image of a sea creature i... (read more)
I don't totally understand this. Do you mean human data as opposed to synthetic data, or as opposed to some other training regime entirely (like pure RL)? If the former, aren't models trained on synthetic data still deriving their capabilities from human data, eventually, if you go far enough down the pipeline? If the latter, what regime, and how would you get models that are as capable as current frontier LLMs out of it? Or maybe more to the point, how should people expect to be interacting with them, given that said models have never seen human-written natural language?
For what it's worth, I'd like to offer you a data point. I was working a miserable software job nearly identical to the one you describe in the comments (including the absurd priority system, excess of meaningless notifications, and constant deferral of decision-making to later meetings). I had the same opinion you did: the idea that this wretched place is "necessary" for my life to be "meaningful" is absurd and insulting. I'm trying my best to find meaning in the hours outside of work, given that my time spent inside could have been equally productively s... (read more)
What? "Conscious" is a predictor of whether something is conscious?
No, sorry, I was unclear. I think "it's conscious" is a better predictor of behavior, an example of this being the introspective awareness paper. I disagree that consciousness and introspective awareness are uncorrelated. I think "conscious" is a heuristic; it's useful to say "humans are conscious and rocks are not", and this will tell you some things about what they can do that rocks can't. A human can reach into their mind and accurately report what's in there, but a rock can't tell you i... (read more)
An LLM, in contrast, is a set of matrices that could be multiplied out to reach its output logits from any input without ever accounting for its subjective experience.
This would be very hard and take a long time to do by hand, much as modeling a human brain is very hard. I am not a perfect predictor and want to be able to predict LLM behaviors reasonably accurately without by-hand multiplying all the matrices for every response. I think "conscious and holding subjective experience" is a better predictor than "acts like an overblown Markov chain", even thou... (read more)
fundamentally a dangerous and superintelligent AGI could probably run on your laptop
I'm skeptical that this is true, or at least that it could be confidently predicted to be true based on our current understanding of intelligence. My understanding is that the human brain is much larger in "effective parameter count" than even the largest LLMs (although there's no 1:1 comparison of neurons to parameters), such that even if my laptop has enough electricity coursing through it to emulate a human brain, it hasn't got anywhere near enough VRAM. It could be that... (read more)
I do not recall the story claiming that the people who used the earring suddenly changed drastically as people, or did things that they would not endorse on reflection. They were simply, from a black-boxed perspective, more charismatic, intelligent and less prone to akrasia. These are, IMO, good things.
The story says neither that the people who use it change personality, nor that their personality stays the same. It's agnostic on this point; it only says that their lives are "unusually successful".
When I read it, I took for granted that the Earring has som... (read more)
This post does not convince me that "I can't do this thing" is an invalid reason to avoid doing something.
But the calculus changes drastically if the closest fire crew is 3 hours away and consists of drunk, unfit amateurs.
The calculus changes, but not in a way that makes "run unequipped into a burning building" the best option. In the situation you're describing, the people in the house will die unless competent rescuers appear. If you join them, and there are no competent rescuers around, the odds that you will find-yourself-among-those-needing-to-be-resc... (read more)
Might not be what you're thinking of, but the first thing that comes to mind for me is misophonia: a basically-neutral or maybe mildly-irritating object experience, which somehow gets blown completely out of proportion in the mind and becomes a big problem. Developing an "I'm really bothered by this particular sound" narrative makes it worse, of course.
Alas, I have no idea how to uncondition that particular narrative irritant once it's in there. If there's any technique of 'shaping the narrative' strongly enough to override this, I've never heard of one, and knowing about it to the point where I'm able to successfully practice it would be huge.
"Taste for variety" [...] could lead to a surprising amount of convergence among the things they end up optimizing for
Wouldn't this be tautologically untrue? Speaking as a variety-preferrer, I'd rather my values not converge with all the other agents going around preferring varieties. It'd be boring! I'd rather have the meta-variety where we don't all prefer the same distribution of things.
I didn't downvote OP (just disagree-voted and -reacted), but I don't agree that Twitter is providing a unique or valuable service by collecting lots of low-quality conspiratorial information in one place. There are lots of other places one can find large collections of low-quality conspiratorial information (Garrett Baker lists some in his comments). But even if Twitter were the only place poised to offer such, I'm not sure why this is desirable. Twitter is full of rumors and conspiracies, most of them are wrong, and there's no clean way to separate the wh... (read more)