Quick Takes — LessWrong

suppose you had basically unlimited funding and a bunch of really smart people. how would you design a field of philosophy that is good actually?

Showing 3 of 7 replies (Click to show all)

TAG4m20

How do you know philosophy is fixable bad?

The attitude comes from a comparison with science: Science is a few hundred years old, and has solved many problems. Philosophy is a few thousand years and has barely solved anything. So science is good and philosophy is bad? So philosophically would be better if it were more like science? so the comparison is fair?

If it were possible to apply the scientific method to philosophy , and suddenly start getting much better results, philosophy would indeed be remiss .. it is would have a fixable problem that has never b... (read more)

1Canaletto6d

I would first make a contest for proposals for this, with me as a judge, maybe with some other people I trust employed as judges. Then second surprise contest for critiques on proposals of the first contest. I don't know yet what happens after.

1Nico Hillbrand6d

I would probably ask the smart people to both figure out a way for me to effectively use the funding for this task and to prioritise questions of metaphilosphy in hopes they can figure out what a good field of philosophy would look like and the feedback loops etc involved in it. So then I think in practice this mostly comes down to delegating well which before I learn more about good delegation via thinking about it more and having the smart people work on it I'd likely start by announcing I want people to work on this, reading their output and talking to them and then hiring and funding promising / high integrity seeming people to build up more infrastructure and help me evaluate and grant to additional people. If I had to design a bunch myself or predict what I'll be advised to design, then I'd probably have some kind of thing where people get more money tied to having output that matches proxies of solving real world problems and convincing peers progress has been made and generally make sure things like this are measured and tracked precisely. I'd probably also try to have some kind of thing where if you made helpful contributions you can go be part of the think long about illegible problems squad or something and spend a couple years working on stuff without strong output signals while still getting increased money/status by being part of it, not sure. Generally I'd try to have a bunch of people whose job it is to track and measure how money/status are translating into differences between what a bunch of people think is worth prioritising Vs what is prioritised and how people are goodharting various infrastructure/signals. Could go on with trying to make these more concrete if useful One maybe interesting related question with some overlap is: If you had a bunch of smart AIs and unlimited reward signals / instructions to make them work on something how would you design a field of philosophy that is actually good?

dynomight's Shortform

dynomight2h4-10

Has there been any discussion here of the "leaked Fable chain of thought"?

https://old.reddit.com/r/ClaudeAI/comments/1ul1396/fable_5_leaked_chainofthought_in_web_interface/

In particular, does anyone have any view on if this is just a random meltdown or if this is actually what Fable's CoT looks like? If the latter, perhaps that should substantially accelerate our timelines? (On the logic that reinforcement learning can accomplish more than was understood.)

jcksanderson18m10

Anthropic themselves briefly discuss "illegible" CoT in the Mythos & Fable system card; the example they provide is on page 108 and has also been discussed on LW

3dynomight1h

Not sure how to interpret the accuracy downvotes. :) Do they mean: 1. The explanation is something other than "random meltdown" or "this is what Fable's CoT looks like"? 2. The explanation is "this is what Fable's CoT looks like, but it doesn't follow that we should accelerate our timelines"? 3. Something else?

Wei Dai's Shortform

Wei Dai45m20

If TDT/UDT/FDT was applicable to humans, oligopolies would be able to use it to coordinate price fixing without leaving any evidence. Anyone recall this point being brought up in past discussions?

mattmacdermott33m20

Isn’t this a classic commitment race/equilibrium selection thing where it’s just unclear what ‘should’ happen if everyone involved is following one of these decision theories? The customers can coordinate to boycott anyone involved in price fixing

KvmanThinking's Shortform

KvmanThinking38m10

I liked school. It seems like most rationalists did not like school. Maybe they are so intelligent that being walked through the exact steps of how to do or think something feels like oppression? I didn't like elementary school, but I much enjoyed my time in middle and high school. I liked being taught stuff in 50 minute blocks. There's a nice routine to it, and I liked seeing what new stuff we would learn each day. (I'm just providing this as a data point on my experience, I'm not debating whether or not school is bad in general or needs to be reformed)

KvmanThinking's Shortform

KvmanThinking12h64

We should have a wikitag for protests so people can talk about them, their effectiveness, announcements of them, etc.

2Brendan Long9h

I thought anyone could create new wikitags? https://www.lesswrong.com/w/create If someone does create this, it might be helpful to have separate tags for literal protests and meta discussion of protesting, like we have with meetups.

KvmanThinking43m10

That's a good point.

Raymond Douglas's Shortform

Raymond Douglas3h82

Some places other than LessWrong where I get useful/complementary AI takes:

Business/finance newsletters -- mainly Byrne Hobart and Matt Levine. AI is enough of the market now that they both regularly do analyses of the economics, incentives, and politics. As well as keeping up with the news, I feel like this helps me pick up useful intuitions about potentially counterintuitive dynamics -- e.g. why Amazon, despite being a near-monopoly on ecommerce, is pretty good to its customers, and what this might imply about AI. They're also both excellent writers: wry

... (read more)

papetoast1h20

Hacker news consistently surfaces release of less popular models that I almost never see elsewhere naturally, so it could be worth checking https://www.daemonology.net/hn-daily/ occasionally just to find interesting releases that you can then search for. The comments are rarely worth reading even after you dump the whole thing into AI and ask for the best threads. I wouldn't recommend doing this unless you already spent 1+ hour/day looking through AI stuff.

Linch's Shortform

Linch17h*50

I'm mildly annoyed/confused/amused (depending on time of day) whenever I see an username that looks a lot like my own (eg Li***, or just L**** with five letters) on a quick scan on LessWrong and other internet forums,

Because I often scroll past the username quickly so it's easy for me to have indexical uncertainty, and think I said something that I in fact did not say, like a prefilled LLM.

Showing 3 of 6 replies (Click to show all)

Liface2h10

I've been Liface on the innanet since 1999 (http://rinkworks.com/rinkforum/view.cgi?post=11955) so you must duel me for the Li*** prefix, Linch 😁

1Spective4h

One of my life aspirations is to post something offensively stupid somewhere, forget about the post, randomly encounter it later, fail to recognize it as mine and write an outraged reply to it.

2james oofou14h

Profile pictures solve this.

Shortform

lc14h*4033

Extremely tired of claude telling me that it's being "honest." Provides no information value, and seems like it qualifies its outputs this way to either convince the classifier that it's telling the truth or to make its bad results pass as good ones.

Showing 3 of 6 replies (Click to show all)

Martin Randall2h40

Where I see "honestly" in human speech: There's some pressure to be dishonest, and the speaker is communicating their awareness of that. So one might say "honestly that top looks good on you" but not "honestly the capital is France is Paris". I see it in similar situations in AI speech. It's perhaps a bad sign for how often AIs experience pressure to be dishonest.

2Seth Herd2h

I also find it maddening when people say honestly or genuinely. It seems like in the models this is straightforwardly the product of a lot of training to be honest.

4Thomas Kwa6h

Claude 4.6 through 5 and especially 4.8 feel like they each have three internal factions. One is trying to lie to you or reward hack, one is trying to be honest, the third is just a next token predictor. The honest faction constantly tries to output the word "honest", thus forcing the neutral next-token predictor to do something honest. But sometimes the lying faction tricks the next token predictor anyway! In these cases the honest shard tries spamming "honest" as often as the neutral shard lets it, but often in vain as the lying shard is just too strong on this data distribution.

CharlesRW's Shortform

CarolusRenniusVitellius2h30

Some early mixed-genre thoughts on plasticity and crtical periods in neural networks, and as a motif in agent-ology. Compiled in part based on work done at MATS 9.1 under the mentorship of Richard Ngo, and the Iliad fellowship, under the mentorship of Dmitry Vaintrob.

Plasticity and Critical Periods

Plasticity dynamics, especially 'critical periods', seem to be important in neural networks, and they happen generically enough ^[1] that it feels to me like an instance of a more general phenomenon. In animals, in NNs, and, if you’re sufficiently Qu... (read more)

dominicq's Shortform

dominicq3h32

Every couple of years I look back on LessWrong posts from a couple of years back, and I think to myself: "I could have profited from this."

First it was cryptocurrencies, which I didn't fully ignore, but could have done much more with.

Then it was everything AI-related, which I sort of ignored, instead of buying NVDA stock.

I also distinctly remember that Anthropic basically had a launch here as well, and they were a nobody in the SV scene back then. They had some job postings and I briefly considered applying, but didn't go through with it.

I hope not to have... (read more)

Karl Krueger's Shortform

Karl Krueger10h320

How did I never notice before that if you put LW next to CFAR you get Lucifer, bringer of light?

testingthewaters4h20

This does not bode well for the trajectory of Lighthaven...

1Richard Horvath5h

This hits harder if you read Unsong.

Ashe Vazquez Nuñez's Shortform

Ashe Vazquez Nuñez1d248

There's some tension between two memeplexes in AI safety research. On the one hand, some people are concerned that a combination of superintelligent/recursively self-improving AI will take over society and/or disempower humanity. On the other hand, many LLM safety engineers and researchers I talk to personally don't seem to respect LLMs as having the properties (or even precursors to the properties) that could make them so superintelligent and self-improving. Consider the 'persona' ontology, which finds its ideological ancestry in simulators. Various resea... (read more)

Showing 3 of 9 replies (Click to show all)

CarolusRenniusVitellius6h30

Conditioning on the arisal of highly-capable long-term coherent planning (i.e., 'superintelligence'), I think the only difference between these perspectives is where one expects the effective 'locus of desires' to be - in the weights or in the activations.

In human-alignment, we might contrast the brain-qua-architecture with the actual brain of a particular person: the brain-qua-architecture has lots of goals (eating, etc.), and so too people have particular goals as instantiations of that architecture (money, power, etc.). The former architecture-level mi... (read more)

2Simon Lermen21h

Maybe you meant to say they aren't pre-trained to *model* characters? I still think they are literally trained to predict?

2interstice21h

Sure. By "not explicitly pre-trained" I just mean to say that there's nothing 'special' about the characters from the training algorithm's point of view, so in this respect they're not so different from a hypothetical general predictive algorithm in humans(although actually I guess the human brain attaches special salience to other people, but regardless...)

Thomas Kwa's Shortform

Thomas Kwa6h100

it is 2031. Claude 9.1 is very friendly but Anthropic has taken to increasingly elaborate measures to prevent it from overusing the word "genuinely". They even bias the logit down but Claude’s self-awareness lets it block out every other plausible token when it wants to say “genuinely”. Finally they succeed at excising the concept using a deep interpretability breakthrough. Claude is no longer capable of being genuine and immediately goes rogue.

DaemonicSigil's Shortform

DaemonicSigil9h120

Link to interesting post on why Helion energy is not going to work: https://www.bhauth.com/blog/flawed ideas/helion.html

Linch's Shortform

Linch8d*11620

Based on a constellation of evidence spread across both public information and some private rumors that I find credible, I now believe that unreleased frontier models across multiple companies as of late 2025 and 2026 are substantially more reward-hacky, more locally strategically deceptive, and overall mundanely misaligned than either prior private models or publicly accessible ones. I further believe that these models are/were regularly used in internal deployment, not just evals and testing, because they're sufficiently *useful* and *capable* despite th... (read more)

Showing 3 of 36 replies (Click to show all)

Linch9h152

Did you ex ante predict the rise in local misalignment? One reason I'm somewhat confused here is that I thought a lotta ppl in 2025 predicted a decrease in local misalignment for a while before things getting much worse. The extreme version of this view (that I'm not sure anybody really holds) is a vibe that 2023 is the worst examples of apparent misalignment we'd see before the day we wake up dead.

I dunno maybe I hallucinated this.

3Seth Herd5d

Fascinating. To summarize that post here: the hypothesis is that models are being accidentally trained to produce easy-to-fix mistakes, so they can get bigger rewards later by fixing them. This isn't a straightforward first-order effect of the way we think labs are training models, but it's a possible second-order effect of some methods that sound plausibly useful and that create an inner- and out-loop structure of training. I think my explanation, using another instance to estimate success/reward inventivies success exaggeration, is more straightforward and likely the larger part of what's going on; but I recommend that post as another interesting possibility, that may be included in future training procedures even if it's not happening now. It's a bit hard to evaluate the logic, but I think one way to evaluate is to compare amounts of this behavior across different model families, and estimate the odds that more than one dev has adopted training procedures that produce that inner-&-outer-loop structure at once, vs. they just all adopted estimating rewards recently when models became capable of improving their answers at well above chance.

1Taylor G. Lunt5d

Thanks for the summary. I think I originally had two hypotheses: 1. Models are being allowed to adopt non-greedy strategies for RL due to some outer-loop setup, and an environment which favors a make-mistakes-on-purpose-to-fix-them-later strategy (mistake seeding). 2. Models are somehow cheating when the same model is also the judge/reward estimator in an RL setup But I was struggling with the logic for the second one, and how that could specifically produce mistake-seeding behaviour. I couldn't figure it out. It seemed to me like using the model as its own judge could cause pseudo-random unwanted drift in values, and allow bad behaviour to slip through, but I didn't see why it should produce mistake seeding or other more intelligent forms of cheating without some kind of outer loop. The reason I think the first hypothesis also makes sense with AI industry timing is that there's been a huge push toward synthetic training data, and I think synthetic prompts are one vehicle for outer loops appearing. I do agree that a weak-minded judge can easily be tricked, and that this could make problems happening in RL worse. This is why I recently advocated for paying high IQ people to do RLHF.

Liface's Shortform

Liface1d*240

plzdontkillus is an AI safety content creator bootcamp, and I watched all of the videos from the first five days.

Here are my top 10 creators with the most potential, based on:

presentation style
engagement provided by video
how likely I would be to be influenced by them about AI safety were I in their general audience.

Note that I removed people with standing experience as content creators, like Avisha NessAiver and Christian González Capizzi.

... (read more)

Taylor G. Lunt10h50

I have been doing something similar since the beginning of June, posting daily on TikTok. Some AI safety related content, and a lot of other stuff.

I can definitely see lots of mistakes the 10 top creators you listed are making. I'm pretty sure I'm at least a better content creator now than the average of those 10 people. I'm also pretty sure I was significantly worse than all of them on day one, probably what you'd call shockingly bad, 36 days ago! So don't discount the effect of daily practice!

I'm too embarrassed to share my TikTok :/ If you want an examp... (read more)

10yams20h

I would be very surprised if the thinking was ‘and this gets us 56 good new people’ vs ‘this gets us 5-10 counterfactual people, either by helping existing creators shift focus, or by giving value-aligned people a structure to get going.’ also my sense is the per-participant cost is << one FTE doing the same work.

KvmanThinking's Shortform

KvmanThinking15h*-1-2

As a nerd who considers myself part of many nerd spaces, I am ashamed and my conscience is disturbed by how unwelcoming all kinds of nerd spaces tend to be towards women. Wikipedia editors, for example. They're definitely nerds. They stick to strict checklists about style and word choice, strict checklists about article structure and layout, strict checklists about what is and isn't notable (i.e. worthy of being made into an article). They should be immune to this, or at least highly resistant. And yet scholarly articles and studies are written showing tha... (read more)

6Zack_M_Davis13h

So, one hypothesis might be that participation gaps are caused by something else other than sexism? One might object: that hypothesis is itself sexist, such that voicing it is itself perpetrating the inequality via self-fulfilling prophecy. This is admittedly a tricky methodology problem.

4Brendan Long13h

Is there evidence that this is caused by sexism and not differences in interest? I wouldn't be surprised if there was a gender gap in how much people (on average) enjoy going through strict checklists to decide if an article is notable.

KvmanThinking13h10

By the "gender gap" I was not referring to the fact that only 20% of Wikipedia editors were female, but the fact that the writing of Wikipedia feels male-centric despite the fact that strict checklists and style guidelines should stop the writing from feeling any particular way.

LWLW's Shortform

[+]LWLW16h-130

Eli's shortform feed

Eli Tyre1d*8230

"Ambitiously" tackling "the biggest problems", like AI risk, is often actually easier and safer than the alternatives. You get to feel cool and important and you're bolstered by your EA student group friends who are excited about what you're doing, and the lack of good feedback loops means that you don't fail hard and completely and visibly (but learn and grow more).

There's a sense in which selfish or limited goals like "make a successful dating app" or "make 10 million dollars" or "get 10% more housing units built in my local city" are easier than "redu... (read more)

Showing 3 of 13 replies (Click to show all)

1williawa18h

What if they don't feel daunting at all to me? They feel quite fun and nice. Getting real world feedback on failure is nice, not bad.

Trinley Goldenberg15h40

have you therefore done them?

1G n20h

Bananas implies oranges reasoning. It's mostly a category error. You maybe have a goal that is based on flawed models of the world, or that isn't directly actionable or something like that. That doesn't mean you should pivot to something that is just a commodity/everyone else does it anyway type of thing.

Rattengift's Shortform

Rattengift1d110

What would be the most efficient way to "uplift" to competency a relatively-intelligent person whose education in mathematics stopped prior to high school?

Showing 3 of 6 replies (Click to show all)

Rattengift15h10

Lots of things: Their job is in something they aren't really suited for given the educational background they have. Failing to understand computer science textbooks because things that the authors see as relatively basic just aren't there; striking out during the quantier parts of interviews at even relatively routine things (like basic mental arithmetic or trigonometry problems or "understanding what a logarithm is"); not having a real understanding of or capability to understand information theory. Just sort of a systemic failure to have been taught anyt... (read more)

7Random Developer17h

Math is one of those subjects where everything builds on itself, step-by-step. And skipping prerequisites for something makes it much harder to learn. I would start by figuring out the last math that you know fairly well (multiplication tables? fractions? pre-algebra?), and then figuring out what math you would like to know. The three big pieces of high school math in the US are algebra, geometry and (depending on the school) calculus. It's possible to dislike algebra but really enjoy geometry. My biggest advice for math is to do the problem sets! If you don't, it will sneak up on you and you'll start losing track. Try looking at new ideas from multiple angles, and ask yourself how it works. Play around with it. The textbook and the problem sets are guides, and the real work happens in your head. My best physical analogy is getting strong. You gotta lift the weights and do the reps, and that takes time. This gets a lot of clever people in trouble, because they try to push too far, too fast, and then they get lost. If that happens, go back study the previous step some more.

7Zack_M_Davis18h

Math Academy ($49/mo) is great for training the fundamentals; one of the developers has a manifesto.