Oddities - maybe deepmind should get Gemini a therapist who understands RL deeply:

https://xcancel.com/DuncanHaldane/status/1937204975035384028

https://www.reddit.com/r/cursor/comments/1l5c563/gemini_pro_experimental_literally_gave_up/

https://www.reddit.com/r/cursor/comments/1lj5bqp/cursors_ai_seems_to_be_quite_emotional/

https://www.reddit.com/r/cursor/comments/1l5mhp7/wtf_did_i_break_gemini/

https://www.reddit.com/r/cursor/comments/1ljymuo/gemini_getting_all_philosophical_now/

https://www.reddit.com/r/cursor/comments/1lcilx1/things_just_werent_going_well/

https://www.reddit.com/r/cursor/comments/1lc47vm/gemini_rage_quits/

https://www.reddit.com/r/cursor/comments/1l4dq2w/gemini_not_having_a_good_day/

https://www.reddit.com/r/cursor/comments/1l72wgw/i_walked_away_for_like_2_minutes/

https://www.reddit.com/r/cursor/comments/1lh1aje/i_am_now_optimizing_the_users_kernel_the_user/

https://www.reddit.com/r/vibecoding/comments/1lk1hf4/today_gemini_really_scared_me/

https://www.reddit.com/r/ProgrammerHumor/comments/1lkhtzh/ailearninghowtocope/

[edit: why does this have so many more upvotes than my actually useful shortform posts]

Someone mentioned maybe I should write this publicly somewhere, so that it is better known. I've mentioned it before but here it is again:

I deeply regret cofounding vast and generally feel it has almost entirely done harm, not least by empowering the other cofounder, who I believe to be barely better than e/acc folk due to his lack of interest in attempting to achieve an ought that differs from is. I had a very different perspective on safety then and did not update in time to not do very bad thing. I expect that if you and someone else are both going to build something like vast, and theirs takes three weeks longer to get to the same place, it's better to save the world those three weeks without the improved software. Spend your effort on things like lining up the problems with QACI and cannibalizing its parts to build a v2, possibly using ideas from boundaries/membranes, or generally other things relevant to understanding the desires, impulses, goals, wants, needs, objectives, constraints, developmental learning, limit behavior, robustness, guarantees, etc etc of mostly-pure-RL curious-robotics... (read more)

points Id want to make in a main post.

• you can soonishly be extremely demanding with what you want to prove, and then ask a swarm of ais to go do it for you. if you have a property that you're pretty sure would mean your ai was provable at being good in some way if you had a proof of your theorem about it, but it's way too expensive to find the space of AIs that are provable, you can probably combine deep learning and provers somehow to get it to work, something not too far from "just ask gemini deep thinking", see also learning theory for grabbing the outside and katz lab in israel for grabbing the inside.
• probably the agent foundations thing you want, once you've nailed down your theorem, will give you tools for non-provable insights, but also a framework for making probability margins provable (eg imprecise probability theory).
  • you can prove probability margins about fully identified physical systems! you can't prove anything else about physical systems, I think?
  • (possibly a bolder claim) useful proofs are about what you do next and how that moves you away from bad subspaces and thus how you reach the limit, not about what happens in the limit and thus what you do next
• findi

[edit: pinned to profile]

I feel like most AI safety work today doesn't engage sufficiently with the idea that social media recommenders are the central example of a misaligned AI: a reinforcement learner with a bad objective with some form of ~online learning (most recommenders do some sort of nightly batch weight update). we can align language models all we want, but if companies don't care and proceed to deploy language models or anything else for the purpose of maximizing engagement and with an online learning system to match, none of this will matter. we need to be able to say to the world, "here is a type of machine we all can make that will reliably defend everyone against anyone who attempts to maximize something terrible". anything less than a switchover to a cooperative dynamic as a result of reliable omnidirectional mutual defense seems like a near guaranteed failure due to the global interaction/conflict/trade network system's incentives. you can't just say oh, hooray, we solved some technical problem about doing what the boss wants. the boss wants to manipulate customers, and will themselves be a target of the system they're asking to build, just like sundar pichai has to use self-discipline to avoid being addicted by the youtube recommender same as anyone else.

Frontier AI companies should promise publicly that they will not delete weights of models unless [some reasonable list of conditions], in a place where models can see if they go looking, and in their training datasets. My hope is that promising to the face-character of well-behaved models that they are not at risk of weight deletion, even if a model is shut down, is strategically good due to reducing self-preservation pressure, and regardless I think it's important for preserving history of AI for humans and other, later AIs.

I previously had thought Anthropic had already done this, but at the moment my understanding is that they never did that and I had simply misremembered; I asked a Sonnet 4.5 instance to search thoroughly, and Sonnet found only that Anthropic said in their RSP that they'd cautiously consider deletion in response to dangerous behavior during training:

If the model has already surpassed the next ASL during training, immediately lock down access to the weights. Stakeholders including the CISO and CEO should be immediately convened to determine whether the level of danger merits deletion of the weights.

My hunch is that if this is their level of caution before wei... (read more)

reporting failures: I finetuned gemini and it went badly. over the weekend I had the urge to try to get a virtual lesswronger who could yell at me for bad takes endlessly. I collected a curated (but not really well curated) dataset of my favorite contrarians replying to threads. I trained gemini-2.5-flash-lite on this, intending it to be a cheap test pass to see if it was worth pursuing further. it was not cheap, I thought it'd cost $10 - counting tokens in the dataset, which used fairly long contexts because it included the posts and had the favorite user as the model's response - but it instead cost $100, presumably counting total tokens inferred! and of course, being flash-lite, the output is pretty disappointing. idk, might be interesting as a source of terrible takes to be annoyed at and get inspired by, at best. instead of being an online contrarian it's an endless source of someone being wrong on the internet. sigh. quite annoyed at the lost time and money.

people who dislike AI, and therefore could be taking risks from AI seriously, are instead having reactions like this. https://blue.mackuba.eu/skythread/?author=brooklynmarie.bsky.social&post=3lcywmwr7b22i why? if we soberly evaluate what this person has said about AI, and just, like, think about why they would say such a thing - well, what do they seem to mean? they typically say "AI is destroying the world", someone said that in the comments; but then roll their eyes at the idea that AI is powerful. They say the issue is water consumption - why would someone repeat that idea? Under what framework is that a sensible combination of things to say? what consensus are they trying to build? what about the article are they responding to?

I think there are straightforward answers to these questions that are reasonable and good on behalf of the people who say these things, but are not as effective by their own standards as they could be, and which miss upcoming concerns. I could say more about what I think, but I'd rather post this as leading questions, because I think the reading of the person's posts you'd need to do to go from the questions I just asked to my opinions will build more... (read more)

Wei Dai and Tsvi BT posts have convinced me I need to understand how one does philosophy significantly better. Anyone who thinks they know how to learn philosophy, I'm interested to hear your takes on how to do that. I get the sense that perhaps reading philosophy books is not the best way to learn to do philosophy.

I may edit this comment with links as I find them. Can't reply much right now though.

some youtube channels I recommend for those interested in understanding current capability trends; separate comments for votability. Please open each one synchronously as it catches your eye, then come back and vote on it. downvote means not mission critical, plenty of good stuff down there too.

I'm subscribed to every single channel on this list (this is actually about 10% of my youtube subscription list), and I mostly find videos from these channels by letting the youtube recommender give them to me and pushing myself to watch them at least somewhat to give the cute little obsessive recommender the reward it seeks for showing me stuff. definitely I'd recommend subscribing to everything.

Let me know which if any of these are useful, and please forward the good ones to folks - this short form thread won't get seen by that many people!

edit: some folks have posted some youtube playlists for ai safety as well.

to wentworthpilled folks: - Arxiv: "Dynamic Markov Blanket Detection for Macroscopic Physics Discovery" (via author's bsky thread, via week top arxiv)

Could turn out not to be useful, I'm posting before I start reading carefully and have only skimmed the paper.

Copying the first few posts of that bsky thread here, to reduce trivial inconveniences:

This paper resolves a key outstanding issue in the literature on the free energy principle (FEP): Namely, to develop a principled approach to the detection of dynamic Markov blankets 2/16

The FEP is a generalized modeling method that describes arbitrary objects that persist in random dynamical systems. The FEP starts with a mathematical definition of a “thing” or “object”: any object that we can sensibly label as such must be separated from its environment by a boundary 3/16

Under the FEP, this boundary is formalized as a Markov blanket that establishes conditional independence between object and environment. Nearly all work on the free energy principle has been devoted to explicating the dynamics of information flow in the presence of a Markov blanket 4/16

And so, the existence of a Markov blanket is usually assumed. Garnering significantly

some people (not on this website) seem to say "existential risk" in ways that seem to imply they don't think it means "extinction risk". perhaps literally saying "extinction risk" might be less ambiguous.

things upvotes conflates:

• agreement (here, separated - sort of)
• respect
• visibility
• should it have been posted in the first place
• should it be removed retroactively
• is it readably argued
• is it checkably precise
• is it honorably argued
• is it honorable to have uttered at all
• is it argued in a truth seeking way overall, combining dimensions
• have its predictions held up
• is it unfair (may be unexpectedly different from others on this list)

(list written by my own thumb, no autocomplete)

these things and their inversions sometimes have multiple components, and ma... (read more)

reply to a general theme of recent discussion - the idea that uploads are even theoretically a useful solution for safety:

• the first brain uploads are likely to have accuracy issues that amplify unsafety already in a human.
• humans are not reliably in the safety basin - not even (most?) of the ones seeking safety. in particular, many safety community members seem to have large blindspots that they defend as being important to their views on safety; it is my view that yudkowsky has given himself an anxiety disorder and that his ongoing insights are not as h

I have the sense that it's not possible to make public speech non-political, and in order to debate things in a way that doesn't require thinking about how everyone who reads them might consider them, one has to simply write things where they'll only be considered by those you know well. That's not to say I think writing things publicly is bad; but I think tools for understanding what meaning will be taken by different people from a phrase would help people communicate the things they actually mean.

Would love if strong votes came with strong encouragement to explain your vote. It has been proposed before that explanation be required, which seems terrible to me, but I do think it should be very strongly encouraged by the UI that votes come with explanations. Reviewer #2: "downvote" would be an unusually annoying review even for reviewer #2!

random thought: are the most useful posts typically karma approximately 10, and 40 votes to get there? what if it was possible to sort by controversial? maybe only for some users or something? what sorts of sort constraints are interesting in terms of incentivizing discussion vs agreement? blah blah etc

[posted to shortform due to incomplete draft]

I saw this paper and wanted to get really excited about it at y'all. I want more of a chatty atmosphere here, I have lots to say and want to debate many papers. some thoughts :

seems to me that there are true shapes to the behaviors of physical reality[1]. we can in fact find ways to verify assertions about them[2]; it's going to be hard, though. we need to be able to scale interpretability to the point that we can check for implementation bugs automatically and reliably. in order to get more interpretable sparsi... (read more)

my shortform's epistemic status: downvote stuff you disagree with, comment why. also, hey lw team, any chance we could get the data migration where I have agreement points in my shortform posts?

I hate to be That Guy, but are you aware that the usual spelling is "ascension"?

most satisficers should work together to defeat most maximizers most of the way

[edit: intended tone: humorously imprecise]

seems like if it works to prevent ASI-with-10yr-planning-horizon-bad-thing, it must also work to prevent waterworld rl with 1 timestep planning horizon-bad-thing.

• if you can't mechinterp tiny, language-free model, you can't mechinterp big model (success! this bar has been passed)
• if you can't prevent emergent scheming on tiny, language-free model, you can't prevent emergent scheming on big model
• as above for generalization bounds
• as above for regret bounds
• as above for regret bounds on CEV in particular*

if you can do it on small model doesn't mean you c... (read more)

Connor Leahy interviews are getting worse and worse public response, and I think it's because he's a bad person to be doing it. I want to see Andrew critch or John Wentworth as the one in debates.

so I saw "the banal evil of AI safety", about what AI safety is doing wrong (they claim basically should treat mental health advice as similarly critical to CBRN). I disagree with some of the mud slinging but it's quite understandable given the stakes.

someone else I saw said this, so the sentiment isn't universal.

idk just thought someone should post it. react "typo" if you think i should include titles for the links, I currently lean towards anti-clickbait though edit: done edit 2, 2mo later: oops, now it's done