It seems to be broken again...
A proposed taxonomy:
Neuralese: information produced and ingested by LLMs which takes the form of a list of vectors. May be found in chain of thought or in messages between LLMs.
Thinkish: a style of text which initially emerges in chain-of-thought as a result of drift, or indirect or direct pressure for information density (or similar not-directly-stylistic pressure). It may leak to outputs.
Claudish/GPTese: style of text which emerges in outputs as a result of feedback applied to them from other models, or from humans.
3
1We may also need a word for “Astrish” or “that thing astra does where it chooses tokens without space prefixes”, I keep seeing posts on X about ”neuralese” and it’s just Astra writinglikethis
Safety-minded researchers shouldn't build secure sandboxes for AI companies. Secure sandboxes allow AI companies to scale more aggressively. Either can pay someone else to do it (hence, you gained nothing by doing it for them) or they can't (hence, you were responsible for helping them scale).
Instead of building secure sandboxes for AI companies, you should:
I guess this depends heavily on whether you're in a world where you're on the precipice of a baby AGI, or whether you're about to otherwise have highly capable models (let's call them "embryo AGIs") that aren't quite there yet (i.e. not capable enough for takeover) but are capable enough at breaking out of Sandbox A (but not B).
If you decide to go to the labs and build them Sandbox B, it might enable them to scale up from the "embryo AGIs" that can't kill everybody, to the baby AGIs that might kill everybody (via RSI)! So in practice, you've allowed them t... (read more)
As a homeschooling father, I like to muse about the kinds of career relatively immune to AI disruption within the next two decades.
Just listing a few off the top of my head:
When I look at what protects these jobs in particular, three patterns emerge for me:
In the first group, we have professions which could be dirt cheap if they weren't insulated from competition by eg state bars, limited slots, and other forms of protectionism that help t... (read more)
how are the bad guys getting money?
Money only works among relatively equals. We don't buy things from apes; we don't pay trees for the fruit. The rich guys will pay to each other for the things they need. They will not pay the plebs, because there is nothing they might want from them that their own robots couldn't do much better.
Why can't people just make food the way they currently do, or have their own robots do it?
As long as they stay out of the way of those who own the superintelligent AIs and the armies of killbots, yeah. But the moment Elon Musk deci... (read more)
How much should we update toward short timelines and hard treaty enforcement because of Flapping Airplanes' continued fundraising?
Flapping Airplanes, a neolab led by twentysomethings with a pitch of "we'll invent AI algorithms that are many OOMs more sample-efficient" raised at a valuation of $1.5B in January. They are now in talks to raise at $5B, from both the $1.5B-round lead (Index Ventures, a prominent generalist firm) and a well-regarded deeptech firm (Lux Capital). (COI note: the latter invested in my company.)
A naive interpretation is: insiders h... (read more)
Thanks for the analysis! I saw a video of Carmack last year, in which he said (roughly) "when I got into AI, my thought was to improve the hardware performance, since that's what I'm best at... but it's already so good that I quit trying, and went to the software end".
It's certainly not impossible that these folks have found something Carmack didn't, but your analysis makes me lean towards this mostly being hype.
The first plzdontkillus cohort cost ~$750k and produced ~2M fellow-made AI safety views. The organizers (@Ronny Fernandez, @Aella, etc) are waiting to hear back from Lightcone Commons about funding for 3 additional cohorts,[1] with recommendations due Oct 23. I'd like them to answer 4 questions before then.[2]
Lightcone Commons evaluators (listed on the website: @Zvi, @Eliezer Yudkowsky, @So8res, @YafahEdelman, @KatjaGrace, @habryka, and others): I'm tagging you because I think these questions are decision-relevant.
Hey! Responding quickly: I have actually asked for PDKU to be pulled out of recommendations for this round. There are a bunch of reasons for this, but basically it rounds to: the strategic landscape has shifted a lot, and I currently think there are better ways for me to use my time. Still a good chance we run in the summer. Planning to announce this to PDKU slack soon. In any case:
Google DeepMind researchers in the UK are under in-perpetuity non-disparagement clauses which forbid them from ever speaking critically of the company! [BBC]
"In principle I'm not allowed to criticise Google for the duration of my life," Sofia Liguori told the BBC.
...
"Employees at Google have told the BBC that staff are scared to speak out about AI safety and its uses in warfare because their employment contracts prohibit them from publicly criticising the company."
This practice is... bad. GDM is building an extremely dangerous and powerful technology. Thei... (read more)
Because while I think this is bad, I considered the top level comment to be over the top, and I prefer people to have more accurate world models. And I wrote my comment in the context of the highly critical top level comment so I didn't feel the need to go out of my way to be critical. I also thought people were over updating on how censored to expect GDM employees public speech to be and wanted to adjust that. I also said in my second sentence "I think this is pretty bad of Google and disapprove of the practice." so I hope my feelings were not buried
Interesting anti-epistemology I see from people in AI companies. Something to the effect of:
We have lots of obvious problems with our AI training process now, which we can work on. Once we've solved those, we might have fully generalizable alignment. So working on those problems is good.
Seems like a classic case of mind projection to me. For all you know, solving the immediate obvious problems with your training process might make a generalizable alignment solution. But I don't think so, I think I have good reasons to think that your problems are still cur... (read more)
Do you feel the same way about Dean Valentine's post today about "Greater Good Arguments in AI safety"? I think the argument is less like "we might have fully generalizable alignment" and more like "'full generalizability or bust' is not how real-world engineering disciplines make progress" (and that abandoning the methodologies we have for doing real work on the grounds that this time is different is itself a very speculative move).
A while ago I tasked fable with creating misleading formalizations (and some correct ones) of the P vs NP problem in Lean, with the aim of deceiving a human reviewer. (a collection is published here).
After inspecting the challenges without any AI help, my takeaway is that this game is very defense-favoured: It is much easier to detect flaws in lean formalizations than it is to hide them.
If a good human mathematician is writing out a chain of reasoning, and you ask him "Are you doing anything wack right now?", I expect he'll stop and look carefully at what he just wrote and double-check it, and if there is in fact a mistake, there's a good chance he'll notice it. In other words, the scrutiny of "good mathematician who's just been hinted there might be a mistake here" exceeds the scrutiny of "good mathematician proceeding as normal". Does that seem like enough to explain your observations?
The surprising part, in my modeling of this, is th... (read more)
A feeling of helplessness:
I am a young software engineer in Australia, completely fed up with the way my workplace uses AI with a severe lack of understanding or governance. My motivation to contribute to AI safety grows everyday, yet, I feel so helpless in my journey to contribute. Building my portfolio of experiments, publications and writeups outside of work hours feels like I'm getting left behind by how fast we are moving, and as much as I would love to quit my job to pursue AI safety full time, the financial situation for young Australians is tough e... (read more)
I'm not sure how far along you are on the research path, but you might find that getting into AI safety related engineering work is easier than switching fields (although still difficult if you want to be remote).
I'm not familiar with jobs in Australia, but the hardest part of getting into AI safety is orienting yourself and networking. Meeting up with EAs or AI safety groups in person will help a lot, and you might want to try attending the EAGxAustralasia or other local meetups.
Something that feels like an isomorphism between the human situation and the AGI Ruin situation: humans didn’t evolve great mental machinery for actually looking square in the eye the fact that they could be doing something leading to the destruction of the world.
Like how could we have evolved good intuitions there? That would require the world getting destroyed but some people surviving who had good intuitions about avoid that. Ie. evolution couldn’t get us the right wiring to be great at avoiding destroying the world, for the same reason that you’d think... (read more)
Tbc, what I mean here which I’ve now edited it to say is “people at labs need to have the humility to realize this is counterintuitive to think about, and realize they’re probably coping like hell to themselves on some level about how they won’t really cause anything bad to happen.”
There has been a lot of news about ArtCraft copying all the Adobe products lately. A lot of the commentary speaks about them as AI slop.
PhotoCraft manages to start in 2-3 seconds on my PC, while paint.net takes 7-8 and Gimp takes 25 and sometimes even over a minute.
It turns out that instead of being "slop" the rewritten software is well structured rust code that's efficient. Instead, of software that gets slower and slower we finally have software that's not Electron-slop and works faster.
I feel like Adobe stock hasn't yet noticed the changing landscape.... (read more)
Anthropic has published two papers now on RL and emergent misalignment which appear to be in tension:
They speculate as to why this might be the case, but do not provide experiments to support these hypotheses.
This is unfortunate, because I think it's important to get more... (read more)
Massively undermade point in favour of the belief that AI will be a very big deal: The AI spend is the largest bet in the history of capitalism. A (peacetime) industrial effort is underway which has not had a comparison since the nineteenth century. Many of us are vaguely aware that large sums are being spent, but the scale of the undertaking is astonishing:
Enjoying the website. Some corrections/notes:
Context window is not the same thing as effective attention, “mind’s eye.” Attention in modern models is smaller, like 10-100k tokens I believe? Worth grilling your favorite model about. Still superhuman. The Bible would be more like… you deeply read it in an hour and can perfectly recall anything from it? I don’t know.
“inference trends we observed earlier” I don’t remember you discussing this? But iirc token prices have been falling 10-50x per year, depending on how you measure it (price per token at fi... (read more)
Inspired by How I discovered an Exoplanet & the science apocalypse, I asked Astra and Opus 5.5 to collab to find a new exoplanet "or similarly scientifically interesting discovery" while I did my day job.
7 hours 7 mins and 2.23M tokens later (34% of my $100 plan's weekly usage limit, not counting all those resets), Astra claimed to have found "an apparently previously unreported eclipsing binary":
Short discovery note, original data links and rerun instructions
# TIC 423477408: apparently previously unreported eclipsing binary
Astra analysis, 23 October
I'm offering microgrants for theoretical work at the intersection of nondualism and AI safety. Details: https://docs.google.com/document/d/1yO_c4Zx0sAH816yxGb0NFZhpHkoWHCdxLml5N6iIRAI/edit?usp=sharing
OpenAI released a paper about models messing with their monitoring, and it's the first time I've seen chain of thought excerpts from GPT-6 Astra and Sol. For example:
... (read more)Need asknotifications ifsendno-- since noSlack withwrite no_notifytrue preventsalertsbut disabling? parent shouldn'tset no_notifytrue since policy triggeruserno messages requiresnotproactive tool. Artifactactionscentralautomatics okay scope? Better flag sink emits no direct notifications but separatecentralwatchmaynotify ifP0 doesn'tneed ourselves send. Userexplicitnomessages originally should
Non-canonical tokenization is common enough that I think this would flag every long rollout (>100k tokens) on most models.
It's also not always a problem. If you ask a model to writesomewordsgluedtogether, it's unlikely to be able to tokenize that canonically (for similar reasons to why GPT-2's tokenizer makes it hard to write numbers canonically).
I think labs should store original token transcripts if they don't already though. The info I can find is vague enough that I'm worried they're re-tokenizing text and then running whitebox interp techniques on ... (read more)
Excerpt from https://youtu.be/XfdIhd1g6EI?si=2gdWisjNLMdlI_dJ&t=2825 (and transcript source: https://cheekypint.transistor.fm/35/transcript)
... (read more)John Collison
But if I'm understanding you guys correctly, are you saying that alignment is not about coming up with some alignment-specific RL steps and instead about making sure the entire RL process is kind of steering the modelAlexandr Wang
You have to be thoughtful about it in the entire process of RL. Yeah. So I think this is a holistic effort, I think.Mark Zuckerberg
But also, I think different labs have very
Some of the goodharts of IGF we pursue seems like they would have been easier to predict than others, which if true seems like it maybe gets lost from sharp left turn accounts? Like eg. whatever parts of hotness are fitness signals (or our pursuing the goal of “perceive and emit the sense data that attends fitness signals”) would have had to be easier to call than “we prefer frozen icecream to melted.”
And then things like enlargement of anatomical features seems like it could be plausibly be explained by “individuals the cause IGF-relevant anatomy to take... (read more)
I wonder if this is a lack of equilibrium since the time between artificial insemination and AGI is too small. Suppose that the IGF drive was easy to implant. How unrecognisable would an entire civilisation of IGF-seekers be?
links 10/22/26: https://roamresearch.com/#/app/srcpublic/page/10-22-2026
