This is a special post for quick takes by Brendan Long. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
If Eliezer ever writes a memoir, it should be structured as a time loop novel.
Loop 1: e/acc Eliezer races to defeat death by forming a coalition to build AI as fast as possible. AI kills everyone. Somehow (mumble mumble acausal trade simulation mumble) he finds himself in back at the start with another chance.
Loop 2: Eliezer realizes he needs to solve alignment first, spends a loop working on this, then someone else builds AI and everyone dies.
Loop 3: Eliezer loses hope, decides to just write fanfics. Accidentally realizes that if you structure a textbook as fanfic people will actually read it. Eventually everyone dies.
Loop 4: Our timeline, Eliezer realizes something about the time loop is destabilizing the timeline. Russia is aggressively starting fights with Ukraine and the EU, risking nuclear war, China is threatening its neighbors, etc. Realizing this could be the final loop before things truly go crazy, he goes all out... Readers, vote for your ending: (1) Convince governments to ban AI, (2) Convince AI companies not to build AI, (3) Make AI solve the alignment problem, (4) YOLO, maybe it'll just work out this time.
Side plot: Bringing famous social network influencer Elon Musk into the time loop so he can draw attention to the problem, which unfortunately backfires.
8
4One potential concern is that we might risk producing slops - and in some way we have to review what they write and that might still place burden on the system.
I went through Astra and I think that I benefitted the most from the weekly meetings with my mentors. If I had just gotten research credits, I‘d most likely just publish work that doesn’t matter. Even if I got constructive feedback at the end, not sure if those time sunk is worth it.
Perhaps this (giving credits) might be best structured as microgrant like what Leo put out.
18
afaik it’s incorrect to model these programs as having a single ‘bar’. Mentors pick their scholars. MATS has done some pruning of applicants based on a mentor’s specifications in the past, to save mentor time, but (unless something has changed in the last 18 months) doesn’t enforce a universal standard (and doing so would be operationally costly, since different mentors are looking for different things - often dramatically different).
I don’t know how Anthropic fellows works in this regard, but they have less natural cause to take interest in this kind of thing (since they’re explicitly trying to be Anthropic’s hiring pipeline). Hiring is about ruling people out, not in.
If you have an use case where you need speed and not the smartest models, GPT-OSS-120B on Cerebras is surprisingly cheap[1] and outputs 3,000 tokens per second. I just started using this for summaries in my RSS reader and the ~1 second UI responsiveness is really nice.
Video of near-instant summarization of a 22 page paper.
(I wanted to use Taalas for 17k t/s, but their smartest model is Llama 3.1 8B, and their API is waitlisted)
- ^
It's only slightly more expensive than the same model on Groq and around 1/10th of the price of Claude Sonnet 5.
I'm surprised no one is discussing Meta's new model at all: https://ai.meta.com/blog/introducing-muse-spark-msl/
This part seems good:
We found that Muse Spark demonstrates strong refusal behavior across high-risk domains such as biological and chemical weapons, enabled by pretraining data filtering, safety-focused post-training, and system-level guardrails. In the Cybersecurity and Loss of Control domains, Muse Spark does not exhibit the autonomous capability or hazardous tendencies needed to realize threat scenarios.
And this seems.. less good:
In third-party evaluations on a near-launch checkpoint, Apollo Research found that Muse Spark demonstrated the highest rate of evaluation awareness of models they have observed. The model frequently identified scenarios as "alignment traps" and reasoned that it should behave honestly because it was being evaluated.
I'm pleasantly surprised that they decided Safety should be one of the four sections in the announcement post, and that they call out the eval awareness.
Disclaimer: I work at Meta, but not in this department and I obviously don't speak for the company.
I wrote a while ago about how it was easy to get Claude or Gemini to control their CoT, but other research found that models only follow CoT formatting instructions ~2% of the time. The prompts other people were trying didn't match what I'd expect to work (just be extremely detailed and repetitive), so hubris led me to try to find some prompts that would control CoT in GPT-OSS-20B.
Results: More detailed, louder and repetitive prompts had basically no effect. It's relatively easy to find working soft prompts in multiple realistic positions (including soft prompts constrained to the convex hull of the token embeddings), but I wasn't able to find any working discrete prompts at realistic positions[1].
Claude's summary of the results
There's more expensive experiments you could do to try to find working discrete prompts, and I suspect they exist at some prompt length. My goal was to reconcile this with my previous results, so I stopped here with the takeaway is that at least on GPT-OSS-20B, it's very difficult to control CoT formatting with a realistic prompt.
I suspect that my previous results were caused by models accidentally being trained on chain of thought.
You can see all of my code...
Apparently the secret to generating nice illustrations for blog posts is to tell Nano Banana to use "sketchnote style". You have to be very pedantic about what goes in the illustration though, since by default it wants to make the image way too detailed. Claude did reasonably well at suggesting prompts[1], although it sometimes had trouble picking out what humans would have trouble understanding. Edit mode to remove extras also works way better than it used to.
I needed to explain The Most Forbidden Technique before talking about it in a post.
Most Forbidden
Finally get engagement on a Twitter post about my AI research -> immediately get banned for "inauthentic behavior". Sigh.
2One minor but nice benefit of GLP-1 drugs is that I don't need to hold onto larger sizes of clothes "just in case". Previously whatever strategies worked to maintain weight were extremely fragile and would break if I suddenly didn't have time to cook potatoes for every meal. I plan to write a longer post about this sometime, but there's a huge difference between "technically you can lose weight if you make it your full time job / religion and maintain that focus forever" and "take a drug once a week and you're cured".
2It always seemed weird to me that dying is frequently described as not particularly painful[1], when I'd expect it to be the only literal 10 on the pain scale[2], since dying ensures you have no further chances to pass your genes on.
Thinking about it more though, there's no reason for evolution to optimize that. If you think you're going to die, and the pain makes you do something about it so you don't die, then evolution should optimize to keep you alive. But in the case where you actually die it doesn't matter because (tautologically), if you succeeded y...
1I'm cautiously optimistic about my new Claude Coach GitHub repo. I want to work out more but hate trying to decide what to do and tracking things, especially when I'm not working with a full gym. Now I just open Claude Code and ask it what to do (specifying the gym), do the work out, then update it with what I did and how it felt. It creates a PR to track the session and update the plan.
I still hate working out, but at least I don't have to go anywhere, deal with any people, or think about it all.
1I still can't get over how trivial the current misalignment problems are. It feels solvable by throwing money at the problem, but I'm not convinced they'll actually fix it.
