My colleagues and I were arguing about the nature of LLM intelligence and generalization. (In particular, they were talking about this paper: [2507.06952] What Has a Foundation Model Found? Using Inductive Bias to Probe for World Models and using Kepler/Newton as an example). This is the only eval I know of that hits the question directly. If you want funding, make a nice website where people can sign up to play it (depending on your worries about data leakage? maybe you'd create a public-set-version?) and you show a good leaderboard. & you can solicit donations. This feels like ARC-AGI-3, sorta. (OTOH, although scientifically interesting, this project might be doom-increasing. "Feeding evals to capabilities labs" and all that. If I were aiming for AGI, this is the benchmark I would hill-climb.)
Yes---but when some people say "I think there is danger here" and others say "I think there is no danger here," most people (reasonably!) resolve that to "huh, there could be some danger here"... and the possibility of danger is a danger.
Would you care to start now by giving an example?
Seconded. The new hat and the pointier, greyer beard have taken him from "internet atheist" to "world-weary Jewish intellectual." We need to be sagemaxxing. (Similarly, Nate-as-seen-on-Fox-News is way better than Nate-as-seen-at-Google-in-2017.)
I like this. My biggest difference with the implicit model is that I really think a "pause-and-resume" changes the game a whole lot. My p(workable alignment solution) is very different based on whether it comes after pausing-and-resuming versus coming in the near term! So I suppose I'd have a "branching" there? But one could argue for other branchings. And you gotta make design choices as some level! Good work.
I second this. "If Anyone Builds It, Everyone Dies" triggers the "find out if this is insane crank horseshit" subroutine. And one of the quickest/strongest ways to negatively resolve that question is credible endorsements from well-known non-cranks.
I suggest restoring "This is the current forefront of both philosophy and political theory. I don't say this lightly" to Grimes's blurb.
But anyhow: congratulations. Exciting. Bullish for the human race. Bullish for lightcone value.
I don't have any connections I can offer to any of the following, but they would be more worth a shot than average: Marjorie Taylor Greene, John Fetterman, Joe Rogan, Rishi Sunak, Leo XIV, or Ecumenical Patriarch Bartholomew.
Thanks for covering this. I urge all New York State residents (and heck, everyone else) to call governor Hochul and urge her to sign the bill. Such interventions really do matter!
I've come to believe that the most (neglected x tractable x important) thing to do right now is Spread The Word. Ordinary folks have quite a bit more power, vigor, and wisdom than they're (implicitly) given credit for around here. ("Agency" is the trendy way to put it, I think.) Estimates of "$0.01 per view on an AI risk explainer video" (https://manifund.org/projects/scaling-ai-safety-awareness-via-content-creators) or "$0.10 per click with a 5-6% share rate" (https://manifund.org//projects/testing-and-spreading-messages-to-reduce-ai-x-risk?tab=comments#1d592d6c-09f7-47bc-8d18-5675fa76556e) seem like darn good deals to me. (Adjust of course for extreme uncertainty and book-talking.)
I intend to fund these efforts, and I would be even more interested in funding an effort run by someone with serious intentions and expertise in Going Viral (or whatever it is getting tiktok famous consists of nowadays). If AI notkilleveryoneism had 1% of the reach of abortion/climate/Holy Land content, we'd live in a much better world.
I think algorithm-optimization (along with its father, Advertising) is a dark art. But darn if I don't want some dark artists on our side.
Edit: I neglected to mention
https://manifund.org/projects/effective-ai-awareness-improving-evidence-based-ai-risk-communication
and https://manifund.org/projects/creating-making-god-an-accessible-feature-documentary-on-risks-from-agi
Yikes. If they're telling the truth about all this---particularly the "useful for RL on hard-to-verify-solution-correctness problems"---then this is all markedly timeline-shortening. What's the community consensus on how likely this is to be true?