Quick Takes

If you’re building any kind of dev tool, stop investing in UIs, CLIs, and MCP. Instead, invest in APIs/SDKs and documentation, which make your tool more useful for LLMs. LLMs are becoming extremely adept at using native low-level tools so it doesn’t make sense to build wrappers around these that aren’t in the training data. * Many have observed that LLMs use bash commands in effective yet alien ways learned through RL. Another phenomenon is that LLMs used to use Playwright to manipulate browsers, but now they just run CDP commands directly. LLMs also prefer to run Python snippets directly instead of CLI wrappers around the same library, especially if the CLI is not in the training data / not well documented. * MCP servers are useful for integrations with apps like Claude Desktop, but for devtools in particular, the raw API is more useful for the LLM (for example, the LLM can write API output to a file instead of processing it in-context). * A good SDK can provide helpful guardrails for the agent. For example, implementing client session management can prevent potential issues and also removes boilerplate. * UIs are still useful but become less useful as LLMs build more user-specific interaction layers for viewing and manipulating data. For example, a database provider like Supabase provides a full UI right now but in the future they may just provide raw visual components which software teams use to construct their own bespoke UIs. Documentation (right now mostly via agent skills) is extremely high ROI and will continue to be higher ROI as more work is delegated to agents. Your service is much less useful if an agent can’t one-shot basic actions when given documentation. * My personal opinion is that documentation is much more useful when a human writes it than an LLM. The LLMs that write documentation are usually not opinionated enough, do not understand which user flows to highlight, and make factual errors. * Documentation is especially helpful in descri

AI safety people often worry about their work having capabilities externalities. I feel like the inverse problem is often a much bigger deal: the stuff you do might be done later by the capabilities/product people in the course of their work, meaning your work isn't counterfactual! I think people should try to distinguish these concerns more clearly.

The Ten People On The Inside likely should prioritize things that capabilities teams aren't going to do anyway and people considering joining labs with similar motivations should likely think about whether they're sure they're going to be able to do work that wouldn't otherwise be done for capabilities reasons. The thing that's top of mind here is reward hacking, which is increasingly becoming a capabilities problem.[1] To the extent research is done here, it seems pretty important to avoid the failure mode of "repeatedly iterate on environments and observable behavior"[2] which is the kind of thing I would expect by default for capabilities reasons. 1. ^ A good example of research on reward hacking that I don't think would've necessarily gotten done "by default for capabilities reasons" is inoculation prompting and similar investigations 2. ^ This can take the form of specific safety training targeting the observable behavior, post-hoc environment fixes, or having an iteration loop between your pre-deployment automated alignment evals and your alignment training

Various actual token spend vs labor cost ROIs on 1-20 hour tasks from SemiAnalysis' recent newsletter: [...]

Is it officially "LessWrong" now? Or is it still "Less Wrong"? Does it matter? I feel like "LessWrong" is more streamlined and futuristic. It's solid at its center of gravity, like a noun, whereas "Less Wrong" feels inelegant as an object in a sentence (try saying "I read posts on Less Wrong" out loud with equal emphasis on the last two words). But Less Wrong seems to be the name the founders intended. Is it left that way in the Sequences just for historical purposes? I get the impression of a gradual shift, endorsed but natural, towards "LessWrong".[1] I think this is the kind of incremental rebranding that non-stagnant organizations undergo naturally.[2] Some people react badly to rebrands (if it ain't broke, don't fix it), but they're a sign of life. 1. ^ e.g. the titles of the welcome posts. 2. ^ Organization, as in, the abstract, intangible hub around which the members orbit, which presents itself to the world through a brand, a self-described purpose, an archetype of the person who is a member, etc. It can be a company, a school, a religious group, a collaborative world-building project...

Claude Opus 4.7 can reliably truesight me on any unpublished blog post draft or internal memo (including on topics I've never written publicly before on). It can also reliably know who I am in 3-5 excerpts of online comments (that I think *probably* is after the training cutoff), including from accounts that I don't publicly acknowledge a connection with (though I don't go out of my way to hide either). It can't quite do it on excerpts of short fiction or private DMs yet (at least when I remove obvious clues like links to my substack), though I feel like this can't be long coming.

The eventual success of rationalists will result in the death of the label "rationalists." Take "liberalism" as an example. In the eighteenth century, liberals were a specific political group that opposed absolute monarchy (I'm oversimplifying a bit). Now, mostly everyone opposes absolute monarchy. The "liberal" label has come to refer to left-wing instead. Similarly, if rationalist thought becomes mainstream in 200 years (assuming we all haven't died), it won't be called rationalism anymore. The stuff in the Sequences will just be obvious thinking habits. So if we want to make less people hate rationalism and effective altruism, then we might have to ditch the labels. There's a lot of mainstream animosity toward rationalists and EAs because news outlets like to portray them as crazy San Francisco doomer polycule vegans, not as a group of people who possess new but reasonable tenets of thinking. By "tenets of thinking" I don't mean AI doomerism—I just mean things like "you should donate to effective charities" and "you should make beliefs pay rent." It should be easier for people to accept these tenets without subscribing to stereotypical rationalist/EA beliefs on AI, animal welfare, etc. LessWrong isn't optimal from this perspective, since it's sort of an echo chamber, and it may overwhelm new users. It will be a good day when some famous politician, perhaps from New York, publicly discusses how they form beliefs without mentioning the word "rationalism" once.