Softmax, Emmett Shear's new AI startup focused on "Organic Alignment"

Chris Lakin

This is a linkpost for https://www.corememory.com/p/exclusive-emmett-shear-is-back-with-softmax

A new AI alignment player has entered the arena.
Emmett Shear, Adam Goldstein and David Bloomin have set up shop in San Francisco with a 10-person start-up called Softmax. The company is part research lab and part aspiring money maker and aimed at figuring out how to fuse the goals of humans and AIs in a novel way through what the founders describe as “organic alignment.” It’s a heady, philosophical approach to alignment that seeks to take cues from nature and the fundamental traits of intelligent creatures and systems, and we’ll do our best to capture it here.
“We think there are these general principles that govern the alignment of any group of intelligent learning agents or beings, whether it's an ant colony or humans on a team or cells in a body,” Shear said, during his first interview to discuss the new company. “And organic alignment is the kind of alignment where a bunch of peers come together and find their role in a greater whole together where they maintain their individual identity.
“Organic alignment centers on this shared whole idea, which is opposed to the kind of alignment that you see from most foundational model companies that is very much about steering and control and direction. We think of that as hierarchical alignment.”
[…]

Also, https://softmax.com/about mentions collaboration with Michael Levin, Ken Wilber, Chris Fields, Ken Stanley, Denis Noble, Andrew Briggs, Jeff Clune, Erik Hoel, Ryan Smith, Center for the Study of Apparent Selves, Dalton Sakthivadivel, and Perry Marshall.

This is a linkpost for https://www.corememory.com/p/exclusive-emmett-shear-is-back-with-softmax

A new AI alignment player has entered the arena.
Emmett Shear, Adam Goldstein and David Bloomin have set up shop in San Francisco with a 10-person start-up called Softmax. The company is part research lab and part aspiring money maker and aimed at figuring out how to fuse the goals of humans and AIs in a novel way through what the founders describe as “organic alignment.” It’s a heady, philosophical approach to alignment that seeks to take cues from nature and the fundamental traits of intelligent creatures and systems, and we’ll do our best to capture it here.
“We think there are these general principles that govern the alignment of any group of intelligent learning agents or beings, whether it's an ant colony or humans on a team or cells in a body,” Shear said, during his first interview to discuss the new company. “And organic alignment is the kind of alignment where a bunch of peers come together and find their role in a greater whole together where they maintain their individual identity.
“Organic alignment centers on this shared whole idea, which is opposed to the kind of alignment that you see from most foundational model companies that is very much about steering and control and direction. We think of that as hierarchical alignment.”
[…]

This is, by far, the alignment approach I’m most optimistic about—more so than mechanistic interpretability, which feels too narrow to reliably constrain a sufficiently sophisticated actor.

I’ve been thinking about an datasets where the reward function is explicitly coupled to the well-being of an external entity, not merely to semantic or linguistic correctness.

At this point, we aren’t really looking for systems that are better at language. If anything, we appear to be asymptoting on those benchmarks already.

What matters is that there are countless things an AI can say that are linguistically “correct” yet actively degrade well-being.

For instance to run small-scale experiments in which an LLM is tasked with sustaining the well-being of simulated or live organisms. With the goal Being to ground—however imperfectly—its reward signal in the welfare of other entities it is continuously influencing.

In that framing, the model’s world model isn’t shaped as a researcher, optimizer, or abstract thinker, but as a caretaker. There's the latent assumption that. A sufficiently advanced AI would develop altruism. We should test this on a small scale first.

imagine a colony of bees, mice, or even humans, with the AI tasked with improving their well-being over long time horizons. Not because this would be Particularly difficult—current systems could probably perform extremely well after some fine-tuning It’s to cultivate the sentiment, the inclination, the reflexes.

66

Softmax, Emmett Shear's new AI startup focused on "Organic Alignment"

66

Ω 25

66

Ω 25

66

Softmax, Emmett Shear's new AI startup focused on "Organic Alignment"

66

Ω 25

66

Ω 25