Ben Smith

Wiki Contributions


A brief review of the reasons multi-objective RL could be important in AI Safety Research

The only resource I'd recommend, beyond MODEM, when that's back up, and our upcoming JAMAAS special issue, is to check out Elicit, Ought's GPT-3-based AI lit search engine (yes, they're teaching GPT-3 about how to create a superintelligent AI. hmm). It's in beta, but if they waitlist you and don't accept you in, email me and I'll suggest they add you. I wouldn't say it'll necessarily show you research you're not aware of, but I found it very useful for getting into the AI Alignment literature for the first time myself.

A brief review of the reasons multi-objective RL could be important in AI Safety Research

That's right. What I mainly have in mind is a vector of Q-learned values V and a scalarization function that combines them in some (probably non-linear) way. Note that in our technical work, the combination occurs during action selection, not during reward assignment and learning.

I guess whether one calls this "multi-objective RL" is semantic. Because objectives are combined during action selection, not during learning itself, I would not call it "single objective RL with a complicated objective". If you combined objectives during reward, then I could call it that.

re: your example of real-time control during hunger, I think yours is a pretty reasonable model. I haven't thought about homeostatic processes in this project (my upcoming paper is all about them!). Definitely am not suggesting that our particular implementation of "MORL" (if we can call it that) is the only or even the best sort of MORL. I'm just trying to get started on understanding it! I really like the way you put it. It makes me think that perhaps the brain is a sort of multi-objective decision-making system with no single combinatory mechanism at all except for the emergent winner of whatever kind of output happens in a particular context--that could plausibly be different depending on whether an action is moving limbs, talking, or mentally setting an intention for a long term plan.

Why I am not currently working on the AAMLS agenda

Interesting comments, thanks. Currently exploring an agenda of my own and this is food for thought.

Signaling Virtuous Victimhood as Indicators of Dark Triad Personalities

I know it's a touchy topic. In my defense, the research is solid, published in social psychology's top journal. I suppose the study deals with rhetoric in a political context. This community has a long history of drawing on social and cognitive psychological research to understand fallacies of thought and rhetoric (HPMOR), and I posted in that tradition. Apologies if I have strayed a little too far into a politicized area.

One needn't see this study as a shot at any particular political side--I can imagine people engaging 'virtuous-victimhood-signalling' within a wide range of different politicized narratives, as well as in completely apolitical contexts.

It also shouldn't be read to delegitimize victims from speaking out about their perspective! But perhaps it does provide evidence that sympathy can be weaponized in rhetorical conflict. We can all recognize this in political opponents and be blind to it amongst political allies.

Supplement to "Big picture of phasic dopamine"

Interesting. Is it fair to say that Mollick's system is relatively more "serial" with fewer parallelisms at the subcortical level, whereas you're proposing a system that's much more "parallel" because there are separate systems doing analogous things at each level? I think that parallel arrangement is probably the thing I've learned most personally from reading your work. Maybe I just hadn't thought about it because I focus too much on valuation and PFC decision-making stuff and don't look broadly enough at movement and other systems.

Apropos of nothing, is there any role for the visual cortex within your system?

I too am puzzled about why some people talk about "mPFC" and others talk about "vmPFC". I focus on "vmPFC", mostly because that's what people in my field talk about. "vmPFC" focuses much more on valuation systems. Theoretically I guess "mPFC" would also include the dorsomedial prefrontal cortex, which includes the anterior cingulate cortex, I guess some systems related to executive control, perhaps response inhibition (although that's usually quite lateral), perhaps abstract processing. Tends to be a bit of a decision-making homunculous of sorts :/ And then there's the ACC, whose role in various things is fairly well defined.

So maybe authors who talk about the mPFC aren't as concerned about distinguishing value processing from all those other things.

Conservatism in neocortex-like AGIs

As you're aware, I'm very much exploring this approach using a multi-objective decision-making approach, with conservativism through only acting when an action is non-negative on the whole set of objective functions that an actor regards.

The alternative, Bayesian AGI approach is also worth thinking about too. A conservative Bayesian AGI might not need multiple objectives. For each action, it just needs a single probability distribution of outcomes. If there are multiple theories of how to translate consequences of its actions into its single utility function, each of those theories might be given some weight, and then they'd be combined into the probability distribution. Then a conservative Bayesian AGI only acts if an action's utility function doesn't exceed below zero. Or maybe there's always some remote possibility of going below zero, and programming this sort of behavior would be absolutely paralyising. In that case maybe we just make it loss-averse rather than strictly avoiding any possibility of a negative outcome.

Propinquity Cities So Far

Two examples come to mind:

Propinquity Cities So Far

in practice, similar proposals (that have actually been implemented, both in communist and nominally capitalist countries) have vastly underestimated the difficulty of this problem, leading to large problems that have made life harder for many people


Singapore and Hong Kong are two generally-capitalist cities that have employed largely government housing development of very dense, tall housing.

It worked REALLY well in capitalist, uber-wealthy Singapore (GDP per capita substantially higher than the USA). ~78% of Singaporeans live in housing developed by the Singapore Government's Housing and Development Board ( It works a bit less well in Hong Kong, but still remarkably well considering how many people are housed in the very small area available.

Propinquity Cities So Far

But just this point is a rabbit hole of questions in itself.

If we equipped every store with tracking devices that measured the amount of time spent by people visiting (!!!!), that might incentivize making products really hard to find in the store, or making really long lines, so people spend more time there.

If it's the raw number of times people spend visiting the store, I am sure there are ways to game that too ("visit us 10 times this week for 10% off your next purchase!"). There could be laws against that, but..

Propinquity Cities So Far

Hi Mako,

First, a quick empirical quibble:

Cost-efficacy, beyond a point, is not rewarded with any increase in market share. The unfit are not selected out.

Not true, I don’t think. I have personally noticed rental rates decline in Auckland City Centre during the covid-19 period because of the abnormally low demand here, and in a Downtown LA housing boom several years ago, it was common for landlords to give away free months' rent. If the amount of supply exceeds demand by a sufficient amount, you will see market forces work. Why wouldn’t you?

I think I have two key objections, not necessarily an exhaustive list but two fairly key objections that come to mind quickly:

(1) Is there any mechanism here for actually increasing the sum total amount of high-demand property? If not, there are still going to be a large amount of people who didn't get to be located in high-demand areas. If so, is that mechanism intrinsic to the Propinquity City model or could it be implemented just as well within our current system of property ownership? For instance, I imagine this system would include rules about land use that would legally permit density where many people want to live. But there is no technical reason we couldn't implement the same rules within our current system. As one can observe in areas of cities where density is in demand and is allowed, over time, density is actually developed by the market according to demand. And that's a hell of a lot easier than developing an entirely new economic system, particularly if it also involves switching to a modular housing system.

(2) And largely as a result of my previous comment, I suspect that the increase in well-being through the implementation of this system would not be particularly large compared to more limited increases that could be obtained with more incremental land use liberalisation plans. That would limit its potential as an "effective cause area" because it would be much less tractable than other reform proposals that accomplish most of the same value as this system.

As you said, it would be very difficult to implement this in existing cities so this is limited to designing in new cities. Initially I thought this was a limitation, but then I reconsidered: in the next few decades there will be billions of people across the developing world moving into new urban areas, so there is plenty of scope for implementation if developers wanted to implement it. 

It would be worth doing an analysis. I recommend, if you proceed to further develop the idea, you do an impact estimate via spreadsheet or other software. Consider the number of people likely to be impacted, the increase in well-being that would result (using QALYs or other empirical measure; be sure to consider anxiety arising from the lack of stability that people may be asked to move at any time), and the amount of political energy that would be need to be invested to achieve the calculated impact. And calculate this relative to the next best solution (maybe a classical Georgist Land Value Tax combined with liberalisation of land use would be a good baseline).

Load More