Oliver Sourbut

Oliver - or call me Oly: I don't mind which!

I'm particularly interested in sustainable collaboration and the long-term future of value. Currently based in London, I'm in my early-ish career working as a senior software engineer/data scientist, and doing occasional AI alignment work with SERI.

I'd love to contribute to a safer and more prosperous future with AI! Always interested in discussions about axiology, x-risks, s-risks.

I enjoy meeting new perspectives and growing my understanding of the world and the people in it. I also love to read - let me know your suggestions! Recently I've enjoyed

  • Ord - The Precipice
  • Pearl - The Book of Why
  • Bostrom - Superintelligence
  • McCall Smith - The No. 1 Ladies' Detective Agency
  • Abelson & Sussman - Structure and Interpretation of Computer Programs
  • Stross - Accelerando

Cooperative gaming is a relatively recent but fruitful interest for me. Here are some of my favourites

  • Hanabi (can't recommend enough; try it out!)
  • Pandemic (ironic at time of writing...)
  • Dungeons and Dragons (I DM a bit and it keeps me on my creative toes)
  • Overcooked (my partner and I enjoy the foody themes and frantic realtime coordination playing this)

People who've got to know me only recently are sometimes surprised to learn that I'm a pretty handy trumpeter and hornist.


Breaking Down Goal-Directed Behaviour

Wiki Contributions


Feature request: voting buttons at the bottom?

Thanks for this response. FWIW I don't say

beside the obvious developer opportunity cost


I didn't know that about GreaterWrong, thanks.

Feature request: voting buttons at the bottom?

To be clear, my more controversial speculation does not go up to scanning the user's brain to confirm their ingestion, digestion, and careful consideration of the post's content[1] before allowing them to cast votes.

It does seem important that people be able to sometimes express 'I would like more people to see this (kind of thing) after my superficial read but I don't have time to digest in detail now', but it might be a good thing if this were necessarily at least a tiny bit deliberate.

  1. (if only because I don't know how to implement this and if I did, I find it hard to imagine a world where the most pressing use of this technology was optimising LessWrong karma UI - but well, maybe) ↩︎

Feature request: voting buttons at the bottom?

EDIT: this response is to the second bit about the object level of this OP, not the first bit regarding agree/reinforce buttons on top level posts, with which I straightforwardly agree.

You only want one set of buttons (sounds reasonable to me). And you want the voting buttons to have a tally next to them (I get it, but I'm not sure it's needed). You re-emphasise the importance of tallies for guiding what to invest reading time into, which means a tally needs to be at the top (I heavily agree in absence of some other guide, perhaps a great recommendation system).

Is it important to render only one tally? Or could a tally appear without buttons at the top/homepage/sequence-page and a duplicate tally appear at the bottom alongside buttons?

"Science Cathedrals"

Some UK examples

London's Natural History Museum is similar and often described as 'a cathedral'. It focuses on natural history (Earth sciences and biology).

Down the road from there is the Science Museum which has SF Exploratorium vibes.

There's a museum in Bristol formerly known as '@ Bristol' and now known as 'We the Curious' which I've not attended in many years but I recall being science-cathedrally.

Another oddly-apt blast from the past is the York Railway Museum which feels like the equivalent but from an earlier era, perhaps.

why assume AGIs will optimize for fixed goals?

I think this is an interesting perspective, and I encourage more investigation.

Briefly responding, I have one caveat: curse of dimensionality. If values are a high dimensional space (they are: they're functions) then 'off by a bit' could easily mean 'essentially zero measure overlap'. This is not the case in the illustration (which is 1-D).

You Only Get One Shot: an Intuition Pump for Embedded Agency

Interesting. I realise now 'one shot' is an overloaded term and perhaps a poor choice. I'm referring to 'one action', 'one chance', rather than 'one training/prompt example' which is how 'one shot' often gets used in ML.

The typical chess AI (or other boardgame-playing RL algorithm) is episode-myopic. Or at least, its training regime is only explicitly incentivising returns over a single episode (e.g. policy gradient or value-based training pressures) - and I don't think we have artefacts yet which reify their goals in a way where it's possible to misgeneralise to non-myopia. It's certainly not action-myopic though (this is the whole point of training to maximise return - aggregate reward - vs single-step reward).

I'm not sure what it would mean entirely for an actor-moment to be myopic, but I imagine it would at minimum have to be 'indifferent' somehow to the presence or absence of relevantly-similar actor-moments in the future.

Information Loss --> Basin flatness

Another aesthetic similarity which my brain noted is between your concept of 'information loss' on inputs for layers-which-discriminate and layers-which-don't and the concept of sufficient statistics.

A sufficient statistic is one for which the posterior is independent of the data , given the statistic

which has the same flavour as

In the respective cases, and are 'sufficient' and induce an equivalence class between s

Information Loss --> Basin flatness

Regarding your empirical findings which may run counter to the question

  1. Is manifold dimensionality actually a good predictor of which solution will be found?

I wonder if there's a connection to asymptotic equipartitioning - it may be that the 'modal' (most 'voluminous' few) solution basins are indeed higher-rank, but that they are in practice so comparatively few as to contribute negligible overall volume?

This is a fuzzy tentative connection made mostly on the basis of aesthetics rather than a deep technical connection I'm aware of.

Information Loss --> Basin flatness

Interesting stuff! I'm still getting my head around it, but I think implicit in a lot of this is that loss is some quadratic function of 'behaviour' - is that right? If so, it could be worth spelling that out. Though maybe in a small neighbourhood of a local minimum this is approximately true anyway?

This also brings to mind the question of what happens when we're in a region with no local minimum (e.g. saddle points all the way down, or asymptoting to a lower loss, etc.)

Load More