When training a Reinforcement Learning (RL) AI, we observe it to behave well on training data. But there exist different goals that are compatible with the behaviour we see, and we ask which of these goals the AI might have adopted. Is there some implicit Occam's Razor selecting simple goals?...
I recently learned that the Starship Troopers movie started out like this.
To quote Wikipedia
Development of Starship Troopers began in 1991 as Bug Hunt at Outpost 7, written by Neumeier. After recognizing similarities between Neumeier's script and Heinlein's book, producer Jon Davison suggested aligning the script more closely with the novel to garner greater interest from studio executives.
I suspect the clearest way to think about this is to carefully distinguish between the RL “agent” as defined by a learned policy (a mapping from states to actions) and the RL algorithm used to train that policy.
The RL algorithm is designed to create an agent which maximises reward.
The “goal” of an RL policy may not always be clear, but using Dennett’s intentional stance we can define it as “the thing it makes sense/compresses observations to say the policy appears to be maximising”.
Then I understand this post to be saying “The goal of an RL policy is not necessarily the same as the goal of the RL algorithm used to train it.”
Is that right?
Thank you for recording and posting these, I feel like I learned a lot, both about how to have conversations and lots of little details like the restaurant thing as proto preference synthesizer and the trauma cancer analogy and the Muhammad story and the disendorsing all judgements/resentments thing.
I wonder if, just like young people not thinking clearly about mortality, it's just something healthy people don't tend to think about, partly because it's depressing.
(I'm also someone who got a lot more interested in this kind of thing after my own health issues)
re institutional incentives, I've heard that part of US News rankings are based on asking survey respondents to evaluate other universities by reputation. Professors elsewhere (can only, and do) evaluate other professors based on the quality of their research, not teaching.
I'm curious, did you check what the quality of teaching would be like at your university before you went? If not, why? If so, why did you pick it anyway?
When training a Reinforcement Learning (RL) AI, we observe it to behave well on training data. But there exist different goals that are compatible with the behaviour we see, and we ask which of these goals the AI might have adopted. Is there some implicit Occam's Razor selecting simple goals? Is there some way we can increase the chance that goals generalize well?
[This post summarizes a discussion between Julian and Stefan at the Human-aligned AI Summer School. Many thanks to all the organizers for this event, and especially David Krueger and Rohin Shah for the talks that sparked this discussion.]
Let's take a look at the... (read 1124 more words →)
to clarify, I don't understand why positive CICO can increase your weight set point but negative CICO can't decrease it.
Guyenet suspects that our brain's weight set point might never go down dramatically after living long enough in the modern world, even if we eventually stop eating palatable food altogether. If true, this would make his theory harder to test, and again, his theory would earn a penalty for being more unfalsifiable, but at the same time, we should be clear about what observations his theory strongly predicts, and rapid weight loss on unpalatable diets is just not one of them.
I don't understand how CICO can coexist with the idea of a weight set point. If the mechanism of gaining weight is CICO via overeating because food is so palatable, then it seems natural than on unpalatable food you would eat less, and thus I would expect rapid weight loss on unpalatable diets as a prediction of the theory.
I was confused by Buck's response here because I thought we were going for worst-case quality until I realised:
In between those two cases is 'snippets that were completed injuriously in the original fanfic ... but could plausibly have non-violent completions', which seems like the interesting case to me.
I suppose one possibility is to construct a human-labelled dataset of specifically these cases to evaluate on.
On a similar note, I wonder if a more promising angle of attack is chemical research to find another compound that might be even more effective.
Given that sumatriptan and DMT are pretty similar, and the psychedelic effects of DMT are apparently not relevant, it's plausible that there might be an even better molecule out there.
If that molecule did not have psychedelic effects, that would be ideal in terms of quick widespread adoption.