[i'm sure people on LW would love to tell me where i can find the answer or how they think about it]
How to properly defer to someone else’s belief instead of their evidence?
If Alice knows much more physics than me, learning that Alice assigns 90% to X is obviously evidence for X.
But blindly averaging toward expert beliefs seems to double-count correlated evidence. It also conflates “probability emanating from a coherently-integrated world-model I can defend the gears-level of” with “probability if I was guessing on a prediction market.”
What are rationalist meta-conceptually über-coherent ways to think about epistemic deference?
How to think about the relative information efficiency of RL vs SFT?
People keep telling me “RL is very informationally inefficient.” It only receives a single bit of information (success/fail) over an entire rollout (tens of thousands of tokens!). Meanwhile, SFT gets a bit or more of information on every token.
But RL can make my warm-started model learn to do what I want in like a dozen steps while I can't get SFT to do like anything useful in a dozen steps.
And RL and SFT on LLMs seem to reduce to the same cross-entropy loss anyway, just with a different coefficient. What is going on??
You know... like StackOverflow....
But I notice Q&A doesn’t seem to be the vibe here, nor quite fit with this particular web of knowledge.
So what can one do instead?
Maybe instead of ...
forum-style “Hallo how do you ask questions?” post (tonally inappropriate, creates spam, doesn't mesh with the web of knowledge)
... one could make a ...
argument-style “LessWrong should support questions” post (and count on people to quickly tell you you’re wrong and LessWrong already supports this via [X], or no, LessWrong shouldn't have questions, but you can do X)
how-to-style “So you want to ask a question on LessWrong...” post (include a thought or two, and let the comments fill in the rest — like writing a continuation-style base LLM prompt)
observation-style “I notice I don't know where to put questions on LW” post (adding an observation to the pile; inviting discussion; maybe noticing a subtle confusion others didn’t realise they were confused about too)
Sometimes I want to post things like:
[here’s my initial stab at it]
[but i have no idea if this is right or wrong]
[i'm sure people on LW would love to tell me where i can find the answer or how they think about it]
If Alice knows much more physics than me, learning that Alice assigns 90% to X is obviously evidence for X.
But blindly averaging toward expert beliefs seems to double-count correlated evidence. It also conflates “probability emanating from a coherently-integrated world-model I can defend the gears-level of” with “probability if I was guessing on a prediction market.”
What are
rationalistmeta-conceptually über-coherent ways to think about epistemic deference?People keep telling me “RL is very informationally inefficient.” It only receives a single bit of information (success/fail) over an entire rollout (tens of thousands of tokens!). Meanwhile, SFT gets a bit or more of information on every token.
But RL can make my warm-started model learn to do what I want in like a dozen steps while I can't get SFT to do like anything useful in a dozen steps.
And RL and SFT on LLMs seem to reduce to the same cross-entropy loss anyway, just with a different coefficient. What is going on??
You know... like StackOverflow....
But I notice Q&A doesn’t seem to be the vibe here, nor quite fit with this particular web of knowledge.
So what can one do instead?
Maybe instead of ...
... one could make a ...