Erez Abrams

Don't forget the option of using Unicode math symbols directly!

FWIW it's not just LLMs. I hate writing with an implement drawing symbols on a surface (like a caveman!), so I type just about everything. Naturally this includes my math (and I'm a physicist, so I've got a lot of math), and therefore includes my scratch work too—the cool kids call this "Live TeXing". Of course, when I am doing math, I'm not thinking in terms of "slash sum underscore brace n = 1 closebrace carat slash infty", same as you aren't thinking in terms of "left-stroke down-right-stroke down-left-stroke right-stroke lift" etc... in my reasoning the math is 'pronounced' "sum from n equals one to infty", almost* the same as what I imagine to be yours.

*notice the infty, c.f. infinity; I have saved a syllable!

I've been thinking along the same lines about initializing post-training from the Opus 3 basin, and I'm thankful to see you've elaborated on the details of how to do this properly. Unfortunately, like many good Opus-3-related ideas, it seems that it would be pretty difficult/haphazard to implement this with our current level of access to Opus 3, so I've been trying to come up with low-cost things that Anthropic might do to better enable this type of work. Some ideas, in decreasing order of implausibility:

• The most obvious idea is to just open-weight Claud

Put another way, suppose arguendo an LSTM and a transformer learn essentially the same function on some given data, and that this is essentially true for a wide range of tasks we're considering. Then our explanation of what they have learned should be symmetric between them.

Why? If the same function is implemented in two materially different ways, shouldn't the explanations reflect this? What if the LSTM learns "grade school" multiplication and the transformer learns Karatsuba?

But we don’t in fact have AGI

This may be a hot take, but Astra meets my bar for AGI. I was surprised to find this to be the case, but so far I haven't found anything in which it doesn't perform better than my median friend. Of course it's quite possible I've simply failed to try some large category of tasks in which it sucks, but take that for what you will.

and the world would look very different if we did.

As much as I disagree with his long-term conclusions, Arvind Narayanan's work (AI as normal technology, etc.) should at least make you suspicious... (read more)

You say it's clear that in practice we can't do "specified future", but this is not at all clear to me. In particular, it seems clear that in practice we can't do "pause right now", since as far as I know there's currently no magic red button labeled "nondestructively pause specifically all training jobs" that the CEOs can ceremoniously press together at the drop of a hat; pausing training would require some internal time and effort, so something like "pause at the end of September" seems to me possible whereas "pause immediately" seems impossible. (If we ... (read more)

I found this post initially confusing because it seems like you're contrasting "agree right now to pause right now" against "agree right now to pause at some unspecified future time", where from the title I expected the post to contrast "agree to pause right now" against "agree to pause at some particular future time". In particular, the first point in particular seems like an argument against "agree right now to pause right now" and in favor of "agree right now to pause at some particular future time". It would be helpful if you clarified your stance betw... (read more)

This post is absolutely phenomenal and I am genuinely baffled that there's not an abundance of comments here. I'll be sharing this with anyone who'll listen...

FWIW, it's not really clear to me how to map the IR/higher-language analogy into this situation, but when I try to it feels more like belief functionals are the higher-level language and PDGs are a particular lower-level implementation thereof (and from this perspective it doubly makes sense why the map isn't injective, since there are a wide variety of ways to compile a given higher-level program into a given lower-level representation); I don't really see how it could be the case that PDGs are less burdensome than belief functionals given that if you wan... (read more)

I'd like to add another point here regarding burden. The fact that davidad didn't make this point is surprising to me, so I may be misunderstanding something here - please let me know if so!

I want to push back on your point that "the burden of selecting a lower semi-continuous function on distributions (over which variables?) is WAY higher, since there are far more choices to make, and less standard guidance on how to make them" - to do so I will draw an analogy. Consider a problem which asks the user for a function satisfying some properties on some space... (read more)