I am a PhD student in computer science at the University of Waterloo, supervised by Professor Ming Li and advised by Professor Marcus Hutter.
My current research is related to applications of algorithmic probability to sequential decision theory (universal artificial intelligence). Recently I have been trying to start a dialogue between the computational cognitive science and UAI communities. Sometimes I build robots, professionally or otherwise. Another hobby (and a personal favorite of my posts here) is the Sherlockian abduction master list, which is a crowdsourced project seeking to make "Sherlock Holmes" style inference feasible by compiling observational cues. Give it a read and see if you can contribute!
See my personal website colewyeth.com for an overview of my interests and work.
I do ~two types of writing, academic publications and (lesswrong) posts. With the former I try to be careful enough that I can stand by ~all (strong/central) claims in 10 years, usually by presenting a combination of theorems with rigorous proofs and only more conservative intuitive speculation. With the later, I try to learn enough by writing that I have changed my mind by the time I'm finished - and though I usually include an "epistemic status" to suggest my (final) degree of confidence before posting, the ensuing discussion often changes my mind again.
It looks like Gemini is self-improving in a meaningful sense:
https://deepmind.google/discover/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/
Some quick thoughts:
This has been going on for months; on the bullish side (for ai progress, not human survival) this means some form of self-improvement is well behind the capability frontier. On the bearish side, we may not expect a further speed up on the log scale (since it’s already factored in to some calculations).
I did not expect this degree of progress so soon; I am now much less certain about the limits of LLMs and less prepared to dismiss very short timelines.
With that said… the problems that it has solved do seem to be somewhat exhaustive search flavored. For instance it apparently solved an open math problem, but this involved arranging a bunch of spheres. I’m not sure to what degree LLM insight was required beyond just throwing a massive amount of compute at trying possibilities. The self-improvements GDM reports are similar - like faster matrix multiplication in I think the 4x4 case. I do not know enough about these areas to judge whether AI is essential here or whether a vigorous proof search would work. At the very least, the system does seem to specialize in problems with highly verifiable solutions. I am convinced, but not completely convinced.
Also, for the last couple of months whenever I’ve asked why LLMs haven’t produced novel insights, I’ve often gotten the response “no one is just letting them run long enough to try.” Apparently GDM did try it (as I expected) and it seems to have worked somewhat well (as I did not expect).
After some thought, I believe I understand everything except the proof of theorem 1, which seems unclear.
The existence proof for the oracles doesn't seem to be rigorous, which is probably the first thing to take care of before further consideration, but I do expect it to go through.
Also, it would be nice to have some upper bound on the power of the class...
That’s an interesting way of putting it, though it really has more to do with Solomonoff induction than AIXI, and isn’t really related to AIXI-tl which uses proof search.
It seems that the OP is simply confused about the fact that preferences should be over the final outcomes - this response equivocates a bit too much.
I suppose the learning process could work in a more legible way - for instance, it's not clear why neural networks generalize successfully. But this seems to be more closely related to the theoretical understanding of learning algorithms than their knowledge representation.
I’ve been saying this for two years: https://www.lesswrong.com/posts/RTmFpgEvDdZMLsFev/mechanistic-interpretability-is-being-pursued-for-the-wrong
Though I have updates somewhat since then - I am slightly more enthusiastic about weak forms of the linear representation hypothesis, but LESS confident that we’ll learn any useful algorithms through mech interp (because I’ve seen how tricky it is to find algorithms we already know in simple transformers).
Can you expand on the first paragraph?
I don’t know if other dynamics are dominating, but I seriously doubt that LLMs are qualitatively changing the dynamics of voting through the mechanism you seem to be suggesting - possibly loose persuasion bots on the internet are affecting voting behavior somewhat, but I don’t think people are intentionally using chatbots to make smarter voting decisions.
Honestly, I am no longer sure I understand what you’re trying to claim at all.
I do not expect voters to actually become much smarter just because in principle they have access to intelligent advice (in some domains, which is sometimes totally wrong). In fact, I think voters have a time-honored tradition of ignoring intelligent advice, particularly when it is hard to distinguish from unintelligent advice.
So, even if this is true in theory, it will not manifest how you're suggesting in practice.
I expect this to start not happening right away.
So at least we’ll see who’s right soon.