(I've been "out of the game" for a couple years.) I would append to
you might think AI alignment is a computer programming problem but it's more of a math problem
a follow-on:
you might think AI alignment is a math problem but it's more of a philosophical / conceptual / mental-phenomenology problem
(I hesitate to use the word "phenomenology" because it will definitely be misunderstood by ~everyone, but there just isn't a better word; an internal phrase that was floated was "core theory of mind", and I mean to gesture at "mathematico-introspective investigation of core theory of mind".) This is elaborated on (cryptically / elliptically, but you could read slowly & think) here: https://www.lesswrong.com/posts/TNQKFoWhAkLCB4Kt7/a-hermeneutic-net-for-agency
In this context, the long and short of it is, I don't think you can feasibly point people at the parts of the problem that matter. I'm just such a special genius that only I could possibly understand the real alignment problem For some reason people don't seem to be inclined to plant difficult questions and let them grow over time, sit with unresolved questions, look at big things and keep their bigness firmly in mind while also somehow making cumulative progress, overhaul key elements or at least keep them firmly provisional, keep staring outside of the streelight's glow for years, and so forth. (Something something Grothendieck something something Peter Scholze something something staring at conceptual foundations.) Or maybe it's the thing about security mindset (Yudkowsky), or overconfidence (Dai), etc. Or maybe we're just not high g enough. Or all of the above or something else, IDK. But again, it just doesn't seem to work to point people at the problem. Or maybe I'm deluded, who could know.
If you take all the alignment research, insofar as I'm aware of it, in the past 2 decades, and multiply that by 3, it still doesn't even come close to solving alignment. This is very rough and grim, but it's what I think.
I'm not sure I buy that most mathematicians will be out of a job. Maybe some of the number theorists, analysts, and combinatorialists? Or something, IDK, I'm making that up. (Based on what fields vaguely seem to have some niches for people doing mostly high-algebraicness work.) Or maybe they will be.
What should they do instead? IDK. Human intelligence amplification is still insanely underresearched, and can definitely use more brainpower. I'd be very happy to be connected with smart motivated scientists / mathematicians who may want to work on empowering humans!! My gmail for that would be: tsvibtcontact
6
1
I think there's a critical opportunity for someone here.
Mathematicians are feeling the doom (mostly in the "lose our jobs" sense).
Academics are freaking out about the daily news that amateurs are asking GPT "prove career-defining theorem, make no mistakes" and it's just working.
Senior researchers are leaving for frontier AI labs. Many are mentally spiralling or flailing about their life's work not mattering anymore.
Last year, when I tried explaining IABIED to colleagues, I would be met with incredulous stares. This year, I'm met with incredulous stares and "So what should I do now?" (I don't have a good answer for them, which is part of why I'm posting this.)
I've quarantined AI discussions on my research discord because otherwise it would overwhelm everything else.
Top mathematicians, people on par in mathematical ability with Critch, Christiano, and Steinhardt, people who have been running leading-edge research groups for decades, people with enormous soft power in academic circles, are going to OpenAI without even having considered x-risk for five minutes.
Many more will be leaving soon.
If you want these folks to hear something at all, to consider some other option in the rest of their life, now is the time.