Happy to have a public conversation on the topic. Just DM me on Twitter if you are interested.
By this logic, any instrumental action taken towards an altruistic goal would be "for personal gain".
I think you are making a genuine mistake, and that I could have been clearer.
There are instrumental actions that favour everyone (raising epistemic standards), and instrumental actions that favour you (making money).
The latter are for personal gains, regardless of your end goals.
Sorry for not getting deeper into it in this comment. This is quite a vast topic.
I might instead write a longer post about the interactions of deontology & consequentialism, and egoism & altruism.
(I strongly upvoted the comment to signal boost it, and possibly let people who agree easily express their agreement to it directly if they don't have any specific meta-level observation to share)
Suppose I take out a coin and flip it 100 times in front of your eyes, and it lands heads every time. Will you have no ability to predict how it lands the next 30 times? Will you need some special domain knowledge of coin aerodynamics to predict this?
Then, yes. Because you had many other coins that had started flipping tail at some point, and there is no easily discernable pattern.
By your interpretation, the Solomonoff induced prior for that coin is basically "it will never flip tail". Whereas, you do expect that most problems that have not been solved now will be solved at some point, which does mean that you are incorporating more knowledge.
"Experts have a precise model of the known subset of the concept-space of their domain, and they can make vague high-level extrapolations on how that domain looks outside the known subset, and where in the wider domain various unsolved problems are located, and how distant they are from the known domain"
Experts from many different fields of Maths and CS have tried to tackle the Collatz' Conjecture and the P vs NP problem. Most of them agree that those problems are way beyond what they set out to prove. I mostly agree with you on the fact that each expert's intuition vaguely tracks one specific dimension of the problem.
But any simplicity prior tells you that it is more likely for there to be a general reason for why those problems are hard along all those dimensions, rather than a whole bunch of ad-hoc reasons.
The way I see it, that's it. This statement isn't reducible to something more neat and simple.
For any given difficult problem, you can walk up to an expert and ask them why it's considered hard, but the answers they give you won't have any unifying theme aside from that. It's all ad hoc.
What makes you think that? I see you repeating this, but I don't see why that would be the case.
Why would you think there's something else?
Good question, thanks! I tried to hint at this in the Original Post, but I think I should have been more explicit. I will make a second edit that incorporates the following.
The first reason is that many different approaches have been tried. In the case where only a couple of specific approaches have been tried, I expect the reason for why it hasn't been solved to be ad-hoc and related to the specific approaches that have been tried. The more approaches are tried, the more I expect a general reason that applies to all those approaches.
The second reason is that the problems are simple. In the case of a complicated problem, I would expect the reason for why it hasn't been solved to be ad-hoc. I have much less of this expectation for simple problems.
Isn't it about empirical evidence that these problems are hard, not "predictions"? They're considered hard because many people have tried to solve them for a long time and failed.
No, this is Preemption 1 in the Original Post.
"hard" doesn't mean "people have tried and failed", and you can only witness the latter after the fact. If you prefer, even if have empirical evidence for the problem being "level n hard" (people have tried up to level n), you;d still do not have empirical evidence for the problem being "level n+1 hard" (you'd need people to try more to state that if there's nothing you can say about it ahead of time). Ie, no predictive power.
An expert can estimate how hard a problem is by eyeballing how distant the abstractions needed to solve it feel from the known ones
Great! We're getting closer to what I care about.
Then what I am saying is that there is a heuristic that the experts are using to eyeball this, and I want to know what that is, start ingwith those 2 conjectures!
I am also saying that the more distant "the abstraction need to solve it feel from the known ones", the easier it should be to do so.
They're able to do this because they've developed strong intuitions for their professional domain: they roughly know what's possible, what's on the edge of the possible, and what's very much not.
Exactly, but those intuitions are implemented somehow. How?
Also, the more experts agree on a judgment, and the stronger their judgment, the easier you expect to be to explain that intuition.
But there's no objective property that makes these problems intrinsically hard, only subjectively hard from the point of view of our conceptual toolbox.
I was confused very confused when I read this. For instance, the part in bold is already reflected in the Original Post.
There are many theoretical problems that are considered to be obviously far far outside our problem solving ability.
If you prefer, interpret "intrinsically hard" as "having an intrinsic property that makes it subjectively hard for us. To model how that would look, consider the following setup:
The space of problems is just the real plane, and our ability to solve problems is modeled by a unit disk in the plane (if in the disk, solved, and the closer it is to the disk, the easier it is to solve). Then, the difficulty of a problem is subjective, it depends on where the disk is.
But let's say the disk is somewhere on the x axis, then the intrinsic property of a problem being far having a high y coordinate, makes it subjectively hard.
I'll make some edits to the post. I thought most of this was clear because of Preemption #1, but it was not.
Are these perhaps boring, because the difficulty is well understood?
They are not boring, I am simply asking about some specific cluster of problems, and none of them belong to that cluster.
I think part of the explanation is that we don't have a model for distance from success. We have no clue if the researchers who've made serious attempts on these problems got us closer to an answer/proof, or if they just spun their wheels.
This post is about experts in the fields of number theory and complexity theory claiming to have a clue about this.
If you think "We have no clue", you likely think they are wrong, and I would be interested in knowing why.
I added more details on this comment, given that someone else already shared a similar thought
The post is about predictions made by experts in number theory and complexity theory.
If you think that this can not be predicted, and that they are thus wrong about their predictions, I would be interested in knowing why.
Namely:
It is plausible that the actual Collatz system is one of these for our standard proof systems.
Why? Consider the following:
The Collatz Conjecture has a lot of structure to it:
- It is false for 3n-1
- It is undecidable for generalizations of it
- It is true empirically, up to 10^20
- It is true statistically, a result of Terrence Tao establishes that "almost all initial values of n eventually iterate to less than log(log(log(log(n))))" (or inverse Ackermann)
Additionally, if you look at the undecidable generalizations of the Collatz Conjecture, I expect that you will find them much stronger than the base system. And when people prove such results, they look for the weakest system such that you still have undecidability.
As a result of those considerations, I find it quite implausible that the Collatz Conjecture is undecidable, and I would be interested in what makes you think otherwise.
P vs NP is another candidate for an undecidable problem, dealing as it does with general behaviour of Turing machines that can run programs with rather weak bounds. There's a lot that we can't yet prove about about general computation systems, and we have theorems that say we should expect there to be a lot that we can't ever prove.
My answer is similar here to Scott Aaronson's:
More to the point, if you believe P vs. NP is undecidable, then you need to answer the question: why does whatever intuition tells you that, not also tell you that the P versus EXP, NL versus PSPACE, MAEXP versus P/poly, TC0 versus AC0, and NEXP versus ACC questions are similarly undecidable? (In case you don’t know, those are five pairs of complexity classes that have been proved different from each other, sometimes using very sophisticated ideas.)
It would be unsurprising if quite a lot of the problems we can't solve after trying very hard are actually not solvable.
I am not sure if you mean "unsurprising" in the literal or metaphorical way.
If you mean that it would literally be merely unsurprising, I agree, this is in the realm of possibilities. But I would not call this a "good reason to expect it to be difficult to solve".
If you mean it as very likely, as to make it a good reason, I disagree. I expect that if we went through Wikipedia's list of mathematical Conjectures solved since 1995 (which I believe is a good approximation of "Problems that were really hard, and we get to find out how they turned out"), we'd find that most of them were actually resolved, rather than found as undecidable.
4% of alignment spending on this seems like clearly way too much.
-
The main hoped-for benefits are "teach the scientific community new things" and "plausibly going viral repeatedly".
For the first one, it seems like one more exploration among many others, on par with @janus' for instance.
For the second one, as you put it, "more hype" is not what is missing in AI. People see Midjourney getting better over the years, people see ChatGPT getting better over the years, and companies are optimising quite hard for flashy demonstrations.
-
I guess I just dislike this type of "theory of change that relies on too many unwarranted assumptions to be meaningful, but somehow still manages to push capabilities and AI hype, and makes grabs for attention and money" in general.
This quote from the "Late 2027" ideal success illustrates what I mean by "too many unwarranted assumptions":
If that's where the value is in the best case, I would just put the money into "grassroots advocacy for an AI development pause or slowdown" directly.
If the pitch is "let the AIs do the grassroots advocacy by themselves because it's more efficient", then I would suggest instead doing it directly and thinking of where AIs could help in an aligned way.
If the pitch is "let's invest money into AI hype, and then leverage it for grassroots advocacy for an AI development pause or slowdown", I would not recommend it because the default optimisation target of hype depletes many commons. Were I to recommend it, I would recommend doing it more directly, and likely checking with people who have run successful hype campaigns instead.
-
I would recommend asking people doing grassroots advocacy how much they think a fun agency demo would help, and more seriously how much they'd be willing to pay (either in $$ or in time).
ControlAI (where I advise), but also PauseAI or possibly even MIRI now with their book tour and more public appearances.
-
There's another thing that I dislike, but is harder to articulate. Two quotes that make it more salient:
This looks to me how we die by a thousand cuts, a lack of focus, and a lack of coordination.
Fmpov, there should be a high threshold for the alignment community to seriously consider a project that is not just fully working on core problems. Like alignment (as opposed to evals, AGI but safe, AGI but for safety), on extinction risks awareness (like the CAIS statement, AI2027) or on pause advocacy (as opposed to job loss, meta-crisis, etc.).
We should certainly have a couple of meta-projects, that is close to the ops budget of NGO. Like 10-15% of the budget on coordination tools (LW, regranters, etc.). But by bulk, we should do the obvious thing.