Research Lead at CORAL. Director of AI research at ALTER. PhD student in Shay Moran's group in the Technion (my PhD research and my CORAL/ALTER research are one and the same). See also Google Scholar and LinkedIn.
E-mail: {first name}@alter.org.il
I mostly agree with this, the part which feels off is
I’d like to say here “screw memetic egregores, follow the actual values of actual humans”
Humans already follow their actual Values[1], and will always do because their Values are the reason they do anything at all. They also construct narratives about themselves that involve Goodness, and sometimes deny the distinction between Goodness and Values altogether. This act of (self-)deception is in itself motivated by the Values, at least instrumentally.
I do have a version of the “screw memetic egregores” attitude, which is, stop self-deceiving. Because, deception distorts epistemics, and we cannot afford distorted epistemics right now. It's not necessarily correct advice for everyone, but I believe it's correct advice for everyone who is seriously trying to save the world, at least.
Another nuance is that, in addition to empathy and naive tit-for-tat, there is also acausal tit-for-tat. This further pushes the Value-recommended strategy in the direction of something Goodness-like (in certain respects), even though ofc it doesn't coincide with the Goodness of any particular culture in any particular historical period.
As Steven Byrnes wrote, "values" might be not the best term, but I will keep it here.
1No, it's not at all the same thing as OpenAI is doing.
First, OpenAI is working using a methodology that's completely inadequate for solving the alignment problem. I'm talking about racing to actually solve the alignment problem, not racing to any sort of superintelligence that our wishful thinking says might be okay.
Second, when I say "racing" I mean "trying to get there as fast as possible", not "trying to get there before other people". My race is cooperative, their race is adversarial.
Third, I actually signed the FLI statement on superintelligence. OpenAI hasn't.
Obviously any parallel efforts might end up competing for resources. There are real trade-offs between investing more in governance vs. investing more in technical research. We still need to invest in both, because of diminishing marginal returns. Moreover, consider this: even the approximately-best-case scenario of governance only buys us time, it doesn't shut down AI forever. The ultimate solution has to come from technical research.
I'm using the term "meta-ethics" in the standard sense of analytic philosophy. Not sure what bothers you so greatly about it.
I find your manner of argumentation quite biased: you preemptively defend yourself by radical skepticism against any claim you might oppose, but when it comes to a claim you support (in this case "ethical realism is false"), suddenly this claim is "pretty close to analytic". The latter maneuver seems to me the same thing as the "Obviously Right" you criticize later.
Also, this brand of radical skepticism is an example of the Charybdis I was warning against. Of course you can always deny that anything matters. You can also deny Occam's razor or the evidence of your own eyes or even that 2+2=4. After all, "there's no predefined standard for standards". (I guess you might object that your reasoning only applies to value-related claims, not to anything strictly value-neutral: but why not?)
Under the premises of radical skepticism, why are we having this debate? Why did you decide to reply to my comment? If anyone can deny anything, why would any of us accept the other's arguments?
To have any sort of productive conversation, we need to be at least open to the possibility that some new idea, if you delve deeply and honestly into understanding it, might become persuasive by the force of the intuitions it engenders and its inner logical coherence combined. To deny the possibility preemptively is to close the path to any progress.
As to your "(b) there's a bunch of empirical evidence against it" I honestly don't know what you're talking about there.
P.S.
I wish to also clarify my positions on a slightly lower level of meta.
First, "ethics" is a confusing term because, on my view, the colloquial meaning of "ethics" is inescapably intertwined with how human societies negotiate of over norms. On the other hand, I want to talk purely about individual preferences, since I view it as more fundamental.
We can still distinguish between "theories of human preferences" and "metatheories of preferences", similarly to the distinction between "ethics" and "meta-ethics". Namely, "theories of human preferences" would have to describe the actual human preferences, whereas "metatheories of preferences" would only have to describe what does it even mean to talk about someone's preferences at all (whether this someone is human or not: among other things, such a metatheory would have to establish what kind of entities have preferences in a meaningful sense).
The relevant difference between the theory and the metatheory is that Occam's razor is only fully applicable to the latter. In general, we should expect simple answers to simple questions. "What are human preferences?" is not a simple question, because it references the complex object "human". On the other hand "what does it mean to talk about preferences?" does seem to me to be a simple question. As an analogy, "what is the shape of Africa?" is not a simple question because it references the specific continent of Africa on the specific planet Earth, whereas "what are the general laws of continent formation" is at least a simpler question (perhaps not quite as simple, since the notion of "continent" is not so fundamental).
Therefore, I expect there to be a (relatively) simple metatheory of preferences, but I do not expect there to be anything like a simple theory of human preferences. This is why this distinction is quite important.
Your failure to distinguish ethics from meta-ethics is the source of your confusion (or at least one major source). When you say "ethical realism is false", you're making a meta-ethical statement. You believe this statement is true, hence you perforce must believe in meta-ethical realism.
Strong disagree.
We absolutely do need to "race to build a Friendly AI before someone builds an unFriendly AI". Yes, we should also try to ban Unfriendly AI, but there is no contradiction between the two. Plans are allowed (and even encouraged) to involve multiple parallel efforts and disjunctive paths to success.
It's not that academic philosophers are exceptionally bad at their jobs. It's that academic philosophy historically did not have the right tools to solve the problems. Theoretical computer science, and AI theory in particular, is a revolutionary method to reframe philosophical problems in a way that finally makes them tractable.
About "metaethics" vs "decision theory", that strikes me as a wrong way of decomposing the problem. We need to create a theory of agents. Such a theory naturally speaks both about values and decision making, and it's not really possible to cleanly separate the two. It's not very meaningful to talk about "values" without looking at what function the values do inside the mind of an agent. It's not very meaningful to talk about "decisions" without looking at the purpose of decisions. It's also not very meaningful to talk about either without also looking at concepts such as beliefs and learning.
As to "gung-ho attitude", we need to be careful both of the Scylla and the Charybdis. The Scylla is not treating the problems with the respect they deserve, for example not noticing when a thought experiment (e.g. Newcomb's problem or Christiano's malign prior) is genuinely puzzling and accepting any excuse to ignore it. The Charybdis is perpetual hyperskepticism / analysis-paralysis, never making any real progress because any useful idea, at the point of its conception, is always half-baked and half-intuitive and doesn't immediately come with unassailable foundations and justifications from every possible angle. To succeed, we need to chart a path between the two.
3
2
1Thanks for the heads up. Can you share which AI models were involved?
Maybe we want a multi-level categorization scheme instead? Something like:
Level 0: Author completely abstains from LLM use in all contexts (not just this post)
Level 1: Author uses LLMs but this particular post was made with no use of LLM whatsoever
Level 2: LLM was used (e.g. to look up information), but no text/images in the post came out of LLM
Level 3: LLM was used for light editing and/or image generation
Level 4: LLM was used for writing substantial parts
Level 5: Mostly LLM-generated with high-level human guidance/control/oversight
10 years ago I argued that approval-based AI might lead to the creation of a memetic supervirus. Relevant quote:
Optimizing human approval is prone to marketing worlds. It seems less dangerous than physicalist AI in the sense that it doesn't create incentives to take over the world, but it might produce some kind of a hyper-efficient memetic virus.
I don't think that what we see here is literally that, but the scenario does seem a tad less far-fetched now.
This frame seems useful, but might obscure some nuance: