I think that the problem is in the way you define the prior. Here is an alternative proposal:

Given a lambda-term $t$, we can interpret it as defining a partial function $f_t:\{0,1{\}}^{\ast }\times \{0,1\}\times N\rightharpoonup Q\cap [0,1]$. This function works by applying $t$ to the (appropriately encoded) inputs, beta-reducing, and then interpreting the result as an element of $Q\cap [0,1]$ using some reasonable encoding. It's a partial function because the reduction can fail to terminate or the output can violate the expected format.

Given $f:\{0,1{\}}^{\ast }\times \{0,1\}\times N\rightharpoonup Q\cap [0,1]$, we define the "corrected" function $\hat{f}:\{0,1{\}}^{\ast }\times \{0,1\}\times N\rightharpoonup Q\cap [0,1]$ as follows. (The goal here is to make it monotonic in the last argument, and also ensure that probabilities sum to $\le 1$.) First, we write $f_{max}(u,b,k)=x$ whenever (i) for all $i\le k$, $(u,b,i)\in \text{dom}\left(f\right)$ and (ii) $x={max}_{i\le k}f(u,b,i)$. If there is no such $x$ (i.e. when condition i fails) then $f_{max}(u,b,k)$ is undefined. Now, we have two cases:

• When $\forall i\le k:f_{max}(u,0,i)+f_{max}(u,1,i)\le 1$ (in particular, the terms on the LHS are defined), we define $\hat{f}(u,b,k):=f_{max}(u,b,k)$.
• In other cases, we define $\hat{f}(u,b,k):=f_{max}(u,b,j)$ where $j$ is maximal s.t. $(u,b,j)$ is in the former case. If there is no such $j$, we set $\hat{f}(u,b,k):=0$.

We can now define the semimeasure ${\mu }_f$ by

For $f=f_t$, this semimeasure is lower-semicomputable. Conversely, any lower-semicomputable semimeasure is of this form. Mixing these semimeasures according to our prior over lambda terms gives the desired Solomonoff-like prior.

I agree, except that I don't think it's especially misleading. If I live on the 10th floor and someone is dangling a tasty cake two meters outside of my window (and suppose for the sake of the argument that it's offered free of charge), I won't just walk out of the window and fall to my death. This doesn't mean I'm not following my values, it just means I'm actually thinking through the consequences rather than reacting impulsively to every value-laden thing.

...The prototypical example of a prior based on Turing machines is Solomonoff's prior. Someone not familiar with the distinction between Turing-complete and Turing-universal might naively think that a prior based on lambda calculus would be equally powerful. It is not so. Solomonoff's prior guarantees a constant Bayes loss compared to the best computable prior for the job. In contrast, a prior based on lambda calculus can guarantee only a multiplicative loss.

Can you please make this precise?

When I think of "a prior based on lambda calculus", I imagine something like the following. First, we choose some reasonable complexity measure $C$ on lambda terms, such as:

• For a variable $x$, we define $C\left(x\right):=1$
• For two terms $t,s$, we define $C\left(ts\right):=C\left(t\right)+C\left(s\right)$
• For a term $t$ and a variable $x$, we define $C(\lambda x.t):=C\left(t\right)+1$

Denote the set of lambda-terms by $\Lambda$. We then choose $\beta >0$ s.t. ${\sum }_{t\in \Lambda }{e}^{-\beta C\left(t\right)}\le 1$. 

Now, we choose some reasonable way to describe lower-semicomputable semimeasures using lambda terms, and make the prior probabilities of different lambda terms proporitional to $e^{-\beta C\left(t\right)}$. It seems to me that the resulting semimeasure dominates every lower-semicomputable semimeasure and is arguably "as good as" the Solomonoff prior. What am I missing?

Contemporary AI is smart in some ways and dumb in other ways. It's a useful tool that you should integrate into your workflow if you don't want to miss out on productivity. However. I'm worried that exposure to AI is dangerous in similar ways to how exposure to social media is dangerous, only more. You're interacting with something designed to hijack your attention and addict you. Only this time the "something" has its own intelligence that is working towards this purpose (and possibly other, unknown, purposes).

As to the AI safety space: we've been saying for decades that AI is dangerous and now you're surprised that we think AI is dangerous? I don't think it's taking over the world just yet, but that doesn't mean there are no smaller-scale risks. It's dangerous not because it's dumb (the fact it's still dumb is the saving grace) but precisely because it's smart.

My own approach is, use AI is clear, compartmentalized ways. If you have a particular task which you know can be done faster by using AI in a particular way, by all means, use it. (But, do pay attention to time wasted on tweaking the prompt etc.) Naturally, you should also occasionally keep experimenting with new tasks or new ways of using it. But, if there's no clear benefit, don't use it. If it's just to amuse yourself, don't. And, avoid exposing other people if there's no good reason.

I mostly agree with this, the part which feels off is

I’d like to say here “screw memetic egregores, follow the actual values of actual humans”

Humans already follow their actual Values^[1], and will always do because their Values are the reason they do anything at all. They also construct narratives about themselves that involve Goodness, and sometimes deny the distinction between Goodness and Values altogether. This act of (self-)deception is in itself motivated by the Values, at least instrumentally.

I do have a version of the “screw memetic egregores” attitude, which is, stop self-deceiving. Because, deception distorts epistemics, and we cannot afford distorted epistemics right now. It's not necessarily correct advice for everyone, but I believe it's correct advice for everyone who is seriously trying to save the world, at least.

Another nuance is that, in addition to empathy and naive tit-for-tat, there is also acausal tit-for-tat. This further pushes the Value-recommended strategy in the direction of something Goodness-like (in certain respects), even though ofc it doesn't coincide with the Goodness of any particular culture in any particular historical period.

1. ^{^}

   As Steven Byrnes wrote, "values" might be not the best term, but I will keep it here.

I'm using the term "meta-ethics" in the standard sense of analytic philosophy. Not sure what bothers you so greatly about it.

I find your manner of argumentation quite biased: you preemptively defend yourself by radical skepticism against any claim you might oppose, but when it comes to a claim you support (in this case "ethical realism is false"), suddenly this claim is "pretty close to analytic". The latter maneuver seems to me the same thing as the "Obviously Right" you criticize later.

Also, this brand of radical skepticism is an example of the Charybdis I was warning against. Of course you can always deny that anything matters. You can also deny Occam's razor or the evidence of your own eyes or even that 2+2=4. After all, "there's no predefined standard for standards". (I guess you might object that your reasoning only applies to value-related claims, not to anything strictly value-neutral: but why not?)

Under the premises of radical skepticism, why are we having this debate? Why did you decide to reply to my comment? If anyone can deny anything, why would any of us accept the other's arguments?

To have any sort of productive conversation, we need to be at least open to the possibility that some new idea, if you delve deeply and honestly into understanding it, might become persuasive by the force of the intuitions it engenders and its inner logical coherence combined. To deny the possibility preemptively is to close the path to any progress.

As to your "(b) there's a bunch of empirical evidence against it" I honestly don't know what you're talking about there.

P.S.

I wish to also clarify my positions on a slightly lower level of meta.

First, "ethics" is a confusing term because, on my view, the colloquial meaning of "ethics" is inescapably intertwined with how human societies negotiate of over norms. On the other hand, I want to talk purely about individual preferences, since I view it as more fundamental. 

We can still distinguish between "theories of human preferences" and "metatheories of preferences", similarly to the distinction between "ethics" and "meta-ethics". Namely, "theories of human preferences" would have to describe the actual human preferences, whereas "metatheories of preferences" would only have to describe what does it even mean to talk about someone's preferences at all (whether this someone is human or not: among other things, such a metatheory would have to establish what kind of entities have preferences in a meaningful sense).

The relevant difference between the theory and the metatheory is that Occam's razor is only fully applicable to the latter. In general, we should expect simple answers to simple questions. "What are human preferences?" is not a simple question, because it references the complex object "human". On the other hand "what does it mean to talk about preferences?" does seem to me to be a simple question. As an analogy, "what is the shape of Africa?" is not a simple question because it references the specific continent of Africa on the specific planet Earth, whereas "what are the general laws of continent formation" is at least a simpler question (perhaps not quite as simple, since the notion of "continent" is not so fundamental).

Therefore, I expect there to be a (relatively) simple metatheory of preferences, but I do not expect there to be anything like a simple theory of human preferences. This is why this distinction is quite important.

Your failure to distinguish ethics from meta-ethics is the source of your confusion (or at least one major source). When you say "ethical realism is false", you're making a meta-ethical statement. You believe this statement is true, hence you perforce must believe in meta-ethical realism.