All of Viktor Rehnberg's Comments + Replies

You can see twin birth rates fell sharply in the late 90s

Shouldn't this be triplet birthrates? Twin birthrates look pretty stable in comparison.

Hmm, yeah it's a bit hard to try stuff when there's no good preview. Usually I'd recommend rot13 chiffer if all else fails but for number sequences that makes less sense.

I knew about 2-4-6 problem from HPMOR, I really like the opportunity to try it out myself. These are my results on the four other problems:

indexA

Number of guesses:

Guess:

Result: Failure

indexB

Number of guesses:

Guess:

Result: Success

indexC

Number of guesses:

Guess... (read more)

See FAQ for spoiler tags, it seems mods haven't seen your request. https://www.lesswrong.com/faq#How_do_I_insert_spoiler_protections_

These problems seemed to me similar to the problems at the International Physicist's Tournament. If you want more problems check out https://iptnet.info

In case anyone else is looking for a source a good search term is probably the Beal Effect. From the original paper by Beal and Smith:

Once the effect is pointed out, it does not take long to arrive at the conclusion that it arises from a natural correlation between a high branching factor in the game tree and having a winning move available. In other words, mobility (in the sense of having many moves available) is associated with better positions

Or a counterexample from the other direction would be that you can't describe a uniform distribution of the empty set either (I think). And that would feel even weirder to call "bigger".

Why would this property mean that it is "bigger"? You can construct a uniform distribution of a uncountable set through a probability density as well. However, using the same measure on a countably infinite subset of the uncountable set would show that the countable set has measure 0.

So we have that

[...] Richard Jeffrey is often said to have defended a specific one, namely the ‘news value’ conception of benefit. It is true that news value is a type of value that unambiguously satisfies the desirability axioms.

but at the same time

News value tracks desirability but does not constitute it. Moreover, it does not always track it accurately. Sometimes getting the news that X tells us more than just that X is the case because of the conditions under which we get the news.

And I can see how starting from this you would get that $U\left(\top \right)=0$. ... (read more)

Skimming the methodology it seems to be a definite improvement and does tackle the short-comings mentioned in the original post to some degree at least.

Isn't that just a question whether you assume expected utility or not. In the general case it is only utility not expected utility that matters.

Anyway, someone should do a writeup of our findings, right? :)

Sure, I've found it to be an interesting framework to think in so I suppose someone else might too. You're the one who's done the heavy lifting so far so I'll let you have an executive role.

If you want me to write up a first draft I can probably do it end of next week. I'm a bit busy for at least the next few days.

Lol. Somehow $1:100000000$ made it more clear that it was meant as a hyperbole than $1:\infty$ did.

You might want to consider cross-posting this to EA forum to reach a larger audience.

I've been thinking about the Eliezer's take on the Second Law of Thermodynamics and while I can't think of a succint comment to drop with it. I think it could bring value to this discussion.

Well I'd say that the difference between your expectations of the future having lived a variant of it or not is only in degree not in kind. Therefore I think there are situations where the needs of the many can outweigh the needs of the one, even under uncertainty. But, I understand that not everyone would agree.

$A⟺\left(\right(A\wedge s_1)\vee (A\wedge s_2\left)\right)$ I agree with as a sufficient criteria to only sum over $\{s_1,s_2\}$, the other steps I'll have to think about before I get them.

I found this newer paper https://personal.lse.ac.uk/bradleyr/pdf/Unification.pdf and having skimmed it seemed like it had similar premises but they defined $U\left(A|B\right)=U(A\wedge B)-U\left(B\right)+U\left(\top \right)=U(A\wedge B)-U\left(B\right)$ (instead of deriving it).

GovAI is probably one of the densest places to find that. You could also check out FHI's AI Governance group.

There is no consensus about what constitutes a moral patient and I have seen nothing convincing to rule out that an AGI could be a moral patient.

However, when it comes to AGI some extreme measures are needed.

I'll try with an analogy. Suppose that you traveled back in time to Berlin 1933. Hitler has yet to do anything significantly bad but you still expect his action to have some really bad consequences.

Now I guess that most wouldn't feel terribly conflicted about removing Hitler's right of privacy or even life to prevent Holocaust.

For a longtermist the ris... (read more)

Didn't you use that $B\vee \neg B=\top$. I can see how to extend the derivation for more steps $s_1\vee s_2\vee \dots \vee s_n$ but only if $\{s_i{\}}_{i=1}^n=\Omega$. The sums

$$\sum _{s\in S}P\left(s|a\right)U(a\wedge s)$$

and

$$\sum _{\omega \in \Omega }P\left(\omega |a\right)U(a\wedge \omega )$$ for arbitrary $U$ are equal if and only if $P(\Omega \setminus S|a)=0$.

The other alternative I see is if (and I'm unsure about this) we assume that $U(z\wedge a)=U\left(z\right)$ and $P\left(z|a\right)=P\left(z\right)$ for $z\in \Omega \setminus S$.

What I would think that $U\left(A|B\right)$ would mean is $U\left(A\right)$ after we've updated probabilities and utilities from the fact that $B$ is certain. I think that would be the first one but I'm not sure. I can't tell which one that would be.

General $S$ (even if mutually exclusive) is tricky I'm not sure the expression is as nice then.

$$\begin{array}{cc}U\left(\top \right)& =\sum _{\omega \in \Omega }P\left(\omega \right)U\left(\omega \right)\\ & =0\end{array}$$ that was one of the premises, no? You expect utility $0$ from your prior.

Some first reflections on the results before I go into examining all the steps.

Hmm, yes my expression seems wrong when I look at it a second time. I think I still confused the timesteps and should have written $$U\left(a\right)=\sum _{\omega \in \Omega }(P\left(\omega |a\right)U(\omega \wedge a)-P\left(\omega \right)U\left(\omega \right))$$

The extra negation comes from a reflex from when not using Jeffrey's decision theory. With Jeffrey's decision theory it reduces to your expression as the negated terms sum to $U\left(\top \right)=0$. But, still I probably should learn not to guess at theorems and properly do all steps in the future. I suppose that is a point in favor f... (read more)

Ah, those timestep subscripts are just what I was missing. I hadn't realised how much I needed that grounding until I noticed how good it felt when I saw them.

So to summarise (below all sets have mutually exclusive members). In Jeffrey-ish notation we say have the axiom

$$U\left(S\right)=\frac{1}{P\left(S\right)}\sum _{s\in S}P\left(s\right)U\left(s\right)$$

and normally you would want to indicate what distribution you have over $S$ in the left-hand side. However, we always renormalize $U$ such that the distribution is our current prior. We can indicate this by labeling the utilities from what timestep (and agent should probabl... (read more)

Well, deciding to do action $a$ would also make it utility 0 (edit: or close enough considering remaining uncertainties) even before it is done. At least if you're committed to the action and then you could just as well consider the decision to be the same as the action.

It would mean that a "perfect" utility maximizer always does the action with utility $0$ (edit: but the decision can have positive utility(?)). Which isn't a problem in any way except that it is alien to how I usually think about utility.

Put in another way. While I'm thinking about which possib... (read more)

Oh, I think I see what confuses me. In the subjective utility framework the expected utilities are shifted to $0$ after each Bayesian update?

So then utility of doing action $a$ to prevent a Doom is $\left(P\right(\text{Doom}|a)-P(\text{Doom}\left)\right)U\left(\text{Doom}\right)+\left(P\right(\neg \text{Doom}|a)-P(\neg \text{Doom}\left)\right)U\left(\neg \text{Doom}\right)$. But when action $a$ has been done then the utility scale is shifted again.

Ok, so this is a lot to take in, but I'll give you my first takes as a start.

My only disagreement prior to your previous comment seems to be in the legibility of the desirability axiom for $U(A\vee B)$ which I think should contain some reference to the actual probabilities of $A$ and $B$.

Now, I gather that this disagreement probably originates from the fact that I defined $U\left(\right\{\left\}\right)=0$ while in your framework $U\left(\top \right)=0$.

Something that appears problematic to me is if we consider the tautology (in Jeffrey notation) $U(\text{Doom}\vee \neg \text{Doom})=P\left(\text{Doom}\right)U\left(\text{Doom}\right)+P\left(\neg \text{Doom}\right)U\left(\neg \text{Doom}\right)=0$. This would mea... (read more)

What I found confusing with $P(A\vee \neg A)U(A\vee \neg A)$ was that to me this reads as $U(A\vee \neg A)$ which should always(?) depend on $P\left(A\right)$ but with this notation it is hidden to me. (Here I picked $\neg A$ as the mutually exclusive event $B$, but I don't think it should remove much from the point).

That is also why I want some way of expressing that in the notation. I could imagine writing as $U_X\left(\Omega \right)$ that is the cleanest way I can come up with to satisfy both of us. Then with expected utility $U_X\left(\Omega \right)=E_X\left[U\right(X\left)\right]$.

When we accept the expected utility hypothesis then we can always write it as a e... (read more)

Hmm, I usually don't think too deeply about the theory so I had to refresh somethings to answer this.

First off, the expected utility hypothesis is apparently implied by the VNM axioms. So that is not something needed to add on. To be honest I usually only think of a coherent preference ordering and expected utilities as two seperate things and hadn't realized that VNM combines them.

About notation, with $U\left(A\right)$ I mean the utility of getting $A$ with certainty and with $pA$ I mean the utility of getting $A$ with probability $p$. If you don't have the expected utility h... (read more)

Having read some of your other comments. I expect you to ask if the top preference of a thermostat is it's goal temperature? And to this I have no good answer.

For things like a thermostat and a toy robot you can obviously see that there is a behavioral objective which we could use to infer preferences. But, is the reason that thermostats are not included in utility calculations that behavioral objective does not actually map to a preference ordering or that their weight when aggregated is 0.

Perhaps for most they don't have this in the back of their mind when they think of utility. But, for me this is what I'm thinking about. The aggregation is still confusing to me, but as a simple case example. If I want to maximise total utility and am in a situation that only impacts a single entity then increasing utility is the same to me as getting this entity in for them more preferable states.

Expected utility hypothesis is that $U\left(pA\right)=pU\left(A\right)$. To make it more concrete suppose that for outcome $A$ is worth $10\text{utils}$ for you. Then getting $A$ with probaillity $1/2$ is worth $5\text{utils}$. This is not necessarily true, there could be an entity that prefers outcomes comparatively more if they are probable/improbable. The name comes from the fact that if you assume it to be true you can simply take expectations of utils and be fine. I find it very agreeable for me.

You could perhaps argue that "preference" is a human concept. You could extend it with something like coherent extrapolated volition to be what the entity would prefer if it knew all that was relevant, had all the time needed to think about it and was more coherent. But, in the end if something has no preference, then it would be best to leave it out of the aggregation.

Could someone who disagrees with the above statement help me by clarifying what the disagreement is?

Seeing as it has -7 on the agreement vote and that makes me think it should be obvious but it isn't to me.

Due to this, he concludes the cause area is one of the most important LT problems and primarily advises focusing on other risks due to neglectedness.

This sentence is confusing me, should I read it as:

1. Due to this, he concludes the cause area is one of the most important LT problems but primarily advises focusing on other risks anyway due to neglectedness.
2. Due to this, he concludes the cause area is not one of the most important LT problems and primarily advises focusing on other risks due to neglectedness.

From this summary of the summary I get the th... (read more)

I agree with 1 (but then it is called alignment forum, not the more general AI Safety forum). But I don't see that 2 would do much good.

All narratives I can think of where 2 plays a significant part sounds like strawmen to me, perhaps you could help me?

I suppose I would just like to see more people start at an earlier level and from that vantage point you might actually want to switch to a path with easier parts.

There’s something very interesting in this graph. The three groups have completely converged by the end of the 180 day period, but the average bank balance is now considerably higher.

Wasn't the groups selected for having currently low income? Shouldn't we expect some regression towards the mean i.e. an increase in average bank balance? Was there any indication for if the observed effect was larger or smaller than expected?

Tackle the [Hamming Problems](https://www.lesswrong.com/posts/Thwfy4gNFx9kHgvov/research-hamming-questions), Don't Avoid Them

I agree with that statement and this statement

Far and away the most common failure mode among self-identifying alignment researchers is to look for Clever Ways To Avoid Doing Hard Things [...]

seems true as well. However, there was something in this section that didn't seem quite right to me.

Say that you have identified the Hamming Problem at lowest resolution be getting the outcome "AI doesn't cause extinction or worse". However, if ... (read more)

I don't think an "actual distribution" over the activations is a thing? The distribution depends on what inputs you feed it.

This seems to be what Thomas is saying as well, no?

[...] look at the network activations at each layer for a bunch of different inputs. This gives you a bunch of activations sampled from the distribution of activations. From there, you can do density estimation to estimate the actual distribution over the activations.

The same way you can talk about the actual training distribution underlying the samples in the training set it should b... (read more)

It seems like it could be worthwhile for you to contact someone in connection to AGI Safety Communications Initiative. Or at the very least check out the post I linked.

To me it seems that the disagreement around this question comes from thinking of different questions.

Has DL produced a significant amount of economic value?

Yes, and I think this has been quite established already. It is still possible to argue about what is meant by significant but I think that disagreement is probably coming better resolved by asking a different question.

(I can imagine that many of these examples technically exist, but not at the level that I mean).

From this and some comments, I think there is a confusion that would be better resolved by asking:

Why don't DL in consumer products seem as amazing as what I see from presentations of research results?

I have not seen anyone do something like this but it sounds like something Anders Sandberg (FHI) would do. If you want a lead or want to find someone that might be interested in researching it, he might be it.

I haven't followed your arguments all the way here but I saw the comment

If I am understanding correctly, you are saying if the sleeping beauty problem does not use a coin toss, but measures the spin of an election instead, then the answer would be different.

and would just jump in and say that others have made a similar arguments. The one written example I've seen is this Master's Thesis. 

I'm not sure if I'm convinced but at least I buy that depending on how the particular selection goes about there can be instances were difference between probabilitie... (read more)

An area where I think there is an important difference between doing explicit search and optimisation through piles of heuristics is in clistering NN à la Filan et al. (link TBD).

A usecase I've been thinking about is to use that kind of technique to help identify mesaoptimisation or more particularly mesaobjectives (with the help of interpretability tools guided by the clustering of the NN).

In the case of explicit search I would expect that it would be more common than not to be able to find a specific part of the network evaluating world states in terms o... (read more)

Oh, I hadn't noticed that. I've got some connections to them and can reach out.