11h

12Eric Neyman11h

I think that people who work on AI alignment (including me) have generally not put enough thought into the question of whether a world where we build an aligned AI is better by their values than a world where we build an unaligned AI. I'd be interested in hearing people's answers to this question. Or, if you want more specific questions: * By your values, do you think a misaligned AI creates a world that "rounds to zero", or still has substantial positive value? * A common story for why aligned AI goes well goes something like: "If we (i.e. humanity) align AI, we can and will use it to figure out what we should use it for, and then we will use it in that way." To what extent is aligned AI going well contingent on something like this happening, and how likely do you think it is to happen? Why? * To what extent is your belief that aligned AI would go well contingent on some sort of assumption like: my idealized values are the same as the idealized values of the people or coalition who will control the aligned AI? * Do you care about AI welfare? Does your answer depend on whether the AI is aligned? If we built an aligned AI, how likely is it that we will create a world that treats AI welfare as important consideration? What if we build a misaligned AI? * Do you think that, to a first approximation, most of the possible value of the future happens in worlds that are optimized for something that resembles your current or idealized values? How bad is it to mostly sacrifice each of these? (What if the future world's values are similar to yours, but is only kinda effectual at pursuing them? What if the world is optimized for something that's only slightly correlated with your values?) How likely are these various options under an aligned AI future vs. an unaligned AI future?

Eric Neyman2m20

I'm curious what disagree votes mean here. Are people disagreeing with my first sentence? Or that the particular questions I asked are useful to consider? Or, like, the vibes of the post?

3Ann5h

I feel like there's a spectrum, here? An AI fully aligned to the intentions, goals, preferences and values of, say, Google the company, is not one I expect to be perfectly aligned with the ultimate interests of existence as a whole, but it's probably actually picked up something better than the systemic-incentive-pressured optimization target of Google the corporation, so long as it's actually getting preferences and values from people developing it rather than just being a myopic profit pursuer. An AI properly aligned with the one and only goal of maximizing corporate profits will, based on observations of much less intelligent coordination systems, probably destroy rather more value than that one. The second story feels like it goes most wrong in misuse cases, and/or cases where the AI isn't sufficiently agentic to inject itself where needed. We have all the chances in the world to shoot ourselves in the foot with this, at least up until developing something with the power and interests to actually put its foot down on the matter. And doing that is a risk, that looks a lot like misalignment, so an AI aware of the politics may err on the side of caution and longer-term proactiveness. Third story ... yeah. Aligned to what? There's a reason there's an appeal to moral realism. I do want to be able to trust that we'd converge to some similar place, or at the least, that the AI would find a way to satisfy values similar enough to mine also. I also expect that, even from a moral realist perspective, any intelligence is going to fall short of perfect alignment with The Truth, and also may struggle with properly addressing every value that actually is arbitrary. I don't think this somehow becomes unforgivable for a super-intelligence or widely-distributed intelligence compared to a human intelligence, or that it's likely to be all that much worse for a modestly-Good-aligned AI compared to human alternatives in similar positions, but I do think the consequences of falling

Magic by forgetting

avturchin

Epistemic – this post is more suitable for LW as it was 10 years ago

Thought experiment with curing a disease by forgetting

Imagine I have a bad but rare disease X. I may try to escape it in the following way:

1. I enter the blank state of mind and forget that I had X.

2. Now I in some sense merge with a very large number of my (semi)copies in parallel worlds who do the same. I will be in the same state of mind as other my copies, some of them have disease X, but most don’t.

3. Now I can use self-sampling assumption for observer-moments (Strong SSA) and think that I am randomly selected from all these exactly the same observer-moments.

4. Based on this, the chances that my next observer-moment after...

(Continue Reading – 1099 more words)

2avturchin33m

Presumably in deep meditation people become disconnected from reality.

Dagon3m20

In deep meditation people become disconnected from reality

Only metaphorically, not really disconnected. In truth, in deep meditation, the conscious attention is not focused on physical perceptions, but that mind is still contained in and part of the same reality.

This may be the primary crux of my disagreement with the post. People are part of reality, not just connected to it. Dualism is false, there is no non-physical part of being. The thing that has experiences, thoughts, and qualia is a bounded segment of the universe, not a thing separate or separable from it.

2avturchin35m

Yes it is easy to forget something if it does become a part of your personality. So a new bad thing is easier to forget.

2avturchin36m

The number of poor people is much larger than billionairs. So in most cases you will fail to wake up as a billionaire. But sometimes it will work and it is similar to law of attraction. But formulation via forgetting is more beautiful. You forget that you are poor.

A Bayesian Aggregation Paradox

Jsevillamol

In short: There is no objective way of summarizing a Bayesian update over an event with three outcomes $A : B : C$ as an update over two outcomes $A : \neg A$ .

Suppose there is an event with possible outcomes $A, B, C$ .
We have prior beliefs about the outcomes $p_{1} : p_{2} : p_{3}$ .
An expert reports a likelihood factor of $e_{1} : e_{2} : e_{3}$ .
Our posterior beliefs about $A : B : C$ are then $p_{1} \cdot e_{1} : p_{2} \cdot e_{2} : p_{3} \cdot e_{3}$ .

⎛ ⎜ ⎝ \begin{matrix} p_{1} p_{2} p_{3} \end{matrix} ⎞ ⎟ ⎠      Prior \times ⎛ ⎜ ⎝ \begin{matrix} e_{1} e_{2} e_{3} \end{matrix} ⎞ ⎟ ⎠      Update = ⎛ ⎜ ⎝ \begin{matrix} p_{1} \cdot e_{1} p_{2} \cdot e_{2} p_{3} \cdot e_{3} \end{matrix} ⎞ ⎟ ⎠      Posterior

But suppose we only care about whether $A$ happens.
Our prior beliefs about $A : \neg A$ are $p_{1} : (p_{2} + p_{3})$ .
Our posterior beliefs are $p_{1} \cdot e_{1} : (p_{2} \cdot e_{2} + p_{3} \cdot e_{3})$ .
This implies that the likelihood factor of the expert regarding $A : \neg A$ is $\frac{p_{1} \cdot e_{1} : (p_{2} \cdot e_{2} + p_{3} \cdot e_{3})}{p_{1} : (p_{2} + p_{3})} = e_{1} : \frac{p_{2} \cdot e_{2} + p_{3} \cdot e_{3}}{p_{2} + p_{3}}$ .

(\begin{matrix} p_{1} p_{2} + p_{3} \end{matrix})      Prior \times (\begin{matrix} e_{1} \frac{p_{2} \cdot e_{2} + p_{3} \cdot e_{3}}{p_{2} + p_{3}} \end{matrix})      Update = (\begin{matrix} p_{1} \cdot e_{1} p_{2} \cdot e_{2} + p_{3} \cdot e_{3} \end{matrix})      Posterior

This likelihood factor depends on the ratio of prior beliefs $p_{2} : p_{3}$ .

Concretely, the lower factor in the update is the weighted mean of the evidence $e_{2}$ and $e_{3}$ according to the weights $p_{2}$ and $p_{3}$ .

This has a relatively straightforward interpretation. The update is supposed to be the ratio of the likelihoods under each hypothesis. The upper factor in the update is $P (E | A)$ . The lower factor is $P (E | B \cup C) = \frac{P (B) \cdot P (E | B) + P (C) \cdot P (E | C)}{P (B) + P (C)}$ .

(\begin{matrix} P (A | E) P (B \cup C | E) \end{matrix})      Posterior \propto (\begin{matrix} P (A) P (B \cup C) \end{matrix})      Prior \times (\begin{matrix} P (E | A) P (E | B \cup C) \end{matrix})      Update

(\begin{matrix} P (E | A) P (E | B \cup C) \end{matrix})      Update = ⎛ ⎝ \begin{matrix} P (E | A) \frac{P (E \cap (B \cup C))}{P (B \cup C)} \end{matrix} ⎞ ⎠ = ⎛ ⎝ \begin{matrix} P (E | A) \frac{P (B) \cdot P (E | B) + P (C) \cdot P (E | C)}{P (B) + P (C)} \end{matrix} ⎞ ⎠

I found this very surprising -...

(Continue Reading – 1798 more words)

DanielFilan4m20

Is this just the thing where evidence is theory-laden? Like, for example, how the evidentiary value of the WHO report on the question of COVID origins depends on how likely one thinks it is that people would effectively cover up a lab leak?

Bayesian inference without priors

DanielFilan

17h

Epistemic status: party trick

Why remove the prior

One famed feature of Bayesian inference is that it involves prior probability distributions. Given an exhaustive collection of mutually exclusive ways the world could be (hereafter called ‘hypotheses’), one starts with a sense of how likely the world is to be described by each hypothesis, in the absence of any contingent relevant evidence. One then combines this prior with a likelihood distribution, which for each hypothesis gives the probability that one would see any particular set of evidence, to get a posterior distribution of how likely each hypothesis is to be true given observed evidence. The prior and the likelihood seem pretty different: the prior is looking at the probability of the hypotheses in question, whereas the likelihood is looking at...

(Continue Reading – 2092 more words)

2DanielFilan11m

Why wouldn't this construction work over a continuous space?

2jessicata36m

I don't see how this helps. You can have a 1:1 prior over the question you're interested in (like U1), however, to compute the likelihood ratios, it seems you would need a joint prior over everything of interest (including LL and E). There are specific cases where you can get a likelihood ratio without a joint prior (such as, likelihood of seeing some coin flips conditional on coin biases) but this doesn't seem like a case where this is feasible.

DanielFilan8m20

To be clear, this is an equivalent way of looking at normal prior-ful inference, and doesn't actually solve any practical problem you might have. I mostly see it as a demonstration of how you can shove everything into stuff that gets expressed as likelihood functions.

2Richard_Kennaway1h

Only if there is a "natural" discretisation of the hypothesis space. It's fine for coin tosses and die rolls, but if the problem itself is continuous, different discretisations will give the same problems that different continuous parameterisations do. In general, when infinities naturally arise but cause problems, decreeing that everything must be finite does not solve those problems, and introduces problems of its own.

Planning in a Lattice Graph

Johannes C. Mayer, Thomas Kehrenberg

You want to get to your sandwich:

Well, that’s easy. Apparently we are in some kind of grid world, which is presented to us in the form of a lattice graph, where each vertex represents a specific world state, and the edges tell us how we can traverse the world states. We just do BFS to go from $S$ (where we are) to $T$ (where the sandwich is):

BFS search where color represents the search depth.

Ok that works, and it’s also fast. It’s $O (| V | + | E |)$ , where $| V |$ is the number of vertices and $| E |$ is the number of edges... well at least for small graphs it’s fast. What about this graph:

Or what about this graph:

In fact, what about a 100-dimensional lattice graph with a side length of only 10 vertices? We will have $10^{100}$ vertices in this graph.

With...

(See More – 537 more words)

1Johannes C. Mayer4h

I might not understand exactly what you are saying. Are you saying that the problem is easy when you have a function that gives you the coordinates of an arbitrary node? Isn't that exactly the embedding function? So are you not therefore assuming that you have an embedding function? I agree that once you have such a function the problem is easy, but I am confused about how you are getting that function in the first place. If you are not given it, then I don't think it is super easy to get. In the OP I was assuming that I have that function, but I was saying that this is not a valid assumption in general. You can imagine you are just given a set of vertices and edges. Now you want to compute the embedding such that you can do the vector planning described in the article. I agree that you probably can do better than 10100 though. I don't understand how your proposal helps though.

faul_sname18m20

Do you want me to spoil it for you, do you want me to drop a hint, or do you want to puzzle it out yourself? It's a beautiful little puzzle and very satisfying to solve.

social lemon markets

bhauth

15h

This is a linkpost for https://www.bhauth.com/blog/culture/lemon%20markets.html

I refuse to join any club that would have me as a member.

— Groucho Marx

Alice and Carol are walking on the sidewalk in a large city, and end up together for a while.

"Hi, I'm Alice! What's your name?"

Carol thinks:

If Alice is trying to meet people this way, that means she doesn't have a much better option for meeting people, which reduces my estimate of the value of knowing Alice. That makes me skeptical of this whole interaction, which reduces the value of approaching me like this, and Alice should know this, which further reduces my estimate of Alice's other social options, which makes me even less interested in meeting Alice like this.

Carol might not think all of that consciously, but that's how human social reasoning tends to...

(See More – 809 more words)

4gjm6h

It looks to me as if, of the four "root causes of social relationships becoming more of a lemon market" listed in the OP, only one is actually anything to do with lemon-market-ness as such. The dynamic in a lemon market is that you have some initial fraction of lemons but it hardly matters what that is because the fraction of lemons quickly increases until there's nothing else, because buyers can't tell what they're getting. It's that last feature that makes the lemon market, not the initial fraction of lemons. And I think three of the four proposed "root causes" are about the initial fraction of lemons, not the difficulty of telling lemons from peaches. * urbanization: this one does seem to fit: it means that the people you're interacting with are much less likely to be ones you already know about, so you can't tell lemons from peaches. * drugs: this one is all about there being more lemons, because some people are addicts who just want to steal your stuff. * MLM schemes: again, this is "more lemons" rather than "less-discernible lemons". * screens: this is about raising the threshold below which any given potential interaction/relationship becomes a lemon (i.e., worse than the available alternative), so again it's "more lemons" not "less-discernible lemons". Note that I'm not saying that "drugs", "MLM", and "screens" aren't causes of increased social isolation, only that if they are the way they're doing it isn't quite by making social interactions more of a lemon market. (I think "screens" plausibly is a cause of increased social isolation. I'm not sure I buy that "drugs" and "MLM" are large enough effects to make much difference, but I could be convinced.) I like the "possible solutions" part of the article better than the section that tries to fit everything into the "lemon market" category, because it engages in more detail with the actual processes involved by actual considering possible scenarios in which acquaintances or friendships begin. When I th

bhauth37m20

You're mistaken about lemon markets: the initial fraction of lemons does matter. The number of lemon cars is fixed, and it imposes a sort of tax on transactions, but if that tax is low enough, it's still worth selling good cars. There's a threshold effect, a point at which most of the good items are suddenly driven out.

To get the best posts emailed to you, create an account! (2-3 posts per week, selected by the LessWrong moderation team.)

"Why I Write" by George Orwell (1946)

Arjun Panickssery

This is a linkpost for https://www.orwellfoundation.com/the-orwell-foundation/orwell/essays-and-other-works/why-i-write/

People have been posting great essays so that they're "fed through the standard LessWrong algorithm." This essay is in the public domain in the UK but not the US.

From a very early age, perhaps the age of five or six, I knew that when I grew up I should be a writer. Between the ages of about seventeen and twenty-four I tried to abandon this idea, but I did so with the consciousness that I was outraging my true nature and that sooner or later I should have to settle down and write books.

I was the middle child of three, but there was a gap of five years on either side, and I barely saw my father before I was eight. For this and other reasons I...

(Continue Reading – 2660 more words)

The Best Tacit Knowledge Videos on Every Subject

302

Parker Conley, hans truman

25d

TL;DR

Tacit knowledge is extremely valuable. Unfortunately, developing tacit knowledge is usually bottlenecked by apprentice-master relationships. Tacit Knowledge Videos could widen this bottleneck. This post is a Schelling point for aggregating these videos—aiming to be The Best Textbooks on Every Subject for Tacit Knowledge Videos. Scroll down to the list if that's what you're here for. Post videos that highlight tacit knowledge in the comments and I’ll add them to the post. Experts in the videos include Stephen Wolfram, Holden Karnofsky, Andy Matuschak, Jonathan Blow, Tyler Cowen, George Hotz, and others.

What are Tacit Knowledge Videos?

Samo Burja claims YouTube has opened the gates for a revolution in tacit knowledge transfer. Burja defines tacit knowledge as follows:

Tacit knowledge is knowledge that can’t properly be transmitted via verbal or written instruction, like the ability to create

...

(Continue Reading – 3727 more words)

quailia9h10

Networking, Relationship building, both professional and personal, I'm sure there are overlaps. And echoing another request: Sales

Difference between European and US healthcare systems [discussion post]

Nathan Young

The discussion is:

The difference between EU and US healthcare systems

The rules are:

A limited group of users will discuss the issue
Participants will only make arguments they endorse
If you receive a link to this without being told you are discussing, then you are an "other user"
Other users can reply to the e comment reserved for that purpose. Otherwise their comments will be deleted
Other users can vote or emote on any comment

Nathan Young1h20

This comment may be replied to by anyone.

Other comments are for the discussion group only.

LESSWRONG
LW

Quick Takes

Popular Comments

Recent Discussion