All of TAG's Comments + Replies

Coping with Undecidability

But (for convenience?), mathematicians/computer scientists use Turing machines/Lambda Calculus/(your favorite programming language) for most purposes instead to model things, even though they have these really peculiar undecidability properties.

FSMs also have undecidability: for each one there an infinity of programmes they can't even run.

Is AI Alignment a pseudoscience?

Yes. I was trying to avoid the downvote demon by hinting quietly.

PS looks like he winged me.

Some thoughts on "The Nature of Counterfactuals"

When talking about counterfactuals, LessWrong tends to focus on a subset of counterfactuals that I would call “decision counterfactuals”. These are counterfactuals with respect to your own actions; e.g. “if I take a drink from my tea, then...”.

There's a subset of lesswrongians , including EY, who have a problem with counterfactuals that other people don't have, so it might be interesting to identify the problem. Many paradoxes arise from self if you assume the central example of a counterfactual is a counterfactual about your own behavio... (read more)

1tailcalled10hThat sort of misattribution seems like a good way to frame it.
Lives of the Cambridge polymath geniuses

A third question is why they were disenchanted with capitalism. But it's not difficult to answer. ... there was the General Strike and the Great Depression.

Someone was bulverising their rejection of capitalism as self-serving, but the cracks in capitalism were obvious.

Some thoughts on "The Nature of Counterfactuals"

. But what if the nondeterminism actually has a bias towards entropy-decreasing events?

What if determinism has a bias towards entropy-decreasing events? There is no contradiction in the idea, it just means that the causal, entropic, and temporal arrows don't necessarily align.

Indeterminism kind of has to increase entropy because there are more high entropic states than low entropic ones, so a random walk starting at a low entropy state will see increasing entropy.

But that's dependent on a low entropy starting state.

The low entropy starting state needs e... (read more)

1tailcalled11hThat's the idea I try to address in my above comment. In our universe, if the same pattern appears twice, then presumably those two appearances have a common cause. But if our universe was just the time-reversed version of an entropy-decreasing universe, it doesn't seem to me like this would need to be the case. Only if you are measure-preserving.
Is AI Alignment a pseudoscience?

This opinion is common on this forum, it puts you in what could be called the ‘pre-paradigmatic’ camp

Maybe it's common now.

During the high rationalist era, early 2010s, there was supposed to be a theory of AGI based on rationality. The problem was that ideal rationality is uncomputable, so that approach would involve going against what is already known about computation, and therefore crankish. (And the claim that any AI is non ideally rational, whilst defensible for some values of non ideallyrational, is not useful, since there are many ways of being non-ideal).

1Koen.Holtman11hI am not familiar with the specific rationalist theory of AGI developed in the high rationalist era of the early 2010s. I am not a rationalist, but I do like histories of ideas, so I am delighted to learn that such a thing as the high rationalist era of the early 2010s even exists. If I were to learn more about the actual theory, I suspect that you and I would end up agreeing that the rationalist theory of AGI developed in the high rationalist era was crankish.
Is AI Alignment a pseudoscience?
Answer by TAGJan 25, 20221

Reasoning about AGI is similar to reasoning about black holes: both of these do not necessarily lead to pseudo-science, though both also attract a lot of fringe thinkers, and not all of them think robustly all of the time

For the two to be similar, there needs to be an equivalent to the laws of physics. Then the cranks would be the people who are ignoring them. But, despite the expenditure of a lot of effort, no specific laws of AGI have been found .

(Of course, AGI is subject to the same general laws as any form of computation).

1Koen.Holtman2dIt is your opinion that despite the expenditure of a lot of effort, no specific laws of AGI have been found. This opinion is common on this forum, it puts you in what could be called the 'pre-paradigmatic' camp. My opinion is that the laws of AGI are the general laws of any form of computation (that we can physically implement), with some extreme values filled in. See my original comment. Plenty of useful work has been done based on this paradigm.
“A Harmful Idea”

It's also telling that there are lots of downvotes, and very little critique.

1clearthis8dThere has been quite a lot of discussion over on the EA Forum: [] Avital Balwit linked to this lesswrong post in the comments of her own response to his longtermism critique (because Phil Torres is currently banned from the forum, afaik): []
On the Impossibility of Interaction with Simulated Agents

Of course jblack meant "given your assumptions.."

Some thoughts on "The Nature of Counterfactuals"

Our universe appears to be a causal universe. By this, I mean that it appears to be characterized by having some particular ‘starting state’, and evolving forwards as a dynamical process according to some causal laws.

Much of what you say depends on on strict determinism rather than causality. Causality is a superset of determinism.

1tailcalled10dThis is definitely where the most likely loophole in my argument is, especially when combined with the quantum/chaos thing too. I'd be curious to know whether it is a serious loophole that might break the argument, or if it's not so important. Let's consider my argument against causality running backwards; I said that this seemed unlikely since entropy is increasing. But what if the nondeterminism actually has a bias towards entropy-decreasing events? In that case, it seems like we could definitely expect a high-entropy universe to evolve into a low-entropy one. To make this more concrete, imagine that we take a "heat death of the universe" state, and then add some imperceptible Gaussian noise to it, leading to a distribution of states possible states. Now imagine that we evolve this forwards for a short while; this is going to lead to a wider distribution of states, with some that are lower entropy than the initial state. Suppose we then perform a Bayesian update on the entropy being low; this leaves us with a narrower distribution of states. And imagine that we then keep repeating this, constantly evolving the distribution forwards and updating on low entropy, to generate an entire trajectory for the universe. Would this method be well-defined? I'm not sure, maybe one eventually does an update on something that is probability 0. (Also, it would be challenging to define 'entropy' properly here...) But assuming it would be well-defined, would the resulting trajectory look similar to a time-reversed version of our universe? My suspicion is that the answer to this is "no", but I'm not sure. It seems like this could generate all sorts of weird information-based shenanigans, like there's nothing preventing a complex pattern from arising twice independently.
On the Impossibility of Interaction with Simulated Agents

This is a serious objection, though phrased frivolously

I particularly enjoyed that phrase.

3Charlie Steiner10dAlways good to start a comment by telling the author you think they won't listen :P (sarcasm!)
Some thoughts on "The Nature of Counterfactuals"

This isn’t quite true. There’s quantum mechanics, which shows that there isn’t determinism with respect to our usual embodied view, and there’s chaos theory which shows that even nominally deterministic laws are not so in practice. I think these effects are small enough that my argument still goes through

Classical chaos can amplify quantum indeterminism indefinitely.

and many people also find it convincing that the problems can be patched (e.g. many worlds interpretation of quantum mechanics);

A decoherent branch of the universal wave function that you happen not to be in is fully's more real than what is usually meant by a real counterfactual.

No Abstraction Without a Goal

You can make the claim that a useful abstraction must serve a purpose, without making the claim that all abstractions are useful.

AMA: I was unschooled for most of my childhood, then voluntarily chose to leave to go to a large public high school.

I’m wondering if there’s a word for “spelling system where each letter corresponds to exactly one sound but that sound can change with consistency based on regional dialect/inflection/other context”.

There's an implementation!

AMA: I was unschooled for most of my childhood, then voluntarily chose to leave to go to a large public high school.

Wha';s called "phonetic" in the context of education is at least partly, given systematic differences in the spoken language, the letters can be interpreted differently.

No Abstraction Without a Goal

The argument given by the OP seems valid to me ... that is the reason to believe that abstractions relate to goals.

Goals are not abstractions in the sense of compressions if an existing territory. When Kennedy asserted a goal to put a man on the moon, he was not representing something that was already true

$1000 USD prize - Circular Dependency of Counterfactuals

That doesn't help. If recursive justification is a particular kind of circular argument that's valid, so that others are invalid, then something makes it valid. But what? EY doesn't say. And how do we know that the additional factor isn't doing all the work?

The Map-Territory Distinction Creates Confusion

Nothing much. A definition of truth doesn't have to make the truth about everything available to every agent

$1000 USD prize - Circular Dependency of Counterfactuals

He doesn't show much sign of embracing the validity of all circular argument ss, and neither do you.

2Chris_Leong20dI never said all circular arguments are valid
The Problem of the Criterion is NOT an Open Problem

Mathematics is not exempt from the problem of the criterion.

The Map-Territory Distinction Creates Confusion

Nota Bene: Just so there’s no misunderstanding, neither materialism nor idealism need be metaphysical assumptions. Questions about materialism and idealism can be investigated via standard methods if we don’t presuppose them. It’s only that a correspondence theory of truth forces these to become metaphysical assumptions by making our notion of truth depend on assuming something about the nature of reality to ground our criterion of truth

Assuming what about the nature of reality? M/T and CToT only require that there is some sort of territory....they dont... (read more)

$1000 USD prize - Circular Dependency of Counterfactuals

It is a matter of perspective whether a world is factual (contains me) or counterfactual.

How so? I would have said the opposite.

I wasn't saying that that is true per se, I was saying it's Lewis's view .

Sorry, I should have been clearer. I wasn’t disagreeing with there being more than one category, but your conclusion from this.

Well,if you think there is a special problem with counterfactuals , then needs a basis other than general Kantian issues.

2Chris_Leong25dAh, okay. I get it now.
$1000 USD prize - Circular Dependency of Counterfactuals

I haven't rejected counterfactual realism. I've pointed out that Lewis's modal realism doesn't deal with counterfactuals as such, because it is a matter of perspective whether a world is factual (ie. contains me) or counterfactual (doesn't).

What I have called moderate realism is the only position that holds counterfactuals to be both intrinsically counterfactual and real.

Presumably, you don’t think moderate realism leads you down this path. Where do you think it leads instead?

Kantianism about counterfactuals might be true, but if it is, you are also... (read more)

2Chris_Leong25dHow so? I would have said the opposite. Yeah, if Kantianism about counterfactuals were true, it would be strange to limit it. My expectation would be that it would apply to a bunch of other things as well. Sorry, I should have been clearer. I wasn't disagreeing with there being more than one category, but your conclusion from this.
$1000 USD prize - Circular Dependency of Counterfactuals

From philosophy of language perspective, I personally like to argue that hypotheticals in past tense are just wrong, but are used in the same way present and future tense versions are: expressing internal belief about how causality will play out for the sake of aligning them in a group

I dont see why philosophy of language would tell you how reality works

Can each event in the world be attributed conditions under which it occurs?
Answer by TAGJan 02, 20221

Your questions seem quite unclear to me. Does...

Can each event in the world be attributed conditions under which it occurs

..mean that it can only occur under certain conditions, or it must occur given certain conditions, or it just happens to occur under certain conditions, without their being particularly relevant?

1Maciej Jałocha1moHi TAG, thanks for the comment, I meant the first and ?third? one but I'm also sure if I understood you correctly. I meant that each observed event can be assigned some events/states etc. that had appeared before/ were happening parallel to the event and be assigned probability of happening this again. (Have you seen the content of my question? Title contained only my first thought for simplicity) * For the first one I think we can assign such conditions when there's >= 0% chance of occuring and when it isn't. So I'd say there are some particular conditions (>=0%) under which event can only have a chance to occur. * Surely I didn't want my post to be interpreted that every event must occur given certain conditions. * I'm not sure of the third, but I think I resolved your question? I've changed my post accordingly.
$1000 USD prize - Circular Dependency of Counterfactuals

Rejecting David Lewis’ Counterfactual Realism as absurd and therefore concluding that counterfactuals must be at least partially a human construction: either a) in the sense of them being an inevitable and essential part of how we make sense of the world by our very nature or b) in the sense of being a semi-arbitrary and contingent system that we’ve adopted in order to navigate the world

There are at least three possibilities. David Lewis level realism, where counterfactual worlds seem fully real to their inhabitants, is an extreme. Moderate realism abou... (read more)

2Chris_Leong1moRegarding moderate realism, if what happened didn't have to happen, then that implies that other things could have happened (these are counterfactuals). But this raises the question, what are these counterfactuals? You've already rejected Counterfactual Realism which seems to lead towards the two possibilities I suggested: a) Counterfactuals are an inevitable and essential part of how we make sense of the world by our very nature b) Counterfactuals are a semi-arbitrary and contingent system that we've adopted in order to navigate the world (Some combination of the two is another possibility.) Presumably, you don't think moderate realism leads you down this path. Where do you think it leads instead? "Even if you accept the Kantian framework, it involves N>1 basic categories" Interesting point. I'm somewhat skeptical of this, but I wouldn't completely rule it out either. (One thing I think plausible is that there could be a category A reducible to a category B which is then reducible back to A; but this wouldn't avoid the circularity) "Well, that's two examples of circular dependency" - Yes, that's what I said. I guess I'm confused why you're repeating it
$1000 USD prize - Circular Dependency of Counterfactuals

I'm still puzzled by your puzzlement.

You are treating httpss:// as though it still an open, but as far as I can see, all the issues raised were answered in the comments .

COVID Skepticism Isn't About Science

Even aside the sacred/secular dichotomy, you still need to be able set an exchange rate between different things (mental health, economic damage, hospitalisations, deaths) to establish whether lockdowns are nett negative .

Or you could just state your conclusions without showing any workings like the OP did .

5Dumbledore's Army1moDownvoted for strawmanning. You are ignoring all of jaspax’ actual points while raising the most extreme flat-Earth view that’s vaguely related to what he said and pretending that it’s somehow relevant.
The Plan

Philosophers are not of a single mind. Some are reductionists, some are illusionists, and so on.

1Blake H.1moGood - though I'd want to clarify that there are some reductionists who think that there must be a reductive explanation for all natural phenomena, even if some will remain unknowable to us (for practical or theoretical reasons). Other non-reductionists believe that the idea of giving a causal explanation of certain facts is actually confused - it's not that there is no such explanation, it's that the very idea of giving certain kinds of explanation means we don't fully understand the propositions involved. E.g. if someone were to ask why certain mathematical facts are true, hoping for a causal explanation in terms of brain-facts or historical-evolutionary facts, we might wonder whether they understood what math is about.
The Plan

Like, yeah. People can be really impressive, but unless you want to make an explicit case for the contrary, people here still think people are made of parts and there exists some way to go from a large cloud of hydrogen to people.

What's important is that it means coming up with a detailed, step-by-step explanation of how some high level concepts like life, shouldness, and intelligence. Just believing that they are natural is not the required explanation. Believing they are unnatural is not the only reason to disbelieve in the possibility of a reduction... (read more)

1rsaarelm1moSo basically the problem is that we haven't got the explanation yet and can't seem to find it with a philosopher's toolkit? People have figured out a lot of things (electromagnetism, quantum physics, airplanes, semiconductors, DNA, visual cortex neuroscience) by mucking with physical things while having very little idea of them beforehand by just being smart and thinking hard. Seems like figuring out human concepts grounding to physics has a similar blocker, we still don't have good enough neuroscience to do a simulation of how the brain goes from neurons to high-level thoughts (where you could observe a simulated brain-critter doing human-like things in a VR environment to tell you're getting somewhere even when you haven't reverse-engineered the semantics of the opaque processes yet). People having that kind of model to look at and trying to make sense of it could come up with all sorts of new unobvious useful concepts, just like people trying to figure out quantum mechanics came up with all sorts of new unobvious useful concepts. But this doesn't sound like a fun project for professional philosophers, a research project like that would need many neuroscientists and computer scientists and not very many philosophers. So if philosophers show up, look at a project like that, and go "this is stupid and you are stupid, go read more philosophy", I'm not sure they're doing it out of purely dispassionate pursuit of wisdom.
The Plan

Why was this downvoted? Sheesh!

What makes for a good "argument"? (Request for thoughts and comments)

Good arguments as in persuausive arguments address the concerns of the other party, use the definitions of the other party, and so on.

Morality and constrained maximization, part 1

The selfish jerks would not be rational, as they wouldn’t be winning

They would be doing as well as you can if you refuse to cooperate. More importantly, co operation isn't a Pareto improvement on defection, because some cooperators take a hit, because not everything is a PD.

But in others, winning may involve doing evil to your neighbours. It would be nice if the morally best action (“What is hateful to you, do not do to your neighbour”) were always to be the selfishly winningest one,

It isn't. It's not a Pareto improvement.

Playing co-operatively can "grow the pie" or produce more overall value even if generating individual losers

Morality and constrained maximization, part 1

Your morality is which part of your goals? If there is no criterion distinguishing moral goals from non moral ones , then a society of selfish jerks who always defect would be 100% moral. But if morality is related to unselfish, cooperative behaviour, as most people believe, game theory is potentially relevant.

(There's a posting where Yudkowsky kind-of-sort argues for the morality-is-goals theory, and a subsequent one where he notices the problem and starts talking about non-jerkish values).

2Richard_Kennaway1moThe selfish jerks would not be rational, as they wouldn't be winning. That's what the game theory is about. The game theory is independent of morality. In some such games, winning happens to involve being good to your neighbours. But in others, winning may involve doing evil to your neighbours. It would be nice if the morally best action ("What is hateful to you, do not do to your neighbour") were always to be the selfishly winningest one, but while the examples have this property, I do not think it has been established in general.
Chapter 8: Positive Bias

It's an older spelling , and AFAIK was pronounced "hiccup", hence the newer spelling. Don't gaol me for saying that! I could shew you some examples, IYL

2Richard_Kennaway1moAs a Brit, I can confirm what TAG says. The word is here pronounced "hiccup" and spelled either like that or as "hiccough". I think "hiccough" is older, and I'm not sure how often it is currently used. Do Americans write "hiccough" and pronounce it "hickoff"?
Some abstract, non-technical reasons to be non-maximally-pessimistic about AI alignment

the problems MIRI has actually already solved are problems that various researchers in the field have told us are impossible for anyone to solve even in principle

And they are..?

Logical induction, Löbian cooperation, reflection in HOL, and functional decision theory are all results where researchers have expressed surprise to MIRI that the results were achievable even in principle.

I think a common culprit is people misunderstanding Gödel's theorems as blocking more things than they actually do. There's also field-specific folklore — e.g., a lot of traditional academic decision theorists seem to have somehow acquired the belief that you can't assign probabilities to your own actions, on pain of paradox.

Where do selfish values come from?

There doesn't seem to be a right answer to counterfactual mugging. Is it the only objection to #1?

2Wei_Dai2moHere's a related objection that may be easier to see as a valid objection than counterfactual mugging: Suppose you're about to be copied, then one of the copies will be given a choice, "A) 1 unit of pleasure to me, or B) 2 units of pleasure to the other copy." An egoist (with perception-determined utility function) before being copied would prefer that their future self/copy choose B, but if that future self/copy is an egoist (with perception-determined utility function) it would choose A instead. So before being copied, the egoist would want to self-modify to become some other kind of agent.
Where do selfish values come from?

By “selfish” I mean how each human (apparently) cares about himself more than others, which needs an explanation because there can’t be a description of himself embedded in his brain at birth.

It's hardly a mystery: you're only plumbed into your own nervous system , so you only feel your own pleasures and pains. That creates a tendency to be only concerned about them as well. It's more mysterious that you would be concerned about pains you can't feel

2Wei_Dai2moYes, I gave this explanation as #1 in the list in the OP, however as I tried to explain in the rest of the post, this explanation leads to other problems (that I don't know how to solve).
Frequent arguments about alignment

It seems to me that the distinction between “alignment” and “misalignment” has become something of a motte and bailey. Historical arguments that AIs would be misaligned used it in sense 1: “AIs having sufficiently general and large-scale motivations that they acquire the instrumental goal of killing all humans (or equivalently bad behaviour)”. Now people are using the word in sense 2: “AIs not quite doing what we want them to do”.

There's an identical problem with "friendliness". Sometimes unfriendliness means we all die, sometimes it means we don't get utopia.

Solve Corrigibility Week

On the last one of your three examples, I feel that ‘mesa optimizers’ is another regrettable example of the forces of linguistic entropy overwhelming any attempts at developing crisply stated definitions which are then accepted and leveraged by the entire community. It is not like the people posting on this site are incapable of using the tools needed to crisply define things, the problem is that many do not seem very interested in ever using other people’s definitions or models as a frame of reference. They’d rather free-associate on the term, and then d

... (read more)
Biology-Inspired AGI Timelines: The Trick That Never Works

There's no strong reason to think the brain does everything with a single algorithm.

2Pattern2moDoes this extend to 'a bunch of algorithms together'? (I.e. how does 'the brain does not do everything with a single algorithm' effect optimality?)
Omicron Variant Post #1: We’re F***ed, It’s Never Over

If the value is in saving lives, then the two are not orthogonal.

How do Bayesians tell what does and doesn't count as evidence (which, e.g., hypotheses may render more or less probable if true)? Is it possible for something to fuzzily-be evidence?

It's not just a case of any two agents having fuzzy approximations to the same world view. In the least convenient case, agents will start off with radically different beliefs, and those beliefs will affect what they consider to be evidence, and how they interpret evidence. So there is no reason for agents to ever converge in the least convenient case .

Aumann's theorem assumes the most convenient case

1JBlack2moAumann's theorem assumes rational agents. Such agents consider every observation to be evidence, and update the probability of every hypothesis in the distribution appropriately. That includes agents who start with radically different beliefs, because for rational agents "belief" is just a distribution over possible hypotheses. The problem is that each hypothesis is a massively multidimensional model, and no real person can even properly fit one in their mind. There is no hope whatsoever that anyone can accurately update weightings over an enormous number of hypotheses on every observation. So we live in an even less convenient world than the "least convenient case" that was proposed. Nobody in the real world is rational in the sense of Aumann's theorem. Not even a superintelligent AGI ever will be, because the space of all possible hypotheses about the world is always enormously more complex than the actual world, and the actual world is more complex than any given agent in it.
Watching Myself Program

Former boss: you don't need comments, you can just look at the code and see what it does..

Me: you need comments to tell you why it does it...

Seeking Truth Too Hard Can Keep You from Winning

What is truth? Rather than get into philosophical debates about this one, let’s use a reasonable working definition that by truth we mean “accurate predictions about our experiences

I would have though that definition was less impacted by the PotC than most. You can check directly that predictive theory is predicting, do you don't need apriori correctness.

This means that you’re going to need some skill at reasoning about non-truth-seeking agents. But the human brain is kinda bad at thinking about minds not like our own. A highly effective way to overco

... (read more)
The Rationalists of the 1950s (and before) also called themselves “Rationalists”

the “critical rationalists” (who are a contemporary movement that involves David Deutsch, the “taking children seriously” people, and some larger set of folks who try to practice a certain set of motions and are based out of the UK, I think)?

Critical rationalism is basically the scientific philosophy of Karl R. Popper. An Austrian, he relocated to the UK in the 30s for similar reasons to Sigmund Freud's. So CR ended as being a kind of UK thing, despite having its roots in the Vienna Circle. (It also has a following in Oz and NZ, but not so much in the s... (read more)

Omicron Variant Post #1: We’re F***ed, It’s Never Over

If you want to determine the balance of evidence, by all means do so, but you can't do that by completely disregarding the alternative explanation.

There was a time when thus place was all about the avoidance of bias.

2ChristianKl2moWhether or not something provides valuable information is orthogonal from it being dangerous. The two aren't alternatives. Something can be valuable&dangerous.
Why Study Physics?

You are responding as though I said something like "physics doesn't work at all", when I actually said it works via idealisations and approximations. To talk of Effective Field Theories concedes my point, since EFTs are by definition approximations .

1Adam Scherlis2moYou said "extremely simplified and idealised situations ... frictionless planes, free fall in a vacuum, and so on". That's a pretty different ballpark than, say, every phenomenon any human before the 1990s had any knowledge of, in more detail than you can see under any microscope (except gravity). Do you consider everything you've experienced in your entire life to have happened in "extremely simplified and idealised situations"?
Load More