All of Peterdjones's Comments + Replies

How to Convince Me That 2 + 2 = 3

Mathematics are so firmly grounded in the physical reality that when observations don't line up with what our math tells us, we must change our understanding of reality, not of math. This is because math is inextricably tied to reality, not because it is separate from it.

On the other hand...

General purpose intelligence: arguing the Orthogonality thesis

I could add: Objective punishments and rewards need objective justification.

General purpose intelligence: arguing the Orthogonality thesis

From my perspective, treating rationality as always instrumental, and never a terminal value is playing around with it's traditional meaning. (And indiscriminately teaching instrumental rationality is like indiscriminately handing out weapons. The traditional idea, going back to st least Plato, is that teaching someone to be rational improves them...changes their values)

The Problem with AIXI

I am aware that humans hav a non zero level of life threatening behaviour. If we wanted it to be lower, we could make it lower, at the expense of various costs. We don't which seems to mean we are happy with the current cost benefit ratio. Arguing, as you have, that the risk of AI self harm can't be reduced to zero doesn't mean we can't hit an actuarial optimum.

It is not clear to me why you think safety training would limit intelligence.

5Rob Bensinger8yYou're cutting up behaviors into two categories, 'safe conduct' and 'unsafe conduct'. I'm making a finer cut, one that identifies systematic reasons some kinds of safe or unsafe behavior occur. If you aren't seeing why it's useful to distinguish 'I dropped an anvil on my head because I'm a Cartesian' from 'I dropped an anvil on my head because I'm a newborn who's never seen dangerous things happen to anyone', consider the more general dichotomy: 'Errors due to biases in prior probability distribution' v. ' Errors due to small or biased data sets'. AIMU is the agent designed to make no mistakes of the latter kind; AIXI is not such an agent, and is only intended to avoid mistakes of the former kind. AIXI is supposed to be a universal standard for induction because it only gets things wrong to the extent its data fails it, not to the extent it started off with a-priori wrong assumptions. My claim is that for a physically implemented AIXI-style agent, AIXI fails in its prior, not just in its lack of data-omniscience. 'You aren't omniscient about the data' is a trivial critique, because we could never build something physically omniscient. ('You aren't drawing conclusions from the data in a computationally efficient manner' is a more serious critique, but one I'm bracketing for present purposes, because AIXI isn't intended to be computationally efficient.) Instead, my main critique is of AIXI's Solomonoff prior. (A subsidiary critique of mine is directed at reinforcement learning, but I won't write more about that in this epistemological setting.) In sum: We should be interested in why the AI is making its mistakes, not just in its aggregate error rate. When we become interested in that, we notice that AIXI makes some mistakes because it's biased, not just because it's ignorant. That matters because (a) we could never fully solve the problem of ignorance, but we might be able to fully solve the problem of bias; and (b) if we build a sufficiently smart self-modifier
The Problem with AIXI

Regarding the anvil problem: you have argued with great thoroughness that one can't perfectly prevent an AIXI from dropping an anvil on its head. However, I can't see the necessity. We would need to get the probability of a dangerously unfriendly SAI as close to zero as possible, because it poses an existential threat. However, a suicidally foolish AIXI is only a waste of money.

Humans have a negative reinforcement channel relating to bodily harm called pain. It isn't perfect, but it's good enough to train most humans to avoid doing suicidal stupid things... (read more)

5Rob Bensinger8yIt's also a waste of time and intellectual resources. I raised this point with Adele [] last month. It's good enough for some purposes, but even in the case of humans it doesn't protect a lot of people from suicidally stupid behavior like 'texting while driving' or 'drinking immoderately' or 'eating cookies []'. To the extent we don't rely on our naturalistic ability to reason abstractly about death, we're dependent on the optimization power (and optimization targets) of evolution. A Cartesian AI would require a lot of ad-hoc supervision and punishment from a human, in the same way young or unreflective humans depend for their survival on an adult supervisor or on innate evolved intelligence. This would limit an AI's ability to outperform humans in adaptive intelligence. Sure. In that scenario, the robot body functions like the robot arm I've used in my examples. Destroying the robot (arm) limits the AI's optimization power without directly damaging its software. AIXI will be unusually bad at figuring out for itself not to destroy its motor or robot, and may make strange predictions about the subsequent effects of its output sequence. If AIXI can't perceive most of its hardware, that exacerbates the problem.
Arguing Orthogonality, published form

An entity that has contradictory beliefs will be a poor instrumental rationalist. It looks like you would need to engineer a distinction between instrumental beliefs and terminal beliefs. While we're on the subject, you might need a firewall to stop an .AI acting on intrinsically motivating ideas, if they exist. In any case, orthogonality is an architecture choice, not an ineluctable fact about minds.

The OT has multiple forms, as Armstrong notes. An OT that says you could make arbitrary combinations of preference and power if you really wanted to, can't p... (read more)

The genie knows, but doesn't care

something previously deemed "impossible"

It's clearly possible for some values of "gatekeeper", since some people fall for 419 scams. The test is a bit meaningless without information about the gatekeepers

The genie knows, but doesn't care

The problem is that I don't see much evidence that Mr. Loosemore is correct. I can quite easily conceive of a superhuman intelligence that was built with the specification of "human pleasure = brain dopamine levels", not least of all because there are people who'd want to be wireheads and there's a massive amount of physiological research showing human pleasure to be caused by dopamine levels.

I don't think Loosemore was addressing deliberately unfriendly AI, and for that matter EY hasn't been either. Both are addressing intentionally friendly... (read more)

The genie knows, but doesn't care

Trying to think this out in terms of levels of smartness alone is very unlikely to be helpful.

0linkhyrule58yWell yes. It is a factor, no more no less. My point is, there is a certain level of general competence after which I would expect convincing someone with an OOC motive to let an IC AI out to be "impossible," as defined below.
The genie knows, but doesn't care

Then solve semantics in a seed.

1Randaly8yYou appear to not understand the Orthogonality Thesis, since you have misstated it. The orthogonality thesis deliberately refers to preferences, not values, because values could also refer to instrumental values [], whereas preferences can only refer to terminal values []. (Obviously, consistency and lack of contradiction are only generally valued instrumentally.) No; if the entity values itself believing contradictions, then it having contradicting beliefs would mean it is a good instrumental rationalist.
The genie knows, but doesn't care

My answer: who knows? We've given it a deliberately vague goal statement (even more vague than the last one), we've given it lots of admittedly contradictory literature, and we've given it plenty of time to self-modify before giving it the goal of self-modifying to be Friendly.

Humans generally manage with those constraints. You seem to be doing something that is kind of the opposite of anthropomorphising -- treatiing an entity that is stipulated as having at least human intelligence as if were as literal and rigid as a non-AI computer.

The genie knows, but doesn't care

Semantcs isn't optional. Nothing could qualify as an AGI,let alone a super one, unless it could hack natural language. So Loosemore architectures don't make anything harder, since semantics has to be solved anyway.

5Rob Bensinger8yIt's a problem of sequence. The superintelligence will be able to solve Semantics-in-General, but at that point if it isn't already safe it will be rather late to start working on safety. Tasking the programmers to work on Semantics-in-General makes things harder if it's a more complex or roundabout way of trying to address Indirect Normativity; most of the work on understanding what English-language sentences mean can be relegated to the SI, provided we've already made it safe to make an SI at all.
The genie knows, but doesn't care

"code in the high-level sentence, and let the AI figure it out."

So it's impossible to directly or indirectly code in the compex thing called semantics, but possible to directly or indirectly code in the compex thing called morality? What? What is your point? You keep talking as if I am suggesting there is someting that can be had for free, without coding. I never even remotely said that.

If the AI is too dumb to understand 'make us happy', then why should we expect it to be smart enough to u

... (read more)
4Eliezer Yudkowsky8yPeterDJones, if you wish to converse further with RobbBB, I ask that you do so on RobbBB's blog rather than here.
5Rob Bensinger8yRead the first section of the article you're commenting on []. Semantics may turn out to be a harder problem than morality, because the problem of morality may turn out to be a subset of the problem of semantics. Coding a machine to know what the word 'Friendliness' means (and to care about 'Friendliness') is just a more indirect way of coding it to be Friendly, and it's not clear why that added indirection should make an already risky or dangerous project easy or safe. What does indirect indirect normativity get us that indirect normativity doesn't?
0Luke_A_Somers8y... and whose fault is that?
0Rob Bensinger8y []
The genie knows, but doesn't care

Goertzel appears to be a respected figuer in the field. Could you point the interested reader to your critique of his work?

2wedrifid8yComments can likely be found on this site from years ago. I don't recall anything particularly in depth or memorable. It's probably better to just look at things that Ben Goertzel says and making one's own judgement. The thinking he expresses is not of the kind that impresses me but other's mileage may vary. I don't begrudge anyone their right to their beauty contests [] but I do observe that whatever it is that is measured by identifying the degree of affiliation with Ben Goertzel is something wildly out of sync with the kind of thing I would consider evidence of credibility.
4Randaly8yGoertzel is also known for approving of people who are uncontroversially cranks. See here. [] It's also known, via his cooperation with MIRI, that a collaboration with him in no way implies his endorsement of another's viewpoints.
5wedrifid8yI agree on this much and am glad that Eliezer was not well received either.
5wedrifid8yRegardless of whether the comments are from Eliezer [] or from some barely known account terrible comments and petty antagonism are not welcome. I don't care one way or the other whether Richard Loosemore stays on the site. However, if he stays and continues to post the same way that he has thus far it would constitute a failure [].
0Rob Bensinger8yYes. If an AI is Friendly at one stage, then it is Friendly at every subsequent stage. This doesn't help make almost-Friendly AIs become genuinely Friendly, though. Yes, but that's stupidity on the part of the human programmer, and/or on the part of the seed AI if we ask it for advice. The superintelligence didn't write its own utility function; the superintelligence may well understand Friendliness perfectly, but that doesn't matter if it hasn't been programmed to rewrite its source code to reflect its best understanding of 'Friendliness'. The seed is not the superintelligence. See: []
The genie knows, but doesn't care

Now, why exactly should we expect the superintelligence that grows out of the seed to value what we really mean by 'pleasure', when all we programmed it to do was X, our probably-failed attempt at summarizing our values?

  • Maybe we didn't do it ithat way. Maybe we did it Loosemore's way, where you code in the high-level sentence, and let the AI figure it out. Maybe that would avoid the problem. Maybe Loosemore has solved FAi much more straightforwardly than EY.

  • Maybe we told it to. Maybe we gave it the low-level expansion of "happy" that we or

... (read more)
3Rob Bensinger8y [] If the AI is too dumb to understand 'make us happy', then why should we expect it to be smart enough to understand 'figure out how to correctly understand "make us happy", and then follow that instruction'? We have to actually code ' correctly understand' into the AI. Otherwise, even when it does have the right understanding, that understanding won't be linked to its utility function. []
The genie knows, but doesn't care

He's achieved about what Ayn Rand achieved, and almost everyone thinks she wasa crank.

5linkhyrule58yBasically this. As Eliezer himself points out, humans aren't terribly rational on average and our judgements of each others' rationality isn't great either. Large amounts of support implies charisma, not intelligence. TDT is closer to what I'm looking for, though it's a ... tad long.
Arguments Against Speciesism

And to be fair, you'd have to give ten or a hundred votes to people with PhD's in political science.

Why Are Individual IQ Differences OK?

It is almost always bad Bayes, or any other kind of reasoning, to make judgements about individuals based on group characeristics, since there is almost always information about them as individuals available which is almost always more reliable.

8gwern8yGroup level information is still useful for shrinkage of estimates and correcting for the always-present unreliability in individual estimates; see for example the long conversation between me and Vaniver on LW somewhere where we work through how you would shrink males and females' SAT scores based on the College Board's published reliability numbers.
Why Are Individual IQ Differences OK?

As evidence for this, he used the fact that right here in America, test scores have gone up over the past several decades. This clearly isn't caused by some genetic change, so the most likely explanation is cultural change.

Is that actually more likely than environmental change?

0Yosarian28yEnvironmental change is certainly possible. For example, the amount of lead the average person gets from the environment has been slowly falling for several decades now. Better pre-natal care, and better education about the effects of consuming even small amounts of alcohol during pregnancy, might also be factors. I think that cultural change is probably the biggest single cause of the Flynn effect, though. Computer use, increased practice taking standardized tests in childhood, ect. Which doesn't necessarily mean that they're smarter, just that they'd better at taking IQ tests.
0zslastman8yIt might be caused by a genetic change. Populations are becoming less inbred. This would be expected to raise IQs, though I don't know what the expected magnitude of change is.
Guardians of Ayn Rand

ith permission from both their spouses, which counts for a lot in my view. If you want to turn that into a "problem", you have to specify that the spouses were unhappy—and then it's still not a matter for outsiders.

I dare say many a guru or cult leader has similar "permission". It often isn't taken to ecuse their actions, because people recognise that such permission can be browbeaten ot of people by someone who seems to them to be an authority figure.

Guardians of Ayn Rand

Rand herself didn't understand emergence (she casting a biologist as the embodiment of scientific corruption, because there is too much complexity in his area of study for any one human brain to be familiar with), and also didn't understand much about cybernetics, etc.

That's hardly the start of it. She opposed relativity and QM, and fence-sat on Evolution.


I don't think "1957" is mcuh of an excuse either, particularly about evolution. For another thing, she never wavered till her death in the 80s. It makes no sense to focus on Bayes, unle... (read more)

Making Rationality General-Interest

Suggestion: teach rationality as an open spirit of enquiry, not as a secular rleigion that will turn you into a clone of Richard Dawkins.

9RomeoStevens9yInstead of down voting maybe we should instead be asking what within the LW community caused Peterdjones to say that?
5wedrifid9yI don't believe you, for most part when there is already a precise term we just use that term already. For most LW jargon it is far more likely that you are confused about the concepts and propose using a wrong term than that there are already precise terms that have the same meaning.
4Vaniver9yFrom the grandparent:

There are already precise terms for most of the concepts LW discusses.

State three examples?

6Viliam_Bur9yI guess that for some LW jargon there already are precise terms, but for other LW jargon there are not. Or sometimes there is a term that means something similar, but can also be misleading. Still, it could be good to reduce the unnecessary jargon. How to do it? Perhaps by making this a separate topic -- find the equivalents to LW jargon, disuss whether they really mean the same thing, and if yes, propose using the traditional expression. What I am saying here is that (1) merely saying "there are precise terms" is not helpful without specific examples, and (2) each term should be discussed, because it may seem like the same thing to one person, but another person may find a difference.
Making Rationality General-Interest

You want to teach philosophy as rationality?

Philosophy includes epistemology, which is kind of important to epistemic ratioanlity.

Philosophy is a toolbox as well as a set of doctrines.

0Lumifer9yVarious philosophies include different approaches to epistemology. Which one do you want to teach? I agree that philosophy can be a toolbox, but so can pretty much any field of human study -- from physics to literary criticism. And here we're talking about teaching rationality, not about the virtues of a comprehensive education.
Making Rationality General-Interest

Or maybe there's a lot of utility in not coming accross geeky and selfish, so they are already being instementally rational.

Seeking examples of people smarter than me who got hung up

Einstein backed local realism and the ensemble interpretation, both of which have been "thrown out".

Seeking examples of people smarter than me who got hung up

It may be correct, but not for the reasons usually given: the non existence of ontological randomness is in now way entailed buy the existence of epistemic inderminism.

low stress employment/ munchkin income thread

As a former boss of mine used to say: "Bloody five o-clocker"

The Robots, AI, and Unemployment Anti-FAQ

. At first, complementary effects dominate, and human wages rise with computer productivity. But eventually substitution can dominate, making wages fall as fast as computer prices now do."

But presumably, productivity would rise as well, increasing the real value of wages of a certain face value.

The Robots, AI, and Unemployment Anti-FAQ

It's plausible we'll never see a city with a high-speed all-robotic all-electric car fleet because the government, after lobbying from various industries, will require human attendants on every car - for safety reasons, of course!

I believe I have alredy pointed out that automatic trains already exist. Putting a human superintendent onto a train with nothing to do except watch it drive itslef would be quite ineffectice, because the job is so boring they are unlikely to concetrate. I believe exisitng driverless trains are monitored by CCTV, which is more effective since the monotirs actually ahve something to do in flicking between channels, and could be appied to driverless cars.

0hairyfigment9yUnless we can harness the power of Desert Bus for Hope []. Sadly, I don't think it applies here.
The Robots, AI, and Unemployment Anti-FAQ

The US educational system is either getting worse at training people to handle new jobs, or getting so much more expensive that people can't afford retraining, for various other reasons. (Plus, we are really stunningly stupid about matching educational supply to labor demand. How completely ridiculous is it to ask high school students to decide what they want to do with the rest of their lives and give them nearly no support in doing so? Support like, say, spending a day apiece watching twenty different jobs and then another week at their top three cho

... (read more)
5AspiringRationalist8yThis the idea of exposing students to actual workplaces. That would provide much better information than full time teachers could.
A case study in fooling oneself

the question of how many worlds can be answered pretty well by ~2 to the power of the average number of decoherence events since the beginning.

Deocherence evernts aren't well defined .. they are always FAPP. That;s the source of the problem.

Problems of the Deutsch-Wallace version of Many Worlds

he Copenhagen interpretation suggests there's a priveliged branch in some way, which is the one we actually perceive. Why should there be? This priveliged branch idea is adding something that we don't need to add.

MWI adds a privileged basis that is also unnecessary.

Many worlds is pretty much that view.

MWI adds a universal quantum state that is not , and cannot be, observed.

If Many-Worlds Had Come First

A very quick but sufficient refutation is that the same math taken as a description of an objectively existing causal process gives us MWI, hence there is no reason to complicate our epistemology beyond this

Or MWI could be said to be complicating the ontology unnecessarily. To be sure, rQM answers epistemologically some questions that MWI answers ontologically, but that isn't obviously a Bad Thing. A realistitc interpretation of the WF is a postive metaphyscial assumption, not some neutral default. A realistic quantum state of the universe is a further assumption that buys problems other interpretations don't have.

Problems of the Deutsch-Wallace version of Many Worlds

The arguments don't apply to interpretations that don't require a real WF or real collapse, and EY has struggled with them,.

Why Many-Worlds Is Not The Rationally Favored Interpretation

The basic wrong assumption being made is that quantum superposition by default equals multiplicity - that because the wavefunction in the double-slit experiment has two branches, one for each slit, there must be two of something there - and that a single-world interpretation has to add an extra postulate to this picture, such as a collapse process which removes one branch. But superposition-as-multiplicity really is just another hypothesis. When you use ordinary probabilities, you are not rationally obligated to believe that every outcome exists somewhere

... (read more)
Why Many-Worlds Is Not The Rationally Favored Interpretation

on a large scale

Which is to say, MWI is what you get if you assume there is a universal state without an observer to observe the state of fix the basis. As it happens, it is possible to reject Universal State AND real collapse.

ESR's New Take on Qualia

But we don't know that qualia aren;t anything and we don't know that about free will either.

Welcome to Less Wrong! (July 2012)

Yep. A morality that leads to the conclusion that we should all work like slaves for a future paradise, the slightest lapse incurring a cost equivalent to untold numbers of dead babies, and the enormity of the task meaning that we shall never experience it ourselves, is prima facie a broken morality.

Welcome to Less Wrong! (July 2012)

It is counterintuitive that you should slave for people you don't know, perhaps because you can't be sure you are serving their needs effectively. Even if that objection is removed by bringing in an omniscient oracle,there still seems to be a problem because the prospect of one generation slaving to create paradise for another isn't fair. the simple version of utilitiarianism being addressed here only sums individual utilities, and us blind to things that can only be defined at the group level like justice and equaliy.

We Change Our Minds Less Often Than We Think

Of course it is unworkable for politicians to stick rigidly to their manifestos. It is also unworkable for them to discard their manifestos on day one.

Load More