All of Houshalter's Comments + Replies

The temporal difference learning algorithm is an efficient way to do reinforcement learning. And probably something like it happens in the human brain. If you are playing a game like chess, it may take a long time to get enough examples of wins and losses, for training an algorithm to predict good moves. Say you play 128 games, that's only 7 bits of information, which is nothing. You have no way of knowing which moves in a game were good and which were bad. You have to assume all moves made during a losing game were bad. Which throws out a lot of informati... (read more)

It's back btw. If it ever goes down again you can probably get it on wayback machine. And yes the /r/bad* subreddits are full of terrible academia snobbery. Badmathematics is the best of the bunch because mathematics is at least kind of objective. So they mostly talk about philosophy of mathematics.

The problem is formal models of probability theory have problems with logical uncertainty. You can't assign a nonzero probability to a false logical statement. All the reasoning about probability theory is around modelling uncertainty in the unkown ext... (read more)

That's unlikely. By the late 19th century there was no stopping the industrial revolution. Without coal maybe it would have slowed down a bit. But science was advancing at a rapid pace, and various other technologies from telephones to electricity were well on their way. It's hard for us to imagine a world without coal, since we took that path. But I don't see why it couldn't be done. There would probably be a lot more investment in hydro and wind power (both of which were a thing before the industrial revolution.) And eventually solar. Cars would be hard, but electric trains aren't inconceivable.

we have nuclear weapons that are likely visible if fired en mass.

Would we be able to detect nuclear weapons detonated light years away? We have trouble detecting detonations on our own planet! And even if we did observe them, how would we recognize it as an alien invasion vs local conflict, or god knows what else.

The time slice between us being able to observe the stars, and post singularity, is incredibly tiny. It's very unlikely two different worlds will overlap so that one world is able to see the other destroyed and rush a singularity. I'm not even sure if we would rush a singularity if we observed aliens, or if it would make any difference.

First of all, the Earth has been around for a very very long time. Even slowly expanding aliens should have hit us by now. The galaxy isn't that big relative to the vast amounts of time they have probably been around. I don't feel like this explains the fermi paradox.

If aliens wanted to prevent us from fleeing, this is a terribly convoluted way of doing it. Just shoot a self replicating nanobot at us near the speed of light, and we would be dealt with. We would never see it coming. They could have done this thousands of years ago, if not millions. And it w... (read more)

Well we have plausible reason to believe in aliens. The copernican principle, that the Earth isn't particularly special and the universe is enormous. There's literally no reason to believe angels and demons are plausible.

And god do I hate skeptics and how they pattern match everything "weird" to religion. Yes aliens are weird. That doesn't mean they have literally the same probability of existing as demons.

I think a concrete example is good for explaining this concept. Imagine you flip a coin and then put your hand over it before looking. The state of the coin is already fixed on one value. There is no probability or randomness involved in the real world now. The uncertainty of it's value is entirely in your head.

From Surely You're Joking Mr. Feynman:

Topology was not at all obvious to the mathematicians. There were all kinds of weird possibilities that were “counterintuitive.” Then I got an idea. I challenged them: "I bet there isn't a single theorem that you can tell me - what the assumptions are and what the theorem is in terms I can understand - where I can't tell you right away whether it's true or false."

It often went like this: They would explain to me, "You've got an orange, OK? Now you cut the orange into a finite number of pieces, put it b

Yudkowsky has changed his views a lot over the last 18 years though. A lot of his earlier writing is extremely optimistic about AI and it's timeline.

This is by far my favorite form of government. It's a great response whenever the discussion of "democracy is the best form of government we have" comes up. Some random notes in no particular order:

Sadly getting support for this in the current day is unlikely because of the huge negative associations with IQ tests. Even literacy tests for voters are illegal because of a terrible history of fake tests being used by poll workers to exclude minorities. (Yes the tests were fake like this one, where all the answers are ambiguous and can be judged as c... (read more)

In the first draft of the lord of the rings, the Balrog ate the hobbits and destroyed middle Earth. Tolkien considered this ending unsatisfactory, if realistic, and wisely decided to revise it.

It's really going to depend on your interests. I guess I'll just dump my favorite channels here.

I enjoy some math channels like Numberphile, computerphile, standupmaths, 3blue1brown, Vi Hart, Mathologer, singingbanana, and some of Vsauce.

For "general interesting random facts" there's Tom Scott, Wendover Productions, CGP Grey, Lindybeige, Shadiversity. and Today I Found Out.

Science/Tech/etc: engineerguy, Kurzgesagt, and C0nc0rdance.

Miscellaneous: kaptainkristian, CaptainDisillusion, and the more recent videos of suckerpinch.

Politics: I unsubscribe... (read more)

Well there is a lot of research into treatments for dementia, like the neurogenesis drug I mentioned above. I think it's quite plausible they will stumble upon general cognitive enhancers that improve healthy people.

Just because it's genetic doesn't mean it's incurable. Some genetic diseases have been cured. I've read of drugs that increase neurogenesis, which could plausibly increase IQ. Scientists have increased the intelligence of mice by replacing their glial cells with better human ones.

I wasn't aware that method had a name, but I've seen that idea suggested before when this topic comes up. For neural networks in particular, you can just look at the gradients of the inputs to see how it's output changes as you change each input.

I think the problem people have, is that just tells you what the machine is doing. Not why. Machine learning can never really offer understanding.

For example, there was a program created specifically for the purpose of training human understandable models. It worked by fitting the simplest possible mathematical exp... (read more)

Same as with the GAN thing. You condition it on producing a correct answer (or whatever the goal is.) So if you are building a question answering AI, you have it model the probability distribution something like P(human types this character | human correctly answers question). This could be done simply by only feeding it examples of correctly answered questions as it's training set. Or you could have it predict what a human might respond if they had n days to think about it.

Though even that may not be necessary. What I had in mind was just having the AI r... (read more)

Consider also that religions that convert more people tend to spread faster and farther than religions that don't. So over time religions should become more virulent.

There are some problems with this analysis. First of all, translation is natural language processing. What task requires more understanding of natural language than translation? Second, the BLEU score mentioned is only a cheap and imperfect measure of translation quality. The best measure is human evaluation. And neural machine translation excels at that. Look at this graph. On all languages, the neural network is closer to human performance than the previous method. And on several languages it's extremely close to human performance, and it's translations ... (read more)

Selecting from a list of predetermined answers extremely limits the AI's ability. Which isn't good if we want it to actually solve very complex problems for us! And that method by itself doesn't make the AI safe, just makes it much harder for it to do anything at all.

Note someone found a way to simplify my original idea in the comments. Instead of using the somewhat complicated GAN thing, you can just have it try to predict the next letter a human would type. In theory these methods are exactly equivalent.

It wasnt until relatively late in the second industrial revolution that coal completely replaced wood. And oil came very late. I think an industrial revolution could happen a second time without fossil fuel.

I like the scenario you presented. 6 months until intelligence explosion changes the entire approach to FAI. More risk is acceptable. More abstract approaches to FAI research seem less useful if they can't lead to tangible algorithms in 6 months.

I think the best strategy would be something like my idea to have AI mimic humans. Then you can task it to do FAI research for you. It could possibly produce years worth of FAI research papers in a fraction of the time. I don't think we should worry too much about the nuances of actually training an FAI directly.

I believe this is the idea of "motivational scaffolding" described in Superintelligence. Make an AI that just learns a model of the world, including what words mean. Then you can describe its utility function in terms of that model - without having to define exactly what the words and concepts mean.

This is much easier said than done. It's "easy" to train an AI to learn a model of the world, but how exactly do you use that model to make a utility function?

No, the choice is to vote for your preferred candidate, or to not vote. Write ins count as "not voting".

Basically yes. First Past The Post does not satisfy the criteria Independence of Irrelevant Alternatives.

If you really believe your candidate is less than 50% likely to be the "correct" candidate, you can just vote for the other one. Then you will necessarily have a >50% confidence you voted for the correct candidate. You can't possibly do worse on a binary decision.

Well the control problem is all about making AIs without "inimical motivations", so that covers the same thing IMO. And fast takeoff is not at all necessary for AI risk. AI is just as dangerous if it takes it's time to grow to superintelligence. I guess it gives us somewhat more time to react, at best.

Are you referring to OP or me? I don't think my estimate of the difference between candidates is ridiculous. It's pretty clear the president can have a massive impact on the world. So large that, even when multiplied by a 1 in 10 million probability, it's still worth your time to vote.

Using dollar amounts might be a bit naive. Instead look at utility directly, perhaps some estimate like QALYs. I think something like health care reform alone has the potential to be worth tens of millions of QALYs. A major war or economic depression can easily cost similar a... (read more)

Yes, from your subjective view your vote is always positive. Thus you should always vote.

Third parties aren't stable. They can appear, but they inevitably split the vote. They always hurt their cause more than help it. Unless they are so popular they can outright replace one of the parties.

False. It requires only a few events, like smarter-than-human AI being invented, and the control problem not being solved. I don't think any of these things is very unlikely.

I can still think the CEV machine is better than whatever the alternative is (for instance, no AI at all.) But yes, in theory, you should prefer to make AIs that have your own values and not bother with CEV.

Having a body is irrelevant. Bodies are just one way to manipulate the world to optimize your goals.

"We convert the resources of the world into the things we want." To some extent, but not infinitely, in a fanatical way. Again, that is the whole worry about AI -- that it might do that fanatically. We don't.

What do you mean by "fanatic... (read more)

In a first past the post election third parties are irrelevant.

More specifically, the calculations above apply to a close election. 538 gives Johnson a less than 0.01% chance of winning. Obviously the probability of you being the tie breaking vote is many many orders of magnitude smaller than is worth calculating.

That's impossible. You can't have less than 50% confidence on a binary decision. You can't literally believe that a coin flip is more likely to select the correct option than you.

...add a primary supergoal which imposes a restriction on the degree to which "instrumental goals" are allowed to supercede the power of other goals. At a stroke, every problem he describes in the paper disappears, with the single addition of a goal that governs the use of instrumental goals -- the system cannot say "If I want to achieve goal X I could do that more efficiently if I boosted my power, so therefore I should boost my power to cosmic levels first, and then get back to goal X."

This is not so simple. "Power" and &... (read more)

I don't see anything in that link that is relevant to this post.

No that's not how this works. We are calculating the expected value of a vote. The expected value is more than just "best case - worst case". You factor in all possibilities weighted by their probability.

As long as the probability you are voting for the correct candidate is higher than 50%, the expected value of a vote is positive. And obviously it's impossible to get odds worse than that, for a binary decision. You can then multiply the probability you are voting for the correct candidate by the expected value of a correct vote, and it's likely ... (read more)

The article gives an upper limit of the expected value of a vote. Even if the lower limit is 2 orders of magnitude lower, it still is quite significant. And makes it worth the time to vote, or to try to convince other people to vote your way.

Scott Alexander estimates the value of a vote more conservatively to be about $300 to $5,000 (the higher number for if you live in a swing state.) He backs this up by pointing to specific policies and actions taken by presidents that cost trillions of dollars.

I don't think it's hard to believe that the president can be... (read more)

I don't see why you do that division. The point of being the decisive vote, is that if you didn't show up to vote, the election would have gone the other way (lets ignore ties for the moment.) You can disregard other people entirely in this model. All that matters is the expected value of your action. Which is enormous.

Well now I see we disagree at a much more fundamental level.

There is nothing inherently sinister about "optimization". Humans are optimizers in a sense, manipulating the world to be more like how we want it to be. We build sophisticated technology and industries that are many steps removed from our various end goals. We dam rivers, and build roads, and convert deserts into sprawling cities. We convert the resources of the world into the things we want. That's just what humans do, that's probably what most intelligent beings do.

The definition of F... (read more)

The CEV process might well be immoral for everyone concerned, since by definition it is compromising a person's fundamental values.

The world we live in is "immoral" in that it's not optimized towards anyone's values. Taking a single person's values would be "immoral" to everyone else. CEV, finding the best possible compromise of values, would be the least immoral option, on average. Optimize the world in a way that dissatisfies the least people the least amount.

That does not necessarily mean "living separately".

Right. I... (read more)

But humans share a lot of values (e.g. wanting to live and not be turned into a dyson sphere.) And a collection of individuals may still have a set of values (see e.g. coherent extrapolated volition.)

It means that when you look an an AI system, you can tell whether it's FAI or not.

Look at it how? Look at it's source code? I argued that we can write source code that will result in FAI, and you could recognize that. Look at the weights of it's "brain"? Probably not, anymore than we can look at human brains and recognize what they do. Look at it's actions? Definitely, FAI is an AI that doesn't destroy the world etc.

I don't see what voting systems have to do with CEV. The "E" part means you don't trust what the real, current humans

No it's not necessarily a negative outcome. I think it could go both ways, which is why I said it was "my greatest fear/hope".

The premise this article starts with is wrong. The argument goes that AIs can't take over the world, because they can't predict things much better than humans can. Or, conversely, that they will be able to take over because they can predict much better than humans.

Well so what if they can predict the future better? That's certainly one possible advantage of AI, but it's far from the only one. My greatest fear/hope of AI is that it will be able to design technology much better than humans. Humans didn't evolve to be engineers or computer programmers. It's r... (read more)

People may have different values (although I think deep down we are very similar, humans sharing all the same brains and not having that much diversity.) Regardless, CEV should find the best possible compromise between our different values. That's literally the whole point.

If there is a difference in our values, the AI will find the compromise that satisfies us the most (or dissatisfies us the least.) There is no alternative, besides not compromising at all and just taking the values of a single random person. From behind the veil of ignorance, the first i... (read more)

No, I'm asking you to specify it. My point is that you can't build X if you can't even recognize X.

And I don't agree with that. I've presented some ideas on how an FAI could be built, and how CEV would work. None of them require "recognizing" FAI. What would it even mean to "recognize" FAI, except to see that it values the kinds of things we value and makes the world better for us.

Learning what humans want is pretty easy. However it's an inconsistent mess which involves many things contemporary people find unsavory. Making it all c