AlphaGo versus Lee Sedol

by gjm1 min read9th Mar 2016183 comments


Personal Blog

There have been a couple of brief discussions of this in the Open Thread, but it seems likely to generate more so here's a place for it.

The original paper in Nature about AlphaGo.

Google Asia Pacific blog, where results will be posted. DeepMind's YouTube channel, where the games are being live-streamed.

Discussion on Hacker News after AlphaGo's win of the first game.

183 comments, sorted by Highlighting new comments since Today at 2:37 PM
New Comment
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

'Yeah, we could maybe have AlphaGo learn everything totally from scratch and reach a superhuman level of knowledge just by playing itself, not using any human games for training material. Of course, reinventing everything that humanity has figured out while playing Go for the last 2,500 years, that's going to take quite a bit of time. Like a few months or so.'

Actually, the AlphaGo algorithm, this is something we’re going to try in the next few months — we think we could get rid of the supervised learning starting point and just do it completely from self-play, literally starting from nothing. It’d take longer, because the trial and error when you’re playing randomly would take longer to train, maybe a few months. But we think it’s possible to ground it all the way to pure learning.

We accidentally had a meetup as the game was ending. For the first time in my life - got to walk in to a room and say; "Who's been watching the big game". It was great, and then about 10mins later the resignation happened. was pretty exciting!

When I started hearing about the latest wave of results from neural networks, I thought to myself that Eliezer was probably wrong to bet against them. Should MIRI rethink its approach to friendliness?

Compared to its competition in the AGI race, MIRI was always going to be disadvantaged by both lack of resources and the need to choose an AI design that can predictably be made Friendly as opposed to optimizing mainly for capability. For this reason, I was against MIRI (or rather the Singularity Institute as it was known back then) going into AI research at all, as opposed to pursuing some other way of pushing for a positive Singularity.

In any case, what other approaches to Friendliness would you like MIRI to consider? The only other approach that I'm aware of that's somewhat developed is Paul Christiano's current approach (see for example, which I understand is meant to be largely agnostic about the underlying AI technology. Personally I'm pretty skeptical but then I may be overly skeptical about everything. What are your thoughts? I don't recall seeing you having commented on them much.

Are you aware of any other ideas that MIRI should be considering?

9paulfchristiano5yDo you have a concise explanation of skepticism about the overall approach, e.g. a statement of the difficulty or difficulties you think will be hardest to overcome by this route? Or is your view more like "most things don't work, and there isn't much reason to think this would work"? In discussion you most often push on the difficulty of doing reflection / philosophy. Would you say this is your main concern? My take has been that we just need to meet the lower bar of "wants to defer to human views about philosophy, and has a rough understanding of how humans want to reflect and want to manage their uncertainty in the interim." Regarding philosophy/metaphilosophy, is it fair to describe your concern as one of: 1. The approach I am pursuing can't realistically meet even my lower bar, 2. Meeting my lower bar won't suffice for converging to correct philosophical views, 3. Our lack of philosophical understanding will cause problems soon in subjective time (we seem to have some disagreement here, but I don't feel like adopting your view would change my outlook substantially), or 4. AI systems will be much better at helping humans solve technical than philosophical problems, driving a potentially long-lasting (in subjective time) wedge between our technical and philosophical capability, even if ultimately we would end up at the right place? My hope is that thinking and talking more about bootstrapping procedures would go a long way to resolving the disagreements between us (either leaving you more optimistic or me more pessimistic). I think this is most plausible if #1 is the main disagreement. If our disagreement is somewhere else, it may be worth also spending some time focusing somewhere else. Or it may be necessary to better define my lower bar in order to tell where the disagreement is.

It seems to be a combination of all of these.

  1. Training an AI to defer to one's eventual philosophical judgments and interim method of managing uncertainty (and not falling prey to marketing worlds and incorrect but persuasive philosophical arguments etc) seems really hard, and made harder by the recursive structure in ALBA and the fact that the first level AI is sub-human in capacity which then has to handle being bootstrapped and training the next level AI. What percent of humans can accomplish this task, do you think? (I'd argue that the answer is likely zero, but certainly very small.) How do the rest use your AI?
  2. Assuming that deferring to humans on philosophy and managing uncertainty is feasible but costly, how many people could resist dropping this feature and the associated cost, in favor of adopting some sort of straightforward utility maximization framework with a fixed utility function that they think captures most or all of their values, if that came as a suggestion from the AI with an apparently persuasive argument? If most people do this and only a few don't (and those few are also disadvantaged in the competition to capture the cosmic commons due to deciding to carry
... (read more)
2paulfchristiano5yRe 1: For a working scheme, I would expect it to be usable by a significant fraction of humans (say, comparable to the fraction that can learn to write a compiler). That said, I would not expect almost anyone to actually play the role of the overseer, even if a scheme like this one ended up being used widely. An existing analogy would be the human trainers who drive facebook's M (at least in theory, I don't know how that actually plays out). The trainers are responsible for getting M to do what the trainers want, and the user trusts the trainers to do what the user wants. From the user's perspective, this is no different from delegating to the trainers directly, and allowing them to use whatever tools they like. I don't yet see why "defer to human judgments and handle uncertainty in a way that they would endorse" requires evaluating complex philosophical arguments or having a correct understanding of metaphilosophy. If the case is unclear, you can punt it to the actual humans. If I imagine an employee who sucks at philosophy but thinks 100x faster than me, I don't feel like they are going to fail to understand how to defer to me on philosophical questions. I might run into trouble because now it is comparatively much harder to answer philosophical questions, so to save costs I will often have to do things based on rough guesses about my philosophical views. But the damage from using such guesses depends on the importance of having answers to philosophical questions in the short-term. It really feels to me like there are two distinct issues: 1. Philosophical understanding may help us make good decisions in the short term, for example about how to trade off extinction risk vs faster development, or how to prioritize the suffering of non-human animals. So having better philosophical understanding (and machines that can help us build more understanding) is good. 2. Handing off control of civilization to AI systems might permanently distort so

The trainers are responsible for getting M to do what the trainers want, and the user trusts the trainers to do what the user wants.

In that case, there would be severe principle-agent problems, given the disparity between power/intelligence of the trainer/AI systems and the users. If I was someone who couldn't directly control an AI using your scheme, I'd be very concerned about getting uneven trades or having my property expropriated outright by individual AIs or AI conspiracies, or just ignored and left behind in the race to capture the cosmic commons. I would be really tempted to try another AI design that does purport to have the AI serve my interests directly, even if that scheme is not as "safe".

If I imagine an employee who sucks at philosophy but thinks 100x faster than me, I don't feel like they are going to fail to understand how to defer to me on philosophical questions.

If an employee sucks at philosophy, how does he even recognize philosophical problems as problems that he needs to consult you for? Most people have little idea that they should feel confused and uncertain about things like epistemology, decision theory, and ethics. I suppose it might be... (read more)

5paulfchristiano5yA general note: I'm not really taking a stand on the importance of a singleton, and I'm open to the possibility that the only way to achieve a good outcome even in the medium-term is to have very good coordination. A would-be singleton will also need to solve the AI control problem, and I am just as happy to help with that problem as with the version of the AI control problem faced by a whole economy of actors each using their own AI systems. The main way in which this affects my work is that I don't want to count on the formation of a singleton to solve the control problem itself. You could try to work on AI in a way that helps facilitate the formation of a singleton. I don't think that is really helpful, but moreover it again seems like a separate problem from AI control. (Also don't think that e.g. MIRI is doing this with their current research, although they are open to solving AI control in a way that only works if there is a singleton.)
4paulfchristiano5yIn general I think that counterfactual oversight has problems in really low-latency environments [] . I think the most natural way to avoid them is synthesizing training data in advance []. It's not clear whether that proposal will work. If your most powerful learners are strong enough to learn good-enough answers to these kinds of philosophical questions, then you only need to provide philosophical input during training and so synthesizing training data can take off time pressure. If your most powerful AI is not able to learn how to answer these philosophical questions, then the time pressure seems harder to avoid. In that case though, it seems quite hard to avoid the time pressure by any mechanism. (Especially if we are better at learning than we would be at hand-coding an algorithm for philosophical deliberation---if we are better at learning and our learner can't handle philosophy, then we simply aren't going to be able to build an AI that can handle philosophy.)
4paulfchristiano5yI replied to your earlier comment. My overall feeling is still that these are separate problems. We can evaluate a solution to AI control, and we can evaluate philosophical work that improves our understanding of potentially-relevant issues (or metaphilosophical work to automate philosophy). I am both less pessimistic about philosophical errors doing damage, and more optimistic about my scheme's ability to do philosophy, but it's not clear to me that either of those is the real disagreement (since if I imagining caring a lot about philosophy and thinking this scheme didn't help automate philosophy, I would still feel like we were facing two distinct problems).
4paulfchristiano5yIs this your reaction if you imagine delegating your affairs to an employee today? Are you making some claim about the projected increase in the importance of these philosophical decisions? Or do you think that a brilliant employees' lack of metaphilosophical understanding would in fact cause great damage right now? I agree that AI may increase the stakes for philosophical decisions. One of my points is that a natural argument that it might increase the stakes---by forcing us to lock in an answer to philosophical questions---doesn't seem to go through if you pursue this approach to AI control. There might be other arguments that building AI systems force us to lock in important philosophical views, but I am not familiar with those arguments. I agree there may be other ways in which AI systems increase the stakes for philosophical decisions. I like the bargaining example. I hadn't thought about bargaining as competitive advantage before, and instead had just been thinking about the possible upside (so that the cost of philosophical error was bounded by the damage of using a weaker bargaining scheme). I still don't feel like this is a big cost, but it's something I want to think about somewhat more. If you think there are other examples like this that might help move my view. On my current model, these are just facts that increase my estimates for the importance of philosophical work, I don't really see it as relevant to AI control per se. (See the sibling, which is the better place to discuss that.) I don't see cases where a philosophical error causes you to lose control, unless you would have some reason to cede control based on philosophical arguments (e.g. in the bargaining case). Failing that, it seems like there is a philosophically simple, apparently adequate notion of "remaining in control" and I would expect to remain in control at least in that sense.
4paulfchristiano5yAre these worse than the principal-agent problems that exist in any industrialized society? Most humans lack effective control over many important technologies, both in terms of economic productivity and especially military might. (They can't understand the design of a car they use, they can't understand the programs they use, they don't understand what is actually going on with their investments...) It seems like the situation is quite analogous. Moreover, even if we could build AI in a different way, it doesn't seem to do anything to address the problem, since it is equally opaque to an end user who isn't involved in the AI development process. In any case, they are in some sense at the mercy of the AI developer. I guess this is probably the key point---I don't understand the qualitative difference between being at the mercy of the software developer on the one hand, and being at the mercy of the software developer + the engineers who help the software run day-to-day on the other. There is a slightly different set of issues for monitoring/law enforcement/compliance/etc., but it doesn't seem like a huge change. (Probably the rest of this comment is irrelevant.) To talk more concretely about mechanisms in a simple example, you might imagine a handful of companies who provide AI software. The people who use this software are essentially at the mercy of the software providers (since for all they know the software they are using will subvert their interests in arbitrary ways, whether or not there is a human involved in the process). In the most extreme case an AI provider could effectively steal all of their users' wealth. They would presumably then face legal consequences, which are not qualitatively changed by the development of AI if the AI control problem is solved. If anything we expect the legal system and government to better serve human interests. We could talk about monitoring/enforcement/etc., but again I don't see these issues as interestingly different
0ESRogs5yI wonder -- are you also relatively indifferent between a hard and slow takeoff, given sufficient time before the takeoff to develop ai control theory? (One of the reasons a hard takeoff seems scarier to me is that it is more likely to lead to a singleton, with a higher probability of locking in bad values.)
5cousin_it5yAs far as I can tell, Paul's current proposal might still suffer from blackmail, like his earlier proposal which I commented on []. I vaguely remember discussing the problem with you as well. One big lesson for me is that AI research seems to be more incremental and predictable than we thought, and garage FOOM probably isn't the main danger. It might be helpful to study the strengths and weaknesses of modern neural networks and get a feel for their generalization performance. Then we could try to predict which areas will see big gains from neural networks in the next few years, and which parts of Friendliness become easy or hard as a result. Is anyone at MIRI working on that?

Then we could try to predict which areas will see big gains from neural networks in the next few years, and which parts of Friendliness become easy or hard as a result. Is anyone at MIRI working on that?

If they did that, then what? Try to convince NN researchers to attack the parts of Friendliness that look hard? That seems difficult for MIRI to do given where they've invested in building their reputation (i.e., among decision theorists and mathematicians instead of in the ML community). (It would really depend on people trusting their experience and judgment since it's hard to see how much one could offer in the form of either mathematical proof or clearly relevant empirical evidence.) You'd have a better chance if the work was carried out by some other organization. But even if that organization got NN researchers to take its results seriously, what incentives do they have to attack parts of Friendliness that seem especially hard, instead of doing what they've been doing, i.e., racing as fast as they can for the next milestone in capability?

Or is the idea to bet on the off chance that building an FAI with NN turns out to be easy enough that MIRI and like-minded researchers can... (read more)

4Gunnar_Zarncke5yThat may be true but that is hindsight bias. MIRIs (or EYs for that matter) approach to hedge against that being true was nonetheless a very (and maybe given the knowledge at the time only) reasonable approach.
4paulfchristiano5yWhat problem do you have in mind here? I thought that the previous problem was mostly psychological, i.e. that if humans were rational agents then this AI would be roughly as vulnerable to blackmail as its designers. So I thought the issue was the psychological strangeness (and great length) of the weird hypothetical. Here we have no such hypothetical, and the system's behavior only depends on the predicted behavior of humans in the real world. That seems to address the narrow version of your concern. I can see two analogous problems: 1. The learning system may believe itself to be in a simulation, as discussed here []. But that seems to cash out to a straightforward technical problem: we want to ensure that as long as there is some important scenario where the AI behaves strangely, we provide feedback on one such scenario. And this technical problem both (1) looks like it should be soluble in principle, (2) looks pretty tractable right now, and (3) is needed to resolve a whole host of other problems (a bunch of exotic failures, but also a bunch of more prosaic difficulties). 2. The behavior of the system depends on what a human would do across many cases. We can sample from these cases at random, but maybe a small fraction of them are sufficiently surprising that they would lead a human to believe they are in a simulation. We would like to sample these situations with disproportionate probability as well so that the human can won't panic when they find themselves in one, but I haven't talked in any detail about how to do that and it's not obvious whether it is possible. (I do think it's possible.) Did you have in mind 1, 2, or something else?
4cousin_it5yI mostly had in mind 2. Not sure how predicting humans is different from putting humans in hypotheticals. It seems like the same problems could happen.
4paulfchristiano5yI agree that the same problem appears for ALBA. I was originally working with proposals where the improbability of the human's situation was bounded, but the recursive structure can lead to arbitrarily large improbability. I hadn't thought about this explicitly. Predicting humans is different from putting humans in hypotheticals, in the sense that in principle you can actually sample from the situations that cause humans to think they are in a simulation or whatever. For example, suppose the human had access to a button that said "This is weird, I'm probably in a simulation," and suppose that we expected the human to press it in any case where they would start behaving weirdly. Then we could potentially sample from the subset of situations where the human presses the button. And if we manage to do that, then the human isn't right to suspect they are in a simulation (any more than they already should believe they are in a simulation, prior to even building the AI). I agree that you shouldn't expect the current version of the scheme to cope with this problem. And it's certainly not obvious that the approach sketched above can work, there are many important questions. But I do think that there is a clear path to fixing this kind of problem. (Interestingly, this is exactly analogous to what the active learning scheme has to do, in order to get the prediction algorithms to predict well.) ETA: although I hadn't thought about this explicitly in the context of ALBA, I have expected to need some way to overweight "weird" situations in order to stop them from being problematic, ever since here [] .
3Lumifer5yIs MIRI even in the AGI race? It certainly doesn't look like it.
9Kaj_Sotala5yThey're working on figuring out what we want the AGI to do, not building one. (I believe Nate has stated this in previous LW comments.)
0Lumifer5yYes, and the point is that MIRI is pondering the situation at the finish line, but is not running in the race.
5Kaj_Sotala5yA different analogy would be that MIRI is looking at the map and the compass to figure out what's the right way to go, while others are just running in any random direction.
3Lumifer5yNot quite. The others are not running around in random directions, they are all running in a particular direction and MIRI is saying "Hold on, guys, there may be bears and tigers and pits of hell at your destination". Which is all fine, but it still is not running.
5Kaj_Sotala5yStill better than running into all the bears and tigers and getting eaten, particularly if it lets you figure out the correct route eventually.
2Lumifer5yThe question was not what is better, the question was whether MIRI is competing in the AGI race.
9Kaj_Sotala5ySure. I wasn't objecting to the "MIRI isn't competing in the AGI race" point, but to the negative connotations that one might read into your original analogy.
-2[anonymous]5yWhich unfortunately presumes that an AGI would be tasked with doing something and given free reign to do so, a truly naïve and unlikely outcome.
5Kaj_Sotala5yHow does it presume that?
0jacob_cannell5yBriefly skimming Christiano's post, this is actually one of the few/first proposals from someone MIRI related that actually seems to be on the right track (and similar to my own loose plans). Basically it just boils down to learning human utility functions with layers of meta-learning, with generalized RL and IRL.

EY was influenced by E.T. Jaynes, who was really against neural networks, in favor of bayesian networks. He thought NNs were unprincipled and not mathematically elegant, and bayes nets were. I see the same opinions in some of EY's writings, like the one you link. And the general attitude that "non-elegant = bad" is basically MIRI's mission statement.

I don't agree with this at all. I wrote a thing here about how NNs can be elegant, and derived from first principles. But more generally, AI should use whatever works. If that happens to be "scruffy" methods, then so be it.

But more generally, AI should use whatever works. If that happens to be "scruffy" methods, then so be it.

This seems like a bizarre statement if we care about knowable AI safety. Near as I can tell, you just called for the rapid creation of AGI that we can't prove non-genocidal.

0dxu5yI don't believe Houshalter was referring to proving Friendliness (or something along those lines); my impression is that he was talking about implementing an AI, in which case neural networks, while "scruffy", should be considered a legitimate approach. (Of course, the "scruffiness" of NN's could very well affect certain aspects of Friendliness research; my relatively uninformed impression is that it's very difficult to prove results about NN's.)
4turchin5yYes, we need to find the way to make existing AIs safe.
3V_V5yNice post. Anyway, according to some recent works (ref [], ref []), it seems to be possible to directly learn digital circuits from examples using some variant of backproagation. In principle, if you add a circuit size penalty (which may be well the tricky part) this becomes time-bounded maximum a posteriori Solomonoff induction.
3Houshalter5yYes binary neural networks are super interesting because they can be made much more compact in hardware than floating point ops. However there isn't much (theoretical) advantage otherwise. Anything a circuit can do, an NN can do, and vice versa. A circuit size penalty is already a very common technique. It's called weight decay, where the synapses are encouraged to be as close to zero as possible. A synapse of 0 is the same as it not being there, which means the neural net parameters requires less information to specify.
1cousin_it5yAgreed on all points. I suppose the main lesson for us can be summarized by the famous verse: A little learning is a dangerous thing; Drink deep, or taste not the Pierian spring: There shallow draughts intoxicate the brain, And drinking largely sobers us again. The sequences definitely qualify as shallow draughts that intoxicate the brain :-(

I think that MIRI did a mistake than decided not be evolved in actual AI research, but only in AI safety research. In retrospect the nature of this mistake is clear: MIRI was not recognised inside AI community, and its safety recommendations are not connected with actual AI development paths.

It is like a person would decide not to study nuclear physics but only nuclear safety. It even may work until some point, as safety laws are similar in many systems. But he will not be the first who will learn about surprises in new technology.

I think that MIRI did a mistake than decided not be evolved in actual AI research [...] MIRI was not recognised inside AI community

Being involved in actual AI research would have helped with that only if MIRI had been able to do good AI research, and would have been a net win only if MIRI had been able to do good AI research at less cost to their AI safety research than the gain from greater recognition in the AI community (and whatever other benefits doing AI research might have brought).

I think you're probably correct that MIRI would be more effective if it did AI research, but it's not at all obvious.

7turchin5yMaybe it should be some AI research which is relevant to safety, like small self evolving agents, or AI-agent which inspects other agents. It would also generate some profit.
9cousin_it5yAgreed on all points. LW was one handshake away from DeepMind, we interviewed [] Shane Legg and referred to his work many times. But I guess we didn't have the right attitude, maybe still don't. Now is probably a good time to "halt, melt and catch fire" as Eliezer puts it.
2hg005yI'm confused what you would have done with the benefit of hindsight (beyond having folks like Jaan Tallin and Elon Musk who were concerned with AI safety become investors in DeepMind, which was in fact done).
2Larks5yWhat do you mean by "one handshake"?
2James_Miller5yGoogle bought DeepMind for, reportedly [], more than $500 million. Other than possibly Eliezer, MIRI probably doesn't have the capacity to employ people that the market places such a high value on.
6turchin5yEY could have such price if he invested more time in studying neural networks, but not in writing science fiction. Lesswrong is also full of clever minds which probably could be employed in any tiny AI project.
6V_V5yHas he ever demonstrated any ability to produce anything technically valuable?
0turchin5yHe has ability to attract groups of people and write interesting texts. So he could attract good programmers for any task.
8V_V5yHe has the ability to attract self-selected groups of people by writing texts that these people find interesting. He has shown no ability to attract, organize and lead a group of people to solve any significant technical task. The research output of SIAI/SI/MIRI has been relatively limited and most of the interesting stuff came out when he was not at the helm anymore.
2Gunnar_Zarncke5yWhile this may be formally right the question is what it shows (or should show)? Because on the other hand MIRI does have quite some research output as well as impact on AI safety - and that is what they set out for.
3V_V5yMost MIRI research output (papers, in particular the peer-reviewed ones) was produced under the direction of Luke Muehlhauser or Nate Soares. Under the direction of EY the prevalent outputs were the LessWrong sequences and Harry Potter fanfiction. The impact of MIRI research on the work of actual AI researchers and engineers is more difficult to measure, my impression is that it has not been very much so far.
2gjm5yWas Eliezer ever in charge? I thought that during the OB, LW and HP eras his role was something like "Fellow" and other people (e.g., Goertzel, Muelhauser) were in charge.
2Gunnar_Zarncke5yThat looks like judgment from availability bias. How do you think MIRI did go about getting researchers and these better directors? And funding? And all those connections that seem to lead to AI safety being a thing now?
2V_V5yIMHO, AI safety is a thing now because AI is a thing now and when people see AI breakthroughs they tend to think of the Terminator. Anyway, I agree that EY is good at getting funding and publicity (though not necessarily positive publicity), my comment was about his (lack of) proven technical abilities.
2dxu5yUnder that hypothesis, shouldn't AI safety have become a "thing" (by which I assume you mean "gain mainstream recognition") back when Deep Blue beat Kasparov?
2V_V5yIf you look up mainstream news article written back then, you'll notice that people were indeed concerned. Also, maybe it's a coincidence, but The Matrix movie, which has AI uprising as it's main premise, came out two years later. The difference is that in 1997 there weren't AI-risk organizations ready to capitalize on these concerns.
0dxu5yWhich organizations are you referring to, and what sort of capitalization?
3cousin_it5yI'm not saying MIRI should've hired Shane Legg. It was more of a learning opportunity.
4James_Miller5yMIRI will never have a comparative advantage in doing the parts of AI research that the big players think will lead to profitable outcomes.
2Manfred5yThey might indeed have comparative advantages, though not absolute ones.
7Raiden5yNeural networks may very well turn out to be the easiest way to create a general intelligence, but whether they're the easiest way to create a friendly general intelligence is another question altogether.

They may be used to create complex but boring part of the real AI like image recognition. DeepMind is no where near to NN, it combines several architectures. So NNs are like ToolAIs inside large AI system: they do a lot of work but on low level.

3skeptical_lurker5yI may be missing something, but why does this matter? An AI has components, as does the human mind. When reasoning about friendliness, what matters is the goal component. Can't the perception/probability estimate module just be treated as an interchangeable black box, regardless of whether it is a DNN, or MCTS Solomov induction approximation, or Bayes nets or anything else?
2Torchlight_Crimson5yNot necessarily. If the goal component what's to respect human preferences, it will be vital that the perception component isn't going to correctly identify what constitutes a "human".
2skeptical_lurker5yThis doesn't seem like a major problem, or one which is exclusive to friendliness - computers can already recognise pictures of humans, and any AGI is going to have to be able to identify and categorise things.
-2bogus5yWell, not quite. []
0Squark5y"Neural networks" vs. "Not neural networks" is a completely wrong way to look at the problem. For one thing, there are very different algorithms lumped under the title "neural networks". For example Boltzmann machines and feedforward networks are both called "neural networks" but IMO it's more because it's a fashionable name than because of actual similarity in how they work. More importantly, the really significant distinction is making progress by trail and error vs. making progress by theoretical understanding. The goal of AI safety research should be shifting the balance towards the second option, since the second option is much more likely to yield results that are predictable and satisfy provable guarantees. In this context I believe MIRI correctly identified multiple important problems (logical uncertainty, decision theory, naturalized induction, Vingean reflection). I am mildly skeptical about the attempts to attack these problems using formal logic, but the approaches based on complexity theory and statistical learning theory that I'm pursuing seem completely compatible with various machine learning techniques including ANNs.
0turchin5yI have one more thought about it. If we work on AI safety problem, we should find the way to secure exiting AIs, not ideal AIs. As if we work on nuclear energy safety, we it would be easy to secure nuclear reactors than nuclear weapons, but knowing that the weapons will be created, we still need to find the way to make the safe. The world had chosen to develop neural net based AI. So we should think how install safety in it.

I found this interesting: AlphaGo's internal statistics predicted victory with high confidence at about three hours into the game (Lee Sedol resigned at about three and a half hours):

For me, the key moment came when I saw Hassabis passing his iPhone to other Google executives in our VIP room, some three hours into the game. From their smiles, you knew straight away that they were pretty sure they were winning – although the experts providing live public commentary on the match weren’t clear on the matter, and remained confused up to the end of the game just before Lee resigned.

Hassabis’s certainty came from Google’s technical team, who pore over AlphaGo’s evaluation of its position, information that isn’t publicly available. I’d been asking Silver how AlphaGo saw the game going, and he’d already whispered back: “It’s looking good”.

And I realised I had a lump in my throat. From that point on, it was crushing for me to watch Lee’s struggle.

Towards the end of the match, Michael Redmond, an American commentator who is the only westerner to reach the top rank of 9 dan pro, said the game was still “very close”. But Hassabis was frowning and shaking his head – he knew that AlphaGo was d

... (read more)
5Gunnar_Zarncke5yThis could be motivated thinking/speaking though.

I watched the whole of both games played so far. In the first game, Redmond definitely thought that Lee Sedol was winning, and at a point close to the end, he said, "I don't think it's going to be close," and I am fairly confident he meant that Lee Sedol would win by a substantial margin. Likewise, he definitely showed real surprise when the resignation came: even at that point, he expected a human victory.

In the second game, he was more cautious and refused to commit himself, but still seemed to think there were points where Lee Sedol had the advantage. However, in this one he did end up admitting that AlphaGo was winning long before the end came.

In particular, I thought Redmond's handling of the top right corner was striking. He identified it as a potential attack for white several times before white's actual attack, and then afterwards thought that a move was 'big' that AlphaGo ignored; later, on calculation, he realized that it was only (if I recall correctly) a one point move.

It looked to me like an example of the human bias towards the corners and walls, combined with his surprise at some of AlphaGo's moves that made significant changes in the center.

6Vaniver5yThe English commentators seemed to take a significant time to come up with score estimates, to the point where I think they were genuinely uncertain in a way that AlphaGo wasn't. (What would be interesting, for example, would be to look at AlphaGo's estimation of the score of historical tournament games that had commentary and see how well the two track each other.)

Myungwan Kim seemed to be quicker to reach the right conclusion - IIRC, by the time that the fighting in the lower right corner ended, he was pretty sure of AlphaGo winning, to the extent of guessing that a move that lost AlphaGo around 1.5 points down there was because AG would win anyway.

-1ChristianKl5yI think it's more likely that the Go professionals (both the commentator and the Lee) simply score certain patterns a few points differently than AlphaGo did then there being motivated thinking in the sense that the commentator wants Lee to win. has some (non-video) comments on the game, and promises more detailed commentary later.

9p Myungwan Kim's commentary (I much preferred this over the official commentary; he's also commenting tomorrow, so recommend following his stream then, though he might start an hour delayed like he did today).

Fun comment from him: "[AlphaGo] play like a god, a god of Go".

Lee Sedol has just resigned the second game.

0WalterL5yI thought he was ahead for a lot of game two. I wonder if that was true, or if AlphaGo was in control all along.

According to this, Lee Sedol said in the post-game press conference that he didn't think he was ahead at any point in the game.

He, you can't really trust that. He'd have said that (or similar) no matter what. It isn't game commentary, its signalling.

There's a sort of humblebrag attitude that permeates all of Go. Every press conference is the same. Your opponent was very strong, you were fortunate, you have deep respect for your opponent and thank him for the opportunity.

In the game commentary you get the real dish. They stop using names and use "White/Black" to talk about either side. There things are much more honest.

2Vaniver5yI thought it was an very likely AlphaGo victory about an hour in, and nearly certain about two hours in.
0SquirrelInHell5yAnd looking at how he used up his time much sooner, he was more cautious today. He still lost and probably also took a psychological hit, so now my estimate of chances of Lee Sedol winning the whole match went down to ~5%.

Ignoring psychology and just looking at the results:

  1. Delta-function prior at p=1/2 -- i.e., completely ignore the first two games and assume they're equally matched. Lee Sedol wins 12.5% of the time.

  2. Laplace's law of succession gives a point estimate of 1/4 for Lee Sedol's win probability now. That means Lee Sedol wins about 1.6% of the time. [EDITED to add:] Er, no, actually if you're using the rule of succession you should apply it afresh after each game, and then the result is the same as with a uniform prior on [0,1] as in #3 below. Thanks to Unnamed for catching my error.

  3. Uniform-on-[0,1] prior for Lee Sedol's win probability means posterior density is f(p)=3(1-p)^2, which means he wins the match exactly 5% of the time.

  4. I think most people expected it to be pretty close. Take a prior density f(p)=4p(1-p), which favours middling probabilities but not too outrageously; then he wins the match about 7.1% of the time.

So ~5% seems reasonable without bringing psychological factors into it.

Laplace's law of succession gives Lee Sedol a 5% chance of winning the match (and AlphaGo a 50% chance of a 5-0 sweep). It gives him a 1/4 chance of winning game 3, a 2/5 chance of winning game 4 conditional on winning game 3, and a 1/2 chance of winning game 5 conditional on winning games 3&4. It's important to keep updating the probability after each game, because 1/4 is just a point estimate for a distribution of true win probabilities and the cases where he wins game 3 tend to come from the part of the distribution where his true win probability is larger than 1/4. It is not a coincidence that Laplace's law (with updating) gives the same result as #3 - Laplace's law can be derived from assuming a uniform prior.

8gjm5yHmm, I explicitly considered whether using LLS we should update after each new game and decided it was a mistake, but on reflection you're right. (Of course what's really right is to have an actual prior and do Bayesian updates, which is one reason why I didn't consider at greater length and maybe get the right answer :-).) Sorry about that.

For me the most interesting part of this match was the part where one of the DeepMind team confirmed that because AlphaGo optimizes for probability of winning rather than expected score difference, games where it has the advantage will look close. It changes how you should interpret the apparent closeness of a game

Qiaochu Yuan, or him quoting someone.

4TheAltar5yI was worried about something like this after the first game. I wasn't sure if expert Go players could discern the difference between AlphaGo playing slightly better than a 9dan versus playing massively better than a 9dan due to how the AI was set up and how difficult it might be to look at players better than the ones already at the top.
2dxu5yThis appears to be a general property of the Monte Carlo search algorithm, which AlphaGo employs.
1Douglas_Knight5yAlmost all algorithms for almost all games play to win. That isn't anything special about AlphaGo. Maybe there's something special about Go or about this algorithm that makes is harder to assess, but that isn't a reason.

Amazing match. Well worth staying up to 2 AM to watch.

Several things I thought were interesting:

  1. The commentator (on the Deepmind channel) calling out several of AlphaGo's moves as conservative. Essentially, it would play an additional stone to settle or augment some group that he wouldn't necessarily have played around. What I'm curious about is how much this reflects an attempt by AlphaGo to conserve computational resources. "I think move A is a 12 point swing, and move B is a 10 point swing, but move B narrows the search tree for future moves in a way that I think will net me at least 2 more points." (It wouldn't be verbalized like that, since it's not thinking verbally, but you can get this effect naturally from the tree search and position evaluator.)

  2. Both players took a long time to play "obvious" moves. (Typically, by this I mean something like a response to a forced move.) 이 sometimes didn't--there were a handful of moves he played immediately after AlphaGo's move--but I was still surprised by the amount of thought that went into some of the moves. This may be typical for tournament play--I haven't watched any live before this.

  3. AlphaGo's willingness to play aggressively and get involved in big fights with 이, and then not lose. I'm not sure that all the fights developed to AlphaGo's advantage, but evidently enough of them did by enough.

  4. I somewhat regret 이 not playing the game out to the end; it would have been nice to know the actual score. (I'm sure estimates will be available soon, if not already.)

What I'm curious about is how much this reflects an attempt by AlphaGo to conserve computational resources.

If I understand correctly, at least according to the Nature paper, it doesn't explicitly optimize for this. Game-playing software is often perceived as playing "conservatively", this is a general property of minimax search, and in the limit the Nash equilibrium consists of maximally conservative strategies.

but I was still surprised by the amount of thought that went into some of the moves.

Maybe these obvious moves weren't so obvious at that level.

5Error5yI don't know about that level, but I can think of at least one circumstance where I think far longer than would be expected over a forced move. If I've worked out the forced sequence in my head and determined that the opponent doesn't gain anything by it, but they play it anyway, I start thinking "Danger, Danger, they've seen something I haven't and I'd better re-evaluate." Most of the time it's nothing and they just decided to play out the position earlier than I would have. But every so often I discover a flaw in the "forced" defense and have to start scrabbling for an alternative.
7WalterL5yThis is very true in Go. If you are both playing down a sequence of moves without hesitation, anticipating a payoff, one of you is wrong (kind of. It's hard to put in words.) It is always worth making double sure that it isn't you.
3Vaniver5ySure. And I'm pretty low as amateurs go--what I found surprising was that there were ~6 moves where I thought "obviously play X," and 이 immediately played X in half of them and spent 2 minutes to play X in the other half of them. It wasn't clear to me if 이 was precomputing something he would need later, or was worried about something I wasn't, or so on. Most of the time I was thinking something like "well, I would play Y, but I'm pretty unconfident that's the right move" and then 이 or AlphaGo play something that are retrospectively superior to Y, or I was thinking something like "I have only the vaguest sense of what to do in this situation." So I guess I'm pretty well-calibrated, even if my skill isn't that great.
4SquirrelInHell5yIf the search tree is narrowed, it is narrowed for both players, so why would it be a gain?
8Vaniver5yThere may be an asymmetry between successful modes of attack and successful modes of defense--if there's a narrow thread that white can win through, and a thick thread that black can threaten through, then white wins computationally by closing off that tree. But thanks for asking: I was confused somewhat because I was thinking about AI vs. human games, but the AI is trained mostly on human vs. human and AI vs. AI games, neither of which will have the AI vs. human feature. Well, except for bots playing on KGS.
0Vaniver5yAs it turns out, we learned later that Fan Hui started working with Deepmind on AlphaGo after their match, and played a bunch of games against it as it improved. So it did have a number of AI vs. human training games.
4gjm5yI saw some blog comment from someone claiming to be (IIRC) an amateur 3-4 dan -- i.e., good enough to estimate this sort of thing pretty well -- reckoning probably 3.5 or 4.5 points in white's favour. That would be after the komi [] of 7.5 points given to white as compensation for moving second, or so I assume from the half-points in the figure. So that would correspond to black being ahead by 3-4 points before komi.
2ChristianKl5yThat wouldn't have given you the actual score as AlphaGo didn't care to maximize the score in the endgame.
0ChristianKl5yWhich specific moves do you mean?
0Vaniver5yI would have to rewatch the game, since the easily available record [] doesn't have the time it took them to make each move.
0ChristianKl5yNo. 2 points is a lot at that level. If the commentator would think a move cost 2 points he wouldn't call it conversative but he would call it an error. Not playing out every move is more about keeping aji open and not wasting possible ko threads. Unfortunately I don't know how to translate aji into English.
0Vaniver5yI think B actually results in more points overall, which is why it would play it; my curiosity is what fraction is due to direct effects vs. indirect effects. For example, one could imagine the board position evaluation function being different for different timing schemes. If you're playing a blitz game where both players have 10 seconds to play each turn, some positions might move from mildly favoring black to strongly favoring black because white needs to do a bunch of thinking to navigate the game tree successfully.
0ChristianKl5yIt's no blitz game and there plenty of time to think through moves. Just for the record at my prime I used to play Go at around 2 kyu.
0polymathwannabe5yI understand aji as potential for future moves that is currently not too usable but may be after the board configuration has evolved.
0ChristianKl5yIt goes in that direction but moves don't have to be used directly to constrain movements elsewhere on the board. When playing around with there was a similar scenario. It's often possible to run a script to get a higher local maxima. However that made the fold more "rigid". The experienced folders did only run the script to search the local maximas at the end when they manually did everything that could be done. With my usage of Go vocabulary running the script to optimize locally beforehand would also be a case of aji-keshi. Aji is for me a phenomological primitive that I learned while playing Go and that I can use outside of Go but which doesn't have an existing English or German word.
0Vaniver5yThe way I think about aji is something fragile on a ledge--sure, it's safe now, but as things shift around, it may suddenly become unsafe.

I'm quite interested in how many of the methods employed in this AI can be applied to more general strategic problems.

From talking to a friend who did quite a bit of work in machine composition, he was of the opinion that tools for handling strategy tasks like go would also apply strongly to many design tasks like composing good music.

7Houshalter5ySure, you can model music composition as a RL task. The AI composes a song, then predicts how much a human will like it. It then tries to produce songs that are more and more likely to be liked. Another interesting thing that alphago did, was start by predicting what moves a human would make. Then it switched to reinforcement learning. So for a music AI, you would start with one that can predict the next note in a song. Then you switch to RL, and adjust it's predictions so that it is more likely to produce songs humans like, and less likely to produce ones we don't like. However automated composition is something that a lot of people have experimented with before. So far there is nothing that works really well.
8ShardPhoenix5yOne difference is that you can't get feedback as fast when dealing with human judgement rather than win/lose in a game (where AlphaGo can play millions of games against itself).
5Houshalter5yYes it would require a lot of human input. However the AI could learn to predict what humans like, and then use that as it's judge. Trying to produce songs that it predicts humans will like. Then when it tests it on actual humans, it can see if it's predictions were right and improve them. This is also a domain with vast amounts of unsupervised data available. We've created millions of songs, which it can learn from. Out of the space of all possible sounds, we've decided that this tiny subset is pleasing to listen to. There's a lot of information in that.
4gwern5yYou can get fast feedback by reusing existing databases if your RL agent can do off-policy learning. (You can consider this what the supervised pre-learning phase is 'really' doing.) Your agent doesn't have to take an action before it can learn from it. Consider the experience replay buffers. You could imagine a song-writing RL agent which has a huge experience replay buffer which is made just of fragments of songs you grabbed online (say, from the Touhou megatorrent with its 50k tracks).
2Vaniver5yEmily Howell?
6Houshalter5yI was thinking more like these examples: [] [] [] []
2gjm5yI think what Vaniver means is: It seems that Emily Howell works pretty damn well, contrary to your claim that nothing does. (By, so far as I understand, means very different from any sort of neural network.)
20mnus5yI know the conversation here has run its course, but I just wanted to add: whether or not Emily Howell is seen as something that "works really well" as an automated system is probably up for debate. It seems to require quite a bit of input from Cope himself in order to come up with sensible, interesting music. For example, one of the most popular pieces from Emily Howell is this fugue: [] - we really don't know how much influence Cope had in creating this piece of music, because the process of composition was not transparent at all.
6ChristianKl5yI think Deep Mind did focus on building this engine because they belief the methods they develop while doing it could potentially be transfered to other tasks.
0MrMind5yAlphaGo has convolutional neural network, supervised learning, self-generated supervised learning, a mix-up strategy between Monte Carlo rollouts and goal function estimation. All these strategies are apted to go because it is a spatial game with a very well defined strategy function. While I do see CNN and supervised learning well worth of being used for music, it is much more difficult to come up with something that resembles the third step in AlphaGo: generating millions of random 'games' (simphonies) with their own label (good music/bad music) to train an 'intuitive' network.
4gwern5yAdversarial generative networks give you a good objective if you want to take a purely supervised approach.
1ChristianKl5yA spotify like service could be used to label the quality. Alternatively it would also be nice to have music that's trained for specific goals like helping people concentrate while they work or reducing stress.
0Gunnar_Zarncke5yI think the basic method could be applied to a more general engine like that of Zillions of Games []. And having an engine that plays any kind of strategy game well would be astonishing.

Go champion Lee Se-dol strikes back to beat Google's DeepMind AI for first time in forth game 3:1

1philh5yHas anyone from Google commented much on AlphaGo's mistakes here? Why it made the mistake at 79, why it didn't notice until later that it was suddenly losing, and why it started playing so badly when it did notice. (I've seen commentary from people who've played other monte-carlo based bots, but I'm curious whether Google has confirmed them.) I don't think I've seen anyone say this explicitly: I would guess that part of the problem was AG hasn't had much training in "mistakes humans are likely to make". With good play, it could have recovered against Lee, but not against itself, and it didn't know it was playing Lee; somehow, the moves it actually played were ones that would have increased its chances of winning if it was playing itself.
0ChristianKl5yI think the DeepMind folks said that they have to get back to London to analyse the case in detail. I don't think that's a good explanation. There's no way that removing it's own ko threats with moves like P14 and O11 would have increased it's chances if it would have played against itself. It look's a bit like belief propagation to update after missing an important move doesn't really work.
0skeptical_lurker5yI think its policy net was only trained on amateurs, not professionals or self-play, making it a little weak. Normally, I suppose that reading large numbers of game trees compensates, but the odds of Lee making his brilliant move 78 (and one other move, but I can't remember which) were 1/10000, so I think that AG never even analysed the first move of that sequence. In other words: I wonder if Google could publish a sgf showing the most probable lines of play as calculated at each move, as well as the estimated probability of each of Lee's moves? I wonder if the best thing to do would be to train nets on: strong amateur games (lots of games, but perhaps lower quality moves?); pro games (fewer games but higher quality?); and self-play (high quality, but perhaps not entirely human-like?) and then take the average of the three nets? Of course, this triples the GPU cycles needed, but it could perhaps be implemented just for the first few moves in the game tree?
0ChristianKl5yI don't think the issue is that 78 was a human like move. It's just a move that's hard to see both for humans and non-humans.
0philh5yNaively, pruning seems like it would cause a mistake at 77 (allowing the brilliant followup 78), not at 79 (when you can't accidentally prune 78 because it's already on the board). But people have been saying that it made a mistake at 79. I don't recall much detail about AG, but I thought the training it did was to improve the policy net? If the policy net was only trained on amateurs, what was it learning from self-play?
0skeptical_lurker5yOf course, but I can't remember which was the other very low-probability move, so perhaps it was one of the later moves in that sequence? I thought the self-play only trained the value net (because they want it to predict human moves, not its own moves), but I might be remembering incorrectly. Pity that the paper is behind a paywall.
1skeptical_lurker5yAlpha Go seems to play really bad moves when it is loosing - this makes some sense as humans also make overplays out of desperation, but it suggests that Alpha Go would be bad at handicap games, unless they change the algorithum to maximise score instead of win probability.
4WalterL5yNothing "bad" about desperate overplays while losing from Alpha Go's perspective. In the same way that it doesn't care about winning by more than a half point, it doesn't mind making its loss more crushing. Invade every territory. If it doesn't work, you lose by a bit more. Boo hoo. If it works, you might winl I'm very interested in the fact that they coded a "resign" function into it. I wouldn't have expected that.
1ChristianKl5yThat's not what happened. T9 wasn't a desperate overplay. It was just bad. J10 might have made more sense as desperate overplay.

Discussion on FiveThirtyEight about experts discussing consequences from the AlphaGo-Lee Sedol match.

2gjm5yYour link is broken -- you have a parenthesis ( instead of a bracket [ at the start of it.

It's worth noting that the match is under Chinese rules and not the more popular Japanese style rules (Korean rules are also Japanese style). That's because Chinese rules are easier for computers.

It would be interesting to have another match played on Japanese style rules.

I listened to the ending press conference. Interestingly, Demis Hassabis discusses AI ethics twice, saying that development will be largely open-sourced to ensure that AI "is for the many, not just the few." So, this gives the impression that Google AI ethics is more thinking along the lines of 'AI based economy renders many unemployed' rather than 'hard takeoff destroys humanity', or at least that is what they are publicly discussing at this time.

On a lighter note, one reporter asked IIRC "How many versions of alphago are there, and how lon... (read more)

2PipFoweraker5yThe thought intrigued me enough to check with a native Korean speaking friend, and they said that cloning doesn't necessarily translate well and it could have been a question about the size of AlphaGo (in terms of copying it or the datasets) or its reproducability / iterations (i.e. are there v1.01, v1.02's floating around).

It is also interesting to know the size of Alphago.

Wiki says: "The distributed version in October 2015 was using 1,202 CPUs and 176 GPUs (and was developed by teem of 100 scientists). Assuming that it was best GPU on the market in 2015, with power around 1 teraflop, total power of AlphaGO was around 200 teraplop or more. (I would give it 100 Teraflop - 1 Petaflop with 75 probability estimate). I also think that the size of the program is around terabytes, but only conclude it from the number of computers in use.

This could provide us with minimal size... (read more)

9gwern5yI don't think that's true. The distributed system for playing is using multiple copies of the CNN value network so each one can do board evaluation during the MCTS on its own without the performance disaster of sending it over the network to a GPU or something crazy like that, not a single one sharded over two hundred servers (CPU!=computer). Similarly for training: each was training the same network in parallel, not 1/200th of the full NN. (You could train something like AlphaGo on your laptop's GPU, it'd just take like 2 years by their wallclock numbers.) The actual CNN is going to be something like 10MB-1GB, because more than that and you can't fit it on 1 GPU to do training. Reading the paper, it seems to be fairly comparable in size to ImageNet competitors: So 500M would be a reasonable guess if you don't want to work out how many parameters that 13-layer network translates to. Not large at all, and model compression would at least halve that. 200 GPUs is not that expensive. Amazon will rent you 1 GPU at spot for ~$0.2/hour, so <$1k/day.
0turchin5yThanks for clarification. If size is rally 500 MB it could easily be stolen or run away, and in 1к a day seems affordable to dedicating hacker.
2ChristianKl5yDemis said that AlphaGo also works on a single computer. The distributed version has 75% winning chance against the one computer version. The hardware they used seem to be the point where there are dimishing return of adding additional hardware.

Does anyone know the current odds being given of Lee Sedol winning any of the three remaining games against AlphaGo? I'm curious if at this point is AlphaGo likely possible to beat by a human player better than Sedol (assuming there are any) or if we're looking at an AI player that is better than a human can be.

3Vaniver5yThe odds I saw for the second match were about 2:3 favoring AlphaGo; my guess is the odds moving forward will be more like 1:4 favoring AlphaGo (but probably it should be closer to something like 1:9). This is my estimation.
4entirelyuseless5yLee Sedol has now won the fourth game, which makes this very improbable. I still think AlphaGo is better than him, but this basically means that its competence can still be measured on a human scale.
2Vaniver5yAgreed that match 4 was a big surprise under my model (I thought it was about 1:20 favoring AlphaGo).

Does AlphaGo use the history of the 1,000,000 games it played against itself to look up similar situations, or is that game history only used to train weights of the algorithm?

9Gunnar_Zarncke5yAlphaGo doesn't do any lookup of positions.

Lee Sedol isn't at the top of Go ratings. How would Ke Jie fare against AlphaGo? A match against the best human player would be a better test of AlphaGo capabilities.

2Dentin5yHonestly that hundred point difference at the top of the Go ratings isn't really going to matter. At best, it probably means that the top player has a ten percent chance of winning a single game, instead of a two percent chance. I wouldn't be surprised at all to hear that AlphaGo is playing at a rating in excess of four thousand, and could be expected to beat the best human players 99% of the time. Frankly, my gut instinct watching the livestream is that AlphaGo is playing at such a high level that even the players at the top of the rankings are having a hard time identifying it. It must be very frustrating to be in that position - you're supposedly one of the best in the world, and for the first half of the game your opponent makes mostly ok but not great moves including some likely mistakes and weird moves that seem pointless. Then near endgame you've somehow ended up 20 points behind with no hope of victory and you're not even sure how it happened.
4Dentin5yUpdate: given the most recent win by Lee Sodol, my hypothesis above seems much less likely. AlphaGo may only be in the 3600-3800 range.