it doesn't do very well at building up a reactive decision tree of strategies (if I scout this, I do that)
In StarCraft terminology, this seems to be completely accurate. But for non-starcraft players, they may get the wrong impression that AlphaStar doesn't scout at all.
AlphaStar does scout, in the sense of seeing whether an attack is coming, and then moving into position to defend against it. It does check to see which units are coming, and then sends the appropriate defense against them. But this is not what the term "scout" is usually used for in StarCraft.
In StarCraft, "scouting" almost always refers to scouting the opponent's buildings, not units. A good player scouts to see whether this or that building exists, and then reacts by doing something that would counter that strategy. So far as I can see, AlphaStar does not do this. (In an hour, Oriol Vinyals and Dario Wünsch will be guests on the Pylon Show, where we may get more details on where AlphaStar is underperforming.) But I do think it is important to clarify that AlphaStar does seem to scout units in the replays I've seen.
It's been two years since AlphaStar used anti-harassment worker oversaturation to great effect, and, as far as I'm aware, not a single progamer has tried it in a GSL match since then. This morning the last GSL match of 2020 marks the 1003rd GSL game since AlphaStar's demonstration, and (to my knowledge) the only games where workers were overmade were when there was a significantly different reason for doing so (being contained, maynarding workers, executing a timing attack). The consensus among SC2 pros appears to be that AlphaStar was wrong to overmake wo... (read more)
DeepMind released their AlphaStar paper a few days ago, having reached Grandmaster level at the partial-information real-time strategy game StarCraft II over the summer.
This is very impressive, and yet less impressive than it sounds. I used to watch a lot of StarCraft II (I stopped interacting with Blizzard recently because of how they rolled over for China), and over the summer there were many breakdowns of AlphaStar games once players figured out how to identify the accounts.
The impressive part is getting reinforcement learning to work at all in such a vast state space- that took breakthroughs beyond what was necessary to solve Go and beat Atari games. AlphaStar had to have a rich enough set of potential concepts (in the sense that e.g. a convolutional net ends up having concepts of different textures) that it could learn a concept like "construct building P" or "attack unit Q" or "stay out of the range of unit R" rather than just "select spot S and enter key T". This is new and worth celebrating.
The overhyped part is that AlphaStar doesn't really do the "strategy" part of real-time strategy. Each race has a few solid builds that it executes at GM level, and the unit control is fantastic, but the replays don't look creative or even especially reactive to opponent strategies.
That's because there's no representation of causal thinking - "if I did X then they could do Y, so I'd better do X' instead". Instead there are many agents evolving together, and if there's an agent evolving to try Y then the agents doing X will be replaced with agents that do X'. But to explore as much as humans do of the game tree of viable strategies, this approach could take an amount of computing resources that not even today's DeepMind could afford.
(This lack of causal reasoning especially shows up in building placement, where the consequences of locating any one building here or there are minor, but the consequences of your overall SimCity are major for how your units and your opponents' units would fare if they attacked you. In one comical case, AlphaStar had surrounded the units it was building with its own factories so that they couldn't get out to reach the rest of the map. Rather than lifting the buildings to let the units out, which is possible for Terran, it destroyed one building and then immediately began rebuilding it before it could move the units out!)
This means that, first, AlphaStar just doesn't have a decent response to strategies that it didn't evolve, and secondly, it doesn't do very well at building up a reactive decision tree of strategies (if I scout this, I do that). The latter kind of play is unfortunately very necessary for playing Zerg at a high level, so the internal meta has just collapsed into one where its Zerg agents predictably rush out early attacks that are easy to defend if expected. This has the flow-through effect that its Terran and Protoss are weaker against human Zerg than against other races, because they've never practiced against a solid Zerg that plays for the late game.
The end result cleaned up against weak players, performed well against good players, but practically never took a game against the top few players. I think that DeepMind realized they'd need another breakthrough to do what they did to Go, and decided to throw in the towel while making it look like they were claiming victory. (Key quote: "Prof Silver said the lab 'may rest at this point', rather than try to get AlphaStar to the level of the very elite players.")
Finally, RL practitioners have known that genuine causal reasoning could never be achieved via known RL architectures- you'd only ever get something that could execute the same policy as an agent that had reasoned that way, via a very expensive process of evolving away from dominated strategies at each step down the tree of move and countermove. It's the biggest known unknown on the way to AGI.