Thanks a lot for writing these! I've really enjoyed these posts and am actually quite worried they won't get the exposure they deserve because you posted all of them at once.
My model says that it's a better idea to space these posts out over a few days to give people time to think, and so that people won't feel like they have to read all the stuff at once, and to make it less likely your posts will get lost in the competition on the frontpage (today in particular is a day in which we have a ton of really great posts, and yours will probably disappear quite quickly).
Bumped back to the top of the frontpage, since I think these probably didn't get the attention they deserve. Also promoted to the frontpage.
This is the first in a three part sequence, examining the scenario in which software development organisations might knowingly take a significant risk of unintentionally launching an AI program into inadequately supervised or controlled self-improvement, by comparing the scenario to the 'press your luck' game mechanic.
This first part explains what that mechanic is.
Many board games bear similarities to each other. The BoardGameGeek website lists some of these common themes. It describes the 'Press your luck' mechanic thus:
Games where you repeat an action (or part of an action) until you decide to stop > due to increased (or not) risk of losing points or your turn. Double or Quits, Keep going or stop, cash your gains or bet them. The idea is not new.
Six players sit down at a table to play a game.
There are six turns, and at the end of the game, each player calculates their total score by adding up the score they got for each turn. The aim is get higher scores than as many other players as possible.
During a turn, a standard die is rolled, once every 20 seconds, in the center of the table where everyone can see it. Between rolls, players can choose to declare that they are banking, in which case their score for the turn is the sum of the rolls so far. But if the die turns up "6", the players still in go bust, and score nothing. When everyone has banked or gone bust, a bonus of 10 is added to the score of the person who banked the most (if there's a tie, nobody gets the bonus).
If there were infinite turns, and the aim were just to make as much as possible per turn, the strategy would be simple. But with this version, how long it makes sense to stay in depends upon how much others made in previous rounds, and whether you are ahead or behind them. This competitive mechanic, were people are tempted to take higher risks in order to beat other people, is known as "Press your luck".