I've thought about this, and as far as I can tell, the big step which will greatly increase algorithmic autonomy is delivery.

For instance, currently we have automated factories, which can even run with the lights off, that the article appears to be getting to the point that: "Typically, workers are necessary to set up tombstones holding parts to be manufactured, and to remove the completed parts" It seems safe to refer to this as delivery, because it seems similar to delivery (Drop off this, pick up that, move it to another spot.) so if you were to automate this, that would allow you to further cut the number of humans needed in the process. http://en.wikipedia.org/wiki/Lights_out_%28manufacturing%29

We are going to have automated vehicles soon, if Google's research and states passing legislation is any indication. This is also an important step in delivering things. http://en.wikipedia.org/wiki/Autonomous_car

So, if we can automate delivery, we could have:

Raw materials are mined, and then automatically delivered to an automated factory which automatically assembles them into a finished product which is automatically delivered to you when you place an order through an automated online system. That's a pretty massive amount of autonomy.

Thinking about it, it seems likely that in the future, automating delivery would cause a huge surge in algorithmic autonomy. At the moment, there are a bunch of separate automatic systems, and all sorts of Humans deliver things from one to the other.

You might have things like:

1. A Hay Trucker drives to a farm and delivers Hay to the Farms Hay Storage.

2. The Farmer delivers Hay to Cows.

3. Cow Milk is delivered to a cheese Manufacturer who manufactures cheese.

4. Cheese is delivered to the warehouse of a Grocery store by another trucker.

5. And is then delivered to the front of the store and to restaurant owners by store stockers/cashiers.

6. Restaurant owners put the cheese into their automatic pizza machine to have it make pizza,

7. Which is then delivered to customers, who eat pizza.

The reason that Pizza required so much human effort seemed to be primarily because humans are delivering things from one automated system (say, an automated milker) to the next automated system(a cheese making machine), to the next automated system(an automatic pizza making machine) to the customers. If you could automate that type of step in general, then you'd have a huge surge in algorithmic autonomy.

I don't want to imply that delivering things is easy or hard to figure out. But it's automation would by far cause the great upward shift in algorithm autonomy that I can think of. And without it's automation, algorithms wouldn't be nearly as autonomous at all, because humans would have to keep picking up things at one end and moving them to the beginning of the next automatic system.

Is this the kind of focus you were thinking of?

The question here is about goal-oriented systems that have done things that the creators did not expect and maybe does not want, but nonetheless which help achieve the goal as originally defined.

I'm not sure autonomy is the right word in reference to an algorithm, as the laws for the algorithm, so to speak, have been set down by another agent, the programmer. Which means the real question from a policy perspective may be what level of autonomy should you grant to the programmers.

The idea that there is a human to balance software has a weird opposite; I frequently see programs given "autonomous" access to data that the programmer isn't permitted to touch. If you wouldn't want programmers making life-and-death decisions, you don't want them codifying those decision-making processes, either.

I suspect the fact that programmers are making life and death decisions might be more alarming to many people than the idea that programs are making life and death decisions. For those who aren't alarmed by that possibility yet, a simple phrase may instill alarm: "These are the same kinds of people who programmed Outlook."