And these are just the opportunities we can take advantage of with existing tech, but the optimizer can also help design new tech.

We can do lots of useful things, sure (this is not a point where we disagree), but they don't add up towards "saving the world". These are just short-term benefits. Technological progress makes it easier to screw stuff up irrecoverably, advanced tech is the enemy. One shouldn't generally advance the tech if distant end-of-the-world is considered important as compared to immediate benefits (this value judgment can well be a real point of disagreement).

I agree with Nesov's response, and would be interested to know if you've changed your mind since writing this comment.

Modeling physical systems is already hard. I don't think we could yet write down the dynamics of the physical systems well enough (or rather, we don't understand what the most important characteristics are) to come up with a precise formulation of the major problems in synthetic biology or nanotechnology. I certainly concede that an optimizer would be helpful in solving many subproblems, and would considerably increase the speed of new developments in pretty much every field. I don't think it solves many problems on its own though.

But even if you could solve narrow existing technological problems or develop new technologies at a steady pace, it seems like you should be able to do more. Suppose the box can do in a minute what takes existing humans a million years. Then our only upper bound on our capabilities using the box is whatever we expect of a million years of progress at the current pace. I don't know about you, but I expect pretty much everything.

There are a few (isomorphic) ways to formulate questions as UTMs. You could provide a description of how to interpret final states as a scoring measure, and ask for the highest-scoring input. Or you could provide a UTM that you know halts and ask what part of its final state will be. These are equally powerful, given a genie with infinite computational resources, since there are trivial wrapper programs to convert between them; but the former degrades better under resource constraints.

Turing machines might not be the best way to think about it, since we don't usually talk about retrieving outputs (other than halts/doesn't halt) from them, although there's no reason why we can't in principle. Perhaps a better alternative to talk about reducing to would be a simple recursive language like lambda calculus. But it's moot anyways, since we wouldn't actually perform the reduction, just confirm that it exists and doesn't involve any intractable steps like simulating a human that we haven't uploaded.

One approach---in the post---is to interpret a program as a scoring measure, and the "question" is to find an input that scores well.

I don't really care if you use that particular UTM, or some other UTM, or some other programming language, or a tractable mathematical definition, because they are all the same modulo the constant effort required to translate (which seems to be within human reach, if only barely, at this point).

Even babelfish gets that one right.

Missing the point. The sentence's nature changes when translated. Once it is in Japanese the sentence doesn't make any sense. So something is lost in the translation.

Incidentally, Hofstadter ran into precisely this issue when the French edition of GEB was being written. In GEB he uses the version "This sentence would be very difficult to translate into French." The problem then became what to do in the French version. They made instead a French sentence which declares itself to be difficult to translate into English.