hold_my_fish

Posts

Sorted by New

Wiki Contributions

Comments

Avoid the abbreviation "FLOPs" – use "FLOP" or "FLOP/s" instead

Not sure what's realistic to do here.

Could write "FLOPS-seconds" maybe, which, although a bit wordy, resembles the common usage of "kilowatt-hours" instead of a more directly joule-based unit.

[Linkpost] Solving Quantitative Reasoning Problems with Language Models

Here's something odd that I noticed in one of the examples in the blogpost (https://ai.googleblog.com/2022/06/minerva-solving-quantitative-reasoning.html).

The question is the one that in part reads "the variance of the first n natural numbers is 10". The model's output states, without any reasoning, that this variance is equal to (n^2 - 1)/12, which is correct. Since no reasoning was used, I think it's safe to assume that the model memorized this formula.

This is not a formula that a random math student would be expected to have memorized. (Anecdotally, I have a mathematics degree and don't know it.) Because of that, I'd expect that a typical (human) solver would need to derive the formula on the spot. It also strikes me as the sort of knowledge that would be unlikely to matter outside a contest, exam, etc.

That all leads me to think that the model might be over-fitting somewhat to contest/exam/etc.-style questions. By that I mean that it might be memorizing facts that are useful when answering such questions but are not useful when doing math more broadly.

To be clear, there are other aspects of the model output, here and in other questions, that seem genuinely impressive in terms of reasoning ability. But the headline accuracy rate might be inflated by memorization.

How much does cybersecurity reduce AI risk?

Regarding the cost, I'd expect the road to AGI to deliver intermediate technologies that reduce the cost of writing provably secure code. In particular, I'd expect Copilot-like code generation systems to stay close to the leading edge of AI technology, if nothing else then because of their potential to deliver massive economic value.

Imagine some future version of Copilot that, in addition to generating code for you, also proves properties of the generated code. There might be reasons to do that beyond security: the requirement to provide specs and proofs in addition to code might make Copilot-like systems more consistent at generating correct programs.

How much does cybersecurity reduce AI risk?

While I can't quantify, I think secure computer systems would help a lot by limiting the options of an AI attempting malicious actions.

Imagine a near-AGI system with uneven capabilities compared to humans. Maybe its GPT-like (natural language interaction) and Copilot-like (code understanding and generation) capabilities pass humans but robotics lags behind. More generally, in virtual domains, especially those involving strings of characters, it's superior, but elsewhere it's inferior. This is all easy to imagine because it's just assuming the relative balance of capabilities remains similar to what it is today.

Such a near-AGI system would presumably be superhuman at cyber-attacking. After all, that plays to its strengths. It'd be great at both finding new vulnerabilities and exploiting known ones. Having impenetrable cyber-defenses would neutralize this advantage.

Could the near-AGI system improve its robotics capabilities to gain an advantage in the physical world too? Probably, but that might take a significant amount of time. Doing things in the physical world is hard. No matter how smart you are, your mental model of the world is a simplification of true physical reality, so you will need to run experiments, which takes time and resources. That's unlike AlphaZero, for example, which can exceed human capabilities quickly because its experiments (self-play games) take place in a perfectly accurate simulation.

One last thing to consider is that provable security has the nice property that you can make progress on it without knowing the nature of the AI you'll be up against. Having robust cyber-defense will help whether AIs turn out to be deep-learning-based or something else entirely. That makes it in some sense a safe bet, even though it obviously can't solve AGI risk on its own.

Can growth continue?

If AGI becomes available then it would replace the labor of AI researchers too. (At least it would if we assume that AGI is cheaper to operate than a human. But that seems almost certain, since humans are very expensive.)

In any case I'm not sure it really makes sense to talk about the productivity of people who aren't employed. I'm considering the economy-wide stat here.

Can growth continue?

Division by zero is undefined and it not guaranteed to correspond to +infinity in all context. In this context there might be a difference whether we are talking about the limit when labor apporaches 0 and when labour is completely absent.

True, and in this context the limiting value when approaching from above is certainly the appropriate interpretation. After all, we're talking about a gradual transition from current use of labor (which is positive) to zero use of labor. If the infinity is still bothersome, imagine somebody is paid to spend 1 second pushing the "start the AGI" button, in which case labor productivity is a gazillion (some enormous finite number) instead of infinity.

If you are benetting from a gift that you don't have to work for at all it not like you "work at infinite efficiency" for it.

You seem to be arguing against the definition of labor productivity here. I think though that I'm using the most common definition. If you consider for example Our World In Data's "productivity per hour worked" graph, it uses essentially the same definition that I'm using.

Can growth continue?

It could also be understood that hitting the AI condition means that human labor productivity becomes 0.

I don't agree with this. Using the formula "labor productivity = output volume / labor input use" (which I grabbed from Wikipedia, which is maybe not the best source, but it seems right to me), if "labor input use" is zero and "output volume" is positive, then "labor productivity" is +infinity.

Can growth continue?

[EDIT: This comment is a bit of a mess, but I haven't figured out yet how to make the reasoning more solid.]

Regarding productivity specifically, it seems relevant that AGI+robotics leads to infinite labor productivity. The reason is that it obsoletes (human) labor, so capital no longer requires any workers to generate output.

Therefore, for there to be any finite limit on labor productivity, it'd need to be the case that AGI+robotics is never developed. That situation, even considered on its own, seems surprising: maybe AGI somehow is physically impossible, or there's some (non-AGI-related) catastrophe leading to permanent collapse of civilization, or even if AGI is physically possible it's somehow not possible to invent it, etc. A lot of those reasons would themselves pose problems for technological progress generally: for example, if a catastrophe prevented inventing AGI, it probably prevented inventing a lot of other advanced technology too.

The AI Countdown Clock

I like the idea of visualizing progress towards a future milestone. However, the presentation as a countdown seems problematic. A countdown implicitly promises that something interesting will happen when the countdown reaches 0, as seen with New Year's countdowns, launch countdowns, countdowns to scheduled events, etc. But, here, because of the uncertainty, it's vanishingly unlikely that anything interesting will happen at the moment the countdown completes. (Not only might it not have happened yet, but it might have happened already!)

I can't think a way to fix this issue. Representing the current uncertainty represented by Metaculus's probability distribution is already hard enough, and the harder thing is to respond to changes in the Metaculus prediction over time. It might be that countdowns inherently only work with numbers known to high levels of precision.

Load More