These are multiple choice questions out of 4 and performance for model sizes ≤7.1B is ~zero, so sure, that's pretty natural, any variation between .4B and 7B is dominated by noise. You can just graph all of the curves to check this.
Noise is so high because there is a minimum competence bar in language understanding before you can start solving complex multi-step problems inside language, and the baseline probability of guessing correctly is significant.
I agree with that example but I don't see the distinction the same way. An optimised measure of that sort is safe primarily because it is within an extremely limited domain without much freedom for there to be a lot of optimality, in some informal sense.
Contrast, capabilities for getting very precise measures of that sort exist in the space of things-you-can-do-in-reality, so there is lots of room for such capabilities to be both benign (an extremely accurate laboratory machine) or dangerous (the shortest program that if executed would have that measurement performed). I wouldn't say that there is an important distinction in it involving an optimizing action—an optimiser—but that the domain is large enough such optimal results within it are dangerous in general.
For instance, the process of optimizing a simple value within a simple domain can be as simple as Newton–Raphson, and that's safe because the domain is sufficiently restricted. Contrast, a sufficiently optimised book ends the world, a widget sufficiently optimised for manufacturability ends the world, a baseball sufficiently optimised for speed ends the world.
While I agree that there are many targets that are harmless if optimised for, like you could have a dumpling optimised to be exactly 1kg in mass, I still see a lot of these outputs being intrinsically dangerous. To me, the key danger of optimal strategies is that they are optimal within a sufficiently broad domain, and the key danger of optimisers is that they produce a lot of optimised outputs.
Yes.
In the long run, if the system successfully bootstrapped itself, I imagine it would start executing some processes with more limited permissions, and do other things to reduce fragility, but those wouldn't come by default.
Appreciate the feedback, I'll see if I can do a pass to clean things up.
I think that's mostly a really good summary. The major distinction I would try to make is that agenthood is primarily a way to actualize power, rather than a source of it.
If you had an agent that wasn't strongly optimized in any sense other than it was an agent, in that it had goals and wanted to solve them, that wouldn't make it dangerous, any more than your dog is dangerous for being an agent. Whereas the converse, if you have something that's strongly optimised in some more generic sense, but wasn't an agent, this still puts you extremely close to a lot of danger. The article was trying to emphasize this by pointing to the most reductive form of agenthood I could see, in that none of the intrinsic power of the resulting system could reasonably be attributed to any intrinsic smartness of the agent component, even if the system was an agent that was powerful.
There is no formal tree structure here. Fundamentally the assumption is that there are a bunch of jobs being scheduled by the outputs of a bunch of model evaluations, which are typically returning code or shell commands. The underlying platform this is executing on is just some computer with access to some bulk computing resource, and those expose controls to close down programs just as a matter of general practicality. Eg. if this was a single fast computer you could just execute a kill command with the process ids, which aren't protected from that by default since everything is running in the same permission space.
There is a tree structure here in the sense that executing some outputs from the model may result in the model being evaluated another number of times and those outputs themselves being executed. Precisely what those nodes are doing is a function of their context, and it can be almost arbitrary given the setup. One thing a model evaluation might want to do, to help illustrate things, is look over a list of running processes and for each process check whether it is running in a sensible and efficient manner, or whether it is doing redundant work.
Absolute government spending numbers on fission research isnt very informative when the government is capable of being arbitrarily inefficient at research. Space flight (SLS/Orion) and fusion (ITER) demonstrate this clearly, with the bulk of government investment largely being unhelpful, despite being large.
I haven't looked into what this spending is on for fission, but I have heard that it is basically impossible to test new designs ever, so I would not expect anybody in the US to be able to demonstrate good progress efficiently even if they were more competently run than those other examples.
I am not 100% sure I did that right but that seems like a more sensible answer.
Eyy, I should trust myself more. Verified on Pile-CC.
(GPT-2/3 BPE)
>>> k = 100000000; k / len(tokenizer(cc[:k])["input_ids"])
4.355680325470372
(T5 sentencepiece)
>>> k = 10000000; k / len(tokenizer(cc[:k])["input_ids"])
4.182535904979476Glad you appreciated it. That is a lot of the appeal of hard science fiction for me, and I agree it's useful to model the world with concrete mechanisms, which really forces you to be able to justify every concrete claim you are making about a scenario.
Which you see comparing the two smaller model's improvements to each other, no?
I don't expect reversion to the mean to be clearly dominant wrt. 7.1→280B because the effect is much smaller than the capability jump there. It's also worth remembering that these smaller model outputs can be arbitrary but not necessarily random; I wouldn't expect one 1000M parameter model's outputs to be fully decorrelated with another 1000M parameter model's outputs even if performance was pure chance.
The PaLM NLU/BigBench numbers do seem to be positively correlated, in contrast, especially when using logits or looking at error rate, as is more reasonable of them given the nonzero performance.
E: Did I say something dumb? I cannot figure out why I've been singularly downvoted here. AFAICT the things I am saying are primarily factual.