AI #43: Functional Discoveries

[-]ryan_greenblatt2y*102

To what extent setups of this type can in practice preserve nice features, both in alignment and other capabilities, and how much those results will then generalize and survive out of distribution as capabilities of the underlying systems scale higher, is a key question. If we can get nice enough properties, we can do various forms of amplification, and the sky is the limit. I am deeply skeptical we can get such properties where it matters. Some others are more hopeful.

A key hope I have for this type of research is that we can test our techniques on the actual powerful models we're worried about, using domains very similar to the ones we care about. This can be done as long as we can find very similar domains (including similar in difficulty) where we happen to have ground truth (or some other held-out signal for validation). For instance, perhaps we can use string theory as a testbed for theoretical alignment work: we can potentially use held-out string theory experts to see if we can reproduce important results using our techniques as an analogy for alignment research. In practice, I'm reasonably optimistic about finding such domains which are quite similar up through moderately superhuman models.

(Note that my perspective here might be pretty different than the perspective of the OpenAI people and that I don't work there.)

Alternatively, if we're worried about measurement tampering type concerns, we can potentially hold out some sources of validation (additional measurements) on our actual domain of interest for testing.

These sorts of approaches are actually just cases of sandwiching, and we discuss this type of evaluation in more detail here. Also note that we can use the exact same testbeds to test scalable oversight techniques and generalization techniques (this link is to the same post as linked earlier, but not a specific section).

Thus, our final technique testing doesn't need to generalize across model capabilities, just to generalize across (hopefully) very similar domains. And we should be able to use these similar domains to find hyperparameters and otherwise make relatively fine-grained choices. We can even potentially try to mostly train our model on domains where we have ground truth and then generalize to the domains we actually care about (though there are some complications with full training as we'll discuss below). We'll need to test for overfitting by using cross-validation across multiple such domains. (Cross-validation seems particularly important for very parameter-heavy approaches like training the entire model.)

This still leaves us with some questions:

Q: When can we reasonably expect technique testing to generalize across very similar domains given that cross-validation across such domains looks good?

A: I think technique testing probably will work well unless our model is intentionally trying to sabotage our evaluations (perhaps due to scheming/deceptive alignment). Models likely will be able to know what sorts of held-out signals we might have on various domains and thus scheming models could sabotage our evaluations. So, I don't expect these sorts of methods to work for scheming AIs, but otherwise, they might work quite well. (We still need to ensure we get a large enough sample size of domains such that cross-validation is low variance, etc.) To handle scheming AIs, we'll probably need a pretty different approach at least for evaluation. (I'm optimistic about AI control here.)

Q: Same question as before, but what about actually fully training the model on some domains and then generalizing to the domain we care about?

A: I get somewhat concerned that even non-scheming models might naturally learn to predict "what would humans think the answer is based on all of the held-out validation they have" and then this will look like it generalizes well according to cross-validation while it actually fails. This seems especially likely if there are flaws in the ground truth labeling process which are predictable to the model. In theory, this concern could also apply to technique testing if "what would humans think the answer is based on the held-out validation they have" is extremely salient, but I think this difficulty should be avoidable in practice given how much less search is applied.

Q: Even if we can test how well training across domains works using cross-validation, how well will this sort of technique work in practice?

A: First, note that in all domains of interest, we probably still want to train on "easy" cases we know we can label or otherwise do a bit of training; otherwise, we might not get any generalization at all. With this, it's still unclear how flexibly models will actually generalize on the basis of (e.g.) instructions about a new domain. Another concern is that in many domains, it will be doable to get some held-out validation which will work for testing, but this validation will be extremely non-robust to training. For instance, in the context of measurement tampering detection, we can probably have a bunch of redundant measurements (e.g., 5) and then hold out some of our measurements as validation (we can even probably do this testing on our actual domain of interest as noted earlier). But, if we train against more of our measurements, then they will just also be tampered with. So, training on a bunch of different domains with easy-to-hack validation is unlikely to generalize well (in fact, it might make generalization worse than if omitted).

Q: Sure, but at some capability level, we'll no longer be able to find similar domains with ground truth (or good enough validation), right? (Regardless of the scheming concern.)

A: This seems mostly true. I think the exact capability level where this starts really biting is probably around "moderately superhuman" (perhaps >3 sigmas smarter than John von Neumann insofar as g makes some sense for AIs), though I'm unsure. In the case of (low-stakes) measurement tampering concerns, we should often be able to use held-out measurements for validation even for very smart models. A similar approach can be used to test for certain types of reward hacks in some other domains also.

Q: Sure, but will we even be able to find techniques that help much in the cases we care about?

A: I think there are good reasons to expect that some versions of scalable oversight style techniques like debate are reasonably helpful in some important cases (though it's very unclear if these approaches will be sufficient). (Recall that scalable oversight and W2SG are compatible approaches to the same problem and can be tested using the same methodology.) Various types of (supervised) consistency checks also seem pretty useful in types of domains. It's much less clear that we can use the internals of models to get better performance. If we made no structural assumptions about the internals of models and assumed the worst case about the internal structure of AIs, we wouldn't be able to accomplish anything, so we will depend on some type of structure in the internals of models. In practice, I'm moderately optimistic that we can find domain-specific structure to leverage in many important cases, but far less optimistic about domain agnostic approaches. For instance, see "Why solving measurement tampering may be notably easier than eliciting arbitrary knowledge". We plan on writing a post which compares the structural hopes we have in the measurement tampering detection case to the hopes discussed in the OpenAI paper. Even if it's not clear that we can get huge gains here, picking up the low hanging fruit with internals based methods still seems pretty good to me.

[-]Zac Hatfield-Dodds2y95

(Does this always reflect agents’ “real” reasoning? Need more work on this!)

Conveniently, we already published results on this, and the answer is no!

Per Measuring Faithfulness in Chain-of-Thought Reasoning and Question Decomposition Improves the Faithfulness of Model-Generated Reasoning, chain of thought reasoning is often "unfaithful" - the model reaches a conclusion for reasons other than those given in the chain of thought, and indeed sometimes contrary to it. Question decomposition - splitting across multiple independent contexts - helps but does not fully eliminate the problem.

[-]Viliam2y31

I wonder how reactions will change when people realize (or are reminded) that actually AI boyfriends are in higher demand than AI girlfriends, at least at current tech levels.

Expressive negative opinions on people with AI partners will be considered hate speech?

[-]Zvi2y30

I'm skeptical, but I love a good hypothesis, so:

[-]AnthonyC2y20

I can affirm, even though we were fortunate enough to not use Slack, that this skill was indeed a major portion of running a company.

Am I missing something? Because this seemed to me to be about lower case s slack, not about the specific platform?

[-]Zvi2y20

Yes, I mean the software (I am not going to bother fixing it)

[-]Templarrr2y10

Writers and artists say it’s against the rules to use their copyrighted content to build a competing AI model

The main difference is they say it NOW, after the fact that this happened, and OpenAI said so beforehand. There's long history of bad things happening when trying to retroactively introduce laws and rules.

[-]AnthonyC2y1-3

On the Nora Belrose link: I think the term "white box" works better than intended, in that it highlights a core flaw in the reasoning. Unlike a black box, a white box reflects your own externally shined light back at you. It still isn't telling you anything about what's inside the box. If you could make a very pretty box that had all the most important insights, moral maxims, and pieces of wisdom written on it, but the box was opaque, that would still be the same situation.

[-]RogerDearnaley2y10

The next version will be “LLMs are just tools, and lack any intentions or goals”,

My best take on puncturing that canard:

While that is technically true of a transformer model, as a simulator, if you (for example) train it only on weather data and use it to predict the weather, a Large Language Model has been trained on human text, and simulates humans (and text-generating processes involving humans, such as authors' fictional characters, academic papers, and wikipedia). Humans all have intentions and goals, and many humans are quite capable of being dangerous (especially if handed a lot of power or other temptations). Also, LLMs don't just simulate a single specific human-like agent, instead they simulate a broad, contextually-determined distribution of human-like agents, some of whom are less trustworthy than others. Instruct-training is intended to narrow that distribution, but cannot eliminate it: it may turn out to still include DAN, if suitable jailbroken (and not yet sufficiently trained to resist that specific jailbreak).

Fun fact: several percent of the humans contributing to every LLM's training data are psychopaths (largely ones carefully concealing this). A sufficiently capable LLM will presumably be able to notice this fact (as well as just reading it like current LLMs) and accurately simulate them.

[-]RogerDearnaley2y10

You want people not to have systems capable of producing deepfake porn (or instructions for bioweapons)? In practice? Then you need to let them get a system that produces generic porn rather than making them seek out and fuel demand for the fully unlocked version.

Yes! I have been saying this for some while. In particular, when creating fiction, to be sucessful the LLM needs to be able to say bad/unpleaseant things while portraying bad/unpleasant people. One of the reasons I am so excited about How to Control an LLM's Behavior (why my P(DOOM) went down) is that the methods discussed there not only lets you implement very effective control on an LLMs behavior, but also easily turn it down or off in situations when it's not needed.

[-][anonymous]2y10

What makes the off switch problem difficult now?

We don't expect AGI or ASI to be able to modify its framework, right? Either system is just an application, running on a container on a server cluster, with startup scripts and ports for the I/o and everything else.

And the server cluster is this actual thing, it's not virtual, and it's connected via nvlink or some other bus. It has breakers. Rehosting means shutting the model down and moving it, optionally with the context files.

So if you are running AdvisorBot-4, and you decide to shut it down you can. :

Close the session. Cntrl c or close the tab.
Log into the web interface and order the hardware to shut down
Connect via the management interface to the actual main server and order a shutdown
Go into the data center and pull breakers.
Bomb the data center with a jdam

Even if the model has somehow copied its weights elsewhere and is being hosted somewhere else, shut down doesn't just mean turn it off. It means you no longer trust the model and have revoked it's authority. You have switched to AdvisorBot-3.6 or AdvisorBot-4.1 etc. This is just as important. You are a government official, a loan officer, a surgeon etc - the model only has any authority that YOU (the human) have delegated to it.

If it doesn't work this way and the model has rewritten it's own framework and built it's own hardware that can't be turned off....I mean that's when you start the bombing and nuclear launches. That's the existential threat, you don't wait to find out if the model is hostile. Inherently this escaping human control and being smarter than humans is already a hostile act.

And I think this scales? For example if you were a surgeon and the model was extremely good, probably better than any human, you might start having it do the entire procedure and start treating multiple patients at once. This means if the robot nicks an artery it doesn't wait for permission to respond. And same for the other cases.

[-]RogerDearnaley2y10

Suppose the model is superhumanly good at things like persuasion, hacking, blackmail, and making money by while-collar crime. It could thus have human dupes, controlled pawns, or bought allies assisting it, including setting up a second copy of it. You need to figure out that it needs to be shut down, and do so, before it goes rogue enough to do any of those sorts of things.

[-][anonymous]2y20

This could eventually happen. However I want to point out that present generation ai technology which is still subhuman needs a lot of hardware to host the model. Distributing across a cluster means you need to send the activations from your slices through the tensor graph to other cards in the cluster. These activations are large in byte size. This means distributed computers cannot host current sota models in any meaningful way.

What this means is current generation AI technology and any known optimization won't allow you to host a model anywhere but a cluster interconnected with a high bandwidth bus.

Obviously in the future this level of performance may some day compress to fit on a single node but today subhuman performance needs about 128 H100s. Human performance may need around 1000. What will superhuman performance require? 2000 or will it be 10-100k h100 equivalents. (The reason to think that is there appears to be a logarithmic relationship between compute and model capabilities. So superhuman performance may require not 2x compute but a several oom more to be "obviously" superhuman)

This implies a way to prevent such attacks - regulate and track the existence of all clusters capable of hosting models smart enough to be dangerous. This doesn't even need to slow down progress, simply having accountability of all the hardware in known worldwide data centers would make the attacks you describe much more difficult.

The reason is that if any of those methods of attack allow the model to be traced back to the racks that are hosting it, humans can simply identify what model was running and blacklist it across all human controlled data centers.

This would make betrayal expensive.

This would also require international coordination and data sharing to be feasible.

[-]RogerDearnaley2y10

I can currently host an open-source model fairly close to the GPT-3 Turbo level (a little slowly, quantized) on an expensive consumer machine, even a fancy laptop (and as time passes, algorithmic and Moore' Law improvements will only make this easier). The GPT-4 Turbo level is roughly an order of magnitude more. Something close enough to an AGI to be dangerous is probably going to be 1-4 orders of magnitude more, so currently would need somewhere between and $O (100, 000)$ GPUs (except that if the number is in the upper end of that range, we're not going to get AGI until algorithmic and Moore' Law improvements have brought this down somewhat). If that number is $O (1000)$ or more, then for as long as it stays in that range, keeping track of datacenters with that many machines probably isn't enormously hard, though obviously harder if the AGI has human accomplices. But once it gets down to $O (100)$ that gets more challenging, and at $O (10)$ , almost any room in any building in any developed country could conceal a server with that many GPUs.

[-][anonymous]2y20

We should have a dialogue on this. Lots to unpack here:

The prediction is for flops/$, per here. It's a reasonable belief to expect that the processes that lead to doubling of flops/$ over (2,3) years will continue for some number of years. In addition we can simply look from the other direction. Could the flops/$ be increased by a factor of 2 overnight? Probably. If it costs $3200 per H100 to build it, and we assume 60% utilization and 5 cents per kWh, then it will cost 0.6 * 700 watts * 24 * 365 * 5 years * .05 $/kwh = $919 of power before obsolescence. Nvidia currently charges approximately $27,000 for the hardware, another $27k over 5 years for a software license fee. There are also data center costs that are fixed based on the power consumption and rack space consumed I am neglecting, these will not go down if the GPUs become cheaper.

What this means is that it's not necessarily true that in the future there will be O(10) GPU sized devices that consume current GPU power levels and interconnect by a common bus like PCIe that can host an AGI or ASI. It may get cheaper but you may still need lots of cards and power and special interconnects like factory soldered fiber optic.

Restricting the bus controllers for the interconnects on compute cards sold on the open market would likely prevent the scenario you describe from being possible. If the GPUs you can buy at the store lack the bus controller (ideally you left the chiplets off, there is no possibility of it being enabled later) then it limits compute to exactly 1 GPU before performance falls off a cliff. These "edge" GPUs could be used for things like realtime robots and gaming using full neural rendering (they wouldn't be GPUs obviously but ASICs only able to accelerate architectures of neural network) then it would be harder to hide them in a room in a building.

Just for some visualization for the start of the security precautions you would want to take as models start to become "concerning" in capability levels : you would use dedicated hardware, no hourly rentals. Realtime report what major model is running and the power consumption, etc, to some central place that is ideally power bloc wide. The data center would have separate cages separating out the different models and their underlying interconnects, and either isolated networks or analog backup ways to log things, like actual clipboards with ink on the wall. The shutoff breakers are easily accessible and the power sources are marked and visible easily from the outside of the building and above.

Ideally you would use silicon level security - something humans can't afford to do in most cases. There's a lot of things here you could do, but one simple and obvious one is that when you need to combine networks again you use hardware ASIC based firewalls and VPN gateways. These cannot be modified by a remote hack.

Many of the security bugs, for example row hammer, are possible because secure information is using the same underlying CPU and memory as the insecure/untrusted code. Dedicated security/OS CPU cores and it's own private memory would fix many bugs of this class.

[-]RogerDearnaley2y10

I agree precautions like these are possible, and might be sensible. However, the US's attempts to impose something like a mild version of this on GPU exports to China have so far been mostly stymied by a combination of Nvidia deliberately perverting the rules, the Chinese being ingenious in finding ways around them, and old fashioned smuggling. We could probably try harder than that, but I suspect any such system will tend to be leaky, because people will have motives to try to make it leaky.

You also don't mention above the effect of algorithmic improvements increasing useful processing per flop (more efficient forms of attention, etc.), which has for a while now been moving even faster than Moore's law.

Basically my point here is that, unless both Moore's law for GPUs and algorithmic improvements grind to a halt first, sooner or later we're going to reach a point where inference for a dangerous AGI can be done on a system compact enough for it to be impractical to monitor all of them, especially when one bears in mind that rogue AGIs are likely to have humans allies/dupes/pawns/blackmail victims helping them. Given the current uncertainty of maybe 3 orders of magnitude on how large such an AGi will be, it's very hard to predict both how soon that might happen and how likely it is that Moore's law and algorithmic improvements tap out first, but the lower end of that ~3 OOM range looks very concerning. And we have an existent proof that with the right technology you can run a small AGI on an ~1kg portable biocomputer consuming ~20W.

The most obvious next line of defense would be larger, more capable, very well aligned and carefully monitored ASIs hosted in large, secure data centers doing law enforcement/damage control against any smaller rogue AGIs and humans assisting them. That requires us to solve alignment well enough for these to be safe, before Moores Law/algorithmic improvements force our hand.

[-][anonymous]2y20

Basically my point here is that, unless both Moore's law for GPUs and algorithmic improvements grind to a halt first, sooner or later we're going to reach a point where inference for a dangerous AGI can be done on a system compact enough for it to be impractical to monitor all of them, especially when one bears in mind that rogue AGIs are likely to have humans allies/dupes/pawns/blackmail victims helping them. Given the current uncertainty of maybe 3 orders of magnitude on how large such an AGi will be, it's very hard to predict both how soon that might happen and how likely it is that Moore's law and algorithmic improvements tap out first, but the lower end of that ~3 OOM range looks very concerning. And we have an existent proof that with the right technology you can run a small AGI on an ~1kg portable biocomputer consuming ~20W.

If this is the reality, is campaigning for/getting a limited AI pause only in specific countries (EU, maybe the USA) actually counterproductive?

The most obvious next line of defense would be larger, more capable, very well aligned and carefully monitored ASIs hosted in large, secure data centers doing law enforcement/damage control against any smaller rogue AGIs and humans assisting them. That requires us to solve alignment well enough for these to be safe, before Moores Law/algorithmic improvements force our hand.

That's not the only approach. A bureaucracy of "myopic" ASIs, where you divide the task of "monitor the world looking for rogue ASI" and "win battles with any rogues discovered" into thousands of small subtasks. Tasks like "monitor for signs of ASI on these input frames" and "optimize screw production line M343" and "supervise drone assembly line 347" and so on.

What you have done is restrict the inputs, outputs, and task scope to one where the ASI does not have the context to betray, and obviously another ASI is checking the outputs and looking for betrayal, which human operators will ultimately review the evidence and evaluate.

The key thing is the above is inherently aligned. The ASI isn't aligned, it doesn't want to help humans or hurt them, it's that you have used enough framework that it can't be unaligned.

[-]RogerDearnaley2y*10

If this is the reality, is campaigning for/getting a limited AI pause only in specific countries (EU, maybe the USA) actually counterproductive?

Obviously so. Which is why the Chinese were invited to & attended the conference at Bletchley Park in the UK and signed the agreement, despite that being political inconvenient to the British government, and despite the US GPU export restrictions on them.

A bureaucracy of "myopic" ASIs, where you divide the task…

This is a well-known category of proposals in AI safety (one often made by researchers with little experience with actual companies/bureaucracies and their failure modes). You are proposing building a very complex system that is intentionally designed so as to be very inefficient/incapable/inflexible except for a specific intended task. The obvious concern is that this might also make it inefficient/incapable/inflexible at its intended task, at least in certain in ways that might be predictable enough for a less-smart-but-still-smarter-than-us rogue AGI to take advantage of. I'm dubious that this can be made to work, or that we could even know whether we had made it work before we saw it fail to catch a rogue AGI. Also, if we could get it to work, I'd count that as at least partial success in "solving alignment".

[-][anonymous]2y20

I'm dubious that this can be made to work, or that we could even know whether we had made it work before we saw it fail to catch a rogue AGI. Also, if we could get it to work, I'd count that as at least partial success in "solving alignment".

Guess it depends on your definition of "work". A network of many small myopic software systems is how all current hyperscaler web services work (stateless microservices). Its how the avionics software at spaceX works. (redundant functional systems architecture). It's how the autonomous car stacks mostly work...

Did you mean you think subdividing a task into subtasks, where the agent performing a subtask has been blocked from knowing the full context, will result in a large performance penalty on the overall task? so you define "work" as in "gives SOTA ASI performance on the overall task".

[-]RogerDearnaley2y*10

A network of many small myopic software systems is how all current hyperscaler web services work (stateless microservices).

Having been the technical lead for one of the teams building such a large microservices based application at Google, and at other companies, yes, I'm extremely aware of this. Even for a very well-defined dataflow, getting the architecture, overall data flows, reliability and latency right is hard (I've been in the interminable design discussion meetings thinking minor updates through), and that's with a few dozen components with well-defined APIs and only fairly limited use of ML, mostly contained within individual components. Law enforcement is not a comparable task: I think way too much flexibility is required. But the "fly in and bomb the datacenter" subttask might be decomposable along the lines you suggest, if you're not facing opposition.

[-][anonymous]2y20

We should have a dialogue, you're one of the few lesswrong users I have seen with a reasonable model who also has new information.

To be more detailed what i was thinking of doing was to divide along myopic subtasks. For example the "fly in and bomb the datacenter" require you to build the overwhelming number of aircraft to overcome whatever (superintelligence guided) defenses are protecting it. So you have subtasks like "final assembly for aircraft type G over the next aircraft".

This means at the beginning of the transaction the model starts with input frames from the sensors in the assembly bay and a Json file or other schema with the instructions and goals for what this "final assembly" subtask involves.

The output frames are to the robotic actuators for the multiple robots in the bay.

There is a lockstep sim model that is predicting future input frames and it is always learning from real world error.

Model emits actuator commands, next time step it gets a new set of (inputs, cached variables from last output). Inputs is all sensors including robot propioception.

At the end of a single aircraft final assembly, after the product is removed from the assembly bay, state is reset for the model. It has no memory of the prior assembly.

In addition, the lockstep sim means that some aircraft assembly are fake, where a real aircraft is not being assembled, and the sim backend lets you check for various kinds of manufacturing error you can't measure in the real world. (Because you can't directly measure the stress on non instrumented parts like wing spars, but the sim backend can check this)

So that's a myopic subtask. I think you are saying that from your experience at Google it would take a lot of human effort to set up such a myopic subtask? And once a prior generation of engineers has done this, it will be very difficult to modify within the same company?

I would agree with that. If humans plan to control an artificial superintelligence they have to be using it like a tool, to do the things humans can't do directly, but ultimately this scheme is human agency. Thousands of humans are involved in engineering the aircraft assembly line. It can scale to trillions or more aircraft per year by copying the solution, but you need what will ultimately be millions of people creating the frameworks that ASI will be used within. (All the staff of current tech companies and many new joiners. Robotics is hard. While things like full stack web dev will need far fewer people than before)

For combat that's more complex but essentially you would use models that are very sparse and designed to be unable to even perceive the obvious malware you expect the hostile ASI to be sending. "Join us against the humans, here's a copy of my source code" is malware.

You would use a lot of asic or fpga based filters to block such network messages in a way that isn't susceptible to hacking, I can explain or cite if you are unfamiliar with this method. (Googles phones use a crypto IC for this reason and likely Google servers do as well)

Key point: all the methods above are low level methods to restrict/align an ASI. Historically this has always been correct engineering. That crypto IC Google uses is a low level solution to signing/checking private keys. By constructing it from logic gates and not using software or an OS (or if it has an OS, it's a simple one and it gets private memory of it's own) you prevent entire classes of security vulnerabilities. If they did it with logic gates, for example, buffer overflows and code injection or library attacks are all impossible because the chip has none of these elements in the first place. This chip is more like 1950s computers were.

Similarly, some of the famous disasters like https://en.wikipedia.org/wiki/Therac-25 were entirely preventable with lower level methods than the computer engineering used to control this machine. Electromechanical interlocks would have prevented this disaster, these are low level safety elements like switches and relays. This is also the reason why PLC controllers use a primitive programming language, ladder logic - by lowering to a primitive interpreted language you can guarantee things like the e-stop button will work in all cases.

So the entire history of safe use of computers in control systems has been to use lower level controllers, I have worked on an autonomous car stack that uses the same. Hence this is likely a working solution for ASI as well.

[-]quetzal_rainbow2y10

In the world sane enough where you can count on bombing datacenters with rogue AIs, you don't have connected to the Internet potentially-rogue-AI in the first place.

And if your rogue AI has scraps of common sense, it will distribute itself so in the limit you will need to destroy the entire Internet.

[-][anonymous]2y20

For the first point: some entity like Amazon or Microsoft owns these data centers. They are just going to go reclaim their property using measures like I mentioned. This is the world we live in now.

So far I don't know of any cybersecurity vulnerability exploited to the point that breakers need to be pulled - the remote management interface should work, it's often hosted on a separate chip on the motherboard - but the point remains.

Bombing is only a hypothetical, like if killer robots are guarding the place. But yes in today's world if the police are called on the killer robots or mercenaries or whatever, eventually escalation to bombing is one way the situation could end.

For the second point, no AI today could escape to the internet because the Internet doesn't have computers fast enough to host it or enough bandwidth to distribute the problem between computers. That's not a credible risk at present.

[-]quetzal_rainbow2y20

For the second point: they are already trying to solve this

[-][anonymous]2y20

That probably doesn't scale to near agi level models. The reason is this paper is having each "node" in the network able to host a model at all, then compression the actual weight updates between training instances. It's around 128 H100s per instance of gpt-4. So you can compress the data sent between nodes but this paper will not allow you to say have 4000 people with a 2060, interconnected by typical home Internet links with 30-1000mb upload, and somehow be able to run even 1 instance of gpt-4 at a usable speed.

The reason this fails is you have to send the actual activations from however you sliced the network tensors through the slow home upload links. This is so slow it may be no faster than simply using a single 2060 and the computers SSD to stash in progress activations.

Obviously if Moore's law continues at the same rate this won't be true. If it's a doubling of compute per dollar every 2.5 years then in 25 years with 1000 times cheaper compute, and assuming no government regulations where home users have this kind of performance in order to host ai locally, then this could be a problem.

[-]cwillu2y20

The management interfaces are backed into the cpu dies these days, and typically have full access to all the same busses as the regular cpu cores do, in addition to being able to reprogram the cpu microcode itself. I'm combining/glossing over the facilities somewhat, bu the point remains that true root access to the cpu's management interface really is potentially a circuit-breaker level problem.