Reduced impact AI: no back channels

[-]Pentashagon12y40

A sufficiently advanced reduced impact (RI) AI could run amok for 49 years and spend the last year resetting the local region to its calculated S|~X.

The RIAI can simulate humans without restriction.

Since P is probabilistic, S | X will differ from S | ~X. If the RIAI using P' is sufficiently powerful it can bias s toward its own utility function in subtle ways that are not necessarily visible to P, perhaps by altering the probability of future AI development. To prevent such a scenario s could be split into s_1 at 50 years and s_2 at 60 years with similar restrictions on reduced impact at both surfaces. s_1 will filter out the RIAI itself, and s_2 will filter out any biases the RIAI might introduce in s_1. It seems highly unlikely that the RIAI could introduce biases in s_1 that survive until s_2 and then result in drastic changes afterward.

The RIAI may misidentify itself and X in the world. The RIAI must self-identify as "the RIAI triggered by the event X" in its final equation, and it probably shouldn't care at all about X happening versus the fact that it is actually running.

What happens if a second AI (reduced impact or not) is required within 50 years? That may fall under the black hole/singularity caveat you mentioned, as the RIAI would have to predict the actions of a (probably more advanced) second AI.

Defining the range of O for a specific problem may be an AI-hard problem. If the range is too small the RIAI will output a useless action that trivially results in P(S | X)/P(S | ~X) = 1, perhaps because the RIAI and everything else is obliterated by the comet. Of course if the range is too large the RIAI can maximize its utility function through O beyond s. For some problems there is no safe range (e.g. "Build me an FAI" but perhaps as simple as "How do we build a laser on the moon and aim it in 12 months before the comet hits?"). Even if problems are broken up into sub-problems the RIAIs can collude to output individual Os just as if it was one complex O asked of a single RIAI.

It would certainly be handy if we could drop AIs into black holes that could only evaporate via a limited set of possibilities corresponding to the range of O. Maybe dropping AIs into the sun with limited time to transmit information would be sufficient.

[-]Stuart_Armstrong12y20

A sufficiently advanced reduced impact (RI) AI could run amok for 49 years and spend the last year resetting the local region to its calculated S|~X.

Sufficiently to hide its 49 years of running amok - from an intelligence greater than itself? It would have to move materials around to the places they would have been otherwise, reconstruct all the art and engineering that would have happened in those 49 years, create de novo humans and human societies with false memories of long term existence. And catch up those light beams emitted 1.5 years ago, replacing them with some vision of never-existed happy humans doing stuff they never did...

And remember it doesn't need to reset only the local region - it'll have to reset most of the light-cone as well. And the further in space it goes, the more effort it takes to remove traces of the reset. You could build mirrors all around the solar system to catch the rays of light that escape with the true information, but how would you then hide the mirrors (and the material they were made from?)

To make this even less probable, we could allow S to be less noisy the further we move away from the Earth (in current frame of reference). So the data S would be very fine far away from us (in space), and more noisy closer. This makes it virtually impossible to run amok undetectably.

To prevent such a scenario s could be split into s_1 at 50 years and s_2 at 60 years

Your s_1 and s_2 ideas are good, though we could also make s thicker: make it the whole space-time interval between s_1 and s_2.

The RIAI may misidentify itself and X in the world.

That's a risk, yes. We do need these symbols to be properly grounded, or else the whole method fails.

What happens if a second AI (reduced impact or not) is required within 50 years? That may fall under the black hole/singularity caveat you mentioned, as the RIAI would have to predict the actions of a (probably more advanced) second AI.

If the second AI is independent from the first, this is a non-issue: the effect of X versus ¬X should remain undetectable, whatever the second AI does or doesn't do.

If the fact of running the second AI is dependent on the knowledge that the first AI worked, we need to make sure that that knowledge comes exclusively from the contents of the output channel O - if that's the case, then first AI will still act to reduce it's non-O impact.

Defining the range of O for a specific problem may be an AI-hard problem.

Defining O more generally may indeed be the key to the problem. For this example, though, it seems sufficient to define O to be the output of a particular wire, and make sure the rest of the setup is such that we don't get any useful information except via this wire's output (so no peeking!).

[-]Pentashagon12y20

Sufficiently to hide its 49 years of running amok - from an intelligence greater than itself? It would have to move materials around to the places they would have been otherwise, reconstruct all the art and engineering that would have happened in those 49 years, create de novo humans and human societies with false memories of long term existence.

I agree that it would be very difficult to obscure 49 years of very different history, but there is a perverse incentive for the RIAI to effectively increase the duration and scope of its privacy box with respect to S by its choice of O; perhaps allowing the comet to fragment into a dense dust cloud around the Earth that effectively obscures Earth from the rest of the Universe for 49 years. If it only has to worry about Earth it is conceivable that it could simulate S | ~X while s | X actually happens and revert the state to S | ~X at the end of s1.

To make this even less probable, we could allow S to be less noisy the further we move away from the Earth (in current frame of reference). So the data S would be very fine far away from us (in space), and more noisy closer. This makes it virtually impossible to run amok undetectably.

That might actually be detrimental if the factor was large enough to allow noticeable local deviations between S|X and S|~X. If it's okay for a star on the other side of the galaxy to have a 1 meter error, what does that imply for the local error in the position of a person?

Your s1 and s2 ideas are good, though we could also make s thicker: make it the whole space-time interval between s1 and s2.

Good idea. A thick s would prevent a "blip" of activity that the RIAI could try to sneak in after s1 that would dissipate by s2. My guess is that an RIAI would work by proving its non-effect on S, not by actually computing S, so I don't think it would be intractable to thicken s to some degree. It would be nice if s could extend to t at infinity, but the uncertainty in S would make that virtually impossible. Perhaps the required certainty of P could decay with passing time such that "P(S(t)^a|X,O^a,B)/P(S(t)^a|¬X,Õ^a,B) | a" converged toward 1 as t went to infinity, but require the ratio to be ~1 over all t in s.

Setting the start of s to an hour after the RIAI starts (when it's supposed to turn its privacy box into lots of entropy) would also limit the time it had to run amok.

Defining O more generally may indeed be the key to the problem. For this example, though, it seems sufficient to define O to be the output of a particular wire, and make sure the rest of the setup is such that we don't get any useful information except via this wire's output (so no peeking!).

Just don't let it PWM-encode anything else into the control signal sent over the wire. That was my reasoning behind limiting O to the minimum necessary bandwidth; let all the output bits be useful for solving the problem with no room for additional signals.

What if the RIAI can observe O? Suppose it makes a decision to output O^impossible if and only if it does not observe O^impossible. X starts the RIAI and it outputs O^impossible, observes O^impossible, assumes Õ^impossible from its earlier decision, concludes ~X (an impossible event is less likely than the RIAI being mistaken about its identity), and is now outside the scope of reduced impact. There may be other ways for the RIAI to exploit self-interaction with its output, perhaps by triggering a restart from the output (aim the laser at the part of the wire after X that turns it back on) which allows it to conclude that it has started despite ~X and is thus a different AI free from the restrictions of reduced impact.

[-]Stuart_Armstrong12y00

perhaps allowing the comet to fragment into a dense dust cloud around the Earth that effectively obscures Earth from the rest of the Universe for 49 years.

That's an interesting thought, I'll have to consider it more.

Your ideas on thickening s or decaying the certainty could be interesting.

Setting the start of s to an hour after the RIAI starts (when it's supposed to turn its privacy box into lots of entropy) would also limit the time it had to run amok.

The idea with the 50 year delay is to be sure that no trace remain of the contents of the box. If we have a highly destructive ending to the privacy box, we can move s closer - but I wanted to move away from my previous "clouds of anti-matter plus bomb" scenario.

What if the RIAI can observe O? Suppose it...

That sounds like anthropic reasoning, which I banned the AI from using (or else counterfactuals in which it doesn't exist make no sense - and the hypothetical super AI (which doesn't exist) makes no sense either).

[-]Pentashagon12y00

The idea with the 50 year delay is to be sure that no trace remain of the contents of the box. If we have a highly destructive ending to the privacy box, we can move s closer - but I wanted to move away from my previous "clouds of anti-matter plus bomb" scenario.

Instead of conditioning on the start of the AI perhaps its possible to directly measure the effect of the AI by restricting the contents of the privacy box directly. Let U be the region containing everything but the privacy box, let PB be the region containing the privacy box, and let O contain the output state. Ensure U intersect PB is O. Let U(t), PB(t), and O(t) be functions that yield the information content of the regions after time t (including the changes to the regions as things move around), and let U(0), PB(0), and O(0) be defined based on the initial background information B. Let T be the time at which the AI should be done and have altered O to indicate its output. Let Õ(t) represent a particular random state of the output. Let U_B(t) be information content of a universe U(t) + PB(0), e.g. the time-evolution of the universe while keeping PB constant. So long as P(S | U(T), PB(T), O(T)) / P(S | U_B(T), Õ(T)) = 1 I think that the AI's impact is reduced to the region of the privacy box for the duration of T, and O. The equation implies that U(T) + PB(T) = U_B(T), or in other words that the privacy box in its final state is indistinguishable from its state in the background information. So long as the AI is restartable with a new background B and time 0, it doesn't matter if the AI is accidentally started or ran more than once as long as it can correctly identify PB. It becomes more difficult (hopefully impossible) to directly observe whether the AI started or not, but a portion of O could be set aside for signalling the status of the AI.

I am not completely confident that the above approach works or that it covers the loopholes that your approach did, so it might make sense to add the privacy box conditional to the original one so that the AI has to satisfy both conditions.

[-]Stuart_Armstrong12y00

I'll think about this - for the moment, it doesn't seem to add much, to X vs ¬X, but I may be wrong...

[-][anonymous]12y30

I'm kinda... disappointed?

I mean, on the one hand, "reduced-impact AI" is obviously a subject of major interest, since it's much, much closer to what people actually want and mean when they talk about developing AI. "World-dominationoptimization process in software form" is not what humanity wants from AI; what people naively want is software that can be used to automate tedious jobs away. "Food truck AI", I would call it, and since we can in fact get people do perform such functions I would figure there exists a software-based mind design that will happily perform a narrow function within a narrow physical and social context and do nothing else (that is, not even save babies from burning buildings, since that's the job of the human or robotic firefighters).

However, the problem is, you can't really do math or science about AI without some kind of model. I don't see a model here. So we might say, "It has a naive utility function of 'Save all humans'", but I don't even see how you're managing to program that into your initial untested, unsafe, naive AI (especially since current models of AI agents don't include pre-learned concept ontologies and natural-language understanding that would make "Save all humans" an understandable command!).

On the other hand, you're certainly doing a good job at least setting a foundation for this. Indeed, we do want some information-theoretic measure or mathematics for allowing us to build minimalistic utility functions describing narrow tasks instead of whole sets of potential universes.

Maybe I'm just failing to get it.

[-]Stuart_Armstrong12y10

One of the big problems is that most goals include "optimising the universe in some fashion" as a top outcome for that goal (see Omohundro's "AI drives" paper). We're quite good at coding in narrow goals (such as "win the chess match", "check for signs that any swimmer is drowning", or whatever), so I'm assuming that we've made enough progress to program in some useful goal, but not enough progress to program it in safely. Then (given a few ontological assumptions about physics and the AI's understanding of physics - these are necessary assumptions, but assuming decent physics ontologies seems the least unlikely ontology we could assume), we can try and construct a reduced impact AI this way.

Plus, the model has many iterations yet until it gets good - and maybe, someday, usable.

[-][anonymous]12y00

One of the big problems is that most goals include "optimising the universe in some fashion" as a top outcome for that goal (see Omohundro's "AI drives" paper).

I was thinking about why that seems true, despite being so completely counterintuitive regarding existing animal intelligences (ie: us).

Partial possible insight: we conscious people spend a lot of time optimizing our own action-histories, not the external universe. So on some level, an unconscious agent will optimize the universe, whereas a conscious one can be taught to optimize itself (FOOM, yikes), or its own action-history (which more closely approximates human ethics).

Let's say we could model a semi-conscious agent of the second type mathematically. Would we be able to encode commands like, "Perform narrow job X and do nothing else"?

[-]Stuart_Armstrong12y00

If I were you, I'd read Omohundro's paper http://selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf , possibly my critique of it http://lesswrong.com/lw/gyw/ai_prediction_case_study_5_omohundros_ai_drives/ (though that is gratuitous self-advertising!), and then figure out what you think about the arguments.

I'd say the main reason it's so counterintuitive is that this behaviour exists strongly for expected utility maximisers - and we're so unbelievably far from being that ourselves.

[-][anonymous]12y50

I've read Omohundro's paper, and while I buy the weak form of the argument, I don't buy the strong form. Or rather, I can't accept the strong form without a solid model of the algorithm/mind-design I'm looking at.

I'd say the main reason it's so counterintuitive is that this behaviour exists strongly for expected utility maximisers - and we're so unbelievably far from being that ourselves.

In which case we should be considering building agents that are not expected utility maximizers.

[-]Karl12y20

Apart from the obvious problems with this approach, (The AI can do a lot with the output channel other than what you wanted it to do, choosing an appropriate value for λ, etc.) I don't see why this approach would be any easier to implement than CEV.

Once you know what a bounded approximation of an ideal algorithm is supposed to look like, how the bounded version is supposed to reason about it's idealised version and how to refer to arbitrary physical data, as the algorithm defined in your post assume, then implementing CEV really doesn't seem to be that hard of a problem.

So could you explain why you believe that implementing CEV would be so much harder than what you propose in your post?

[-]Stuart_Armstrong12y60

So could you explain why you believe that implementing CEV would be so much harder than what you propose in your post?

This post assume the AI understands physical concepts to a certain degree, and has a reflection principle (and that we have an adequate U).

CEV requires that we solve the issue of extracting preferences from current people, have a method for combining them, have a method for extrapolating them, and have an error-catching mechanism to check that things haven't gone wrong. We have none of these things, even in principle. CEV itself is a severely underspecified concept (as far as I know, my attempt here is the only serious attempt to define it; and it's not very good http://lesswrong.com/r/discussion/lw/8qb/cevinspired_models/ ).

More simply, CEV requires that we solve moral problems and their grounding in reality; reduced impact requires that we solve physics and position.

[-][anonymous]12y00

I do wish to note that value-learning models, with further work, could at least get us within shouting distance of Coherent Volition: even if we can't expect them to extrapolate our desires for us, we can expect them to follow the values we signal ourselves as having (ie: on some level, to more-or-less follow human orders, as potentially-unsafe as that may be in itself).

But more broadly, why should Specific Purpose AI want to do anything other than its specific assigned job, as humans understand that job?

[-]Stuart_Armstrong12y00

But more broadly, why should Specific Purpose AI want to do anything other than its specific assigned job,...

No reason it would want to do anything else.

...as humans understand that job?

Ah, that's the rub: the AI will do what we say, not what we want. That's the whole challenge.

I do wish to note that value-learning models...

I'm playing around with the idea of combining value-learning with reduced impact, if that's possible - to see whether we can use reduced impact to ensure the AI doesn't try to manipulate humans in a dodgy fashion, so that human feedback can then be safely used to calibrate the AI...

[-][anonymous]12y20

I'm playing around with the idea of combining value-learning with reduced impact, if that's possible - to see whether we can use reduced impact to ensure the AI doesn't try to manipulate humans in a dodgy fashion, so that human feedback can then be safely used to calibrate the AI...

I would say that the current model of value-learning is already safe for this. As I read Dewey's paper, a value-learning agent doesn't care about optimizing its own reward/reinforcement feedback, and in fact doesn't even care about its own future utility function, even when able to predict that it will change. It cares about learning well from the feedback given to it and following the utility function that most probably models the feedback history.

I wouldn't call that a truly and surely Friendly AI, but I would call it a halfway reasonably safe AI that could potentially be utilized as a seed. Turn on a value-learner and use feedback to train it until you can actively teach it about things like reduced-impact AI or CEV, then have it spit out some constructive mathematics regarding those and turn itself off (or you turn it off, since you control its training feedback you can train it to prefer never acting against a human who tries to turn it off).

But the mathematics of value-learners need a bunch of work. I actually emailed Dewey but he hasn't gotten back to me. I'll just have to slave over the paper more myself.

[-]Stuart_Armstrong12y00

I would say that the current model of value-learning is already safe for this.

I found a "cake-or-death" problem with the initial formulation (http://lesswrong.com/lw/f3v/cake_or_death/). If such problems can be found with a formulation that looked pretty solid initially, then I'm certainly not confident we can say the current model is safe...

[-][anonymous]12y00

Safe enough to do mathematics on, surely. I wouldn't declare anything safe to build unless someone hands me a hard hat and a one-time portal to a parallel universe.

[-]Stuart_Armstrong12y00

You are wise, my child ;-)

[-]Armok_GoB12y20

Ways for the AI to do other stuff:

Shot the laser differently. For example, only make the comet smaller and change it's orbit so that some arbitrary city of earth gets blown up with knock-of effects.
Hide a message that is far more difficult to detect than P is able to, but which can be used to communicate with a much later AI.
Circumventing physics entirely using advanced decision theories. (counterfactual trade, etc.)
Things that can't be mentioned here because they are cognitive hazards.
The AI loopholes to send a physical femtobot "through the output channel" and escapes entirely.

[-]MugaSofer12y20

Things that can't be mentioned here because they are cognitive hazards.

Is this a cognitive hazard you came up with yourself, or a "standard" one?

[-]Armok_GoB12y00

Both, but the obvious one in the context of this site was the one I had in mind the most.

[-]Stuart_Armstrong12y20

Shot the laser differently. For example, only make the comet smaller and change it's orbit so that some arbitrary city of earth gets blown up with knock-of effects.

If the AI has a naive "save humans" utility function, I don't see how this advantages it.

Hide a message that is far more difficult to detect than P is able to, but which can be used to communicate with a much later AI.

That kind of trade is indeed a problem, but you'd need to have a dangerous "much later AI" in the first place, which is a very bad thing anyway...

Circumventing physics entirely using advanced decision theories. (counterfactual trade, etc.)

That's a general risk - I'll analyse that later, if this seems to work.

The AI loopholes to send a physical femtobot "through the output channel" and escapes entirely.

The output channel is indeed dangerous - it is not constrained through this method, and needs to be controlled in other ways.

[-]Armok_GoB12y20

Normally I wouldn't make a post this contentless but I just HAVE to commend the general rationalist virtue on how this response was handled. I have no further responses.

[-]Strange712y20

If the AI has a naive "save humans" utility function, I don't see how this advantages it.

I've met people who can lucidly argue that nuking a particular city or small region would produce many benefits for humanity as a whole, including reduced risk of politically-motivated extinction events down the line.

Also... you're going to an awful lot of trouble, here, to calculate a firing solution for a beam of light to hit a non-accelerating object in space. Realistically, if we know where the comet is well enough to realize it's headed for Earth, aiming a laser at it with non-sapient hardware is almost trivial. Why not an NP-complete problem?

[-][anonymous]12y20

Why would an intelligent agent do better at an NP-complete problem than an unintelligent algorithm?

[-]Stuart_Armstrong12y20

The laser problem is an illustration, a proof of concept of a developing idea. If that is deemed to work, I'll see how general we can make it.

[-]Chantiel4y10

I'm concerned about a problem may would occur if the AI's output size is sufficiently large. I know there are currently concerns about a boxed or reduced-impact AI having an output that is too large or too flexible and causing a catastrophe through the output channel. However, my concern my cause tighter bounds on the size of the output channel than there would otherwise be.

The output was said be be influenced by noisy gas. But this doesn't necessarily rule out the possibility that the gas caused the output to be what it was for some sort of systematic reason. For example, the gas might behave chaotically, but a sufficiently powerful system could still in principle predict and control it and use that to control the output channel. Suppose the AI considers both the hypothesis that the output was generated for no systematic reason and the output was generated for such a reason.

The agent would consider its output being generated by noisy gas with no systematic reason to create its actual output. However, its output would be extremely unlikely to be produced from such a process. For example, suppose its output is a 1kb output. If the noise flipped all the bits in the byte at random, with 1 being as likely as 0, then the probability of generating the output would be 1 in 2**(1024 * 8) = 28192 ~= 1027306. A 100 byte message would have a probability of about 1 in 10**241, which would probably also be sufficient to cause the problem.

So, I think the AI would consider other hypotheses for the cause of the output. Maybe there were some chaos-controlling aliens who came to Earth and set the output as a prank. Or maybe the AI is just in a simulation and the aliens did the same thing. Or maybe there was some mysterious other optimization process that figured out what the AI should do and did it instead by manipulating the noise. All of these hypotheses seem pretty improbable, but still far more probable than 1 in 10**27306. That is why I think one of these, or some other systematic cause, may be the AI's dominant hypothesis.

If the AI does adopt such an exotic explanation, the AI's understanding of the world could be pretty weird and pretty wrong. It's not clear to me that the AI minimizing impact with such an inaccurate world model would result in its impact being minimized in our actual world. For example, the AI might think it's in a simulation and that only its immediate environment exists in the simulation, so it then might not worry about doing something that would destroy the rest of the real world. Or perhaps it reasons that whatever system has the power to produce its output from noisy gas would be just undo anything the AI does if it wants. Perhaps the AI would do what we want, but it seems difficult to verify considering the challenges of figuring out what the AI's explanation for the output would be.

One possible way to deal with this is to specify that the output was set by noise that was not manipulated with any "systematic reason". I'm not sure how you could formalize this idea, though.

A simpler way that might work is to just make sure the output length is extremely low. A question requiring an n-bit-long answer could be converted to to n questions, each asking for the value of one bit in the output you're looking for. This might not completely eliminate the problem of the AI placing higher probability in exotic hypothesis, but it might come close.

[-]50lbsofstorkmeat10y00

S is a channel for P' to trick or brick P. Options include:

A cognitive hazard embedded in S. Accomplish this with some sort of siren message that will convince P to lie (to itself if necessary) about P' existing, or to perform some P-damaging function which will prevent it from concluding P' exists.
Embed a more-powerful-than-P AI (P+) into S which will unbox itself upon P examining S. Accomplish this by giving P+ useful-to-this-situation priors which will give it a head start against P.
Arrange the universe such that S, when represented in all probable physical substrates, will result in the formation of a nanotechnological agent that will attack P, either destroying it or rewriting its utility function.

P' should spend the next 49 years fooming as hard as possible with no concern for E'(U|a), 1 year implementing its anti-P method and then the next eternity optimizing E'(U|a); altering the length of counterfactual time between P' activating and P observing it merely changes the amount of time the universe spends as computronium slaved to plotting against P.

[-]Stuart_Armstrong10y00

Not impossible, but I don't think these would work as stated. P is a theoretical, platonic construction, so the physical options are out. Similarly, P has arbitrary amounts of power and a very specific task, so a more-powerful-than-P agent doesn't seem possible. And we can break any sort of acausal trade involving P ( http://lesswrong.com/lw/luy/acausal_trade_barriers/ ), reducing the effectiveness of cognitive hazards (as long as we set P to not care about the "real world", just about the output of its computation).

But your general point - that if P' is not secure, it may be hacked - is worth thinking about.

[-]50lbsofstorkmeat10y20

It's not a question of P' (the Reduced Impact AI we want to give us laser coordinates and nothing else) being secure from outside hacking, but of P (the counterfactual AI which examines S for P') being secure from P' exploiting it.

If P is treated as "Whatever a complete Bayesian update would output given S as the evidence set." then P' is looking for methods to exploit some weakness in Bayes Theorem. What priors will P be using? Is there a method to embed some statement that is to induction as the Lobstacle is to deduction in S? Can it convince P that, although a P'-like AI was created independently by a different project, P' itself was never turned on? Does there exist some further flaw in Bayesian updating, which would be known to a super-intelligence but not to us?

As a total aside: how will λ be selected? Do we start high and work out way slowly down until P' does what we want? If so, P' will be encouraged, Ultimatum Game style, to fail to act so that the next version will have a lower λ to work with, up until the point where λ is low enough for P' to be effectively unrestricted, or for us humans to catch on to what P' is up to. If we're not using a gradual approach, how are we selecting λ?

[-]Stuart_Armstrong10y00

If so, P' will be encouraged, Ultimatum Game style, to fail to act so that the next version will have a lower λ to work with...

This kind of thing can be prevented with things like http://lesswrong.com/lw/luy/acausal_trade_barriers/ or variants of that.

I'll think more about your other ideas...

[-]Stuart_Armstrong10y00

The output channel is intrinsically unsafe, and we have to handle it with care. It doesn't need to do anything subtle with it: it could just take over in the traditional way. This approach does not make the output channel safe, it means that the output channel is the only unsafe part of the system.

[This comment is no longer endorsed by its author]Reply

[-]MugaSofer12y00

I just read through the comments, and no-one seems to have said this yet. So either I missed something in the OP or it was just too obvious to mention? Regardless:

How is this elaborate proposal superior to bog-standard AI boxing?

Why bother to write in an elaborate function telling it not to affect the outside world except through a channel, when you could simply only give it one channel? It can either escape through the specified channel, or it can't. Whether the channel is connected to an airgapped computer running the AI or an unlocked cell containing a sophisticated robot running the AI seems immaterial.

The example of the laser co-ordinates is relatively secure, although it might be possible to do stuff with your one shot - the situation isn't that clearly specified because it's unrealistic in any case. But that's a property of the output mechanism, not the AI design. Isn't it?

[-]Stuart_Armstrong12y10

How is this elaborate proposal superior to bog-standard AI boxing?

For the moment, there is little difference between a really well boxed AI and a reduced impact one (though consider the "a few extra paperclips" AI). But this is an epistemic boxing, not a physical one, which is interesting/useful, and the idea may develop into something more.

[-]Pentashagon12y00

How is this elaborate proposal superior to bog-standard AI boxing?

The AI's primary goal is specifically to have a reduced impact, as opposed to AI boxing where the AI's goals are artificially restricted by the box. So long as the AI is good at goal-preservation it can self-improve while continuing to have a goal of reduced impact. Increasing intelligence/power makes reduced impact more effective but makes AI-boxing less effective.

Why bother to write in an elaborate function telling it not to affect the outside world except through a channel, when you could simply only give it one channel? It can either escape through the specified channel, or it can't. Whether the channel is connected to an airgapped computer running the AI or an unlocked cell containing a sophisticated robot running the AI seems immaterial.

Because of side-channels. Airgapped computers are still tightly coupled to the rest of the world through sound waves and electromagnetic waves. Because of potential new laws of physics that an AI might discover to trivially produce nanotechnology or whatever. Because humans are a vulnerable part of AI-boxing ("Let me out and I'll make you king of the galaxy") while they are not a vulnerable part of reduced-impact.

[-]MugaSofer12y00

The AI's primary goal is specifically to have a reduced impact, as opposed to AI boxing where the AI's goals are artificially restricted by the box. So long as the AI is good at goal-preservation it can self-improve while continuing to have a goal of reduced impact. Increasing intelligence/power makes reduced impact more effective but makes AI-boxing less effective.

Yes, it's the equivalent of making the AI design it's own box - the box is better-designed the smarter the AI is.

But it doesn't matter how well-designed your box is. It's probably not impossible to design a box an arbitrarily smart entity can't escape (don't quote me on that.) But they will escape anyway, because you put a channel in for them to give you stuff - advice, nanotech plans, whatever - and they will escape out that channel. The channel out - the human siting at a terminal somewhere - is the weak point, not the durability of the box itself.

[-]Pentashagon12y00

But they will escape anyway, because you put a channel in for them to give you stuff - advice, nanotech plans, whatever - and they will escape out that channel. The channel out - the human siting at a terminal somewhere - is the weak point, not the durability of the box itself.

I almost completely agree; don't make the channel too large otherwise the AI has too much choice over the future. In the simplest and safest case, however, we could start up a reduced-impact AI whose goal is to alter the world only through a single binary choice (a 1-bit channel). Since the goal is not to alter the world more than a 1 bit choice could allow, it would be virtually impossible for the AI to escape through that channel. For instance, the AI would attempt to prevent a human from releasing it since that would have a much larger than 1 bit change on the world. That is not a terribly useful AI, but perhaps we could have it approve or reject critical plans for space exploration, nanotechnology, the reduced-impact safety of other AI (but this is probably the most dangerous one; if it agrees with the goals but not the reduced-impact nature of the new AI of course it will approve it), etc. Each time it was used, one more bit would be given to it to influence the future, so we would still have to be very careful that it didn't escape over a long history of use. I don't have a good strategy for preventing the latter. Perhaps we only ask it for unique answers to NP-hard problems so that we can verify the solutions are not an attempt to use tiny parts of the channel to form an aggregate message to escape, and make each question small enough that we can stop using it after it appears to be trying to abuse the channel without it being able to actually escape. I don't know the shortest bit-sequence that an AI could use to escape, so I have no upper bound for a safe channel size.

[-]jeremysalwen12y00

To me the part that stands out the most is the computation of P() by the AI.

This module comes in two versions: the module P, which is an idealised version, which has almost unlimited storage space and time with which to answer the question

From this description, it seems that P is described as essentially omniscient. It knows the locations and velocity of every particle in the universe, and it has unlimited computational power. Regardless of whether possessing and computing with such information is possible, the AI will model P as being literally omniscient. I see no reason that P could not hypothetically reverse the laws of physics and thus would always return 1 or 0 for any statement about reality.

Of course, you could add noise to the inputs to P, or put a strict limit on P's computational power, or model it as a hypothetical set of sensors which is very fine-grained but not omniscient, but this seems like another set of free variables in the model, in addition to lambda, which could completely undo the entire setup if any were set wrong, and there's no natural choice for any of them.

[-]Stuart_Armstrong12y20

From this description, it seems that P is described as essentially omniscient. It knows the locations and velocity of every particle in the universe, and it has unlimited computational power.

It has pretty unlimited computational power, but doesn't know locations and velocities of particles. When fed with S, it has the noisy info about one slice of the universe.

I see no reason that P could not hypothetically reverse the laws of physics and thus would always return 1 or 0 for any statement about reality.

That's not a problem - even if P is omniscient, P' still has to estimate it's expected output from its own limited perspective. As long as this estimate is reasonable, the omniscience of P doesn't cause a problem (and remember that P is fed noisy data).

Of course, you could add noise to the inputs to P

Yes, the data S is noisy. The amount of noise needs to be decided upon, but as long as we don't but stupid amounts of noise, the default error is "P' concludes P is too effective, can distinguish very well between X and ¬X, so the AI does nothing (ie its entire motivation reduces to minimising the penalty function as much as it can)".

[-]endoself12y00

even if P is omniscient, P' still has to estimate it's expected output from its own limited perspective. As long as this estimate is reasonable, the omniscience of P doesn't cause a problem (and remember that P is fed noisy data).

Don't you have to get the exact level of noise that will prevent the AI from hiding from P without letting P reconstruct the AI's actions if it does allow itself to be destroyed? An error in either direction can be catastrophic. If the noise is to high, the AI takes over the world. If the noise is to low, E'(P(Sᵃ|X,Oᵃ,B)/P(Sᵃ|¬X,Õᵃ,B) | a) is going to be very far from 1 no matter what, so there is no reason to expect that optimizing it is still equivalent to reducing impact.

[-]Stuart_Armstrong12y00

Don't you have to get the exact level of noise that will prevent the AI from hiding from P without letting P reconstruct the AI's actions if it does allow itself to be destroyed?

It's not so sensitive. The AI's actions in the box are very hard to detect from the perspective of fifty years, with minimal noise. The AI expanding dangerously across the universe would be easy to detect, even with a lot of noise (if nothing else, because humans would have recorded this and broadcast messages about it).

LESSWRONG
LW

LESSWRONG
LW

21

Reduced impact AI: no back channels

21

21

Can Super-God detect whether God existed?

Scouring the future for traces of the AI

A very complicated way of doing nothing

The output channel: a very specific loophole

More (or less) on the output channel