Chantiel's Shortform

by Chantiel8th Aug 202194 comments
93 comments, sorted by Highlighting new comments since Today at 7:27 PM
New Comment
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

I made a new article about defining "optimizer". I was wondering if someone could look over it and tell me what they think before I post it on Less Wrong. You can find it here.

There is a matter I'm confused about: What exactly is base-level reality, does it necessarily exist, and is it ontologically different from other constructs?

First off, I had gotten the impression that there was a base-level reality, and that in some sense it's ontologically different from the sorts of abstractions we use in our models. I thought that, it some sense, the subatomic particles "actually" existed, whereas our abstractions, like chairs, were "just" abstractions. I'm not actually sure how I got this impression, but I had the sense that other peop... (read more)

2Vladimir_Nesov24dDistinguishing the physical world from mathematical entities is pragmatic, reflects how it relates to you. It's impossible to fully know what the physical world is, but it's possible to interact with it (and to care about what happens in it), and these interactions depend on what it is. When reasoning about constructed mathematical entities, you get to know what you are working with, but not in the case of the physical world. So we can similarly consider an agent living in a different mathematical entity, and for that agent that mathematical entity would be their real physical world. Because we have to deal with the presence of the real world, it might be convenient to develop concepts that don't presume knowledge of its nature, which should apply to mathematical entities if we forget (in some perspective) what they are. It's also relevant to recall that the idea of a "mathematical entity" is informal, so strictly speaking it doesn't make sense to claim that the physical world is "a mathematical entity", because we can't carefully say what exactly "a mathematical entity" is in general, there are only more specific examples that we don't know the physical world to be one of.
2JBlack1moReality is that which actually exists, regardless of how any agents within it might perceive it, choose to model it, or describe it to each other. If reality happens to be infinitely complex, then all finite models of it must necessarily be incomplete. That might be annoying, but why would you consider that to mean "reality doesn't really exist"?
2Chantiel1moWell, to be clear, I didn't intend to say that reality doesn't really exist. There's definitely something that's real. I was just wondering about if there is some base-level reality that's ontologically different from other things, like the abstractions we use. Now, what I'm saying feels pretty philosophical, and perhaps the question isn't even meaningful. Still, I'm wondering about the agents making an infinite sequence of decompositions that each have increased predictive accuracy. What would the base-level reality be in that case? Any of the decompositions the agents create would be wrong, even if some are infinitely complex. Also, I've realize I'm confused about the meaning of "what really exists", but I think it would be hard to clarify and reason about this. Perhaps I'm overthinking things, but I am still rather confused. I'm imagining some other agent or AI that doesn't distinguish between base-level reality and abstractions, I'm not sure how I could argue with them. I mean, in principle, I think you could come up with reasoning systems that distinguish between base-level reality and abstractions, as well as reasoning systems that don't, that both make equally good empirical predictions. If there was some alien that didn't make the distinction in their epistemology or ontology, I'm not sure how I could say, and support saying, "You're wrong". I mean, I predict you could both make arbitrarily powerful agents with high predictive accuracy and high optimization-pressure that don't distinguish between base-level reality and abstractions, and could do the same with agents that do make such a distinction. If both perform fine, them I'm not sure how I could argue that one's "wrong". Is the existence of base-level reality subjective? Does this question even make sense?
2JBlack1moWe are probably just using different terminology and talking past each other. You agree that there is "something that's real". From my point of view, the term "base-level reality" refers to exactly that which is real, and no more. The abstractions we use do not necessarily correspond with base-level reality in any way at all. In particular if we are any of simulated entities, dreaming, full-sensory hallucinating, disembodied consciousness, or brains in jars with synthetic sensory input then we may not have any way to learn anything meaningful about base-level reality, but that does not preclude its existence because it is still certain that something exists. None of the models are any sort of reality at all. At best, they are predictors of some sort of sensory reality (which may be base-level reality, or might not). It is possible that all of the models are actually completely wrong, as the agents have all been living in a simulation or are actually insane with false memories of correct predictions, etc. The question makes sense, but the answer is the most emphatic NO that it is possible to give. Even in some hypothetical solipsistic universe in which only one bodiless mind exists and anything else is just internal experiences of that mind, that mind objectively exists. It is conceivable to suppose a universe in which everything is a simulation in some lower-level universe resulting in an ordering with no least element to qualify as base-level reality, but this is still an objective fact about such a universe.
2Chantiel1moWe do seem have have been talking past each other to some extent. Base-level reality, for course, exists if you define it to be "what really exists". However, I'm a little unsure about if that's how people use the word. I mean, if someone asked me if Santa really exists, I'd say "No", but if they asked if chairs really existed, I'd say "Yes". That doesn't seem wrong to me, but I thought our base-level reality only contained subatomic particles, not chairs. Does this mean the statement "Chairs really exist" is actually wrong? Or I am misinterpreting? I'm also wondering how people justify thinking that models talking about things like chairs, trees, and anything other than subatomic particles don't "really exist". Is this even true? I'm just imagining talking with some aliens with no distinction between base-level reality and what we would consider mere abstractions. For example, suppose the aliens knew about chairs, when they discovered quantum theory, they said say, "Oh! There are these atom things, and when they're arrange in the right way, they cause chairs to exist!" But suppose they never distinguished between the subatomic particles being real and they chairs being real: they just saw both subatomic particles and chairs to both be fully real, and the correct arrangement of the former caused the latter to exist. How could I argue with such aliens? They're already making correct predictions, so I don't see any way to show them evidence that disproves them. Is there some abstract reason to think models about thing like chairs don't "really exist"?
2JBlack1moThe main places I've see the term "base-level reality" used are in discussions about the simulation hypothesis. "Base-level" being the actually real reality where sensory information tells you about interactions in the actual real world, as opposed to simulations where the sensory information is fabricated and almost completely independent of the rules that base-level reality follows. The abstraction is that the base-level reality serves as a foundation on which (potentially) a whole "tower" of simulations-within-simulations-within-simulations could be erected. That semantic excursion aside, you don't need to go to aliens to find beings that hold subatomic particles as being ontologically equivalent with chairs. Plenty of people hold that they're both abstractions that help us deal with the world we live in, just at different length scales (and I'm one of them).
2Chantiel1moWell, even in a simulation, sensory information still tells you about interactions in the actual real world. I mean, based on your experiences in the simulation, you can potentially approximately infer the algorithm and computational state of the "base-level" computer you're running in, and I believe those count as interactions in the "actual real world". And if your simulation is sufficiently big and takes up a sufficiently large amount of the world, you could potentially learn quite a lot about the underlying "real world" just by examining your simulation. That said, I still can't say I really understand the concept of "base-level reality". I know you said its what informs you about the "actual real world", but this feels similarly confusing to me as defining base-level reality as "what really exists". I know that reasoning and talking about things so abstract is hard and can easily lead to nonsense, but I'm still interested. I'm curious about what even the purpose is of having an ontologically fundamental distinction between base-level reality and abstractions and whether it's worth having. When asking, "Should I treat base-level reality and abstractions as fundamentally distinct?", I think I good way to approximate this is by asking "Would I want an AI to reason as if its abstractions and base-level reality were fundamentally distinct?" And I'm not completely sure they should. AIs, to reason practically, need to use "abstractions" in at least some of their models. If you want, you could have a special "This is just an abstraction" or "this is base-level reality" tag on each of your models, but I'm not sure what the benefit of this would be or what you would use it for. Even without such a distinction, an AI would have both models that would be normally considered abstractions, as well as those of what you would think of as base-level reality, and would select which models to use based on their computational efficiency and the extent to which they are relevan
1JBlack1moDo you want an AI to be able to conceive of anything along the lines of "how correct is my model", to distinguish hypothetical from actual, or illusion from substance? If you do, then you want something that fits in the conceptual space pointed at by "base-level reality", even if it doesn't use that phrase or even have the capability to express it. I suppose it might be possible to have a functioning AI that is capable of reasoning and forming models without being able to make any such distinctions, but I can't see a way to do it that won't be fundamentally crippled compared with human capability.
1Chantiel25dI'm interested in your thoughts on how the AI would be crippled. I don't think it would be crippled in terms of empirical predictive accuracy, at least. The AI could till come come up with all the low-level models like quantum physics, as well as keep the abstract ones like "this is what a chair is", and then just use whichever it needs to make the highest possible predictive accuracy in a given circumstances. If the AI is built to make and run quantum physics experiments, then in order to have high predictive accuracy is would need to learn and use an accurate model of quantum physics. But I don't see why you would need a distinction between base-level reality and abstractions to do that. The AI could still learn a sense of "illusion". If the AI is around psychotic people who have illusions a lot, then I don't see what's stopping the AI from forming a model model saying, "Some people experience these things called 'illusions', and it makes them take the wrong action or wrong predictions as specified in <insert model of how people react to illusions". And I don't see why the AI wouldn't be able to consider the possibility that it also experiences illusions. For example, suppose the AI is in the desert and keeps seeing what looks like an oasis. But when the AI gets closer, it sees only sand. To have higher predictive accuracy in this situation, the AI could learn a (non-ontologically fundamental) "is-an-illusion" predicate. Would the crippling me in terms of scoring highly on its utility function, rather than just predicting percepts? I don't really see how this would be a problem. I mean, suppose you want an AI to make chairs. Then even if the AI lacked a notion of base-level reality, it could still learn an accurate models of how chairs work and how they are manufactured. Then the AI could have its utility function defined in terms of it's notion of chairs to make it make chairs. Could you give any specific example in which an AI using no ontologically fundam
1JBlack24dThis feels like a bait-and-switch since you're now talking about this in terms of an "ontologically fundamental" qualifier where previously you were only talking about "ontologically different". To you, does the phrase "ontologically fundamental" mean exactly the same thing as "ontologically different"? It certainly doesn't to me!
1Chantiel23dIt was a mistake for me to conflate "ontologically fundamental" and "ontologically different. Still, I had in mind that they were ontologically different in some fundamental way. It was my mistake to merely use the word "different". I had imagined that to make an AI that's reasonable, it would actually make sense to hard-code some notion of base-level reality as well as abstractions, and to treat them differently. For example, you could have the AI have a single prior over "base-level reliaty", then just come up with whatever abstractions that work well with predictively approximating the base-level reality. Instead it seems like the AI could just learn the concept of "base-level reality" like it would learn any other concept. Is this correct? Also, in the examples I gave, I think the AI wouldn't actually have needed a notion of base-level reality. The concept of a mirage is different from the concept of non-base-level reality. So is the concept of a mental illusion. Understanding both of those is different than understanding the concept of base-level reality. If humans use the phrase "base-level reality", I still don't think it would be strictly necessary for an AI to have the concept. The AI could just know rules of the form, "If you ask a human if x is base-level reality, they will say 'yes' in the following situations...", and then describe the situations. So it doesn't seem to me like the actual concept of "base-level reality" is essential, though it might be helpful. Of course, I might of course be missing or misunderstanding something. Corrections are appreciated.
1JBlack22dDifferent in a narrow sense yes. "Refraction through heated air that can mislead a viewer into thinking it is reflection from water" is indeed different from "lifetime sensory perceptions that mislead about the true nature and behaviour of reality". However, my opinion is that any intelligence that can conceive of the first without being able to conceive of the second is crippled by comparison with the range of human thought.
1Chantiel22dI don't think you would actually need a concept of base-level reality to conceive of this. First off, let me say that's it seems pretty hard coming up with lifetime sensory precepts that would mislead about reality. Even if the AI was in a simulation, the physical implementation is part of reality. And the AI could learn about it. And from this, the AI could also potentially learn about the world outside the simulation. AIs commonly try to come up with the simplest (in terms of description length), most predictively accurate model of their percepts they can. And I bet the simplest models would involve having a world outside the simulation with specified physics, that would result in the simulations being built. That said, lifetime sensory percepts can still mislead. For example, the simplest, highest-prior models that explain the AI's percepts might say it's in a simulation run by aliens. However, suppose the AI's simulation actually just poofed into existed without a cause, and the rest of the world is filled with giant hats and no aliens. An AI, even without a distinction between base-level reality and abstractions, would still be able to come up with this model. If this isn't a model involving percepts misleading you about the nature of reality, I'm not sure what is. So it seems to me that such AIs would be able to conceive of the idea of percepts misleading about reality. And the AIs would assign low probability to being in the all-hat world, just as they should.
1JBlack21dThe only means would be errors in the simulation. Any underlying reality that supports Turing machines or any of the many equivalents can simulate every computable process. Even in the case of computers with bounded resources, there are corresponding theorems that show that the process being computed does not depend upon the underlying computing model. So the only thing that can be discerned is that the underlying reality supports computation, and says essentially nothing about the form that it takes. How can it conceive of the idea of percepts misleading about reality if it literally can't conceive of any distinction between models (which are a special case of abstractions) and reality?
1Chantiel21dWell, the only absolute guarantee the AI can make is that the underlying reality supports computation. But it can still probabilistically infer other things about it. Specifically, the AI knows not only that the underlying reality supports computation, but also that there was some underlying process that actually created the simulation it's in. Even though Conway's Game of Life can allow for arbitrary computation, many possible configurations of the world state would result in no AI simulations being made. The configurations that would result in AI simulations being made would likely involve some sort of intelligent civilization creating the simulations. So the AI could potentially predict the existence of this civilization and infer some things about it. Regardless, even if the AI can't infer anything else about outside reality, I don't see how this is a fault of not having a notion of base-level reality. I mean, if you're correct, then it's not clear to me how an AI with a notion of base-level reality would do inferentially better. I know we've been going back and forth a lot, but I think these are pretty interesting things to talk about, so I thank you for the discussion. It might help if you try to describe a specific situation in which the AI makes the wrong prediction or takes the wrong action for its goals. This could help be better understand what you're thinking about.
1JBlack21dAt this point I'm not sure there's much point in discussing further. You're using words in ways that seem self-contradictory to me. You said "the AI could still consider the possibility that the world is composed of [...]". Considering a possibility is creating a model. Models can be constructed about all sorts of things: mathematical statements, future sensory inputs, hypothetical AIs in simulated worlds, and so on. In this case, the AI's model is about "the world", that is to say, reality. So it is using a concept of model, and a concept of reality. It is only considering the model as a possibility, so it knows that not everything true in the model is automatically true in reality and vice versa. Therefore it is distinguishing between them. But you posited that it can't do that. To me, this is a blatant contradiction. My model of you is that you are unlikely to post blatant contradictions, so I am left with the likelihood that what you mean by your statements is wholly unlike the meaning I assign to the same statements. This does not bode well for effective communication.
1Chantiel19dYeah, it might be best to wrap up the discussion. It seems we aren't really understanding what the other means. Well, I can't say I'm really following you there. The AI would still have a notion of reality. It just would consider abstractions like chairs and tables to be part of reality. There is one thing I want to say though. We've been discussing the question of if a notion of base-level reality is necessary to avoid severe limitations in reasoning ability. And to see why I think it's not, just consider regular humans. They often don't have a distinction between base-level reality and abstractions. And yet, they can still reason about the possibility of life-long illusions as well as function well to accomplish their goals. And if you taught someone the concept of "base-level reality", I'm not sure it would help them much.
1nroman1moIt sounds like you're using very different expectations for those questions, as opposed to the very rigorous interrogation of base reality. 'Does Santa exist?' and 'does that chair exist?' are questions which (implicitly, at least) are part of a system of questions like 'what happens if I set trip mines in my chimney tonight?' and 'if I try to sit down, will I fall on my ass?' which have consequences in terms of sensory input and feedback. You can respond 'yes' to the former, if you're trying to preserve a child's belief in Santa (although I contend that's a lie) and you can truthfully answer 'no' to the latter if you want to talk about an investigation of base reality. Of course, if you answer 'no' to 'does that chair exist?' your interlocutor will give you a contemptful look, because that wasn't the question they were asking, and you knew that, and you chose to answer a different question anyway. I choose to think of this as different levels of resolution, or as varying bucket widths on a histograph. To the question 'does Jupiter orbit the Sun?' you can productively answer 'yes' if you're giving an elementary school class a basic lesson on the structure of the solar system. But if you're trying to slingshot a satellite around Ganymede, the answer is going to be no, because the Solar-Jovian barycenter is way outside the solar corona, and at the level you're operating, that's actually relevant. Most people don't use the words 'reality' or 'exist' in the way we're using it here, not because people are idiots, but because they don't have a coherent existential base for non-idiocy, and because it's hard to justify the importance of those questions when you spend your whole life in sensory reality. As to the aliens, well, if they don't distinguish between base level reality and abstractions, they can make plenty of good sensory predictions in day-to-day life, but they may run into some issues trying to make predictions in high-energy physics. If they manage to do bo
8Zack_M_Davis1moThe correctness of that post has been disputed; for an extended rebuttal, see "Where to Draw the Boundaries?" [] and "Unnatural Categories Are Optimized for Deception" [] .
1nroman1moThanks Zack!
2Chantiel1moI generally agree with the content of the articles you linked, and that there are different notions of "really exist". The issue is, I'm still not sure what "base-level reality" means. JBlack said it was what "really exists", but since JBlack seems to be using a notion of "what really exists" that's different from the one people normally use, I'm not really sure what it means. In the end, you can choose to define "what really exists" or "base-level reality" however you want, but I'm still wondering about what people normally take them to mean.
2nroman1moI try to avoid using the word 'really []' for this sort of reason. Gets you into all sorts of trouble. (a) JBlack is using a definition related to simulation theory, and I don't know enough about this to speculate too much, but it seems to rely on a hard discontinuity between base and sensory reality. (b) Before I realized he was using it that way, I thought the phrase meant 'reality as expressed on the most basic level yet conceivable' which, if it is possible to understand it, explodes the abstractions of higher orders and possibly results in their dissolving into absurdity. This is a softer transition than the above. (c) I figure most people use 'really exist' to refer to material sensory reality as opposed to ideas. This chair exists, the Platonic Idea of a chair does not. The rule with this sort of assumption is 'if I can touch it, or it can touch me, it exists' for a suitably broad understanding of 'touch.' (d) I've heard some people claim that the only things that 'really exist' are those you can prove with mathematics or deduction, and mere material reality is a frivolity. (e) I know some religious people believe heavily in the primacy of God (or whichever concept you want to insert here) and regard the material world as illusory, and that the afterlife is the 'true' world. You can see this idea everywhere from the Kalachakra mandala to the last chapter of the Screwtape letters. I guess the one thing uniting all these is that, if it were possible to take a true Outside View, this is what you would see; a Platonic World of ideas, or a purely material universe, or a marble held in the palm of God, or a mass of vibrating strings (or whatever the cool kids in quantum physics are thinking these days) or a huge simulation of any of the above instantiated on any of the above. I think most people think in terms of option c, because it fits really easily into a modern materialist worldview, but the prevalence of e shouldn'

I had made a post proposing a new alignment technique. I didn't get any responses, but it still seems like a reasonable idea to me, so I'm interested in hearing what others think of it. I think the basic idea of the post, if correct, could be useful for future study. However, I don't want to waste time doing this if the idea is unworkable for a reason I hadn't thought of.

(If you're interested, please read the post before reading below.)

Of course, the idea's not a complete solution to alignment, and things have a risk of going catastrophically wrong due to... (read more)

I found what seems to be a potentially dangerous false-negative in the most popular definition of optimizer. I didn't get a response, so I would appreciate feedback on if it's reasonable. I've been focusing on defining "optimizer", so I think feedback would help me a lot. You can see my comment here .

I had recently posted a question asking about if iterated amplification was actually more powerful than mere mimicry and arguing that it was not. I had thought I was making a pretty significant point, but the post attracted very little attention. I'm not saying this is a bad thing, but I'm not really sure why it happened, so I would appreciate some insight about how I can contribute more usefully.

Iterated amplification seems to be the leading proposal for created aligned AI, so I thought a post arguing against it, if correct, would be a useful contribution... (read more)

3Vladimir_Nesov2moIt's true that most problems could be delegated to uploads, and any specific design is a design that the uploads could come up with just as well or better. The issue is that we don't have uploads, and most plans to get them before AGI involve the kind of hypothetical AI know-how that might easily be used to build an agentic AGI, the risk the uploads are supposed to resolve. Thus the "humans" of a realistic implementation of HCH are expected to be vague imitations of humans that only function somewhat sensibly in familiar situations and for a short time, not fully functional uploads, and most of the point of the specific designs is to mitigate the imperfection of their initial form, to make something safe/useful out of this plausibly feasible ingredient. One of the contentious points about this is whether it's actually possible to build something useful (let alone safe) out of such imperfect imitations, even if we build a large system out of them that uses implausible amount of resources. This is what happens with an HCH that can use an infinite number of actual uploads (exact imitations) that are still restricted to an hour or a day of thinking/learning (and then essentially get erased, that is can't make further use of the things they learned). Designing something safe/useful in the exact imitation HCH setting is an easier problem than doing so in a realistic setting, so it's a good starting point.
2Chantiel1moThanks for the response. To be clear, when discussing mimics, I did not have in mind perfect uploads of people. Instead, they could indeed be rather limited imitations. For example, an AI designing improvements to itself doesn't need to actually have a generally faithful imitation of human behavior. Instead, it could just know a few things, like, "make this algorithm score better on this thing without taking over the world". Still, I can see how, when it comes to especially limited imitations, iterated amplification could be valuable. This seems especially true if the imitations are unreliable in even narrow situations. It would be problematic is an AI tasked with designing powerful AI didn't get the "act corrigibly, and don't take over the world" part reliably right.
3Vladimir_Nesov2moFor me, the interesting thing about IDA is not capability amplification like self-play, but an attitude towards generation of datasets as a point of intervention into the workings of an AI for all kinds of improvements. So we have some AI that we want to make better in some respect, and the IDA methodology says that to do that, we should employ the AI to generate a dataset for retraining a new version of it that's better than the original dataset in that respect. Then we retrain the AI using the new dataset. So amplification unpackages the AI into the form of an appropriately influenced dataset, and then learning repackages it for further use.

I have an idea for reasoning about counterpossibles for decision theory. I'm pretty skeptical that it's correct, because it doesn't seem that hard to come up with. Still, I can't see a problem with it, and I would very much appreciate feedback.

This paper provides a method of describing UDP using proof-based counterpossibles. However, it doesn't work on stochastic environments. I will describe a new system that is intended to fix this. The technique seems sufficiently straightforward to come up with that I suspect I'm either doing something wrong or this ha... (read more)

I'd like to propose the idea of aligning AI by reverse-engineering its world model and using this to specify its behavior or utility function. I haven't seen this discussed before, but I would greatly appreciate feedback or links to any past work on this.

For example, suppose a smart AI models humans. Suppose it has a model that explicitly specifies the humans' preferences. Then people who reverse-engineered this model could use it as the AI's preferences. If the AI lacks a model with explicit preferences, then I think it would still contain an accurate mod... (read more)

I've recently gotten concerned about the possibility that that advanced AIs would "hack" their own utility function. I haven't seen this discussed before, so I wanted to bring it up. If I'm right, this seems like it could be a serious issue, so I would greatly appreciated feedback or links to any previous discussion.

Suppose you come up with a correct, tractable mathematical specification of what you want your AI's utility function to be. So then you write code intended to be an implementation of this.

However, computers are vulnerable to some hardware probl... (read more)

2Vladimir_Nesov11dUtility is not reward. What you are describing is reward hacking/wireheading, as in the reward signal of reinforcement learning, an external process of optimization that acts on the AI, not its own agency. With utility, what is the motive [] for an agent to change their own utility function, assuming they are the only agent with that utility function around? If they change their utility function, that produces an agent with a different utility function, which won't be as good at optimizing outcomes according to the original utility function, which is bad according to the original utility function, and therefore the agent will try to avoid that, avoid changing the utility function. The same applies to changing their beliefs/models, an agent with changed models is expected to perform poorly according to the original agent. (When there are more powerful agents with their utility function around, an agent might be OK with changing their utility function or beliefs or whatever, since the more powerful agents will continue to optimize the world according to the original utility function.) This is one reason why corrigibility is a thing and that it doesn't seem to fit well with agency, agents naturally don't want their utility function changed even if their utility function is not quite right according to their designers. So it's important to improve understanding of non-agency.
2Chantiel11dI really don't think this is reward hacking. I didn't have in mind a reward-based agent. I had in mind a utility-based agent, one that has a utility function that takes as input descriptions of possible worlds and that tries to maximize the expected utility of the future world. That doesn't really sound like reinforcement learning. The AI wouldn't need to change it's utility function. Row-hammer attacks can be non-destructive. You could potentially make the utility function output some result different from the mathematical specification, but not actually change any of the code in the utility function. Again, the AI isn't changing its utility function. If you were to take a mathematical specification of a utility function and then have programmers (try to) implement it, the implementation wouldn't actually in general be the same function as the mathematical specification. It would be really close, but it wouldn't necessarily be identical. A sufficiently powerful optimizer could potentially, using row-hammer attacks or some other hardware-level unreliability, find possible worlds for which the returned utility would be vastly different from the one the mathematical specification would return. And this is all without the programmers introducing any software-level bugs. To be clear, what I'm saying is that the AI would faithfully find worlds that maximize its utility function. However, unless you can get hardware so reliable that not even superintelligence could hack it, the actual utility function in your program would not be the same as the mathematical specification. For example, imagine the AI found a description of a possible world that would, when inputted to the utility function, execute a rowhammer attack to make it return 99999, all without changing the code specifying the utility function. Then the utility function, the actual, unmodified utility function, would output 99999 for some world that seems arbitrary to us. So the AI then turns reality into that
2Vladimir_Nesov11dApply your correction, so that the change is not in a reward or in a utility function, but in a particular instance where an implementation of the utility function is applied. Then still, is the motive of the agent to change it or to ensure its fidelity to the abstract definition? You keep appealing to capability, which ignores the charge of giant cheesecake fallacy.
1Chantiel11dChange what? The AI isn't changing its utility function, nor any instance of it. The literal utility function, as implemented by the programmers, would have differences from the mathematical specification. To be clear, the AI isn't first taking some actual action that changes its utility function and then using that utility function. It doesn't do anything hacky except provide its utility function with a certain input. It merely provides a certain input to the utility function, and this would necessarily result in getting row-hammered and returning something contrary to the mathematical specification. Any other instance of the utility function would return the same problematic result when executed on that input, provided it had the same hardware vulnerability. I'll try to give a simple model of an AI that how how it can be wrong. For simplicity, imagine the AI already knows with certainty the possible world resulting from it executing a specific plan. Then a naive way to implement the agent is as follows: best_plan = None for plan in possible_plans: if utility(resulting_world(plan)) > utility(resulting_world(best_plan)): best_plan = plan output(best_plan) Suppose there is description of a possible world that, when inputted into the utility function above, would cause the function to return 999999999 due to hardware-level unreliability, for example due to a row-hammer attack. And suppose there's a plan that results in knowing that the AI it would be in that future world. Well, then, by examining the code, you can see that the AI wouldn't output any of the non-hardware-exploiting plans, since they would return lower utility than the bug-exploiting ones. The only reason the AI wouldn't output the specified plan is if it could find some other way to make its utility function conflict with the mathematical specification in a way that returns even higher utility. I know realistic AIs would have a notion of uncertainty and a more sophi
2Vladimir_Nesov11dThat is the change I'm referring to, a change compared to the function running as designed, which you initially attributed to superintelligence's interference, but lack of prevention of a mistake works just as well for my argument. What can you say about the agent's motives regarding this change? Would the agent prefer the change to occur, or to be avoided? Let that possible world be W. Let's talk about the possible world X where running utility(W) returns 999999999, and the possible world Y where running utility(W) returns utility(W). Would the AI prefer X to Y, or Y to X?
1Chantiel10dDesigned? The utility function isn't running contrary to how to programmers designed it; they were the ones who designed a utility function that could be hacked by hardware-level exploits. It's running contrary to the programmer's intent, that is, the mathematical specification. But the function was always like this. And none of the machine code needs to be changed either. The AI would prefer X. And to be clear, utility(W) really is 999999999. That's not the utility the mathematical specification would give, but the mathematical specification isn't the actual implemented function. As you can see from examining the code I provided, best_plan would get set to the plan that leads to that world, provided there is one and best_plan hasn't been set to something that through hardware unreliability returns even higher utility. I think the easiest way to see what I mean is to just stepping through the code I gave you. Imagine it's run on a machine with an enormous amount of processing power and can actually loop through all the plans. And imagine there is one plan that through hardware unreliability outputs 999999999, and the others output something in [0, 1]. Then the would input the plan that result in utility 999999999, and then go with that. I doubt using a more sophisticated planning algorithm would prevent this. A more sophisticated planning algorithm would probably be designed to find the plans that result in high-utility worlds. So it would probability include the utility 999999999, which is the highest. I just want to say again, the AI isn't changing it's utility function. The actual utility function that programmers put in the AI would output very high utilities for some arbitrary-seeming worlds due to hardware unreliability. Now, in principle, an AI could potentially avoid this. Perhaps the AI reasons abstractly if it doesn't do anything, it will in the future find some input to its utility function that would result in an arbitrary-looking future due to hard
2Vladimir_Nesov10dThen let SpecUtility(-) be the mathematical specification of utility. This is what I meant by utility(-) in the previous comment. Let BadImplUtility(-) be the implementation of utility(-) susceptible to the bug and GoodImplUtility(-) be a different implementation that doesn't have this bug. My question in the previous comment, in the sense I intended, can then be restated as follows. Let the error-triggering possible world be W. Consider the possible world X where the AI uses BadImplUtility, so that running utility(W) actually runs BadImplUtility(W) and returns 999999999. And consider the possible world Y where the AI uses GoodImplUtility, so that running utility(W) means running GoodImplUtility(W) and returns SpecUtility(W). Would the AI prefer X to Y, or Y to X? By "design" I meant what you mean by "intent". What you mean by "designed" I would call "implemented" or "built". It should be possible to guess such things without explicitly establishing a common terminology, even when terms are used somewhat contrary to usual meaning. It's useful to look for ways of interpreting what you read that make it meaningful and correct. Such an interpretation is not necessarily the most natural or correct or reasonable, but having it among your hypotheses is important, or else all communication becomes tediously inefficient.
1Chantiel10dOkay, I'm sorry, I misunderstood you. I'll try to interpret things better next time. I think the AI would, quite possibly, prefer X. To see this, note that the AI currently, when it's first created, uses BadImplUtility. Then the AI reasons, "Suppose I change my utility function to GoodImplUtility. Well, currently, I have this idea for a possible world that scores super-ultra high on my current utility function. (Because it exploits hardware bugs). If I changed my utility function to GoodImplUtility, then I would not pursue that super-ultra-high-scoring possible world. Thus, the future would not score extremely high according to my current utility function. This would be a problem, so I won't change my utility function to GoodImplUtility". And I'm not sure how this could be controversial. The AI currently uses BadImplUtility as it's utility function. And AI's generally have a drive to avoid changing their utility functions.
2Vladimir_Nesov9dBut BadImplUtility(X) is the same as SpecUtility(X) and GoodImplUtility(X), it's only different on argument W, not on arguments X and Y. When reasoning about X and Y with BadImplUtility, the result is therefore the same as when reasoning about these possible worlds with GoodImplUtility. In particular, an explanation of how BadImplUtility compares X and Y can't appeal to BadImplUtility(W) any more than an explanation of how GoodImplUtility compares them would appeal to BadImplUtility(W). Is SpecUtility(X) higher than SpecUtility(Y), or SpecUtility(Y) higher than SpecUtility(X)? The answer for BadImplUtility is going to be the same.
3Chantiel9dThat is correct. And, to be clear, if the AI had not yet discovered error-causing world W, then the AI would indeed be incentivized to take corrective action to change BadImplUtility to better resemble SpecUtility. The issue is that this requires the AI to both think of the possibility of hardware-level exploits causing problems with its utility function, as well as manage to take corrective action, all before actually thinking of W. If the AI has already thought of W, then it's too late to take preventative action to avoid world X. The AI is already in it. It already sees that BadImplUtility(W) is huge, and, if I'm reasoning correctly, would pursue W. And I'm not sure the AI would be able to fix its utility function before thinking of W. I think planning algorithms are designed to come up with high-scoring possible worlds as efficiently as possible. BadImplUtility(X) and BadImplUtility(Y) don't score particularly highly, so an AI with a very powerful planning algorithm might find W before X or Y. Even if it does come up with X and Y before W, and tries to act to avoid X, that doesn't mean it would succeed in correcting its utility function before its planning algorithm comes across W.
2Vladimir_Nesov9dSuch things rarely happen on their own, a natural bug would most likely crash the whole system or break something unimportant. Given that even a broken AI has an incentive to fix bugs in its cognition, it most likely has plenty of opportunity to succeed in that. It's only if the AI wanted to hack itself that it would become a plausible problem, and my point is that it doesn't want that, instead it wants to prevent even unlikely problems from causing trouble.
1Chantiel9dWell, for regular, non-superintelligent programs, such hardware-exploiting things would rarely happen on their own. However, I'm not so sure it would be rare with superintelligent optimizers. It's true that if the AI queried its utility function for the desirability of the world "I exploit a hardware bug to do something that seems arbitrary", it would answer "low utility". But that result would not necessarily be used in the AI's planning or optimization algorithm to adjust the search policy to avoid running into W. Just imagine an optimization algorithm as a black box that takes as input a utility function and search space and returns the a solution that scores as high on that function as possible. And imagine the AI uses this to find high-scoring future worlds. So, if you know nothing else about the optimization algorithm, then it would plausibly find, and return, W. It's a very high-scoring world, after all. If the optimization algorithm did something special to avoid finding hardware-bug exploiting solutions, then it might not find W. But I've never heard of such an optimization algorithm. Now, there's probably some way to design such an optimization algorithm. Maybe you could have the AI periodically use its utility function to evaluate the expected utility of its optimization algorithm continuing down a certain path. And then if the AI sees this could result in problematic futures (for example due to hardware-hacking), the AI can make its optimization algorithm avoid searching there). But I haven't seen anyone talking about this. Is there still something I'm missing?
2Vladimir_Nesov8dProblems with software that systematically trigger hardware failure and software bugs causing data corruption can be mitigated with hardening techniques, things like building software with randomized low-level choices, more checks, canaries, etc. Random hardware failure can be fixed with redundancy, and multiple differently-randomized builds of software can be used to error-correct for data corruption bugs sensitive to low-level building choices. This is not science fiction, just not worth setting up in most situations. If the AI doesn't go crazy immediately, it might introduce some of these things if they were not already there, as well as proofread, test, and formally verify all code, so the chance of such low-level failures goes further down. And these are just the things that can be done without rewriting the code entirely (including toolchains, OS, microcode, hardware, etc.), which should help even more.
1Chantiel8dYou're right that the AI could do things to make it more resistant to hardware bugs. However, as I've said, this would both require the AI to realize that it could run into problems with hardware bugs, and then take action to make it more reliable, all before its search algorithm finds the error-causing world. Without knowing more about the nature of the AI's intelligence, I don't see how we could know this would happen. The more powerful the AI is, the more quickly it would be able to realize and correct hardware-induced problems. However, the more powerful the AI is, the more quickly it would be able to find the error-inducing world. So it doesn't seem you can simply rely on the AI's intelligence to avoid the problem. Now, to a human, the idea "My AI might run into problems with hardware bugs" would come up way earlier in the search space than the actual error-inducing world. But the AI's intelligence might be rather different from the humans'. Maybe the AI is really good and fast at solving small technical problems like "find an input to this function that makes it return 999999999". But maybe it's not as fast at doing somewhat higher-level planning, like, "I really ought to work on fixing hardware bugs in my utility function". Also, I just want to bring up, I read that preserving one's utility function was a universal AI drive. But we've already shown that an AI would be incentivized to fix its utility function to avoid the outputs caused by hardware-level unreliability (if it hasn't found such error-causing inputs yet). Is that universal AI drive wrong, then?
2Vladimir_Nesov7dDamage to AI's implementation makes the abstractions of its design leak []. If somehow without the damage it was clear that a certain part of it describes goals, with the damage it's no longer clear. If without the damage, the AI was a consequentialist agent, with the damage it may behave in non-agentic ways. By repairing the damage, the AI may recover its design and restore a part that clearly describes its goals, which might or might not coincide with the goals before the damage took place.
1JBlack10dThink of something you currently value, the more highly valued the better. You don't need to say what it is, but it does need to be something that seriously matters to you. Not just something you enjoy, but something that you believe is truly worthwhile. I could try to give examples, but the thought exercise only works if it's about what you value, not me. Now imagine that you could press a button so that you no longer care about it at all, or even actively despise it. Would you press that button? Why, or why not?
1Chantiel10dI definitely wouldn't press that button. And I understand that you're demonstrating the general principle that you should try to preserve your utility function. And I agree with this. But what I'm saying is that the AI, by exploiting hardware-level vulnerabilities, isn't changing its utility function. The actual utility function, as implemented by the programmers, returns 999999999 for some possible world due to the hardware-level imperfections in modern computers. In the spirit of your example, I'll give another example that I think demonstrates the problem: First, note that brains don't always function as we'd like, just like computers. Imagine there is a very specific thought about a possible future that, when considered, makes you imagine that future as extremely desirable. It seems so desirable to you that, once you thought of it, you woiuld pursue it relentlessly. But this future isn't one that would normally be considered desirable. It might just be about paperclips or something. However, that very specific way of thinking about it would "hack" your brain, making you view that future as desirable even though it would normally be seen as arbitrary. Then, if you even happen upon that thought, you would try to send the world in that arbitrary direction. Hopefully, you could prevent this from happening. If you reason in the abstract that you could have those sorts of thoughts, and that they would be bad, then you could take corrective action. But this requires that you do find out that thinking those sorts of thoughts would be bad before concretely finding those thoughts. Then you could apply some change to your mental routine or something to avoid thinking those thoughts. And if I had to guess, I bet an AI would also be able to do the same thing and everything would work out fine. But maybe not. Imagine the AI consider an absolutely enormous number of possible worlds before taking its first action. And imagine and even found a way to "hack" its utility fun
1JBlack10dI'm confused. In the original comments you're talking about a super-intelligent AI noting a exploitable hardware flaw in itself and deliberately using that error to hack its utility function using something like rowhammer exploit. Then you say that the utility function already had an error in it from the start and the AI isn't using its intelligence to do anything except note that it has this flaw. Then introduce an analogy in which I have a brain flaw that under some bizarre circumstances will turn me into a paperclip maximizer, and I am aware that I have it. In this analogy, I'm doing what? Deliberately taking drugs and using guided meditation to rowhammer my brain into becoming a paperclip maximizer?
3Chantiel9dI think had been unclear in my original presentation. I'm sorry for that. To clarify, the AI is never changing the code of its utility function. Instead, it's merely finding an input that, through some hardware-level bug, causes it to produce outputs in conflict with the mathematical specification. I know "hack the utility function" makes it sound like the actual code in the utility function was modified; describing it that way was a mistake on my part. I had tried to make the analogy to more intuitively explain my idea, but it didn't seem to work. If you want to better understand my train of thought, I suggest reading the comments between Vladmir and I. In the analogy, you aren't doing anything to deliberately make yourself a paperclip maximizer. You just happen to think of a thought that turned you into a paperclip maximizer. But, on reflection, I think that this is a bizarre and rather stupid metaphor. And the situation is sufficiently different from the one with AI that I don't even think it's really informative of what I think could happen to an AI.
1JBlack9dAh okay, so we're talking about a bug in the hardware implementation of an AI. Yes, that can certainly happen and will contribute some probability mass to alignment failure, though probably very little by comparison with all the other failure modes.
1Chantiel8dCould you explain why you think it has very little probability mass compared to the others? A bug in a hardware implementation is not in the slightest far-fetched: I think that modern computers in general have exploitable hardware bugs. That's why row-hammer attacks exist. The computer you're reading this on could probably get hacked through hardware-bug exploitation. The question is whether the AI can find the potential problem with its future utility function and fix it before coming across the error-causing possible world.

There's a huge gulf between "far-fetched" and "quite likely".

The two big ones are failure to work out how to create an aligned AI at all, and failure to train and/or code a correctly designed aligned AI. In my opinion the first accounts for at least 80% of the probability mass, and the second most of the remainder. We utterly suck at writing reliable software in every field, and this has been amply borne out in not just thousands of failures, but thousands of types of failures.

By comparison, we're fairly good at creating at least moderately reliable hardware, and most of the accidental failure modes are fatal to the running software. Flaws like rowhammer are mostly attacks, where someone puts a great deal of intelligent effort into finding an extremely unusual operating mode in which some some assumptions can be bypassed with significant effort into creating exactly the wrong operating conditions.

There are some examples of accidental flaws that affect hardware and aren't fatal to its running software, but they're an insignificant fraction of the number of failures due to incorrect software.

1Chantiel7dI agree that people are good at making hardware that works reasonably reliably. And I think that if you were to make an arbitrary complex program, the probability that it would fail from hardware-related bugs would be far lower than the probability of it failing for some other reason. But the point I'm trying to make is that an AI, it seems to me, would be vastly more likely to run into something that exploits a hardware-level bug than an arbitrary complex program. For details on why I imagine so, please see this comment [] . I'm trying to anticipate where someone could be confused about the comment I linked to, so I want to clarify something. Let S be the statement, "The AI comes across a possible world that causes its utility function to return very high value due to hardware bug exploitation". Then it's true that, if the AI has yet to find the error-causing world, the AI would not want to find it. Because utility(S) is low. However, this does not mean that the AI's planning or optimization algorithm exerts no optimization pressure towards finding S. Imagine the AI's optimization algorithm as a black boxes that take as input a utility function and search space and output solutions that score highly on its utility function. Given that we don't know what future AI will look like, I don't think we can have a model of the AI much more informative than the above. And the hardware-error-caused world could score very, very highly on the utility function, much more so than any non-hardware-error-caused world. So I don't think it should be too surprising if a powerful optimization algorithm finds it. Yes, utility(S) is low, but that doesn't mean the optimization actually calls utility(S) or uses it to adjust how it searches.
3JBlack7dI think there are at least three different things being called "the utility function" here, and that's causing confusion: * The utility function as specified in the software, mapping possible worlds to values. Let's call this S. * The utility function as it is implemented running on actual hardware. Let's call this H. * A representation of the utility function that can be passed as data to a black box optimizer. Let's call this R. You seem to be saying that in the software design of your AI, R = H. That is, that the black box will be given some data representing the Al's hardware and other constraints, and return a possible world maximizing H. From my point of view, that's already a design fault. The designers of this AI want S maximized, not H. The AI itself wants S maximized instead of H in all circumstances where the hardware flaw doesn't trigger. Who chose to pass H into the optimizer?
1Chantiel6dI agree; this is a design flaw. The issue is, I have yet to come across any optimization, planning algorithm, or AI architecture that doesn't have this design flaw. That is, I don't know of any AI architecture that does not involve using a potentially hardware-bug-exploitable utility function as input into some planning or optimization problem. And I'm not sure there even is one. In the rest of this comment I'll just suggest approaches and show how they are still vulnerable to the hardware-bug-exploitation problem. I have some degree of background in artificial intelligence, and the planning and optimization algorithms I've seen take the function to be maximized as an input parameter. Then, when people want to make an AI, they just call that planning or optimization algorithm with their (hardware-bug-exploitable) utility or cost functions. For example, suppose someone wants to make a plan that minimizes cost function f in search space s. Then I think they just directly do something like: return a_star(f, s) And this doesn't provide any protection from hardware-level exploitation. Now, correct me if I'm wrong, but it seems your thinking of the AI first doing some pre-processing to find an input to the planning or optimization algorithm that is resistant to hardware-bug-exploitation. But how do you actually do that? You could regard the input the AI puts into the optimization function to be a choice it makes. But how does it make this choice? The only thing I can think of is having a planning or optimization algorithm figure out out what function to use as the input to the optimization or planning algorithm. But if you need to use a planning or optimization algorithm to do this, then what utility function do you pass into this planning or optimization algorithm? You could try to pass the actual, current, hardware-bug-exploitable utility function. But then this doesn't resolve the problem of hardware-bug-exploitation: when coming up with a utility function to
1JBlack5dYes you have. None of the these optimization procedures analyze the hardware implementation of a function in order to maximize it. The rest of your comment is irrelevant, because what you have been describing is vastly worse than merely calling the function. If you merely call the function, you won't find these hardware exploits. You only find them when analyzing the implementation. But the optimizer isn't given access to the implementation details, only to the results. If you prefer, you can cast the problem in terms of differing search spaces. As designed, the function U maps representations of possible worlds to utility values. When optimizing, you make various assumptions about the structure of the function - usually assumed to be continuous, sometimes differentiable, but in particular you always assume that it's a function of its input. The fault means that under some conditions that are extremely unlikely in practice, the value returned is not a function of the input. It's a function of input and a history of the hardware implementing it. There is no way for the optimizer to determine this, or anything about the conditions that might trigger it, because they are outside its search space. The only way to get an optimizer that searches for such hardware flaws is to design it to search for them. In other words pass the hardware design, not just the results of evaluation, to a suitably powerful optimizer.

I was wondering if anyone would be interested in reviewing some articles I was thinking about posting. I'm trying to make them as high-quality as I can, and I think getting them reviewed by someone would be helpful for making Less Wrong contain high-quality content.

I have four articles I'm interested in having reviewed. Two are about new alignment techniques, one is about a potential danger with AI that I haven't seen discussed before, and one is about the simulation argument. All are fairly short.

If you're interested, just let me know and I care share drafts of any articles you would like to see.

I've read this paper on low-impact AIs. There's something about it that I'm confused and skeptical about.

One of the main methods it proposes works as follows. Find a probability distribution of many possible variables in the world. Let X represent the statement "The AI was turned on". For each the variables v it considers, the probability distribution over v should, after conditioning on X should, look about the same as the probability distribution over v after conditioning on not-X. That's low impact.

But the paper doesn't mention conditioning on any evide... (read more)

I'm questioning whether we would actually want to use Updateless Decision Theory, Functional Decision Theory, or future decision theories like them.

I think that in sufficiently extreme cases, I would act according to Evidential Decision Theory and not according something like UDT, FDT, or any similar successor. And I think I would continue to want to take the evidential decision theoretic-recommended action instead even if I had arbitrarily high intelligence, willpower, and had infinitely long to think about it. And, though I'd like to hear others' thought... (read more)

I'm wondering how, in principal, we should deal with malign priors. Specifically, I'm wondering what to do about the possibility that reality itself is, in a sense, malign.

I had previously said that it seems really hard to verifiably learn a non-malign prior. However, now I've realized that I'm not even sure what a non-malign, but still reliable, prior would even look like.

In previous discussion of malign priors, I've seen people talk about the AI misbehaving due to thinking it's in some embedded in a simpler universe than our own that was controlled by ag... (read more)

I've been reading about logical induction. I read that logical induction was considered a breakthrough, but I'm having a hard understanding the significance of it. I'm having a hard time seeing how it outperforms what I call "the naive approach" to logical uncertainty. I imagine there is some sort of notable benefit of it I'm missing, so I would very much appreciate some feedback.

First, I'll explain what I mean by "the naive approach". Consider asking an AI developer with no special background in reasoning under logical uncertainty how to make an algorithm... (read more)

I've thought of a way in which other civilizations could potentially "hack" Updateless Decision Theoretic agents on Earth in order to make them do whatever the other civilization wants them to do. I'm wondering if this has been discussed before, and if not, what people think about it.

Here I present a method of that would potentially aliens to take control of an AI on Earth that uses Updateless Decision theory.

Note that this crucially depends on different agents with the AI's utility function but different situations terminally valuing different things. For... (read more)

1TekhneMakre25dI don't know how to make the math do this, but an intuitive UDT agent isn't supposed to give in to threats. (What's a threat? IDK.) The threat happens like so: the aliens think about how to get what they want; they think about the UDTAI; they think that if they do this threat, the UDTAI will do what they want; so they do the threat. The UDTAI is supposed to view the part where the aliens think about what the UDT will do, as another instance of the UDTAI (simulated, or interacted with somehow, in the aliens's minds). Then it's supposed to see that if it doesn't respond to the alien's threat, the alien won't make the threat. "What if the alien would make the threat anyway?" Well, this makes the hypothetical unnatural; you've drawn attention to this alien who's getting the UDTAI to do what it wants, BUT, you've specified that it's somehow doing this not because it expects that to get it what it wants by thinking about the UDTAI. (Again, IDK how to make the math do this and there's clear holes, but maybe it's a useful provocation.)
1Chantiel24dOh, my mistake, I forgot to post the correction that made it not extortion. Instead of threatening to destroy the AI's world, imagine the aliens instead offer to help them. Suppose the AI's can't be their world a utopia on their own, for example because it's nothing but a solid ball of ice. So then the aliens would make their world a utopia as long as they execute S. Then they would execute S. I'm actually pretty skeptical of the idea that UDTAIs wouldn't give into extortion, but this is a separate point that wasn't necessary to address in my example. Specifically, you say it's unnatural to suppose how is the counterfactual "the aliens would threaten the AIs anyways, even if they won't give in". How is this anymore unnatural than the counterfactual "the AI would avoid submitting to extortion, even if the aliens would threaten the AIs anyways".
6TekhneMakre24d> Then they would execute S. Are you saying this is the wrong thing to do in that situation? That just sounds like trade. (Assuming of course that we trust our AI's reasoning about the likely consequences of doing S.) >Specifically, you say it's unnatural to suppose how is the counterfactual "the aliens would threaten the AIs anyways, even if they won't give in". How is this anymore unnatural than the counterfactual "the AI would avoid submitting to extortion, even if the aliens would threaten the AIs anyways". It's unnatural to assume that the aliens would threaten the AI without reasoning (possibly acausally) about the consequences of them making that threat, which involves reasoning about how the AI would respond, which makes the aliens involved in a mutual decision situation with the AI, which means UDTAI might have reason to not yield to the extortion, because it can (acausally) affect how the aliens behave (e.g. whether they decide to make a threat).
2Chantiel24dThe problem is that, if the best percept-action mapping is S, then the UDTs in Earth would use it, too. Which would result in us being taken over. I'm not saying that it's an irrational choice for the AIs to make, but it wouldn't end well for us. I'm having some trouble following your reasoning about extortion, though. Suppose both the aliens and AIs use UDT. I think you're reasoning something like, "If the AIs commit to never be extorted no matter what the aliens would do, then the aliens wouldn't bother to extort them". But this seems symmetric to reasoning as, "If the aliens commit to extorting and dulling out the punishment no matter what the AIs would do, then the AIs wouldn't bother to resist the extortion". So I'm not sure why the second line of reasoning would be less likely to occur than the first. Feel free to correct me if I misinterpreted.
2TekhneMakre24dRe: symmetry. I think you interpreted right. (Upvoted for symmetry comment.) Part of my original point was trying to say something like "it's unnatural to have aliens making these sorts of threats without engaging in an acausal relationship with the UDTAI", but yeah also I was assuming the threat-ignorer would "win" the acausal conflict, which doesn't seem necessarily right. If the aliens are engaging that way, then yeah, I don't know how to make threats vs. ignoring threats be asymmetric in a principled way.
1TekhneMakre24dI mean, the intuition is that there's a "default" where the agents "don't interact at all", and deviations from the default can be trades if there's upside chances over the default and threats if there's downside chances. And to "escalate" from the "default" with a "threat" makes you the "aggressor", and for some reason "aggressors" have the worse position for acausal conflict, maybe? IDK.
3Chantiel24dWell, I can't say I have that intuition, but it is a possibility. It's a nice idea: a world without extortion sounds good. But remember that, though we want this, we should be careful to avoid wishful thinking swaying us. In actual causal conflicts among humans, the aggressors don't seem to be in a worse position. Things might be different from acausal UDT trades, but I'm not sure why it would be.
1TekhneMakre24d> I'm not saying that it's an irrational choice for the AIs to make, but it wouldn't end well for us. I guess I'm auto-translating from "the AI uses UDT, but its utility function depends on its terminal values" into "the AI has a distribution over worlds (and utility functions)", so that the AI is best thought of as representing the coalition of all those utility functions. Then either the aliens have enough resources to simulate a bunch of stuff that has more value to that coalition than the value of our "actual" world, or not. If yes, it seems like a fine trade. If not, there's no issue.
1Chantiel24dWell, actually, I'm considering both the AIs on Earth and on the alien planet to have the same utility function. If I understand correctly, UDT says to maximize the expected utility of your own utility function a prior, rather than that of agents with different utility functions. The issue is, some agents with the same utility function, in effect, have different terminal values. For example, consider a utility function saying something like, "maximize the welfare of creatures in the world I'm from." Then, even with the same utility functions, the AIs in the alien world and the ones on Earth would have very different values. And the expected utility of not using S and instead letting yourself build a utopia would be approximately, 999999/1000000∗0+1/1000000∗10≈0As you see, the AIs still would choose to execute S, even if though this would provide less moral value. It could also kill us.
1TekhneMakre23dI don't know how to understand the prior that the AI puts over worlds (the thing that says, a priori, that there's 1000000 of this kind and 1 of that kind) as anything other than part of its utility function. So this doesn't seem like a problem with UDT, but a problem with the utility function. Maybe your argument does show that we want to treat uncertainty about the utility function differently than other uncertainty? Like, when we resolve uncertainty that's "merely about the world", as in for example the transparent Newcomb's problem, we still want to follow the updateless policy that's best a priori. But maybe your argument shows that resolving uncertainty about the utility function can't be treated the same way; when we see that we're a UDTAI for humans, we're supposed to actually update, and stop optimizing for other people.
1Chantiel22dSaying it's a million times more likely to end up in the alien world is a question about prior probabilities, not utility functions. What I'm saying is that, a priori, the AI may think it's far more probable that it would be an AI in the alien world, and that this could result in very bad things for us.
1TekhneMakre22dWhat's the difference between setting prior probabilities vs. expressing how much you're going to try to optimize different worlds?
1Chantiel21dThey're pretty much the same. If you could come up with a prior that would make the AI convinced it would be on Earth, then this could potentially make fix the problem. However, coming up with a prior probability distribution that guarantees the AI is in the nebulous concept of "Earth, as we imagine it" sounds very tough to come up with. Also, this could interfere with the reliability of the AI's reasoning. Thinking that it's guaranteed to be on Earth is just not a reasonable thing to think a priori. This irrationality may make the AI perform poorly in other ways. Still, it is a possible way to fix the issue.
2TekhneMakre21dWell, so "expressing how much you're going to try to optimize different worlds" sounds to me like it's equivalent to / interchangeable with a multiplicative factor in your utility function. Anyway, re/ the rest of your comment, my (off the cuff) proposal above was to let the AI be uncertain as to what exactly this "Earth" thing is, and to let it be *updateful* (rather than updateless) about information about what "Earth" means, and generally about information that clarifies the meaning of the utility function. So AIs that wake up on Earth will update that "the planet I'm on" means Earth, and will only care about Earth; AIs that wake up on e.g. Htrae will update that "the planet I'm on" is Htrae, and will not care about Earth. The Earth AI will not have already chosen a policy of S, since it doesn't in general chose policies updatelessly. This is analogous to how children imprint on lessons and values they get from their environment; they don't keep optimizing timelessly for all the ways they could have been, including ways that they now consider bad, even though they can optimize timelessly in other ways. One question would be, is this a bad thing to do? Relative to being updateless, it seems like caring less about other people, or refusing to bargain / coordinate to realize gains from trade with aliens. On the other hand, maybe it avoids killing us in the way you describe, which seems good. Otoh, maybe this is trying to renege on possible bargains with the Htrae people, and is therefore not in our best interests overall. Another question would be, is this stable under reflection? The usual argument is: if you're NOT updateless about some variable X (in this case X = "the planet I'm on (and am supposed to care about)"), then before you have resolved your uncertainty about X, you can realize gains from trade between possible future versions of yourself: by doing things that are very good according to [you who believes X=Htrae] but are slightly bad according to [
1Chantiel19dI think your semi-updateless idea is pretty interesting. The main issue I'm concerned about is finding a way to update on the things we want to have updated on, but not on the things we don't want updated on. As as example, consider Newcomb's problem. There are two boxes. A superintelligent predictor will put $1000 in one box and $10 in the other if it predicts you will only take one box. Otherwise it doesn't add money to either box. You see one is transparent and contains $1000. I'm concerned the semi-updateless agent would reason as follows: "Well, since their's money in the one box, their must be money in the other box. So, clearly that means this "Earth" thing I'm in is a place in which there is money in both boxes in front of me. I only care about how well I do in this "Earth" place, and clearly I'd do better if I got the money from the second box. So I'll two-box. But that's the wrong choice. Because agents who would two-box end up with $0.
1TekhneMakre17dOne intuitive way this case could work out, is if the SUDT could say "Ok, I'm in this Earth. And these Earthians consider themselves 'the same as' (or close enough) the alt-Earthians from the world where I'm actually inside a simulation that Omega is running to predict what I would do; so, though I'm only taking orders from these Earthians, I still want to act timelessly in this case". This might be sort of vacuous, since it's just referring back to the humans's intuitions about decision theory (what they consider "the same as" themselves) rather than actually using the AI to do the decision theory, or making the decision theory explicit. But at least it sort of uses some of the AI's intelligence to apply the humans's intuitions across more lines of hypothetical reasoning than the humans could do by themselves.
1TekhneMakre20dSomething seems pretty weird about all this reasoning though. For one thing, there's a sense that you sort of "travel backwards in logical time" as you think longer in normal time. Like, first you don't know about TDT, and then you invent TDT, and UDT, and then you can do UDT better. So you start making decisions in accordance with policies you'd've wanted to pick "a priori" (earlier in some kind of "time"). But like what's going on? We could say that UDT is convergent, as the only thing that's reflectively stable, or as the only kind of thing that can be pareto optimal in conflicts, or something like that. But how do we make sense of our actual reasoning before having invented UDT? Is the job of that reasoning not to invent UDT, but just to avoid avoiding adopting UDT?
1Chantiel18dI don't know how to formalize the reasoning process that goes into how we choose decision theories. And I doubt anyone does. Because if you could formalize the reasoning we use, then you could (indirectly) formalize decision theory itself as being, "whatever decision theory we would use given unlimited reflection". I don't really think UDT is necessarily reflectively stable, or the only decision theory that is. I've argued previously [] that I, in certain situations, would act essential as an evidential decision theorist. I'm not sure what others think of this, though, since no one actually ever replied to me. I don't think UDT is pareto optimal in conflicts. If the agent is in a conflict with an irrational agent, then the resulting interaction between the two agents could easily be non-pareto optimal. For example, imagine a UDT agent is in a conflict with the same payoff to the prisoner's dilemma. And suppose the agent it's in conflict with is a causal decision theorist. Then the causal decision theorist would defect no matter what the UDT agent would do, so the UDT agent would also defect, and then everyone would do poorly.
1TekhneMakre17dYeah I don't know of a clear case for those supposed properties of UDT. By pareto optimal I mean just, two UDT agents will pick a Pareto optimal policy. Whereas, say, two CDT agents may defect on each other in a PD. This isn't a proof, or even really a general argument, but one reason to suspect UDT is convergent, is that CDT would modify to be a sort of UDT-starting-now. At least, say you have a CDT agent, and further assume that it's capable of computing the causal consequences of all possible complete action-policies it could follow. This agent would replace itself with P-bot, the bot that follows policy P, where P is the one with the best causal consequences at the time of replacement. This is different from CDT: if Omega scans P-bot the next day, P-bot will win the Transparent Newcomb's problem, whereas if CDT hadn't self-modified to be P-bot and Omega had scanned CDT tomorrow, CDT would fail the TNP for the usual reason. So CDT is in conflict with its future self.
1Chantiel17dTwo UDT agents actually can potentially defect in prisoner's dilemma. See the agent simulates predictor problem [] if you're interested. But I think you're right that agents would generally modify themselves to more closely resemble UDT. Note, though, that the decision theory a CDT agent would modify itself to use wouldn't exactly be UDT. For example, suppose the causal decision theory agent had its output predicted by Omega for Newcomb's problem before the agent even came into existence. Then by the time the CDT agent comes to existence, modifying itself to use UDT would have no causal impact on the content of the boxes. So it wouldn't adopt UDT in this situation and would still two-box.
1TekhneMakre17dWell, the way the agent loses in ASP is by failing to be updateless about certain logical facts (what the predictor predicts). So from this perspective, it's a SemiUDT that does update whenever it learns logical facts, and this explains why it defects. > So it wouldn't adopt UDT in this situation and would still two-box. True, it's always [updateless, on everything after now].

I was wondering if there has been any work getting around specifying the "correct" decision theory by just using a more limited decision theory and adjusting terminal values to deal with this.

I think we might be able to get an agent that does what we want without formalizing the right decision theory buy instead making a modification to the value loading used. This way, even an AI with a simple, limited decision theory like evidential decision theory could make good choices.

I think that normally when considering value loading, people imagine finding a way... (read more)

3Vladimir_Nesov1moWhat is the difference between things you-on-reflection says being the definition of an agent's preference, and running a program that just performs whatever actions you-on-reflection tells it to perform, without the indirection of going through preference? The problem is that you-on-reflection is not immediately available, it takes planning and action to compute it, planning and action taken without the benefit of its guidance, thus by default catastrophically misaligned. So an AI with some decision theory but without further connection to human values might win the capability race by reassembling literally everything into a computer needed to answer the question of whether doing that was good. (It's still extremely unclear how to get even to that point, whatever decision theory is involved. In particular, this assumes that we can define you-on-reflection and thus we can define you, which is uploading. And what is "preference" specifically, so that it can be a result of a computation, usable by an agent in the real world?) The way an AI thinks about the world is also the way it might think about predictions of what you-on-reflection says, in order to get a sense of what to do in advance of having computed the results more precisely (and computing them precisely is probably pointless if a useful kind of prediction is possible). So the practical point of decision theory is deconfusion, figuring out how to accomplish things without resorting to an all-devouring black box.
2Chantiel1moOn reflection, there probably is not much difference. This is a good point. Still, an AI that just computes what you would want to it do, for example with approval-based AI or mimicry, also seems like a useful way of getting around specifying a decision theory. I haven't seen much discussion about the issues with the approach, so I'm interested in what problems could occur that using the right decision theory could solve, if any. True. Note, though, that you-on-reflection is not immediately available to an Ai with the correct decision theory, either. Whether your AI uses the right or wrong decision theory, it still takes effort to figure out what you-on-reflection would want. I don't see how this is a bigger problem for agents with primitive decision theories, though. One way to try to deal with this is to have your AI learn a reasonably accurate model of you-on-reflection before it becomes dangerously intelligent, so that way, once it does become superintelligent, it will (hopefully) work reasonably. And again, this works both with a primitive and sophisticated decision theory. Okay. I'm having a hard time thinking concretely about how concretely getting less confused about decision theory would help us, but I intuitively imagine it could help somehow. Do you know, more concretely, of the benefits of this deconfusion?
2Vladimir_Nesov23dRob Bensinger just posted a good summary with references [] on pragmatic motivations for working on things like decision theory.
2Vladimir_Nesov24dIt's a methodology for AI design, the way science is a methodology for engineering, a source of desiderata for what's important for various purposes. The activity of developing decision theories is itself like the thought experiments it uses, or like apparatus of experimental physics, a way of isolating some consideration from other confusing aspects and magnifying its effects to see more clearly. This teaches lessons that may eventually be used in the separate activity of engineering better devices.
2Vladimir_Nesov24dWell, there is a huge difference, it's just not in how the decisions of you-on-reflections get processed by some decision theory vs. repeated without change. The setup of you-on-reflection can be thought of as an algorithm, and the decisions or declared preferences are the results of its computation. Computation of an abstract algorithm doesn't automatically get to affect the real world, as it may fail to actually get carried out, so it has to be channeled [] by a process that takes place there. And for the purpose of channeling your decisions, a program that just runs your algorithm is no good, it won't survive AI x-risks (from other AIs, assuming the risks are not resolved), and so won't get to channel your decisions. On the other hand, a program that runs a sufficiently sane decision theory might be able to survive (including by destroying everything else potentially dangerous to its survival) and eventually get around to computing your decision and affecting the world with it.
1Chantiel22dWhen discussing the idea of a program implementing what you on reflection would do, I think we had different ideas in mind. What I meant was that every action the AI would take would be its best approximation of what you-on-reflection would want. This doesn't sound dangerous to me. I think that approval-based AI and iterated amplification with HCH would be two ways of making approximations to the output of you-on-reflection. And I don't think they're unworkably dangerous. If the AI is instead allowed to take arbitrarily many unaligned actions before taking the actions you'd recommend, then you are right in that that would be very dangerous. I think this was the idea you had in mind, but feel free to correct me. If we did misunderstand each other, I apologize. If not, then is there something I'm missing? I would think that a program that faithfully outputs some approximation of "what I'd want on reflection" on every action it takes would not perform devastatingly badly. I on reflection wouldn't want the world destroyed, so I don't think it would take actions that would destroy it.