All of sbenthall's Comments + Replies

This point about Ukrainian neo-Nazis is very misunderstood by the West.

During the Maidan revolution in Ukraine in 2014, neo-Nazi groups occupied government buildings and brought about a transition of government.

Why are there neo-Nazis in Ukraine? Because during WWII, the Nazis and the USSR were fighting over Ukraine. Ukraine is today quite ethnically diverse, and some of the 'western' Ukrainians who were resentful of USSR rule and, later, Russian influence, have reclaimed nazi ideas as part of a far-right Ukrainian nationalism. Some of these nazi groups th... (read more)

Thanks for writing this. I have been fretting for some time and realized that what I needed was a rational take on the war. I appreciate the time you've taken you write this out and I'll check out your other posts on this.

This seems correct to me. Thank you.

You don't know anything about how cars work?

I have a model of what inputs produce what outputs ("pressing on the gas pedal makes the engine go; not changing the oil every few months makes things break"). I do not have a causal model of the internals of the system. At best I can make understandish-sounding noises [] about engines, but I could not build or repair one, nor even identify all but the most obvious parts.

It's possible to predict the behavior of black boxes without knowing anything about their internal structure.


That says a lot more about your personal values then the general human condition.

I suppose you are right.

The models of worms might be a bit better at predicting worm behavior but they are not perfect.

They are significantly closer to being perfect than our models of humans. I think you are right in pointing out that where you draw the line is somewhat arbitrary. But the point is the variation on the continuum.

Do you think it is something external to the birds that make them migrate?

Norbert Wiener is where it all starts. This book has a lot of essays. It's interesting--he's talking about learning machines before "machine learning" was a household word, but envisioning it as electrical circuits.

I think that it's important to look inside the boxes. We know a lot about the mathematical limits of boxes which could help us understand whether and how they might go foom.

Thank you for introducing me to that Concrete Mathematics book. That looks cool.... (read more)

Thank you for the book. Just ordered it. You're welcome. It is the most fun math book I ever read. Currently it is just a bunch of PDEs on paper. But I really want to write a post on this as this could provide some mathematical footing for many of the fooming debates. One problem I'm stumbling with is the modelling of hard practical physical limits on computational processes. And I mean really practical limits that take thermodynamic into account, not these computronium bounds that are much too high. Something that takes entropic cost of replication and message transfer into account.

Do you think that rationalism is becoming a religion, or should become one?

"Religion" means too many different things. To a sociologist, religion is not just a creed, it's a social behavior; it's something people do, not only something they believe. People get together and do various things together, which they explain in various terms — a Zen Buddhist meditation session doesn't look very much like a High Church service, except that both involve a lot of people in a hall together.
Rationalism is a toolset with which to approach problems, not a belief system. If you had a functioning brain, you would know that.
Depends on what connotations of relgion you mean. But ensuring ulturally that the scientific process doesn't stop - and provides value by forming a religion-like commuity around it does sound like a viable way. Reminds me of the approach proposed for warning of radioactive waste dumps for 10000 years []:

Thanks. That criticism makes sense to me. You put the point very concretely.

What do you think of the use of optimization power in arguments about takeoff speed and x-risk?

Or do you have a different research agenda altogether?

As an informal concept, it's a good one (and better than "intelligence"). Just as long as its not taken too literally.

That makes sense. I'm surprised that I haven't found any explicit reference to that in the literature I've been looking at. Is that because it is considered to be implicitly understood?

One way to talk about optimization power, maybe, would be to consider a spectrum between unbounded, LaPlacean rationality and the dumbest things around. There seems to be a move away from this though, because it's too tied to notions of intelligence and doesn't look enough at outcomes?

It's this move that I find confusing.

There are people in my department who do work in this area. I can reach out and ask them.

I think Mechanical Turk gets used a lot for survey experiments because it has a built-in compensation mechanism and there are ways to ask questions in ways that filter people into precisely what you want.

I wouldn't dismiss Facebook ads so quickly. I bet there is a way to target mobile app developers on that.

My hunch is that like survey questions, sampling methods are going to need to be tuned case-by-case and patterns extracted inductively from that. Good social scientific experiment design is very hard. Standardizing it is a noble but difficult task.

Thanks. That's very helpful.

I've been thinking about Stuart Russell lately, which reminds me...bounded rationality. Isn't there a bunch of literature on that?

Have you ever looked into any connections there? Any luck with that?

You might say bounded rationality is our primary framework for thinking about AI agents, just like it is in AI textbooks like Russell & Norvig's. So that question sounds to me like it might sound to a biologist if she was asked whether her sub-area had any connections to that "Neo-Darwinism" thing. :)

1) This is an interesting approach. It looks very similar to the approach taken by the mid-20th century cybernetics movement--namely, modeling social and cognitive feedback processes with the metaphors of electrical engineering. Based on this response, you in particular might be interested in the history of that intellectual movement.

My problem with this approach is that it considers the optimization process as a black box. That seems particularly unhelpful when we are talking about the optimization process acting on itself as a cognitive process. It's eas... (read more)

Interesting. I know a bit about cybernetics but wasn't consciously aware of a clear analog between cognitive and electrical processes. Maybe I'm missing some background. Could you give a reference I could follow up on? That is a plausible interpretation. Fooming is actually the only valid interpretation given an ideal black-box AI modelled this way. We have to look into the box which is comparable to looking at non-ideal op-amps. Fooming (on human time-scales) may still be be possible, but to determine that we have to get a handle on the math going on inside the box(es). One could formulate discrete [] analogs to the continuous equations relating self-optimization steps. But I don't think this gains much as we are not interested in the specific efficiency of a specific optimization step. That wouldn't work anyway simply because the effect of each optimization step isn't known precisely, not even its timing. But maybe your proposal to use complexity results from combinatorial optimization theory for specific feedback types (between the optimization stages outlined by EY) could provide better approximations to possible speedups. Maybe we can approximate the black-box as a set of nested interrelated boxes.

Could you please link to examples of the kind of marketing studies that you are talking about? I'd especially like to see examples of those that you consider good vs. those you consider bad.

I did a poor job at the introduction. I'm assuming the studies exist, because if they don't that's full on false advertising. Not to pick on anyone in particular here are some I recently encountered: * ("Scientifically proven to measure you 20% more accurately than a professional tailor." - no details are provided on how this was measured, hard to believe claim, YC company) * Nightwave Sleep Assistant [] - list of effects, no source. * Basically anything in whole foods [] :p The probiotics section at wholefoods (and my interactions with customers who believed those claims or were skeptical of my claims given the state of the supplement market) was what finally caused me to post this thread. As a perplexing counterbalance to wholefoods are companies which don't advertise any effects whatsoever, even though you'd expect they would. List of companies where a lack of studies/objective claims caught my imagination: * & - these are huge companies doing science stuff. Why don't they have "opptimizely users make X% more revenue after 9 months" rather than testimonials? * The five companies I did customer development with (Beeminder, HabitRPG, Mealsquares, Complice, Apptimize)

I am confused. Shouldn't the questions depend on the content of the study being performed? Which would depend (very specifically) on the users/clients? Or am I missing something?

I am hopeful that at minimum we can create guidelines for selecting questions. I also think that some standardized health & safety questions by product category would be good (for nutritional supplements I would personally be interested in seeing data for nausea, diarrhea, weight change, stress/mood, and changes in sleep quality). For productivity solutions I'd be curious about effects on social relationships, and other changes in relaxation activities. Within a given product category, I'm also hopeful we can reuse a lot of questions. Soylent's test and Mealsquares' test shouldn't require significantly different questions.

I would worry about sampling bias due to selection based on, say, enjoying points.

For studies in which people have to actively involve themselves and consent to participate, I believe that there is always going to be some sampling bias. At best we can make it really really small, at worst, we should state clearly what we believe are those biases in our population. At worst, we will have a better understanding of what goes into the results. Also, for some studies, the sampled population might, by necessity, be a subset of the population.

The privacy issue here is interesting.

It makes sense to guarantee anonymity. Participants recruited personally by company founders may be otherwise unwilling to report honestly (for example). For health related studies, privacy is an issue for insurance reasons, etc.

However, for follow-up studies, it seems important to keep earlier records including personally identifiable information so as to prevent repeatedly sampling from the same population.

That would imply that your organization/system needs to have a data management system for securely storing the p... (read more)

He then takes whatever steps we decide on to locate participants.

Even if the group assignments are random, the prior step of participant sampling could lead to distorted effects. For example, the participants could be just the friends of the person who created the study who are willing to shill for it.

The studies would be more robust if your organization took on the responsibility of sampling itself. There is non-trivial scientific literature on the benefits and problems of using, for example, Mechanical Turk and Facebook ads for this kind of work. There is extra value added for the user/client here, which is that the participant sampling becomes a form of advertising.

Yeah, this is a brutal point. I wish I knew a good answer here. Is there a gold standard approach? Last I checked even the state of the art wasn't particularly good []. Facebook / Google / StumbleUpon ads sound promising in that they can be trivially automated, and if only ad respondents could sign up for the study, then the friend issue is moot. Facebook is the most interesting of those, because of the demographic control it gives. How bad is the bias? I performed a couple google scholar searches but didn't find anything satisfying. To make things more complicated, some companies will want to test highly targeted populations. For example, Apptimize is only suitable for mobile app developers -- and I don't see a facebook campaign working out very well for locating such people. A tentative solution might be having the company wishing to perform the test supply a list of websites they feel caters to good participants. This is even worse than facebook ads from a biasing perspective though. At minimum it sounds like disclosing how participants were located prominently will be important.

Thanks for your thoughtful response. I'm glad that I've been more comprehensible this time. Let me see if I can address the problems you raise:

1) Point taken that human freedom is important. In the background of my argument is a theory that human freedom has to do with the endogeneity of our own computational process. So, my intuitions about the role of efficiency and freedom are different from yours. One way of describing what I'm doing is trying to come up with a function that a supercontroller would use if it were to try to maximize human freedom. The i... (read more)

Thanks for your response! 1) Hmmm. OK, this is pretty counter-intuitive to me. 2) I'm not totally sure what you mean here. But, to give a concrete example, suppose that the most moral thing to do would be to tile the universe with very happy kittens (or something). CEV, as I understand, would create as many of these as possible, with its finite resources; whereas g/g* would try to create much more complicated structures than kittens. 3) Sorry, I don't think I was very clear. To clarify: once you've specified h, a superset of human essence, why would you apply the particular functions g/g to h? Why not just directly program in 'do not let h cease to exist'? g/g do get around the problem of specifying 'cease to exist', but this seems pretty insignificant compared to the difficulty of specifying h. And unlike with programming a supercontroller to preserve an entire superset of human essence, g/g* might wind up with the supercontroller focused on some parts of h that are not part of the human essence- so it doesn't completely solve the definition of 'cease to exist'. (You said above that h is an improvement because it is a superset of human essence. But we can equally program a supercontroller not to let a superset of human essence cease to exist, once we've specified said superset.)

I see, that's interesting. So you are saying that while the problem as scoped in §2 may take a function of arbitrary complexity, there is a constraint in the superintelligence problem I have missed, which is that the complexity of the objective function has certain computational limits.

I think this is only as extreme a problem as you say in a hard takeoff situation. In a slower takeoff situation, inaccuracies due to missing information could be corrected on-line as computational capacity grows. This is roughly business-as-usual for humanity---powerful enti... (read more)

On the one hand "close enough" is adequate for horseshoes, but probably not good enough for THE FATE OF THE UNIVERSE (grabs algebra nerd by lapels and shakes vigorously) On the other hand, supergeniuses like Ben Goertzel have suggested that a takeoff might follow a "semi-hard" trajectory. While others have suggested "firm takeoff" (Voss), and even "tumescent takeoff" Like most of humanity, I'll start getting concerned when the computers finally beat us in chess. ( camera whispering) On a more serious note, the superhuman AI that polices this site just had a most unwelcome message for me: You are trying to eject messages too fast. Rehydrate and try again in 3 minutes. The machines! ...they're takin' over! They're already layin' down the law!
While I would agree that not all portions of h are needed to solve the value problem, I think it's very plausible that it would take all of h to be certain that you'd solved the value problem. As in, you couldn't know that you had included everything important unless you knew that you had everything unimportant as well. Also, I don't think I'm sympathetic to the idea that a slow takeoff buys you time to correct things. How would you check for inaccuracies? You don't have a less-flawed version to compare things to; if you did, you'd be using that version. Some inaccuracies will be large and obvious, but that's rarely, if ever, going to catch the kinds of errors that lead to hyperexistential catastrophe, and will miss many existential catastrophes.

Could you flesh this out? I'm not familiar with key-stretching.

A pretty critical point is whether or not the hashed value is algorithmically random. The depth measure has the advantage of picking over all permissible starting conditions without having to run through each one. So it's not exactly analogous to a brute force attack. So for the moment I'm not convinced on this argument.

Thanks for your encouraging comments. They are much appreciated! I was concerned that following the last post with an improvement on it would be seen as redundant, so I'm glad that this process has your approval.

Regarding your first point:

  • Entropy is not depth. If you do something that increases entropy, then you actually reduce depth, because it is easier to get to what you have from an incompressible starting representation. In particular, the incompressible representation that matches the high-entropy representation you have created. So if you hold hum

... (read more)

Maybe. Can you provide an argument for that?

As stated, that wouldn't maximize g, since applying the hash function once and tiling would cap the universe at finite depth. Tiling doesn't make any sense.

I don't think it's literally tiling. More hash stretching all the way.

Your point about physical entropy is noted and a good one.

One reason to think that something like D(u/h) would pick out higher level features of reality is that h encodes those higher-level features. It may be possible to run a simulation of humanity on more efficient physical architecture. But unless that simulation is very close to what we've already got, it won't be selected by g.

You make an interesting point about the inefficiency of physics. I'm not sure what you mean by that exactly, and am not in a position of expertise to say otherwise. However, I ... (read more)

So, the key issue is whether or not the representations produced by the paperclip optimizer could have been produced by other processes. If there is another process that produces the paperclip-optimized representations more efficiently than going through the process of humanity, then that process dominates the calculation of D(r).

In other words, for this objection to make sense, it's not enough for the humanity to have been sufficient for the R scenario. It must be necessary for producing R, or at least necessary to result in it in the most efficient possible way.

What are your criteria for a more concrete model than what has been provided?

Do you know if there are literally entries for these outcomes on Should there be?

3slutbunwaller9y []

I think what the idea in the post does is that it gets at the curvature of the space, so to speak.

thanks I've been mispelling that for a while now. I stand corrected.

That is of course one of the questions on the table: who has the power to implement and promote different platforms.

I guess I disagree with this assessment of which problem is easier.

Humanity continues to exist while people stub their toes all the time. I.e., currently the humanity existing problem is close to solved, and the toe stubbing problem has by and large not been.

this is the sort of thing that gets assigned in seminars. Maybe 80% correct but ultimately weak sauce IMO

So there's some big problems of picking the right audience here. I've tried to make some headway into the community complaining about newsfeed algorithm curation (which interests me a lot, but may be more "political" than would interest you) here:

which is currently under review. It's a lot softer that would be ideal, but since I'm trying to convince these people to go from "algorithms, how complicated! Must be evil" to "oh, they ... (read more)

ethically motivated algorithmic curation.

Is that a polite expression for "propaganda via software"? Whose ethics are we talking about?


Re: Generality.

Yes, I agree a toy setup and a proof are needed here. In case it wasn't clear, my intentions with this post was to suss out if there was other related work out there already done (looks like there isn't) and then do some intuition pumping in preparation for a deeper formal effort, in which you are instrumental and for which I am grateful. If you would be interested in working with me on this in a more formal way, I'm very open to collaboration.

Regarding your specific case, I think we may both be confused about the math. I think you are right... (read more)

I'm still confused as to what D(u/h) means. It looks like it should refer to the number of logical steps you need to predict the state of the universe - exactly, or up to a certain precision - given only knowledge of human history up to a certain point. But then any event you can't predict without further information, such as the AI killing everyone using some astronomical phenomenon we didn't include in the definition of "human history", would have infinite or undefined D(u/h).

Non-catastrophic with respect to existence, not with respect to "human values." I'm leaving values out of the equation for now, focusing only on the problem of existence. If species suicide is on the table as something that might be what our morality ultimately points to, then this whole formulation of the problem has way deeper issues.

My point is that starting anew without taking into account the computational gains, you are increasing D(u) efficiently and D(u/h) inefficiently, which is not favored by the objective function.

If there's something ... (read more)

I feel that a good FAI designer shouldn't dismiss these objections so easily. The toe-stubbing problem is much simpler than preserving human existence. To solve the toe-stubbing problem, you just need to correctly order two outcomes A and B. To preserve human existence, you need to order outcome A against all possible outcomes B that a powerful creative intelligence might come up with. Any solution to the bigger problem that doesn't allow you to solve smaller problems is very suspicious to me.

Maybe this will be more helpful:

If the universe computes things that are not computational continuations of the human condition (which might include resolution to our moral quandaries, if that is in the cards), then it is, with respect to optimizing function g, wasting the perfectly good computational depth achieved by humanity so far. So, driving computation that is not somehow reflective of where humanity was already going is undesirable. The computational work that is favored is work that makes the most of what humanity was up to anyway.

To the extent t... (read more)

Re: your first point:

As I see it, there are two separate problems. One is preventing catastrophic destruction of humanity (Problem 1). The other is creating utopia (Problem 2). Objective functions that are satisficing with respect to Problem 1 may not be solutions to Problem 2. While as I read it the Yudkowsky post you linked to argues for prioritizing Problem 2, on the contrary my sense of the thrust of Bostrom's argument is that it's critical to solve Problem 1. Maybe you can tell me if I've misunderstood.

Without implicating human values, I'm claiming th... (read more)

General response: I think you should revise the chances of this working way downwards until you have some sort of toy model where you can actually prove, completely, with no "obvious" assumptions necessary, that this will preserve values or at least the existence of an agent in a world. But I think enough has been said about this already. Specific response: "Looks like noise" here means uncompressability, and thus logical shallowness. I'll try again to explain why I think that relative logical depth turns out to not look like human values at all, and you can tell me what you think. Consider an example. Imagine, if you will, logical depth relative to a long string of nearly-random digits, called the Ongoing Tricky Procession. This is the computational work needed to output a string from its simplest description, if our agent already knows the Ongoing Tricky Procession. On the other hand, boring old logical depth is the computational work needed to output a string from its simplest description period. The logical depth of the Ongoing Tricky Procession is not very big, even though it has a long description length. Now imagine a contest between two agents, Alice and Bob. Alice knows the Ongoing Tricky Procession, and wants to output a string of high logical depth (to other agents who know the Ongoing Tricky Procession). The caveat is Bob has to think that the string has low logical depth. Is this possible? The answer is yes. Alice and Bob are spies on opposite sides, and Alice is encrypting her deep message with a One Time Pad. Bob can't decrypt the message because, as every good spy knows, One Time Pads are super-duper secure, and thus Bob can't tell that Alice's message is actually logically deep. Even if the Ongoing Tricky Procession is not actually that K-complex, Alice can still hide a message in it - she just isn't allowed to give Bob a simple description that actually decomposes into the OTP and the message. This is almost the opposite of the Slow Growth

First, I'm grateful for this thoughtful engagement and pushback.

Let's call your second dystopia the Universal Chinese Turing Factory, since it's sort of a mash-up of the factory variant of Searle's Chinese Room argument and a universal Turing Machine.

I claim that the Universal Chinese Turing Factory, if put to some generic task like solving satisfiability puzzles, will not be favored by a supercontroller with the function I've specified.

Why? Because if we look at the representations computed by the Universal Chinese Turing Factory, they may be very logical... (read more)

Well, certainly I've been coming from the viewpoint that it doesn't capture these intuitions :P Human values are complicated (I'm assuming you've read the relevant posts here (e.g. [])), both in terms of their representation and in terms of how they are cashed out from their representation into preferences over world-states. Thus any solution that doesn't have very good evidence that it will satisfy human values, will very likely not do so (small target in a big space). I'd say, gains relative to what? When considering an enslaved Turing Factory full of people versus a happy Montessori School full of people, I don't see why there should be any predictable difference in their logical depth relative to fundamental physics. The Turing Factory only shows up as "simple" on a higher level of abstraction. If we restrict our search space to "a planet full of people doing something human-ish," and we cash out "logical depth relative to humanity" as operating on a high level of abstraction where this actually has a simple description, then the process seems dominated by whatever squeezes the most high-level-of-abstraction deep computation out of people without actually having a simple pattern in the lower level of abstraction of fundamental physics. Idea for generally breaking this: If we cash out the the "logical depth relative to humanity" as being in terms of fundamental physics, and allowing us to use a complete human blueprint, then we can use this to encode patterns in a human-shaped object that are simple and high-depth relative to the blueprint but look like noise relative to physics. If both "logical depth relative to humanity" and "logical depth" are on a high, humanish level of abstraction, one encodes high-depth computational results in slight changes to human cultural artifacts that look like noise relative to a high-level-of-abstraction description that doesn't have the blueprints for those artifacts.

I can try. This is new thinking for me, so tell me if this isn't convincing.

If a future is deep with respect to human progress so far, but not as deep with respect to all possible incompressible origins, then we are selecting for futures that in a sense make use of the computational gains of humanity.

These computational gains include such unique things as:

  • human DNA, which encodes our biological interests relative to the global ecosystem.

  • details, at unspecified depth, about the psychologies of human beings

  • political structures, sociological structures,

... (read more)
This is not obvious to me. I concur with Manfred's point that "any solution that doesn't have very good evidence that it will satisfy human values, will very likely not do so (small target in a big space)." Why couldn't they just scan everyone's brain then store the information in a big hard drive in a maximum-security facility while the robots wipe every living person out and start anew? Perhaps it's possible that by doing that you vastly increase resilience to exogenous shocks, making it preferable. And about 'using the computational gains of humanity', that could just as easily be achieved by doing the opposite of what humans would have done.

1) Thanks, that's encouraging feedback! I love logical depth as a complexity measure. I've been obsessed with it for years and it's nice to have company.

2) Yes, my claim is that Manfred's doomsday cases would have very high D(u) and would be penalized. That is the purpose of having that term in the formula.

I agree with your suspicion that our favorite future have relatively high D(u/h) / D(u) but not the highest value of D(u/h) / D(u). I suppose I'd defend a weaker claim, that a D(u/h) / D(u) supercontroller would not be an existential threat. One reason f... (read more)

Many utility functions have the same feature. For example, I could give the AI some flying robots with cameras, and teach it to count smiling people in the street by simple image recognition algorithms. That utility function would also assign a high score to our favorite future, but not the highest score. Of course the smile maximizer is one of LW's recurring nightmares [], like the paperclip maximizer. Any function that's computationally hard to optimize would have the same feature. What other nice features does your proposal have?

A good prediction :)

Logical depth is not entropy.

The function I've proposed is to maximize depth-of-universe-relative-to-humanity-divided-by-depth-of-universe.

Consider the decision to kill off people and overwrite them with a very fast SAT solver. That would surely increase depth-of-universe, which is in the denominator. I.e. increasing that value decreases the favorability of this outcome.

What increases the favorability of the outcome, in light of that function, are the computation of representations that take humanity as an input. You could imagine the s... (read more)

Ah, okay, my bad for just thinking of it as maximizing relative depth. So what's really pushed are things that are logically deep in their simplest expression in terms of humanity, but not logically deep in terms of fundamental physics. Depending on how this actually gets cashed out, the "human" that encodes deep computational results rather than actually living is still a very desirable object. Here's a slightly more alive dystopia: Use humanity to embody a complicated turing machine (like how the remark goes that chimpanzees are turing complete because they can be trained to operate a turing machine). Deep computational work appears to be being done (relative to humanity), but from a fundamental physics point of view it's nothing special. And it's probably not much fun for the humans enslaved as cogs in the machine.

Corporations exist, if they have any purpose at all, to maximize profit. So this presents a sort of dilemma: their diminishing returns and fragile existence suggest that either they do intend to maximize profit but just aren't that great at it; or they don't have even that purpose which is evolutionarily fit and which they are intended to by law, culture, and by their owners, in which case how can we consider them powerful at all or remotely similar to potential AIs etc?

Ok, let's recognize some diversity between corporations. There are lots of differen... (read more)

As far as I know, large corporations are almost as ephemeral as small corporations. Which tells you something about how valuable it is, and how ineffective each of the many ways is, no?

Yes, but Git has a bottleneck: there are humans in the loop, and there are no plans to remove or significantly modify those humans. By "in the loop", I mean humans are modifying Git, while Git is not modifying humans or itself.

I think I see what you mean, but I disagree.

First, I think timtyler makes a great point.

Second, the level of abstraction I'm talking about is that of the total organization. So, does the organization modify its human components, as it modifies its software component?

I'd say: yes. Suppose Git adds a new feature. Then t... (read more)

This seems to be quite similar to Robin Hanson's Ubertool argument. The problems with wetware are not that it's hard to change the hardware -- it's that there is very little that seems to be implemented in modifiable software. We can't change the algorithm our eyes use to assemble images (this might be useful to avoid autocrorecting typos). We can't save the stack when an interrupt comes in. We can't easily process slower in exchange for more working memory. We have have limits in how much we can self-monitor. Consider writing PHP code which manually generates SQL statements. It would be nice if we could remember to always escape our inputs to avoid SQL injection attacks. And a computer program could self-modify to do so. A human could try, but it is inevitable that they would on occasion forget (see Wordpress's history of security holes). We can't trivially copy our skills -- if you need two humans who can understand a codebase, it takes approximately twice as long as it takes for one. If you want some help on a project, you end up spending a ton of time explaining the problem to the next person. You can't just transfer your state over. None of these things are "software", in the sense of being modifiable. And they're all things that would let self-improvement happen more quickly, and that a computer could change. I should also mention that an AI with a FPGA could change its hardware. But I think this is a minor point; the flexibility of software is simply vastly higher than the flexibility of brains.

Ok, thanks for explaining that.

I think we agree that organizations recursively self-improve.

The remaining question is whether organizational cognitive enhancement is bounded significantly below that of an AI.

So far, most of the arguments I've encountered for why the bound on machine intelligence is much higher than human intelligence have to do with the physical differences between hardware and wetware).

I don't disagree with those arguments. What I've been trying to argue is that the cognitive processes of an organization are based on both hardware and ... (read more)

Ah, so you're concerned that an organization could solve the friendly AI problem, and then make it friendly to itself rather than humanity? That's conceivable, but there are a few reasons I'm not too concerned about it. Organizations are made mostly out of humans, and most of their agency goes through human agency, so there's a limit to how far an organization can pursue goals that are incompatible with the goals of the people comprising the organization. So at the very least, an organization could not intentionally produce an AGI that is unfriendly to the members of the team that produced the AGI. It is also conceivable that the team could make the AGI friendly to its members but not to the rest of humanity, but future utopia made perfect by AGI is about as far a concept as you can get, so most people will be idealistic about it.

They can't use one improvement to fuel another, they would have to come up with the next one independently

I disagree.

Suppose an organization has developers who work in-house on their issue tracking system (there are several that do--mostly software companies).

An issue tracking system is essentially a way for an organization to manage information flow about bugs, features, and patches to its own software. The issue tracker (as a running application) coordinates between developers and the source code itself (sometimes, its own source code).

Taken as a w... (read more)

Yes, it can fuel improvement. But not to the same level that an AGI that is foom-ing would. See this thread for details: [] I agree that organizations may be seen as similar to an AGI that has supra-human intelligence in many ways, but not in their ability to self modify.

should the word "corporation" in the first sentence be "[organization]"?

Yes, at least to be consistent with my attempt at de-politicizing the post :) I've corrected it. Thanks.

I wasn't sure what sort of posts were considered acceptable. I'm glad that particular examples have come up in the comments.

Do you think I should use particular examples in future posts? I could.

I think that as a general rule, specific examples and precise language always improve an argument.

I find this difficult to follow. Is there a concrete mathematical definition of 'recursion' in this sense available anywhere?

I've realized I didn't address your direct query:

(Aside: Is the theory of "communicative rationality" specified well enough that we can measure degrees of it, as we can with Bayesian rationality?)

Not yet. It's a qualitatively described theory. I think it's probably possible to render it into quantitative terms, but as far as I know it has not yet been done.

There are many reasons why the intelligence of AI+ greatly dwarfs that of human organizations; see Section 3.1 of the linked paper.

Since an organization's optimization power includes optimization power gained from information technology, I think that the "AI Advantages" in section 3.1 mostly apply just as well to organizations. Do you see an exception?

This sounds similar to a position of Robin Hanson addressed in Footnote 25 of the linked paper.

Ah, thanks for that. I think I see your point: rogue AI could kill everybody, whereas a domina... (read more)

Thanks for the quick reply. I agree that certain "organizations" can be very, very dangerous. That's one reason why we want to create AI...because we can use it to beat these organizations (as well as fix/greatly reduce many other problems in society). I hold that Unfriendly AI+ will be more dangerous, but, if these "organizations" are as dangerous as you say, you are correct that we should put some focus on them as well. If you have a better plan to stop them than creating Friendly AI, I'd be interested to hear it. The thing you might be missing is that AI is a positive factor in global risk as well, see Yudkowsky's relevant paper [].
Load More